Data format

[shgnag]

Hi,
I'm trying to run the code for a different dataset. By any chance can you please tell the format of the data (toy example) passed from the dataloader to the model.

Thanks

[fuankarion]

Hi, the code expects the data to be arranged in two directories (Audio, Video).

The audio directory contains the wav files associated with a visual tracklet. That is it contains the audio track from the very moment the tracklet becomes visible to the very last frame it is visible (Notice this creates some overlap in the audio files, but that is ok).

The video directory contains 1 sub-directory for every visual tracklet in the video. Inside those sub-directories you have just a bunch of facial crops.

Audio
-- Tracklet1.wav
-- Tracklet2.wav
...
-- Trackletn.wav

Video
-- Tracklet1
-- -- FaceCrop1.jpg
-- -- FaceCrop2.jpg
.. ..
-- -- FaceCropM.jpg

-- TrackletK
-- -- FaceCrop1.jpg
-- -- FaceCrop2.jpg
.. ..
-- -- FaceCropM.jpg

Adapting to a different dataset is not easy if you don't have the facial tracklets. But once you have them just arrange the audio and video in this way everything else kind of fits.
Don't forget to create a csv file that follows the structure of the original AVA files