Train on custom dataset

[ahsan856jalal]

Hi @ehcalabres @melkilin can you please guide how to train model on a custom data which is similar to cub-100 data format , folders of classes format.,

[melkilin]

Thanks for your interest in our work, @ahsan856jalal . We'll get back to you ASAP.

[IoSonoMarco]

hi @ahsan856jalal, for any custom dataset we recommend following the template code in cub200_FSCIL.py or miniimagenet_FSCIL.py (avoid referring to the CIFAR template). We don't provide code for making the splits, it's up to you. If you want to replicate exactly our templates without making any changes and run our code strictly as it is, we recommend organizing your data as it is shown in cpe-clip/splits .

1. Create the <your_dataset_name> folder and put it into the splits folder.
2. Populate the <your_dataset_name> folder with just class_to_id.pkl, bass_class.pkl, and exp[n].pkl files for the n experiences. Pickle experience files (and base_class) contain train and test dictionaries with image paths, labels, and category names (as you can see in the split folder examples you can download).
3. Create a <your_dataset_name>_FSCIL.py and put it in datasets module .

Note: you don't have to use pickle format necessarily, also pandas dataframes are ok. The important thing is that you specify image paths, labels and category names for every experience. If you want to use panda dataframes remember to modify the loading procedure in your <your_dataset_name>_FSCIL.py accordingly.

[ahsan856jalal]

@IoSonoMarco thank you , I am able to run your model on a custom data, fortunately I did the same steps three days ago as you suggested :)
Just one question , how can I save and load model and then pass a test image to get inference, is there any snippet for that

[brian-mann-math]

Do you have any tips to use your method for datasets with very large numbers of classes? My use case has around 9000 base classes and I'm running out of GPU memory (23GB on an AWS G5 instance).