CMC for small custom dataset

[HaoliangZhou]

Thanks for open-sourcing your work, I have been trying to use CMC on my custom toy dataset which has 2 views (RGB, Depth). The number of RGB and Depth images is 4000. I have 2 dataloader, one for RGB data and another for depth. And use corresponding RGB and depth to be positive pair.
When I use NCECriterion, the reslut is like,

If use Softmax-CE, the result becomes,

Could you please give me some suggestion about train with dual-modal datasets or small datasets ?

Thanks a lot!