investigate why 1K supervised evaluation shows as high accuracy as full training set

[taeil]

• Supervised s1/s2 (1024)
• Supervised s1/s2 (full)
• Moco s1/s2 (1024)

[taeil]

rerunning supervised training/eval on s1, s2, s1/s2 to make sure it was not just recording issue.

[taeil]

rerun the following supervised training to make sure we can reproduce. So far, cannot reproduce.

https://wandb.ai/cal-capstone/SEN12MS/runs/rqn33k8c/overview

[taeil]

rerunning supervised training/eval on s1, s2, s1/s2 to make sure it was not just recording issue.

reproduced the high accuracy.

[taeil]

retrying on smaller dataset (512) for training both supervised and pretraining (visionary).

[suryagutta]

We can try the following:

1. 1024 images from the original training and not from the 35k(removed overlap images)
2. use 100 epochs

Update: It seems that the current train_list_1024.pkl is written from the train examples only and not from 35k. Then option 1 is invalid.

[suryagutta]

I was able to reproduce the issue. Performance was 60%.
After using the same hyperparameters as the full pre-trained model provided by SEN12MS, I saw a decrease in performance. Step logic that we added is adding a performance improvement by ~3.5 percent. If we don't use that and use the same hyperparameters as SEN12MS, we see that there is a decrease in performance for 1024 samples. Also, as expected, S1 < S2 < S1S2.