SimDino

[daniel-z-kaplan]

Acceptance criteria: Implement SimDINO into our current codebase.
At this time, that would adding a flag that replaces our current setup with SimDino.

An interested future step would be seeing if there's value in combining SimDino with regular dino (kinda unlikely, but interesting exploration!)
https://robinwu218.github.io/SimDINO/

[Dojo2024]

I've gone through the SimDINO paper and am very interested in taking on this implementation task.
My initial thought is that this would involve creating a new SimDINO model class, similar to the DINO class that is already built, which would be containing the student and teacher networks. The key change would be in the training engine, where we would need to implement the three specific loss terms that is mentioned at the paper.
I'd be happy to start working on a draft PR for this. Please let me know if this is the right direction or if there are any other details I should consider.

[daniel-z-kaplan]

Yes, I think that's approximately the approach we are looking at.