Unstable training of the OmniLearned-medium models

[gregorkrz]

Loss sometimes jumps to >10 (from O(1))
Doesn't happend with regression, only classification

EDIT: seems to also happen with regression sometimes, just less frequently. maybe a good idea would be to freeze the backbone and only train the heads for OL-medium