Skip to content

[Feature][WIP] Implement KoLeoLoss for DINOv2/v3#367

Open
d3tk wants to merge 1 commit intogalilai-group:mainfrom
d3tk:feature/koleo
Open

[Feature][WIP] Implement KoLeoLoss for DINOv2/v3#367
d3tk wants to merge 1 commit intogalilai-group:mainfrom
d3tk:feature/koleo

Conversation

@d3tk
Copy link

@d3tk d3tk commented Dec 7, 2025

Description

This pull request implements the KoLeo regulrizer from Sablayrolles et al., 2019 as a loss regularization for DINOv2 as described in Oquab et al., 2024.

This resolves #319

I referenced the KoLeo Loss implementations from DINOv2 and DINOv3 GitHubs

Changes

  • Added KoLeoLoss class in stable_pretraining/losses/dino.py:
  • Implements the efficient version using pairwise dot products (torch.mm) and the diagonal filling trick.

Updated DINOv2Loss:

  • Added koleo_loss_weight argument (default 0.0 to maintain backward compatibility).
  • Updated forward to accept student_cls_features and compute KoLeo loss if enabled.

Added Tests:

  • Created stable_pretraining/tests/unit/test_koleo_loss.py.
  • Includes unit tests for initialization, forward pass (shapes, basic logic), and integration with DINOv2Loss.

Checklist

  • I have read the Contributing document.
  • The documentation is up-to-date with the changes I made (check build artifacts).
  • All tests passed, and additional code has been covered with new tests.
  • I have added the PR to the RELEASES.rst file.

@d3tk d3tk changed the title Implement KoLeoLoss for DINOv2/v3 [Feature] Implement KoLeoLoss for DINOv2/v3 Dec 7, 2025
@d3tk d3tk changed the title [Feature] Implement KoLeoLoss for DINOv2/v3 [Feature][WIP] Implement KoLeoLoss for DINOv2/v3 Dec 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add KoLeo Regularization Loss for DINOv2/v3

1 participant