Fine-Tuning SONAR

[stamatisvas]

Hi,
I’m currently working with the SONAR embeddings and I'm interested in fine-tuning the model/text encoder on my custom dataset (pair of sentences) for tasks like classification and semantic similarity. Is there any documentation or guidance on how to fine-tune SONAR for these purposes?

[avidale]

Hi @stamatisvas!
SONAR encoders (as well as any other fairseq2 models) are PyTorch modules, and thus compatible with any training framework you prefer (e.g. PyTorch Lightning, Accelerate, or recently released fairseq2 trainer. Alternatively, you can write a training loop in pure pytorch, without any frameworks.

Any specific details of the fine-tuning would depend on the task, language and dataset for which you are fine-tuning the model.

[Sibgat-Ul]

Hello @avidale, i was thinking of doing some experiments (distilling to a smaller version of sonar embedding), retrain the last layers of the encoders (and maybe the decoder also) for a certain task. I mostly needed help with training, as i am not sure how do i load the encoders and only train the last layer, and use in a teacher-student framework. Can you help me with the process?

Thanks

[mauro-belgiovine]

Hi, any guidance on this? I am also interested in fine-tuning the decoder. @avidale any chance you could provide a minimal code example in pure pytorch? Thanks!

[yhamidullah]

Hi @mauro-belgiovine If you want a pure pytorch + torch no fairseq, we have shared ours here https://huggingface.co/mtmlt/sonar-nllb-200-1.3B. You can train it using regular pytorch or HF trainer.

[mauro-belgiovine]

@yhamidullah thanks, although it seems the link you shared is broken?

[yhamidullah]

@mauro-belgiovine It is not broken, we just put it private since Monday. Due to an ongoing paper submission, because of review policy. We will put it back to public soon.

[avidale]

@mauro-belgiovine @Sibgat-Ul I added a notebook in #70 with an example of fine-tuning SONAR text encoder with a classifier head. I hope it would help, at least as an inspiring example!

Later, we could add examples of other training task (e.g. teacher-student training of encoders).

[mauro-belgiovine]

Thanks @avidale ! This is super helpful, I will take a look and let you know if there's any questions/comments.

[Sibgat-Ul]

thanks alot @avidale that will help a ton.

[Sibgat-Ul]

Thanks guys, I had to ask that is there any straight forward way to get the logits for every steps easily?

Thanks.

[yhamidullah]

@Sibgat-Ul you can check here if it is what you want. https://huggingface.co/mtmlt/sonar-nllb-200-1.3B. It's now open.

[Sibgat-Ul]

yes thanks alot this will save lives lol. Seriously, thanks. @yhamidullah