Finetuning Speech Encoders further

[King-Rafat]

Hi,

I tried finetuning the Swahili speech encoder but the performance only increases to 9.6 BLEU from a base BLEU score of 7.5 on your already finetuned encoder. I finetuned the speech encoder for 5 epochs with augmented data. I am not willing to try more epochs as the performance increase is not I had imagined. I finetuned with about 30hrs of data. The MSE loss in the last epoch was 1.5*10^-6. Any different approach that might help achieve a better BLEU?

Also, what is the finetuned decoder model checkpoint that I read in the paper does well for Swahili? When I try to use it I get the error - ValueError: The input sequence length must be less than or equal to the maximum sequence length (512), but is 513 instead which I do not get for the normal decoder. All my audios are less than or equal to 30 sec.

Thank you for your time!

[avidale]

Hi @King-Rafat!

Any different approach that might help achieve a better BLEU?

If you need the best model for automatic speech recognition, using SONAR is not advised: because of the sentence-level representation bottleneck, it is going to be less accurate than encoder-decoder models without bottleneck (e.g. SeamlessM4T or Whisper) and maybe even than CTC models (e.g. MMS).

If you want specifically to improve speech SONAR encoder, the bottleneck may be in the student speech encoder itself underfitting to the data, or the text teacher encoder producing embeddings for your language that are not accurate enough; hard to tell without looking at the data.

When I try to use it I get the error - ValueError: The input sequence length must be less than or equal to the maximum sequence length (512), but is 513 instead which I do not get for the normal decoder

Could you please share the code that reproduces the error? We'll try to reproduce and fix it!.