This line outputs the following message below which might confuse the user to think that flores101 tokenizer is activated even when the tokenizer flores200 is selected. Better trigger the warning only when spm or flores101 tokenizers are selected
|
sacrelogger.warn("Tokenizer 'spm' has been changed to 'flores101', and may be removed in the future.") |
Outputted message
2022-10-24 12:27:01 | WARNING | sacrebleu | Tokenizer 'spm' has been changed to 'flores101', and may be removed in the future.