Skip to content

Remove the flores101 related tokenizer logging message when the selected tokenizer is flores200  #219

@hadyelsahar

Description

@hadyelsahar

This line outputs the following message below which might confuse the user to think that flores101 tokenizer is activated even when the tokenizer flores200 is selected. Better trigger the warning only when spm or flores101 tokenizers are selected

sacrelogger.warn("Tokenizer 'spm' has been changed to 'flores101', and may be removed in the future.")

Outputted message

2022-10-24 12:27:01 | WARNING | sacrebleu | Tokenizer 'spm' has been changed to 'flores101', and may be removed in the future.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions