Unicode normalization?

[davidweichiang]

Currently no Unicode normalization (e.g., NFKC or NFKD) is done, so that (say) für and für would not count as a match. Would it be possible to add this or would it break too many things?

[ZJaume]

Had similar issues with special unicode symbols. May not be suitable for every scenario but for that I used --tok spm, as SentencePiece already does NFKC normalization by default. The inconvenience is that BLEU scores tend to be higher (there are more tokens) and you cannot compare to other BLEUs using default tokenization unless you recompute scores with spm.