Request for tokenizer

[ruoyxue]

Thanks for the perfect work!
I am working on a new English lip reading dataset, could you please publish the tokenizer model and token list in this work?

[ahaliassos]

Hi, you can have a look at https://github.com/ahaliassos/usr/blob/main/utils/labels/unigram1000_units.txt for the vocabulary and at https://github.com/ahaliassos/usr/blob/main/utils/utils.py#L6 for how the final token list is obtained (i.e., with the blank and eos tokens).

Hope this helps!