Python script for manipulating the existing tokenizer.
The solution was tested on Llama3-8B tokenizer.
Installation from PyPI:
pip install tokenizerchanger- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4
1kkiRen/Tokenizer-Changer
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|