A PHP implementation of OpenAI's BPE tokenizer tiktoken.
-
Updated
Jan 25, 2025 - PHP
#
special-tokens
Here are 2 public repositories matching this topic...
Custom BPE Tokenizer, trained for 1256 merges. Can be trained further, we can load the previously trained tokenizer and also there's an option to add or remove special tokens that you need.
-
Updated
Jun 23, 2025 - Python
Improve this page
Add a description, image, and links to the special-tokens topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the special-tokens topic, visit your repo's landing page and select "manage topics."