-
Notifications
You must be signed in to change notification settings - Fork 0
Custom BPE Tokenizer, trained for 1256 merges. Can be trained further, we can load the previously trained tokenizer and also there's an option to add or remove special tokens that you need.
aj-arson/BPE_Tokenizer
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Custom BPE Tokenizer, trained for 1256 merges. Can be trained further, we can load the previously trained tokenizer and also there's an option to add or remove special tokens that you need.
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published