Support for other models via Hugging Face tokenizers#9

simonw wants to merge 2 commits intomainsimonw/ttok:mainfrom

other-tokenizerssimonw/ttok:other-tokenizersCopy head branch name to clipboard