feat: add Llama 3 tokenizer support #4
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add comprehensive support for Meta's Llama 3 family (3.0 through 3.3), including all official special tokens and extended agent vocabulary. Implementation: - Add LLAMA3_PATTERN constant and llama3.tiktoken vocabulary - Support "llama3", "llama3.1", "llama3.2", "llama3.3" in from_pretrained() - Add all Meta standard tokens (begin_of_text, eot_id, python_tag, etc.) - Implement 54 agent tokens starting at 128300 (avoiding Meta's reserved range) - Align <|image|> token with official Meta 3.2-Vision ID (128256) - Export LLAMA3_AGENT_TOKENS Python API with all token constants Documentation: - Add Llama 3 to supported vocabularies table - Document Meta's official tokens and chat format - Add comprehensive special_tokens.md section for Llama 3 - Include Python usage examples for both OpenAI and Llama 3 models - Document version-specific tokens (3.1 tool use, 3.2 vision) This maintains full compatibility with the Llama 3 token allocation while extending the vocabulary with splintr's agent tokens for chat, reasoning, tool calling, and RAG applications.
Add test coverage for cl100k, o200k, and llama3 tokenizers in both Rust and Python implementations. Rust tests (tests/): - Encode/decode roundtrip validation - Special token handling - Batch encoding operations - Streaming decoder functionality - Edge case handling - Agent token constant validation Python tests (python/tests/): - Mirror Rust test coverage - PyO3 binding validation - Cross-language consistency checks Also includes uv.lock for Python dependency management.
Update version across all configuration files and enhance pre-commit hook to automatically update uv.lock when pyproject.toml changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
<|image|>token with official Meta Llama 3.2-Vision ID (128256)Changes
LLAMA3_PATTERN, bundled vocabulary,LLAMA3_AGENT_TOKENSPython API