Add tokenize unit tests#12

Open

cesaregarza wants to merge 1 commit intomainfrom

codex/create-and-test-tokenizer-with-mappings

Owner

cesaregarza commented May 17, 2025

Summary

test tokenize with and without preexisting mappings
ensure ability/weapon lists contain ints
verify <PAD> is appended last
check that weapon mappings using prefixes are handled correctly

Testing

poetry run pytest -q


          Add tokenize unit tests

929e1c5

cesaregarza added the codex label

— with

ChatGPT Codex Connector

Owner Author

cesaregarza commented May 18, 2025

merge conflict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels