Add package tiktoken

[zlaazlaa]

Referenced b6d8f3c

• Closes tiktoken #828

[mhsmith]

Thanks very much. In order to merge this, you'll need to:

• Deal with the comments below.
• Add a basic test script, as described in server/pypi/README.md.
• Let me know which combinations of Python version and ABI you've tested. You don't need to do them all, but you should do at least one.

[mhsmith]

Based on the cryptography recipe, we can set this to the minimum supported version of Python, and then build it for any later version. So can we reduce this to Python 3.10 or even lower?

[zlaazlaa]

Based on the cryptography recipe, we can set this to the minimum supported version of Python, and then build it for any later version. So can we reduce this to Python 3.10 or even lower?

I tested the following compilation situation of tiktoken:

• Under Python 3.10, both version 0.8.0 and 0.9.0 failed to compile. It was not until version 0.7.0 that they were successfully compiled.
• The 0.9.0 version was successfully compiled under Python 3.12.

Do you think we should be using the old version of tiktoken (0.7.0) and abi3-py310 in the warehouse, or should we use the latest version of tiktoken (0.9.00) and abi3-py312 instead?

[mhsmith]

Let's stay on the latest version of tiktoken. For future reference, the errors when reducing the abi3 to py310 and building against Python 3.10 are apparently all to do with the buffer API. Here are the first couple:

error[E0412]: cannot find type `Py_buffer` in module `pyo3::ffi`
   --> src/py.rs:185:31
    |
185 |         view: *mut pyo3::ffi::Py_buffer,
    |                               ^^^^^^^^^ not found in `pyo3::ffi`

error[E0425]: cannot find value `PyBUF_WRITABLE` in module `pyo3::ffi`
   --> src/py.rs:191:32
    |
191 |         if (flags & pyo3::ffi::PyBUF_WRITABLE) == pyo3::ffi::PyBUF_WRITABLE {
    |                                ^^^^^^^^^^^^^^ not found in `pyo3::ffi`

However, Python 3.11 builds fine.

[zlaazlaa]

test passed:

2025-07-21 18:36:15.864  9665-26801 python.stdout           com.chaquo.python.pkgtest3           I  Encoded tokens for empty string: []
2025-07-21 18:36:15.864  9665-26801 python.stdout           com.chaquo.python.pkgtest3           I  Decoded text for empty string: 
2025-07-21 18:36:15.864  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  ok
2025-07-21 18:36:15.864  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  test_encode_decode (chaquopy.test.tiktoken.TestTiktoken) ... 
2025-07-21 18:36:15.864  9665-26801 python.stdout           com.chaquo.python.pkgtest3           I  Encoded tokens: [31373, 995]
2025-07-21 18:36:15.864  9665-26801 python.stdout           com.chaquo.python.pkgtest3           I  Decoded text: hello world
2025-07-21 18:36:15.864  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  ok
2025-07-21 18:36:15.864  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  test_token_count (chaquopy.test.tiktoken.TestTiktoken) ... 
2025-07-21 18:36:15.864  9665-26801 python.stdout           com.chaquo.python.pkgtest3           I  Token count: 4 Tokens: [11505, 20185, 12586, 0]
2025-07-21 18:36:15.864  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  ok
2025-07-21 18:36:15.865  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  
2025-07-21 18:36:15.865  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  ----------------------------------------------------------------------
2025-07-21 18:36:15.865  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  Ran 3 tests in 0.222s
2025-07-21 18:36:15.865  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  
2025-07-21 18:36:15.865  9665-26801 python.stderr           com.chaquo.python.pkgtest3           W  OK

[mhsmith]

The tests pass from Python 3.11 to 3.13 on all my ARM test devices and emulators, and on x86_64 and x86 with API level 24. But on x86_64 with API level 35, it hangs on the first test. This happens on Python 3.11 and 3.12, regardless of whether it's built against abi3-py311 or abi3-py312. I don't see anything relevant in the logs.

What ABI and API level did you test on?

It's possible this could be resolved by moving to a different version of tiktoken, but I'll leave it to you to try that.