Cloudflare worker Uncaught ReferenceError: FinalizationRegistry is not defined

[TranquilMarmot]

This is a duplicate of #46 but that's closed, and I don't think it should be 😄

Following the instructions here in the README:

https://github.com/dqbd/tiktoken?tab=readme-ov-file#cloudflare-workers

When you try to deploy the function to Cloudflare, it will fail with:

Uncaught ReferenceError: FinalizationRegistry is not defined

It's happening at:

node_modules/.pnpm/tiktoken@1.0.13/node_modules/tiktoken/lite/tiktoken_bg.js:167:30

Using js-tiktoken here isn't an option since the package for a Cloudflare worker has to be < 1MiB (1024 KiB) after gzip, so WASM is really the only route to go if you want to run this in a Cloudflare worker.

Note; you can use gpt-tokenizer in a Cloudflare worker with something like:

const myFunction = async () => {
   const { encode, decode } = await import("gpt-tokenizer");
   // ... can use functions here, but the overall bundle size is still small
}

It seems like the Cloudflare worker instructions should be removed from the README for now, or at the very least the ✅ should be changed to a ❌.

[RBSUS]

I came across this today. It's preventing me from deploying my NodeJs API. As far as I'm aware, I didn't update this package and yet the error has come about out of thin air. Therefore, this may be something to do with the Google Cloud Functions environment in which I'm running the api. Currently assessing older versions of this package to see if the error persists.