forked from openai/tiktoken
-
Notifications
You must be signed in to change notification settings - Fork 71
Open
Description
This is a pretty cool library! I'm doing something like this:
while (enc.encode(raw).length > length) {
raw = raw.slice(0, -1000)
}
to truncate big data, however this is quite slow (running this 100 times can take like 1-2 seconds).
Is there any better way to do this or can I optimize this somehow?
Metadata
Metadata
Assignees
Labels
No labels