Utility to calculate LLM context length of an arbitrary string.
Based on https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them.
str_to_tokens(str: str) -> floatstr_to_processing_tokens(str: str) -> intsurfly/llm-context-length
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|