Bug + Proposal: ekg-embedding search fails on long Chinese notes due to naive truncation

[minkieyume]

Hi, I chose to use EKG because I was tired of the rigid hierarchical structure of traditional file systems. The categorization is too singular, making notes hard to retrieve, each note requires a manually defined title, and it's inconvenient for LLMs to flexibly query and access notes in real time.

However, I’ve encountered an issue with embeddings:
When a note's content is too long, using ekg-embedding-search sometimes fails to return the relevant note—even when the searched keyword exists verbatim within the text or even in the note’s tags.

I suspect the problem is related to the current truncation mechanism in ekg-embedding. The ekg-text-selector function appears to truncate long text based on English word delimiters, which might cause semantic loss in Chinese or improper segmentation of long Chinese passages for embedding.

Additionally, I’ve noticed that this relatively simple truncation strategy might be optimized for OpenAI models. For local models, compatibility issues may arise due to differences in tokenizer implementations—leading to inconsistent token counts.

While it is necessary to limit context length for LLMs, forcibly truncating text during vectorization could lead to contextual loss. Inspired by the LlamaIndex project, I’d like to propose the following improvements:

Allow users to define a custom tokenizer, and provide a Python-compatible tokenizer invocation module that lets users calculate token counts based on their model of choice.

Change the note processing logic: When a note exceeds the token limit, split it into chunks. Each chunk should retain metadata linking it to its original note. All chunks can then be embedded independently. This should help mitigate context loss and improve local retrieval reliability.

Support flexible text splitting methods, such as by paragraphs, Chinese/English punctuation marks, or even individual Chinese characters. When the resulting chunks still exceed the token limit, they could be further sliced using word spaces or character-based delimiters.

Include a small number of trailing words or characters from one chunk in the beginning of the next, to prevent semantic breakage. This overlap size should be a tunable parameter, with a suggested default of around 5–10 words or characters.

[ahyatt]

Yes, this is true - the llm library doesn't perform well on Asian languages without spaces. In addition, we don't estimate tokens correctly for all sorts of languages. There is ahyatt/llm#14 which is an idea to depend on an elisp tiktoken implementation, which probably would be an improvement even for non-Open AI models.

But ekg itself, as you point out, also depends on word boundaries all over the place. I think a variable that is a custom truncation function would be useful. Having different methods such as based on punctionation should be good. But remember that it's just one embedding per note, so it's not really splitting, and there's no multiple chunks to include text from.

Notes should ideally have multiple chunks, each with an embedding, for shorter parts of text, all decontextualized. That's a significant effort, though, one I haven't yet started on. It may not be possible due to limitation in embedding search efficiency in sqlite at the moment.

[minkieyume]

Personally, I think providing a configuration option that allows users to use an external vector database, such as Chroma or Qdrant, might help solve the bottleneck issue with SQLite in vector search. I'm not sure if there's any existing discussion or implementation on this.

[ahyatt]

I like your idea; let me experiment with it! It might actually be a reasonable idea for a new library.