New OutlineTextSplitter class

[Stevenic]

One of the thing that drives me batty about RAG is that it often retrieves chunks that contain partial lists. It'll retrieve a chunk that contains Step 2 and 3 of a task but completely drop Step 1 and 4. My Document Sections is designed to improve that situation by at least keeping the order of steps correct but it doesn't solve for dropped steps and list items. The core issue is that many times these missing steps are in in adjacent chunks that aren't semantically relevant to the query.

In an effort to solve this issue, I'm working with my tool (GPT-5.2) to design a new OutlineTextSplitter class. The idea is to break text splitting into a 2 phase problem. You first create an outline that identifies the structure of the document you want to split, ignoring any token counts, and then you split the document based on the outline, not delimiters. This should result in a better chance of a sequence of steps landing in the same chunk. The Document Sections algorithm will then do the rest of the work.

Bellow is a partial discussion with my tool thinking through the problem:

[Stevenic]

what if we did this... we build a new TextSplitter class that makes two passes. In pass 1 it identifies the structure of a document by generating an outline (ignoring token counts.) It then makes a second pass where it does the actual splitting, using the outline as a guide

Great idea. A two-pass splitter can preserve structure first, then enforce token budgets, which should yield more coherent chunks than a single recursive pass.

Proposed design: Outline-aware two-pass TextSplitter

Pass 1: Build a structural outline (no tokenization)

• Goal: Identify document blocks and their character spans (startPos/endPos) without considering token counts.
• OutlineNode shape: { type, startPos, endPos, children[], meta }.
• Detectors by docType (guided by current separator knowledge):
  • Markdown: headings (##..######), fenced code blocks, lists (bulleted/numbered), tables, paragraphs. The existing markdown-oriented separators are a good starting list of boundaries to scan for. [1]
  • HTML: tags like <h1..h6>, <p>, <li>, <ul>/<ol>, <div>, <table>, etc. The current html separators list provides the tags to anchor. [1]
  • Code: classes, functions/methods, control-flow beginnings, and explicit region markers (e.g., // LLM-REGION). The existing per-language separator lists already encode these landmarks. [1]
• How to build it:
  • Single linear scan of the text, recognizing block starts/ends using docType-specific patterns and simple state machines (e.g., in-fence for ``` blocks, in-list while consecutive list-item lines persist).
  • Produce a shallow tree: sections → subsections → blocks (paragraph/list/code/table/etc.). Each node records exact character offsets, preserving the original text.
  • Keep this pass tokenizer-free to stay fast and match your idea.

Pass 2: Token-budgeted packing guided by the outline

• Inputs: outline nodes, chunkSize, chunkOverlap, tokenizer (defaults to GPT-3-style). [1]
• Packing policy:
  • Treat nodes as atomic when possible. Greedily pack whole nodes into a chunk until adding the next would exceed chunkSize (by tokens). If so, flush the current chunk and continue.
  • Overlong single node: recursively split inside the node using its internal structure:
    • Markdown list → split by list items; only split inside an item if it still exceeds chunkSize, then fall back to sentence/space splitting for that item.
    • Paragraph → split by sentences, then words/spaces as last resort (akin to the current space-based fallback). [1]
    • Code block → split by inner function/statement boundaries if detectable; otherwise by newlines, then halves as absolute fallback (mirrors current “halve when no separators remain”). [1]
    • HTML → split by child tags, then text nodes, then spaces.
• Overlap handling:
  • After emitting the ordered chunk list, attach startOverlap/endOverlap token slices exactly like the current implementation (copy tail/head tokens from neighbors up to chunkOverlap). This preserves cross-boundary context without altering chunk text. [1]
• Combining:
  • Because packing already respects chunkSize, skip the legacy combineChunks step for two-pass mode to avoid inserting spaces that could disturb formatting (lists/tables). In single-pass fallback, keep using combineChunks as today. [1]
• Tokenization strategy:
  • Tokenize only the node text you’re considering for the current chunk. Cache token counts per node slice to avoid re-encoding when nodes are reconsidered. The existing tokenizer interface is reused; default remains GPT3Tokenizer. [1]

API and integration

• New class OutlineTextSplitter implements the same split(text): TextChunk[] contract so LocalDocumentIndex and downstream code remain unchanged. Chunks still contain text, tokens, startPos/endPos, and overlaps. [1]
• Config additions (non-breaking):
  • twoPass: boolean (default true for docTypes with strong structure like md/html/code).
  • atomicNodeTypes: string[] (e.g., ["listItem", "tableRow", "function"]).
  • fallbackSeparators: string[] (inherits from getSeparators(docType) to echo today’s behavior when structure is unknown). [1]
  • keepSeparators: honored as today, but typically off for two-pass chunks so text remains as-is. [1]
  • chunkSize, chunkOverlap, tokenizer remain as-is with the same validations. [1]

Algorithm sketch

Pass 1 (outline):

• outline = []
• scan text using docType detectors (derived from getSeparators and small parsers for lists/code fences/tables)
• emit nodes with startPos/endPos; nest where appropriate (e.g., headings contain lists/paragraphs)

Pass 2 (packing):

• chunks = []
• cur = empty, curTokens = 0
• for node in outline in document order:
  • t = tokens(node) [cached]
  • if curTokens + t <= chunkSize: append node to cur; curTokens += t
  • else:
    • if cur not empty: flush cur to chunks
    • if t <= chunkSize: start new cur with node
    • else: for each slice in splitNodeToFit(node): emit slices as chunks or pack slices greedily
• after loop: flush cur
• attach startOverlap/endOverlap tokens between adjacent chunks as in current splitter [1]

Edge cases and fallbacks

• Unknown or low-structure documents: if outline yields a single flat span, fall back to today’s recursive separator-based splitting so behavior remains predictable. [1]
• Extremely large single blocks: descend to sentence/space splitting, then the existing “cut in half” safety valve when no lower separators exist. [1]
• Formatting preservation: since chunks are contiguous spans by character offsets, list bullets, numbering, and code indentation stay intact.

Why this helps

• Semantic coherence: whole list items, functions, or paragraphs stay together, improving retrieval quality compared to newline-first recursion. [1]
• Efficiency: one structural pass over characters plus targeted tokenization is often cheaper than repeatedly re-encoding during deep recursion. [1]
• Compatibility: same TextChunk shape, same overlap semantics, same tokenizer defaults/limits. [1]

Migration plan

• Implement OutlineTextSplitter alongside the current TextSplitter.
• Gate via config (twoPass=true) and docType allowlist (md, html, code families).
• Add unit tests: lists, nested headings, big code blocks, hybrid prose+lists, HTML with nested lists/tables.
• Benchmark against current splitter on token accuracy, speed, and retrieval metrics.

If you want, I can draft concrete detectors for md lists/code fences and a splitNodeToFit implementation to drop into your codebase. [1]

[Stevenic]

One note about code... Comments are often above the class/type/method/function they define. It's always bugged me that these got moved to an adjacent chunk. Ideally the comment would be in the same chunk as the thing its documenting.

[Stevenic]

A related issue I've seen is that super long documents with little structure but a lot of text can take for every to split... like minutes... Here's a good example:

https://en.wikipedia.org/wiki/Taylor_Swift

If you just use a regular tokenizer it can take on the order of 5 minutes to chunk that article. It's because you have to do a ton of recursive tokenization's as your trying to work your way down to your desired chunk size. I solved that in my other project but creating a FastTokenizer class...

The FastTokenizer class lets you pass in a target chunk size. It then looks at the character length of the text being tokenized and estimates the token count by multiplying that length by 0.75. If that value is more than 110% of the target chunk size it lies and returns the estimated token length, This dropped my text splitting for long docs down from minutes to under 1 second. I can now tokenize several GB of text in just a couple of minutes...