Phase A: The Sharpest Code Reader — Minimal Mode, Token Budget, AST Truncation & New Languages

[dean0x]

Phase A: The Sharpest Code Reader

Parent: #17 (Project Horizon)
Priority: Highest
Goal: Make skim the most useful code reading tool an AI agent can reach for — not via hooks, but by being so good that agents and humans actively choose it.

Strategic Rationale

Modern AI coding agents (Claude Code, Cursor, Windsurf) already have intelligent built-in file reading — they support offsets, limits, and selective reading. Hooking cat adds marginal value since agents rarely use raw cat as their primary file read path.

The real opportunity: features that no agent's built-in Read tool can replicate — AST-aware truncation, token budget optimization, and semantic code compression. These make skim the tool agents choose when they need to understand code efficiently.

Hooks become valuable in Phase B when we add command output compression (test runners, git, builds) — that's where agents have no built-in intelligence.

Deliverables

1. ✅ --mode=minimal — Light-Touch Filtering (PR #22)

Fill the gap between full (no transformation) and structure (bodies stripped).

• Strip single-line comments (preserve doc comments: ///, /**, #[doc)
• Strip block comments (preserve Python docstrings """)
• Normalize excessive blank lines (3+ consecutive → 2)
• Keep all function bodies intact
• Language-aware comment pattern detection via tree-sitter node types
• Expected savings: 15-30% token reduction on typical source files
• Default mode when minimal context cleanup is needed

2. --max-lines N — AST-Aware Smart Truncation

Unique differentiator. No other tool truncates at AST boundaries.

• Given a line budget, select the most informative lines
• Priority hierarchy: type definitions > function signatures > imports > structural braces > body lines
• Cut at AST node boundaries — never mid-expression, never mid-function-signature
• Insert // ... N lines omitted markers at truncation points
• Composable with any mode: skim file.rs --mode=structure --max-lines 50
• Handles files shorter than N lines gracefully (no-op)

3. --tokens N — Token Budget Mode

Novel capability — no existing tool does this. Pull forward from Phase D.

• skim file.rs --tokens 2000 — fit the most informative view into exactly N tokens
• Uses tiktoken (cl100k_base) for accurate token measurement
• Algorithm: start with structure mode, progressively add detail until budget is reached
  • Step 1: All type definitions + function signatures (minimum useful view)
  • Step 2: Add import statements
  • Step 3: Add doc comments on public items
  • Step 4: Add short function bodies (< 5 lines)
  • Step 5: Add remaining bodies by size (smallest first)
• Output always within budget (never exceed, may underuse by <5%)
• Use case: agent says "I have 4K tokens of context budget, give me what fits"
• Works with multi-file: skim 'src/**/*.rs' --tokens 8000 — budget across all files

4. New Language Support — C/C++ and TOML

More languages = more useful. Low effort, high visibility.

• C/C++ support via tree-sitter-c and tree-sitter-cpp (feat: Add C/C++ language support #10)
• TOML support — evaluate tree-sitter-toml vs serde_toml approach (feat: Add TOML language support #8)
• Extensions: .c, .h, .cpp, .cc, .cxx, .hpp, .hxx, .toml
• Full test fixtures for each (minimum 4 per language per CLAUDE.md)
• Update README language table
• Brings total to 11-12 supported languages (good for launch optics)

5. Stdin & Pipe Improvements

Agents pipe things. Skim should be the best pipe-friendly code compressor.

• git diff | skim --lang rust --mode=structure — works perfectly
• git show HEAD:file.rs | skim --mode=signatures — language auto-detected from filename hint
• --lang flag for explicit language override when piping (no file extension to detect from)
• Ensure BufWriter streaming works correctly for large piped inputs
• No regression on existing stdin support

Sequence

1. --mode=minimal              ← Foundation (needed by other features)
2. --max-lines N               ← Quick differentiator
3. --tokens N                  ← Novel capability, killer feature
4. C/C++ + TOML languages      ← Parallelizable, independent work
5. Stdin/pipe improvements     ← Polish

Acceptance Criteria

• All existing 151 tests still pass
• New tests for minimal mode, max-lines, token budget, and new languages
• --mode=minimal achieves 15-30% reduction on test fixtures
• --max-lines never cuts mid-AST-node
• --tokens output is always within budget (verified by tiktoken count)
• C/C++ and TOML have ≥4 test fixtures each
• Performance: all new features maintain <50ms for 1000-line files

Out of Scope (Moved to Phase B)

• skim init hook installation → Phase B (hooks are valuable when we have command output compression)
• skim rewrite command engine → Phase B
• Multi-agent hook support → Phase B

---

Phase A focuses on making skim indispensable by capability, not by interception.

[dean0x]

Step 1: --mode=minimal — Implementation Plan

Summary

Add a new Minimal transformation mode that strips non-doc comments and normalizes blank lines while keeping all function bodies intact. Fills the gap between full (no transformation) and structure (bodies stripped). Expected savings: 15-30% token reduction on typical source files.

Prerequisites: Fixtures and Shared Utilities

Existing test fixtures contain only doc comments (/**, """, //!) — zero regular // or # comments. Before writing any transform code, we need comment-rich fixtures to test against.

Fixture-first approach:

1. Create tests/fixtures/{lang}/comments.{ext} for each tree-sitter language containing:
   • Regular single-line comments
   • Block comments
   • Doc comments (to verify preservation)
   • Inline comments (code + comment on same line)
   • Comments inside function bodies
   • Shebangs (Python fixture)
2. Then write failing tests against these fixtures
3. Then implement the transform

This step also introduces src/transform/utils.rs — a shared utility module for helpers needed across Steps 1-3:

• get_comment_prefix(language) -> &'static str — returns //, #, or <!-- per language
• is_inside_function_body(node, language) -> bool — checks if a node has a function body ancestor

Files to Change

Core library (crates/rskim-core/):

1. src/types.rs — Add Mode::Minimal variant and is_serde_based() helper

   • Add Minimal to the Mode enum (between Full and Structure conceptually)
   • Add "minimal" to Mode::parse() and Mode::name()
   • Add pub fn is_serde_based(&self) -> bool to Language — returns true for Json and Yaml. Step 3's cascade and Step 4's TOML extension depend on this method. Definition: matches!(self, Self::Json | Self::Yaml)

2. src/transform/utils.rs — New file, shared utilities

   • pub fn get_comment_prefix(language: Language) -> &'static str — exposed as pub (not pub(crate)) because Step 3's budget.rs in the CLI crate needs it for truncation markers
     • TypeScript/JavaScript/Rust/Go/Java → "//"
     • Python/YAML → "#"
     • Markdown → "<!--"
     • JSON → "//" (best-effort; JSON has no comments — omission markers in truncated output will not be valid JSON)
     • (Step 4 extends: C/C++ → "//", TOML → "#")
   • pub fn get_comment_suffix(language: Language) -> &'static str — closing counterpart for get_comment_prefix(), exposed as pub for the same reason. Returns " -->" for Markdown (closing the HTML comment), "" for all other languages. Without this, Markdown omission markers like <!-- ... 5 lines omitted would be unclosed HTML comments that swallow all subsequent content in renderers.
   • pub(crate) fn is_inside_function_body(node: Node, language: Language) -> bool
     • Walks ancestors looking for function body nodes (statement_block, block, compound_statement)

3. src/transform/minimal.rs — New file, core transformation logic

   • transform_minimal(source, tree, language, config) -> Result<String>

   • Three-pass approach:

     • Pass 1 (AST): Walk tree-sitter AST, collect byte ranges of non-doc comment nodes to remove
     • Pass 2 (String): Remove collected ranges, trim trailing whitespace on affected lines
     • Pass 3 (String): Normalize blank lines (3+ consecutive → 2)

   • Comment node types per language:

     Language	Single-line	Block	Doc patterns to KEEP
     TypeScript/JS	comment starts with //	comment starts with /*	/** (JSDoc)
     Python	comment (starts with #)	—	Docstrings: expression_statement > string that is the first child statement of a function_definition, class_definition, or module body
     Rust	line_comment	block_comment	///, //!, /** */, /*! */
     Go	comment	comment	Any // comment immediately preceding a declaration (see Go adjacency algorithm below)
     Java	line_comment	block_comment	/** (Javadoc)

   • For JSON/YAML/Markdown: minimal mode returns source unchanged (no comments to strip in data formats; content is preserved)

   • Security limits: define MAX_AST_DEPTH = 500 and MAX_AST_NODES = 100_000 constants (matching the values used in other transform modules — each module defines its own copy)

4. src/transform/mod.rs — Add routing

   • Add pub mod minimal; and pub mod utils;
   • Add Mode::Minimal => minimal::transform_minimal(...) to transform_tree() dispatcher

5. src/lib.rs — Re-export get_comment_prefix and get_comment_suffix from transform::utils as public API. The CLI crate's budget.rs (Step 3) needs both for well-formed truncation markers (Markdown requires closing -->). Other utils.rs functions (is_inside_function_body) remain pub(crate) since they use tree-sitter types that shouldn't leak across crate boundaries.

CLI (crates/rskim/):

6. src/main.rs — Add CLI option

   • Add Minimal to ModeArg enum
   • Add ModeArg::Minimal => Mode::Minimal to the From impl
   • Update help text examples

7. src/cache.rs — No changes needed (cache key already uses Mode's Debug format)

Tests:

8. tests/fixtures/{lang}/comments.{ext} — New comment-rich fixtures per language (created first)

9. crates/rskim-core/tests/integration.rs — Add tests per language

   • Test: regular comments stripped, doc comments preserved
   • Test: blank line normalization (3+ → 2)
   • Test: function bodies intact (including comments INSIDE bodies — preserved)
   • Test: inline comments removed, code preserved, no trailing whitespace
   • Test: shebangs preserved
   • Test: JSON/YAML/Markdown passthrough
   • Minimum 2 tests per tree-sitter language

10. crates/rskim/tests/cli.rs — CLI integration test

    • Test: --mode minimal flag accepted and produces expected output

Algorithm Detail

fn transform_minimal(source, tree, language, config):
    if language.is_serde_based() || language == Language::Markdown:
        return Ok(source.to_string())
    
    // Pass 1: Collect comment ranges to remove
    let removals = Vec<(usize, usize)>  // (start_byte, end_byte)
    walk AST nodes:
        if node is comment type for this language:
            // PRESERVE: shebangs (#! at byte 0)
            if node.start_byte() == 0 && node_text.starts_with("#!"):
                continue
            // PRESERVE: doc comments (language-specific patterns)
            if is_doc_comment(node, language):
                continue
            // PRESERVE: comments inside function bodies
            if is_inside_function_body(node, language):
                continue
            removals.push((node.start_byte(), node.end_byte()))
    
    // Pass 2: Build output without removed comments + trim trailing whitespace
    let result = apply_removals(source, removals)
    // For inline comments (e.g., `let x = 5; // comment`), removing the comment
    // leaves trailing whitespace. Trim each affected line.
    let result = trim_trailing_whitespace_on_removal_lines(result)
    
    // Pass 3: Normalize blank lines
    normalize_blank_lines(result)  // 3+ consecutive → 2

is_doc_comment() dispatcher: An internal function in minimal.rs that matches on language and delegates to language-specific detection: Go → is_go_doc_comment() (see below), Rust → check for ///, //!, /** */, /*! */ prefixes, TypeScript/JavaScript/Java → check for /** prefix, Python → always returns false (Python # comments are never doc comments; Python docstrings are expression_statement > string nodes — not comment types — and are automatically preserved because they never enter the comment removal path; see Python docstring detection below for their structure).

Go doc comment adjacency algorithm:

In Go, doc comments are multi-line // blocks immediately preceding a declaration. Each // line is a separate comment node in tree-sitter. The algorithm walks the full contiguous chain — a comment is a doc comment if there exists an unbroken chain of comment siblings (no blank lines) between it and a declaration:

fn is_go_doc_comment(comment_node) -> bool:
    // Walk forward through sibling nodes to find next non-comment sibling
    let next_decl = find_next_non_comment_sibling(comment_node)
    if next_decl is not a declaration (function_declaration, type_declaration, etc.):
        return false

    // Verify no blank lines exist between this comment and the declaration,
    // allowing only intervening comment nodes (contiguous doc block)
    let mut current = comment_node
    loop:
        let next = current.next_named_sibling()
        if next == next_decl:
            // Reached the declaration — check final adjacency
            return current.end_row() + 1 == next_decl.start_row()
        if next.kind() != "comment":
            return false  // Non-comment node between us and declaration
        if next.start_row() > current.end_row() + 1:
            return false  // Blank line breaks the doc block
        current = next

fn find_next_non_comment_sibling(node) -> Option<Node>:
    let mut sibling = node.next_named_sibling()
    while sibling.is_some() && sibling.kind() == "comment":
        sibling = sibling.next_named_sibling()
    sibling

Go adjacency edge cases: The adjacency algorithm above handles the common case (contiguous // block immediately before a declaration), but Go doc comment patterns in real codebases have edge cases worth verifying during implementation: (1) a blank line inside what appears to be a doc block splits it into a non-doc comment group and a doc comment group — the algorithm correctly rejects the earlier group via the start_row() > end_row() + 1 check, (2) a /* */ block comment immediately before a declaration is also a doc comment in Go convention — the algorithm handles this since tree-sitter represents it as a single comment node whose find_next_non_comment_sibling() resolves to the declaration, and (3) a comment block followed by a blank line then a declaration is not a doc comment — the final adjacency check (end_row() + 1 == start_row()) correctly rejects this. Validate against Go stdlib files (e.g., net/http/server.go, fmt/print.go) which exercise all three patterns. Add at least one test fixture per edge case.

Python docstring detection:

fn is_python_docstring(node) -> bool:
    // Must be expression_statement > string
    // AND must be the first child statement of a function/class/module body
    if node.kind() != "expression_statement": return false
    if node.child(0)?.kind() != "string": return false
    let parent = node.parent()?
    let body_children = get_body_statements(parent)
    return body_children.first() == Some(node)

fn get_body_statements(parent) -> Vec<Node>:
    // For function_definition/class_definition: children of the `block` child node
    // For module (root): direct named children that are statements
    if let Some(body) = parent.child_by_field_name("body"):
        body.named_children().collect()
    else:
        parent.named_children().collect()  // module level

Design Decisions

preserve_comments config field: The existing TransformConfig.preserve_comments field is currently unused (never read by any transform function — _config in structure.rs). Minimal mode is the first consumer: preserve_comments: true (default) means strip regular comments but keep doc comments. preserve_comments: false means strip ALL comments including doc comments. We do NOT activate this field for other modes in this PR — that's a separate concern.

Comments inside function bodies: Minimal mode ONLY strips top-level/module-level comments. Comments inside function bodies are preserved, consistent with "Keep all function bodies intact." This means minimal mode operates on the same scope as structure mode but with a lighter touch — it removes noise without removing structure.

Acceptance Criteria

• All existing 151 tests still pass
• --mode=minimal achieves 15-30% reduction on comment-rich test fixtures
• Doc comments (///, /**, """) are preserved
• Regular comments (//, #, /* */) are stripped
• Shebangs (#!/usr/bin/env python) are preserved
• Inline comments removed cleanly (no trailing whitespace left)
• Comments inside function bodies are preserved
• Blank lines normalized (3+ consecutive → 2)
• JSON/YAML/Markdown: minimal = full (passthrough)
• Cache works correctly with new mode
• Performance: <50ms for 1000-line files
• README and CLAUDE.md updated (new mode in tables, updated mode count)

Estimated Scope

~250-350 lines in minimal.rs, ~60 lines in utils.rs, ~20 lines of wiring, ~200 lines of test fixtures. Medium complexity — the doc comment detection (especially Go adjacency and Python docstrings) is the fiddly part.

[dean0x]

Step 2: --max-lines N — Implementation Plan

Summary

Add AST-aware smart truncation that fits the most informative lines within a line budget. Unlike head -n, this cuts at AST node boundaries and prioritizes high-value content (types, signatures, imports over body lines).

Key differentiator: No other tool truncates at AST boundaries.

Architecture: Truncation Inside the Transform Pipeline

Truncation lives inside transform_tree() in the core library, applied as a post-processing step after the mode transform but while the tree-sitter AST is still in scope. This avoids re-parsing transformed output (which is often invalid code — structure mode produces { /* ... */ } bodies that don't parse in Python, signatures mode produces bare fragments).

// In transform/mod.rs — transform_tree()
fn transform_tree(source, tree, language, config) -> Result<(String, Vec<NodeSpan>)> {
    let (transformed, spans) = match config.mode {
        Mode::Full => {
            let line_count = source.split('\n').count();  // Use split('\n') not lines() — must match truncate_to_lines() indexing
            Ok((source.to_string(), vec![NodeSpan {
                original_range: 0..line_count,
                transformed_range: 0..line_count,
                node_kind: "source_file",
            }]))
        },
        Mode::Minimal => minimal::transform_minimal(source, tree, language, config),
        Mode::Structure => structure::transform_structure(source, tree, language, config),
        // ... (Signatures, Types follow the same tuple return pattern)
    }?;

    let output = if let Some(max_lines) = config.max_lines {
        truncate::truncate_to_lines(&transformed, &spans, language, max_lines)?
    } else {
        transformed
    };

    Ok((output, spans))
}

Serde languages (JSON/YAML): These bypass transform_tree() entirely — they go through Language::transform_source() which calls json::transform_json() / yaml::transform_yaml() directly. Truncation for serde languages is applied in transform_source() after the serde path via a separate simple_line_truncate() function that takes the first N lines with an omission marker. No AST scoring is needed since data formats have flat, uniform structure.

// In types.rs — Language::transform_source()
pub(crate) fn transform_source(self, source: &str, config: &TransformConfig) -> Result<String> {
    let result = match self {
        Self::Json => json::transform_json(source)?,
        Self::Yaml => yaml::transform_yaml(source)?,
        _ => return tree_sitter_path(source, self, config),  // truncation inside transform_tree()
    };
    // Serde languages: apply simple line truncation here (not in transform_tree())
    if let Some(max_lines) = config.max_lines {
        return truncate::simple_line_truncate(&result, self, max_lines);
    }
    Ok(result)
}

Composability: --max-lines composes with any mode: skim file.rs --mode=structure --max-lines 50

Multi-file semantics: --max-lines applies per file (not total across files). For total budget across files, use --tokens instead.

Files to Change

Core library (crates/rskim-core/):

1. src/types.rs and src/lib.rs — Add config field and auto-detection API

   • Add max_lines: Option<usize> to TransformConfig
   • Update Default impl (default: None)
   • Add builder method: pub fn max_lines(mut self, n: usize) -> Self
   • Update transform_source() call to transform_tree() — destructure to extract the String and discard the Vec<NodeSpan>, since transform_source() returns Result<String>: let (output, _spans) = crate::transform::transform_tree(source, &tree, self, config)?;
   • Add pub fn transform_auto_with_config(source: &str, path: &Path, config: &TransformConfig) -> Result<String> to src/lib.rs — mirrors transform_auto() but accepts a full TransformConfig instead of just a Mode. The CLI's process_file() switches from transform_auto(source, path, mode) to transform_auto_with_config(source, path, &config) to thread max_lines through the auto-detection path. Without this, max_lines has no path from CLI to core — transform_auto() accepts only mode and creates a default config internally with max_lines: None. Similarly, the fallback path when auto-detection fails (transform(&contents, language, options.mode) in the Err branch of process_file()) must switch to transform_with_config(&contents, language, &config) — otherwise --max-lines is silently lost when the user overrides language detection with --language.

2. src/transform/truncate.rs — New file, truncation logic

   • pub(crate) fn truncate_to_lines(transformed: &str, spans: &[NodeSpan], language: Language, max_lines: usize) -> Result<String> — AST-aware truncation using the NodeSpan mapping built during transformation. Scores nodes by kind, selects greedily using transformed line counts, builds output from transformed line ranges.
   • pub(crate) fn simple_line_truncate(text: &str, language: Language, max_lines: usize) -> Result<String> — Non-AST truncation for serde-based languages (JSON/YAML, and TOML in Step 4). Takes the first N lines and appends a language-appropriate omission marker. Called from transform_source(), not transform_tree().
   • Uses utils::get_comment_prefix(), utils::get_comment_suffix(), and utils::score_node_kind() from the shared module created in Step 1

3. src/transform/utils.rs — Extend with new helpers (created in Step 1)

   • Add pub(crate) fn score_node_kind(kind: &str, language: Language) -> u8 — node priority scoring. Takes a &str node kind (from NodeSpan.node_kind), not a tree-sitter Node, to decouple scoring from the AST.
   • Reuse existing get_comment_prefix() and get_comment_suffix() for omission markers

4. src/transform/mod.rs — Wire truncation into the pipeline

   • Add pub mod truncate;
   • Define NodeSpan struct (or in truncate.rs) with original_range, transformed_range, and node_kind fields
   • Change transform_tree() return type from Result<String> to Result<(String, Vec<NodeSpan>)> — each mode-specific transform populates spans as it emits output. This requires updating the return types of all mode-specific functions: transform_structure(), transform_signatures(), transform_types() in their respective files, and transform_minimal() (added in Step 1). Mode::Full is handled inline — return (source.to_string(), vec![single NodeSpan covering all lines with kind "source_file"]) to represent the full file as one span. Mode::Minimal follows the same single-span pattern — since minimal mode preserves all code structure (only stripping comments and normalizing blank lines), return a single NodeSpan covering all output lines with kind "source_file". This means --max-lines with minimal mode uses head-style truncation rather than AST-priority selection, which is reasonable given that minimal output retains full function bodies and has no clear node-level priority hierarchy. Additionally, extract_markdown_headers() in structure.rs must also return Result<(String, Vec<NodeSpan>)> — it is called as an early return by transform_structure(), transform_signatures(), and transform_types() for Markdown input, and all three call sites must compile with the new tuple return. Return a single NodeSpan covering all output lines with kind "atx_heading". Each mode-specific function populates Vec<NodeSpan> as it walks the AST and emits output lines.
   • Add post-transform truncation step in transform_tree() that passes spans to truncate_to_lines() (see architecture above)
   • The sole caller (transform_source() in types.rs) destructures to (String, _) — see types.rs changes above

CLI (crates/rskim/):

5. src/main.rs — Add CLI arg and pipeline threading

   • Add --max-lines arg: #[arg(long, help = "Maximum output lines per file (AST-aware truncation)")]
   • Add max_lines: Option<usize> to ProcessOptions struct and its new() constructor. Wire from CLI args through ProcessOptions into both TransformConfig (for the core library) and cache function calls (for key generation).
   • Validate: must be > 0
   • Stdin path: The stdin handler in main() (current line 640) calls transform(&buffer, language, mode) directly, bypassing process_file(). Update to build a TransformConfig with max_lines and call transform_with_config(&buffer, language, &config) instead, so --max-lines works with stdin input (e.g., cat file.ts | skim - --language=ts --max-lines 50).

6. src/cache.rs — Extend cache key to include max_lines

   • The current cache key is format!("{}|{}|{}", canonical_path, mtime_secs, mode_str). Since truncation happens inside transform_tree(), the cached output is the truncated result. Running skim file.rs --max-lines 50 and then skim file.rs (no --max-lines) must not return the truncated cached output.
   • Add max_lines: Option<usize> parameter to cache_key(), read_cache(), and write_cache() function signatures — cache_key() incorporates it into the SHA256 hash input; read_cache() and write_cache() pass it through to cache_key()
   • Extend the cache key: format!("{}|{}|{}|{:?}", canonical_path, mtime_secs, mode_str, max_lines)
   • This means (path, mode, None) and (path, mode, Some(50)) produce different cache entries
   • Migration note: Adding the fourth field changes the hash input, so all pre-existing cache entries become misses on first run after upgrade. This is safe (misses just trigger re-transformation) but leaves orphaned files from the old 3-field key format. skim --clear-cache cleans them up. No explicit migration logic needed.
   • Thread max_lines from ProcessOptions through to the cache calls in the processing pipeline
   • Pre-existing issue: test_clear_cache_command in crates/rskim/tests/cli_cache.rs fails intermittently on macOS ("Directory not empty", os error 66). This is a race condition in the existing test, not caused by Step 2 changes. Fix or skip this test before extending cache behavior.

Tests:

7. crates/rskim-core/tests/integration.rs — Truncation tests
   • Test: output never exceeds N lines
   • Test: never cuts mid-function-signature (verify AST boundary)
   • Test: omission markers present and use language-appropriate comment syntax
   • Test: file shorter than N lines → no-op (passthrough)
   • Test: composable with each mode (structure + max-lines, signatures + max-lines)
   • Test: types/imports prioritized over body lines
   • Test: --max-lines 1 returns at least one meaningful line
   • Test: --show-stats reflects truncated output, not pre-truncation

Algorithm Detail

The truncation algorithm works across two coordinate systems: original source lines (where the AST operates) and transformed output lines (what the user sees). After transformation, line numbers diverge — e.g., a function at original lines 10-30 may occupy transformed lines 10-12 after structure mode strips the body. A NodeSpan mapping bridges these coordinates.

/// Returned alongside the transformed string from transform_tree().
/// Built during transformation — each mode's transform function already walks
/// the AST and knows which original lines map to which output lines.
/// INVARIANT: spans must be non-overlapping and sorted by transformed_range.
/// The truncation algorithm sums span lengths for budget tracking — overlapping
/// spans would cause double-counting and overshoot max_lines.
struct NodeSpan {
    original_range: Range<usize>,     // line range in source
    transformed_range: Range<usize>,  // line range in output
    node_kind: &'static str,          // for priority scoring — zero-alloc since tree-sitter Node::kind() returns &'static str
}

The transform_tree() function returns (String, Vec<NodeSpan>) instead of just String. Each mode-specific transform (structure, signatures, types, minimal) populates the spans as it emits output — no re-parsing or separate mapping pass needed.

fn truncate_to_lines(transformed, spans: &[NodeSpan], language, max_lines):
    lines = transformed.split('\n')
    if lines.len() <= max_lines:
        return Ok(transformed)  // No truncation needed

    // Score each node span by priority using node_kind from the original AST
    scored_spans = Vec<(priority, &NodeSpan)>
    for span in spans:
        priority = score_node_kind(&span.node_kind, language)
        scored_spans.push((priority, span))

    // Greedy selection: highest priority first, using TRANSFORMED line counts
    // Tie-break by source position (ascending) for deterministic output
    scored_spans.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.transformed_range.start.cmp(&b.1.transformed_range.start)))

    let marker_budget = 2  // Initial estimate for omission marker lines
    let effective_budget = max_lines.saturating_sub(marker_budget)
    selected = Vec<&NodeSpan>
    total_lines = 0
    for (_, span) in scored_spans:
        node_lines = span.transformed_range.len()
        if total_lines + node_lines <= effective_budget:
            selected.push(span)
            total_lines += node_lines

    // Fallback: if no span fits within budget (all spans larger than
    // effective_budget), take the first max_lines lines from the highest-
    // priority span. Guarantees non-empty output for any max_lines >= 1.
    if selected.is_empty() && !scored_spans.is_empty():
        let best = scored_spans[0].1  // Already sorted: highest priority first
        let start = best.transformed_range.start
        let take = min(max_lines, best.transformed_range.len())
        let partial_lines = lines[start..start + take].join("\n")
        let omitted = lines.len() - take  // Total lines minus kept (accounts for gaps before AND after span)
        if omitted > 0:
            let comment_prefix = get_comment_prefix(language)
            let comment_suffix = get_comment_suffix(language)
            return Ok(format!("{}\n{} ... {} lines omitted{}", partial_lines, comment_prefix, omitted, comment_suffix))
        return Ok(partial_lines)

    // Re-sort by transformed position (preserve reading order)
    selected.sort_by_key(|s| s.transformed_range.start)

    // Post-selection correction: actual marker lines may differ from initial estimate
    let marker_lines = count_gaps(selected) + if selected.last().end < lines.len() { 1 } else { 0 }
    while total_lines + marker_lines > max_lines && selected.len() > 1:
        // Drop the lowest-priority span among selected (by score_node_kind)
        let weakest = selected.iter().min_by_key(|s| score_node_kind(s.node_kind, language))
        total_lines -= weakest.transformed_range.len()
        selected.remove(weakest)
        marker_lines = count_gaps(selected) + if selected.last().end < lines.len() { 1 } else { 0 }

    // Build output using TRANSFORMED line ranges with omission markers between gaps
    let comment_prefix = get_comment_prefix(language)
    let comment_suffix = get_comment_suffix(language)  // "" for most languages, " -->" for Markdown
    output = String::new()
    for (i, span) in selected.iter().enumerate():
        if i > 0 && span.transformed_range.start > prev_end + 1:
            omitted = span.transformed_range.start - prev_end - 1
            output += format!("{} ... {} lines omitted{}\n", comment_prefix, omitted, comment_suffix)
        output += lines[span.transformed_range].join("\n")
        prev_end = span.transformed_range.end

    // Trailing omission marker if needed
    if prev_end < lines.len():
        omitted = lines.len() - prev_end
        output += format!("\n{} ... {} lines omitted{}", comment_prefix, omitted, comment_suffix)

    return Ok(output)

fn score_node_kind(kind: &str, language) -> u8:  // Lives in utils.rs
    match kind:
        // Priority 5 (highest): Type definitions, Markdown headings
        "type_alias_declaration" | "interface_declaration" |
        "struct_item" | "trait_item" | "enum_item" | "enum_declaration" |
        "struct_specifier" | "enum_specifier" | "type_definition" |
        "atx_heading" | "setext_heading" => 5

        // Priority 4: Function/method signatures, forward declarations, templates
        "function_declaration" | "function_item" | "method_declaration" |
        "function_definition" | "method_definition" |
        "declaration" |              // C/C++ forward declarations and prototypes
        "template_declaration" => 4  // C++ templates (wrapping functions, classes, or types)

        // Priority 3: Imports/exports
        "import_statement" | "use_declaration" | "import_declaration" |
        "preproc_include" => 3

        // Priority 2: Class/module declarations
        "class_declaration" | "module_declaration" | "impl_item" |
        "class_definition" | "class_specifier" | "namespace_definition" => 2

        // Priority 1 (lowest): Everything else
        _ => 1

Edge Cases

• Empty files: Return empty string
• --max-lines 1: Return the first line of the highest-priority node. Never return empty output.
• Single-node files: Return full content if it fits; if over budget, return first N lines of that node
• Deeply nested code: Score based on outermost node, not children
• C/C++ declaration breadth: The declaration node kind (priority 4) covers function prototypes, forward declarations, and top-level variable declarations. In practice this is acceptable: top-level declaration nodes in C files are predominantly prototypes and typedefs (local variables live inside compound_statement, scored as part of their parent node). Verify priority behavior against real C headers during implementation.
• JSON/YAML: Simple line-based truncation applied in transform_source() via simple_line_truncate() (not transform_tree(), which serde languages bypass entirely). Takes first N lines with an omission marker.
• Line mapping: NodeSpan mapping is constructed during transformation — no re-parsing or separate mapping pass needed. Each mode-specific transform already walks the AST and knows which original lines map to which output lines.
• Markdown: Truncate by header sections
• Trailing newlines: split('\n') is used consistently for line indexing (matching how NodeSpan ranges are computed in transform_tree()). A trailing newline creates an extra empty element — e.g., "a\nb\n" splits to 3 items. This means --max-lines 2 on a 2-visible-line file with trailing newline triggers truncation. This is consistent within the coordinate system and produces correct output (the empty trailing element is counted but the omission marker accounts for it).
• --show-stats interaction: Token counting happens AFTER truncation, so stats reflect the final output

Acceptance Criteria

• Output never exceeds --max-lines N lines
• Never cuts mid-function-signature or mid-type-definition
• Omission markers use language-appropriate comment syntax (//, #, <!--)
• Composable: --mode=structure --max-lines 50 works correctly
• Files shorter than N lines pass through unchanged
• Priority ordering: types > signatures > imports > structure > bodies
• Per-file semantics in multi-file mode (not total)
• --max-lines 1 returns at least one meaningful line
• Cache key includes max_lines — different --max-lines values produce different cache entries
• Running without --max-lines after a cached --max-lines 50 run returns full output
• Performance: <50ms for 1000-line files
• README and CLAUDE.md updated (new --max-lines flag documented)

Estimated Scope

~250-350 lines in truncate.rs, ~40 lines added to utils.rs (score_node_kind), ~30 lines wiring in mod.rs and main.rs. High complexity — the node scoring and boundary detection logic are the challenging parts, but the architecture (truncation inside the pipeline) keeps it clean.

[dean0x]

Step 3: --tokens N — Implementation Plan

Summary

Token budget mode: fit the most informative view of a file into exactly N tokens. Uses a mode cascade approach — tries progressively less aggressive modes until the output fits within budget. If no mode fits, truncates the most aggressive mode's output to fit.

Novel capability — no existing tool does this.

Design: Mode Cascade

v1 approach: Simple mode cascade — try the least aggressive mode first, fall back to more aggressive modes until output fits within the token budget:

Full → Minimal → Structure → Signatures → Types → truncate-to-fit

Each step is a clean, well-tested transform. The cascade is simple, predictable, and covers the primary use case: "give me the most informative view that fits in N tokens."

Composability with --mode: If the user specifies --tokens N --mode structure, we use structure mode output directly and truncate to fit if needed — no cascade.

Why not progressive layering? An alternative design builds up from a minimal view (types → add signatures → add imports → add short bodies). This was considered but deferred: merging different mode outputs is complex (types and signatures produce incompatible formats), and the cascade covers 90% of use cases with 10% of the complexity.

Architecture: Token Counting Stays in CLI

The tiktoken dependency stays in the CLI crate. The core library remains pure (no tokenizer dependency). The --tokens feature orchestrates multiple calls to the core transform() function from the CLI layer. The final truncation step uses a separate truncate_to_token_budget() function in budget.rs — a binary search on line count to fit within the token budget. This is distinct from Step 2's AST-aware truncate_to_lines(), which scores nodes by kind. The two features are complementary but use independent truncation algorithms.

Files to Change

CLI (crates/rskim/):

1. src/main.rs — Add CLI arg, orchestration, and fix --mode default

   • Add --tokens N arg: #[arg(long, help = "Token budget — fit output within N tokens")]
   • Validate: must be > 0
   • --tokens and --max-lines are mutually exclusive (clear error if both specified)
   • Change mode from ModeArg to Option<ModeArg>: The current mode field has default_value = "structure", which means args.mode is always ModeArg::Structure — even when the user doesn't pass --mode. There is no way to distinguish "user explicitly set --mode structure" from "clap applied the default." This breaks the cascade: it would never run because mode always has a value. Fix:

     // Before (current)
     #[arg(short, long, value_enum, default_value = "structure")]
     mode: ModeArg,
     
     // After
     #[arg(short, long, value_enum)]
     mode: Option<ModeArg>,

   • Defaulting logic in run(): When mode is None and --tokens is NOT specified, default to Mode::Structure (preserves current behavior — no breaking change). When mode is None and --tokens IS specified, pass None to the cascade so it runs the full cascade. When mode is Some(m) and --tokens is specified, use that mode's output and truncate to fit.
   • Affected locations in main.rs: Line 609 (let mode = Mode::from(args.mode)) is the primary breakpoint — change to args.mode.map(Mode::from). Lines 640, 666, 679, 689 all consume the derived mode variable and work unchanged once line 609 is fixed. The defaulting (unwrap_or(Mode::Structure)) should happen at line 609 for the non---tokens path.
   • This change is backwards-compatible: users who don't pass --mode still get structure mode. Only the --tokens cascade path observes the difference.
   • Help text update: Since Option<ModeArg> removes clap's [default: structure] indicator from --help output, update the --mode help text to explicitly state the default: "Transformation mode (default: structure unless --tokens triggers cascade)".
   • Add token_budget: Option<usize> to ProcessOptions and change mode: Mode to mode: Option<Mode>: The existing struct has mode, explicit_lang, use_cache, include_original. Add the budget field and change mode to Option<Mode> so the cascade can distinguish "user specified a mode" from "no mode specified." Also apply the same mode: Option<Mode> change to MultiFileOptions (main.rs:345-353, used by process_files()). In non---tokens code paths, use options.mode.unwrap_or(Mode::Structure) to preserve the current default behavior. The cascade's apply_token_budget() receives options.mode directly as its explicit_mode parameter.
   • Conditional dispatch in process_file(): If token_budget.is_some(), first resolve language explicitly via rskim_core::detect_language_from_path(path).or(opts.explicit_lang) — the cascade's apply_token_budget() requires an explicit Language parameter because it calls transform() multiple times (unlike transform_auto() which detects internally). Then call budget::apply_token_budget() with the resolved language. The budget function returns (String, usize) — the output and its token count. When --show-stats is active, reuse this count instead of re-counting.
   • Budget calculation before parallel dispatch: In process_files() (or the caller in main()/run()), compute per-file budget as total_budget / paths.len() before dispatching to rayon. Each file's ProcessOptions receives its share.
   • Stdin path: The stdin handler in main() calls transform() directly, bypassing process_file(). When --tokens is specified with stdin, resolve language from --language (or --file-name if Step 5 is landed) and call budget::apply_token_budget() directly. When --tokens is not specified but --max-lines is (Step 2), build a TransformConfig with the resolved mode and max_lines and call transform_with_config(). This mirrors the process_file() dispatch logic for the stdin code path. Cache bypass is automatic since stdin has no disk path to cache against.

2. src/tokens.rs — Add budget helpers

   • pub fn fits_in_budget(text: &str, budget: usize) -> Result<bool>
   • pub fn tokens_remaining(text: &str, budget: usize) -> Result<isize>

3. src/budget.rs — New file, mode cascade algorithm

   • pub fn apply_token_budget(source: &str, language: Language, budget: usize, explicit_mode: Option<Mode>) -> Result<(String, usize, Mode)> — returns the output, its token count, and the mode that produced the output (enables --show-stats to report which cascade level was selected)
   • Uses rskim_core::transform() internally for each cascade step
   • Uses rskim_core::get_comment_prefix() (public API, exposed in Step 1's lib.rs re-export) for truncation markers
   • Algorithm (see below)

Core library (crates/rskim-core/):

4. No core changes needed for v1. The cascade calls existing transform() with different modes. Each call re-parses with tree-sitter (~5ms), but the total cascade is ~25-50ms — within the 100ms target. If profiling shows this is too slow, we can expose a parse_once() + transform_tree() API to avoid redundant parsing.

Tests:

5. crates/rskim/tests/cli_tokens.rs — Extend with budget tests
   • Test: output never exceeds N tokens (verified by tiktoken)
   • Test: output uses at least 80% of budget (reasonable efficiency)
   • Test: small budget → types/signatures only (more aggressive mode auto-selected)
   • Test: large budget → full source returned
   • Test: --tokens N --mode structure → structure output, truncated if needed
   • Test: --tokens N --max-lines M → clear error message
   • Test: works with multi-file (skim 'src/*.rs' --tokens 8000)

Algorithm Detail

// In run(), derive explicit_mode from the now-optional args.mode:
//   let explicit_mode: Option<Mode> = args.mode.map(Mode::from);
// This is None when the user didn't pass --mode, Some(mode) when they did.

fn apply_token_budget(source, language, budget, explicit_mode) -> Result<(String, usize)>:
    // If user specified a mode, use that mode and truncate to fit
    if let Some(mode) = explicit_mode:
        let output = transform(source, language, mode)?
        let count = count_tokens(&output)?
        if count <= budget:
            return Ok((output, count))
        return truncate_to_token_budget(&output, language, budget)

    // Short-circuit for serde-based languages (JSON/YAML — extended to include TOML in Step 4):
    // They ignore mode — all cascade steps produce identical output.
    // Trying 5 modes would waste cycles for zero benefit.
    if language.is_serde_based():
        let output = transform(source, language, Mode::Full)?
        let count = count_tokens(&output)?
        if count <= budget:
            return Ok((output, count))
        return truncate_to_token_budget(&output, language, budget)

    // No explicit mode — cascade from least to most aggressive
    let modes = [Mode::Full, Mode::Minimal, Mode::Structure, Mode::Signatures, Mode::Types]

    let mut last_output = String::new();
    let mut last_count = 0;
    for mode in modes:
        last_output = transform(source, language, mode)?
        last_count = count_tokens(&last_output)?
        if last_count <= budget:
            return Ok((last_output, last_count))

    // Even Types mode (last in cascade) exceeds budget — truncate to fit
    truncate_to_token_budget(&last_output, language, budget)

fn truncate_to_token_budget(text, language, budget) -> Result<(String, usize)>:
    // Binary search for the right number of lines that fits within budget
    let lines: Vec<&str> = text.split('\n').collect()  // Use split('\n') not lines() — matches Step 2's convention for trailing newline consistency
    // Reserve tokens for the omission marker (e.g., "// ... 47 lines omitted" ≈ 8 tokens)
    let marker_reserve = 10
    let content_budget = budget.saturating_sub(marker_reserve)
    let mut lo = 1  // Always return at least 1 line, even if it exceeds budget
    let mut hi = lines.len()

    while lo < hi:
        let mid = (lo + hi + 1) / 2
        let candidate = lines[..mid].join("\n")
        if fits_in_budget(&candidate, content_budget)?:
            lo = mid
        else:
            hi = mid - 1

    let truncated = lines[..lo].join("\n")
    let omitted = lines.len() - lo
    let result = if omitted > 0:
        let prefix = rskim_core::get_comment_prefix(language)  // Public API from Step 1
        let suffix = rskim_core::get_comment_suffix(language)  // " -->" for Markdown, "" otherwise
        format!("{}\n{} ... {} lines omitted{}", truncated, prefix, omitted, suffix)
    else:
        truncated
    let final_count = count_tokens(&result)?
    Ok((result, final_count))

Performance Considerations

The cascade calls transform() up to 5 times, each re-parsing with tree-sitter (~5ms per parse). Token counting via tiktoken is fast after first init (~50-100ms for loading BPE vocabulary, then ~0.5ms per count). Worst case: ~25ms for parsing + ~2.5ms for token counting = ~28ms per file. Well within the 100ms target.

The cascade short-circuits: if Full mode fits, we never try other modes. For most files with generous budgets, only 1-2 modes are tried.

Token count reuse: apply_token_budget() returns (String, usize, Mode) — the output, its final token count, and the mode that was used. When --show-stats is active, the token count is reused directly (avoiding a redundant tiktoken pass), and the selected mode is printed to stderr (e.g., mode: structure (auto-selected by cascade)) so agent workflows can see which cascade level was chosen.

Caching Strategy

Do NOT cache --tokens N output. The output varies by budget value, which would cause cache explosion. Instead:

• Bypass both cache reads and cache writes when --tokens is active. Guard the existing cache read at the top of process_file() with if options.use_cache && options.token_budget.is_none() — without this, a prior cached result (e.g., from running skim file.rs which caches structure mode output) would short-circuit the cascade and return stale output that ignores the token budget entirely.
• The cascade calls rskim_core::transform() directly (a pure library function with no cache interaction), so intermediate mode transforms are not cached. This is acceptable for v1 — the cascade completes in ~28ms worst case (5 parses × ~5ms + token counting). If profiling shows intermediate caching would help, a future optimization can call cache::read_cache() before each transform() and fall back on cache miss.

Multi-file Budget Distribution

For skim 'src/*.rs' --tokens 8000:

• v1: Equal split — max(budget / num_files, 1) tokens per file. Simple and predictable. Guard: if glob matches zero files, return early with no output (no division by zero).
• v2 (future): Proportional to each file's structure-mode token count. Transform all files in structure mode first, measure tokens, then distribute budget proportionally.

Pipeline integration: The per-file budget is computed in process_files() (or the caller in run()) before rayon parallel dispatch. Each file's ProcessOptions carries its token_budget: Option<usize> share. Inside process_file(), resolve the language and mode, then dispatch:

// Resolve language: auto-detect from path, fall back to --language flag
let language = rskim_core::detect_language_from_path(path)
    .or(opts.explicit_lang)
    .ok_or_else(|| anyhow!("Could not detect language for {}", path.display()))?;
let mode = opts.mode.unwrap_or(Mode::Structure);

let (output, token_count) = if let Some(budget) = opts.token_budget {
    // Budget path: cascade needs explicit language (calls transform() per mode)
    let (out, count) = budget::apply_token_budget(&source, language, budget, opts.mode)?;
    (out, Some(count))
} else {
    // Non-budget path: build config with mode + optional max_lines (from Step 2)
    let mut config = TransformConfig::with_mode(mode);
    if let Some(n) = opts.max_lines {
        config = config.max_lines(n);  // Step 2's builder method
    }
    let output = rskim_core::transform_with_config(&source, language, &config)?;
    let token_count = if opts.include_original { Some(count_tokens(&output)?) } else { None };
    (output, token_count)
};

Cache writes are skipped when --tokens is active. The cascade calls rskim_core::transform() directly, which has no cache interaction (see caching strategy above), so intermediate mode transforms are also not cached.

Edge Cases

• Budget of 0: Error ("--tokens must be at least 1")
• Budget larger than full file: Return full mode output
• Very small budget (< 50): Return as many type signatures as fit
• Empty file: Return empty
• JSON/YAML/TOML: Short-circuits — serde languages produce identical output regardless of mode, so the cascade tries once then truncates if needed. No wasted cycles on redundant transforms.
• Multi-file: Each file gets equal share of total budget (v1)
• --tokens N --mode structure: Use structure output, truncate if it exceeds N tokens
• Trailing newlines: split('\n') (used in truncate_to_token_budget) creates an extra empty element for text ending with \n — e.g., "a\nb\n" splits to 3 items. This affects binary search bounds (hi = lines.len()) and omission line counts. Behavior is consistent with Step 2's truncate_to_lines() coordinate system.

Acceptance Criteria

• Output fits within N tokens (verified by tiktoken count) — with two caveats: (1) always returns at least one line even if that line alone exceeds budget, and (2) the omission marker adds ~5-10 tokens that are accounted for by a 10-token reserve in the binary search
• Cascade tries modes in correct order (Full → Minimal → Structure → Signatures → Types)
• Small budget → more aggressive mode automatically selected
• Large budget → full source returned
• --tokens N --mode X → uses mode X output, truncates if needed
• --tokens N --max-lines M → clear error message
• Works with multi-file (equal budget distribution per file)
• --tokens bypasses both cache writes and cache reads — the cascade calls rskim_core::transform() directly with no cache interaction
• Performance: <100ms for 1000-line files
• README and CLAUDE.md updated (new --tokens flag documented)

Estimated Scope

~200-250 lines in budget.rs, ~30 lines in tokens.rs, ~40 lines wiring in main.rs (includes Option<ModeArg> change and defaulting logic). Medium complexity — the cascade is straightforward. The Option<ModeArg> change, binary search truncation, and cache bypass are the tricky parts.

Note on Dependencies

This step depends on Step 1 for three items: Mode::Minimal (cascade order), Language::is_serde_based() (serde short-circuit), and rskim_core::get_comment_prefix() (truncation markers). Implement Step 1 first, or stub these three items directly if developing concurrently.

[dean0x]

Step 4: C/C++ and TOML Language Support — Implementation Plan

Summary

Add C/C++ via tree-sitter grammars and TOML via serde. Standard language additions following the established patterns in the codebase. Part A (C/C++) is fully independent of Steps 1-3 and can be developed on a separate branch concurrently. Part B (TOML) depends on Step 1's is_serde_based() method — develop Part B after Step 1 lands, or add the method directly in types.rs if developing concurrently.

Part A: C/C++ Support

Approach: Two separate Language variants (C and Cpp) with their own node type mappings. Tree-sitter-c and tree-sitter-cpp are well-maintained grammars with stable APIs.

.h files default to C++: Header files are used by both C and C++ projects. Since tree-sitter-cpp correctly parses valid C code (C is largely a subset), defaulting .h to C++ is the safer choice — it handles both C and C++ headers. Defaulting to C would fail on templates, classes, namespaces. This matches clangd and other modern tooling. Users can override with --language c if needed.

Node type discovery: The node types listed below are based on tree-sitter documentation, but must be verified by running tree-sitter parse on the test fixtures during implementation. Do not rely solely on documentation — parse real files and inspect the AST.

Files to Change:

1. Cargo.toml (workspace)

   • Add tree-sitter-c and tree-sitter-cpp dependencies — use latest versions compatible with workspace tree-sitter = "0.25" (ABI 15). Check crates.io for compatible releases; as of writing, some published versions target ABI 14 which may require a tree-sitter-language = "0.1" compatibility shim, or newer 0.25+ releases may be available. Verify with cargo check before proceeding.

2. crates/rskim-core/src/types.rs

   • Add Language::C and Language::Cpp variants
   • from_extension: .c → C; .h → Cpp; .cpp, .cc, .cxx, .hpp, .hxx → Cpp
   • name(): "C", "C++"
   • to_tree_sitter(): tree_sitter_c::LANGUAGE.into(), tree_sitter_cpp::LANGUAGE.into()

3. crates/rskim-core/src/transform/structure.rs — Add node types

   • C: function: "function_definition", method: "function_definition"
   • C++: function: "function_definition", method: "function_definition"
   • Body node: compound_statement (already in find_body_node() — no change needed there)

4. crates/rskim-core/src/transform/signatures.rs — Add signature node types

   • C: function_definition, declaration (for prototypes/forward declarations)
   • C++: above + template_declaration

5. crates/rskim-core/src/transform/types.rs — Add type node types

   • C: struct_specifier, enum_specifier, type_definition (typedef)
   • C++: above + class_specifier, template_declaration (template classes), namespace_definition

6. crates/rskim-core/src/transform/minimal.rs — Add comment detection rules (if Step 1 is merged)

   • C/C++ comment types: comment node for both // and /* */
   • Doc comment patterns to preserve: /** (Doxygen block) and /// (Doxygen line — standard in C++)
   • If Step 1 is not yet merged, defer this to a follow-up PR

7. crates/rskim-core/src/transform/utils.rs — Extend shared utilities

   • Add C, Cpp to get_comment_prefix() → "//", and Toml → "#" (TOML's comment character, needed by simple_line_truncate() for omission markers)
   • Extend score_node_kind() with 3 C/C++ kinds not already in Step 2's table: class_specifier → priority 2 (alongside class_declaration), template_declaration → priority 4, and declaration (C prototypes/forward declarations) → priority 4. Step 2's definition already includes struct_specifier, enum_specifier, type_definition, preproc_include, function_definition, and namespace_definition — those need no changes. If implementing before Step 2, include all 9 kinds when creating the function.

8. crates/rskim-core/src/lib.rs — Update supported_languages() to include C and Cpp

9. crates/rskim/src/main.rs — Add to CLI

   • Add C and Cpp to LanguageArg enum with aliases (c, cpp, cxx, cc)

Test Fixtures (minimum 4 per language):

10. tests/fixtures/c/

    • simple.c — functions, includes, typedefs
    • struct.c — struct definitions, function pointers
    • header.h — header file with declarations, include guards
    • preprocessor.c — macros, conditionals, #define function-like macros
    • comments.c — regular // and /* */ comments, Doxygen /** */ doc comments (to test minimal mode preservation/stripping), inline comments on code lines

11. tests/fixtures/cpp/

    • simple.cpp — functions, namespaces
    • class.cpp — class with methods, constructors, destructors
    • template.cpp — template functions and classes
    • modern.cpp — auto, lambdas, range-for, smart pointers
    • comments.cpp — regular // and /* */ comments, Doxygen /** */ and /// doc comments (to test minimal mode preservation/stripping), inline comments on code lines

12. crates/rskim-core/tests/integration.rs — Tests per mode per language

---

Part B: TOML Support

Decision: serde approach (like JSON/YAML).

TOML is a configuration format (data), not code — same reasoning that led to JSON using serde_json and YAML using serde_yaml_ng. The toml crate (v1.0, serde-based) is mature and well-maintained. This keeps the pattern consistent: serde for data formats, tree-sitter for code.

Files to Change:

1. Cargo.toml — Add toml = "1.0" dependency to rskim-core

2. crates/rskim-core/src/types.rs

   • Add Language::Toml variant
   • from_extension: .toml → Toml
   • name(): "TOML"
   • to_tree_sitter(): None (uses serde, not tree-sitter)
   • Extend Language::is_serde_based() (added in Step 1) to include Self::Toml
   • transform_source(): Route to crate::transform::toml::transform_toml(source). Since TOML is serde-based, all transformation modes produce identical key-only output (the --mode flag is effectively ignored, matching JSON/YAML behavior). --max-lines truncation is handled by the simple_line_truncate() pattern established in Step 2 — applied in transform_source() after the serde path, not in transform_tree(). TOML inherits this automatically with no additional truncation code needed.

3. crates/rskim-core/src/transform/toml.rs — New file

   • pub(crate) fn transform_toml(source: &str) -> Result<String>
   • Key-only output matching JSON/YAML pattern:

     # Input                          # Output
     [package]                        [package]
     name = "skim"                    name
     version = "1.0.0"               version
     
     [dependencies]                   [dependencies]
     tree-sitter = "0.23"            tree-sitter

   • Inline tables: point = { x = 1, y = 2 } → expand to nested form (matching JSON/YAML pattern of showing nested structure):

     [point]
     x
     y
     

   • Array of tables: [[servers]] → preserve the double-bracket [[servers]] header in output (semantically distinct from single-bracket [servers] — indicates an array of tables, not a single table). Show keys from the first table entry only; subsequent entries are omitted (matching how JSON handles arrays of objects)
   • Dotted keys: owner.name = "Tom" → expand to nested form:

     [owner]
     name
     

   • Security limits: MAX_TOML_DEPTH = 500, MAX_TOML_KEYS = 10_000

4. crates/rskim-core/src/transform/mod.rs — Add pub mod toml;

5. crates/rskim-core/src/lib.rs — Update supported_languages()

6. crates/rskim/src/main.rs — Add Toml to LanguageArg with alias toml

Test Fixtures (minimum 4):

7. tests/fixtures/toml/

   • simple.toml — basic key-value pairs
   • nested.toml — nested tables, dotted keys, inline tables
   • array.toml — array of tables, inline arrays
   • cargo.toml — real-world Cargo.toml example

8. crates/rskim-core/tests/integration.rs — TOML transform tests

---

Acceptance Criteria

• C files: .c detected and transformed in all modes
• C++ files: .cpp, .cc, .cxx, .hpp, .hxx, .h detected and transformed
• .h defaults to C++ (overridable with --language c)
• TOML: .toml detected and transformed (key-only output)
• TOML inline tables, array of tables, dotted keys handled correctly
• ≥4 test fixtures each for C, C++, and TOML
• All existing tests still pass
• All modes work for C/C++ (structure, signatures, types, full, and minimal if Step 1 is merged)
• supported_languages() returns 12 languages
• Performance: C/C++ transforms <50ms for 1000-line files
• README and CLAUDE.md updated (language count updated to 12, new extensions in language tables)

Estimated Scope

• C/C++: ~120 lines of node type mappings + ~200 lines of test fixtures + ~50 lines in minimal.rs/utils.rs
• TOML: ~150 lines in toml.rs + ~80 lines of test fixtures
• Total: ~600 lines. Low-medium complexity — follows established patterns, but C++ node type discovery requires care.