feat: intent + structured query for deep_search

[igrigorik]

Intent — steer every pipeline stage

Optional intent parameter across all search tools. When a query is ambiguous
(e.g., "performance" → web-perf vs health), intent provides background context
that steers results toward the caller's interpretation.

{ query: "performance", intent: "web performance and latency" }

Intent flows through five stages in deep_search:

1. Query expansion: LLM prompt becomes "Context: {intent}\nExpand: {query}"
2. Strong-signal bypass: disabled — obvious BM25 match isn't what the caller wants
3. Chunk selection: intent terms (weight 0.5) augment query terms (1.0)
4. Reranking: intent prepended to query for Qwen3-Reranker
5. Snippet extraction: intent terms (weight 0.3) nudge toward relevant lines

Structured query — caller-provided expansions bypass LLM

deep_search has a local 1.7B LLM that expands queries into keywords, concepts,
and a hypothetical passage. An upstream LLM caller (Claude, GPT) already knows
what the user wants — routing through 1.7B is a capability downgrade.

Structured query lets the caller provide expansions directly, skipping LLM:

  string (auto — LLM expands):        structured (manual — no LLM):

  "performance"                        { text, keywords, concepts, passage }
        │                                 │       │         │         │
        ▼                                 ▼       ▼         ▼         ▼
  ┌────────────┐                        BM25     FTS     vector    vector
  │ Qwen3 1.7B │ → {lex,vec,hyde}        └───────┴─────────┴─────────┘
  └────────────┘                                      │
        │                                             ▼
        ▼                                       RRF → rerank
  FTS + vector → RRF → rerank
                                               ~240ms (89% faster)
  ~2s (LLM bottleneck)

Fields route directly: keywords → FTS, concepts → vector, passage → vector.
Intent remains orthogonal — steers scoring in both modes.

Eval (2115-doc corpus, 8 ambiguous queries)

Each query has multiple valid interpretations (e.g. "performance" → web-perf vs
health, "scaling" → infra vs business).

Metric	Baseline (LLM)	Intent (steered)	Structured (no LLM)
Signal Density @5	0.500	0.675 (+0.175)	0.725 (+0.225)
MRR	0.738	0.938 (+0.200)	1.000 (+0.262)
Avg latency	2114ms	1781ms	240ms (89% faster)

MRR of 1.000 = first result always relevant. Quality wins from the upstream
caller being better at disambiguation than the local 1.7B model.

Implementation highlights

• StructuredQuery type: { text, keywords?, concepts?, passage? } with normalizeQuery() stripping empty fields and hasCallerExpansions() detecting caller-driven mode
• Unified routing: FTS and vector paths shared between caller/LLM modes — ~95 → ~65 lines
• RRF weight fix: replaced positional heuristic (i < 2) with primaryIndices: Set<number> tracking which ranked lists represent the original query
• MCP schema: z.union([string, object]) → anyOf in JSON Schema. Empirically tested: Opus/Sonnet handle correctly, Haiku fails safe via additionalProperties: false
• extractSnippet refactor: 6 positional params → options object, eliminates undefined placeholders at ~31 callsites
• Intent tokenization: stop word set + punctuation stripping, short domain terms (API, SQL) survive
• Collection filtering: MCP instructions now direct callers to filter by collection when intent maps