gitindex: optimize git index time by ~21%

[clemlesne]

Summary

• Eliminate redundant enry classification: rank() was calling enry.IsGenerated/IsVendor/IsTest per document, duplicating work already done by DetermineFileCategory. Now uses cached doc.Category. Moved DetermineFileCategory to Builder.Add() to overlap with catfile I/O.
• Skip redundant binary check: ShardBuilder.Add() re-scanned all content for null bytes even though DocChecker.Check already did this. Now skipped when Category is already set.
• Single-byte varint fast path: ~80% of posting list deltas are < 128. append(slice, byte) is cheaper than append(slice, slice...).
• Slab allocator for file content: Replaced ~29K individual make([]byte, size) with 16MB slab sub-slicing (3-index slices for safety), reducing GC pressure.
• GC tuning for batch indexing: GOGC=-1 + GOMEMLIMIT=2GiB minimizes madvise syscall overhead. Both respect user env var overrides.

Benchmark (hyperfine)

Measured with hyperfine on kubernetes/kubernetes (29,226 files, ~230MB, no ctags):

hyperfine --warmup 3 --min-runs 15 \
  --prepare 'rm -rf /tmp/zoekt-index-hyperfine' \
  'zoekt-git-index-baseline -index /tmp/zoekt-index-hyperfine -disable_ctags /tmp/k8s-bench' \
  'zoekt-git-index-optimized -index /tmp/zoekt-index-hyperfine -disable_ctags /tmp/k8s-bench'

Command	Mean [s]	Min [s]	Max [s]	Relative
baseline (main)	5.987 ± 0.419	5.590	7.001	1.24 ± 0.09
optimized (PR)	4.813 ± 0.100	4.690	5.058	1.00

1.24x faster (mean), with notably lower variance (σ=0.100 vs σ=0.419).

Behavioral notes

• rank() now uses DetermineFileCategory's mutually exclusive categories instead of independent enry.IsGenerated/IsVendor/IsTest checks. A file matching multiple categories (e.g. generated test file) previously received multiple rank penalties; now it receives only the highest-priority one. This changes intra-shard document ordering slightly (<0.02% shard size difference) but does not affect search correctness.
• For SkipReasonTooManyTrigrams files, DetermineFileCategory now receives the original content (previously it received the marker string due to content replacement in ShardBuilder.Add). This is more correct — DetermineFileCategory was already designed to handle this case.

Test plan

• go test ./... full suite passes
• Output shards have identical file counts
• ShardBuilder.Add still performs binary check + DetermineFileCategory for direct callers (merge path, AddFile) where Category == FileCategoryMissing
• Slab sub-slices use 3-index slices to prevent accidental overwrites
• GOGC and GOMEMLIMIT env vars are respected when set by the user

🤖 Generated with Claude Code

[keegancsmith]

Thanks, I'm out for easter weekend so will take a look Tuesday.