Record: 7-gram N-gram Cache (0.8960 bpb)

[armantsaturian]

7-gram N-gram Cache + XSA-All + LeakyReLU² + Parallel Muon

val_bpb: 0.8960 (3-seed mean, std 0.0004) | ~15.92 MB | 8×H100 SXM

Results

Seed	Pre-cache bpb	Post-cache bpb	Cache gain	Eval time	Artifact
1337	1.1488	0.8956	-0.2532	136s	15,917,799
42	1.1488	0.8959	-0.2529	136s	15,921,427
2025	1.1500	0.8966	-0.2534	138s	15,911,611
Mean	1.1492	0.8960	-0.2532	~137s

Approach: Streaming 7-gram N-gram Cache at Eval

A single-pass streaming n-gram cache built during sliding window evaluation. For each scored token:

1. Score neural model NLL under torch.inference_mode()
2. Cache lookup across n-gram orders 2-7, with near-zero backoff beta (1e-6) — longest match dominates
3. Blend with fixed 80/20 cache/neural mix (alpha=0.20)
4. Update cache with scored token's n-grams (score-before-update, single-pass)

No TTT, no online mixer — the cache alone delivers -0.2532 bpb over the base model.

Legality

• ✅ Single-pass, score-before-update (cache updated only after scoring)
• ✅ All eval under torch.inference_mode() (no gradients, no weight mutation)
• ✅ No training on validation data
• ✅ No multi-pass replay or phase-2 re-scoring
• ✅ No external downloads — fully self-contained

Timing

Phase	Time
Training	600s (10 min cap)
Eval (sliding window + 7-gram cache)	~137s
Total	~737s

Built on

• PR #549 — LeakyReLU² + Legal TTT + Parallel Muon (base model)
• PR #741 — N-gram cache inspiration
• PR #745 — XSA-all, Hedge mixer concepts
• PAQ/cmix compression literature — backoff and blending techniques

See full README for ablation, architecture details, and what we tried and discarded.