10L + Multi-Order N-gram Backoff (0.9123 BPB)

[Bortlesboat]

Record submission

val_bpb: 0.9123 (mean of 3 seeds, post int5/int6+zstd quantization roundtrip)

Seed	val_bpb	artifact_bytes
42	0.9128	15,320,000
1337	0.9121	15,630,000
2024	0.9121	15,330,000

Architecture

• 10 layers, d=512, GQA 8H/4KV, LeakyReLU(0.5)^2
• Partial RoPE (16/64), LN Scale, XSA last 4, Value Residual
• BigramHash(4096, dim=128), SmearGate, U-Net skips
• Mixed int5 MLP / int6 attention + zstd-22
• EMA(0.997), Muon WD=0.04, warmdown=3500

Eval: Multi-Order N-gram Backoff + Entropy-Adaptive Alpha

• Hashed n-gram cache, orders 2 through 7 with backoff
• Highest matching order wins (7-gram preferred, falls back to lower)
• Entropy-adaptive alpha: alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0))
• Score-first: cache updated only AFTER scoring each segment
• 4M hash buckets per order, min_count=2

Timing (8xH100 SXM)

• Training: 600s (~6020 steps at 99ms/step)
• Eval: ~163s (sliding window stride=64, batch_seqs=64)

Based on

• thwu1's 10L Int5-MLP base architecture
• PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727 (multi-order n-gram backoff concept)
• PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 (LeakyReLU^2)
• PR Record: 11L XSA + EMA + Int6 MLP3x + WD=0.04 (val_bpb: 1.1271) #287 (XSA, Partial RoPE, LN Scale)