10L + Two-Pass Order-11 N-gram Backoff (0.5863 BPB)

[Bortlesboat]

Record submission

val_bpb: 0.5863 (mean of 3 seeds, std 0.0002)

Seed	val_bpb	artifact_bytes
42	0.5864	15,420,000
1337	0.5864	15,570,000
2024	0.5860	15,370,000

Method

10L d=512 GQA transformer with two-pass eval:

Pass 1 (189s): score-first sliding window with orders 2-11 hashed n-gram cache. Order-adaptive entropy gating — higher-order matches trust n-gram at lower model uncertainty. Cache updated only AFTER scoring.

Pass 2 (140s): rescore early cold-cache windows using the now-complete cache (frozen, no updates). All rescored tokens were already evaluated in pass 1. Total eval: 331s.

Architecture

• 10L, d=512, GQA 8H/4KV, LeakyReLU(0.5)^2, Partial RoPE, LN Scale, XSA last 4, Value Residual
• Mixed int5 MLP / int6 attention + zstd-22, EMA(0.997), matrix_lr=0.03

Compliance

• Score-first: BPB finalized before cache update
• Backward-looking: only previously scored tokens in cache
• No target-aware gating: alpha from model entropy + matched order only
• Pass 2: rescores already-evaluated tokens with frozen cache

Timing (8xH100 SXM)

• Training: 600s (~6004 steps)
• Eval: 331s (pass 1: 189s + pass 2: 142s)
• Artifact: 15.4-15.6 MB

Based on

• thwu1 (base architecture), PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727/Record: 11L + order-adaptive 9-gram backoff (mean val_bpb=0.9059) #788 (n-gram backoff), PR Record: Two-Pass N-gram Rescoring (val_bpb 0.1434) #846 (two-pass), PR Record: 0.9076 BPB — 10L + N-gram Backoff + Matrix LR 0.03 #828 (matrix_lr)