Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# 11L XSA-all + Order-Adaptive 11-gram Backoff

## Results

| Seed | val_bpb |
|------|---------|
| 42 | 0.8883 |
| 1337 | 0.8886 |
| 2024 | 0.8875 |
| **Mean** | **0.8881** |
| Std | 0.0006 |

Artifact: 13.99 MB. Train: 600s on 8xH100 SXM. Eval: ~160s.

## Architecture

- 11 layers, 512 dim, 8/4 GQA heads
- XSA-all, LeakyReLU(0.5)^2, 3x MLP
- BigramHash (10240, 128d), SmearGate
- Value Residual, Gated Attention, LN scaling
- GPTQ-lite int6 + zstd-22
- EMA(0.997), Tight SWA, Late QAT

## Order-Adaptive N-gram Eval Cache

- Orders 2 through 11, highest-order-first backoff
- Order-adaptive entropy gating: higher-order matches trust n-gram at lower model uncertainty
- Separate hash tables per order
- Score-first, deterministic, no TTT
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"name": "11L XSA-all + order-adaptive 11-gram backoff",
"val_loss": 0.8881,
"bytes_total": 13991619,
"blurb": "11L XSA-all with order-adaptive entropy-gated 11-gram backoff. Mean 0.8881 (std 0.0006) across 3 seeds.",
"author": "hypery11",
"github_id": "hypery11",
"date": "2026-03-26"
}
Loading