Int6 GPTQ-lite + LeakyReLU(0.5)^2 + EMA + 11L MLP3x

records/track_10min_16mb/2026-03-25_Int6_LeakyReLU2_EMA_11L_MLP3x/README.md

@@ -0,0 +1,24 @@
+# Int6 GPTQ-lite + LeakyReLU(0.5)^2 + EMA + 11L MLP3x
+
+## Improvements over baseline
+
+| Change | Expected BPB Delta |
+|--------|-------------------|
+| 11 layers (vs 9) | -0.005 |
+| MLP 3x (vs 2x) | -0.005 |
+| LeakyReLU(0.5)^2 | -0.003 |
+| Int6 GPTQ-lite + zstd-22 | -0.005 |
+| Late QAT (STE, LR < 0.15) | -0.003 |
+| EMA (decay 0.997) | -0.005 |
+| Sliding window eval (stride 64) | -0.005 |
+
+## Run Command
+
+```bash
+torchrun --standalone --nproc_per_node=8 train_gpt.py
+```
+
+Single GPU test:
+```bash
+torchrun --standalone --nproc_per_node=1 train_gpt.py
+```

records/track_10min_16mb/2026-03-25_Int6_LeakyReLU2_EMA_11L_MLP3x/submission.json

@@ -0,0 +1,9 @@
+{
+  "name": "Int6 GPTQ-lite + LeakyReLU(0.5)^2 + EMA + 11L MLP3x",
+  "val_bpb": 0.0,
+  "bytes_total": 0,
+  "blurb": "11 layers (vs 9 baseline) + MLP 3x expansion + LeakyReLU(0.5)^2 activation + Int6 per-row GPTQ-lite quantization + zstd-22 compression + Late QAT via STE + EMA(0.997) weight averaging + sliding window eval (stride=64).",
+  "author": "zeytx",
+  "github_id": "zeytx",
+  "date": "2026-03-25"
+}