What's actually legal TTT? A summary of Issue #402 rulings

[dentity007]

TL;DR for anyone confused by the TTT situation (I was too):

I spent $65 on an 8×H100 pod running 10-epoch TTT before discovering it was banned. Here's what I learned so others don't make the same mistake.

ILLEGAL (your PR will be closed):

• Multi-epoch TTT — training on validation data more than once
• Any approach where the model sees the same token twice during eval
• AdamW/SGD optimization loops over validation chunks

LEGAL:

• Score-first single-pass TTT — score the token, THEN adapt, never revisit
• LoRA TTT with document-level reset (PR [record bpb=1.195] sliding window + LoRA TTT #77 approach)
• Any adaptation where information flows strictly forward

Key organizer quotes (from @valerio-oai in Issue #402):

• "Multi-epoch TTT is not valid"
• "Training on the validation set" = closed

Submissions closed so far:

• PR Record: 11L XSA4 + LeakyReLU(0.5)² + Cosine TTT 50ep (val_bpb=1.0622) #518 (1.0622), PR Record: 11L EMA + AdamW TTT 10ep (mean val_bpb=1.1027) #442 (1.0992), PR Record: Cosine TTT scheduling with per-layer lr — mean val_bpb=1.0970 (3 seeds) #481 (1.0970), PR Record*: val_bpb=0.978 BPB — Goldfish ML Autonomous Research (100ep Cosine *leaky* TTT) #517 (0.978), PR Record: pcloadloveletter v6 — Novel Codebook+Huffman Compression + AdamW TTT (val_bpb=1.0487) #532 (1.0487)

The realistic leaderboard looks very different once you filter these out. Check the interactive dashboard and filter to "Open Only" to see where you actually stand.

Hope this saves someone a few hours and a few dollars.