Feat: Lightweight downstream Zero-Shot Evaluation (Tiny-Eval)

[aniruddhaadak80]

Problem

The success metric for the agent is currently pure val_bpb (validation bits-per-byte). While a great proxy, isolated cross-entropy loss can sometimes disguise models that have collapsed in specific linguistic capabilities or are cheating via strange formatting.

Proposal

Integrate a very lightweight zero-shot evaluation loop at the end of the TIME_BUDGET. This could test against miniature versions of MMLU, Hellaswag, or a simple QA format. Emitting a real "accuracy" percentage in results.tsv alongside val_bpb will give the AI a secondary, qualitative anchor to ensure its architectural hacks are improving actual model reasoning.

[MustakimShaikh01]

This makes sense. Relying only on val_bpb could hide regressions in reasoning or language understanding.

Adding a small zero-shot evaluation (even with tiny subsets of MMLU or QA datasets) would provide a more interpretable metric like accuracy. That would make it easier to verify whether architectural changes are improving real model capabilities rather than just lowering cross-entropy.