Eval: loss-only path + compute receipt#27
Open
StanByriukov02 wants to merge 1 commit into1x-technologies:mainfrom
Open
Eval: loss-only path + compute receipt#27StanByriukov02 wants to merge 1 commit into1x-technologies:mainfrom
StanByriukov02 wants to merge 1 commit into1x-technologies:mainfrom
Conversation
This adds a fast path for CE loss evaluation that skips MaskGIT refinement sampling and the decode/LPIPS pipeline when you only care about loss. Also prints compute_logits_calls (and per-frame) so you can see exactly how many full forward logits passes happened. Notes: - Guaranteed: with --loss_only, compute_logits calls per generated frame drop from ~maskgit_steps to 1 (so maskgit_steps=2 is ~2x on the GPU-compute-heavy part). - Scales: if someone runs maskgit_steps=K for quality/sampling, the compute ceiling is ~Kx. - Not promised: end-to-end Joules/frame depends on your full eval setup; I only measured a small forward-pass microbench on H100 as a sanity check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was evaluating GENIE-style checkpoints and kept running into the same thing: sometimes I only need CE loss (teacher-forced logits), but the current eval loop still does MaskGIT refinement + decode/LPIPS/acc work.
This PR adds a clean fast path for that case.
What changed
Guarantees / scaling / what I’m not claiming
H100 sanity check (forward-pass microbench)
Repro (receipt)
python genie/evaluate.py --checkpoint_dir --maskgit_steps 2
python genie/evaluate.py --checkpoint_dir --maskgit_steps 2 --loss_only
You should see compute_logits_calls_per_frame drop ~2.00 -> ~1.00 for maskgit_steps=2.