Update evals for 5 envs #229
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dmahan93
reviewed
Aug 14, 2025
Collaborator
dmahan93
left a comment
dmahan93
left a comment
There was a problem hiding this comment.
next time please break this into individual PRs, fix the 'eval' removal and it's good to merge
|
|
||
| async def rollout_and_score_eval(self, question: str, answer: str) -> dict: | ||
| """Rollout and score evaluation with detailed sample data collection.""" | ||
| eval_temperature = 0.0 |
Collaborator
There was a problem hiding this comment.
this seems a bit weird want to just make eval_temperature a base config param?
Collaborator
There was a problem hiding this comment.
I do it in mine (though mine are not usually 0.0) - I think this is good though as with reasoning models you want a diff eval temp than non-reasoners or with the weird 3.0+ temp RL'ed model etc
Collaborator
There was a problem hiding this comment.
yeah I don't disagree but if we're changing stuff here may as well make it able to be set by the user
| n=1, | ||
| max_tokens=self.config.max_generation_tokens, | ||
| temperature=self.config.eval_temperature, | ||
| split="eval", |
teknium1
approved these changes
Sep 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Type
📝 General Information
Description
Add file-based eval logging to five environments - answer format, instruction following, tool calls, reasoning gym, letter counting.
Type of Change
Testing
python tool_calling_server.py evaluate \ --env.max_eval_samples $MAX_SAMPLES \ --env.data_dir_to_save_evals tc_test \ --openai.base_url "$BASE_URL" \ --openai.model_name "$MODEL_NAME" \ --openai.api_key "$OPENAI_API_KEY"(replacing
tool_calling_server.pywith any of the other 4 as well)