Refactor evaluators #107
Open
Refactor evaluators #107
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the evaluation system to be more flexible and config-driven, removing the Julia integration, adding new model architectures (Sequential, GPSTransformer), and expanding test coverage. Key changes include:
- Renamed evaluators to
Evaluator,Validator,Testerwith configurable monitor tasks - Made early stopping optional and updated Trainer schema to support both constructor and evaluator config formats
- Added comprehensive tests for evaluators, sequential models, and GPS transformers
- Removed Julia worker integration and tests
- Updated dataset handling for Zarr stores without root groups
Reviewed changes
Copilot reviewed 34 out of 36 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
evaluate.py |
Refactored evaluators with DataFrame-based results and configurable monitor tasks |
train.py, train_ddp.py |
Updated trainer to use new evaluator names and optional early stopping |
test_evaluate.py |
Added comprehensive evaluator tests with custom monitors |
test_sequential.py, test_gps_transformer.py |
New tests for Sequential and GPSTransformer models |
test_trainer.py |
Updated tests for default evaluators and trainer configuration |
test_tune.py |
Refactored fixture to use yaml_config fixture |
conftest.py |
Replaced Julia-based fixtures with pure Python data generation |
config_utils.py |
Added support for list sweeps and numeric string conversion |
dataset_base.py, dataset_ondisk.py |
Updated to handle Zarr stores without root groups |
gnn_model.py |
Fixed pooling layer validation logic |
sequential.py, gps_transformer.py |
New configurable model architectures |
models/__init__.py |
Exported new Sequential model |
utils.py |
Added maybe_number utility |
pyproject.toml |
Removed juliacall, added huggingface-hub |
| Documentation files | Updated guides and API docs for new evaluator system |
| Config files | Added example configs for training |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| original_weights = [param.clone() for param in trainer.model.parameters()] | ||
|
|
||
| training_data, valid_data = trainer.run_training( |
There was a problem hiding this comment.
Variable training_data is not used.
|
|
||
| original_weights = [param.clone() for param in trainer.model.parameters()] | ||
|
|
||
| training_data, valid_data = trainer.run_training( |
There was a problem hiding this comment.
Variable valid_data is not used.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
depends on #101