Add benchmarks framework #901
Conversation
benchmarks/README.md
Outdated
| uv sync --extra benchmark | ||
|
|
||
| # Run all benchmarks | ||
| uv run pytest benchmarks/ |
There was a problem hiding this comment.
We've standardized on hatch as the main entry point for everything else in this repo. How would someone run these tests through hatch?
There was a problem hiding this comment.
Modifying pyproject.toml to install the benchmark extras in the test env and then running hatch run test:pytest benchmarks/ did it.
It would be nice to have an alias, though, something like hatch run benchmark.
There was a problem hiding this comment.
You're right, missed that. I'll update it (although I think migrating to uv might be a good idea and should be fairly simple, but that's a separate discussion / task).
It's still a very initial draft I just completed and wanted to push so it's backed up and not just stored locally.
I'll give a more detailed explanation about the design and workflows once it's ready for review.
There was a problem hiding this comment.
OK, sounds good.
Running the tests gives me error: unrecognized subcommand 'download' from uv pip download. Maybe I need a newer uv?
$ uv --version
uv 0.9.22 (Homebrew 2026-01-06)
I had been very focused on profiling tools to find hotspots in the code, but I like the idea of having tests like these that we can run as we refine those sections, so thank you for working on this.
There was a problem hiding this comment.
Looks like an LLM hallucination I need to fix as well (this sub-command doesn't appear to exist) :)
Used AI to write everything for the first phase (with a design researched and reviewed manually), and now I'm at the phase of reviewing everything it wrote and refining / fixing the actual code files (also testing the new GitHub Action pipeline on this PR).
The hope here is to utilize CodSpeed (available free for open-source projects. Used by teams of popular projects like Astral and Pydantic) for easy tracking and visibility of performance metrics over time (or more accurately, over commits / merged branched), while having agnostic benchmarks that don't rely on CodSpeed and are easy to migrate if needed.
You can see an example of the metrics for Pydantic's repo here:
https://codspeed.io/pydantic/pydantic
Add performance benchmarking infrastructure for fromager including GitHub Actions workflows for nightly and on-demand runs. Signed-off-by: Michael Yochpaz <myochpaz@redhat.com>
MichaelYochpaz
force-pushed
the
add-benchmarks-framework
branch
from
8782f26 to
105874a
Compare
|
Will recreate the PR from a fork |
|
|
Add performance benchmarking infrastructure for fromager including GitHub Actions workflows for nightly and on-demand runs.
See the newly created README for full information.
Closes #734