List view
**Requirements** Monitor the execution time of the tests, such as for XRADIO tests that seem to be taking longer. Codecov does provide some information, but it is only a single average number. Something like air speed velocity (as Andrew set up for CASA) would be great, allowing us to have graphs that show execution time per commit. Possible options to look into: - airspeed velocity (asv) - pytest-benchmark - Benchopt - Conbench **Note** Monitoring the runtime of existing tests is different from benchmarking the code that the test is testing. Mostly because in general, test cases apply assertions and comparisons that can affect the true runtime of the function it is testing. Therefore it is best to have separated performance tests that only call the function and does not do any assertion or analysis. Libraries such as **numpy** have separated benchmark tests for ASV in the same way that CASA has. The only drawback is that API changes to the functions will need to be updated also on the benchmark tests. From the suggested list of options for benchmark tests, ASV and pytest-benchmark seem more fit to XRADIO and could be explored first. We have good experience with ASV that could be the first one to use for this experiment. pytest-benchmark is simple to implement and we could also create a small prototype using it. **ASV** ASV stands as the definitive benchmarking framework for scientific Python projects (numpy, scipy, astropy, CASA), offering sophisticated commit-to-commit regression detection. ASV builds comprehensive performance histories across Git commits using binary search regression finding and statistical significance testing. The framework excels at tracking five benchmark types crucial for I/O libraries: timing benchmarks for execution performance, memory benchmarks for object size tracking, peak memory benchmarks monitoring maximum resident memory, raw timing benchmarks measuring import overhead, and custom tracking benchmarks for domain-specific metrics. ASV's dual-repository architecture separates concerns effectively - the main repository contains source code and benchmark definitions while an archive repository stores results and deploys GitHub Pages dashboards automatically. It also provides configuration for I/O-heavy scientific computing that requires careful setup of conda environments with complete dependency matrices, extended install timeouts for complex packages, and setup_cache methods to avoid repeated expensive data generation. It is not possible to use the same existing XRADIO tests to collect ASV benchmarks because ASV requires naming prefixes for test discovery **pytest-benchmark** pytest-benchmark provides minimal-friction integration with existing pytest test suites, requiring only fixture injection rather than separate benchmark files. The framework offers sophisticated calibration and statistical analysis with automatic grouping into statistically significant rounds, comprehensive metrics including operations per second and outlier detection, and built-in comparison capabilities for regression detection. pytest-benchmark focuses on single-session testing and does not build a history per commit. Memory profiling integration comes through complementary tools like pytest-monitor for comprehensive resource tracking, pytest-memray for advanced allocation analysis with native library support, and custom monitoring fixtures for domain-specific requirements. These combinations provide complete coverage of runtime and memory performance without abandoning pytest workflows. For projects already using pytest-benchmark, **RapidsAI**'s solution provides immediate integration: - asvdb: Programmatic library for ASV database manipulation - rapids-pytest-benchmark: Plugin extending pytest-benchmark - command: `pytest --benchmark-asv-output-dir=./asv_results` This approach converts pytest-benchmark results directly to ASV format without function renaming. **Conclusions** The benchmark tests have been setup for XRADIO and AstroVIPER using ASV. The tests are implemented in the repository https://github.com/casangi/benchviper
No due date•1/1 issues closedCreate a report or slides summarizing the experiment with testspace and make recommendations.
No due dateLearn and experiment with the testspace tool (https://www.testspace.com) to collect test results from TestVIPER Actions and display them in the testspace dashboard. Explore the available metrics in the dashboard and report the results.
No due date•2/2 issues closedCreate workflows for minimum build and test that passes in GitHub Actions. The first setup should be as minimum as possible to create separated yaml files for: build: install current release of relevant components from PYPI and testviper dependencies test: run existing integration tests from testviper:main publish: create coverage reports in html that can be visualized in codecov
No due date•3/3 issues closed