-
Notifications
You must be signed in to change notification settings - Fork 97
Open
Description
When I run the same benchmark, once full and once a subset (to save execution time):
./bin/cub.bench.transform.babelstream.base -d 0 --stopping-criterion entropy --json base.json
./bin/cub.bench.transform.babelstream.base -d 0 -a "T{ct}=I16" -a "Elements{io}[pow2]=28" --stopping-criterion entropy --json new.json
and then compare the results:
PYTHONPATH=./_deps/nvbench-src/scripts python3 ./_deps/nvbench-src/scripts/nvbench_compare.py ./base.json ./new.json
['./base.json', './base_bif.json']
# Summary
- Total Matches: 0
- Pass (diff <= min_noise): 0
- Unknown (infinite noise): 0
- Failure (diff > min_noise): 0
I get an empty comparison results. I think the script should report the result for each benchmark which is present in both JSON files and only skip the ones present in only one JSON file.
The current workaround is to create a partial base.json as well, which is 100% unnecessary.
Metadata
Metadata
Assignees
Labels
No labels