Skip to content

INI benchopt benchmark to run sbibm easily#75

Open
tomMoral wants to merge 1 commit intosbi-benchmark:mainfrom
tomMoral:BENCH_using_benchopt
Open

INI benchopt benchmark to run sbibm easily#75
tomMoral wants to merge 1 commit intosbi-benchmark:mainfrom
tomMoral:BENCH_using_benchopt

Conversation

@tomMoral
Copy link
Copy Markdown

After discussion with @janfb and @dgedon, I started to work on a benchopt version to run the sbibm benchmark.

benchopt is a framework to organize, run and share benchmarks, that makes it easy to run, maintain and extend benchmarks.
Here, I just propose a proof of concept so we can have a discussion about it. The goal is not to replace sbibm but to complement it with a set of tools to run the evaluation.
The current design uses tasks directly defined in sbibm, fixes some training samples, and only evaluate the c2st. But this can change depending on the evaluation

The benefit of benchopt is that:

  • it provides some config files and allow to run the benchmark in parallel on large scale infrastructures.
  • It provides caching for the runs that were already run.
  • it allows to provide an interface to visualize the results. The interface is customizable.
  • It makes it easier to manage dependencies on a per-solver basis
  • It sets up a set of tests to make the benchmark maintanable in the long run.

Note that benchopt can also evolve to add features that are necessary, as this would probably help other benchmarks too.

This version is inspired of previous work by @JuliaLinhart, @francois-rozier and @MouzaouiMatthieu in https://github.com/JuliaLinhart/benchmark_sbi/, which was targeting a benchmark accross framework.

Co-authored-by: JuliaLinhart <julia_linhart@yahoo.com>
Co-authored-by: francois-rozet <francois.rozet@outlook.com>
Co-authored-by: MouzaouiMatthieu <mouzaouimatthieu@gmail.com>
Co-authored-by: agramfort <agramfort@fb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant