INI benchopt benchmark to run sbibm easily#75
Open
tomMoral wants to merge 1 commit intosbi-benchmark:mainfrom
Open
INI benchopt benchmark to run sbibm easily#75tomMoral wants to merge 1 commit intosbi-benchmark:mainfrom
tomMoral wants to merge 1 commit intosbi-benchmark:mainfrom
Conversation
Co-authored-by: JuliaLinhart <julia_linhart@yahoo.com> Co-authored-by: francois-rozet <francois.rozet@outlook.com> Co-authored-by: MouzaouiMatthieu <mouzaouimatthieu@gmail.com> Co-authored-by: agramfort <agramfort@fb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After discussion with @janfb and @dgedon, I started to work on a
benchoptversion to run thesbibmbenchmark.benchoptis a framework to organize, run and share benchmarks, that makes it easy to run, maintain and extend benchmarks.Here, I just propose a proof of concept so we can have a discussion about it. The goal is not to replace
sbibmbut to complement it with a set of tools to run the evaluation.The current design uses tasks directly defined in
sbibm, fixes some training samples, and only evaluate thec2st. But this can change depending on the evaluationThe benefit of
benchoptis that:Note that
benchoptcan also evolve to add features that are necessary, as this would probably help other benchmarks too.This version is inspired of previous work by @JuliaLinhart, @francois-rozier and @MouzaouiMatthieu in https://github.com/JuliaLinhart/benchmark_sbi/, which was targeting a benchmark accross framework.