Skip to content

Allow running the benchmark workflow with different parameter sets #405

@rcannood

Description

@rcannood

The run_benchmark needs to be able to run methods with multiple parameter settings. Perhaps with the addition of something like this?

        - name: "--parameter_sets"
          type: file
          description: |
            A yaml file containing the parameter sets for each method. Example format:

            - method_id: ...
              run_id: ...
              parameters:
                ...
            - method_id: ...
              run_id: ...
              parameters:
                ...

If the run_id is not set, it defaults to the method_id. If no parameters are specified, it defaults to {}.

Alternative formats are possible, such as:

        - name: "--parameter_sets"
          type: file
          description: |
            A yaml file containing the parameter sets for each method. Example format:

           method_1:
             param_set1:
               par_a: ...
               par_b: ...
             param_set2:
               par_a: ...
               par_b: ...
           method_2:
             param_set1:
               par_a: ...
               par_b: ...
             param_set2:
               par_a: ...
               par_b: ...

The paramset_id / run_id would need to be stored in the .uns, and downstream components would need to be adapted accordingly, so let's think about how the api specs would change and which downstream components are affected.

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions