[SRBench 2024] Requirements for the implementations

[folivetti]

We need to decide what we are the requirements for the participants. What we agreed so far:

• They must provide fully running docker (based on a template) with srbench and the algorithm implementation
• The algorithm must have a scikit-learning like API, similar to the previous srbench
• It should have a method that returns a sympy compatible string representation of the expression

What we need to decide:

• They need to provide a time_limit argument in their API that runs the algorithm up to a specified time limit
• They need to write a file with the best of the generation or the whole population (if we decide to do that, the time limit argument is unnecessary)
• They need to provide a Pareto front (MOO only), this can be an optional track when analysing the results

[folivetti]

@lacava @MilesCranmer @gbomarito @hengzhe-zhang @gAldeia @foolnotion

[gAldeia]

I would also suggest asking participants two other requirements that should not be hard to achieve:

1. Thy should provide a param_grid for performing a grid-search (we can set a maximum limit of n configurations, e.g. 100), and
2. Having default values that are compatible with the previous SRBench version.

My reasons for that are the following:

1. Original authors have more in-depth knowledge of their algorithms and hyperparameter effects, so we are no better than them at defining a hyperparameter tuning search space. They should define the grid respecting a number of at most n possible combinations (for the sake of running the experiments), considering previous SRBench experimental setup (i.e., number of evaluations, execution time, etc.).
2. Having off-the-shelf capabilities, I believe, is a desideratum of people outside our community, and it could greatly improve the usability of our research. Our work in symbolic regression or evolutionary algorithms has the potential to impact other science areas. However, people from outside our community may find it challenging to understand each algorithm's hyperparameters, making our work less accessible. To address this, we could report the experiments in two ways: one with parameter tuning and the other with off-the-shelf performance.

[gAldeia]

All of the above goes hand in hand with having a sklearn interface. It makes everything easier to use (for both us and other users interested in SR)