Benchmarking Models

[JamesPHoughton]

It would be good to have a set of test models of varying complexity that can serve as benchmarks for speed tests. Then, as development progresses, we can run the benchmarks on old and new versions, and see if the code is getting faster. Also, we can see just how much slower PySD is than SDEverywhere. (I dread to think.)

I don't know much about benchmarking, but I expect there are some well understood best practices we could implement. I imagine we want a dozen or so different models of different complexities and sizes that exercise the range of behaviors we see in SD models. Ideally, they would be models the community understands, so they would be easy to communicate about. Maybe World3, the market growth model, the beer game, The C-Roads model, etc. would be good candidates. Tom Fiddaman has some of these on his blog - https://metasd.com/model-library/ - we could ask his permission to use them.

I'd propose putting these models together in a /benchmarks/ directory, and possibly listing the number of variables, subscript dimensionality, etc. associated with each one.

@enekomartinmartinez, @alexprey, @julienmalard, @ivan-perl, @travisfranck, what are your thoughts?

[julienmalard]

Excellent idea! I think that would help me a lot in finalising the dimensions functionality that I had been working on a while back (I didn't complete the pull request because I had seen that my new code based on pure xarray instead of xarray inside of numpy arrays was significantly (over 10x) faster in the case of a large, multidimensional model but at the same time slowed down a simple model with no dimensions quite a bit). I could contribute these models to the test directory if you would like. I myself know very little about how to run a benchmarking suite; would it be similar to the regular test suite? Is this something we could integrate into CI?
Please let me know how to participate! I am very enthousiastic about this idea.

[enekomartinmartinez]

I think it is really necessary to do that, and it will be really useful.
We should include a wide range of models from simple and fast to complex and slow. We have to be careful because some models may not be able to run with the latest versions of PySD or SDEverywhere, and we will not be able to assess changes in performance...

Moreover,I think it is also interesting to create or use specific tests for processes that may require more time. For example, you can run for a considerable time a test of Delays (for example tests/subscripted_delays/test_subscripted_delays.mdl) to see if it has improved the performance in this particular case.

By the way, I have also no experience in how we should implement that in PySD, so I am open to suggestions.

[JamesPHoughton]

Different types of complexity are also probably important. For example, these may all have different behavior, and are likely to be slow in different places:

1. A model with 100 independent stocks
2. A model with a single 100-order delay
3. A model with 10 stocks, each of which is dependent on the other 9
4. A model with a single stock of subscript dimensions [1x100]
5. A model with 10 stocks of subscript dimensions [1x10]
6. A model with 10 stocks, for 10 times the time horizon (or 1/10th the timestep)

I suppose there is no reason why all passing integration tests (from the tests folder) should not also be included as a performance benchmark, so long as you run the same subset of tests to compare across versions...

[travisfranck]

I encourage people to use the SDX.org models along with the models we have in the SDEverywhere.org repo. We should try to build a single repo that we can all use.

Also, @enekomartinmartinez @julienmalard have you considered working with SDE code to output Python code? SDE has functions that "emit" the C could. Those functions could be rewritten to output python code. The benefit is that subscripts, subscript ranges, dead code trimming and everything else has already been implemented.

[julienmalard]

Thanks for the suggestion @travisfranck ! I am eventually trying to build a software (and interface tool, in JS) that would allow for the construction and simulation of SD models. I really like the idea of editing SDE to output Python (and WebAssembly) code; my main question would be whether this could work dynamically (e.g., from a user-built model) through pure JS functions without needing to recompile using MacOS and XCode. Please do let me know if this would be possible!
Many thanks.

[travisfranck]

Apologies for the delay... We have SDEverywhere running in GitHub Actions, so I think the answer is that it should be easy to have SDE help dynamically behind a JS tool. Put another way, we don't use Xcode or macOS as part of our daily tools. (The installation directions mention Xcode because that is the easiest way to install some developer tools on macOS.)

Also, you might want to connect with the people at sdcloud.io, who are also building a SD tool that I think is web-based (and likely using JS).

[julienmalard]

Thanks @travisfranck ! I will check.