Simultaneous Scenario Comparison Feature

[philmard]

Hi,

I'm developing an Interactive Simulation Environment for an Integrated Assessment Model using the SDEverywhere tool, and I'd appreciate your advice on an important feature, as experts in SDEverywhere's codebase.

The feature I want to incorporate in the ISE is this:

I'd like users to be able to compare the results of two distinct input configurations (scenarios) in a single graph.
(!) This isn’t about QA tools like comparisons.yaml; rather, it’s about letting users tweak two different sets of inputs at once and observe the results in a single combined graph, where eg. one line in the graph corresponds to scenario1's output, and another line corresponds to scenario2's output.

I'm currently implementing an approach, where I basically initialize two model instances (instead of one) in index.js when the app loads:

model1 = await createModel();
model2 = await createModel();

I've also extended the CSV parsing logic of the package "@sdeverywhere/plugin-config" so that more columns are parsed in graphs.csv.

This is an example of how graphs.csv looks like:

id, ... , scenario, ... , scenario mode, ...
id1scenario1, ... , Scenario1, ... , separate, ... // separate graph
id1scenario2, ... , Scenario2, ... , separate, ... // separate graph
 
id2scenario1, ... , Scenario1, ... , combined, ... // these will be combined
id2scenario2, ... , Scenario2, ... , combined, ... // in a single graph

(!) the "scenario" column links each row to its corresponding scenario.
(!) "scenario mode" is useful, because having the two scenarios on a single graph is not always desirable; sometimes we want them to be on two separate graphs. (eg. when using stacked area charts)

There's a handful of additional changes, but I won't go into more detail here.
(if -- for any reason -- you want to have a deeper look into the code, it's hosted in branch "dashboard" of the following repo: https://github.com/ntantaroudas/choice-web/tree/dashboard)

The above approach works (and it is -- rather -- simple in implementation) but it creates some inefficiencies that I'd preferably avoid. I'll try to explain briefly.

In the ideal solution:

• model1 would generate output variables only for scenario 1, and
• model2 would generate output variables only for scenario 2.

However, in the current implementation, model1 generates output variables for scenario 1 AND scenario 2, and model2 does the same. As a result, instead of having (ideally) 2x output variables (one set for each scenario), we end up having 4x output variables. (We have 2 extra sets of outputs that are unnecessary and are not shown in the graphs at all.)

So, the current approach doubles resource usage compared to an ideal solution, as each model instance processes both scenarios.

(!) I've also tried utilizing a single row in graphs.csv for both scenario1 and scenario2, but I can't think of a way to "split" a single graphId into two separate running models.
(!) I've also tried utilizing only a single model instance for this, by messing with ModelScheduler, ModelRunner, without much success.
(!) I've recently taken a deeper look into "@sdeverywhere/runtime-async" and "@sdeverywhere/runtime" packages -- especially functions like initModel, initWasmModel, and the WasmModel class --, but I'm hesitant to extend or modify those packages the way I did with "@sdeverywhere/plugin-config".

The questions I have for you (and feel free to be as broad or as specific/technical as you want) are the following:

• How important do you think it is -- if at all -- to solve this "double resource" problem? Will the app significantly slow down when multiple more variables get introduced? (At the moment, we're using roughly 100 output variables)
• Is there something in the SDEverywhere architecture that I'm missing, which would make this type of "scenario comparison" easily and intuitively implementable, without modifications to low‑level or package code?
• What would be your recommended approach to implement this type of “scenario comparison” in an ideal (computation-wise) way?

Here's a link to the ISE: https://ntantaroudas.github.io/choice-web/

Thank you very much for your time and any pointers you can share!

[chrispcampbell]

Hi @philmard, thanks for your detailed message; you've come to the right place and I'd be happy to help! I was really glad to hear that your project is using SDEverywhere. It's always cool to see different kinds of UIs and use cases built on top of SDEverywhere.

It just so happens that we've had some code to support one of our internal projects that should help with your situation. I had plans to migrate that code into the public SDEverywhere repo and finally got back to working on that last week. I'm hoping to wrap up the changes today or early tomorrow and will post a more detailed reply soon after. I'll also provide answers to your questions at that time so that I can reference the newly added runtime API and sample code. Stay tuned!

[chrispcampbell]

@philmard: I published new versions a couple packages, along with a new Svelte-based project template/demo, to help support your use case.

• I added a new MultiContextModelScheduler class to the @sdeverywhere/runtime package. This class allows you to more efficiently have multiple contexts (input/output scenarios) in your app while using the same underlying model instance. See Add model scheduler variant that supports multiple input/output contexts #625 for details.

• I updated @sdeverywhere/plugin-config so that you can now set the "plot N source" in graphs.csv to something like "Scenario1" or "ScenarioN", and it will get treated like a normal output variable, but allows your app to choose an output associated with a particular scenario/context. See Update plugin-config to handle graph datasets that have a "Scenario" source #626 for details. (It sounds like you may have forked plugin-config to suit your own purposes, which is a perfectly valid thing to do. Take a look at the PR to see the change I made and pull that into your fork if needed.)

• I added a new Svelte-based project template that also demonstrates how to use multiple input/output contexts. See Add Svelte-based project template #629 for more details and screenshots.

To try the new template, you can create an empty folder and run the following command (it'll set it up with the SIR example model by default; you can hit enter to select the default option for each question):

mkdir my-project
cd my-project
npm create @sdeverywhere@latest -- --template svelte
npm run dev

By the way, I would recommend this Svelte template (and Svelte in general as a web framework) as it's much more extensible than the jQuery-based "default" template. The main reason we only had the jQuery-based template before is that I wanted to have a very simple template to avoid having lots of moving parts for newcomers. As you've seen, it all fits in a single js file and is meant to be conceptually simple, but (as you also may have seen), it gets unwieldy to grow into a more complex application. The Svelte template is more in line with how we develop En-ROADS and other projects, and though it has more files, I think it has a better and more extensible/maintainable/understandable structure.

To answer your earlier questions:

How important do you think it is -- if at all -- to solve this "double resource" problem? Will the app significantly slow down when multiple more variables get introduced? (At the moment, we're using roughly 100 output variables)

I think it's always a good idea to keep an eye on performance and memory usage, but it really depends on your model:

• The performance of your generated model will depend on the size and complexity of the model.
• The memory use of your generated model will depend on the number of outputs, the time span, and your "saveper" value. For example, if you have 100 outputs, and your model runs from 1950 to 2100, and you save one value per year, that's 15000 values. Let's say 8 bytes per value, that's 120,000 bytes.

That's not too much, but if you have more outputs, or you use a smaller "saveper", or you have multiple copies of the model in memory, or your model gets more complex, then maybe you'll start to notice it. Also, if you generate a WebAssembly model, that may require more memory than a plain JavaScript model. The former will usually run faster, but it may require more memory.

All of that to say, it depends :)

Is there something in the SDEverywhere architecture that I'm missing, which would make this type of "scenario comparison" easily and intuitively implementable, without modifications to low‑level or package code?

Before I added this new MultiContextModelScheduler class, the next best alternative would be to initialize multiple instances of the Model in your "core" package in your project, but it would require more modifications.

If you take a look at the new Svelte template, it uses MultiContextModelScheduler under the hood, and the app is also set up to show how to work with multiple contexts (like, having multiple groups of sliders, and showing multiple scenarios in graphs, either in the same graph or in multiple graphs).

What would be your recommended approach to implement this type of “scenario comparison” in an ideal (computation-wise) way?

Same answer as above.

Apologies for the verbose reply! Please let me know if anything is unclear, or if you have any issues with the new APIs or examples. (Or if you have any suggestions or comments in general.)

[philmard]

Thanks a lot! This is more than enough. I hope it wasn't too much trouble!

[chrispcampbell]

@philmard: No trouble at all! I was glad to have a real-world use case to help motivate me to finish up those branches finally :)

[travisfranck]

@philmard I'm curious what IAM your using. It would be great to know the name and/or subject (climate, health, energy etc) that of it. And you're able to share what group it is. Small effort at building a SDE community. ;). Thanks.

[RyanTanYiWei]

@travisfranck Hi! I'm Philip's colleague, working on the same project and have been lurking on this thread. It's going to be based on a global IAM FeliX model. The scope of the SDE tool is not completely decided since we are still in its preliminary stages, but it will generally revolve around the food system and demand-side mitigation options. Some public information about the project and the consortium at large can be found here. I'd be happy to share more once the project is more firmly established; for now, I can only whet your interest :)

[travisfranck]

That project looks great and FeliX is a nice model to start from. Seems like you've got a lot of good partners already, but I'm happy to have a chat about the project, if you'd like. I have experience in IAMs, land, food, etc.