Improve and adapt notebooks with latest environment and optional wildcards

[huyhoang-mike]

This PR resolves the following issues:

• Rerun and check hydrogen_market.ipynb, hydrogen_financing.ipynb, and network_analysis_sec.ipynb.
• Update notebooks latest pypsa-earth environment (Python 3.10.15 to Python 3.11.13) and other libraries.
• Added optional wildcards to the network string builder.

[huyhoang-mike]

Hi @doneachh, I've updated the hydrogen_market.ipynb to the new pypsa-earth env and enabled the optional wildcards. It would be nice to hear your review :) Then I can proceed with other notebooks using the same approach. Basically, the notebook would search for all matching networks to the config.yaml file and choose one from them to analyse the results. The priorities are sector networks -> prenetworks -> electricity networks if it finds multiple networks.

[doneachh]

@huyhoang-mike hydrogen_financing.ipynb, hydrogen_market.ipynb, network_analysis_sec.ipynb do work perfectly fine!

When trying to run network_analysis.ipynb with a sector coupled network, i get some errors. Can you try to adjust this notebook, that you can analyze sector coupled as well as electricity only networks? :)

[huyhoang-mike]

Hi @doneachh, I have created another PR-1625 which is related to the missing tech_colors in this PR. If it is the right approach, we can then remove those manual color handling in the network_analysis.

[doneachh]

@ekatef can you merge the two PRs? :) thank you!

[huyhoang-mike]

Hi @doneachh, I would like to just add some heads-up:

• I have tried some configurations to model the sector-coupled network and could not reproduce those carriers with missing tech_colors as you described in the network_analysis. For example, 'lignite' and 'coal' are already in the config.default.yaml, but they are missing in your network. My assumption is your network is modeled a long time ago that the config file didn't have many tech_colors, isn't it?
• If it is the case, I would suggest removing all manual color handling cells before merging this PR. Otherwise, we would override the default colors with the current code.

[doneachh]

Hi @doneachh, I would like to just add some heads-up:

• I have tried some configurations to model the sector-coupled network and could not reproduce those carriers with missing tech_colors as you described in the network_analysis. For example, 'lignite' and 'coal' are already in the config.default.yaml, but they are missing in your network. My assumption is your network is modeled a long time ago that the config file didn't have many tech_colors, isn't it?
• If it is the case, I would suggest removing all manual color handling cells before merging this PR. Otherwise, we would override the default colors with the current code.

@huyhoang-mike exactly. Feel free to go ahead and tag Katja as soon as the PR is ready to merge.

[huyhoang-mike]

Hi @ekatef, this PR has been completely finished. It would be nice if you can review it :)

[ekatef]

Hello @doneachh and @huyhoang-mike!
Thank you for taking care about keeping the notebooks up-to-date!

@huyhoang-mike it looks like there may be still something wrong with network_analysis:

• the plot of current capacity of the energy system seems to present load shedding and CCGT in a very similar way which can easily lead to a confusion (since load shedding itself isn't a perfectly clear concept)
• it appears that installed capacity is NaN for Onshore Wind
• plots are empty for the potential density

I see that cells are numbered in network_analysis from 229 which could probably be a reason of the data issues. @huyhoang-mike could you please re-run network_analysis and make sure those issues are resolved?

[ekatef]

As an additional technical comment, the implementation includes quite sophisticated system of searching files including management of wildcards. I wonder if it may be a good idea to store those helpers scripts in a dedicated file and import them into all the notebooks where they can be needed? It would make the notebooks much lighter and facilitate maintenance significantly thanks to reducing duplication

@huyhoang-mike would you be interested to work on that?

[huyhoang-mike]

@ekatef yes, I'm totally up for this topic. Thanks for your suggestion! Modularizing the network searching process would make the notebook way simpler and more approachable for these tutorials. I will add a helper script in the next PR :)

[ekatef]

@ekatef yes, I'm totally up for this topic. Thanks for your suggestion! Modularizing the network searching process would make the notebook way simpler and more approachable for these tutorials. I will add a helper script in the next PR :)

Hey @huyhoang-mike, happy you like the idea! I think it could be a great next step.

In the meantime, would you mind to add fixes to network_analysis, so we can merge this PR? I mean the points which seem to need attention mentioned here #107 (comment)

[huyhoang-mike]

Hi @ekatef, thanks a lot for your comprehensive review! I have discussed with @doneachh, and solved the mentioned concerns in the #107 (comment). Here is the summary:

• I have modified the input data and potential density is no longer empty.
• Installed capacity for Onshore Wind seems to be NaN since the value is too minor. I've changed the code to filter out those small values which end users cannot see in order to avoid confusion.
• Excluded load shedding from the energy system as it is not an actual generation source.

If everything is fine now, I will proceed with the network searching helper in the next PR. Thanks for your wonderful collaboration :)

[doneachh]

@huyhoang-mike @ekatef Hi!
just wanted to ask whats the status here as some of my students are using these notebooks, it would be nice if we could merge it soon.

@huyhoang-mike are the notebooks running for sector coupled + electricity only results of the newest main?
@ekatef did you have a look on the modifications? :)

[doneachh]

Feel free to contact me in case I can assist :)

[ekatef]

Hey @huyhoang-mike apologies have completely missed your response in December (for future, please feel free to ping us if you don't get answer on the revisions during a few days). I think network_analysis looks great now, and filtering by a threshold makes sense. Thank you so much for introducing changes!

For network_analysis_sec and hydrogen_market I see cells numbering starts from [20] which can potentially lead to some confusion. Could you please re-start both of them and commit the changes? After that, I think the PR is ready to be merged 🚀

[huyhoang-mike]

Hi @ekatef, I’ve rebased and then rerun the two notebooks you mentioned and fixed the execution‑count issue. While testing the hydrogen_market notebook, I also ran into a column‑name mismatch, so I modified that part to make the logic more generic. Additionally, I corrected the revenue calculation: previously, negative export quantities resulted in negative revenue, but I adjusted it so that revenue is zero in such cases, which makes more sense in my opinion.

Let me know if this works for you :) Then we can go ahead and merge it.

[huyhoang-mike]

Hi @doneachh, the network_analysis has been adapted to both sector-coupled and electricity. If your students face any problem with one of these notebooks, I'd be happy to further improve it then :)

[ekatef]

Hi @ekatef, I’ve rebased and then rerun the two notebooks you mentioned and fixed the execution‑count issue. While testing the hydrogen_market notebook, I also ran into a column‑name mismatch, so I modified that part to make the logic more generic. Additionally, I corrected the revenue calculation: previously, negative export quantities resulted in negative revenue, but I adjusted it so that revenue is zero in such cases, which makes more sense in my opinion.

Let me know if this works for you :) Then we can go ahead and merge it.

Hey @huyhoang-mike, thank you so much for the update. I think the changes make perfect sense, and the contribution looks great. I think we are ready to merge unless @doneachh has any further comments.

[ekatef]

A maintenance note: a default GitHib render seems be incorrect for rebase. The commits which are still in main are still highlighted in changes, while git diff shows that PR modifies those and only those files which it's intended to modify (checked locally with Sublime Merge):

[doneachh]

Hi @ekatef, I’ve rebased and then rerun the two notebooks you mentioned and fixed the execution‑count issue. While testing the hydrogen_market notebook, I also ran into a column‑name mismatch, so I modified that part to make the logic more generic. Additionally, I corrected the revenue calculation: previously, negative export quantities resulted in negative revenue, but I adjusted it so that revenue is zero in such cases, which makes more sense in my opinion.
Let me know if this works for you :) Then we can go ahead and merge it.

Hey @huyhoang-mike, thank you so much for the update. I think the changes make perfect sense, and the contribution looks great. I think we are ready to merge unless @doneachh has any further comments.

@ekatef from my side no further comments :)

[ekatef]

Merged! 🎉 🎉 🎉
@huyhoang-mike Thank you so much for getting the update done. I think it's really a great improvement 🥇
@doneachh thank you for the fantastic supervision! 💚

[huyhoang-mike]

@ekatef My pleasure 🎉 Thanks a lot for your reviewing and helpful feedback.