Resolve ubuntu-latest+pip issue

[liamhuber]

Problem: running pip install... breaks on the runner "ubuntu-24.04" with an "This environment is externally managed".

#135 patches over this by setting the default runner from "ubuntu-latest" back to "ubuntu-22.04". This is a temporary solution.

There is a live issue for this upstream: actions/runner-images#10781

If this is resolved by the image provider such that we can use pip again, then the default should be reverted to "ubuntu-latest". Otherwise, we'll need to come up with another solution in the direction of venvs so our pip install necessities work (and on all platforms), and that they continue to play well with where we conda install things.

@mbruns91, I'm sorry, I thought this was a bug in the image release, but it looks like a fundamental change in the ubuntu policy as of the 2023 release. I dismissed your other suggestions in favour of just waiting out the image fix, but if this is not a bug but a legit change we're going to need them... Want to copy your favourite(s) here?

[liamhuber]

There's still a chance that they'll update the image to come with a non-system python interpreter as the default, so I don't think we need to act on this quite yet, but let's be prepared with an outline of how we would solve it if they don't provide such convenience.

[mbruns91]

So, my preference would be to expose the used runner everywhere as a non-required input, defaulting to ubuntu-22.04. My reasoning for why I don't favor using ubuntu-latest I have written down here; wrapped up I find it cleaner and less ambiguous.

Approaches

There are numerous options we have to deal with the issue of "python being externally managed":

1. Install packages via the runners native package manager
   This means apt/brew/choco install <package> for Linux/MacOS/windows. This has the downside that the package actually needs to be packed for the resp. manager, which will not always be the case.
2. Install packages into a venv managed by the runner's user account
   The easiest way to do this would be using the pipx package (which, in turn has to be installed via the system's native package manager, but that's working as far as I know), because we avoid actually having to set up/select/manage/... a venv via shell commands. This solution would certainly require some testing, especially if we really care that our actions are by default running on all 3 OSes. I have no hands-on experience with using pipx.
3. Mess around with system python
   Ubuntu 24.04 is doing the whole "dare-you-pip-install-systemwide" to protect system-python, as it is more and more central for the system running stable. For any client/ server, where actually work is done on "manually", this is really good practice¹. However, I don't see there is a big issue for a container being fired up, running some notebooks or shell-commands and then being killed again. I guess, typically the packages we install in CI/CD tasks are not breaking any system. So we could simply do pip install <package> --break-system-packages and see how that works out.
4. Use the setup-python-action from github
   Maybe this one is the way to go. This actions installs a selectable version of python and pypy and adds it to $PATH (or $PYTHONPATH? The doc says the former.). This way we can also for tasks, where we normally would not use conda specify the python-version unambiguously.
5. Use conda/mamba
   This makes a lot of sense, since we mostly use conda for dealing with Python packages in other places.

How to keep the default runners updated?

Somehow, when we pin the default runner to specific versions, we also should have a look how to automate updating them.
Collection of approaches:

1. Use dependabot
   This would be the preferred solution. However, a request for this feature is an open issue on the dependabot-core repo since Feb. 2023. I'm not sure how confident we should be, that this is addressed soon(ish)².

Any of these solutions requires some more involvement or at least awareness, so we should mention its details and our reasoning somewhere prominently.

Footnotes

1. This is actually a thing; I once screwed up a system by messing with pip install ... or even worse sudo pip install .... ↩

2. This translates to: I don't think it will be addressed within a time-frame someone reasonable would call "soon". ↩

[mbruns91]

Just as a note: I'll edit my post from above, whenever I find another way to approach this.

[liamhuber]

@mbruns91, just FYI they patched a workaround for this, but intended to remove the patch next week: actions/runner-images#10897. I intend to check back there to see if they present a palatable "suggested" way to accomplish this.

From what I can see, the only places we use pip are inside cached-miniforge and update-pyproject-dependencies. Almost all our other environment modification here goes through cached-miniforge. I would be open to moving back to "ubuntu-latest" and simply putting the necessary "fix" in these two scripts (probably a setup-python job invocation).