CI failure debugging

[liamhuber]

Uploading to coveralls has been spotty for a while, but four days ago the daily test started to fail consistently with:

 ======================================================================
ERROR: test_draw (unit.test_node.TestNode.test_draw)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyiron_workflow/pyiron_workflow/tests/unit/test_node.py", line 347, in test_draw
    p.unlink()
  File "/home/runner/work/pyiron_workflow/pyiron_workflow/cached-miniforge/my-env/lib/python3.12/pathlib.py", line 1342, in unlink
    os.unlink(self)
IsADirectoryError: [Errno 21] Is a directory: '/home/runner/work/pyiron_workflow/pyiron_workflow/start/cached-miniforge'

----------------------------------------------------------------------
Ran 224 tests in 40.683s

Everything is fine on my local machine, this test only produces the three image files and these get cleaned up ok. On the CI it really is putting conda env information inside the node directory. This isn't just an unlucky overlap of paths, in #631 I played around with it and it's consistently and exactly putting it where the node directory is created. Force-removing the whole directory with shutil.rmtree(...) does the trick (cf. #632), but really shouldn't be necessary.

What's more, even if I fix that, the docs CI job is still failing:

Run mkdir public_html
/home/runner/work/_temp/54c68941-f48b-40bf-89f5-3538d59d9f93.sh: /home/runner/work/pyiron_workflow/pyiron_workflow/cached-miniforge/my-env/bin/sphinx-build: cached-miniforge/my-env/bin/python3.12: bad interpreter: No such file or directory

That isn't part of the dailies, but at the latest was working two weeks ago when executorlib was bumped (#627).

So, it really looks like something took a wrong turn in the upstream CI env management exactly between 4 and 5 days ago.

[liamhuber]

The dailies are still running fine for other projects using pyiron/actions. Thankfully, there is one project under very active development in this period -- semantikon. And indeed, looking at the CI logs there, I can see that the push-pull was passing build-docs five days ago, but all the more recent runs fail to build-docs with (modulo pathing) the exact same message I'm getting here:

Run mkdir public_html
/home/runner/work/_temp/f1d7eaa1-50c6-4652-9a3e-59d120956195.sh: /home/runner/work/semantikon/semantikon/cached-miniforge/my-env/bin/sphinx-build: cached-miniforge/my-env/bin/python3.12: bad interpreter: No such file or directory

So four days ago, somebody broke something.

[liamhuber]

I really want to blame conda incubator, but we use v3 and the last time this was touched was way back in January for the 3.1.1 release.

[liamhuber]

Aha aha ahaaa, but their Mamba example started failing at the exact same as our CI -- 5 days ago!

[liamhuber]

From what I can see, they are asking for the same thing

...
   Do we need to update conda-build?
  /home/runner/miniconda3/condabin/conda install --name base mamba=1.5.*
...
     added / updated specs:
      - mamba=1.5
...

but now suddenly they get a solver error

...
  Warning: 
  LibMambaUnsatisfiableError: Encountered problems while solving:
    - package mamba-1.5.3-py312h9460a1c_2 requires conda >=23.9,<23.11.0, but none of the providers can be installed
  
  Could not solve for environment specs
...

No one has commented on this yet. Since miniforge should now be as fast as mamba, and it is only their mamba example which is failing, I'm going to experiment with simply not using mamba to see if that provides an easy way around the problem here.

[liamhuber]

@samwaseda because your doc tests are failing, and @niklassiemer just FYI because you do a lot of the infrastructure: here is an executive summary of what is going wrong with the CI:

Five days ago, everything was fine.
Four days ago:

• Our build-docs jobs started failing because of claims an interpreter could not be found (e.g. [on semantikon(https://github.com/pyiron/semantikon/actions/runs/14732644949/job/41350600739)])
• My pyiron_workflow dailies started failing because a folder unexpectedly had extra env data
• The conda-incubator/setup-miniconda Mamba example started failing because of a LibMambaUnsatisfiableError

I can't find anyone complaining about this -- i.e. I found nothing on google, there are no issues on conda-incubator/setup-miniconda, and when I poked around other conda/mamba repos explicitly I didn't see anything. However, I think the timing of this is too suspicious for these to be independent, and pyiron_workflow and setup-miniconda didn't have any changes to their own code-base between their working and failing states.

Given the breadth of problem, I suspect that four days ago someone in the conda-sphere pushed something even farther upstream that conda-incubator and broke stuff.

I dug into things here and nailed down my unit test problem as stemming from passing un-normalized environment names to the setup-minconda input argument activate-environment (i.e. foo_bar and ./foo/bar are both ok, but simply foo/bar breaks stuff).

I've already opened a patching branch on pyiron/actions, where we could fix this (I'm sure it's wonky for more than just pyiron_workflow, it just doesn't seem to be wonky in a way that breaks tests), but this doesn't resolve our docs issue and I don't even know where to start attacking that.

In light of all this, my current strategy is to sit and wait out the weekend, to see if whoever broke things upstream fixes them 🤞 I would ultimately still like to polish and merge the actions PR as it pulls in other useful updates from setup-miniconda, but at the moment I'm not even sure where to start debugging the broken docs jobs.

[liamhuber]

After sleeping on it, prepending the ./ to the paths is not bad in and of itself. I then thought to move on to getting stuff switched from coveralls to codecov, but I'd really rather do that in a CI environment that is otherwise successful. So rather than simply waiting, I'm going to take a crack at fixing the docs build in #637 after all.

[liamhuber]

Well, ok, that was embarrassingly simple. pyiron/actions#145 just also fixes the docs problem, as seen in #637.