Give precedence to $ORIGIN locations in RPATH wrappers

[Crivella]

Fixes:

• Potential wrong order in default include dirs for rpath wrappers #5037

[Crivella]

Builds that worked with this branch checked out (while having an existing installation of that software that would previously cause trouble in the tests)

• {tools}[foss/2023b] ADIOS2 v2.10.2 easybuild-easyconfigs#24316 (comment)
• {tools}[foss/2023b] ADIOS2 v2.10.2 easybuild-easyconfigs#24316 (comment)

[lexming]

Following-up on the discussion in the conf call:

• $ORIGIN can be any subfolder in the install/build/test directory (depends on what is being loaded)
• it seems redundant to put both the relative path $ORIGIN/../lib and the absolute path.join(self.installdir, 'lib'), this would only be useful for binaries loaded from non-standard subfolders (i.e. something else than {installdir}/*) that link to libraries in a parent non-standard lib subfolder and also link to other libraries in the standard {installdir}/lib folder.
• changing the order is probably not enough and the extend should be changed to a prepend, otherwise easyblocks and custom paths will keep causing this issue

I see two solutions to this issue:

• A. we just remove the absolute paths from the list

  '$ORIGIN',
  '$ORIGIN/../lib',
  '$ORIGIN/../lib64',
  

• B. we only use the absolute paths and delete the installdir before rebuilding

  os.path.join(self.installdir, 'lib'),
  os.path.join(self.installdir, 'lib64'),
  $ORIGIN
  

Option A seems the most practical solution.

@boegel any thoughts on this?

[ocaisa]

I'm pretty sure the reason the paths in A exist is so that when you run the test step for something not yet installed it can still find the libraries that it needs (which will ultimately end up in os.path.join(self.installdir, 'lib') but are not there yet).

Specifically, the scenario is temp_location/bin/something being run which depends on temp_location/lib/librequirement.so before the install step. My memory is running on fumes, but I believe we saw this for LAMMPS at least.

[Crivella]

I'm pretty sure option A exists so that when you run the test step for something not yet installed it can still find the libraries that it needs (which will ultimately end up in os.path.join(self.installdir, 'lib') but are not there yet).

I think the rpaths to installdir are always there, they are just commonly ignored since that location does not exists.
The problem comes in when that location does exist, and the library there gets used instead of the newly built one.
In particular i had trouble with this when testing different dependencies for ADIOS2, and also with LLVM we had to enforce deleting the installdir before the build otherwise the library picked up by clang when starting the stage2 would have different mangled symbol names than the expected ones (at the time i did not think of going the rpath wrappers route i guess)

[Crivella]

changing the order is probably not enough and the extend should be changed to a prepend, otherwise easyblocks and custom paths will keep causing this issue

I think the only thing that can get in before $ORIGIN right now would be overrides given with the build_option rpath_override_dirs which i guess should be fine as it expects the user is doing it intentionally

[boegel]

I think this change makes sense, but this should be tested quite broadly...

We may have had a reason to use the particular order we're using now, although maybe not and we're just using whatever made most sense at the time, I would have to dig into the Git history to check for clues.

@Crivella Can you set up a broad test with this change in place, building a whole bunch of things from scratch (and then some rebuilds of things that are known to be problematic with the currently used order), and report back?

If no problems emerge from broad testing, we can merge this change.
But since this only really fixes a problem for rebuilds, which is quite specific, I'm reluctant to go YOLO on this (merge it and see what happens during pre-release regression test, for example).