Pilot writes to home directory of executing user during verify_proxy

[olifre]

In jobs running in isolated / containerized environments as users with non-existing or read-only home directory, the following warnings are observed:

2021-08-10 14:33:48,037 | DEBUG    | pilot.user.atlas.proxy           | interpret_proxy_info      | stderr = /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/swConfig/asetup/createUserASetup.sh: line 44: /home/centos/.asetup: No such file or directory
WARNING: Failed to create directory /home/centos/.arc
WARNING: Unable to create /home/centos/.arc directory.
WARNING: Failed to create directory /home/centos/.arc
WARNING: Unable to create /home/centos/.arc directory.

Indeed, in case the home directory exists, files are created there and read from there, potentially breaking job isolation, and of course these errors are misleading when running with dummy users.

I traced these errors back to verify_arcproxy:

pilot2/pilot/user/atlas/proxy.py

Line 77 in d973077

def verify_arcproxy(envsetup, limit, proxy_id="pilot", test=False):

The issue is caused in two places:

• /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/swConfig/asetup/createUserASetup.sh is called as part of setupATLAS, and creates an .asetup file in the user home directory.
• arcproxy itself parses /etc/passwd and extracts the home directory, then creates its config files in there.

The first could in principle be worked around by adding unset HOME to the envsetup string (even though this is not a nice workaround), or adapting ALRB itself (but since ALRB is often used interactively, I presume other such cases may come up in the future).
The latter can not easily be solved, since arcproxy does not have a --no-config option nor does it accept arcproxy -z /dev/null. I wonder if flipping the order (i.e. preferring voms-proxy-info) is a viable option, now that SL6 is gone?

[PalNilsson]

Hi,
I think this goes beyond the pilot and I would like to add a few people who do not have GitHub accounts. I'll create a mail thread instead for this.
Cheers, Paul

[olifre]

Hi,
I see, indeed this may also be interpreted as something requiring a change in ALRB. The issue has been observed by my colleagues at KIT running in containerized environments, so we should also add them to the discussion. I'll send you a mail with their mail addresses shortly (after confirming with them who wants to join in) — in fact, there's also a related issue in ALRB which we can then maybe include from the start.
Cheers and thanks,
Oliver