local group not added anymore to My.CMSGroups ?

[belforte]

There at least one case where the user membership in local groups is not added to CMSGroups classAd
See https://cms-talk.web.cern.ch/t/issue-with-processing-private-mc-samples-using-crab/138451/4

pasting here most relevant text

 matching doesnt work because of stringListsIntersect(CMSGroups,“T3_US_FNALLPC”)`

The user is among the local users at LPC:

[mmascher@vocms0106 ~]$ grep jingyu /cvmfs/cms.cern.ch/SITECONF/T3_US_FNALLPC/GlideinConfig/local-users.txt
jingyu

As @mmascher pointed out, CRAB should be setting the group information here?

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Lines 405 to 408 in b84f876

	## Add group information (local groups in SITECONF via CMSGroupMapper, VOMS groups via task info in DB)
	groups = set.union(map_user_to_groups(task['tm_username']), task['user_groups'])
	groups = ','.join(groups) # from the set {'g1','g2'...,'gN'} to the string 'g1,g2,..gN'
	jobSubmit['My.CMSGroups'] = classad.quote(groups)

[belforte]

as simple as "/cvmfs" is not mounted inside the container.

crab3@crab-prod-tw01:/data/srv/TaskManager$ ls -ld /cvmfs
drwxr-xr-x. 2 root root 0 Aug 21 15:43 /cvmfs
crab3@crab-prod-tw01:/data/srv/TaskManager$ ls -ld /cvmfs/cms.cern.ch/
ls: cannot access '/cvmfs/cms.cern.ch/': No such file or directory
crab3@crab-prod-tw01:/data/srv/TaskManager$ 

[belforte]

It turned out to be the old problem that at times the mount of CVMFS inside the container goes belly up.
I fixed it by restarting the container (docker stop; docker start) on prod and canary TW.
The more drastic procedure described in https://cmscrab.docs.cern.ch/crab-components/crab-taskworker/how-to.html#cvmfs-failed-to-mount to address the case when CVMFS fails at "VM-level" was not necessary.

We can do a few things, not necessarily one excluding the other:

1. find a way to set an alert/alarm so that we do not run long time with this
2. ask CVMFS support if there is more robust solution than fuse mount, like https ? (all in all CMVFS can be cached via squid)
3. bypass CVMFS, clone https://gitlab.cern.ch/SITECONF in the container and do a periodic git pull (e.g. when memory cache expires, see code in CMSGroupMapper.py
4. do 3. in the VM in /data disk, refresh via crontab and mount the local directory in the container
5. like 4. but instead of clone/fetch, do a straight cp from CVMFS
6. like 5. but only propagate changes
   • rsync -av /cvmfs/cms.cern.ch/SITECONF/ /data/container/SITECONF/
   • inside the container that is available as /data/hostdisk/SITECONF/ so we should modify the code

     CRABServer/src/python/CMSGroupMapper.py

     Line 65 in 3f5c584

     base_dir = '/cvmfs/cms.cern.ch/SITECONF'

FYI @aspiringmind-code I defer to you on deciding how to proceed

[belforte]

keeping open until a follow-up strategy is defined and ad-hoc issue(s) opened.

[belforte]

@aspiringmind-code about 1. the place where to catch the problem could be

CRABServer/src/python/CMSGroupMapper.py

Lines 65 to 73 in 0ac5f00

	base_dir = '/cvmfs/cms.cern.ch/SITECONF'
	cache = {}
	user_re = re.compile(r'[-_A-Za-z0-9.]+')
	sites = None
	try:
	if os.path.isdir(base_dir):
	sites = os.listdir(base_dir)
	except OSError as ose:
	log_function("Cannot list SITECONF directories in cvmfs:" + str(ose))

where today os.path.isdir(base_dir) was simply returning False. No exception raised and no action taken.

[belforte]

the clear symptom was that ls inside the container showed no directory
so maybe a simple alert could be crontab in the TW which does something like
docker exec -ti TaskWorker ls /cvmfs/cms.cern.ch/SITECONF
and prints to stdout in case of error, generating a mail to operators

e.g.

belforte@crab-dev-tw04/~> docker exec -ti TaskWorker ls /cvmfs/cms.cern.ch/SITECONF
ls: cannot access '/cvmfs/cms.cern.ch/SITECONF': No such file or directory
belforte@crab-dev-tw04/~> 

even if the first time I executed it returned nothing, the second the error.