Container transparency

[lmoureaux]

Problem description

In the context of the widespread migration to Alma9, containers are needed for the widely-used CMSSW 10_6 series powering Run 2 UltraLegacy analyses. Images are available to be used through the cmssw-cc7 command. Unfortunately, this comes with an important caveat for interactive usage because the images are not designed for this use case:

• Editors are very limited, even vim is not installed and one has to resort to the antiquated vi
• Singularity overrides the user's default prompt, removing important contextual information located there
• Command autocompletion does not work
• ls colors are disabled
• etc

It is unfortunately not possible to build more comprehensive images based on CentOS 7 because the corresponding repositories have been removed.

In practice, the above means that users need to juggle between two shells: one running in Singularity to run scram and CMSSW commands, and another running in Alma9 for everything else. This is extremely frustrating.

Desired behavior

I think part of the solution to these issues is for SCRAM to add container support. The behavior would be as follows:

• When a release for the host architecture is not available, scram project creates a release for the production architecture.
• When scram runtime is used in an area whose architecture doesn't match the host, it sets up environment variables telling SCRAM to use a container.
• Other SCRAM commands transparently start a container to run the command inside.
• A new command, say scram run, allows running a command in the container environment. This is needed for commands like rivet made available by CMSSW.

Implementing this would greatly alleviate the pain of using CMSSW 10_6_X that many analyzes are currently facing.

In fact, I think SCRAM should embrace containers even further and use them regardless of the host architecture. Always running in a container will prevent the same issue from reappearing in the future.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @lmoureaux.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

@lmoureaux , thanks for the suggestion. We will look in to it

By the way , cmssw-elX container e.g. cmssw-cc7 or cmssw-el7 are not to replace lxplus type env. These are there to provide basic env to build/run cmssw executables. These containers are also used for running grid jobs so we would like to keep the size of these container in control. That is why many tools (condor for job submission, editors e.g.) which are available on build nodes/lxplus are not there in these containers.

By the way, did you try cmssw-el7 --interactive which should start bash in interactive mode and then ls coloring, or command autocompletion should work. vim is already installed in containers.

[smuzaffar]

By the way, if one knows that he/she wants to work with cc7/el7 releases then the following two can do what you requested

alias scram-el7='cmssw-el7 -- scram'
function scram-run(){
eval "cmssw-el7 -- 'cmsenv; $@'"
}

and then

> scram-el7 p CMSSW_10_6_30
> cd CMSSW_10_6_30
> scram-run git cms-addpkg FWCore/Utilities
> scram-el7 build -j 10
> scram-el7 build runtests -j 10
> scram-run which cmsRun
/cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_30/bin/slc7_amd64_gcc700/cmsRun
> scram-run cmsRun --help
cmsRun [options] [--parameter-set] config_file
Allowed options:
  -h [ --help ]                         produce help message
  -p [ --parameter-set ] arg            configuration file
  -j [ --jobreport ] arg                file name to use for a job report file:
                                        default extension is .xml
  -e [ --enablejobreport ]              enable job report files (if any)
                                        specified in configuration file
  -m [ --mode ] arg                     Job Mode for MessageLogger defaults -
                                        default mode is grid
  -n [ --numThreads ] arg               Number of threads to use in job (0 is
                                        use all CPUs)
  -s [ --sizeOfStackForThreadsInKB ] arg
                                        Size of stack in KB to use for extra
                                        threads (0 is use system default size)
  -t [ --multithreadML ]                MessageLogger handles multiple threads
                                        - default is single-thread
  --strict                              strict parsing

[lmoureaux]

Thanks @smuzaffar!

I had checked the cmssw-el7 script 6 months ago and it wasn't working so well back then, so I moved on and didn't check again. Adding vim, bash, and zsh to the images were certainly nice additions and they now appear more suitable for interactive use.

The scram-el7 and scram-run commands you propose are also extremely useful. A nice side-effect of using scram-run is that one doesn't need to run cmsenv explicitly ever again. The drawback is that setting up the container takes about 1s and so does running cmsenv, so scram-run echo takes 2s.

Since scram the tool used to manage the runtime environment of cmssw, I still think it should understand when a container is needed. Minimally, the error message when no build is available for the host architecture should provide instructions for setting up a container (and recommend one for the production architecture).