Spack deployment discussion

[pramodk]

@matz-e @tristan0x : here are notes from our discussion :

Where are packages being built?

The tests scripts that I am running are under:

/gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy

The packages that I built can be seen as:

export SOFTS_DIR_PATH=/gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy/workspace/deployment
export SPACK_ROOT=/gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy/workspace/HOME_DIR/spack
export PATH=$SPACK_ROOT/bin:$PATH
source $SPACK_ROOT/share/spack/setup-env.sh

spack find

Issue with SSD filesystem?

Don't use tmp_compile as I am seeing weired install package corruption. I don't think is triggering that but CS is also not seeing somehting obvious:

/gpfs/bbp.cscs.ch/ssd/tmp_compile

Errors I saw was:

# cmake package is correuprted

  >> 3    CMake Error: Could not find CMAKE_ROOT !!!
     4    CMake has most likely not been installed correctly.
     5    Modules directory not found in

# setuptools is corrupted

Error: option --single-version-externally-managed not recognized

If you want to use ssd, use HPC specific directory:

mkdir -p /gpfs/bbp.cscs.ch/apps/hpc/test/$USER

Caveats

Here are list of errors that I have seen:

I have seen above below error which is not exactly related to spack but appears from bash. Its not fatal. I asked on slack channel and someone suggested to use spack --color never install .... but that doesn't fix the issue. I don't know how to fix this. This is not fatal and it seems ok to ignore it.

...
==> zlib is already installed in /gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy/workspace/deployment/install/linux-rhel7-x86_64/gcc-6.4.0/zlib-1.2.11-w43e56
==> zlib@1.2.11 : marking the package explicit
==> Error: 'ascii' codec can't encode character u'\u2018' in position 154659: ordinal not in range(128)

Is parquet / arrow failing? I am seeing below:

     7924    /gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy/workspace/HOME_DIR/spack/var/spack/stage/p
             arquet-1.4.0-qxq76a5srn5ls42n6rnc3ar76y37is2x/parquet-cpp-apache-parquet-cpp-1.4.0/src/parquet/
             column_reader.cc:40:55:   required from here
  >> 7925    /gpfs/bbp.cscs.ch/apps/compilers/install/gcc-6.4.0/include/c++/6.4.0/bits/unique_ptr.h:74:22: e
             rror: invalid application of ‘sizeof’ to incomplete type ‘arrow::RleDecoder’
     7926      static_assert(sizeof(_Tp)>0,
     7927                          ^
     7928    /gpfs/bbp.cscs.ch/apps/compilers/install/gcc-6.

Omega-h failing with :

1 error found in build log:
     40    -- TeuchosParameterList_CONFIG: /gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy/workspace/de
           ployment/install/linux-rhel7-x86_64/gcc-6.4.0/trilinos-12.12.1-3qf4e4/lib/cmake/TeuchosParameterL
           ist/TeuchosParameterListConfig.cmake
     41    -- Omega_h_USE_TeuchosComm: ON
     42    -- TeuchosComm_PREFIX:
     43    -- TeuchosComm_CONFIG: /gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/spack-deploy/workspace/deployment/
           install/linux-rhel7-x86_64/gcc-6.4.0/trilinos-12.12.1-3qf4e4/lib/cmake/TeuchosComm/TeuchosCommCon
           fig.cmake
     44    -- Omega_h_USE_TeuchosParser: ON
     45    -- TeuchosParser_PREFIX:
  >> 46    CMake Error at cmake/bob.cmake:303 (find_package):
     47      By not providing "FindTeuchosParser.cmake" in CMAKE_MODULE_PATH this
     48      project has asked CMake to find a package configuration file provided by
     49      "TeuchosParser", but CMake did not find one.
     50
     51      Could not find a package configuration file provided by "TeuchosParser"
     52      with any of the following names:

Creation of Mirror

• Mirror should be created to store all source tarballs. Should be in central location. Somewhere in:

/gpfs/bbp.cscs.ch/apps/hpc/test/central-mirror/

You have write permission to populate this mirror.

Order of installations

Should we skip intel compiler?

• making first pass with gcc@6.4.0
• later we can introduce intel but the workflow should be multi-compiler aware

Other packages

• compilers : should be OK
• system tools : OK
• python packages : ~OK
  • skip large packages!!
  • activate most of them
• serial libraries : OK
• parallel libraries : should be OK
• applications : should be OK

where can I get compiler tarballs?

Intel compilers take long time to download and PGI requires manual download from account. We have downloaded tarballs in:

/gpfs/bbp.cscs.ch/apps/hpc/test/kumbhar/download

How am I testing it?

cd /gpfs/bbp.cscs.ch/apps/hpc/test/$USER
git clone https://github.com/pramodk/spack-deploy.git -b base-packages

cd spack-deploy

./check_specs.sh
./create_mirror.sh
./install_specs.sh

Structure of deployment

Here is what we thought about:

I want to trigger:
    - build all stack into prefix /gpfs/something
    - building everything using compiler 2017-11-01


jenkins-deployment
    - mirror
    - install
        - compilers
            - YYYY-MM-DD
            - latest (point to last : option in jenkins => )
        - tools
            - YYYY-MM-DD
        - libraries
            - YYYY-MM-DD
            - YYYY-MM-DD
            - YYYY-MM-DD
                - gnu-xyz-x86_64
                    - my-library
                - data
                    - .spack/... (local config)
                    - specs.txt
                    - spack_deploy.env (environment dump)
                    - spack_deploy.version (git hash)
                    - spack.env (environment dump)
                    - spack.version (git hash)
                    - stack.xml
                    - compilers.yaml
                    - packages.yaml
        - applications
            - YYYY-MM-DD
    - modules
    - deploy
        - venv  (
        - spack (git checkout used)

[matz-e]

Parquet/arrow: we should not build parquet by itself anymore, it's included in arrow+parquet now. I thought I had mentioned that before, but seems like I forgot?

[pramodk]

Parquet/arrow: we should not build parquet by itself anymore, it's included in arrow+parquet now. I thought I had mentioned that before, but seems like I forgot?

you mentioned but I didn't remove. I thought its still buildable. I will remote it.

[matz-e]

Sorry, should have stated that more explicitly. Since we added parquet to Spack, I'm thinking about filing a PR upstream to remove it, too.

[tristan0x]

To fix omega-h build error, we should use trilinos@develop instead of trilinos@12.12.1

[pramodk]

@tristan0x : Ok. As you know, develop is handled bit specially in Spack. For reproducibility we need version rather than develop. Do you think adding our own version with latest commit would be sufficient? If so, could you add such version?

[tristan0x]

@pramodk omega_h works with trilinos git tag xsdk-0.4.0-rc1. I replaced it in #2 and BlueBrain/spack#139