SLURM error when trying to run on Perlmutter #1231
Description
I was trying to revisit cupynumeric after a while (previous discussion). I tried out the latest install instructions with conda on Perlmutter:
conda create -n myenv -c conda-forge -c legate cupynumeric
when I try to run examples with legate --launcher srun ./program.py, I get the following error
nid001236:1733247] OPAL ERROR: Unreachable in file pmix3x_client.c at line 111
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:
version 16.05 or later: you can use SLURM's PMIx support. This
requires that you configure and build SLURM --with-pmix.
Versions earlier than 16.05: you must use either SLURM's PMI-1 or
PMI-2 support. SLURM builds PMI-1 by default, or you can manually
install PMI-2. You must then build Open MPI using --with-pmi pointing
to the SLURM PMI library location.
Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[nid001236:1733247] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Are there any specific instructions on how to install/run on Perlmutter? I mainly want to test the multi-node multi-GPU Choleksy, linear solves, and Eigenvalue decompositions. Please let me know if there is anything specific to install for those.
Thanks for your help!