Skip to content

Documentation to have climada 6 work in Euler #1103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
spjuhel wants to merge 1 commit into
base: develop
Choose a base branch
from
+169 −0
Open

Documentation to have climada 6 work in Euler #1103

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 169 additions & 0 deletions doc/development/Guide_Euler.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,175 @@
"\n",
"Once the Jupyterlab has started, open a Python notebook and stick to the default kernel."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fallback 2: Using Spack"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The ETH Zurich **Euler cluster** now supports **Spack**, a flexible package manager for high-performance computing (HPC). Spack simplifies installing and managing multiple versions and configurations of scientific software.\n",
"\n",
"-----\n",
"\n",
"### The Role of Spack on Euler\n",
"\n",
"Spack addresses the complexity of installing and managing scientific software, especially when multiple versions and configurations are required, which is common in HPC environments like Euler.\n",
"\n",
" * **Custom Dependency Management:** Spack ([official website](https://spack.readthedocs.io/en/latest/)) allows users to **compile and install recent versions** of a lot of application/libraries, such as **GDAL**, **PROJ**, and **HDF5**.\n",
" * **Cluster Compatibility:** It ensures that all dependencies are built with the correct compilers and settings to be **compatible with Euler's architecture** and software stack."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Commands to Create and Install the Environment\n",
"\n",
"Follow these steps to set up and install the CLIMADA environment using Spack on the Euler cluster."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Make Spack Available\n",
"\n",
"First, you need to load the centralized Spack installation on the cluster. (The command sources the setup script to initialize the Spack environment and commands.)\n",
"\n",
"`. /cluster/software/stacks/2025-06/setup-env.sh`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Create the Environment File\n",
"\n",
"Create a new spack environment with the following:\n",
"\n",
"`spack env create climada6`\n",
"\n",
"By default it should create it in your home folder. As these environment can get big, you can also use `spack env create -d /cluster/project/climate/$USER/spack/climada6`.\n",
"\n",
"Next, put the dependencies in the spack.yaml file:\n",
"\n",
"```\n",
"cat << 'EOF' > climada-6.1.0-env/spack.yaml\n",
"# This is a Spack Environment file.\n",
"#\n",
"# It describes a set of packages to be installed, along with\n",
"# configuration settings.\n",
"spack:\n",
" # add package specs to the `specs` list\n",
" packages:\n",
" all:\n",
" compiler: [gcc@12.2.0]\n",
" providers:\n",
" mpi: [openmpi]\n",
" specs:\n",
" - hdf5\n",
" - geos\n",
" - sqlite\n",
" - eccodes\n",
" - proj\n",
" - swig\n",
" - gdal@3.10.0+hdf5+netcdf+python\n",
" - llvm+python\n",
" - py-fiona\n",
" - py-rasterio\n",
" - py-pyproj\n",
" - py-xarray\n",
" - py-sparse\n",
" - py-dask\n",
" - py-geopandas\n",
" - py-pip\n",
" view: true\n",
" concretizer:\n",
" unify: true\n",
"EOF\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Activate the Environment\n",
"\n",
"You must **activate** the environment before you can configure or install packages within it.\n",
"\n",
"`spack env activate climada6` (you can add `-p` to show the name of the env in your prompt if you want). Note that it can take a bit of time.\n",
"\n",
"(Note that you can deactivate the environment with `spack env deactivate`)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Install the Environment\n",
"\n",
"With the environment active, you can now install the environment. Spack will read the `spack.yaml` file, resolve all dependencies, compile packages from source (where necessary, like **GDAL**), and install them into a unique path.\n",
"This command may take a significant amount of time, thus it is best to do it within an interactive session instead than on the login node:\n",
"\n",
"First request a computing node:\n",
"`srun --time=06:00:00 --ntasks=1 --cpus-per-task=1 --mem-per-cpu=4G --pty bash`\n",
"\n",
"Then check that you are indeed in the spack env with `spack find`\n",
"The first line should state `==> In environment climada6`\n",
"\n",
"Now you can invoke the `spack install` command to begin the installation process (It can take a few hours, some required libraries take a lot of time to compile (llvm and GDAL) but spack does everything for you)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Install climada and petal\n",
"\n",
"Now the environment has everything non-pythonic available for climada. You can now `pip install climada` and `pip install climada-petals` and shoud get the latest version of climada installed.\n",
"\n",
"Note that climada will only be available and work in this environment. So you have to activate that environment in order to use it.\n",
"If you exit and log back in, spack needs to be reloaded:\n",
"\n",
"```\n",
". /cluster/software/stacks/2025-06/setup-env.sh\n",
"spack env acticate climada6\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Making it work with JupyterHub\n",
"\n",
"JupyterHub is available via a different software stack than spack, thus you cannot make your spack environment work in JupyterHub without a few additional steps.\n",
"\n",
"First you need to modify the `jupyterlabrc` file (in `/cluster/home/$USER/.config/euler/jupyterhub/jupyterlabrc`) (you should save a copy of the original) with the following:\n",
"\n",
"```\n",
"module load stack/2025-06 r/4.4.1\n",
". /cluster/software/stacks/2025-06/setup-env.sh\n",
"spack env activate climada6\n",
"```\n",
"\n",
"This will tell the script loading JupyterHub to load spack and your environment. Now the problem is that everything related to jupyterhub is not present in that stack. But that's fine, you can actually just install it:\n",
"\n",
"```\n",
"pip install pip-system-certs jupyter==1.1.1 jupyter_client==8.6.3 jupyter-console==6.6.3 jupyter_core==5.9.1 jupyter-events==0.12.0 jupyter-lsp==2.3.0 jupyter-resource-usage==1.2.0 jupyter_server==2.17.0 jupyter_server_fileid==0.9.3 jupyter_server_proxy==4.4.0 jupyter_server_terminals==0.5.3 jupyter-server-ydoc==2.1.1 jupyter-ydoc==3.1.0 jupyterhub==5.4.2 jupyterlab==4.4.10 jupyterlab_pygments==0.3.0 jupyterlab_server==2.28.0 jupyterlab_widgets==3.0.14\n",
"```\n",
"\n",
"And now you should be good! Climada will be available in your default python3 kernel in jupyter.\n",
"\n",
"Note that this now loads JupyterHub in a different stack so you other environment will not be available / not work. You can revert `jupyterlabrc` to its original version to load it back from the original stack."
]
}
],
"metadata": {
Expand Down
Loading