Can't build JAX when using local Clang and the CUDNN installed under /usr

[medaminezghal]

When I download CUDNN from the official website, it's installed par default under /usr. this will broke the toolchains for Clang because Bazel will link /usr/include to the CUDNN as mesntioned in this comment.
NVSHMEM and NCCL are installed in same location but it didn't break the toolchains because it only link the files not the entire folder.

[ybaturina]

Usage of LOCAL_CUDA_PATH is generally discouraged. This feature was added only for NVIDIA team needs.
If for some reason you need custom CUDA version, which is not available from NVIDIA repository yet, the installation of the toolkit shouldn't be at the same path as system libs/headers (like /usr/lib, '/usr/include` etc.).

[lahwaacz]

If for some reason you need custom CUDA version, which is not available from NVIDIA repository yet, the installation of the toolkit shouldn't be at the same path as system libs/headers (like /usr/lib, /usr/include etc.).

Note that CUDNN is not part of the CUDA toolkit and therefore it is (almost) never installed in the CUDA root directory. Check e.g. NVIDIA's official container images:

$ podman run --rm -it nvcr.io/nvidia/cuda:13.0.1-cudnn-devel-ubuntu24.04 find /usr | sort | grep cudnn
/usr/include/x86_64-linux-gnu/cudnn_adv.h
/usr/include/x86_64-linux-gnu/cudnn_adv_v9.h
/usr/include/x86_64-linux-gnu/cudnn_backend.h
/usr/include/x86_64-linux-gnu/cudnn_backend_v9.h
/usr/include/x86_64-linux-gnu/cudnn_cnn.h
/usr/include/x86_64-linux-gnu/cudnn_cnn_v9.h
/usr/include/x86_64-linux-gnu/cudnn_graph.h
/usr/include/x86_64-linux-gnu/cudnn_graph_v9.h
/usr/include/x86_64-linux-gnu/cudnn.h
/usr/include/x86_64-linux-gnu/cudnn_ops.h
/usr/include/x86_64-linux-gnu/cudnn_ops_v9.h
/usr/include/x86_64-linux-gnu/cudnn_v9.h
/usr/include/x86_64-linux-gnu/cudnn_version.h
/usr/include/x86_64-linux-gnu/cudnn_version_v9.h
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.13.0
/usr/lib/x86_64-linux-gnu/libcudnn.so
/usr/lib/x86_64-linux-gnu/libcudnn.so.9
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.13.0
/usr/share/doc/libcudnn9-cuda-13
/usr/share/doc/libcudnn9-cuda-13/changelog.Debian.gz
/usr/share/doc/libcudnn9-cuda-13/copyright
/usr/share/doc/libcudnn9-dev-cuda-13
/usr/share/doc/libcudnn9-dev-cuda-13/changelog.Debian.gz
/usr/share/doc/libcudnn9-dev-cuda-13/copyright
/usr/share/doc/libcudnn9-headers-cuda-13
/usr/share/doc/libcudnn9-headers-cuda-13/changelog.Debian.gz
/usr/share/doc/libcudnn9-headers-cuda-13/copyright

$ podman run --rm -it nvcr.io/nvidia/cuda:13.0.1-cudnn-devel-rockylinux9 find /usr | sort | grep cudnn
/usr/include/cudnn_adv.h
/usr/include/cudnn_adv_v9.h
/usr/include/cudnn_backend.h
/usr/include/cudnn_backend_v9.h
/usr/include/cudnn_cnn.h
/usr/include/cudnn_cnn_v9.h
/usr/include/cudnn_graph.h
/usr/include/cudnn_graph_v9.h
/usr/include/cudnn.h
/usr/include/cudnn_ops.h
/usr/include/cudnn_ops_v9.h
/usr/include/cudnn_v9.h
/usr/include/cudnn_version.h
/usr/include/cudnn_version_v9.h
/usr/lib64/libcudnn_adv.so
/usr/lib64/libcudnn_adv.so.9
/usr/lib64/libcudnn_adv.so.9.13.0
/usr/lib64/libcudnn_cnn.so
/usr/lib64/libcudnn_cnn.so.9
/usr/lib64/libcudnn_cnn.so.9.13.0
/usr/lib64/libcudnn_engines_precompiled.so
/usr/lib64/libcudnn_engines_precompiled.so.9
/usr/lib64/libcudnn_engines_precompiled.so.9.13.0
/usr/lib64/libcudnn_engines_runtime_compiled.so
/usr/lib64/libcudnn_engines_runtime_compiled.so.9
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.13.0
/usr/lib64/libcudnn_graph.so
/usr/lib64/libcudnn_graph.so.9
/usr/lib64/libcudnn_graph.so.9.13.0
/usr/lib64/libcudnn_heuristic.so
/usr/lib64/libcudnn_heuristic.so.9
/usr/lib64/libcudnn_heuristic.so.9.13.0
/usr/lib64/libcudnn_ops.so
/usr/lib64/libcudnn_ops.so.9
/usr/lib64/libcudnn_ops.so.9.13.0
/usr/lib64/libcudnn.so
/usr/lib64/libcudnn.so.9
/usr/lib64/libcudnn.so.9.13.0
/usr/share/licenses/libcudnn9-cuda-13
/usr/share/licenses/libcudnn9-cuda-13/LICENSE

[ybaturina]

We probably need to update the docs and mention that the default CUDA toolkit installation shouldn't be used for LOCAL_CUDA_PATH value. Please see the explanation here.

[lahwaacz]

So you don't support using the local CUDA/CUDNN/NCCL/NVSHMEM installations at all and insist on "downloading half of the internet" as stated in the comment you refer to?

[ybaturina]

We support the local paths of CUDA redistributions provided they are not located at default system paths, and they have the structure described in the documentation. "LOCAL_CUDA_PATH" is not the same as "default CUDA toolkit path found in NVIDIA docker images, or obtained via NVIDIA CUDA installer".

May I ask, why do you need local redistributions instead of the ones available in NVIDIA repositories?
Also please note that "half of the internet" was related to XLA builds, that indeed have a lot of third-party dependencies.

[medaminezghal]

@ybaturina I’m not sure if the provided reason exists or not because for building packages provided by Linux distributions maintainers, it’s always built under clean chroot so Bazel cache will note cause any problem at all. The case mentioned in the comment are for one who is building for its custom use case not for someone who want to deliver a stable package that can works perfectly within its environment.

In Arch Linux, TensorFlow 2.17 and JAX 0.4.31 was built against local CUDA/cuDNN without any problems. First problem has introduced within CUPTI location as you assumed that it should be under $CUDA_HOME not $CUDA_HOME/extras/CUPTI and now from the cuDNN that is located under /usr.

[ybaturina]

TensorFlow 2.17 and JAX 0.4.31 are pretty old versions, I think they didn't have hermetic CUDA rules at all.

CUPTI headers location in the individual redistributions is different than in CUDA toolkit location in the images - you can download the artifacts from this location and look at the structure.

We have no control over CUDA/CUDNN headers available in the redistributions. If we create individual symlinks, we might miss new headers/libraries in the new toolkit versions.

NVSHMEM and NCCL have the same approach: they create symlinks for the folders: see here and here.

The majority of the team is on holidays. I'll discuss the issues mentioned here after my colleagues get back, and will post the updates here.

[lahwaacz]

May I ask, why do you need local redistributions instead of the ones available in NVIDIA repositories?

The question should be the opposite: why do you need to always download your preferred copy of CUDA/CUDNN/NCCL/NVSHMEM? I emphasize this because it seems that you are actively working on preventing users to use locally installed software under any circumstances.

Anyway, the answer should be obvious: using locally installed software saves network bandwidth, disk space & wear, and CPU cycles. Here is some longer explanation. To reiterate a bit, when we redistribute a software that is built using rules_ml_toolchain, we typically start with a clean environment of that particular distribution, where there is no bazel cache (hence the resource waste) or even integration between bazel and the distribution build system. Distribution packages should have dependencies among them, they should not vendor their own copies of everything they need. What we want is to install other packages of the distribution, take sources of the software, and build it, providing an artifact that other packages can depend on. Ideally, the build process should be reproducible even in an offline environment.