Jaxlib Errors

[sundeep-vedithi]

When I run AlphaPulldown Multimer in homomer mode, I seen the following error. I tried updating Jax installation on Cuda 11.8 but the error remained the same....

2023-12-09 14:59:51.012309: E external/xla/xla/stream_executor/cuda/cuda_dnn.cc:415] There was an error before creating cudnn handle (302): cudaGetErrorName symbol not found. : cudaGetErrorString symbol not found.
Traceback (most recent call last):
File "/home/user/miniconda3/envs/AlphaPulldown/bin/run_multimer_jobs.py", line 457, in
app.run(main)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/home/user/miniconda3/envs/AlphaPulldown/bin/run_multimer_jobs.py", line 451, in main
predict_multimers(multimers)
File "/home/user/miniconda3/envs/AlphaPulldown/bin/run_multimer_jobs.py", line 386, in predict_multimers
model_runners, random_seed = create_model_runners_and_random_seed(
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/alphapulldown/utils.py", line 355, in create_model_runners_and_random_seed
model_params = data.get_model_haiku_params(model_name=model_name, data_dir=data_dir)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/alphafold/model/data.py", line 33, in get_model_haiku_params
return utils.flat_params_to_haiku(params)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/alphafold/model/utils.py", line 116, in flat_params_to_haiku
hk_params[scope][name] = jnp.array(array)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/numpy/lax_numpy.py", line 2073, in array
out_array: Array = lax_internal._convert_element_type(out, dtype, weak_type=weak_type)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/lax/lax.py", line 555, in _convert_element_type
return convert_element_type_p.bind(operand, new_dtype=new_dtype,
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/core.py", line 386, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/core.py", line 389, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/core.py", line 821, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/dispatch.py", line 120, in apply_primitive
compiled_fun = xla_primitive_callable(
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/util.py", line 263, in wrapper
return cached(config._trace_context(), *args, **kwargs)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/util.py", line 256, in cached
return f(*args, **kwargs)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/dispatch.py", line 153, in xla_primitive_callable
compiled = computation.compile()
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/interpreters/pxla.py", line 2263, in compile
executable = UnloadedMeshExecutable.from_hlo(
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/interpreters/pxla.py", line 2611, in from_hlo
xla_executable, compile_options = _cached_compilation(
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/interpreters/pxla.py", line 2518, in _cached_compilation
xla_executable = compiler.compile_or_get_cached(
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/compiler.py", line 289, in compile_or_get_cached
return backend_compile(backend, computation, compile_options,
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/profiler.py", line 314, in wrapper
return func(*args, **kwargs)
File "/home/user/miniconda3/envs/AlphaPulldown/lib/python3.10/site-packages/jax/_src/compiler.py", line 251, in backend_compile
return backend.compile(built_c, compile_options=options)
jaxlib.xla_extension.XlaRuntimeError: FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
I0000 00:00:1702133991.342979 4070 tfrt_cpu_pjrt_client.cc:352] TfrtCpuClient destroyed.

Kindly help...Sundeep

[dingquanyu]

Hi @scv26

Many thanks for your interests in our package. The error message: "FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details." seems to indicate that you haven't installed or loaded corresponding cuDNN library. Please check if you have also correctly install the cuDNN library that is compatible with you CUDA. Hope it helps.

Yours
Dingquan

[sundeep-vedithi]

Hi Dingquan Yu,

Thank you for your response. I managed to resolve the error.