Core 0x23 stalled on current (21.11.0) image

[sanori]

Log.txt

00:29:18:WU02:FS00:Download complete
00:29:19:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:12261 run:0 clone:236 gen:81 core:0x23 unit:0x000000ec0000005100002fe500000000
00:29:19:WU02:FS00:Starting
00:29:19:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /fah/cores/cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/0x23-8.0.3/Core_23.fah/FahCore_23 -dir 02 -suffix 01 -version 706 -lifeline 1 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
00:29:19:WU02:FS00:Started FahCore on PID 50
00:29:19:WU02:FS00:Core PID:54
00:29:19:WU02:FS00:FahCore 0x23 started
00:29:19:WARNING:WU02:FS00:FahCore returned: WU_STALLED (127 = 0x7f)

Inspection

Core 0x23 seems to require OpenCL 3.0. But, OpenCL 3.0 does not work properly on CUDA 11.2.2.

$ docker exec -it fah0 clinfo
Number of platforms                               1
  Platform Name                                   NVIDIA CUDA
  Platform Vendor                                 NVIDIA Corporation
  Platform Version                                OpenCL 3.0 CUDA 12.2.148
  Platform Profile                                FULL_PROFILE
(snip)
ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.2.11
  ICD loader Profile                              OpenCL 2.1
	NOTE:	your OpenCL library only supports OpenCL 2.1,
		but some installed platforms support OpenCL 3.0.
		Programs using 3.0 features may crash
		or behave unexepectedly

Inference

According to the NVIDIA Technical Blog, NVIDIA supports OpenCL 3.0 since Linux driver version 465.19.1. The matching CUDA version would be 11.3.1 according to the CUDA release notes

Therefore, I guess that the CUDA version of base image should be updated at least 11.3.1.

[sanori]

I found that clinfo package on Ubuntu 18.04 and 20.04 does not support OpenCL 3.0.
The minimum CUDA version of official docker image that is built on Ubuntu 22.04 is 11.7.1-base-ubuntu22.04, currently.
Therefore, minimum CUDA version would be at least 11.7.1, I guess.

[sanori]

I also found current FahCore_23 binary requires libexpat.so.1 which does not exist neither in docker image nor in core 23 package.
(I don't know why core 23 requires libexpat1 which is not required in 0xa8 or 0x22 cores.)
I succeeded to run FahCore_23 in the docker container by changing some parameters in Dockerfile.
I'll send a pull-request.

[beberg]

We hold back versions intentionally for maximum compatibility.

The only new requirement for core 23 is that the CPU must support SSE4.1, so this may be what's triggering the cascade of issues. It only needs OpenCL 1.2, but you should not get assigned that core if you don't have SSE4.1.

[sanori]

I understand. Thank you for the explanation. I misunderstood that OpenMM 8 uses OpenCL 3.

Then, it appears that the main cause of Fahcore 23 stalling is the absence of libexpat1.
Which one would be the solution of WU_STALLED? Adding libexpat1 in the docker image? Or wating for the dependency change of Fahcore23 package?

[cgint]

I faced the same issue as described in the title.

@sanori Thx for your PR #32.

That fixed it for me and now everything works inside docker the same way as on my host machine.