Error with Resource Tracker in MPI Parallelism Using ProcessPoolExecutor

[HITOfficial]

Hello,

I'm currently working on a Python script that utilizes MPI for distributing tasks across multiple nodes, and I am using concurrent.futures.ProcessPoolExecutor() for parallelism on each node.

I am encountering an error when parallelizing across multiple nodes, while the script works fine on a single node. The error message I receive is as follows:

Error
lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:314: UserWarning: resource_tracker: There appear to be 1 leaked folder objects to clean up at shutdown

To gain more insights, I added logging to the _unlink_resources method in resource_tracker.py. Here are some example names of the leaked folders:

/dev/shm/joblib_memmapping_folder_238470_267115d5d22146a0aac9c084e6c08cda_371f2046d4ca4d58a3704d85c4641ad6
/dev/shm/joblib_memmapping_folder_238488_151df3b700ff4a73917d18881b3eef34_899604ea80c3414e9f8a72aa791d15de
/dev/shm/joblib_memmapping_folder_238472_01d82d7561b54ff1a04ac14d5a713c49_2b7d6b2c3a814f978557dd3947f46266

Environment Details:
Python version: 3.11
Loky version: 3.4.1
Joblib version: 1.4.2 (build py311h06a4308_0)
Environment created with Conda

[ogrisel]

It's curious that there is single node/multi node discrepancy since those temporary files should be visible on a single node (/dev/shm should not be not a shared filesystem folder).

I have no idea how to debug without access to an MPI cluster unfortunately, and I am not familiar enough with MPI software to deploy a virtual debug cluster locally using docker compose or similar.