Issues in resource tracker for Python 3.13.7: `ModuleNotFoundError: No module named '_posixsubprocess'` on Windows and warnings `Cannot register X for automatic cleanup: unknown resource type Y` on posix

[lesteve]

This was seen in scikit-learn CI scikit-learn/scikit-learn#31955 (comment)

from loky.backend import resource_tracker

resource_tracker.register("this_folder", "folder")

Windows

This works fine on Python 3.13.6 but yield the following error on Python 3.13.7:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 1
----> 1 resource_tracker.register("this_folder", "folder")

File ~\AppData\Roaming\uv\python\cpython-3.13.7-windows-x86_64-none\Lib\multiprocessing\resource_tracker.py:244, in ResourceTracker.register(self, name, rtype)
    242 def register(self, name, rtype):
    243     '''Register name of resource with resource tracker.'''
--> 244     self._send('REGISTER', name, rtype)

File ~\AppData\Roaming\uv\python\cpython-3.13.7-windows-x86_64-none\Lib\multiprocessing\resource_tracker.py:261, in ResourceTracker._send(self, cmd, name, rtype)
    256 if len(msg) > 512:
    257     # posix guarantees that writes to a pipe of less than PIPE_BUF
    258     # bytes are atomic, and that PIPE_BUF >= 512
    259     raise ValueError('msg too long')
--> 261 self._ensure_running_and_write(msg)

File ~\AppData\Roaming\uv\python\cpython-3.13.7-windows-x86_64-none\Lib\multiprocessing\resource_tracker.py:220, in ResourceTracker._ensure_running_and_write(self, msg)
    218         msg = None  # message was sent in probe
    219     else:
--> 220         self._launch()
    222 while True:
    223     try:

File ~\AppData\Roaming\uv\python\cpython-3.13.7-windows-x86_64-none\Lib\multiprocessing\resource_tracker.py:185, in ResourceTracker._launch(self)
    183     if _HAVE_SIGMASK:
    184         prev_sigmask = signal.pthread_sigmask(signal.SIG_BLOCK, _IGNORED_SIGNALS)
--> 185     pid = util.spawnv_passfds(exe, args, fds_to_pass)
    186 finally:
    187     if prev_sigmask is not None:

File ~\AppData\Roaming\uv\python\cpython-3.13.7-windows-x86_64-none\Lib\multiprocessing\util.py:515, in spawnv_passfds(path, args, passfds)
    514 def spawnv_passfds(path, args, passfds):
--> 515     import _posixsubprocess
    516     import subprocess
    517     passfds = tuple(sorted(map(int, passfds)))

ModuleNotFoundError: No module named '_posixsubprocess'

Linux (probably posix in general)

You get a warning indicating that the resource was not registered correctly:

Traceback (most recent call last):                                                                                                                                                                                                                                                                                              
  File "/home/lesteve/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register this_folder for automatic cleanup: unknown resource type folder

CPython change

The change in CPython seems to be python/cpython#131787 which was back-ported in 3.13 branch in python/cpython#137738. This hasn't been backported in the 3.14 branch yet python/cpython#137737.

Got this by looking at the recent changes in the CPython code and this matches the git log command.

❯ git log v3.13.6..v3.13.7 -- Lib/multiprocessing/resource_tracker.py
commit d22276dc7d907d868f59e7b65f9d1c1af80bdfbe
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Thu Aug 14 12:52:30 2025 +0200
 
    [3.13] gh-131788: make resource_tracker re-entrant safe (GH-131787) (#137738)
    
    gh-131788: make resource_tracker re-entrant safe (GH-131787)
    (cherry picked from commit f24a012350f71141648cbd61081a25a458dd7fff)
    
    Co-authored-by: Thomas Grainger <tagrain@gmail.com>
    Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
    Co-authored-by: Gregory P. Smith <greg@krypto.org>

Note: for now Python 3.13.7 is not available on conda-forge see conda-forge/python-feedstock#807. I installed Python 3.13.7 with uv.

[lesteve]

After having looked into it a bit, here is my current understanding:

• loky.resource_tracker.ResourceTracker inherits from multiprocessing.resource_tracker.ResourceTracker and overrides ensure_running
• before the CPython change: loky register = multiprocessing register -> multiprocessing _send -> ensure_running. By overriding ensure_running we make sure that the loky code is called
• after the CPython change, register -> _send -> _ensure_running_and_send, so loky ensure_running is not called anymore.
• it is unclear to me whether this can be considered a regression in CPython or not ... deriving from multiprocessing ResourceTracker and overriding ensure_running seems a valid use case. At the same time, copying some code from CPython and modifying it (Windows support, additional resource types) is bound to be brittle in the long run.
• there are other changes in the CPython resource tracker that weren't ported to loky e.g. some reentrant things. The cleanest diff I could find between loky and CPython was CPython 3.8.

[ogrisel]

We might indeed try to resync the loky fork of the resource tracker machinery with a more recent version of Python, but this might be challenging because we still want to be compatible with Python 3.9 (or at least Python 3.10 if we decide to drop Python 3.9 support in the process).