Potential reason for fork-mode processes never starting / ending

[zetxtech]

During my use of Mpire, I encountered a peculiar situation randomly:

1. The program would hang indefinitely on the last few tasks.
2. Using Ctrl+C to terminate the program would fail to exit.

I tried to gain worker_insights for the hanging state, which showed that some workers were not running any tasks and were stuck in the waiting_time state for a long time. At the same time, by monitoring the main thread state using debugpy, I found that the main thread was waiting for these workers (either waiting for them to start or to finish), and these worker processes did exist.

I attempted to connect and debug these hanging processes using debugpy, but I couldn't attach (later I learned it was due to blocking instructions preventing the connection). Eventually, I successfully used py-spy to capture their stack traces. I found that they were stuck at _flush_std_streams when being started by multiprocessing:

def _flush_std_streams():
    try:
        sys.stdout.flush()
    except (AttributeError, ValueError):
        pass
    try:
        sys.stderr.flush()
    except (AttributeError, ValueError):
        pass

(located in multiprocessing.util)

I inspected sys.stdout and found that it was never overwritten with another stream, so it was unlikely that the flush operation would hang.

Later, I found a related issue on Python's GitHub: python/cpython#91776. This issue remains unresolved and is believed to be caused by internal locks during writing. In some cases, users might be running other threads while executing WorkerPool, and these threads might be writing to sys.stdout or sys.stderr. If a fork occurs at this moment, the forked process, controlled by mpire and multiprocessing, executes _flush_std_streams. However, if the internal lock of stdout or stderr is in a writing state, the flush operation will wait indefinitely, causing a hang.

Even without threads, there is a small probability of hanging (much lower, as mentioned in your code when waiting for the task_queue to end with a timeout), possibly due to file descriptor delays related to the operating system, for unknown reasons.

Although this issue is not directly related to mpire, the fork mode is a primary usage method for mpire, and writing to stdout/stderr from other threads when running a WorkerPool is a common operation. So I'm recording it here in case someone needs it:

Reopening stdout and stderr:

def reopen_std_streams():
    sys.stdout = os.fdopen(1, "w")
    sys.stderr = os.fdopen(2, "w")

os.register_at_fork(after_in_child=reopen_std_streams)

(This is only effective on Linux & python >= 3.7)

Another approach is to provide a thread lock for the process's fork (_start_worker) to help synchronize other logging threads, and with documentation.

If you could delve deeper into solving this issue, I would greatly appreciate it.

[sybrenjansen]

Thnx for the elaborate description. This is indeed a known issue. See https://pythonspeed.com/articles/python-multiprocessing/ for a similar example. Global locks (especially in combination with threads) and forking do not always play nice.

Can you confirm that reopening the std streams method makes the deadlock go away? If so, I can add it (or you can make a PR), no problem.

[zetxtech]

Thanks for your prompt reply.

Yes, using reopen_std_streams solved the problem for me, and my program no longer randomly freezes. (My use case is using Manager to transmit logs to the main process for display, and the log display thread in the main process causes stdout to be locked frequently during fork)

However, I would consider it potentially risky to put this operation in the library by default, because someone may overwrite sys.stdout before forking, for processing before output to console. Although this is not a common practice, reopening the std streams will destroy this setup. Therefore, it may be helpful to have a parameter to control this behavior.

I will provide a PR when I have free time.