Support for PyPy processes

[davidbrochart]

It could be interesting to offload the parallel jobs to PyPy processes instead of the main Python interpreter, which would be mostly CPython if one wants to plot data, use Pandas or any other library that is not (yet) supported by PyPy. In this use case I see PyPy as a fast computation backend similar to Numba and its @jit decorator. Furthermore, Numpy arrays could be passed in parameters and handled by Numpypy, provided that pickling/unpickling is compatible between the two.
For it to work it is necessary to use multiprocessing.set_start_method('spawn') and set_executable() to set the interpreter to PyPy. Unfortunately this option of starting a fresh Python interpreter process is only available from Python 3.4, and PyPy only supports 3.3 for now. There is also this multiprocess fork of multiprocessing, which uses dill for a better serialization, so it could be worth integrating the py3.5 version of multiprocess into deco, so that set_start_method can be back-ported to an older version of Python and available in PyPy.
What do you think?

[alex-sherman]

Thanks for mentioning this, I've never used PyPy before, so let me make sure I understand its usefulness in this multiprocessing context correctly. I think it has two main benefits over the traditional Python runtime:

• JIT compilation for performance
• Sandboxing for security

I also see some pretty big downsides and complications:

• PyPy possibly has a different AST
• PyPy's JIT may be incompatible with the AST parsing deco performs
• The sandboxing feature is likely not available when using Pool, or is only as secure as vanilla Python's Pool
• multiprocessing.set_executable is only available on Window's

All things considered I think it would be a pretty huge undertaking to make deco capable of spawning PyPy processes. Monumentally harder to do it in a way that's portable, since we'd have to implement a lot of what Pool does. Probably the simplest alternative is to just run deco in PyPy, which on its own might not work. Out of curiosity have you been able to run a script using deco in a PyPy interpreter?

Lastly about using multiprocess rather than multiprocessing, I'm pretty hesitant to include extra dependencies that don't address some common issue. From the dill documentation, I think the main benefit seems to be better serialization of functions. My feeling, which could be totally wrong, is that it would be pretty rare to pass a function as an argument to an @concurrent function. However, even in this case I think my goal would be providing a better API for choosing the serialization method or providing an API for custom serialization rather than adding some function like use_dill().

In summary, I definitely think deco should give a user the flexibility to use PyPy or dill. At the same time, I think deco should probably not include PyPy or dill directly, but rather expose an API that would allow that flexibility. Already there is a tiny start towards this with concurrent.custom but it could use some documentation and some improvements. My preference will be to improve that API as a resolution to this issue.

Thanks again for commenting about this. I'm interested in your feedback on the above, mostly because it's possible I've missed the point of using PyPy or that I'm wrong about something.

[davidbrochart]

I was really thinking of PyPy as a replacement for Numba, plus it can speed up the execution of general code (not only Numpy code) that Numba can't.
I believe multiprocessing.set_executable is also available on Linux since Python 3.4, but as I said PyPy only supports Python 3.3 for now.
Running deco in PyPy means we would be lacking a lot of good things that only work on CPython. I know PyPy is catching up with supporting C extension libraries but still, CPython is the mainstream.
Let me play a bit with deco and PyPy, and I'll get back with more feedback.
Thanks for your answer.

[davidbrochart]

What I saw is that It works out of the box with CPython 3.4 spawning CPython 3.4 processes. I couldn't see if it works with CPython spawning PyPy processes because of some language features required by multiprocessing and not present in PyPy 3.3. So I guess I'll wait for PyPy 3.5 to get released. As you mentioned, there might be other problems related to a different AST.
Thanks for your feedback.