Using juliapomdps and PyCall across python executables

[markkho]

msdm/msdm/algorithms/juliapomdps.py

Line 13 in f489c1d

except (ImportError, julia.core.UnsupportedPythonError):

juliapomdps relies on PyCall, which is installed in a way that links it to a particular python executable (e.g., in a virtual environment). If I try to run python -m msdm.tools.install_julia_pomdps and I already have PyCall set up (e.g., I set it up in a separate virtual environment), then it throws a julia.core.UnsupportedPythonError. The error message suggests two workarounds:

The easiest workaround is to pass `compiled_modules=False` to `Julia`
constructor.  To do so, first *reboot* your Python REPL (if this happened
inside an interactive session) and then evaluate:

    >>> from julia.api import Julia
    >>> jl = Julia(compiled_modules=False)

Another workaround is to run your Python script with `python-jl`
command bundled in PyJulia.  You can simply do:

    $ python-jl PATH/TO/YOUR/SCRIPT.py

See `python-jl --help` for more information.

For more information, see:

    https://pyjulia.readthedocs.io/en/latest/troubleshooting.html

PyCall has some workarounds as well (see discussion of virtual environments), but they all would require the user to interact with julia.

[cgc]

just did some testing. When I make a virtualenv with a different python version, I get the error as above. However, a second virtualenv of the same python version seems to be just fine. Wonder if your above error only happens across python versions @markkho? My guess is my env with matched python version is built against the same libpython, which is what PyCall needs to match.

Seems like our options are:

1. disable compiled modules (from above). bad for loading performance, and maybe generally bad for performance? Here's the docs

This is equivalent to julia’s command line option --compiled-modules=no and disables the precompilation cache mechanism in Julia. Note that this option slows down loading and using Julia packages especially for complex and large ones.

2. Run scripts via python-jl (from above). Not sure how this would influence other parts of the workflow people have; like how does jupyter work with this? It looks like this let's julia run the supplied python script, which means that it will simply be using the python it is linked to, not the python in the current virtualenv. script here

3. make the user rebuild for the new virtualenv (from PyCall readme). uses a command like: rm(Pkg.dir("PyCall","deps","PYTHON")); Pkg.build("PyCall")

4. supply the runtime as an environment variable (from PyCall readme). Given that julia is basically embedded, I'm not really sure how well this can work? We'd have to run code like the below before pyjulia is imported at the very least. However, this requires that the executable's libpython matches that of PyCall

import os, sys
os.environ['PYCALL_JL_RUNTIME_PYTHON'] = sys.executable

5. figure out some sort of scheme to have different julia package installs per libpython. This feels like the only surefire way to avoid headaches. I think this would roughly work as follows: 1) get some kind of reference/path to the current libpython. One strategy might be from distutils import sysconfig; sysconfig.get_config_var("LIBDIR"). Another could use pyjulia's code directly, turns out to be as easy as from julia import find_libpython; find_libpython.linked_libpython(). 2) before importing packages in pyjulia, make sure to enter the current julia environment using Pkg.activate($PATH) (docs). 3) this will require some scheme mapping libpython (maybe hash the path) to some reasonable place to store the julia env (not sure where this would live, especially when it's pip installed)

It seems like a fundamental constraint that PyCall can only build against a specific libpython, so I think the best solutions are either to rebuild (option 3) or do the work to figure out per-libpython installs (option 5).