Remove hardcoded "pip" install from configure script

[ejmeitz]

Hello,

I am trying to build legate in a cross-compilation environment and the recent change which hardcodes pip into the configure script is blocking progress. The specific line is in the ensure_aedifix function here.

My build environment does have pip3, and my attempt to work around this is to manually clone the aefidix repository and add it to the PYTHONPATH. For some reason, the version parsing in the ensure_aedifix function cannot parse the version of aedifix when it is installed this way and I cannot make progress on the build.

Would it be possible to remove the assumption that the pip program is actually called pip (instead of say pip3) or provide another way of installing aedifix as that is also a legate library. Thanks!

[manopapad]

@Jacobfaib @bryevdv wdyt?

[Jacobfaib]

Hmmmm, normally this kind of issue only crops up when you call pip directly:

$ pip install whatever

instead of going through python -m pip. This would seem to imply that the pip package is called pip3. @ejmeitz is that the case on your end? For example, what does

$ pip list

Show for you?

[ejmeitz]

@Jacobfaib I did not appreciate that the generic pip or pip3 is irrelevant. The issue is due to an oddity of my build environment I think. I do not have much control over this environment. Its just the sandbox (alpine linux) Julia uses to cross-compile external libraries in a repeatable manner. The version of python shipped in this sandbox is 3.9 (too low for the configure script). I am able to install python 3.10, but for various reasons things like python3.10 -m ensurepip do not work. As a result, when I run the configure script with python3.10 the call to pip here crashes.

My current work around to get other dependencies like rich, is to manually download the wheels/git repo and add them to PYTHONPATH. This works for aedifix, but the version metadata doesn't seem to get parsed right when the package is installed this way and I fall into the except block of the ensure_aedifix function. This is the error when calling version("aedifix"):

importlib.metadata.PackageNotFoundError: No package metadata was found for aedifix

This is all that is returned by vars(aedifix)

{'__name__': 'aedifix', '__doc__': None, '__package__': 'aedifix', '__loader__': <_frozen_importlib_external._NamespaceLoader object at
 0x7f1663820c10>, '__spec__': ModuleSpec(name='aedifix', loader=<_frozen_importlib_external._NamespaceLoader object at 
0x7f1663820c10>, submodule_search_locations=_NamespacePath(['/workspace/srcdir/config_deps/aedifix'])), '__file__': None,
 '__path__': _NamespacePath(['/workspace/srcdir/config_deps/aedifix'])}

I can think of a couple hacky changes to ensure_pip that could help me get around this, but I'm also sure there's a reason it was designed this way in the first place. More than happy to make a PR once I understand the requirements!

[Jacobfaib]

The version of python shipped in this sandbox is 3.9 (too low for the configure script)...I am able to install python 3.10

Ahhh yes, that would be why pip refuses to download it... I hate to be the bearer of more bad news but... nv-legate/aedifix#7. The same is actually true for Legate as well. They both require 3.11. But due to a typo in our pyproject.toml, pip wasn't enforcing this.

Are you able to install Python 3.11 in your sandbox?

This works for aedifix, but the version metadata doesn't seem to get parsed right when the package is installed this way

Hmmm yeah... I guess this is expected, since you aren't technically "installing" the package in the normal sense. You are placing it on the system and essentially doing LD_LIBRARY_PATH=.... So it stands to reason that the normal packaging infrastructure knows nothing about your version of the library.

So instead of PYTHON_PATH, can you try editably installing aedifix:

$ git clone https://aedifix....
$ cd aedifix
$ pip install -e .

then import_lib should "know" about it

[ejmeitz]

Ah didn't see that the python version got bumped too. I can probably get 3.11, it will require me to update some other packages on the Julia side so it will take a few days. I'll let ping you once I'm able to try that.

[bryevdv]

Just reiterating what @Jacobfaib you would need to run pip install . or pip install -e . in the cloned aedifix repo, in order for setuptools-scm to actually do the work to vendor in the version information to the installed package. That said, if you get to a point where you could run either of those, I'm not sure why ensure_aedifix would also not just start working.

[ejmeitz]

Figured out a way to get 3.11. There are more issues, but this one is no longer a sticking point for me.