pip install from PyPI broken

[bjlittle]

@pletzer The existing mechanism within setup.py is conda centric, in that it is searching for include files for vtk and netcdf.

However, when installing the mint dependencies from PyPI, these typically are installed by pip as binary wheels, and as such don't contain the source code include files, only the dynamic libraries.

As such, attempting to install mint==1.13.1 with pip fails (even when the pip dependencies are pre-installed):

> pip install python-mint==1.13.1
Collecting python-mint==1.13.1
  Using cached python-mint-1.13.1.tar.gz (81 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/bin/python3.8 /tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /var/tmp/tmp77o_acxy
       cwd: /var/tmp/pip-install-xksexzsc/python-mint_f7e92944d84743caa20f8a75797a088c
  Complete output (25 lines):
  Traceback (most recent call last):
    File "setup.py", line 36, in getCondaVTK
      version = list(include_dir.glob("vtk-*"))[-1].name
  IndexError: list index out of range
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
      main()
    File "/tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/var/tmp/pip-build-env-91_ztpfc/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/var/tmp/pip-build-env-91_ztpfc/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/var/tmp/pip-build-env-91_ztpfc/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 82, in <module>
      vtklib = getCondaVTK()
    File "setup.py", line 38, in getCondaVTK
      raise RuntimeError('ERROR: you need to "conda install vtk"')
  RuntimeError: ERROR: you need to "conda install vtk"
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/a5/e5/dc2d51f7ec2bdd483ae395fc08c8addb8d2e25e46f9cf12174ab4446213a/python-mint-1.13.1.tar.gz#sha256=ff09e3dbe5cb752d6e108b6efacd180110e0f42dbee32ccd1db620b9644eb5be (from https://pypi.org/simple/python-mint/) (requires-python:<3.9,>3.6). Command errored out with exit status 1: /tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/bin/python3.8 /tmp/persistent/itwl/miniconda3/envs/mint-install-pypi/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /var/tmp/tmp77o_acxy Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement python-mint==1.13.1 (from versions: 1.8.0, 1.13.1)
ERROR: No matching distribution found for python-mint==1.13.1

[bjlittle]

@pletzer python-mint 1.13.1 is now available on PyPI, see here, but sadly it's un-installable as is 😢

[pletzer]

@bjlittle: What are the options?

• Maybe one possibility would be to look for the include files under ~/miniconda3/envs/mintenv/include and the libraries under ~/miniconda3/envs/mintenv/lib? This will of course change from user to user - I'm assuming that pip install actually runs python setup.py on the computer where the library is being installed.
• Supplying the missing include files as part fo the source code could be another option
• Skip support for python-mint
• ?

[bjlittle]

@pletzer I've never quite encountered this issue before, but that's only because I've only had the opportunity to contribute to a couple of Python/C++ packages.

That said, I think it might be possible, but I'll need to experiment a wee bit to understand things a tad deeper.

My current thinking/assumption is that you only need to link against headers when building mint from source, and this is typically only done by a developer - and the deal here is that you've pre-installed all the relevant package dependencies with the necessary headers et al.

The issue we're hitting is that we've packaged mint as a sdist rather than a binary wheel, and this might be the nub of the problem. We've effectively painted ourselves into a corner, so to speak. If we were able to package mint as a self-contained binary wheel, then this wouldn't be a problem, I believe. If this is true, then this means the following:

• setup.py needs the smarts to only look for headers et al for an sdist, not when attempting to install as a wheel, but certainly when building a wheel
• we need to crack the nut of building a binary wheel for mint
  • I hit issues doing this during the last release (v1.8.0) and subsequently abandoned getting this to work (due to time constraints)... however, it looks like we may not be able to avoid this any longer, assuming my line of thinking is correct.

Is it okay to leave this with me? Is the clock ticking on this one? i.e., do we need to support PyPI for you to hit a specific deadline? Or is there some slack here for chin scratching, coffee drinking and experimentation? 🤔 ☕ 💥

[bjlittle]

This is purely a reference breadcrumb trail for me to follow though on, but it's worthwhile documenting here:

• https://stackoverflow.com/questions/59451069/binary-wheel-cant-be-uploaded-on-pypi-using-twine
• https://stackoverflow.com/questions/46599004/any-limitations-on-platform-constraints-for-wheels-on-pypi
• https://github.com/pypa/manylinux
• https://pypi.org/project/auditwheel/
• https://github.com/pypa/python-manylinux-demo

> docker run --rm -i -e PLAT=<plat> -v /path/to/dir/dist:/io /usr/bin/bash
$ auditwheel repair /io/<bdist_wheel> --plat ${PLAT} -w <wheelhouse_dir>

e.g., where <plat>=manylinux2014_x86_64

[bjlittle]

@pletzer This is a question related to building manylinux_x_y binary wheels for linux... but what the minimum acceptable version of Python you want to support for mint?

Is a minimum bar of py38 too high? i.e., do we need to support py36 and/or py37?

Note that, numpy 1.20.0 does not support py36, and thus is effectively forcing the conversation regarding the decommissioning of py36.

[pletzer]

@bjlittle happy to drop anything below python 3.8. Thanks

[pletzer]

Hi @bjlittle . Thanks for your investigations. Yes pip install fails because of missing headers (netcdf, vtk). Following are all the options we can think of:

• build a binary wheel. Potential problems might arise if the compilation linked against system libraries that are not available on the target. For instance, we may link against mesa but the target might have another, more performant OpenGL library installed. Or maybe no OpenGL at all.
• drop support for pip install. We may have reached the point where pip is not the right tool. Might be worth enquiring with the commnunity via stackflow to check if there is a solution we're missing.
• add the source of vtk and netcdf. Have setup.py compile everything. Problems will arise for environment that have module A depending on vtk version X and mint linked against vtk version Y, X != Y. (Same for netcdf.).
• Have pip vtk and libnetcdf recipes that store the include files. We might have to produce (and maintain) those ourselves.
  I'm going to focus on the mint implementation and will leave the packaging with you. So far I'm super happy to have a conda install working :)
  --Alex

[bjlittle]

@pletzer Okay, a quick update... the plot thickens 🧐

So I just release our python-stratify package from SciTools, which is a Python/Cython based package, and I took the opportunity to figure out how to build a bdist_wheel against a specific glibc for the first time (rather than only offering an sdist package) on PyPI, see https://pypi.org/project/stratify/0.2.post0/#files. I was targeting manylinux2014_x84_64, which supports CPython 3.7.8+, 3.8.4+, 3.9.0+ and pip >=19.3, and done this in a suitable docker container.

I think this might be the way forward... but I need to experiment a bit further.

It seems that given the contents of the mint platform specific linux wheel i.e., generated by python setup.py bdist_wheel, I need to use the auditwheel repair tool to convert it to a manylinux generic type wheel, which will then allow you to upload it to PyPI, as it guarantees a minimal compatible glibc for specific linux (this includes osx btw) platforms.

However, the easy win seems to be to build/repair the wheel as a manylinux_x_y wheel, which is restricted to CPython 3.8.10+, 3.9.5+, 3.10.0+. As you mentioned above, you're cool with that, so I'll see how far I can take it and whether it generates a wheel that works as we expect and need to.

I'll let you know how I get on.

At the very least I'm certainly learning a tonne of cool new stuff, so I'm totally happy chipping away at this 👍

[pletzer]

Sounds good Bill!

[kinow]

Saw this issue last night but forgot to comment. @bjlittle at Cylc we use a package called pyzmq, for communication between processes and machines. It uses a library called zmq, that is written in C++.

If I recall correctly, they have a setup.py that detects the platform of the build (Windows, Linux) and the C++ library version. I think it uses the headers only too, and produces a wheel that is compatible for the platform in which it was built.

The CI process (a mix of GH Actions and Circle CI) then allocates hosts for each platform to produce the wheels, and uploads everything to PYPI. Finally, when I pip install pyzmq pip finds the correct wheel in PYPI. See their 22.1.0 release files for an example of what's produced: https://pypi.org/project/pyzmq/#files

The code is hosted at https://github.com/zeromq/pyzmq. The README has more info about their building process, as well as their Wiki page https://github.com/zeromq/pyzmq/wiki/Building-and-Installing-PyZMQ.

So maybe the same could be done for mint, build the wheels with the right headers/libs we need, and publish to PYPI. Just in case that's helpful 👍

Bruno

[pletzer]

Thanks @kinow! I'll have to defer to @bjlittle. Happy to set up a meeting between the 3 of us to nail this down.