Investigate problems with Buildout on Python 3.10 and macOS >= 11

[amotl]

About

Coming from crate/crate-python#428, where we found GHA croaks when running buildout with Python 3.10 on macOS 11 and macOS 12, this is a dedicated repository to reproduce the observations. pypa/setuptools#2514 is the best reference we could find about the issue.

The problem only appears in the context of Buildout, when the package to be installed pulls in transitive dependencies as wheels, fails on that, and then wants to invoke the gcc compiler. As an example package, we used SQLAlchemy in this example, which transitively pulls in the greenlet package.

Symptoms

Assessment

• On Python 3.9, everything works smoothly.
• Python 3.10 has problems on both macOS 11 and 12.
• Python 3.10 works well on macOS 10, though.
• Using Buildout 2.13.7 vs. the upcoming Buildout 3.0.0rc3 does not make any difference.

Appendix

The error output does not reveal much, but the problem is that the Buildout process is not able to install SQLAlchemy.

No eggs found in /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/easy_install-uvggdcg2/zope.interface-5.2.0/egg-dist-tmp-s7mlm8qo (setup script problem?)

Invoking tests with CrateDB 4.8.0 and SQLAlchemy 1.4.37
Requirement already satisfied: pip in /Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages (22.1.2)
Collecting zc.buildout==2.13.4
  Downloading zc.buildout-2.13.4-py2.py3-none-any.whl (153 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.0/154.0 kB 7.3 MB/s eta 0:00:00
Collecting setuptools<52,>=8.0
  Downloading setuptools-51.3.3-py3-none-any.whl (786 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 786.2/786.2 kB 9.8 MB/s eta 0:00:00
Installing collected packages: setuptools, zc.buildout
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
Successfully installed setuptools-51.3.3 zc.buildout-2.13.4
Creating directory '/Users/runner/work/crate-python/crate-python/eggs'.
Creating directory '/Users/runner/work/crate-python/crate-python/bin'.
Creating directory '/Users/runner/work/crate-python/crate-python/parts'.
Creating directory '/Users/runner/work/crate-python/crate-python/develop-eggs'.
Develop: '/Users/runner/work/crate-python/crate-python/.'
Getting distribution for 'zc.recipe.testrunner==2.2'.
Got zc.recipe.testrunner 2.2.
Getting distribution for 'zope.testrunner==5.2'.
Got zope.testrunner 5.2.
Getting distribution for 'zc.recipe.egg==2.0.7'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
Got zc.recipe.egg 2.0.7.
Getting distribution for 'zope.interface==5.2.0'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching 'coverage.xml' found anywhere in distribution
warning: no previously-included files matching 'appveyor.yml' found anywhere in distribution
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'benchmarks'
No eggs found in /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/easy_install-uvggdcg2/zope.interface-5.2.0/egg-dist-tmp-s7mlm8qo (setup script problem?)
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface==5.2.0'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
 Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 2174, in main
    getattr(buildout, command)(args)
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 701, in install
    [self[part]['recipe'] for part in install_parts]
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 701, in <listcomp>
    [self[part]['recipe'] for part in install_parts]
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1324, in __getitem__
    options._initialize()
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1432, in _initialize
    self.initialize()
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1438, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1381, in _install_and_load
    zc.buildout.easy_install.install(
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 957, in install
    return installer.install(specs, working_set)
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 730, in install
    for dist in self._get_dist(req, ws):
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 574, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 1745, in _move_to_eggs_dir_and_compile
    [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: not enough values to unpack (expected 1, got 0)

[amotl]

When using SYSTEM_VERSION_COMPAT=1, as suggested at pypa/setuptools#2514 (comment) by @idgserpro, we are able to observe what is going wrong. Beforehand, the output was too thin.

As we can clearly see from the output below ¹, it looks like the build process attempts to compile extensions and croaks because it thinks the gcc compiler is not installed or not available.

Getting distribution for 'greenlet!=0.4.17'.
[...]
gcc: error: sh -c '/Applications/Xcode_13.2.1.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find gcc 2> /dev/null' failed with exit code 256: (null) (errno=Invalid argument)
xcode-select: Failed to locate 'gcc', requesting installation of command line developer tools.
error: Setup script exited with error: command '/usr/bin/gcc' failed with exit code 72

This observation raises two questions:

1. Why does the procedure try to invoke the compiler at all and not pull binary wheels from PyPI instead?
2. Why does the procedure not see the gcc compiler when it is falling back to it?

Footnotes

1. https://github.com/crate-workbench/test-gha-macos-buildout-sqlalchemy/runs/6948865602#step:5:324 ff. ↩

[amotl]

@proddata just reported he is observing the same thing with Python 3.10 on his workstation running macOS 12.3.1 (21E258), when running buildout on the original crate-python repository.

% buildout -N
Develop: '/Users/gtraar/Developer/crate/crate-python/.'
Getting distribution for 'zope.interface==5.2.0'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching 'coverage.xml' found anywhere in distribution
warning: no previously-included files matching 'appveyor.yml' found anywhere in distribution
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'benchmarks'
No eggs found in /var/folders/q3/c5v154l92g93jzxf50bjvgdc0000gn/T/easy_install-w80sr0_w/zope.interface-5.2.0/egg-dist-tmp-igh7377t (setup script problem?)
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface==5.2.0'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/gtraar/Developer/crate/crate-python/.venv/lib/python3.10/site-packages/zc/buildout/buildout.py", line 2174, in main
    getattr(buildout, command)(args)
[...]
  File "/Users/gtraar/Developer/crate/crate-python/.venv/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1438, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
  File "/Users/gtraar/Developer/crate/crate-python/.venv/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1381, in _install_and_load
    zc.buildout.easy_install.install(
  File "/Users/gtraar/Developer/crate/crate-python/.venv/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 957, in install
    return installer.install(specs, working_set)
[...]
  File "/Users/gtraar/Developer/crate/crate-python/.venv/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 574, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/gtraar/Developer/crate/crate-python/.venv/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 1745, in _move_to_eggs_dir_and_compile
    [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: not enough values to unpack (expected 1, got 0)

[amotl]

With 37f70c6, we added jobs to CI/GHA for probing Linux and Windows (workflows/linux.yml, workflows/windows.yml) in order to get a broader picture. The outcome confirms that it is really only failing on macOS (workflows/macos.yml).

[amotl]

We just reported our observations to the people at Buildout per buildout/buildout#609.

[amotl]

We also reported our observations to the people at GHA per actions/runner-images#5791.