Build Failure

[smokestacklightnin]

When running pip install . (with or without -e, Python versions 3.9-3.11), there is a build failure when Bazel (tried with version 6.5.0 and 7.6.1) runs:

$ pip install .
Processing /tmp/tmp.6hQYuJFamo/ml-metadata
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting absl-py<2.0.0,>=0.9 (from ml-metadata==1.18.0.dev0)
  Using cached absl_py-1.4.0-py3-none-any.whl.metadata (2.3 kB)
Collecting attrs<24,>=20.3 (from ml-metadata==1.18.0.dev0)
  Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting grpcio<2,>=1.8.6 (from ml-metadata==1.18.0.dev0)
  Downloading grpcio-1.71.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting protobuf<5,>=4.25.2 (from ml-metadata==1.18.0.dev0)
  Using cached protobuf-4.25.7-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting six<2,>=1.10 (from ml-metadata==1.18.0.dev0)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Downloading grpcio-1.71.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 1.9 MB/s eta 0:00:00
Using cached protobuf-4.25.7-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: ml-metadata
  Building wheel for ml-metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for ml-metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [92 lines of output]
      /tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/dist.py:332: InformationOnly: Normalizing '1.18.0.dev' to '1.18.0.dev0'
        self.metadata.version = self._normalize_version(self.metadata.version)
      /tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: Apache Software License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running bazel_build
      Computing main repo mapping:
      Loading:
      Loading: 0 packages loaded
      Analyzing: target //ml_metadata:move_generated_files (0 packages loaded, 0 targets configured)
      Analyzing: target //ml_metadata:move_generated_files (0 packages loaded, 0 targets configured)
      
      ERROR: Traceback (most recent call last):
          File "/home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/internal/rule_support.bzl", line 226, column 36, in <toplevel>
                  deps_cfg = apple_common.multi_arch_split,
      Error: 'apple_common' value has no field or method 'multi_arch_split'
      ERROR: error loading package '@@com_github_grpc_grpc//src/compiler': at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:33:6: at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/ios.bzl:26:5: at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/internal/testing/ios_rules.bzl:22:5: at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/internal/testing/apple_test_bundle_support.bzl:62:5: initialization of module 'apple/internal/rule_support.bzl' failed
      ERROR: /tmp/tmp.6hQYuJFamo/ml-metadata/WORKSPACE:182:5: error loading package '@@com_github_grpc_grpc//src/compiler': at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:33:6: at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/ios.bzl:26:5: at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/internal/testing/ios_rules.bzl:22:5: at /home/william/.cache/bazel/_bazel_william/1304b14177bf42c21170b06ca3f0b90c/external/build_bazel_rules_apple/apple/internal/testing/apple_test_bundle_support.bzl:62:5: initialization of module 'apple/internal/rule_support.bzl' failed and referenced by '//external:grpc_python_plugin'
      ERROR: Analysis of target '//ml_metadata:move_generated_files' failed; build aborted: Analysis failed
      INFO: Elapsed time: 0.145s, Critical Path: 0.00s
      INFO: 1 process: 1 internal.
      ERROR: Build did NOT complete successfully
      FAILED:
      ERROR: Build failed. Not running target
      Traceback (most recent call last):
        File "/tmp/tmp.6hQYuJFamo/ml-metadata/.pixi/envs/default/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
          main()
        File "/tmp/tmp.6hQYuJFamo/ml-metadata/.pixi/envs/default/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/tmp.6hQYuJFamo/ml-metadata/.pixi/envs/default/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
          return _build_backend().build_wheel(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 435, in build_wheel
          return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 423, in _build
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 512, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 135, in <module>
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 115, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1102, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1102, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1102, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-3m0s9lux/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "<string>", line 108, in run
        File "/tmp/tmp.6hQYuJFamo/ml-metadata/.pixi/envs/default/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/usr/local/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '//ml_metadata:move_generated_files']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ml-metadata
Failed to build ml-metadata
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ml-metadata)

[smokestacklightnin]

CC: @peytondmurray

[peytondmurray]

I think this can't be built as is with bazel>6. Building with 6.x yields a different error:

$ USE_BAZEL_VERSION=6.x python setup.py bdist_wheel
...
ERROR: /home/pdmurray/.cache/bazel/_bazel_pdmurray/fb0f67d0f9901fc69841d7d3eee005d0/external/libmysqlclient/BUILD.bazel:14:8: Executing genrule @libmysqlclient//:configure failed: (Exit 1): bash failed: error executing command (from target @libmysqlclient//:configure) 
  (cd /home/pdmurray/.cache/bazel/_bazel_pdmurray/fb0f67d0f9901fc69841d7d3eee005d0/sandbox/linux-sandbox/42/execroot/ml_metadata && \
  exec env - \
    PATH=/home/pdmurray/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:/home/pdmurray/.pyenv/versions/mlm/bin:/usr/share/pyenv/libexec:/usr/share/pyenv/plugins/xxenv-latest/bin:/usr/share/pyenv/plugins/python-build/bin:/usr/share/pyenv/plugins/pyenv-virtualenv/bin:/home/pdmurray/.luarocks/bin:/home/pdmurray/.volta/bin:/home/pdmurray/.pyenv/shims:/opt/google-cloud-cli/bin/:/opt/miniconda3/bin/:/usr/lib/emscripten:/home/pdmurray/.config/scripts:/home/pdmurray/.cargo/bin:/home/pdmurray/bin:/home/pdmurray/go/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin:/usr/lib/rustup/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; export INSTALL_DIR=$(pwd)/bazel-out/k8-opt/bin/external/libmysqlclient/build
export TMP_DIR=$(mktemp -d -t build.XXXXXX)
mkdir -p $TMP_DIR
cp -R $(pwd)/external/libmysqlclient/* $TMP_DIR
cd $TMP_DIR
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release ${CMAKE_ICONV_FLAG-}
cd ..
cp -R ./build/* $INSTALL_DIR
rm -rf $TMP_DIR')
# Configuration: 84924e504f40736b447898638cca4d25136a7d578b52c0c2f3bd9723fd3721f9
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
CMake Error at CMakeLists.txt:5 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.


-- Configuring incomplete, errors occurred!
...
...
Target //ml_metadata:move_generated_files failed to build
ERROR: /home/pdmurray/.cache/bazel/_bazel_pdmurray/fb0f67d0f9901fc69841d7d3eee005d0/external/libmysqlclient/BUILD.bazel:42:11 Middleman _middlemen/@libmysqlclient_S_S_Clibmysqlclient-cc_library-compile failed: (Exit 1): bash failed: error executing command (from target @libmysqlclient//:configure) 
  (cd /home/pdmurray/.cache/bazel/_bazel_pdmurray/fb0f67d0f9901fc69841d7d3eee005d0/sandbox/linux-sandbox/42/execroot/ml_metadata && \
  exec env - \
    PATH=/home/pdmurray/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:/home/pdmurray/.pyenv/versions/mlm/bin:/usr/share/pyenv/libexec:/usr/share/pyenv/plugins/xxenv-latest/bin:/usr/share/pyenv/plugins/python-build/bin:/usr/share/pyenv/plugins/pyenv-virtualenv/bin:/home/pdmurray/.luarocks/bin:/home/pdmurray/.volta/bin:/home/pdmurray/.pyenv/shims:/opt/google-cloud-cli/bin/:/opt/miniconda3/bin/:/usr/lib/emscripten:/home/pdmurray/.config/scripts:/home/pdmurray/.cargo/bin:/home/pdmurray/bin:/home/pdmurray/go/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin:/usr/lib/rustup/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; export INSTALL_DIR=$(pwd)/bazel-out/k8-opt/bin/external/libmysqlclient/build

So it seems like the version of libmysqlclient used here requires cmake<3.5. Bazel never provides as much build isolation as is promised :/

cmake==3.5 was released 7 years ago, maybe we can bump libmysqlclient to a later version?