Fix PreCommit YAML Xlang job hanging by aIbrahiim · Pull Request #37685 · apache/beam

aIbrahiim · 2026-02-23T14:15:23Z

Fixes: #34798
Successful run: https://github.com/aIbrahiim/beam/actions/runs/22308583106
Related to #32603

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-02-23T14:15:46Z

Summary of Changes

Hello @aIbrahiim, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the stability and performance of Python dependency resolution within the build system. By upgrading 'pip', integrating the 'uv' package manager, and introducing a 'constraints.txt' file with explicit dependency pinning, it resolves a critical issue causing the PreCommit YAML Xlang job to hang. These changes ensure more reliable and faster package installations, particularly for complex ML-related dependencies.

Highlights

Dependency Management Tool Upgrade: Upgraded 'pip' to version '26.0.1' for improved resolution performance and bug fixes, and introduced 'uv' as a new, faster Python package installer.
Migration to 'uv' for Package Installation: Switched the package installation logic from 'pip install' to 'uv pip install' within the build process to address 'resolution-too-deep' issues.
Introduction of 'constraints.txt': Added a new 'constraints.txt' file to explicitly pin versions of critical Python packages, aiming to reduce backtracking during dependency resolution for 'installGcpTest'.
Explicit Dependency Pinning: Explicitly pinned 'namex' and 'optree' packages to specific versions ('0.0.9' and '0.16.0' respectively) in 'setup.py' and 'constraints.txt' to prevent conflicts, particularly those arising from 'tensorflow' and 'keras' dependencies.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
- Updated the 'pip' installation command to upgrade to 'pip==26.0.1' and also install 'uv'.
- Modified the package installation logic to use 'uv pip install' instead of 'pip install', incorporating a newly defined 'constraints.txt' and explicitly anchoring 'namex' and 'optree' packages.
sdks/python/constraints.txt
- Added a new file to define pinned versions for 'numpy', 'googleapis-common-protos', 'grpc-google-iam-v1', 'google-api-core', 'optree', and 'namex' for the 'installGcpTest' environment.
sdks/python/setup.py
- Added explicit version pins for 'namex==0.0.9' and 'optree==0.16.0' to the 'tensorflow-transform' extra.
- Added explicit version pins for 'namex==0.0.9' and 'optree==0.16.0' to the 'p312_ml_test' and 'p313_ml_test' extras.

Activity

No specific activity (comments, reviews, or progress updates) was provided in the context for this pull request.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Abacn · 2026-02-23T15:15:10Z

sdks/python/setup.py

          ] + ml_base,
          'p312_ml_test': [
              'datatable',
+              'namex==0.0.9',


cc: @tvalentyn is it fine to pin versions in "_test" extension? We usually set an upper bound only.

We can set an upper/lower bound, but agreed that it would be better to at least keep a range of valid values rather than pinning to a specific version.

@aIbrahiim do these actually need to be constrained this tightly?

@damccorm Agreed, using ranges instead of exact pins i updated to namex>=0.0.9,<0.2.0 and optree>=0.16.0,<0.19.0 so these constrain the resolver and avoid resolution too deep while still allowing minor updates if we hit resolver or compatibility issues we can narrow further

+1 to use ranges, and constrain the ranges to help the resolver pick the right one. We can use uv locally to find what compatible ranges are if pip doesn't solve this.

Overall, guidance is: https://cwiki.apache.org/confluence/display/BEAM/Dependency+management+guidelines+for+Beam+Python+SDK+maintainers.

github-actions · 2026-02-23T17:28:43Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

damccorm · 2026-02-23T21:06:51Z

sdks/python/setup.py

          ] + ml_base,
          'p312_ml_test': [
              'datatable',
+              'namex==0.0.9',


We can set an upper/lower bound, but agreed that it would be better to at least keep a range of valid values rather than pinning to a specific version.

@aIbrahiim do these actually need to be constrained this tightly?

damccorm · 2026-02-23T21:08:10Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy


+          def pythonSdkDir = project.project(":sdks:python").projectDir
+          // Python 3.13 requires numpy>=2.1.0; constraints.txt pins numpy<2 for py310-312.
+          def constraintsName = "3.13".equals(project.ext.pythonVersion) ? "constraints_py313.txt" : "constraints.txt"


Does constraints_py313.txt exist? Also, you can do this in a single file like https://github.com/aIbrahiim/beam/blob/89b4b5e0b80731943149dd7596feb93768507af3/sdks/python/setup.py#L378

Ahh yes thast's a better approach, I refactored to use the single file approach. constraints_py313.txt has been removed and Numpy is now specified in setup.py (and pyproject.toml)

Tried switching namex/optree to version ranges (>=0.0.9,<0.2.0 and >=0.16.0,<0.19.0) but that caused installGcpTest to fail in the Yaml_Xlang_Direct workflow so Im reverting to exact pins (namex==0.0.9, optree==0.16.0) so the ML stack installs reliably and we can revisit ranges once keras pins these dependencies

damccorm · 2026-02-23T21:11:47Z

sdks/python/setup.py

              # tensorflow-transform requires dill, but doesn't set dill as a
              # hard requirement in setup.py.
              'dill',
+              # keras deps namex/optree lack version bounds - pin to avoid resolver issues


What are the actual issues we're running into here?

I investigated the install failures and traced them to three main causes:

Resolver backtracking: Pip 25.1/26.0.1 was hitting its resolver limit on the tensorflow keras namex/optree chain amd since Keras doesnt pin namex/optree, adding version bounds reduces excessive backtracking

Also NumPy/Pandas ABI mismatch when uv selected NumPy 2.x but Pandas expected 1.x the tests failed with ValueError numpy.dtype size changed so constraining NumPy to 1.x for Python 3.10–3.12 fixed this

Also for Python 3.13 support NumPy 1.x doesnt support Python 3.13 so we require numpy>=2.1.0 for 3.13 via python_version markers

And i switched to uv since its resolver handles this dependency tree more reliably as the namex/optree version ranges (instead of exact pins) provide enough structure without triggering deep backtracking

tvalentyn · 2026-02-24T00:08:51Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

-                "pip install --pre --retries 10 --upgrade tox --no-cache-dir"
+                "pip install --pre --retries 10 --upgrade pip==26.0.1 --no-cache-dir && " +
+                "pip install --pre --retries 10 --upgrade tox --no-cache-dir && " +
+                "pip install uv"


i am vary of using uv in our tests, unless it were so common that our users were doing the same, and docs instructed users to do so.

tvalentyn · 2026-02-24T00:11:25Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

+          // Use uv instead of pip - pip was hitting resolution-too-deep on tensorflow->keras->namex/optree.
+          // Include namex/optree as explicit deps to constrain resolution.
+          // --prerelease allow: envoy-data-plane depends on betterproto==2.0.0b6 (beta).
+          def anchorPkgs = "namex==0.0.9 optree==0.16.0"


I think these deps are specific to a certain test suite, and thus configuring them is a matter of the test suite that needs them; BeamModulePlugin should concern with global configuration matters for the project; adding these deps doesn't feel like the right direction to me

tvalentyn · 2026-02-24T00:14:26Z

sdks/python/pyproject.toml

    "distlib==0.3.9",
-    # Numpy headers
-    "numpy>=1.14.3,<2.5.0", # Update setup.py as well.
+    # Numpy headers. py313 requires 2.1+; py<3.13 use 1.x for pandas ABI compat.


I don't follow why we need to cap numpy to < 2 for Python 3.10-3.12.

<2.5.0 seems like a wide enough bound , and i'd expect each python version to pick the right version that is compatible

tvalentyn · 2026-02-24T00:15:04Z

sdks/python/setup.py

-          # Use a strict upper bound.
-          'numpy>=1.14.3,<2.5.0',  # Update pyproject.toml as well.
+          # numpy: py310-312 use 1.x; py313 needs 2.x (1.x unsupported; avoids  # pylint: disable=line-too-long
+          # pandas ABI mismatch).


are there some detaills on this mismatch?

tvalentyn · 2026-02-24T00:23:17Z

sdks/python/setup.py

              # hard requirement in setup.py.
              'dill',
+              # namex/optree: pin to avoid resolver issues (lack version bounds)  # pylint: disable=line-too-long
+              'namex==0.0.9',


not an opinion, just wondering - should this be defined in ml_base instead? will

github-actions · 2026-02-24T03:07:01Z

Assigning reviewers:

R: @shunping for label python.
R: @damccorm for label build.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Fix PreCommit YAML Xlang job hanging

f9361c6

github-actions bot added python build labels Feb 23, 2026

Fix Python 3.13 dependency resolution for installGcpTest

89b4b5e

Abacn reviewed Feb 23, 2026

View reviewed changes

Amar3tto requested a review from damccorm February 23, 2026 15:22

damccorm reviewed Feb 23, 2026

View reviewed changes

aIbrahiim added 2 commits February 24, 2026 00:37

Fix Python 3.13 deps and resolver issues

dbda4f4

fix numpy 1.x in base_image_requirements for py<3.13

9709643

github-actions bot added the docker label Feb 23, 2026

aIbrahiim added 2 commits February 24, 2026 01:13

revert namex/optree to exact pins for installGcpTest

d03ed05

Fix setup.py lint and constraints for py313

7c0a8d3

tvalentyn reviewed Feb 24, 2026

View reviewed changes

github-actions bot added the Next Action: Reviewers label Feb 24, 2026

Comments

Conversation

aIbrahiim commented Feb 23, 2026 • edited by Amar3tto Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Feb 23, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aIbrahiim commented Feb 23, 2026 •

edited by Amar3tto

Loading