Refactor update compatibility checks and pipeline options access

[claudevdm]

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @claudevdm, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors how compatibility version checks are managed within the Apache Beam Python SDK. By centralizing the is_compat_version_prior_to logic into the PipelineOptions class, it streamlines the process of determining pipeline behavior based on compatibility settings. This change simplifies the API for version checks, enhances consistency across the codebase, and improves the testability of version-dependent logic by allowing compatibility options to be scoped effectively.

Highlights

• Centralized Compatibility Version Checks: The logic for checking compatibility versions (is_compat_version_prior_to) has been moved from a utility function in apache_beam.transforms.util directly into the PipelineOptions class. This centralizes version comparison logic and makes it accessible directly from pipeline options.
• Simplified Coder Initialization: The DeterministicFastPrimitivesCoderV2 class no longer requires update_compatibility_version as a direct parameter during initialization. Instead, it dynamically retrieves pipeline options to determine compatibility behavior.
• Dynamic Dill Usage Determination: The _should_force_use_dill function now fetches pipeline options dynamically to decide whether to force dill usage, removing its dependency on a registry parameter.
• Streamlined Testing for Compatibility: Test cases across various modules (coders_test_common.py, bigquery_file_loads_test.py, util_test.py) have been updated to use scoped_pipeline_options and mock.patch for setting and testing compatibility versions, eliminating direct manipulation of a global typecoders.registry.
• Removed Redundant Utility Functions: The is_v1_prior_to_v2 and is_compat_version_prior_to utility functions, along with their associated tests, have been removed from apache_beam.transforms.util as their functionality is now encapsulated within PipelineOptions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• .github/trigger_files/beam_PreCommit_Python_Dill.json
  • Updated the revision number from 2 to 3.
• sdks/python/apache_beam/coders/coders.py
  • Modified DeterministicFastPrimitivesCoderV2.__init__ to remove the update_compatibility_version parameter and fetch pipeline options dynamically.
  • Updated _should_force_use_dill to retrieve pipeline options dynamically instead of using a registry parameter.
  • Adjusted _update_compatible_deterministic_fast_primitives_coder to call _should_force_use_dill without arguments and DeterministicFastPrimitivesCoderV2 without the compatibility version parameter.
• sdks/python/apache_beam/coders/coders_test_common.py
  • Imported PipelineOptions and scoped_pipeline_options.
  • Removed the tearDown method that reset typecoders.registry.update_compatibility_version.
  • Wrapped test_deterministic_coder and test_deterministic_map_coder_is_update_compatible tests with scoped_pipeline_options.
  • Removed direct assignment to typecoders.registry.update_compatibility_version.
  • Updated test_cross_process_encoding_of_special_types_is_deterministic to use scoped_pipeline_options and remove direct typecoders.registry manipulation.
  • Removed typecoders import from the embedded script.
• sdks/python/apache_beam/coders/typecoders.py
  • Removed the self.update_compatibility_version = None assignment from TypeCoderRegistry.__init__.
• sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
  • Replaced util.is_compat_version_prior_to with p.options.is_compat_version_prior_to.
• sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
  • Replaced direct typecoders.registry.force_dill_deterministic_coders manipulation with mock.patch for _should_force_use_dill.
  • Removed typecoders import.
• sdks/python/apache_beam/options/pipeline_options.py
  • Added the is_compat_version_prior_to method to the PipelineOptions class.
• sdks/python/apache_beam/options/pipeline_options_test.py
  • Added a new CompatVersionTest class to thoroughly test the is_compat_version_prior_to method with various version strings.
• sdks/python/apache_beam/pipeline_test.py
  • Updated test_dir methods to include is_compat_version_prior_to in the expected list of options for PipelineOptions.
• sdks/python/apache_beam/transforms/external.py
  • Replaced the call to the is_compat_version_prior_to utility function with pcolls.pipeline._options.is_compat_version_prior_to.
• sdks/python/apache_beam/transforms/ptransform.py
  • Replaced the call to the is_compat_version_prior_to utility function with options.is_compat_version_prior_to.
• sdks/python/apache_beam/transforms/util.py
  • Removed the pipeline_options import.
  • Removed the is_v1_prior_to_v2 and is_compat_version_prior_to utility functions.
  • Replaced calls to is_compat_version_prior_to with pcoll.pipeline.options.is_compat_version_prior_to.
• sdks/python/apache_beam/transforms/util_test.py
  • Replaced direct typecoders.registry.force_dill_deterministic_coders manipulation with mock.patch for _should_force_use_dill.
  • Removed typecoders import.
  • Removed the CompatCheckTest class and its associated tests for is_v1_prior_to_v2.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[claudevdm]

/gemini review

[gemini-code-assist]

Code Review

This pull request effectively refactors the compatibility version checking mechanism by centralizing the logic within the PipelineOptions class. The is_compat_version_prior_to method is a welcome addition, improving code organization and testability. All affected code paths and tests have been updated to utilize this new approach, and the removal of the old utility functions and registry attributes streamlines the codebase. The changes are well-implemented and contribute positively to the maintainability of the project.

[github-actions]

Assigning reviewers:

R: @damccorm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

/gemini review

[damccorm]

LGTM unless Gemini catches something

[gemini-code-assist]

Code Review

This pull request provides a significant and well-executed refactoring of how compatibility version checks are handled. By introducing the is_compat_version_prior_to method on PipelineOptions and migrating all existing checks to use it, the logic is now centralized and more object-oriented. The removal of the old utility functions from transforms.util cleans up the codebase. Furthermore, the changes to the test suite, replacing global state manipulation with scoped_pipeline_options and mocking, are a great improvement for test isolation and reliability. The changes are thorough and consistent across the codebase. This is an excellent refactoring with no issues found.

[damccorm]

Maybe the first Gemini review I've seen without review comments 🎉