Revert "[yaml] : switch js2py to pythonmonkey"

[derrickaw]

Reverts #37560

[gemini-code-assist]

Summary of Changes

Hello @derrickaw, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request reverses a prior change, restoring js2py as the JavaScript execution engine for YAML-based UDFs within Apache Beam. The primary goal is to ensure compatibility and stability by reverting the integration of pythonmonkey and re-establishing the js2py framework, including necessary adaptations for object serialization and dependency management.

Highlights

• Reverted JavaScript Engine: The pull request reverts the JavaScript execution engine used in Apache Beam's YAML module from pythonmonkey back to js2py.
• Dependency Management: The setup.py file was updated to remove the pythonmonkey dependency and reintroduce js2py, with a version constraint and a condition for Python versions less than 3.12.
• Codebase Cleanup: Extensive pythonmonkey-related code, including a custom dispatcher and callable classes, was removed from yaml_mapping.py. Corresponding js2py imports and logic were reinstated.
• Pickling Support for js2py: A custom _CustomJsObjectWrapper class was added to yaml_mapping.py to address pickling issues with js2py.JsObjectWrapper objects.
• Test Updates: Unit tests in yaml_udf_test.py were adjusted to reflect the switch back to js2py, updating import checks and skip conditions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• sdks/python/apache_beam/yaml/standard_io.yaml
  • Removed num_shards parameter from the WriteToJson transform definition.
• sdks/python/apache_beam/yaml/yaml_mapping.py
  • Removed imports related to atexit, importlib, os, queue, sys, threading, and uuid.
  • Reintroduced conditional imports for js2py and JsObjectWrapper.
  • Added _CustomJsObjectWrapper class to provide pickling support for js2py objects.
  • Removed the PythonMonkeyDispatcher and JavaScriptCallable classes.
  • Refactored _expand_javascript_mapping_func to use js2py for JavaScript evaluation and result conversion, including a new _js_object_to_py_object helper function.
  • Updated the error message for missing JavaScript support to reference js2py and Python 3.12 compatibility.
• sdks/python/apache_beam/yaml/yaml_udf_test.py
  • Removed importlib import.
  • Updated the JavaScript engine availability check from pythonmonkey to js2py.
  • Modified unittest.skipIf decorators to check for js2py installation status.
• sdks/python/setup.py
  • Replaced the pythonmonkey dependency with js2py, specifying a version range (>=0.74,<1) and a Python version constraint (python_version<"3.12").

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[derrickaw]

Run Prism_Python PreCommit 3.13

[derrickaw]

Run Python_Coverage PreCommit 3.10

[derrickaw]

Run Python_Examples PreCommit 3.11

[derrickaw]

Run Python_Integration PreCommit 3.13

[derrickaw]

Run Python_ML PreCommit 3.11

[tvalentyn]

Re test failure:

RuntimeError: Pipeline construction environment and pipeline runtime environment are not compatible. If you use a custom container image, check that the Python interpreter minor version and the Apache Beam version in your image match the versions used at pipeline construction time. Submission environment: beam:version:sdk_base:apache/beam_python3.13_sdk:2.73.0.dev. Runtime environment: beam:version:sdk_base:apache/beam_python3.13_sdk:2.72.0.dev.

seems like we have a might have regression, possibly in boot.go. The problem is that for the Dataflow test job (2026-02-20_14_47_19-12481031046029561904) we pass --sdk_location, but the artifact (new SDK, 2.73.0.dev), is not being installed.

[tvalentyn]

rootcause the py3.13 wheel format apache_beam-2.73.0.dev0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl doesn't match expected format:

beam/sdks/python/container/boot.go

Line 422 in 9ff96df

wheelName := fmt.Sprintf("cp%s-cp%s-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", pyVersion, pyVersion)

[Abacn]

What's the progress on this PR? Python tests are largely broken for a few days

[derrickaw]

What's the progress on this PR? Python tests are largely broken for a few days

Hi @Abacn, @tvalentyn is working on a fix for the broken tests. I think once those are fixed, then this can go in.

[tvalentyn]

there are several breakages, we might have to merge some PRs on red and work on stabilizing builds

[tvalentyn]

is this PR a pure rollback or there are changes on top of the rollback?

[github-actions]

Assigning reviewers:

R: @tvalentyn for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[derrickaw]

is this PR a pure rollback or there are changes on top of the rollback?

Almost pure rollback, there was one small conflict with setup.py.