[3/3] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid]

[mohamedawnallah]

Description

• Update the Enrichment Documentation to include Milvus Search Enrichment Handler integration
• Add Python Example for Milvus Search Enrichment to the Documentation
• Add Jupyter Notebook Example for Milvus Search Enrichment
• Add Release Note for the added documentation and examples for Milvus enrichment transform

Closes #36176.
Towards #35046.
Prev #35577.

Motivation and Context

Follow-up on the discussion here #35216 (comment) and #35216 (comment).

Website Update Demo

website_update_demo.mov

Website Build: https://apache-beam-website-pull-requests.storage.googleapis.com/35467/documentation/transforms/python/elementwise/enrichment-milvus/index.html

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[mohamedawnallah]

Merging the master branch here after this PR #35216 merged

[mohamedawnallah]

This appears to be the last PR (in a 3-part series) after Apache Beam 2.67.0 to be released, changing the title

[github-actions]

Assigning reviewers:

R: @shunping for label python.
R: @kennknowles for label website.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[mohamedawnallah]

perhaps this not supposed to be in review queue yet until Apache Beam 2.67.0 version is released? stopping reviewer notifications

[mohamedawnallah]

stop reviewer notifications

[github-actions]

Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment assign set of reviewers

[damccorm]

R: @damccorm

[mohamedawnallah]

@damccorm –This PR is now ready for review. The precommit python ml test failure is unrelated and seems a transient failure:

=================================== FAILURES ===================================
_____________ PyODIForestTest.test_scoring_with_unmatched_features _____________
[gw2] linux -- Python 3.9.22 /runner/_work/beam/beam/sdks/python/test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-ml/py39-ml/bin/python
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-09-16T19:57:35.819235229+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>

During handling of the above exception, another exception occurred:

self = <apache_beam.ml.anomaly.detectors.pyod_adapter_test.PyODIForestTest testMethod=test_scoring_with_unmatched_features>

    def test_scoring_with_unmatched_features(self):
      # The model is trained with two features: a, b, but the input features of
      # scoring has one more feature (target).
      # In this case, we should either get rid of the extra feature(s) from
      # the scoring input or set `features` when creating the offline detector
      # (see the `test_scoring_with_matched_features`)
      detector = PyODFactory.create_detector(self.pickled_model_uri)
      options = PipelineOptions([])
      # This should raise a ValueError with message
      # "X has 3 features, but IsolationForest is expecting 2 features as input."
      with self.assertRaisesRegex(Exception, "is expecting 2 features"):
        with beam.Pipeline(options=options) as p:
>         _ = (
              p | beam.Create(self.get_test_data_with_target())
              | beam.Map(
                  lambda x: beam.Row(
                      **dict(zip(["a", "b", "target"], map(int, x)))))
              | beam.WithKeys(0)
              | AnomalyDetection(detector=detector))
E         AssertionError: "is expecting 2 features" does not match "<_MultiThreadedRendezvous of RPC that terminated with:
E         	status = StatusCode.DEADLINE_EXCEEDED
E         	details = "Deadline Exceeded"
E         	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-09-16T19:57:35.819235229+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
E         >"

apache_beam/ml/anomaly/detectors/pyod_adapter_test.py:149: AssertionError
----------------------------- Captured stderr call -------------------------

[mohamedawnallah]

Website build: https://apache-beam-website-pull-requests.storage.googleapis.com/35467/documentation/transforms/python/elementwise/enrichment-milvus/index.html

[damccorm]

Thanks! Had some notes, but overall it LGTM

[mohamedawnallah]

When running Milvus tests locally with version 2.5.10, I encountered the error panic: embedded etcd can not be used under distributed mode. After searching online, I found this appears to be a recent issue e.g appearing at milvus-io/milvus#43760. It looked like the ETCD_USE_EMBED process environment variable was no longer working as expected.

I initially tried upgrading the pymilvus client SDK to v2.6.X to align with the latest release mainly to fix that issue as stated by one of the milvus maintainers there. However, this caused compatibility Beam issues with the upper bounds of grpc-related python packages.

I resolved the issue by updating the intermediary Milvus daemon YAML file in this commit: b7c064f. This fixed the etcd while still being on version 2.5.10

[mohamedawnallah]

Thanks a lot @damccorm for the thoughtful review comments. I apologize for the significant delay in addressing them. I've now addressed all of them, and all relevant tests are passing in the CI

[damccorm]

Thanks - just had one last question, otherwise LGTM

[damccorm]

Thanks! Letting checks run, then will merge