Rename Chunk to EmbeddableItem

[claudevdm]

The naming Chunk is confusing because it is associated with splitting a text document into smaller documents. EmbeddableItem keeps a unit of embeddable content more generic.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @claudevdm, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request undertakes a significant refactoring effort by renaming the core Chunk data type to EmbeddableItem. This change aims to improve clarity and generality, decoupling the data structure from the specific concept of text splitting. The refactoring involved a comprehensive update of all dependent components within the RAG framework, including embedding adapters, enrichment handlers, and ingestion writers, to align with the new naming convention. The introduction of a type alias ensures a smooth transition for existing users by maintaining backward compatibility.

Highlights

• Core Type Renaming: The central Chunk data type has been renamed to EmbeddableItem to provide a more generic and less ambiguous representation for content units that can be embedded.
• Backward Compatibility: An alias Chunk = EmbeddableItem has been introduced in apache_beam.ml.rag.types to ensure existing codebases using Chunk continue to function without immediate changes.
• API Consistency: All references, type hints, docstrings, and variable names across various RAG-related modules (embeddings, enrichment, ingestion) have been updated to consistently use the new EmbeddableItem type.
• Convenience Constructor: A new class method EmbeddableItem.from_text() was added to simplify the creation of EmbeddableItem instances from plain text.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• sdks/python/apache_beam/ml/rag/embeddings/base.py
  • Renamed create_rag_adapter to create_text_adapter and updated internal functions to use EmbeddableItem.
  • Added a backward compatibility alias create_rag_adapter = create_text_adapter.
• sdks/python/apache_beam/ml/rag/embeddings/base_test.py
  • Updated imports and function calls to use create_text_adapter.
  • Adjusted expected error messages to reflect EmbeddableItem.
• sdks/python/apache_beam/ml/rag/embeddings/huggingface.py
  • Updated EmbeddingsManager initialization to use create_text_adapter.
• sdks/python/apache_beam/ml/rag/embeddings/vertex_ai.py
  • Updated EmbeddingsManager initialization to use create_text_adapter.
• sdks/python/apache_beam/ml/rag/enrichment/bigquery_vector_search.py
  • Replaced Chunk with EmbeddableItem in parameters, docstrings, and internal logic for BigQuery vector search.
  • Updated error messages to refer to Item instead of Chunk.
• sdks/python/apache_beam/ml/rag/enrichment/milvus_search.py
  • Replaced Chunk with EmbeddableItem in type definitions, method signatures, and error messages for Milvus search.
• sdks/python/apache_beam/ml/rag/enrichment/milvus_search_test.py
  • Adjusted expected error messages in tests to reflect the EmbeddableItem naming.
• sdks/python/apache_beam/ml/rag/ingestion/alloydb.py
  • Updated docstring reference from Chunk to EmbeddableItem schema.
• sdks/python/apache_beam/ml/rag/ingestion/base.py
  • Updated base ingestion classes and transforms to accept EmbeddableItem instead of Chunk.
• sdks/python/apache_beam/ml/rag/ingestion/bigquery.py
  • Renamed ChunkToDictFn to EmbeddableToDictFn and updated related functions and docstrings.
• sdks/python/apache_beam/ml/rag/ingestion/cloudsql.py
  • Updated docstring reference from Chunk to EmbeddableItem schema.
• sdks/python/apache_beam/ml/rag/ingestion/milvus_search.py
  • Updated Milvus ingestion configurations and converters to handle EmbeddableItem objects.
• sdks/python/apache_beam/ml/rag/ingestion/mysql.py
  • Updated MySQL ingestion builders and converters to work with EmbeddableItem.
• sdks/python/apache_beam/ml/rag/ingestion/mysql_common.py
  • Updated common MySQL ingestion utilities, ColumnSpec definitions, and value functions to use EmbeddableItem.
• sdks/python/apache_beam/ml/rag/ingestion/postgres.py
  • Updated PostgreSQL ingestion builders and converters to work with EmbeddableItem.
• sdks/python/apache_beam/ml/rag/ingestion/postgres_common.py
  • Updated common PostgreSQL ingestion utilities, ColumnSpec definitions, and value functions to use EmbeddableItem.
  • Corrected type hints for metadata fields from intfloat to float and intstr to str.
• sdks/python/apache_beam/ml/rag/ingestion/spanner.py
  • Updated Spanner ingestion column specifications and converters to handle EmbeddableItem objects.
  • Adjusted error messages to refer to EmbeddableItem.
• sdks/python/apache_beam/ml/rag/types.py
  • Renamed Chunk dataclass to EmbeddableItem.
  • Added EmbeddableItem.from_text factory method for convenient construction.
  • Aliased Chunk to EmbeddableItem for backward compatibility.
  • Updated Embedding dataclass to include an optional metadata field.
  • Refined type hints for dense_embedding and sparse_embedding properties.
• sdks/python/apache_beam/ml/transforms/embeddings/vertex_ai.py
  • Updated type hints for contextual_text and docstrings to refer to EmbeddableItem.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[claudevdm]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the codebase by renaming Chunk to the more generic EmbeddableItem. The change is applied consistently across multiple files, including type hints, docstrings, and variable names. A backward compatibility alias Chunk = EmbeddableItem is thoughtfully included. Overall, this is a great improvement for code clarity. I've added a couple of minor suggestions to improve consistency in PTransform labels and code formatting.

[claudevdm]

R: @damccorm

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[damccorm]

Thanks - we can get an extra pass from Gemini, but this LGTM

[damccorm]

/gemini review

[gemini-code-assist]

Code Review

This pull request effectively renames the Chunk class to EmbeddableItem across the codebase, which is a positive change for clarity and generality as outlined in the PR description. The introduction of EmbeddableItem as a universal container for embeddable content, along with factory methods like from_text, significantly improves the API. Backward compatibility is well-maintained through the use of aliases. The changes are consistently applied to imports, type hints, variable names, and docstrings throughout the affected files. Additionally, the Embedding dataclass has been enhanced with a metadata field, which is a valuable functional improvement for richer embedding representations. Minor inconsistencies in example code and the scope of the Embedding dataclass change have been noted for further refinement.

[damccorm]

Thanks!

[Abacn]

Sounds a breaking change: #37646