Add bolt support for paimon#8
Merged
WangGuangxin merged 1 commit intoWangGuangxin:add_bolt_backendfrom Mar 12, 2026
Merged
Conversation
|
Run Gluten Clickhouse CI on x86 |
Owner
|
@ZacBlanco there are conflicts, can you help resolve it? |
This change introduces all of the necessary paimon-specific changes that were included in olap/gluten's master branch but were not ported over when switching to OSS gluten's main branch The following changes were made - PaimonScanTransformer is now AbstractPaimonScanTransformer in order to support a other backend's implementation for paimon. Bolt-specific features are now included in an extended class BoltPaimonScanTransformer to facilitate the bolt-specific requirements. - Added support for adding extension info from the scan transformer into the advanced_extension field for a ReadRel. Also added protos for the paimon-specific advanced extension which are automatically generated and compiled for both Java and C++ versions. - The above also coincides with adding support for passing the "tableParameters" field of a HiveTableHandle when converting the substrait plan into bolt. - Added protos specific to paimon splits in order to serialize and deserialize paimon-specific split information from gluten into bolt. Previously this was done in a hacky way using strings and comma-separated lists. This new version uses protobuf both on the Java and C++ side to communicate paimon-specific split information via a LocalFiles' file_format field oneof definition in algebra.proto. One of the new options added is for "PaimonReadOptions" which contains the required paimon information. - Added a bolt-specific paimon suite which mirrors all of the previous test cases from the master branch.
Author
|
It is fixed now |
|
Run Gluten Clickhouse CI on x86 |
WangGuangxin
merged commit 6dbdc7c
into
WangGuangxin:add_bolt_backend
3 of 12 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces support for paimon in the bolt backend
The following changes were made
PaimonScanTransformer is now AbstractPaimonScanTransformer in order to support a other backend's implementation for paimon. Bolt-specific features are now included in an extended class BoltPaimonScanTransformer to facilitate the bolt-specific requirements.
Added support for adding extension info from the scan transformer into the advanced_extension field for a ReadRel. Also added protos for the paimon-specific advanced extension which are automatically generated and compiled for both Java and C++ versions.
The above also coincides with adding support for passing the "tableParameters" field of a HiveTableHandle when converting the substrait plan into bolt.
Added protos specific to paimon splits in order to serialize and deserialize paimon-specific split information from gluten into bolt. Previously this was done in a hacky way using strings and comma-separated lists. This new version uses protobuf both on the Java and C++ side to communicate paimon-specific split information via a LocalFiles' file_format field oneof definition in algebra.proto. One of the new options added is for "PaimonReadOptions" which contains the required paimon information.
Added a bolt-specific paimon suite which mirrors all of the previous test cases from the old branch.