Refactor StreamGraphTranslator and GlutenOperators#32
Open
lgbo-ustc wants to merge 3 commits intoflink_202512from
Open
Refactor StreamGraphTranslator and GlutenOperators#32lgbo-ustc wants to merge 3 commits intoflink_202512from
lgbo-ustc wants to merge 3 commits intoflink_202512from
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
lgbo-ustc
force-pushed
the
flink_202512_lgbo
branch
3 times, most recently
from
2e7fb8c to
b147241
Compare
KevinyhZou
reviewed
Dec 25, 2025
| sourceContext.collect(row); | ||
| if (element.isRecord()) { | ||
| StatefulRecord record = element.asRecord(); | ||
| if (outClass.isAssignableFrom(RowData.class)) { |
Author
There was a problem hiding this comment.
下游算子的两种情况
- 下游算子不能offload。转成rowdata，再传输
- 下游算子可以offload。直接传递rowvector的指针
|
|
||
| @Override | ||
| public StatefulRecord deserialize(DataInputView source) throws IOException { | ||
| LOG.error("xxx deserialize"); |
| import java.io.Serializable; | ||
|
|
||
| // This bridge is used to convert the input data to RowVector. | ||
| public class VectorInputBridge<IN> implements Serializable { |
There was a problem hiding this comment.
感觉这块最好实现一个类：
StatefulRowData implements RowData {
private StatefulRecord record;
public int getInt() {
return record.getInt();
}
......
这样各个operator 的输入输出就可以统一使用RowData类型，也不用专门搞一个Bridge 来转换
| StatefulRecord record, | ||
| BufferAllocator allocator, | ||
| RowType outputType) { | ||
| if (outClass.isAssignableFrom(RowData.class)) { |
gluten-flink/runtime/src/main/java/org/apache/gluten/util/VectorInputBridge.java
Outdated
Show resolved
Hide resolved
lgbo-ustc
force-pushed
the
flink_202512_lgbo
branch
from
b147241 to
de835f1
Compare
|
Run Gluten Clickhouse CI on x86 |
lgbo-ustc
force-pushed
the
flink_202512_lgbo
branch
from
de835f1 to
c544115
Compare
lgbo-ustc
force-pushed
the
flink_202512_lgbo
branch
2 times, most recently
from
66498be to
406d5f1
Compare
lgbo-ustc
force-pushed
the
flink_202512_lgbo
branch
from
406d5f1 to
6e591b2
Compare
lgbo-ustc
force-pushed
the
flink_202512_lgbo
branch
from
651eddb to
c220c90
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
重构了 StreamGraphTranslator
StreamGraphTranslator 代码实现进行了一定的简化。 多个 operator 不再合并为一个，但仍然预留了可合并的可能。
重构了 GlutenOperator 的具体实现
之前为每个 Operator 实现了两个版本， 如 GlutenOneInputOperator 和 GlutenVectorOneInputVector。在这里，通过泛型统一到一种实现，只根据输入的数据类型采取不同的转换操作。
解决 RowVector 在算子间如何传递的问题
RowVector在算子间传递涉及到
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)