Skip to content

[GLUTEN-9525][FLINK]Support tps metric for source task#10023

Merged
shuai-xu merged 10 commits intoapache:mainfrom
KevinyhZou:support_tps_metrics
Oct 11, 2025
Merged

[GLUTEN-9525][FLINK]Support tps metric for source task#10023
shuai-xu merged 10 commits intoapache:mainfrom
KevinyhZou:support_tps_metrics

Conversation

@KevinyhZou
Copy link
Contributor

@KevinyhZou KevinyhZou commented Jun 20, 2025

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #9525)

this pr depends on: bigo-sg/velox#4

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@github-actions github-actions bot added the FLINK label Jun 20, 2025
@github-actions
Copy link

#9525

@KevinyhZou
Copy link
Contributor Author

Testing on both Kafka and Nexmark source is feasible, please take a review at this pr. @shuai-xu

@shuai-xu
Copy link
Contributor

How does flink update the counter?

@KevinyhZou
Copy link
Contributor Author

KevinyhZou commented Aug 28, 2025

How does flink update the counter?

The numRecodsOut & numBytesOut counter have already registered in flink, so we just get them from the registry, and update the value of them by velox metrics, and flink will get the updated value.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class SourceTaskMetrics {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does flink have some class that could be used for source metrics?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, such as KafkaSourceReaderMetrics. But it is a single class that can not be reused here, so we create this class which is smilar to flink's.

@shuai-xu shuai-xu merged commit 8b66b60 into apache:main Oct 11, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL][FLINK] Support tps metric for nexmark

2 participants