Spark 4.1: Display write metrics on SQL UI #15104
Open
+978
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull request overview
This PR ports write metrics display functionality from Spark 4.0 to Spark 4.1, enabling write operation metrics to be shown in the Spark SQL UI. The changes introduce custom metric classes for tracking various write operations and integrate them with Spark's connector API.
Changes:
- Added 25 new custom metric classes extending
CustomSumMetricto track data files, delete files, records, and file sizes for added/removed/total categories - Integrated metrics reporting into
SparkWriteandSparkPositionDeltaWriteby implementingreportDriverMetrics()andsupportedCustomMetrics() - Enhanced
BaseTableto support combining multiple metrics reporters
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| TotalRecords.java | Defines metric for tracking total record count |
| TotalPositionalDeletes.java | Defines metric for tracking total positional deletes |
| TotalFileSizeInBytes.java | Defines metric for tracking total file size |
| TotalEqualityDeletes.java | Defines metric for tracking total equality deletes |
| TotalDeleteFiles.java | Defines metric for tracking total delete files |
| TotalDataFiles.java | Defines metric for tracking total data files |
| RemovedRecords.java | Defines metric for tracking removed records |
| RemovedPositionalDeletes.java | Defines metric for tracking removed positional deletes |
| RemovedPositionalDeleteFiles.java | Defines metric for tracking removed positional delete files |
| RemovedFileSizeInBytes.java | Defines metric for tracking removed file size |
| RemovedEqualityDeletes.java | Defines metric for tracking removed equality deletes |
| RemovedEqualityDeleteFiles.java | Defines metric for tracking removed equality delete files |
| RemovedDeleteFiles.java | Defines metric for tracking removed delete files |
| RemovedDataFiles.java | Defines metric for tracking removed data files |
| AddedRecords.java | Defines metric for tracking added records |
| AddedPositionalDeletes.java | Defines metric for tracking added positional deletes |
| AddedPositionalDeleteFiles.java | Defines metric for tracking added positional delete files |
| AddedFileSizeInBytes.java | Defines metric for tracking added file size |
| AddedEqualityDeletes.java | Defines metric for tracking added equality deletes |
| AddedEqualityDeleteFiles.java | Defines metric for tracking added equality delete files |
| AddedDeleteFiles.java | Defines metric for tracking added delete files |
| AddedDataFiles.java | Defines metric for tracking added data files |
| SparkWriteBuilder.java | Adds custom metrics support to write builder |
| SparkWrite.java | Integrates metrics reporter and implements reportDriverMetrics() |
| SparkPositionDeltaWrite.java | Integrates metrics reporter and implements custom metrics methods |
| SparkWriteUtil.java | Provides utility methods for creating custom metrics and task metrics |
| InMemoryMetricsReporter.java | Adds commitReport() method to retrieve commit metrics |
| BaseTable.java | Adds combineMetricsReporter() method for combining metrics reporters |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...k/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/metrics/AddedFileSizeInBytes.java
Outdated
Show resolved
Hide resolved
...v4.1/spark/src/main/java/org/apache/iceberg/spark/source/metrics/RemovedFileSizeInBytes.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Jan 22, 2026
| return (ScanReport) metricsReport; | ||
| } | ||
|
|
||
| @Nullable |
Contributor
There was a problem hiding this comment.
why the deviation from the behavior defined in scanReport()?
Member
Author
There was a problem hiding this comment.
commitReport is called in every Spark write while only some of them could have commits to Iceberg tables.
Contributor
There was a problem hiding this comment.
yeah that's why you should either end up with a null report or an actual commit report
...k/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/metrics/AddedFileSizeInBytes.java
Outdated
Show resolved
Hide resolved
...v4.1/spark/src/main/java/org/apache/iceberg/spark/source/metrics/RemovedFileSizeInBytes.java
Outdated
Show resolved
Hide resolved
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
Show resolved
Hide resolved
manuzhang
force-pushed
the
spark-4.1-write-metrics
branch
2 times, most recently
from
1c63813 to
2da3d86
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.