Skip to content
Build

[SPARK-XXXXX][SS] Add V1 to V2 offset log auto-upgrade #1307

Sign in to view logs

[SPARK-XXXXX][SS] Add V1 to V2 offset log auto-upgrade

[SPARK-XXXXX][SS] Add V1 to V2 offset log auto-upgrade #1307

Triggered via push March 3, 2026 22:49
@ericm-dbericm-db
pushed 9eab4f6
streaming-offset-v1-to-v2-migration
Status Failure
Total duration 1h 47m 45s
Artifacts 26

build_main.yml

on: push
Run  /  Check changes
1m 6s
Run / Check changes
Run  /  Base image build
1m 26s
Run / Base image build
Run  /  Protobuf breaking change detection and Python CodeGen check
1m 15s
Run / Protobuf breaking change detection and Python CodeGen check
Run  /  Java 17 build with Maven
44m 50s
Run / Java 17 build with Maven
Run  /  Java 25 build with Maven
40m 36s
Run / Java 25 build with Maven
Run  /  Run TPC-DS queries with SF=1
1h 22m
Run / Run TPC-DS queries with SF=1
Run  /  Run Docker integration tests
1h 27m
Run / Run Docker integration tests
Run  /  Run Spark on Kubernetes Integration test
0s
Run / Run Spark on Kubernetes Integration test
Run  /  Run Spark UI tests
33s
Run / Run Spark UI tests
Matrix: Run / build
Run  /  Build modules: sparkr
27m 51s
Run / Build modules: sparkr
Run  /  Linters, licenses, and dependencies
2m 39s
Run / Linters, licenses, and dependencies
Run  /  Documentation generation
36m 46s
Run / Documentation generation
Matrix: Run / pyspark
Fit to window
Zoom out
Zoom in

Annotations

12 errors and 1 warning
Run / Linters, licenses, and dependencies
Process completed with exit code 1.
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect
XXSC0
Run / Build modules: sql - other tests
Process completed with exit code 18.
OffsetSeqLogSuite.checkpoint version wins on restart (v1 to v2): org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite#L1
org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = a75b8aa7-a1a4-42b1-9250-ec83b8ef4732, runId = f27db403-e43c-4bc9-9bf6-e55bbd168a09] terminated with exception: Offset log is in V1 format but V2 format was requested via spark.sql.streaming.offsetLog.formatVersion=2. To migrate the offset log, set spark.sql.streaming.offsetLog.v1ToV2.autoUpgrade.enabled=true. Important: This is a one-way migration that cannot be rolled back. Ensure all batches are committed before enabling. See documentation for details. SQLSTATE: XXKST === Streaming Query === Identifier: [id = a75b8aa7-a1a4-42b1-9250-ec83b8ef4732, runId = f27db403-e43c-4bc9-9bf6-e55bbd168a09] Current Committed Offsets: {} Current Available Offsets: {} Current State: ACTIVE Thread State: RUNNABLE Logical Plan: ~WriteToMicroBatchDataSourceV1 FileSink[file:/home/runner/work/spark/spark/target/tmp/spark-f712a3e1-079f-46e9-95be-430e25214dbf], a75b8aa7-a1a4-42b1-9250-ec83b8ef4732, [path=/home/runner/work/spark/spark/target/tmp/spark-f712a3e1-079f-46e9-95be-430e25214dbf, checkpointLocation=/home/runner/work/spark/spark/target/tmp/spark-0880698c-69e2-43ee-99c6-953c9b9f2788], Append +- ~StreamingDataSourceV2ScanRelation[value#666273] MemoryStreamDataSource
Run / Build modules: pyspark-core, pyspark-errors, pyspark-streaming, pyspark-logger
No files were found with the provided path: **/target/test-reports/*.xml **/target/surefire-reports/*.xml. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size Digest
ericm-db~spark~2LAXXD.dockerbuild
44.3 KB
sha256:e6c2f2a3ebf7cd388192a05bd0fc02eacfc84d791982d41e0e5c10cc8c39a9d3
ericm-db~spark~2N3ROU.dockerbuild
37.2 KB
sha256:1867cd19725c2e65a8c41b73716dd7a9b15837b8c7e27c859647bbe87238298e
ericm-db~spark~NYW2NK.dockerbuild
41.1 KB
sha256:c24227940be4737466e419d50018d64eb4552420e01d7794a8907d13529b4d39
ericm-db~spark~QBRBV6.dockerbuild
39.4 KB
sha256:b9c36bbbe041863c93997742189456ba6c7b2ca55fa16e4d314c842a64fec565
site Expired
65.3 MB
sha256:aff1164fbb48c3d3490073318be9de9c4d20602ad8bb864469832dcf6fc32fc8
test-results-api, catalyst, hive-thriftserver--17-hadoop3-hive2.3
713 KB
sha256:39619441ad0dcc8d5450432c348381fd631972db7f8b389fac3b842116e1be4b
test-results-core, unsafe, kvstore, avro, utils, utils-java, network-common, network-shuffle, repl, launcher, examples, sketch, variant--17-hadoop3-hive2.3
37.6 KB
sha256:45a532200e0e8b749a7c409cbc7eb0f18bcc73dac13545f44b0d22fc9033e528
test-results-docker-integration--17-hadoop3-hive2.3
53.1 KB
sha256:ea2f7e88442b17d13bd8459370062633c14b625be97647647cb6ab3d085add8d
test-results-hive-- other tests-17-hadoop3-hive2.3
239 KB
sha256:2ee84d668562860eea13357ffa713618a562bca0b8d83ccddaf4f5244b537db6
test-results-hive-- slow tests-17-hadoop3-hive2.3
229 KB
sha256:7947e47c32bec3c5836f6db7e45078054fc5547919571055cf8e15b399321a95
test-results-mllib-local, mllib, graphx, profiler, pipelines--17-hadoop3-hive2.3
433 KB
sha256:0ddc922f1fa08310b9015b0f5ceeb352199a0ec28830bc2ed4e68548997eac9c
test-results-pyspark-connect--17-hadoop3-hive2.3-python3.12
202 KB
sha256:e7f86b61d83fc29b84a3e24c71dafeb8ddff75be0f921244aa16954faaf65c30
test-results-pyspark-mllib, pyspark-ml, pyspark-ml-connect, pyspark-pipelines--17-hadoop3-hive2.3-python3.12
99.6 KB
sha256:b9e400d4ef9599f955903a7b84010d0ea66663bae424e575ac115202529787ac
test-results-pyspark-pandas--17-hadoop3-hive2.3-python3.12
132 KB
sha256:74d9ad6b51f16aafea45d4803b3e0e877250ced36f5b01ce0ac0327a34b34388
test-results-pyspark-pandas-connect--17-hadoop3-hive2.3-python3.12
144 KB
sha256:17e9fcd2426bbcdd72974626f6f503f578bb14c2b5b037ec6c34dc783d1e239d
test-results-pyspark-pandas-slow--17-hadoop3-hive2.3-python3.12
108 KB
sha256:62a617381ed6fe042977eccbee0da9ef8bc5760df87e6f196a6290b25943ba27
test-results-pyspark-pandas-slow-connect--17-hadoop3-hive2.3-python3.12
116 KB
sha256:9102dd47df8218d3158506102b5e83a29bd20547c7c10a51d5a3d2eb8579b9ee
test-results-pyspark-sql, pyspark-resource, pyspark-testing--17-hadoop3-hive2.3-python3.12
204 KB
sha256:8a2d7fcff11a4bcfcf856ec1cb197456ed2368fb10aad313a4fb8d8807cd067b
test-results-pyspark-structured-streaming, pyspark-structured-streaming-connect--17-hadoop3-hive2.3-python3.12
31.2 KB
sha256:bbcd41822738a2bfa24c1abae8dc5625162cf7f14c1acd8f08d365c73c262cbb
test-results-sparkr--17-hadoop3-hive2.3
17.2 KB
sha256:6dda445d1150617d8ffd22244eb7dc0740340f889615350d7ee81b0de7fd77c1
test-results-sql-- extended tests-17-hadoop3-hive2.3
1.35 MB
sha256:3dbe30e11a5aeb46e59411f830e920dbc7ed9a833659ee095c41f1209e376afa
test-results-sql-- other tests-17-hadoop3-hive2.3
1.6 MB
sha256:4c27e95222df384bfc19585ce49fc465eacd82d0dddcbf8f76cfa8d38ab127b9
test-results-sql-- slow tests-17-hadoop3-hive2.3
1.36 MB
sha256:d3e3097844e573c9523bcd39bda82dcebf8df3a5114f6165e62000e16903598d
test-results-streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect--17-hadoop3-hive2.3
248 KB
sha256:c8ad9ba4674e3ee9e273a400e9bd8ff7c04783587ed35d235839a570295b2ca3
test-results-tpcds--17-hadoop3-hive2.3
4.91 KB
sha256:faf20f381519254874f5509f446a772737938de1c05fa9751bb953e4830a4c37
unit-tests-log-sql-- other tests-17-hadoop3-hive2.3
62.8 MB
sha256:65a1e8f2212613d4dfb33bc71f4ee353452b56bd6693d727ddb0a041cc9a535f