SF1000 Status update (Aug 28 2025)

[GregoryKimball]

SF1000 Status update

logs and profiles

10 queries running on 4 drivers
+6 can run with 1 driver
+6 assuming we can fix errors, add spilling and match cudf-polars
=4.4x speedup

and then we can begin performance tuning

Action items

Query	Issues	Status
Q4	Segfault for 1 and 4 drivers *** Signal 11 (SIGSEGV) (0xf6e560010000) received by PID 22125 (pthread TID 0xf6f40d5cd840) (linux TID 22149) (code: address not mapped to object), stack trace: ***	tbd
Q10	segfault for 1 drivers, "nvCOMP crash" for 4 drivers	tbd
Q13	see SF100
Q17	OOM. see analysis below. maybe just needs spilling	tbd
Q18	OOM. Strangely OOMs with 1 driver but passes with 4 drivers
Q21	concatenate exceeds row limit. needs a streaming join? for anti or right semi (filter) perhaps	tbd

"nvCOMP crash" impacting Q3,5,8,9,10,14,20 on 4 drivers Exceptions.h:66] Line: /workspace/velox/velox/exec/Driver.cpp:574, Function:operator(), Expression: Operator::getOutput failed for [operator: TableScan, plan node ID: 3]: CUDF failure at: /buildcache/release/_deps/cudf-src/cpp/src/io/parquet/reader_impl_chunking_utils.cu:627: Error during decompression, Source: RUNTIME, ErrorCode: INVALID_STATE

Q17 OOM

Steadily builds up tablescan-join-aggregate

cudf-polars solves this with spilling in UVM

https://drive.google.com/file/d/1EiYrMthoQHYWleLpNzHocFx8E5vCoSDt/view?usp=drive_link

Q18 OOM

Polars spills heavily
https://drive.google.com/file/d/1c5Ev4fI27yQur5aYlnxbUKuqaMmqipvZ/view?usp=drive_link

System information

GH200
https://github.com/rapidsai/velox-testing on main
Velox on mattgara/upstream-tpch-bench-fixes

[GregoryKimball]

Running with latest branches is showing a new set of TableScan failures.

Branches:
https://github.com/mattgara/velox/tree/upstream-tpch-bench-fixes-25.10
https://github.com/rapidsai/velox-testing/tree/mattgara/velox-build-with-nvjitlink

4 drivers

Q3, Q10, Q11, Q12,
reader_impl_chunking_utils.cu:635: Error during decompression

Q5, Q7, Q8
cuco/detail/storage/counter_storage.cuh96: cudaErrorIllegalAddress an illegal memory access was encountered
Comment: "The illegal memory access occurs during a memcpy from device to host, most likely because the device buffer was freed before the memcpy, leading to the error."
This error only occurs for multi-driver.

Q17
parallel_for failed: cudaErrorIllegalAddress: an illegal memory access was encountered

Q21
Exceptions.h:66] Line: /workspace/velox/velox/exec/Driver.cpp:574, Function:operator(), Expression: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: CUDF failure at:/buildcache/release/_deps/cudf-src/cpp/src/io/parquet/reader_impl_preprocess_utils.cu:263: Parquet header parsing failed with code(s) while counting page headers 0x1, Source: RUNTIME, ErrorCode: INVALID_STATE

Q4 segfault and Q13 stringview error continue

1 driver

the cuco, most of the decompression and page header errors go away. perhaps multi-driver is freeing memory too early.

Q4 Q10 segfault
Q13 error with known fix
Q17, Q18 OOM
Q20 reader_impl_chunking_utils.cu:635: Error during decompression
Q21 row count overflow

[wence-]

I added another column to the SF1K sheet from my run of cudf-polars on 2025-09-09 using a slightly tweaked (non-default) block size of 8_000_000_000, and RMM async MR (no spilling required, at that scale we can just squeak in on a GH200), timings are run each query three times, run all queries back to back, take best time for each query. That shaves another ~10% from the total runtime compared to your run on 2025-08-27.