[Discussion] What’s the appropriate KVIKIO_NTHREADS setting for GDS?

[JigaoLuo]

Hi KVIKIO team,

I’d like to share a performance debugging story from this afternoon that I think is somewhat unique, and hopefully useful to others.

Let me start by describing the setup:

• hardware: a DGX A100 system with 4 PCIe4.0 SSDs, each capable of ~6.4 GB/s bandwidth, measured by GDSIO.
  • Previously, these SSDs were configured as a single RAID-0 volume.
  • This week, we split the RAID and mounted each SSD as an independent filesystem.
• my workload involves multistream, multithreaded Parquet reading, which is a strong assumption: I have implemented a multistream, multithreaded Parquet reader based on libcudf. In my design, each Parquet read call retrieves only a small chunk (such as a row group like 32MB), and multiple worker threads concurrently issue these Parquet read calls. This approach differs from traditional readers, which may process larger chunks.

My Parquet reader setup is a bit different—I operate at the row group level, which means I typically launch around 60 separate cudf read_parquet calls per file.

With RAID-0

Life was simple with RAID-0. I could easily saturate the SSD bandwidth:

• GDSIO: Using 4 threads and large read chunks
• In my parquet reader: with KVIKIO_NTHREADS=16 and 8 worker threads issuing cudf read_parquet calls

Performance was excellent—reading Parquet at ~23 GB/s into one A100. Decoding and decompression were pipelined, and the overhead from 16 KVIKIO threads plus 8 cudf workers was kind of minimal. Everything worked beautifully thanks to KVIKIO and GDS.

Without RAID

For various reasons, we decided to un-RAID the SSDs and mount them individually. I reran my previous setup:

• With GDSIO, I could still saturate a single SSD (~6.4 GB/s) using 4 threads.
  • By issuing 4 concurrent GDSIO commands—each targeting a different SSD and using 4 threads—I again reached ~23 GB/s. That was expected.
• Using my Parquet reader with KVIKIO_NTHREADS=16 and 8 cudf worker threads, I could read from a single SSD at ~6.5 GB/s. Also expected.

So far, everything looked good.

Performance hunting

Then I tried to scale up and saturate all 4 SSDs simultaneously, reading parquets. Theoretically, this shouldn’t be hard. I launched 8 threads per SSD to read Parquet files, and scaled KVIKIO_NTHREADS linearly to 64. But with this setup, I only achieved ~15 GB/s, with huge fluctuations in bandwidth.

(I spent two hours checking thread pool setup, PCIe traffic, iostat, RMM MR, and nsys profiling—but I’ll skip those details here.)

Eventually, I found the root cause: KVIKIO_NTHREADS was simply too low. To saturate all 4 SSDs and PCIe, I needed to set KVIKIO_NTHREADS=128. With that, I saw stable bandwidth and full utilization.

Discussions

I’ll share more observations later, but I wanted to start a discussion around a few questions (you can also add anything you like):

• What’s the rule of thumb for choosing a good KVIKIO_NTHREADS value?
• What are standard workarounds when working with multiple SSDs?
• (I will share nsys profile) How do you handle thread pool contention in this context?
• And a comment: I now realize that removing RAID comes with a significant CPU cost—something I hadn’t anticipated when making the change.

Looking forward to hearing your thoughts and experiences!

[JigaoLuo]

Here I’m showing my GDS bandwidth monitoring results by plotting bandwidth numbers reported from /proc/driver/nvidia-fs/stats alongside nsys.

• There are all in the later case: Without RAID
• The workload is just reading parquet with 10 repeats.

I also manually annotated the KVIKIO I/O wait times, which stood out prominently in the nsys traces—they’re quite significant. It seems the thread pool is experiencing heavy contention, and I haven’t found a clean fix for it yet:

• Reducing either KVIKIO_NTHREADS or the number of cudf worker threads prevents the system from reaching peak I/O bandwidth.

KVIKIO_NTHREADS = 64

KVIKIO_NTHREADS = 128

[JigaoLuo]

I hope everything above makes sense—if not, I’d be happy to share my nsys or any other details on my end, and walk you through it.

If I were to summarize the TLDR takeaway from this discussion issue: keeping a RAID-0 setup might actually be a very good idea.

[kingcrimsontianyu]

Thanks Jigao for raising the issue with great details. There are quite a few variables here, the multi-SSD setup, GDS, and parquet reading, where their handling in KvikIO/libcudf may demand further improvement.

Would it be possible to start the investigation by using KvikIO alone without parsing the parquet file (i.e. just making I/O operations), and having each thread that calls kvikio::FileHandle::pread() read the file with a unique offset? I'd like to know if the reduced bandwidth could be observed again in this simplified setup, if this makes sense.

[JigaoLuo]

Yes, that’s something I overlooked in the earlier observation because I did not have time.

One question: is there a tool similar to GDSIO or FIO that I could use to benchmark raw I/O? I noticed a Python file in the benchmark folder—would that be the closest equivalent to GDSIO or FIO?

It’s worth discussing with you what defines a fair comparison. From my perspective, this analysis is structured around contrasting RAID0 and non-RAID configurations.

---

One point I’d like to make:

• with RAID0 in place, setting KVIKIO_NTHREADS to around 16 is sufficient for my parquet experiments. From my side, it is a rather fair comparison.
• without RAID, I launch GDSIO four times per SSD—16 threads in total—to fully saturate 4 SSDs.

[JigaoLuo]

One thing that comes to mind today: reading from more SSDs also increases memory pressure—could the contention be coming from KVIKIO’s memory allocation via RMM?

• Also, I remember KVIKIO used to have NVTX color labels, but I don’t see them recently anymore.

[kingcrimsontianyu]

is there a tool similar to GDSIO or FIO that I could use to benchmark raw I/O? I noticed a Python file in the benchmark folder—would that be the closest equivalent to GDSIO or FIO?

I think FIO and GDSIO are the most feature complete, reliable and versatile tools out there for benchmark purpose. The Python benchmark code in KvikIO is intended to evaluate the implementation performance and focuses on sequential read and write, and I think we need to revisit this script some time in the future for improvement, for example, to ensure the setup be consistent with FIO/GDSIO defaults or expose more tunable parameters.

reading from more SSDs also increases memory pressure—could the contention be coming from KVIKIO’s memory allocation via RMM?

KvikIO uses CUDA driver API and does not internally use RMM. The overhead of per-thread, bounce buffer (16 MiB size by default, pinned memory, for H2D copies) allocations in KvikIO only exists in the first time the bounce buffer is needed, as can be seen in a nsys profile for a simple FileHandle::pread(). The bounce buffer is reused and not deallocated. So subsequence H2D copy does not have the allocation overhead.

I remember KVIKIO used to have NVTX color labels, but I don’t see them recently anymore.

Could you share a screenshot? I cannot reproduce this issue on my side. On the main thread, FileHandle::pread() is an asynchronous call and returns soon after invoked, so its range is typically very short. On the worker threads, each individual I/O task is colored.

[JigaoLuo]

Thanks! I’ll reply to the comments shortly. Now I’m able to saturate reads above 25 GB/s, though it requires over 150 threads to achieve that. I’ll include plots and try to have a clean comparison.

(Some might be delayed due to my paper deadline, but I’ll definitely return to this and follow through on everything I’ve promised.)

[kingcrimsontianyu]

There are a few directions I could think of to help with the investigation of the core issue.

• Launch 4 processes of your program, each process reading from a separate SSD, and see what number of threads are needed for the max bandwidth. In this setting, each process has its own thread pool in KvikIO, which should help to relieve thread pool contention issue if any.
• Check if this thread issue can still be observed under KVIKIO_COMPAT_MODE=ON which uses POSIX instead of cuFile API. In this setting we need to manually drop the page cache before each of the 10 repetitions in your test. The C++ utility for cache dropping is bool success = kvikio::clear_page_cache() in KvikIO.
• Use the batch API in KvikIO or vanilla cuFile API. I could not provide further guidance on this, since I don't have the performance numbers to reference at the moment. This can potentially be an effective way to reduce userland thread count, and I'll perform studies on this some time in the next couple of months.

[JigaoLuo]

Thanks. That makes sense. I will keep in mind.

(Let me tag my previous question as well: #835

[kingcrimsontianyu]

I have reproduced this scalability problem using a KvikIO microbenchmark with the following simplified setting:

• Sequentially read two 4 GiB binary files located on two separate NVMes
• Read each file into a host buffer
• Use the compatibility mode

I will continue the investigation. In the mean time, a new feature of opportunistic Direct I/O (#863) will be available in 25.12, which will improve POSIX read/write performance for unbuffered I/O (i.e. you will have to temporarily switch away from GDS). For file read, the way to activate this feature is to set the env var export KVIKIO_AUTO_DIRECT_IO_READ=1. Hope this helps to some extent.

[JigaoLuo]

Hi @kingcrimsontianyu , thanks for the follow-up. I’m still in deadline mode for December and can’t really move other things forward right now. One request from my side is a CLI tool similar to GDSIO/FIO, but for KVIKio.

I would also like to continue improving the GDS interaction side. I’m aware that with GDS there is a higher cost compared to the compatibility mode when trying to reach the bandwidth limits of SSDs: larger block sizes and significantly more CPU threads are required (as in this issue). However, I’m looking at this from another perspective — the number of CPU threads observed in this issue seems potentially excessive. I believe we could reduce them to a minimal set.

---

Why am I so stubborn about GDS? First, even with this issue, the maximum bandwidth of 4 SSDs is still achievable with KVIKio-GDS. Second, my goal is to keep the system as minimally CPU-dependent as possible.
(The second point does not fully hold due to this issue, but it provides a good starting point for discussing how we might still move toward reducing CPU threads.)