feat: add smoke-benchmarks (smoking bench?)

[melodywang060]

Add a benchmarking subcommand that can run a quick performance test to verify that RAPIDS operations are running at expected performance levels on the given hardware

[jacobtomlinson]

I like this idea!

[ncclementi]

@jayavenkatesh19 and I chatted a bit about this in our sync, and it wasn't very clear what's the goal we are going for here. I'm dropping some questions below that popped up when we were discussing this issue.

We should probably scope this better. Some thing to answer are:

• How do we determine expected performance?
  • by hardware version
  • by cudf/cuml/cupy/etc version
  • are we gettign some statistics?
• What is an acceptable result range for what is expected?
• If the result is not expected under whatever criteria is defined, what are the follow up actions to recommend?

Jaya, feelfree to add anything I might have missed.

[jayavenkatesh19]

Thanks for the comprehensive summary of our discussion @ncclementi!

Just following in line with how the doctor piece is designed for checks, we can add an extensible benchmark piece that uses entry-points allowing developers can use to add their own benchmarks for various libraries. Results can be presented to the user through rapids-cli.

However, adding to Naty's comment above, what needs scoping in such a framework is also the time it takes to run these benchmarks. Checks by nature are expected to be relatively quick to execute, however benchmarks might not always be so. In this scenario, users might end up waiting too long on the CLI waiting for the benchmarking to finish.

Even with simple benchmarks, we are still looking at stuff like hardware variance, performances changes due to software version etc.

[jacobtomlinson]

If rapids doctor answers the question "did I install everything correctly?" then rapids benchmark answers the question "is this GPU software faster than standard CPU libraries?".

I think what I had imagined here is that for each benchmark libraries would define two functions, a CPU and GPU implementation.

Then our tool would run them both a bunch of times and then show the outputs. I would expect each benchmark duration to be of the order of milliseconds to seconds, compared to many existing benchmarks that can be of the order of seconds to minutes.

Total duration of the suite could be single-figure minutes, provided the output is interesting to watch. For a UI I was thinking something similar to the way hyperfine presents things.

[btepera]

Thanks all for the discussion in this issue and the work done so far!

My primary question is whether we need to show timings (e.g. "this ran faster on the GPU"?) or just that something ran on the GPU vs the CPU?

• Finding standard benchmarks that consistently run x-times faster than CPU can be challenging because it often requires different dataset sizes for different operations or different libraries. When you add in the requirement that it should show a significant improvement on ANY type of GPU it becomes significantly more difficult. The same dataset that might show a good speedup on an 80 GB H100 might cause an out-of-memory error on a 8 or 16 GB GPU. On the flip side, a smaller workload may not see that big of a speedup on a server grade GPU with a high-end CPU that we're comparing against.
• If we did show timings, what would we tell users about the times they see compared to the benchmarks we publish? I can see a world in which a user runs a utility that only shows a 3x speedup on their system compared to a 10x speedup that we publish and gets discouraged from using RAPIDS
• If we show that something just runs on the GPU (possibly with the GPU time, but without making a CPU comparison) that could still show a user that they have set up RAPIDS properly and are taking advantage of the GPU, which I believe is the intent here

[jacobtomlinson]

The goal here is to give the user confidence that they have correctly installed and configured their system to use RAPIDS. Given that the value proposition of RAPIDS is to accelerate your Python Data Science code I'm keen to find a way to demonstrate to the user that they have done so.

I agree that giving the user raw benchmark numbers could lead to misinterpretation and misrepresentation of what their system is capable of. I think pass/fail style feedback could still give the same assurance they things are working as expected.

We already have rapids doctor for running smoke tests and giving pass/fail feedback. So perhaps it would be better to focus on adding checks there that confirm the user has accelerated hardware rather than building out a parallel entrypoint focused on performance.

This leads us on to the question of "how do we write a pass/fail test to check that the user has the ability to accelerate their code?".

[betatim]

Maybe a way to think about this is to think about what a user could configure that would lead to bad performance but not outright breakage.

For example, if you install cupy and can execute X @ Y, what could you have misconfigured that would lead to that not being as fast as it can be?

For a library like cudf I'd also assume that if you can run some vanilla cudf code, then "it works and is as fast as it can be" (less clear for cudf.pandas). Maybe for cudf the user can (mis)configure things related to data transfer between the CPU and GPU? In that case we could try and test/bench that?

In both cases it might be nice to show the user a message like "you are performing X matrix multiplications per second with cupy and Y with numpy" (I assume we'd always get X to be bigger than Y).

---

Back in the day when you had to compile a BLAS library yourself there were implementations that auto-tuned themselves to your hardware. They used to show some "untuned" numbers for common operations and some after spending hours(?) on tuning. As a naive user it was nice to see that the tuned ones were bigger (even though I had no idea what they should be for the hardware that I had).

---

tl;dr: what can users configure that would degrade performance but not outright crash? Answers to that might lead to ideas for "benchmarks"

[betatim]

Maybe call it "performance test" or rapids perf-test? A command to test the performance of the user's setup.

Or even "performance report"? That way you make it clear that it is reporting the performance, not comparing it or telling you what it should/could be. A bit like fast.com tells me how fast my internet is, without telling me if that is good or not, how it compares to what I pay for, etc