Reproducibility Gap on Ryzen 7 8845HS

[HemaSwetha25]

I am unable to reproduce the reported AMD results. Below are my evaluation results for GPT-4.1 (Attempt = 10) and DeepSeek V3 (deepseek-chat) for different RAG settings.
CPU: AMD Ryzen 7 8845HS
Attempts: N = 10
Models Tested: GPT-4.1, DeepSeek V3 (deepseek-chat)

GPT-4.1 Results (Attempt = 10)
Configuration | Passed | Failed | Pass Rate (%) | Avg Vector Score | Avg Total Cycles
N = 10 (No RAG) | 75 | 27 | 73.53 | 1.517 | 38076.36
N = 10, RAG = 1 | 64 | 38 | 62.75 | 2.117 | 35839.72
N = 10, RAG = 2 | 69 | 33 | 67.65 | 3.228 | 27555.90
N = 10, RAG = 3 | 69 | 33 | 67.65 | 2.661 | 28977.23
N = 10, RAG = 5 | 69 | 33 | 67.65 | 2.786 | 34524.62

To fit within token limits, I modified the RAG setup by removing some kernel comments to reduce prompt size.

DeepSeek V3 Results (deepseek-chat)
Configuration | Passed | Failed | Pass Rate (%) | Avg Vector Score | Avg Total Cycles
N = 10 (No RAG) | 57 | 45 | 55.88 | 4.237 | 28504.61
N = 10, RAG = 3 | 53 | 49 | 51.96 | 2.978 | 18697.75

Clariification:

1. Are there any other parameters that I can tune to improve reproducibility and performance?
2. Does reducing the RAG prompt significantly affect performance?

[skalade]

Hi @HemaSwetha25, apologies for late reply

1. I don't believe so - original results were obtained on a phoenix machine (7940HS). My recommendation is to run and measure canonical solutions, then compare your results against that which will give you more of an apples to apples comparison for your specific setup. I'd refer to the blog results since those are more up-to-date than the paper and speedup is more meaningful than vector score.
2. I can't say for sure, but it will definitely influence results for better or worse.

[HemaSwetha25]

Hi @skalade, thank you for your reply.

I need to clarify a few more points. I have calculated the speedup with and without RAG for attempt 10 runs. How can I ensure that these results are within the expected range defined by the evaluation?

Model | AMD | My result

DeepSeek V3 | - | 28.34x
GPT‑4.1 | 11.1x | 9.7x
GPT‑4o‑mini | 1.3x | 0.9x

Model | AMD | My result

DeepSeek V3 | - | 49.68x
GPT‑4.1 | 33.4x | 33.3x
GPT‑4o‑mini | 31.6x | 24.5x

While there is no speedup value reported for the DeepSeek V3 model in the blog, I think it would likely be close to the value I observed.

I am seeing one specific case where, for the DeepSeek V3 model with RAG (attempt-10), the vector score is lower compared to the non-RAG, but the speedup is higher for the RAG case. In this situation, how should performance be interpreted (By speedup or vector score as my vector score is showing more difference) ?

From my understanding, RAG is not showing better performance in my experiments. I am following the same pipeline as mentioned in NPUEval.

[skalade]

They ranges can vary drastically - if the kernels are functionally correct then the speedup values are valid.

Vector score (VPU usage) is a useful metric to track, but just because the VPU is being used does not mean it is being used effectively that's why you might see vector score high without much real perf improvement.

Ultimately the source of truth is cycle count - so I'd prioritized speedup over vector score.

[HemaSwetha25]

Thank You @skalade for clarifying the things.

[HemaSwetha25]

When I calculated the speedup for DeepSeek R1 model, larger deviation was seen between my results and AMD’s speedup values. In contrast, for the GPT-based models, the speedup values are very close to AMD’s values, with only minor differences. For DeepSeek R1, both the vector score and speedup has large variation.

Model | AMD Speedup | My Speedup | Difference
DeepSeek R1 (no rag) | 41.9x | 38.4x | -3.5x
DeepSeek R1 (rag = 2) | 58.2x | 50.3x | -7.9x

In considering reproducibilty, is this level of deviation in models is okay or need any checks to proceed with next level ?

[skalade]

The original R1 results at the time would've been with the 0528 release. If you're using the official DeepSeek API then it's defaulting to the DeepSeek-V3.2 checkpoint these days - provider choice could be source of a diff.

A single kernel can have a wild impact on these results since it's possible to get a ~300x speedup over scalar baseline after applying vectorization. Your diff of -7.9x could just be a matter of 2-4 kernels failing. I would double check that none of your API calls timed out resulting in a failed kernel.

Other than that reproducibility is tricky - different device, how LLM is deployed, RAG changes will have an effect. In my opinion 3.5-7.9x aren't a big deviation because like I mentioned this could be a matter of a couple kernels.