Different results running benchmarks with and without the action

[andresliszt]

Hello! Thanks for this amazing project. Maybe I'm too new on this, but I'm trying to configure the codspeed action, I'm getting different results in some scenarios. I'm working on a project which mixes python and rust through pyo3, and I have a test here, Which behaves completely different when I run it with and without the action.

import time

import pytest
import numpy as np
from scipy.spatial.distance import cdist  # type: ignore

from pymoors._pymoors import cross_euclidean_distances  # type: ignore


def test_cdist_pymoors(benchmark):
    matrix = np.random.rand(5000, 5000)
    result = benchmark(cross_euclidean_distances, matrix, matrix)
    assert result.shape == (5000, 5000)


def eculidean_distance_plus_time(matrix):
    start_time = time.perf_counter()
    result = cross_euclidean_distances(matrix, matrix)
    end_time = time.perf_counter()
    total = end_time - start_time
    return result, total


@pytest.mark.parametrize("n", [1000, 1500, 2000, 2500])
def test_compare_scipy_cdist_vs_pymoors(benchmark, n):
    matrix = np.random.rand(n, n)
    result_pymoors, total_time = benchmark(eculidean_distance_plus_time, matrix)
    # Measure the time for scipy's cdist using a timer
    start_time = time.perf_counter()
    result_scipy = cdist(matrix, matrix, metric="sqeuclidean")
    end_time = time.perf_counter()
    time_scipy = end_time - start_time
    # Check the result is the same
    np.testing.assert_allclose(result_pymoors, result_scipy, rtol=1e-5, atol=1e-8)
    # Check the time
    assert total_time < time_scipy

If I use the action the tests run like if I've deployed my package in debug mode, comparable with the time i got locally when i run them in debug mode

- name: Run benchmarks
  uses: CodSpeedHQ/action@v3
  with:
    token: ${{ secrets.CODSPEED_TOKEN }}
    run: |
      pytest -s -vvv tests/benchmarks --codspeed

But when I remove the action, and run the benchmark directly, the tests run just fine, like in release mode

- name: Run benchmarks
  run: pytest -s -vvv tests/benchmarks --codspeed

Is the action introducing a change in the runner machine? Is there a doc where I can see what is the action doing?

Thanks!

[andresliszt]

I fixed it using

- name: Run benchmarks
  uses: CodSpeedHQ/action@v3
  with:
    run: uv run pytest -s -vvv tests/benchmarks --codspeed
  env:
    CODSPEED_RUNNER_MODE: "walltime"

I can't find info related to the env var CODSPEED_RUNNER_MODE, I found it looking at the runner code

[adriencaccia]

Hey @andresliszt, thanks for taking the time to create an issue.

By default, the CodSpeedHQ/action will run your code using the CPU instrument. As explained in the docs, it will run your benchmark once on a simulated CPU and count its cycles, instead of measuring real time. This allow to have consistent measurements not subject to the noisyness of CI runners.

The CODPSEED_RUNNER_MODE=walltime env variable enables the Walltime instrument, which will measure the real wall time of your code. But this mode is subject to noisyness, this is why we recommend using our Macro Runners–bare-metal runners optimised for performanced provided and managed by CodSpeed– instead of the ones provided by GitHub.
We did not document this env variable since using this mode does not make sense on regular runners, and is enabled by default on our Macro Runners.

I hope I clarified things a bit, let me know if something is not clear!

[andresliszt]

Thank for answering this!. I have updated my test, the one showed in the issue is the one on main, working fine with walltime. Note carefully that in the test i'm comparing my execution time given by

result_pymoors, total_time = benchmark(eculidean_distance_plus_time, matrix)

against scipy execution time, total_time comes from the function eculidean_distance_plus_time, which as I understand is completely independent from the codspeed stuff, so that time should be the same for both runner modes. My test with the default runner mode fails in the assertion assert total_time < time_scipy, where in that scenario my execution time is close to the debug execution time. With the walltime mode, my execution time is similar to the release one. Is that expected?

[adriencaccia]

Code that is being called passed in the benchmark fixture will be instrumented on the simulated CPU, so its execution will be slower than what it would be on a normal CPU.

In your case, with the current benchmark code, you are comparing the time it took to run your implementation under the CodSpeed instrumentation to the time it took to run scipy's implementation without the CodSpeed instrumentation.

So for you assertion to be relevant, you would have to change your benchmarked code and wrap both the execution and time tracking of both your code and scipy's in the same function, and use the benchmark fixture on that function.
That way both implementations will be instrumented on the simulated CPU, and you will have relevant things to compare.