gh-21: Attempt to make regression tests more reliable

[connoraird]

Use mean instead on min as it should be more reliable and make less strict

[paddyroddy]

Good idea. Assume 15% was required for this to pass? Seems a fairly high level of degradation

[connoraird]

Good idea. Assume 15% was required for this to pass? Seems a fairly high level of degradation

TBH the 15% is quite arbitrary. It was very inconsistent running on my local machine. However, the inconsistencies were usually less than 15% but not always.

[paddyroddy]

Can the tolerance in the tests be changed?

[connoraird]

Can the tolerance in the tests be changed?

As in on a per test basis? Or do you mean not hard code in nox and pass it in posargs? Posargs would be easy.

[paddyroddy]

I had meant in the tests. But whatever you think is best. I just think 15% degradation seems too high.

[connoraird]

I had meant in the tests. But whatever you think is best. I just think 15% degradation seems too high.

I can't see anyway to have a per test comparison. Unless we filtered the tests within nox via pytest -k ...

[connoraird]

Looking at the runs for this PR, most of the stats seem reasonable but you do get some strange ones. For example, this line has a standard deviation much larger than the mean

Name (time in us)                                                                                  Min                    Max                  Mean              StdDev                Median                 IQR            Outliers          OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_broadcast_first[numpy]                                                                    29.3750 (1.72)     43,014.5230 (661.64)      35.0362 (1.92)     382.9808 (134.39)      30.8480 (1.75)       0.8800 (2.31)       1;1213  28,541.9155 (0.52)      12597           1

[connoraird]

This snippet from pytest-benchmark docs seems relevant

[paddyroddy]

Does asv have this issue? Or pytest-codspeed?

[connoraird]

Does asv have this issue? Or pytest-codspeed?

In both pytest-benchmark and pytest-codspeed, there is a "pedantic" mode in which you can specify how the stats should be collected. Perhaps that will be useful?

Also we could try using a different timer function as suggested here

[connoraird]

Does asv have this issue? Or pytest-codspeed?

I can't see why there would be a difference in relation to this unreliability. In my opinion, the issue is related to the fact we cannot guarantee the state of the machine/VM we are running on which will be the case no matter what tool we use.

[paddyroddy]

Also we could try using a different timer function as suggested here

Let's give this a go

[connoraird]

Closing as moved to glass repo glass-dev/glass#780