Performance regression test

[guibou]

krank performances highly depends on the performance of the regular expression we use to find issues.

We need to setup a performance regression test, but I don't know how.

[gbataille]

Hum so I see 2 ways:

In any case, get a lib that can perform X times the same test to get a reasonable average/percentile. Then

1. Define an absolute threshold. This is difficult to do and this is made even more difficult with a metric that is going to have quite a bit of volatility
2. (something I did at work a few years ago) each run on master saves the current "acceptable" performance in some permanent storage (AWS DDB, commit back to the repo, I don't know if circle CI has some storage). Any CI run on a feature branch compares its performance to this baseline. And we define 2 threshold: a relative one that says how little the baseline is allowed to get worse, and an absolute one because getting worse bit by bit could mean eventually becoming unacceptable