Compare revmc perf

[dajuguan]

Benchmark for go-llvm

git co po_llvm_host
taskset -c 1 go test  -run=^$ -bench=BenchmarkContracts   -v -tags=llvm18

MSTORE/MLOAD with host function

BenchmarkContracts/weth/NoGas             283316              4111 ns/op
BenchmarkContracts/weth/Gas               270841              4235 ns/op
BenchmarkContracts/weth/Section           273950              4124 ns/op
BenchmarkContracts/weth/Interp            681439              1836 ns/op

MSTORE/MLOAD with LLVM

BenchmarkContracts/weth/NoGas             340348              3502 ns/op
BenchmarkContracts/weth/Gas               334497              3520 ns/op
BenchmarkContracts/weth/Section           334764              3466 ns/op
BenchmarkContracts/weth/Interp            693616              1806 ns/op

After caching GetFunctionAddress

BenchmarkContracts/weth/NoGas             436989              2633 ns/op
BenchmarkContracts/weth/Gas               440812              2706 ns/op
BenchmarkContracts/weth/Section           439948              2678 ns/op
BenchmarkContracts/weth/Interp            737246              1853 ns/op

After free memory (very counterintuitive! might be because GC takes up lots of time):

BenchmarkContracts/weth/NoGas             484111              2508 ns/op
BenchmarkContracts/weth/Gas               462482              2497 ns/op
BenchmarkContracts/weth/Section           497572              2529 ns/op
BenchmarkContracts/weth/Interp            687445              1748 ns/op

[dajuguan]

revmc benchmark with Rust bench

Benchmarking fibonacci/native
Benchmarking fibonacci/native: Warming up for 2.0000 s
Benchmarking fibonacci/native: Collecting 20 samples in estimated 5.0000 s (52M iterations)
Benchmarking fibonacci/native: Analyzing
fibonacci/native        time:   [96.018 ns 96.415 ns 96.853 ns]
                        change: [+0.6302% +0.9172% +1.2301%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 20 measurements (15.00%)
  1 (5.00%) high mild
  2 (10.00%) high severe
Benchmarking fibonacci/revmc/default
Benchmarking fibonacci/revmc/default: Warming up for 2.0000 s
Benchmarking fibonacci/revmc/default: Collecting 20 samples in estimated 5.0000 s (16M iterations)
Benchmarking fibonacci/revmc/default: Analyzing
fibonacci/revmc/default time:   [307.09 ns 307.94 ns 309.04 ns]
                        change: [+12.399% +12.698% +13.016%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 20 measurements (10.00%)
  2 (10.00%) high severe
Benchmarking fibonacci/revmc/no_gas
Benchmarking fibonacci/revmc/no_gas: Warming up for 2.0000 s
Benchmarking fibonacci/revmc/no_gas: Collecting 20 samples in estimated 5.0000 s (50M iterations)
Benchmarking fibonacci/revmc/no_gas: Analyzing
fibonacci/revmc/no_gas  time:   [100.43 ns 100.81 ns 101.20 ns]
                        change: [+3.0267% +3.5958% +4.2417%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high mild
Benchmarking fibonacci/revm-interpreter
Benchmarking fibonacci/revm-interpreter: Warming up for 2.0000 s
Benchmarking fibonacci/revm-interpreter: Collecting 20 samples in estimated 5.0006 s (1.5M iterations)
Benchmarking fibonacci/revm-interpreter: Analyzing
fibonacci/revm-interpreter
                        time:   [3.2042 µs 3.2178 µs 3.2350 µs]
                        change: [-7.8045% -7.1027% -6.1246%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high severe

Benchmarking fibonacci-calldata/revmc/default
Benchmarking fibonacci-calldata/revmc/default: Warming up for 2.0000 s
Benchmarking fibonacci-calldata/revmc/default: Collecting 20 samples in estimated 5.0002 s (1.6M iterations)
Benchmarking fibonacci-calldata/revmc/default: Analyzing
fibonacci-calldata/revmc/default
                        time:   [3.2106 µs 3.2162 µs 3.2224 µs]
                        change: [-15.911% -15.681% -15.448%] (p = 0.00 < 0.05)
                        Performance has improved.
Benchmarking fibonacci-calldata/revmc/no_gas
Benchmarking fibonacci-calldata/revmc/no_gas: Warming up for 2.0000 s
Benchmarking fibonacci-calldata/revmc/no_gas: Collecting 20 samples in estimated 5.0001 s (12M iterations)
Benchmarking fibonacci-calldata/revmc/no_gas: Analyzing
fibonacci-calldata/revmc/no_gas
                        time:   [416.84 ns 418.49 ns 420.30 ns]
                        change: [+0.5815% +0.9383% +1.3585%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high mild
Benchmarking fibonacci-calldata/revm-interpreter
Benchmarking fibonacci-calldata/revm-interpreter: Warming up for 2.0000 s
Benchmarking fibonacci-calldata/revm-interpreter: Collecting 20 samples in estimated 5.0041 s (115k iterations)
Benchmarking fibonacci-calldata/revm-interpreter: Analyzing
fibonacci-calldata/revm-interpreter
                        time:   [43.141 µs 43.452 µs 43.816 µs]
                        change: [-9.2501% -8.3248% -7.1813%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 20 measurements (10.00%)
  1 (5.00%) high mild
  1 (5.00%) high severe

Benchmarking factorial/revmc/default
Benchmarking factorial/revmc/default: Warming up for 2.0000 s
Benchmarking factorial/revmc/default: Collecting 20 samples in estimated 5.0003 s (1.4M iterations)
Benchmarking factorial/revmc/default: Analyzing
factorial/revmc/default time:   [3.6216 µs 3.6292 µs 3.6397 µs]
                        change: [-4.2431% -3.9336% -3.5739%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 20 measurements (20.00%)
  2 (10.00%) high mild
  2 (10.00%) high severe
Benchmarking factorial/revmc/no_gas
Benchmarking factorial/revmc/no_gas: Warming up for 2.0000 s
Benchmarking factorial/revmc/no_gas: Collecting 20 samples in estimated 5.0002 s (1.4M iterations)
Benchmarking factorial/revmc/no_gas: Analyzing
factorial/revmc/no_gas  time:   [3.5789 µs 3.5862 µs 3.5958 µs]
                        change: [+0.9023% +1.2011% +1.5030%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Benchmarking factorial/revm-interpreter
Benchmarking factorial/revm-interpreter: Warming up for 2.0000 s
Benchmarking factorial/revm-interpreter: Collecting 20 samples in estimated 5.0080 s (102k iterations)
Benchmarking factorial/revm-interpreter: Analyzing
factorial/revm-interpreter
                        time:   [50.527 µs 50.747 µs 51.038 µs]
                        change: [+0.5853% +1.1847% +1.8751%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 20 measurements (10.00%)
  2 (10.00%) high mild

Benchmarking counter/revmc/default
Benchmarking counter/revmc/default: Warming up for 2.0000 s
Benchmarking counter/revmc/default: Collecting 20 samples in estimated 5.0000 s (29M iterations)
Benchmarking counter/revmc/default: Analyzing
counter/revmc/default   time:   [173.56 ns 173.92 ns 174.23 ns]
                        change: [+0.5949% +1.5522% +2.5186%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) low mild
Benchmarking counter/revmc/no_gas
Benchmarking counter/revmc/no_gas: Warming up for 2.0000 s
Benchmarking counter/revmc/no_gas: Collecting 20 samples in estimated 5.0000 s (29M iterations)
Benchmarking counter/revmc/no_gas: Analyzing
counter/revmc/no_gas    time:   [173.80 ns 174.12 ns 174.42 ns]
                        change: [-1.1726% +0.4535% +1.8000%] (p = 0.60 > 0.05)
                        No change in performance detected.
Benchmarking counter/revm-interpreter
Benchmarking counter/revm-interpreter: Warming up for 2.0000 s
Benchmarking counter/revm-interpreter: Collecting 20 samples in estimated 5.0000 s (12M iterations)
Benchmarking counter/revm-interpreter: Analyzing
counter/revm-interpreter
                        time:   [416.59 ns 417.70 ns 419.23 ns]
                        change: [+4.1253% +4.7257% +5.3006%] (p = 0.00 < 0.05)
                        Performance has regressed.

Benchmarking counter-eof/revmc/default
Benchmarking counter-eof/revmc/default: Warming up for 2.0000 s
Benchmarking counter-eof/revmc/default: Collecting 20 samples in estimated 5.0000 s (27M iterations)
Benchmarking counter-eof/revmc/default: Analyzing
counter-eof/revmc/default
                        time:   [182.68 ns 182.93 ns 183.27 ns]
                        change: [+0.9222% +1.2379% +1.5386%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high mild
Benchmarking counter-eof/revmc/no_gas
Benchmarking counter-eof/revmc/no_gas: Warming up for 2.0000 s
Benchmarking counter-eof/revmc/no_gas: Collecting 20 samples in estimated 5.0000 s (28M iterations)
Benchmarking counter-eof/revmc/no_gas: Analyzing
counter-eof/revmc/no_gas
                        time:   [175.83 ns 176.39 ns 176.90 ns]
                        change: [+2.0619% +2.3272% +2.6088%] (p = 0.00 < 0.05)
                        Performance has regressed.
Benchmarking counter-eof/revm-interpreter
Benchmarking counter-eof/revm-interpreter: Warming up for 2.0000 s
Benchmarking counter-eof/revm-interpreter: Collecting 20 samples in estimated 5.0000 s (12M iterations)
Benchmarking counter-eof/revm-interpreter: Analyzing
counter-eof/revm-interpreter
                        time:   [414.09 ns 414.81 ns 415.80 ns]
                        change: [+0.2116% +0.4644% +0.7399%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high mild

Benchmarking snailtracer/revmc/default
Benchmarking snailtracer/revmc/default: Warming up for 2.0000 s
Benchmarking snailtracer/revmc/default: Collecting 20 samples in estimated 6.0800 s (100 iterations)
Benchmarking snailtracer/revmc/default: Analyzing
snailtracer/revmc/default
                        time:   [60.598 ms 60.714 ms 60.852 ms]
                        change: [-0.3538% -0.0461% +0.2595%] (p = 0.76 > 0.05)
                        No change in performance detected.
Found 2 outliers among 20 measurements (10.00%)
  2 (10.00%) high mild
Benchmarking snailtracer/revmc/no_gas
Benchmarking snailtracer/revmc/no_gas: Warming up for 2.0000 s
Benchmarking snailtracer/revmc/no_gas: Collecting 20 samples in estimated 6.0071 s (100 iterations)
Benchmarking snailtracer/revmc/no_gas: Analyzing
snailtracer/revmc/no_gas
                        time:   [58.446 ms 59.070 ms 59.662 ms]
                        change: [-1.9082% -0.9166% +0.0005%] (p = 0.10 > 0.05)
                        No change in performance detected.
Benchmarking snailtracer/revm-interpreter
Benchmarking snailtracer/revm-interpreter: Warming up for 2.0000 s
Benchmarking snailtracer/revm-interpreter: Collecting 20 samples in estimated 5.0715 s (40 iterations)
Benchmarking snailtracer/revm-interpreter: Analyzing
snailtracer/revm-interpreter
                        time:   [126.72 ms 126.99 ms 127.26 ms]
                        change: [-13.426% -11.295% -8.8548%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking snailtracer-eof/revmc/default
Benchmarking snailtracer-eof/revmc/default: Warming up for 2.0000 s
Benchmarking snailtracer-eof/revmc/default: Collecting 20 samples in estimated 5.4998 s (100 iterations)
Benchmarking snailtracer-eof/revmc/default: Analyzing
snailtracer-eof/revmc/default
                        time:   [54.738 ms 54.864 ms 54.991 ms]
                        change: [-3.3594% -3.0118% -2.6909%] (p = 0.00 < 0.05)
                        Performance has improved.
Benchmarking snailtracer-eof/revmc/no_gas
Benchmarking snailtracer-eof/revmc/no_gas: Warming up for 2.0000 s
Benchmarking snailtracer-eof/revmc/no_gas: Collecting 20 samples in estimated 5.3633 s (100 iterations)
Benchmarking snailtracer-eof/revmc/no_gas: Analyzing
snailtracer-eof/revmc/no_gas
                        time:   [53.511 ms 53.642 ms 53.780 ms]
                        change: [-4.5019% -4.1463% -3.8156%] (p = 0.00 < 0.05)
                        Performance has improved.
Benchmarking snailtracer-eof/revm-interpreter
Benchmarking snailtracer-eof/revm-interpreter: Warming up for 2.0000 s
Benchmarking snailtracer-eof/revm-interpreter: Collecting 20 samples in estimated 5.0897 s (40 iterations)
Benchmarking snailtracer-eof/revm-interpreter: Analyzing
snailtracer-eof/revm-interpreter
                        time:   [127.33 ms 127.62 ms 127.92 ms]
                        change: [-11.847% -11.562% -11.251%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking weth/revmc/default
Benchmarking weth/revmc/default: Warming up for 2.0000 s
Benchmarking weth/revmc/default: Collecting 20 samples in estimated 5.0000 s (11M iterations)
Benchmarking weth/revmc/default: Analyzing
weth/revmc/default      time:   [467.52 ns 468.61 ns 469.76 ns]
                        change: [-6.0077% -5.7724% -5.5099%] (p = 0.00 < 0.05)
                        Performance has improved.
Benchmarking weth/revmc/no_gas
Benchmarking weth/revmc/no_gas: Warming up for 2.0000 s
Benchmarking weth/revmc/no_gas: Collecting 20 samples in estimated 5.0000 s (11M iterations)
Benchmarking weth/revmc/no_gas: Analyzing
weth/revmc/no_gas       time:   [467.10 ns 468.26 ns 469.57 ns]
                        change: [-5.9308% -5.6908% -5.4251%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high mild
Benchmarking weth/revm-interpreter
Benchmarking weth/revm-interpreter: Warming up for 2.0000 s
Benchmarking weth/revm-interpreter: Collecting 20 samples in estimated 5.0001 s (6.7M iterations)
Benchmarking weth/revm-interpreter: Analyzing
weth/revm-interpreter   time:   [740.16 ns 742.38 ns 745.46 ns]
                        change: [-7.8690% -7.6143% -7.3343%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 20 measurements (15.00%)
  3 (15.00%) high mild

Benchmarking hash_10k/revmc/default
Benchmarking hash_10k/revmc/default: Warming up for 2.0000 s
Benchmarking hash_10k/revmc/default: Collecting 20 samples in estimated 5.3162 s (2100 iterations)
Benchmarking hash_10k/revmc/default: Analyzing
hash_10k/revmc/default  time:   [2.5273 ms 2.5368 ms 2.5511 ms]
                        change: [-5.3774% -5.0974% -4.7531%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 20 measurements (20.00%)
  4 (20.00%) high severe
Benchmarking hash_10k/revmc/no_gas
Benchmarking hash_10k/revmc/no_gas: Warming up for 2.0000 s
Benchmarking hash_10k/revmc/no_gas: Collecting 20 samples in estimated 5.2956 s (2100 iterations)
Benchmarking hash_10k/revmc/no_gas: Analyzing
hash_10k/revmc/no_gas   time:   [2.5181 ms 2.5230 ms 2.5285 ms]
                        change: [-5.0502% -4.7064% -4.3038%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 20 measurements (10.00%)
  2 (10.00%) high mild
Benchmarking hash_10k/revm-interpreter
Benchmarking hash_10k/revm-interpreter: Warming up for 2.0000 s
Benchmarking hash_10k/revm-interpreter: Collecting 20 samples in estimated 5.4730 s (1470 iterations)
Benchmarking hash_10k/revm-interpreter: Analyzing
hash_10k/revm-interpreter
                        time:   [3.7156 ms 3.7242 ms 3.7330 ms]
                        change: [+2.0728% +2.3517% +2.6627%] (p = 0.00 < 0.05)
                        Performance has regressed.

Benchmarking hash_10k-eof/revmc/default
Benchmarking hash_10k-eof/revmc/default: Warming up for 2.0000 s
Benchmarking hash_10k-eof/revmc/default: Collecting 20 samples in estimated 5.4144 s (2100 iterations)
Benchmarking hash_10k-eof/revmc/default: Analyzing
hash_10k-eof/revmc/default
                        time:   [2.6473 ms 2.6767 ms 2.7075 ms]
                        change: [+1.2723% +2.2333% +3.3511%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 20 measurements (10.00%)
  1 (5.00%) high mild
  1 (5.00%) high severe
Benchmarking hash_10k-eof/revmc/no_gas
Benchmarking hash_10k-eof/revmc/no_gas: Warming up for 2.0000 s
Benchmarking hash_10k-eof/revmc/no_gas: Collecting 20 samples in estimated 5.3619 s (2100 iterations)
Benchmarking hash_10k-eof/revmc/no_gas: Analyzing
hash_10k-eof/revmc/no_gas
                        time:   [2.6181 ms 2.6386 ms 2.6701 ms]
                        change: [-0.8159% -0.0685% +0.9691%] (p = 0.90 > 0.05)
                        No change in performance detected.
Found 3 outliers among 20 measurements (15.00%)
  1 (5.00%) high mild
  2 (10.00%) high severe
Benchmarking hash_10k-eof/revm-interpreter
Benchmarking hash_10k-eof/revm-interpreter: Warming up for 2.0000 s
Benchmarking hash_10k-eof/revm-interpreter: Collecting 20 samples in estimated 5.2200 s (1470 iterations)
Benchmarking hash_10k-eof/revm-interpreter: Analyzing
hash_10k-eof/revm-interpreter
                        time:   [3.5487 ms 3.5536 ms 3.5592 ms]
                        change: [-3.4092% -2.7235% -2.1464%] (p = 0.00 < 0.05)
                        Performance has improved.

[qizhou]

I guess we also need to evaluate the perf with tevm