Performance statistics report

[jeromekelleher]

It would be very useful to have some simple microbenchmarks we could run from time-to-time. We've tried using ASV, but it's very heavyweight and collects masses of data that's never used.

Requirements:

• Make a script perf_benchmark.py which runs a standard set of microbenchmarks and outputs the result to a file. The idea is this file should be updated before every release, so that we can spot any perf regressions and also so that we have a per-release history of the benchmarks in the history.
• Update the developer docs to include how and when to run these benchmarks as part of the release process.

CPU time performance benchmarks:

Given a standard file

• time to load
• time to save
• time to access ts.tables in a loop
• time to access tables.nodes, tables.individuals etc
• time to access columns, nodes.flags, etc
• time to get first tree, ts.first()
• time to seek to middle tree
• time to iterate over all trees
• time to access tree arrays, tree.parent etc.
• time to decode all variants
• time to iterate over all rows, with and without metadata
• time to write to vcf (writing to devnull)

additional as I thought of them (BJ):

• tree node arrays (postorder et al)
• tree row accessors (ts.node(42))
• tree accessors (tree.left_sib(42))
• iterate over tree.nodes

It would be nice to track the memory usage here, but this may be much more difficult and not worth the effort.

[benjeffery]

Done some research here, measuring the variance of different methods. The main aim is to avoid this phenomenon in micro benchmarks, where running code over a period of 20s reveals two performance regimes:

Using timeit and having a balance between "number of trials in the tight loop" and "number of repetitions of the loop" seems to avoid this phenomena and gives lowest variance. I've converged on:

def autotime(setup: str, code: str)
    t = timeit.Timer(setup=setup, stmt=code)
    # We want each test to last around 1-2 seconds, and to have a good number of trials and
    # repetitions as this gives the lowest variance in testing.
    one_run = t.timeit(number=1)
    num_trials = int(max(1, 0.01 / one_run))
    num_repeats = math.ceil(2 / (one_run * num_trials))
    t1 = time.time()
    results = t.repeat(repeat=num_repeats, number=num_trials)
    return np.mean(results) / num_trials

For configuration this is the proposed format:

setup: |
  import sys
  import tskit
  import msprime
  import time
  ts = msprime.simulate(10, random_seed=1)

benchmarks:
  - code: ts.tables
  - code: tables.{table}
    setup: tables = ts.tables
    parameters:
      table:
        - nodes
        - edges
        - sites
        - mutations
        - populations
        - individuals
        - migrations
        - provenances
  - code: "{table}.{column}"
    setup: |
      tables = ts.tables
      {table} = tables.{table}
    parameters:
      table:
        nodes:
          column:
            - flags
        edges:
          column:
            - left
            - right
            - parent
            - child
....

This allows parameterisation in both the setup and code.

WIP at main...benjeffery:bench

[jeromekelleher]

Very nice, I like it!

[benjeffery]

Tests with side effects are tricky as timeit does setup once, then repeats them. So for example seek only does meaningful work on the first iteration of the loop. For seek I've fixed this by seeking backwards and forwards in the loop. If I come across one where such a trick is not possible we'll just have to only have one trial per loop.

[benjeffery]

Closed by #2454