Per round tx blooms

[agodnic]

Summary

This pull request introduces bloom filters to improve the performance of ledger.LookupTxid, in particular for the case of looking up a TXID that does not exist in the given round.

Performance

This pull request includes several benchmarks. Those benchmarks aim towards proving that:

1. The speed of ledger.LookupTxid is improved drastically when the TXID doesn't exist.
2. When calling ledger.LookupTxid for a TXID that does exist, the performance penalty introduced by bloom filters relatively low.
3. When calling ledger.AddBlock, the performance penalty introduced by bloom filters is relatively low.

Case 1: Measure miss performance

This benchmark measures the performance of calling ledger.LookupTxid with a non-existent Txid.
Measurements are made with and without bloom filters.

cd data
go test -bench "BenchmarkReadMiss.*BloomFilter" -run $^

Results

The benchmark showed a ~100% speedup when using bloom filters.

go test -bench "Benchmark.*BloomFilter" -run $^
goos: linux
goarch: amd64
pkg: github.com/algorand/go-algorand/data
cpu: AMD Ryzen 9 5900X 12-Core Processor            
BenchmarkReadMissWithBloomFilter-24       	   60523	     17652 ns/op
BenchmarkReadMissWithoutBloomFilter-24    	   31374	     35086 ns/op
PASS
ok  	github.com/algorand/go-algorand/data	5.722s

Case 2: Measure hit performance

This benchmark measures the performance of calling ledger.LookupTxid with a valid Txid.
Measurements are made with and without bloom filters.

cd data
go test -bench "go test -bench "BenchmarkInsertion.*BloomFilter" -run $^.*BloomFilter" -run $^

Results

The benchmark showed a ~3% slowdown when using bloom filters.

go test -bench "Benchmark.*BloomFilter" -run $^
goos: linux
goarch: amd64
pkg: github.com/algorand/go-algorand/data
cpu: AMD Ryzen 9 5900X 12-Core Processor            
BenchmarkReadHitWithBloomFilter-24        	   29918	     34620 ns/op
BenchmarkReadHitWithoutBloomFilter-24     	   31610	     33817 ns/op
PASS
ok  	github.com/algorand/go-algorand/data	5.722s

Case 3: Measure bloom filter insertion cost

This benchmark measures the performance of calling ledger.AddBlock with and without bloom filters.

cd ledger
go test -bench "BenchmarkInsertion.*BloomFilter" -run $^

Results

The benchmark showed a ~4% slowdown when using bloom filters

go test -bench "BenchmarkInsertion.*BloomFilter" -run $^
goos: linux
goarch: amd64
pkg: github.com/algorand/go-algorand/ledger
cpu: AMD Ryzen 9 5900X 12-Core Processor            
BenchmarkInsertionWithBloomFilter-24       	    1098	   1064862 ns/op
BenchmarkInsertionWithoutBloomFilter-24    	    1052	   1026313 ns/op
PASS
ok  	github.com/algorand/go-algorand/ledger	2.802s