splinter_test --perf --num-bg-threads > 0 causes a ref count assert to fail on query

[ajhconway]

Running the --perf splinter_test with a non-zero number of background threads successfully performs the inserts, but then hits a ref count assertion as the query phase of the test starts.

[gapisback]

Attempted to reproduce this issue after some clean-up has been under PR #469 to improve support for bg-threads.
Now we can invoke driver_test splinter_test --perf with options slightly renamed as: --num-memtable-bg-threads 2 --num-normal-bg-threads 2.

Could not reproduce the exact assertion failure mentioned in this issue. Attempted with following variations.
Tried following repro steps off of /main as of this commit:

bcd7933 Jon Howell 9 days ago Sat, 24-Dec-2022, 08:17:19 AM (Authored: Sat, 24-Dec-2022, 08:17:19 AM)
 Jonh move extent funcs to allocator (#504)

$ build/release/bin/driver_test splinter_test --perf --num-insert-threads 4 --num-lookup-threads 8 --tree-size-gib 2

$ build/release/bin/driver_test splinter_test --perf --num-insert-threads 4 --tree-size-gib 1 --num-memtable-bg-threads 2 --num-normal-bg-threads 2

But was able to reproduce some other bundle-related assertion with this combination:

$ build/release/bin/driver_test splinter_test --perf --num-insert-threads 4 --num-memtable-bg-threads 2 --num-normal-bg-threads 2

The assertion appears like:

Running splinter_test with 1 caches
splinter_test: SplinterDB performance test started with 1 tables
splinter_perf_inserts() starting num_insert_threads=4, num_threads=20, num_inserts=27185152 (~27 million) ...
Thread 2 inserting  57% complete for table 0 ... OS-pid=3493221, OS-tid=3493223, Thread-ID=2, Assertion failed at src/trunk.c:1969:trunk_get_new_bundle(): "(node->hdr->end_bundle != node->hdr->start_bundle)". No available bundles in trunk node. page disk_addr=393216, end_bundle=4, start_bundle=4

... and is duplicate of the failure reported under issue #474.

The 'resolution' for that seems to be to increase the # of bg-threads, so did not pursue investigating this further.

A rework of the error-handling when trunk_get_new_bundle() fails to get a new bundle and currently asserts has already been requested under issue #129.

[rtjohnso]

Fixed by #618