[RC tests] timestamp order with multiple threads and multiple regions tests

[KFilipek]

FEAT: timestamp order with multiple threads and multiple regions tests

Description

Prerequisites:
(Prereq 1) Add generator derivated by pmemstream_test_base for that case:

• Generates multiple regions
• Multithreaded synchronously append entries to regions

(Prereq 2) Add generator derivated by pmemstream_test_base for that case:

• Generates multiple regions
• Multithreaded asynchronously append entries to regions

Cases for (prereq 1 and 2):
Case 1:
In every region, all timestamps increase.

Case 2:
In stream, all timestamps increase (duplications).

Case 3:
Add regions, remove single region with entries, add an empty region and then add new entries then check Case 2.

Implementation details

N/A

[lukaszstolarczuk]

as for "prereqs" - you can use pmemstream_with_multi_empty_regions which generates multiple regions; MT sync/async appends can be done in a body of a test (that probably needs some new tests' helpers).

[igchor]

Still, you need to make sure that the region_runtimes map in helpers is prepopulated in a concurrent test (or always pass NULL as region_runtime).

Also, are you suggesting that the generator should also perform multithreaded appends? I don't think this is a good idea - concurrent/multithreaded appends should be explicitly stated in the test as @lukaszstolarczuk suggested. The only job of a generator is to provide an initial, correct state (and appending concurrently might not result in a consistent state, this is what we want to test).

[igchor]

I would also like to extend case 1 or 2 with value validation (similarly as we do in concurrent_iterate_with_append).

[igchor]

Also, please make sure that the code which iterates over all entries in the stream (all regions) and checks that timestamps monotonically increase is added as a helper to stream_helpers. We could even call this code every time in the destructor of stream helper (as we always expect stream to end up in consistent state).