Intermittent eunit failure - `riak_kv_ttaaefs_manager:take_first_workitem_test/0`

[martinsumner]

There is an intermittent eunit test failure with a riak_kv_ttaaefs_manager test:

riak_kv_ttaaefs_manager: take_first_workitem_test...*failed*
in function riak_kv_ttaaefs_manager:take_first_workitem_test/0 (/home/runner/work/riak_kv/riak_kv/src/riak_kv_ttaaefs_manager.erl, line 1655)
in call from eunit_test:'-mf_wrapper/2-fun-0-'/2 (eunit_test.erl, line 273)
in call from eunit_test:run_testfun/1 (eunit_test.erl, line 71)
in call from eunit_proc:run_test/1 (eunit_proc.erl, line 531)
in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 356)
in call from eunit_proc:handle_test/2 (eunit_proc.erl, line 514)
in call from eunit_proc:tests_inorder/3 (eunit_proc.erl, line 456)
in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 346)
**error:{assertMatch,[{module,riak_kv_ttaaefs_manager},
              {line,1655},
              {expression,"PromptS3YetMoreSeconds > PromptS2YetMoreSeconds"},
              {pattern,"true"},
              {value,false}]}
  output:<<"">>

[martinsumner]

This is as a result of the ?SCHEDULE_JITTER:

riak_kv/src/riak_kv_ttaaefs_manager.erl

Lines 80 to 81 in f4ed77e

	-define(SCHEDULE_JITTER, 60).
	% Jitter each schedule start time by up to a minute, to avoid overlaps

The schedule is determined by dividing up the 24 hour day by the number of work items, then by the number of nodes (so that each node doesn't overlap) and then by the cluster_slice (so that same nodes on different clusters don't coordinate) - and then the ?SCHEDULE_JITTER is added.

In the unit test, there are 48 work items - and so 30 minutes to schedule the work. There are then 8 nodes, and 4 cluster slices - and so each cluster slice and node is just less than 1 minute apart. The jitter may differ by 59s, so ordering is not guaranteed. ClusterSlice N + 1 may be triggered before ClusterSlice N.

When sizing the jitter, having 1500 full-syncs per day wasn't expected as a potential configuration. The actual assertion failed isn't vital, but it does mean that two clusters will very rarely in this setup trigger concurrent changes, rather than spaced out changes.

The jitter should probably reduce when the scheduling is so tight

[martinsumner]

Removed label (to fix in 3.4), as this is not a significant issue. Fix can be added to future patch release