Nhse o34 orkv.i61 repairoptions #14
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allows for type-specific list of options to be added by Module:handoff_started/2, and also if the Module exports a handoff_encoding_fun/2 an alternative encoder can be provided rather than Module:encode_handoff_item/2.
This allows for a riak_core_vnode_manager that is managing a repair to return negative filters when a second handoff is requested for the same target partition - so that data already sent in the first handoff is not resent. If using a fold_heads, this will avoid the deferred fetch.
Increase the width of repairs so that more nodes can contribute - but using the NegativeFilters to avoid duplication of work. To use a broader width - double-pair should be set as the riak_core/repair_span, not pair.
Previously the riak_core_repair filter used the a riak_core_bucket function to return an NValMap - but that doesn't include bucket types within the map. There's no equivalent for typed buckets - so lookup dynamically, and use the process dict to avoid repeated lookups. Also as ring_sizes beyond 64K are not supported - make life easier by checking with 16-bit integers only
Will otherwise produce hundreds of logs. Also make the core worker pool logs metric logs
As it may be use multiple times with a negative filter
As we only need log2(RingSize) bits of the hash - just work with these small integers.
Contributor
Author
Contributor
Author
This was referenced Feb 27, 2025
ThomasArts
approved these changes
Mar 10, 2025
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
FilterFun must only return boolean according to type - so NotTrue must be false
Snappy issues now reoslved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are four main parts to this refactoring:
riak_core_handoff_senderfromMod:handoff_started/2(this is then used by riak_kv to turn fold_objects queries into fold_heads with deferred fetching).The net effect is that multiple repairs for each vnode will be triggered, but the the losers of the race to start will have less work to do (and potentially no work). By using fold_heads with deferred fetch, there is minimal work when losing the race. This allows for busier nodes to avoid repair work, and for less busy nodes to pick up the slack.