Feature Proposal: AAE Exclude Bucket Property

[martinsumner]

Background

AAE is currently all-or-nothing in a cluster - enabling TictacAAE will turn on AAE for all buckets.

There may be certain buckets where data resilience is less important, and so it would be beneficial to exclude these buckets from AAE - so that the AAE caches do not contain knowledge of these buckets, and those buckets could have other properties that would otherwise conflict with intra-cluster AAE and inter-cluster full-sync.

This would mean that real-time replication could be optional for those buckets, as a failure to replicate would not lead a full-sync failure. So if a specific bucket was to be local, whereas the rest of the buckets were to be replicated - AAE Exclude bucket property could be enforced for that bucket, the bucket could be excluded from the real-time repl config - and full-sync would still work.

This would also allow for a TTL to be applied for that specific bucket (without the need for an eraser job), as this would not cause AAE failures after tree rebuilds

Proposal

Add an aae_exclude bucket property, which is false by default.

Keys could still be repaired by read_repair, and still included in hinted handoffs (and transfers). So this is suited to short-lived data (i.e. less than 3 months), where long-term entropy is not an issue.

As an extension there could be a ttl option possible on a bucket, which may be enabled if and only the aae_exclude option is set - as keys could now be auto-expired, without impacting AAE.

Design

Work ongoing:

• In parallel mode AAE, should the key still be added to the keystore, so that stats-style AAE queries would still work (i.e. the keys are excluded from the cache but not the store).
• In this case how should such results be filtered from fetch_clocks style queries, in the fold, or should it be the responsibility of the client to exclude. This might be a configurable option for the fold.
• Should there be a persistent term to add excluded_buckets, which is potentially updated every time there is an update to bucket properties (if the list of excluded buckets has changed). This would allow exclusion in folds to be a simple lists:member/1 check.

Alternative Design Ideas

.If a per-bucket TTL property is to be added, then it will probably need to be a leveled-only feature. For bitcask, an expiry setting is store-wide, and there is not an obvious and simple way of making it more specific - and certainly getting the same behaviour as with leveled (for example setting expiry per object at PUT time).

For this reason, it may be preferable to stick with eraser jobs as an answer to expiry, rather than try and make significant changes to bitcask. Currently eraser depends on operator prompting, but perhaps instead there should be a riak_kv_eraser_manager - that would look for buckets with expiry enabled, and periodically trigger eraser activity to garbage collect these buckets.

Testing

OpenRiak/riak_test#9

Caveats

...

Pull Requests

martinsumner/kv_index_tictactree#131

This PR adds support for a key_filter function to both the aae_controller and the aae_exchange. For each key/clock to be added to the cached tree (either due to aae_put or a a rebuild/rehash) the controller key_filter will be run against the inputted Bucket/Key pair - if it returns true, it will be added to the cache, otherwise it will not be added. Additionally on an exchange, the key filter will filter the Keys/Clocks being passed into compare_clocks.

In our case it is expected that the filter function will look only at the bucket part of the key, and check the bucket properties and return true unless the aae_tree_exclude property is present. It is assumed that there may be a desire to cache results of this proporty check, so an aae_controller:aae_reset_key_filter/1 function has been added to the API which will send a reset atom to the filter function to clear any cache.

Planned Release for Inclusion

Riak 3.4

[martinsumner]

With regards to naming, there is the question of whether the bucket property should be named by "what it does" or " what we expect it to be used for"?

For example, this is a feature intended for temporary objects that don't need the usual anti-entropy (and potentially inter-cluster resilience): so the bucket property could be {temporary, true|TTL} - what it is expected to be used for. This might, by some, be deemed more user friendly.

But perhaps for expert users, that keep a mental model of how things work, a name which is more explicitly about what it does might be preferable (e.g. the proposed aae_exclude). Also such a bucket property might be used as a workaround for some other issue in the future, in which case it would be confusing for it to be named after what we expected it to be used for.

[tburghart]

I think there's an argument to be made for "both" - aae_exclude does precisely what it says, while temporary might serve as a) a readable flag; b) an umbrella that implicitly turns on aae_exclude and potentially other settings; c) a hint to a bucket-type-aware backend as to how to store the object; d) etc ...

[martinsumner]

OpenRiak/riak_kv#42

Some detail of the planned implementation. As part of this it is suggested that the property be renamed aae_tree_exclude to make it clear that the exclusion is related to aae tree caches, and excluded buckets will still appear as results in aae_folds.