Feature Proposal: JSON Projected Attributes on Secondary Indexes and MGET_IF

[martinsumner]

Background

The Query API in Riak 3.4 replaces the secondary index feature for handling range queries with overloaded terms. Whereas in the original Riak 2.0 2i solution, attributes projected onto overloaded terms had to be filtered using regular expressions, in the Query API they could be converted into a map of attributes using an eval expression, before being filtered with a filter expression.

The design of this feature was focused on being backwards compatible with existing index entries, but this is not the most obvious and easy way of working with projected attributes on queries.

Further, as part of 4.0, the intention is to retire the Map/Reduce feature. Although there were issues with the feature, it is hoped to retain some of the power of the solution, but with a significantly reduced maintenance overhead. The primary use for Map/Reduce was as a mechanism for supporting multiple gets, that were conditional (i.e. MGET_IF). The expectation is to have a simple way of supplying a "stream" of multiple objects that have passed some filter - either filter on an index, or a filter on the object metadata.

Proposal

There are three proposed extensions to the Query API:

• Add support within the Object API for a _j64 index (i.e. rather than _bin). These indexes should be base64 encoded json with two elements to the json document - a $sort_key and a $attr_map; where the $sort_key is a binary string, and the $attr_map is a map of Key/Value pairings. This will then be stored as an index key as a {binary(), maps:map()} tuple. Querying this will not require an eval expression to create an initial map of projected attributes, as the map will simply be deserialised.
• Add support to the Query API to fold heads (not just index entries), such that the Key/Value pairs of the user metadata will be treated as a map of projected attributes, and an filter expression can be applied on that map of metadata to filter the results during the fold.
• Add support to allow for an accumulation_option of queue, where the output of a query (of object keys) to be pushed to a queue (e.g. just as with range_repl and other aae_folds), and so that a reference to the queue will be returned to the client (not a list of results), and where the client may then use the fetch API to fetch the resultant objects one by one.
  • the queue will exist on one node, but will be routable from all nodes;
  • the queue will timeout after inactivity;
  • the results will be sent through the query server, not streamed directly to the queue, so that the query server can add a special $end_of_results message that will arrive after all results;
  • results on the queue will be keys, the process of calling fetch will lead to their expansion into objects;
  • the merge_option will be added to the fetch API, and so the fetched object will be the result of the merge_strategy callback handle_get_response_body/2

There are some potential further extensions:

• In a _j64 index, a key in the $attr _map, can use a $b64. prefix, and be converted by the API into a binary for internal storage.
  • There may be a more general need for defining types of values in an attribute map, by tagging keys - so perhaps all keys should have a prefix, with initially the only prefix being $str. and $b64.
• A merge strategy behaviour may have an optional callbacks for filter expressions so that bespoke functions can be defined, and then used in filter expressions.

Design

Most of the scaffolding for the solution already exists within Riak, so the change in code footprint from adding these features should be small.

Using a fetch queue for the processing and receipt of a stream of objects, should be simpler from an API and riak client perspective than trying to stream objects.

This should provide functionality similar to what is available via riak_pip/map-reduce, but without some of the challenging side-effects:

• no dynamic addition of code at runtime via the API, all bespoke code controlled through the behaviour module;
• simpler model with respect to back-pressure.

Although there will be no capability to fail-over work in the event of node down (i.e. a riak_pipe_vnode can do handoff), however the usefulness of this feature is limited relative to the cost of implementing and maintaining test for it.

Alternative Design Ideas

Testing

Caveats

Depends in-part on the merge strategy behaviour bucket property.

Pull Request

Planned Release for Inclusion

Riak 4.0