Revisiting the member extraction algorithm (MEA) section

[pietercolpaert]

Opening an issue for an ongoing discussion in the TREE CG meetings:

The Member Extraction Algorithm (MEA) is optional to implement (MAY), as it depends on the goals of the client (e.g., if the client wants to select a specific BGP, it does not need the MEA). Nevertheless, it is very important for LDES, where a consumer needs to be able to replicate a set of quads per member. We will therefore talk about TREE/LDES below.

Problem

The member extraction algorithm (MEA) is difficult to implement as it is written trying to solve member extraction in all possible cases, supporting all possible RDF-based APIs. In reality, we see clients that only partially support member extraction based on the needs of their ecosystem. This however hinders cross-domain interoperability of TREE/LDES clients and we argue that we might want to move the goal posts on what we want to achieve. Instead of having a member extraction algorithm that works for everything, it would already be nice to have a member extraction algorithm subset that works for TREE/LDES.

E.g., a member extraction algorithm for TREE/LDES probably does not need support for partial out of band members, but this is added in the algorithm for supporting APIs like LDP or IIIF-CD, without any practical benefit for TREE/LDES.

What is the minimum that we want to mandate? And what do we make optional to implement? And if something is optional, how do we indicate what a client wants?

Related issue:

• Extending the MEA with blank node graphs for associating context: Member extraction algorithm: supporting multiple named graphs in one member #113
• Clearing out the definition of tree:shape: Clear out tree:shape definition #75

Proposed re-work

The MEA should include:

1. CBD on the first focus node (subject-based star patterns including blank nodes recursively)
2. Named Graphs support
3. Somehow support for blank node graphs and more elaborate context associations cfr. what is being proposed/discussed in Member extraction algorithm: supporting multiple named graphs in one member #113?

When these 3 steps would result in no triples at all, then an extra HTTP requests will be done to the focus node.

While the tree:shape MAY thus point at more data than the CBD and the named graph(s), it is not automatically fulfilled. The client MAY implement a mode however to fetch all data needed to perform a validation of the tree:shape. This however will be optional to implement, and we can then still refer to the shape topology algorithm that can evolve separately from the TREE spec.

[smessie]

I’m still not convinced that the shape topology part of the MEA should be optional to implement. I’m still in favor of a MUST implement requirement.

Members can go beyond the complexity of CBD and named graphs, even if they are fully in-band. However, next to that, members can also be partially or fully out-of-band.

Using a publisher provider shape remains the only way to ensure that the client is extracting members as intended by the publisher.

Sure, it can be a bit more challenging to implement, but I believe that ease of implementation should never compromise the correctness of a specification. Furthermore, a client not implementing shape topology should not be considered a complete implementation, especially if you want to maintain domain-agnosticism with TREE.

If you only want to use CBD and named graphs to extract members, you should restrict the publisher to publish members in that format only (and to be clear, I’m not proposing to do so here).

[pietercolpaert]

The problem is that shape topologies is also only a partial solution. E.g., it at this moment does not allow to include literals adhering to a regular expression and excluding literals adhering to another. How far we go into fixing RDF that does not have a formal way to indicate a “package” properly will always be a heuristic / a trade-off on functionality we need.

We started with setting the goal that we want to support how APIs such as IIIF-CD, ActivityPub, or LDP do it today. This had the requirement that we should support partially out of band triples, for which we therefore needed shape topologies. The shape topologies spec tries to formalize a generalized Member Extraction Algorithm for any kind of API. However, this quickly became a scope creep which indeed becomes more difficult to implement, however I agree that that should not be the main reason to not do something in a spec.

Today I’m proposing to move that goal a bit: sacrificing generalizability towards a more easy to understand system for both clients and providers. When an API needs to be able to work with a TREE client, they can also take into account the fact that by default the client will extract the set of quads in a more specific way; and we thus would sacrifice compatibility with specs like LDP, Hydra, ActivityPub, etc.

On MUST/SHOULD/MAY: today the MEA as a whole is a MAY for TREE clients to implement. SPARQL or autocompletion clients for example would not need to implement the MEA. It’s only useful for clients that are domain agnostic and want to pass on the package of triples as intended by the initial provider. If the provider wants to be clear about what set of triples it provides, it is better to use a packaging technique, such as a named graph, or maybe even a context association. I think as the MEA in itself is already a MAY, I think we therefore can specify both the TREE MEA as well as the generalized MEA which then would be the Shape Topology algorithm. LDES could then indicate they use the TREE MEA, but can enable the shape topologies algorithm when it is interesting to do so. Therefore I think the MUST/SHOULD/MAY in the LDES client of which MEA to support should be in the LDES spec, and not in the TREE spec.

On making it mandatory for servers: well, the client spec is of course steering the server primers we’re building. When the client works in a certain way, servers must follow of course.