Requirements evaluation

[bruth]

Here is the discussion that defines many of the needs. I would to try and describing the requirements using the Spine model.

Needs

• Ability to create concepts and domains
• Ability to make changes to concepts and domains
• Ability to provide information about the change
• Ability to propose changes for review
• Ability to comment on proposed changes
• Ability to do all of the above without disrupting live concepts being consumed by clients or users (i.e. staging vs. production)
• HTTP+JSON API for performing operations programmatically
• Web client to facilitate create/refine, review, publish workflow

Values

Principles

Practices

Tools

[l-k-]

Needs

• Ability to maintain and view the history if changes to concepts and domains
• Ability to revert to a prior version of a concept or domain
• Ability to evaluate the effect of a proposed concept change on existing queries

Values

Q: What do we optimize for?

A: Average time that it takes to perform the tasks outlined below,

along with the average resulting user frustration level:

• create a simple (representative?) model
• create a change, document it, and make it available for review
• review a proposed concept change (i.e. understand the nature and the effects of the proposed change). [Also: we might need a measure of the quality of reviews (e.g. number of missed undesired effects?)]
• identify the [timestamp / revision context] and authorship of a particular concept change
• propose and review reverting to a prior version of a concept or domain
• merge reviewed changes into the main (production) branch available for client consumption

[bruth]

This is good. A few more things to consider:

Concepts from at least one query evaluator will be imported by the system itself. These will be identified by some configured unique name (provided by the source or configured locally??). If changes to concepts are made by the evaluator, they will be propagated to the local registry (diagram for context).

Admins of the concepts would need to review the changes, specifically ones that may be breaking. There are a few different situations that can arise each with different ways of handling things.

# compare concepts
for concept in remote.concepts
    if concept not in local.concepts   # new, check using id
        return [ review is optional ]

    if concept is not in local.mapped
        return [ review is optional ]

    diff = compare(concept, local.concept)
    if diff.non_breaking
        return [ review is optional | auto-merge ]

    local.concept.disable = true  # disable local concept
    return [ review is required ]

# look for removed concepts
for concept in local.concepts
    if concept not remote.concepts:
        local.concept.disable = true
        return [ review is required ]

As you point out, what has changed needs to be present. Since this is an automated process, why something changed in the upstream will likely not be present. This may be feasible to handle in the local registry for the evaluator, but not the local registry for the app.

Thinking about this more, this is not necessary a proposed change, but rather a conflict of state between the local and remote registries. This is slightly different from what is described above since the changes have not been applied yet (and therefore being proposed). In this case, the changes have been made upstream already and are basically forcing the downstream to deal with it.

@gerpsh This brings up an interesting point. If concepts are always derived from the evaluator, then there is little insight to why a change occurs (add, remove, change). I still believe a registry could be bootstrapped from an evaluator, but then a best practice from there would be to good through the change, propose, review process being updating the evaluator to implement the change.

The proposed change could theoretically be shared with subscribers to evaluate if any of the mappings would break before the change is committed. Basically acknowledging dependent subscribers.

[bruth]

To summarize the last part, a fresh setup would look like this:

1. Deploy evaluator and registry together
2. Initial import of concepts from evaluator
3. Initial review of concepts to make available to consumers (subscribers)
4. Now the evaluator is available to be subscribed to

Next, the coordinator and local registry for a particular use case.

1. Coordinator and local registry come online, subscribes to evaluator
2. Pulls in available concepts locally
3. Initial review of which to make available locally
4. Optionally make additional changes such as names, etc.
5. Optionally create domains (subsets of concepts for organization)
6. Ready

Now let's say there is a a new concept being requested that requires a new implementation. The most logical approach would be to add a new entry in the evaluator registry, propose the changes, implement the concept, publish the concept. Once published this would be pushed to subscribers which will go through the logic mentioned above, however this would be a single concept rather than everything. In fact, except for the initial import, the changes would always be in incremental. An import is really just a bunch of new entries anyway.

Another case is that the implementation needs to change, but does not impact the concept definition. I feel that this should still be recorded somehow. Maybe an "implementation version" in the concept metadata. Since this could like to breakages without anyone realizing it.

Another case is a change in the concept description itself. This could be done locally if it is domain specific, or it could be done upstream in the evaluator (if you have control over it). The same review process would happen.

Based on what I described, this leads me to believe that changes will be largely driven by the evaluators. The only changes that make sense locally are changing the names of things to suit the domain.