Feature: Explicit review checkpoints

[jamesmunns]

(from this bluesky thread)

IMO, it would be useful to have some kind of manual/human review checkpoint system.

For each requirement, ask a reviewer:

• Does this requirement make sense? (or any other quality checklist criteria)
• Then, for each impl tag linked to that requirement, ask "does this tagged block actually and correctly implement a part of the associated requirement?"
• Then for ALL impl tags of a requirement, ask "do these impl blocks FULLY implement the associated requirement?"
• Then for each verify block, ask "does this tagged block actually and correctly verify a part of the associated requirement?"
• Then for ALL verify blocks, ask "do these verify blocks FULLY verify the associated requirement?"

When the manual review "signs off" on this chain, mark them as "reviewed". If any of the three aspects change, invalidate the chain.

I think the web interface could be used to drive this in a PR-like manner, or as an interactive step-by-step process.

My intended use for this would be to have a human-in-the-loop, letting the agent write/expand requirements, implement code, and write verification tests, but to periodically "QA" the process, and track when local changes potentially have larger ripple effects that need to be updated.

I think this could even be fed-forward into the agent (if it's not already by the MCP?), "if you touch this code, make sure you put effort into resolving any changes in requirements or tests".

I personally think the agent shouldn't be doing the manual review step, but I think it would also be okay to have that as a configurable option for people who just want to use the metadata but still work "open loop".

CC @eikopf

[jamesmunns]

(this is extremely derivative of the DO-178 review process, and part of the traceability lifecycle tracking required for those systems. I do think that automation for this is a huge force-multiplier for humans-in-the-loop, even if agents are doing most of the lift on the line-level work)

[orual]

if an agent does the manual review step, it must be via calling out to another agent with an appropriately rigorous review prompt. agents are completely incapable of reviewing their own work, and even delegated review agents with good prompts tend to trust what I call "cope comments" from the implementer far too much, unless directed to be paranoid and adversarial.

[eikopf]

I'll add that while it would be nice for this to be accessible through the web UI, it would also be useful to extract the functionality so that editors (and maybe also agents?) can interact directly with and drive the review process. I'm not sure whether this can be jammed into an LSP message, but at the very least it should be accessible through the CLI.

[eikopf]

"cope comments"

Claude trusts me too much this is very true

[eikopf]

I'll add that while it would be nice for this to be accessible through the web UI, it would also be useful to extract the functionality so that editors (and maybe also agents?) can interact directly with and drive the review process. I'm not sure whether this can be jammed into an LSP message, but at the very least it should be accessible through the CLI.

I think we can have the daemon track the state of the review process and expose CLI commands for driving it? I can't find anything in the LSP spec that we could retrofit for this purpose...

[orual]

"cope comments"

Claude trusts me too much this is very true

hehe it trusts itself far too much as well.

[eikopf]

I have a couple of open questions about the design:

1. Should the review process generate an artifact that can be used to tell what it did? Or is that just a VCS concern?
2. Is this a local-only process? Or does it have some state that can be committed and modified by multiple reviewers?

I imagine this as something local-only and which you can track by committing any changes after the fact, but maybe that doesn't scale when you have a significant number of requirements? Or maybe it's fine that it doesn't scale because the user can restrict the set of requirements they're reviewing?

[orual]

I think it should produce an artifact of some kind. that guards against a bunch of error modes in agent workflows in general.

[jamesmunns]

I don't know if tracey already maintains a hierarchical tree of req -> impl -> verify anywhere, but if it does, I would assume it could be kept as metadata in that kind of tree, potentially with some info about what each review did (who did it, what they verified, maybe comments?).

In tools I've used before, we would have a pane of "links that require (re)-review)" as just a running list, essentially as open todos. It was expected that during development, there would be some "open" links, but before we hit release (or pre-release checkpoints), all links would be closed with review (and sometimes, even do a full manual re-review).

In those tools, there would be a pane with basically just a list of selectable links that needed to be validated, and you could select one or multiple at a time to initiate a review. So, the tool would need to:

• Keep the metadata of which links have presently validated reviews
• Produce a list of unvalidated links, and a list of validated links
• Ability to select+review a set of links (maybe even including already-valid links)
• Detect changes to tagged "blocks", which allow for the invalidation of relevant links

[eikopf]

What information do you care about tracking? I'm thinking about a receipt here that lists the reviewed requirements and describes the outcome for each of them.

[eikopf]

In tools I've used before, we would have a pane of "links that require (re)-review)" as just a running list,

Do you know how this list was calculated? Or was it a manual thing?

[jamesmunns]

In those tools, "reviews" were first class metadata objects, like requirements or other tags. It was automatically maintained by the tool, no idea what the internal data structures or databases looked like, though imo for Tracey it would need to live as content inside the repo.

[eikopf]

I guess we could revive the old .tracey/ directory to use for this kind of shared state? Tracey has a notion of requirements belonging to a containing code object (function/struct/impl/etc.), so we could store a hash of the code object when it was last reviewed for each requirement and maybe some other metadata like the date and the reviewer.

[fasterthanlime]

so we could store a hash of the code object when it was last reviewed for each requirement and maybe some other metadata like the date and the reviewer.

yes, even though.. that doesn't capture deps of that code object, but it's a start. for tracey to really work it needs to be integrated into a build system... 😎