docs: add validation framework requirements and detailed design by hdamker · Pull Request #447 · camaraproject/ReleaseManagement

hdamker · 2026-03-17T16:21:07Z

What type of PR is this?

documentation

What this PR does / why we need it:

Requirements and detailed design for the CAMARA Validation Framework, split into two documents:

Requirements (CAMARA-Validation-Framework-Requirements.md) — what the framework must do and what constraints apply:

Use cases and validation contexts (profiles, rule model, execution contexts)
MVP scope definition
Check inventory and rule architecture
Bundling and external ref resolution (Commonalities consumption, cache sync)
Artifact and findings surfacing
Workflow contract (reusable workflow inputs/outputs)
Rollout strategy (central config, stage model)
Release automation integration points

Detailed Design (CAMARA-Validation-Framework-Detailed-Design.md) — architecture decisions, processing flows, and implementation detail for developers:

Rule metadata model (YAML structure, condition evaluation, Spectral pass-through)
Bundling pipeline ($ref interaction, dependency categories, version matrix)
Artifact surfacing detail (naming, temporary branches, wip handling)
Caller workflow design (token resolution, GitHub App, triggers, ref resolution, templates)
Rollout implementation (central config alternatives, rulesets, caller update strategy)
Release automation implementation (handoff models, file restriction checks, context model)

The detailed design document is expected to migrate to the tooling repository alongside the implementation code.

Which issue(s) this PR fixes:

Related to #435

Special notes for reviewers:

The requirements document is the primary review target — it defines what the framework must do. The detailed design document provides supporting context for reviewers who want to understand architectural choices.

Presented at WG meeting 2026-03-17.

Changelog input

release-note
n/a (supporting document)

Additional documentation

docs
Added documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md
Added documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

Initial draft of the CAMARA Validation Framework requirements document covering use cases, validation contexts, rule architecture, bundling, caller workflow design, and rollout strategy. Work in progress — Sessions 5-6 (integration points, consolidation) still pending.

…on 5) Add sections 11 (Release Automation Integration) and 12 (Operational Views) to the validation framework requirements document. Update cross-references in sections 2.2, 2.3, 3, and 7.7. Resolve deferred bundling/snapshot interaction topic.

Split the 1,178-line combined document into a focused requirements document (522 lines) and a companion detailed design document (634 lines). The requirements document is now reviewable by WG members without implementation detail. The detailed design document preserves all architecture decisions, processing flows, YAML structures, and workflow templates for implementers.

Add sections 8 and 9 to the detailed design document, completing the workflow architecture needed for implementation: - Section 8: job architecture, checkout strategy, context builder, engine orchestration, composite action boundaries, common findings model - Section 9: post-filter processing, profile application, output formatting, truncation strategy, line number mapping, error handling, bundled spec artifact handoff (Model B) Also updates: - Section 6.2: central config schema with fork_owners allowlist - Section 7.1: settle on artifact handoff model (Model B) - Section 7.4: clarify mode input for release automation invocation - Requirements section 9.2: add mode input to workflow inputs table

hdamker · 2026-03-18T19:29:19Z

Added sections 8 and 9 to the detailed design document (Session 7):

Section 8 — End-to-end processing flow: single-job workflow architecture, checkout strategy, context builder with branch/trigger/profile derivation, engine orchestration (yamllint → bundling → Spectral/gherkin/Python), common findings model
Section 9 — Output pipeline: post-filter processing, profile-based blocking, output formatting (summary, annotations, PR comment, commit status), truncation, line number mapping, error handling, bundled spec artifact handoff to release automation

Also updated: central config schema (section 6.2) with fork_owners allowlist, settled on artifact handoff model for bundled specs (section 7.1), added mode input to workflow contract (section 7.4 + requirements 9.2).

tanjadegroot

Looks good to me - a few suggestions for consideration

/LGTM

documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

…Detailed-Design.md Co-authored-by: Tanja de Groot <87864067+tanjadegroot@users.noreply.github.com>

- Add CAMARA-Validation-Framework-Design-Guide-Audit.md: comprehensive audit of 106 machine-checkable rules from both design guides, cross- referenced against existing Spectral rules, OWASP rules (tooling#95), Linting-rules.md, and api_review_validator_v0_6.py - Update section 2.2 with complete check areas by engine - Add Appendix A: naming conventions (full list from design guide) - Clean internal tracking identifiers from document

Audit of 65 machine-checkable rules from API-Testing-Guidelines.md, cross-referenced against gherkin-lintrc (25 rules) and api_review_validator_v0_6.py test alignment checks.

Cross-reference identified discrepancies, gaps, and disabled rules against prior Commonalities and tooling issues/PRs to document known rationale and decisions.

camara-language-spelling was never intended for Spectral — marked "No/No" from day one. Spelling checks are better suited to AI-based review tools. Move from discrepancies to other items section.

hdamker · 2026-03-19T09:20:16Z

Two working documents added as input for the validation framework check inventory:

CAMARA-Validation-Framework-Design-Guide-Audit.md — Audit of 106 machine-checkable rules extracted from CAMARA-API-Design-Guide.md and CAMARA-API-Event-Subscription-and-Notification-Guide.md, cross-referenced against existing Spectral rules, the OWASP rules from tooling#95, Linting-rules.md, and api_review_validator_v0_6.py. Includes coverage mapping, gap identification, version sensitivity analysis (r3.4 vs r4.x), and a summary of prior Commonalities discussions related to the findings.
CAMARA-Validation-Framework-Testing-Guidelines-Audit.md — Audit of 65 machine-checkable rules from API-Testing-Guidelines.md, cross-referenced against the gherkin-lintrc configuration and the v0_6 validator test alignment checks.

Both documents are working documents to support the design review — feedback on completeness and accuracy is welcome.

The check areas summary in the detailed design (section 2.2) and a naming conventions appendix (Appendix A) have been updated based on the audit findings.

Section 2.2: add gherkin-lint engine category listing test definition file checks (structural rules, step ordering, tagging, formatting). Section 8.4: add gherkin-lint column to findings normalization source mapping table.

Conditional check for stable public APIs per API Readiness Checklist. File existence is automated; content adequacy remains manual.

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

rartych · 2026-03-19T11:44:56Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+
+### Cross-cutting finding
+
+The incremental rollout approach was established in [PR#74](https://github.com/camaraproject/Commonalities/pull/74) (Oct 2023), where rules were documented before implementation. Several rules from that era remain documented but not yet implemented. No existing issue specifically tracks the gap between documented rules in Linting-rules.md and implemented rules in .spectral.yml.


That's true.
Most of missing rules requires TypeScript function implementation, some are even hard to implement within spectral.
BTW, camara-schema-type-check rule is implemented partially, I have prepared function that looks into more complex schema definitions that seems to work. Probably new rule should be added.
It happens that type: object is missing - it works with even Swagger - as properties: implicitly indacates it is an object - so Swagger doesn't complain.

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

documentation/SupportingDocuments/CAMARA-Validation-Framework-Testing-Guidelines-Audit.md

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

Kevsy · 2026-03-23T15:36:38Z

Two of the documents use uppercase directives - so I've suggested to add the boilerplate RFC 2119 text, and whatever heading you want for that (API Design Guide has 'Conventions').

Otherwise LGTM!

hdamker · 2026-03-23T15:45:19Z

Two of the documents use uppercase directives - so I've suggested to add the boilerplate RFC 2119 text, and whatever heading you want for that (API Design Guide has 'Conventions').

Both documents are not themselves authoritative, but they are reflecting what was found in the guideline documents. But sure, the RFC 2119 text is doing no harm here.

…Design-Guide-Audit.md Co-authored-by: Kevin Smith <Kevsy@users.noreply.github.com>

…Testing-Guidelines-Audit.md Co-authored-by: Kevin Smith <Kevsy@users.noreply.github.com>

Kevsy

LGTM

tanjadegroot

comments on CAMARA_Validation_Framework_Detailed_Design.md

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

…ent-based inclusion Clarify in section 3.1 that bundling converts external $ref to internal $ref with component inclusion in the appropriate components/ subsection, covering all component types (schemas, securitySchemes, headers, parameters, responses, examples) with deduplication. Add explicit bundling tool requirements in section 8.4 step 6: component-based inclusion (not raw JSON pointer inlining) and deduplication of multiply-referenced components.

Replace hardcoded file list (CAMARA_common.yaml, notification-as-cloud-event.yaml) with reference to Commonalities#603 restructuring. The tooling must map commonalities_release versions to the correct source files without assuming a fixed file list.

Replace concrete dependency declaration design with a neutral statement that the framework should not preclude cross-repository schema sharing, without committing to a specific organizational model. The how/where is an open question outside the scope of this design.

documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md

tanjadegroot · 2026-03-24T21:44:09Z

I finished reviewing all files.

I put info in one comment which now can be resolved (in design-guide-audit on the PRIVATE_KEY_JWT)
I had 2 comments on the requirements doc that we did not check in the call today. One may imply some additional work (presence check of release assets.

@hdamker please check those 2 comments and then I can approve.

Add API-Testing-Guidelines.md and artifacts/CAMARA_common.yaml as authoritative sources for validation rules. Note that further Commonalities artifacts depend on restructuring (Commonalities#603).

Clarify that artifact presence checks (per target API status) do not run on the release review PR — already validated at snapshot creation time and artifacts are immutable on the snapshot branch.

hdamker · 2026-03-24T22:29:53Z

@hdamker please check those 2 comments and then I can approve.

@tanjadegroot Addressed both comments. Added #447 (comment) to close another conversation (about breaking changes).

Add implementation guardrails for UC-03 (local validation) to the detailed design. Classifies processing steps by environment portability, adds `local` trigger type to the rule metadata vocabulary, and defines five constraints to keep the framework implementation reusable for local tooling. Move UC-03 from MVP to post-MVP — the design guardrails are MVP scope, but the local entry point and tooling are not.

hdamker · 2026-03-24T23:48:29Z

Added Appendix B: Local Validation Considerations to the detailed design, and moved UC-03 (local validation) from MVP to post-MVP.

The local entry point is not on the critical path for replacing pr_validation v0. The appendix ensures the framework implementation stays modular enough for local reuse later — classifying processing steps by environment portability, adding local to trigger_types, and defining implementation guardrails (e.g., engines don't read GitHub env vars directly, output supports terminal, tooling path is configurable).

Design guardrails remain MVP scope; only the local entry point itself is deferred.

Commit: ef6dc56

tanjadegroot

Great all comments addressed

/LGTM

hdamker · 2026-03-26T13:41:54Z

I merge the documents as the baseline for the implementation and will open a new PR with changes arising from the implementation and to collect further feedback.

hdamker added 3 commits March 17, 2026 17:18

hdamker changed the title ~~docs: add validation framework requirements and design~~ docs: add validation framework requirements and detailed design Mar 17, 2026

hdamker requested review from a team and Kevsy March 17, 2026 22:25

hdamker marked this pull request as ready for review March 17, 2026 22:26

hdamker requested review from rartych, soadeyemo and tanjadegroot as code owners March 17, 2026 22:26

This was referenced Mar 18, 2026

Implement CAMARA Validation Framework v1 #448

Open

Implement PR Validation Framework (C1 Workflow) #353

Closed

tanjadegroot previously approved these changes Mar 18, 2026

View reviewed changes

Update documentation/SupportingDocuments/CAMARA-Validation-Framework-…

5ff2df3

…Detailed-Design.md Co-authored-by: Tanja de Groot <87864067+tanjadegroot@users.noreply.github.com>

hdamker dismissed tanjadegroot’s stale review via 5ff2df3 March 19, 2026 07:14

hdamker added 5 commits March 19, 2026 08:28

docs: expand naming conventions examples in check areas summary

31bc028

docs: add testing guidelines audit

21d8949

Audit of 65 machine-checkable rules from API-Testing-Guidelines.md, cross-referenced against gherkin-lintrc (25 rules) and api_review_validator_v0_6.py test alignment checks.

docs: add prior discussions section to design guide audit

b2a1480

Cross-reference identified discrepancies, gaps, and disabled rules against prior Commonalities and tooling issues/PRs to document known rationale and decisions.

docs: reclassify spelling rule as deliberate exclusion

8d90eb0

camara-language-spelling was never intended for Spectral — marked "No/No" from day one. Spelling checks are better suited to AI-based review tools. Move from discrepancies to other items section.

hdamker added 2 commits March 19, 2026 10:26

docs: add user story file existence check to Python engine

77b728a

Conditional check for stable public APIs per API Readiness Checklist. File existence is automated; content adequacy remains manual.

rartych reviewed Mar 19, 2026

View reviewed changes

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md Show resolved Hide resolved

rartych reviewed Mar 19, 2026

View reviewed changes

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md Outdated Show resolved Hide resolved

hdamker mentioned this pull request Mar 20, 2026

Release Issue Not Automatically Created for SessionInsights Repo #451

Closed

hdamker requested a review from rartych March 23, 2026 14:01

Kevsy reviewed Mar 23, 2026

View reviewed changes

documentation/SupportingDocuments/CAMARA-Validation-Framework-Testing-Guidelines-Audit.md Outdated Show resolved Hide resolved

Kevsy reviewed Mar 23, 2026

View reviewed changes

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md Outdated Show resolved Hide resolved

hdamker and others added 2 commits March 23, 2026 16:45

Update documentation/SupportingDocuments/CAMARA-Validation-Framework-…

4f6e3ab

…Design-Guide-Audit.md Co-authored-by: Kevin Smith <Kevsy@users.noreply.github.com>

Update documentation/SupportingDocuments/CAMARA-Validation-Framework-…

df625b4

…Testing-Guidelines-Audit.md Co-authored-by: Kevin Smith <Kevsy@users.noreply.github.com>

Kevsy previously approved these changes Mar 23, 2026

View reviewed changes

tanjadegroot reviewed Mar 23, 2026

View reviewed changes

hdamker dismissed Kevsy’s stale review via 9c9abc3 March 24, 2026 14:37

hdamker added 4 commits March 24, 2026 16:03

docs: clarify step reference points to validation pipeline (section 6.2)

cfcc77a

docs: fix API name in examples (qos-on-demand → quality-on-demand)

d817b23

hdamker requested review from Kevsy and tanjadegroot March 24, 2026 16:51

tanjadegroot reviewed Mar 24, 2026

View reviewed changes

documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md Show resolved Hide resolved

documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md Show resolved Hide resolved

hdamker added 2 commits March 24, 2026 23:12

docs: add testing guidelines and common schemas to authoritative sources

1f9707d

Add API-Testing-Guidelines.md and artifacts/CAMARA_common.yaml as authoritative sources for validation rules. Note that further Commonalities artifacts depend on restructuring (Commonalities#603).

docs: add release readiness checks to release review PR table

87b09b6

Clarify that artifact presence checks (per target API status) do not run on the release review PR — already validated at snapshot creation time and artifacts are immutable on the snapshot branch.

hdamker force-pushed the docs/validation-framework-requirements branch from 346acd9 to ef6dc56 Compare March 24, 2026 23:50

tanjadegroot approved these changes Mar 25, 2026

View reviewed changes

hdamker merged commit 3c8478e into camaraproject:main Mar 26, 2026
1 check passed

hdamker deleted the docs/validation-framework-requirements branch March 26, 2026 13:49

hdamker mentioned this pull request Mar 26, 2026

docs: update validation framework design for hint field split and bundling clarification #463

Draft


		### Cross-cutting finding

		The incremental rollout approach was established in [PR#74](https://github.com/camaraproject/Commonalities/pull/74) (Oct 2023), where rules were documented before implementation. Several rules from that era remain documented but not yet implemented. No existing issue specifically tracks the gap between documented rules in Linting-rules.md and implemented rules in .spectral.yml.

Conversation

hdamker commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for reviewers:

Changelog input

Additional documentation

Uh oh!

hdamker commented Mar 18, 2026

Uh oh!

tanjadegroot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hdamker commented Mar 19, 2026

Uh oh!

Uh oh!

rartych Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kevsy commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hdamker commented Mar 23, 2026

Uh oh!

Kevsy left a comment

Choose a reason for hiding this comment

Uh oh!

tanjadegroot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tanjadegroot commented Mar 24, 2026

Uh oh!

hdamker commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hdamker commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tanjadegroot left a comment

Choose a reason for hiding this comment

Uh oh!

hdamker commented Mar 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hdamker commented Mar 17, 2026 •

edited

Loading

rartych Mar 19, 2026 •

edited

Loading

Kevsy commented Mar 23, 2026 •

edited

Loading

hdamker commented Mar 24, 2026 •

edited

Loading

hdamker commented Mar 24, 2026 •

edited

Loading