diff --git a/documentation/SupportingDocuments/Commonalities-Consumption-and-Bundling-Design.md b/documentation/SupportingDocuments/Commonalities-Consumption-and-Bundling-Design.md new file mode 100644 index 0000000..862cd9c --- /dev/null +++ b/documentation/SupportingDocuments/Commonalities-Consumption-and-Bundling-Design.md @@ -0,0 +1,343 @@ +# Commonalities Consumption and Bundling Design + +> **Scope:** Design proposal for consuming `CAMARA_common.yaml` and similar shared schemas in CAMARA API repositories, including validation, synchronization, reviewer visibility, and release-time bundling. +> +> **Related discussion:** [Commonalities issue #577](https://github.com/camaraproject/Commonalities/issues/577) + +--- + +## Executive Summary + +Several CAMARA API repositories need a supported way to consume shared schemas from `CAMARA_common.yaml` without copy-paste. This became urgent after recent Commonalities schema updates, including OWASP-related changes, because repeated manual copying into API repositories does not scale and destroys provenance. + +This proposal recommends: + +1. **Controlled local copy on `main`:** API repositories consume Commonalities schemas through a local cached copy, referenced via relative `$ref`. +2. **`release-plan.yaml` as source of truth:** `dependencies.commonalities_release` defines the intended Commonalities dependency. +3. **CI-managed synchronization:** the local cached Commonalities file is treated as cache, not as a manually maintained source file. +4. **Source-only `main`, bundled release artifacts:** bundled API definitions are not committed on `main`, but are generated for PR review and committed on snapshot/release branches and tags. +5. **In-place replacement:** source files and bundled files occupy the same path (`code/API_definitions/api-name.yaml`). On `main` the file may contain external `$ref` to `common/` or `modules/`; on snapshot/release branches these are replaced with the referenced content. There is no separate source or output directory. + +This design aligns Commonalities consumption with the release automation concepts already introduced for CAMARA release preparation. + +--- + +## 1. Motivation and Problem Statement + +Section 2.1 of the CAMARA API Design Guide points to `CAMARA_common.yaml` as a shared source of common schemas, but does not define a standard consumption model. In practice this has led to: + +- copy-paste of shared schemas into API repositories +- unclear provenance of consumed Commonalities content +- avoidable maintenance effort whenever Commonalities evolves +- inconsistent tooling support across repositories + +The immediate pressure is increased by recent updates in `CAMARA_common.yaml`, where API repositories should be able to adopt shared schema changes without re-solving the same problem by local duplication. + +--- + +## 2. Design Goals + +The solution should provide: + +- **Traceability:** the exact Commonalities release used by an API repository must be identifiable +- **Tooling compatibility:** source files must work with standard OpenAPI tooling and local development workflows +- **Reviewer visibility:** codeowners must be able to inspect the effective API impact, including changes caused by shared schema updates +- **Release artifact quality:** published releases must expose complete standalone API definitions +- **Identical layout across branches:** the repository structure does not change between `main` and release branches — only file content changes (source → bundled). Tools and processes that operate on `code/API_definitions/` work the same way regardless of branch. +- **Low contributor friction:** the model should remain understandable and practical for normal API repository contributors + +--- + +## 3. Repository Model on `main` + +API repositories should adopt a controlled local copy model with a canonical layout: + +```text +code/ + API_definitions/ # source API definitions + common/ # local cached copy of CAMARA_common.yaml + modules/ # optional project-local reusable schemas +``` + +Rules on `main`: + +- API source files reference Commonalities content via relative `$ref` into `code/common/` +- project-local reusable schemas may be placed in `code/modules/` and also referenced via relative `$ref` +- remote URL-based `$ref` are not used for normative schema consumption +- bundled standalone API definitions are not committed on `main` + +### 3.1 Source of Truth + +`release-plan.yaml.dependencies.commonalities_release` is the authoritative declaration of the intended Commonalities dependency. + +The local `code/common/CAMARA_common.yaml` file is only a cache of that declaration. It is not a manually curated source. + +### 3.2 Cache Ownership + +Manual edits to `code/common/CAMARA_common.yaml` are not supported. The cache is expected to be synchronized by automation. + +--- + +## 4. Dependency Synchronization Model + +The synchronization mechanism should be permanent, not a one-time migration helper. + +Supported triggers: + +- automatic trigger after relevant `release-plan.yaml` changes +- manual trigger to re-sync `code/common/CAMARA_common.yaml` later if required + +Expected behavior: + +- synchronization is idempotent +- if no change is needed, no PR is created +- the synchronization result is surfaced through a normal PR +- snapshot creation validates that the cache is exactly in sync with the declared dependency + +This follows an already established principle in CAMARA release discussions: + +- `release-plan.yaml` defines the intended target +- PR validation on `main` reports what still needs to change before a release can be created +- the final hard block occurs at snapshot creation time + +An out-of-sync Commonalities cache is therefore expected to be: + +- **a warning on `main`** +- **a blocker for snapshot creation** + +--- + +## 5. Bundling Decision + +Bundled API definitions should **not** be committed on `main`. + +Instead: + +- `main` keeps source files only (containing `$ref` to `common/` and `modules/`) +- PR workflows generate bundled artifacts and bundled diffs for reviewer visibility +- release automation replaces external `$ref` with the referenced content in place on snapshot/release branches and tags — the file path (`code/API_definitions/api-name.yaml`) stays the same + +### 5.1 Rationale + +Committing generated bundled files on `main` would: + +- pollute PRs and git history with generator output +- increase merge conflicts +- obscure the real source-level changes +- create a risk that contributors start editing generated files directly + +Keeping source only on `main` preserves a clean development model while still allowing reviewers and consumers to see the complete API through generated outputs. + +--- + +## 6. Reviewer Visibility and Diff Priorities + +The issue discussion highlighted that codeowners want to see the effective API changes, especially when `CAMARA_common.yaml` changes. That need is valid, but it does not require committing bundled files on `main`. + +PR workflows should therefore provide review surfaces in the following priority order: + +1. **Source diff** + The normal git diff remains the primary review surface. + +2. **Bundled artifact** + The fully bundled API for the PR head should be uploaded as an artifact for each affected API. + +3. **Bundled diff** + A diff between the bundled API from the PR base and the bundled API from the PR head should be generated for affected APIs. + +4. **Optional API-aware summary** + A compact semantic summary of changes in paths, operations, schemas, responses, and examples may be added later. + +This gives reviewers the complete effective API view without polluting `main`. + +--- + +## 7. Validation Policy + +Validation should depend on the operational context. + +### 7.1 PR Validation on `main` + +Purpose: + +- validate declared intent +- provide early feedback +- allow merge of planned-but-not-yet-releaseable states + +Behavior: + +- invalid configuration remains blocking +- most release-readiness issues are warnings +- out-of-sync `code/common/CAMARA_common.yaml` is a warning, not a blocker +- warning messages should point to the expected sync PR or manual sync trigger + +Mental model: + +- `release-plan.yaml` defines the target state +- PR validation indicates what still needs to be changed before a release snapshot can be created + +### 7.2 Validation Before Snapshot Creation + +Purpose: + +- ensure the exact source commit intended for release is ready + +Behavior: + +- previously tolerated release-readiness warnings become blockers unless explicitly allowed +- local source references must resolve +- `code/common/CAMARA_common.yaml` must exactly match the declared `commonalities_release` +- APIs intended for release must pass the stricter release profile + +Mental model: + +- the last PR before snapshot creation should be green +- technically, snapshot creation still re-validates the exact source commit + +### 7.3 Snapshot/Release Artifact Validation + +Purpose: + +- validate the bundled release artifact rather than the editable source state + +Behavior: + +- bundled API definitions must validate as standalone artifacts +- release metadata and dependency provenance must be correct +- release branches and tags expose consumer-facing artifacts, not editable source structure + +--- + +## 8. Commonalities Content Expectations + +`CAMARA_common.yaml` must be directly consumable via `$ref` without preprocessing. This requires two changes on the Commonalities side, which can be addressed independently: + +### 8.1 Placeholder Removal + +`CAMARA_common.yaml` currently contains placeholder patterns (`{{SPECIFIC_CODE}}`, `{{field}}`) inside Generic error response schemas and examples. These make the file unparseable as a `$ref` target because the placeholder values are not valid OpenAPI content. + +The recommended approach is to **comment out** placeholder-containing lines rather than delete them: + +- Commented lines are ignored by YAML parsers, bundling tools, and Spectral +- The placeholders remain visible as human-readable documentation, showing API teams where to add API-specific content +- The Generic error responses retain their common error codes (e.g., `INVALID_ARGUMENT`, `NOT_FOUND`) as valid enum values +- The change is minimal — approximately 30 lines across the file + +This makes `CAMARA_common.yaml` immediately consumable without requiring structural refactoring. + +### 8.2 API-Specific Error Code Extension + +APIs need a way to add their own error codes beyond the common ones provided by the Generic responses. This is a separate concern from placeholder removal — it does not need to be solved before `CAMARA_common.yaml` becomes consumable. + +Possible extension mechanisms include `allOf` composition, property overrides, or defining API-specific responses that reference only the shared `ErrorInfo` schema. The right approach may vary by API pattern and can be worked out incrementally as APIs adopt the `$ref` model. + +Example/template API source files should demonstrate the recommended extension pattern once it is established. + +--- + +## 9. Relationship to Release Automation + +The current release automation direction already provides part of the required model: + +- `release-plan.yaml` already carries `dependencies.commonalities_release` +- release-state derivation already exposes that dependency +- release automation already resolves the semantic Commonalities version +- release metadata already records dependency provenance +- source-commit traceability already exists through `src_commit_sha` and `source/rX.Y` reference tags + +The main missing pieces are: + +- synchronization of the local Commonalities cache +- bundling/dereferencing of API source files during snapshot creation +- PR-time generation of bundled review artifacts and bundled diffs + +### 9.1 Release Branch Strategy + +The release process applies in-place replacement (see executive summary, point 5): + +- on `main`, `code/API_definitions/api-name.yaml` is the source file which may contain external `$ref` to `common/` or `modules/` +- on snapshot/release branches, the same file at the same path is replaced with the standalone bundled version +- the `common/` and `modules/` directories are removed from the snapshot branch since their content is now embedded in the bundled files + +This follows the same pattern as the existing release automation transformer, which replaces `info.version` and server URLs in place on the snapshot branch. No separate output directory is created. + +This keeps the familiar filename in all contexts while maintaining a clean source model on `main`. + +--- + +## 10. Bridge Strategy Before Validation Framework v1 + +A broader `validation-framework-v1` is expected to take time. It should eventually provide: + +- one shared validation core +- context-specific profiles +- a common findings model for hints, warnings, and errors +- multiple reporting surfaces +- replacement of MegaLinter-based validation + +Before that is ready, a minimal bridge can still enable the Commonalities consumption model. + +### 10.1 Minimal Bridge Scope + +Short-term work should: + +- keep `pr_validation v0` mostly intact +- introduce narrow pilot support for repositories adopting the local Commonalities model +- add synchronization support for `code/common/CAMARA_common.yaml` +- add strict pre-snapshot validation and bundling into release automation + +This allows early adoption for pilot repositories without waiting for the complete validation rewrite. + +### 10.2 Tooling Ref Consistency + +If pilot repositories use non-`v0` refs of `camaraproject/tooling`, each reusable workflow must be internally consistent with its own tooling ref. + +This is especially relevant for PR validation: + +- changing only the caller ref is not sufficient if the workflow still hardcodes internal calls to `@v0` +- pilot behavior should therefore resolve its internal actions, scripts, and configuration from the same tooling ref + +This matters for cooperation between PR validation and release automation during the pilot phase. + +--- + +## 11. Recommended Phased Plan + +### Phase 1: Decision and Pilot Preparation + +- confirm the controlled local copy model +- confirm source-only `main` plus bundled release artifacts +- align Commonalities on direct consumability of shared content +- select pilot repositories + +### Phase 2: Pilot Implementation + +- implement sync PR generation for `code/common/CAMARA_common.yaml` +- generate bundled artifacts and bundled diffs in PR workflows +- add strict pre-snapshot validation and bundling to release automation + +### Phase 3: Generalization + +- introduce the shared validation core in validation framework v1 +- replace MegaLinter-centered PR validation +- roll the model out across API repositories + +--- + +## 12. Decisions Requested + +1. Confirm the controlled local copy model as the standard way to consume Commonalities schemas. +2. Confirm that bundled API definitions are not committed on `main`. +3. Confirm that PR workflows must provide bundled artifacts and bundled diffs for reviewer visibility. +4. Confirm that an out-of-sync Commonalities cache is a warning on `main`, but a blocker for snapshot creation. +5. Confirm that Commonalities should evolve `CAMARA_common.yaml` toward direct consumability without placeholders. + +--- + +## 13. Expected Follow-Up Artifacts + +The expected follow-up artifacts around this design are: + +- a summary comment in [Commonalities issue #577](https://github.com/camaraproject/Commonalities/issues/577) linking to this design and the ReleaseManagement discussion +- a ReleaseManagement enhancement issue that serves as the umbrella for implementation work +- implementation issues in tooling, Commonalities, and pilot API repositories linked back to that umbrella