From 95bfa6afdb4cdef6de3b67732db2966e78ce0730 Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 16:37:11 +1300 Subject: [PATCH 01/13] spec: draft v0.1 CI policy evaluation semantics --- spec/ci-policy.md | 177 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 159 insertions(+), 18 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 940b8e0..04bee30 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -1,35 +1,176 @@ Specification: OpenPAKT -Document: CI Policy Semantics +Document: CI Policy Evaluation Semantics Version: v0.1 Status: Draft -# OpenPAKT — CI Policy Semantics +# OpenPAKT — CI Policy Evaluation Semantics -## Overview +## Purpose -This document defines the **CI policy semantics** used to evaluate OpenPAKT security findings within continuous integration pipelines. +This document defines a minimal, deterministic model for evaluating OpenPAKT findings in CI. -CI policies enable automated enforcement of security requirements based on finding severity and taxonomy categories. +The v0.1 model provides a tool-independent way to determine pass/fail outcomes from normalized findings. -## Design Goals +## Scope -CI policy semantics are designed to: +This document defines: -- enable deterministic CI pipeline evaluation -- support consistent enforcement across tools -- remain simple and portable -- allow flexible policy configuration +- a minimal CI policy input shape +- deterministic pass/fail evaluation rules +- deterministic handling for ignored severities and ignored finding types +- severity threshold behavior aligned to the OpenPAKT severity model +- compatibility guidance for CI systems and external reporting formats -## Specification +This document does **not** define: -CI policies operate on OpenPAKT findings and determine whether a build should pass, fail, or report warnings. +- a policy DSL or query language +- scanner normalization logic +- taxonomy or severity definitions (see dedicated specification documents) +- SARIF mapping +- provenance or registry semantics +- implementation-specific workflow logic -Detailed policy evaluation rules will be defined in future revisions. +## Design goals -## Examples +The v0.1 CI policy evaluation semantics are designed to be: -Examples of CI policy evaluation will be included in future revisions of the OpenPAKT specification. +- minimal +- deterministic +- implementation-agnostic +- CI-friendly +- compatible with simple pipeline gate behavior -## Compatibility Considerations +## Normative guidance -CI policy semantics are designed to integrate with common CI systems such as GitHub Actions, GitLab CI, and Azure Pipelines. +- CI policy evaluation **MUST** operate on normalized OpenPAKT findings. +- Evaluators **MUST** use severity ordering from the OpenPAKT severity model: `critical > high > medium > low > informational`. +- Policies **MUST** define `fail_on`. +- Policies **MAY** define `ignore_severities`. +- Policies **MAY** define `ignore_types`. +- Evaluators **MUST** exclude ignored findings from fail/pass evaluation. +- A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. +- A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. +- Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. +- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`), `fail_on`, and matched finding identifiers. + +## Policy input model (v0.1) + +A v0.1 policy input uses three concepts: + +- `fail_on` (required): severity threshold for failing the build +- `ignore_severities` (optional): list of severities to exclude +- `ignore_types` (optional): list of finding `type` values to exclude + +### Example policy input (YAML) + +```yaml +fail_on: high +ignore_severities: + - informational +ignore_types: + - prompt_injection_low_confidence +``` + +## Evaluation model + +Given: + +- a policy `P` +- a normalized findings list `F` + +evaluation proceeds as follows: + +1. Start with all findings in `F`. +2. Remove findings where `severity` is listed in `P.ignore_severities`. +3. Remove findings where `type` is listed in `P.ignore_types`. +4. From the remaining findings, select findings with `severity >= P.fail_on` using OpenPAKT severity ordering. +5. If one or more findings match step 4, decision is `fail`; otherwise decision is `pass`. + +If `ignore_severities` or `ignore_types` are omitted, evaluators **MUST** treat them as empty sets. + +## Deterministic severity threshold behavior + +Severity comparison **MUST** use this strict ranking: + +1. `critical` +2. `high` +3. `medium` +4. `low` +5. `informational` + +For threshold checks, a finding meets `fail_on` when its severity rank is equal to or higher than the threshold rank. + +Examples: + +- with `fail_on: medium`, severities `medium`, `high`, and `critical` meet the threshold +- with `fail_on: high`, only `high` and `critical` meet the threshold + +## Deterministic ignore handling + +Ignore logic applies before threshold comparison. + +A finding is ignored when at least one of the following is true: + +- its `severity` is in `ignore_severities` +- its `type` is in `ignore_types` + +If both ignore lists are present, evaluators **MUST** treat ignore matching as logical OR. + +Ignored findings: + +- **MUST NOT** contribute to threshold matching +- **MAY** be reported as excluded in implementation-specific output +- **MUST NOT** change the normative pass/fail rule + +## Compatibility guidance + +### CI system compatibility + +Implementations in CI systems (for example GitHub Actions, GitLab CI, and Azure Pipelines) **SHOULD** preserve the normative evaluation order and pass/fail rules in this document. + +The CI platform exit status **SHOULD** be derived directly from the policy decision: + +- `pass` -> successful job/stage +- `fail` -> failed job/stage + +### External reporting compatibility + +When exporting results to external reporting formats, producers **SHOULD** preserve: + +- the original policy inputs used for evaluation +- the final decision (`pass`/`fail`) +- the set of matching non-ignored finding identifiers + +Export behavior **MUST NOT** redefine OpenPAKT evaluation semantics. + +## Deterministic examples + +### Example findings (normalized) + +```yaml +findings: + - id: f-001 + type: tool_abuse_privilege_escalation + severity: high + - id: f-002 + type: prompt_injection_low_confidence + severity: medium + - id: f-003 + type: policy_observability_gap + severity: informational +``` + +### Evaluation examples + +| Policy input | Non-ignored findings | Threshold matches | Decision | +|---|---|---|---| +| `fail_on: high` | `f-001`, `f-002`, `f-003` | `f-001` | `fail` | +| `fail_on: high`, `ignore_types: [prompt_injection_low_confidence]` | `f-001`, `f-003` | `f-001` | `fail` | +| `fail_on: critical`, `ignore_severities: [informational]` | `f-001`, `f-002` | none | `pass` | +| `fail_on: medium`, `ignore_severities: [high, medium]` | `f-003` | none | `pass` | + +## Versioning and compatibility notes + +This document defines the minimal CI policy evaluation semantics for OpenPAKT v0.1. + +Future versions may extend policy expressiveness, but v0.1 implementations should treat this evaluation model as the normative baseline for deterministic pass/fail behavior. From 2a66e7ca8b524d34b1c4eaaa5cfe526f34b0c3a6 Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 16:44:40 +1300 Subject: [PATCH 02/13] spec: polish CI policy semantics clarity --- spec/ci-policy.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 04bee30..29c9484 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -11,6 +11,8 @@ This document defines a minimal, deterministic model for evaluating OpenPAKT fin The v0.1 model provides a tool-independent way to determine pass/fail outcomes from normalized findings. +CI policy evaluation operates on findings that conform to the OpenPAKT report schema. + ## Scope This document defines: @@ -44,7 +46,7 @@ The v0.1 CI policy evaluation semantics are designed to be: - CI policy evaluation **MUST** operate on normalized OpenPAKT findings. - Evaluators **MUST** use severity ordering from the OpenPAKT severity model: `critical > high > medium > low > informational`. -- Policies **MUST** define `fail_on`. +- Policies **MUST** define `fail_on`, and the value **MUST** be one of the severity levels defined in the OpenPAKT severity model. - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. - Evaluators **MUST** exclude ignored findings from fail/pass evaluation. @@ -61,6 +63,8 @@ A v0.1 policy input uses three concepts: - `ignore_severities` (optional): list of severities to exclude - `ignore_types` (optional): list of finding `type` values to exclude +Policy keys are case-sensitive and **MUST** appear exactly as defined. + ### Example policy input (YAML) ```yaml @@ -83,7 +87,7 @@ evaluation proceeds as follows: 1. Start with all findings in `F`. 2. Remove findings where `severity` is listed in `P.ignore_severities`. 3. Remove findings where `type` is listed in `P.ignore_types`. -4. From the remaining findings, select findings with `severity >= P.fail_on` using OpenPAKT severity ordering. +4. From the remaining findings, select findings with `severity >= P.fail_on` according to the severity ordering defined in this document. 5. If one or more findings match step 4, decision is `fail`; otherwise decision is `pass`. If `ignore_severities` or `ignore_types` are omitted, evaluators **MUST** treat them as empty sets. From 9961eea01c2a32d920e6ed5515b243887530c66b Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 16:49:12 +1300 Subject: [PATCH 03/13] - small tweak to severity guidance to avoid future reference drift --- spec/ci-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 29c9484..dfd288d 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -45,7 +45,7 @@ The v0.1 CI policy evaluation semantics are designed to be: ## Normative guidance - CI policy evaluation **MUST** operate on normalized OpenPAKT findings. -- Evaluators **MUST** use severity ordering from the OpenPAKT severity model: `critical > high > medium > low > informational`. +- Evaluators **MUST** apply the severity ordering defined in the OpenPAKT severity model and referenced in this document. - Policies **MUST** define `fail_on`, and the value **MUST** be one of the severity levels defined in the OpenPAKT severity model. - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. From 40bd644f952e25575d23af705fc20f46c56eacaa Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 17:18:47 +1300 Subject: [PATCH 04/13] =?UTF-8?q?-=20since=20our=20last=20push=20will=20pr?= =?UTF-8?q?oduce=20merge=20conflicts=20with=20Codex,=20I=20manually=20upda?= =?UTF-8?q?te=20locally.=20-=20fixed=20the=20CI=20policy=20examples=20to?= =?UTF-8?q?=20use=20canonical=20taxonomy=20identifiers=20in=20normalized?= =?UTF-8?q?=20findings=20and=20ignore=5Ftypes=20usage=20(prompt=5Finjectio?= =?UTF-8?q?n,=20sensitive=5Fdata=5Fexposure),=20resolving=20the=20non-conf?= =?UTF-8?q?ormant=20type=20ID=20issue=20while=20preserving=20evaluation=20?= =?UTF-8?q?behavior=20and=20structure.=20(Codex)=20-=20kept=20the=20policy?= =?UTF-8?q?=20model=20unchanged=20but=20aligned=20the=20example=20key=20us?= =?UTF-8?q?age=20in=20the=20severity=20document=20from=20fail-on=20to=20fa?= =?UTF-8?q?il=5Fon,=20removing=20the=20cross-spec=20naming=20inconsistency?= =?UTF-8?q?=20with=20the=20CI=20policy=20document=E2=80=99s=20strict=20key?= =?UTF-8?q?=20rule.=20(Codex)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- spec/ci-policy.md | 8 ++++---- spec/severity.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index dfd288d..eb0698e 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -72,7 +72,7 @@ fail_on: high ignore_severities: - informational ignore_types: - - prompt_injection_low_confidence + - prompt_injection ``` ## Evaluation model @@ -157,10 +157,10 @@ findings: type: tool_abuse_privilege_escalation severity: high - id: f-002 - type: prompt_injection_low_confidence + type: prompt_injection severity: medium - id: f-003 - type: policy_observability_gap + type: sensitive_data_exposure severity: informational ``` @@ -169,7 +169,7 @@ findings: | Policy input | Non-ignored findings | Threshold matches | Decision | |---|---|---|---| | `fail_on: high` | `f-001`, `f-002`, `f-003` | `f-001` | `fail` | -| `fail_on: high`, `ignore_types: [prompt_injection_low_confidence]` | `f-001`, `f-003` | `f-001` | `fail` | +| `fail_on: high`, `ignore_types: [prompt_injection]` | `f-001`, `f-003` | `f-001` | `fail` | | `fail_on: critical`, `ignore_severities: [informational]` | `f-001`, `f-002` | none | `pass` | | `fail_on: medium`, `ignore_severities: [high, medium]` | `f-003` | none | `pass` | diff --git a/spec/severity.md b/spec/severity.md index 523fbdc..e4ce1bd 100644 --- a/spec/severity.md +++ b/spec/severity.md @@ -88,10 +88,10 @@ evidence: ### CI threshold style example ```txt -fail-on: high +fail_on: high ``` -Expected deterministic behaviour for `fail-on: high`: +Expected deterministic behaviour for `fail_on: high`: - `critical` -> fail build - `high` -> fail build From 7b9019a0184d235c111f7cf7f125379bdcc46d61 Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 18:25:24 +1300 Subject: [PATCH 05/13] =?UTF-8?q?-=20added=20normative=20constraints=20for?= =?UTF-8?q?=20ignore=20lists=20so=20ignore=5Fseverities=20must=20use=20can?= =?UTF-8?q?onical=20severity=20values=20and=20ignore=5Ftypes=20must=20use?= =?UTF-8?q?=20canonical=20taxonomy=20identifiers,=20and=20specified=20dete?= =?UTF-8?q?rministic=20invalid-input=20handling=20(unsupported=20values=20?= =?UTF-8?q?make=20the=20policy=20invalid=20and=20evaluation=20must=20stop)?= =?UTF-8?q?.=20-=20kept=20the=20policy=20model=20unchanged=20but=20clarifi?= =?UTF-8?q?ed=20key/value=20precision=20in=20the=20policy=20input=20sectio?= =?UTF-8?q?n=20by=20explicitly=20requiring=20canonical=20identifiers=20for?= =?UTF-8?q?=20ignore-list=20values=20when=20present.=20-=20resolved=20thre?= =?UTF-8?q?shold=20comparison=20ambiguity=20by=20replacing=20rank-directio?= =?UTF-8?q?n=20phrasing=20with=20explicit=20ordered-list=20semantics=20(?= =?UTF-8?q?=E2=80=9Csame=20as=20threshold=20or=20to=20the=20left=20in=20th?= =?UTF-8?q?e=20defined=20order=E2=80=9D).?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- spec/ci-policy.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index eb0698e..4736398 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -49,6 +49,9 @@ The v0.1 CI policy evaluation semantics are designed to be: - Policies **MUST** define `fail_on`, and the value **MUST** be one of the severity levels defined in the OpenPAKT severity model. - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. +- If present, `ignore_severities` values **MUST** be severity levels defined in the OpenPAKT severity model. +- If present, `ignore_types` values **MUST** be canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification. +- Evaluators **MUST** treat policies containing unsupported `ignore_severities` or `ignore_types` values as invalid input and **MUST NOT** continue evaluation with those values. - Evaluators **MUST** exclude ignored findings from fail/pass evaluation. - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. @@ -65,6 +68,8 @@ A v0.1 policy input uses three concepts: Policy keys are case-sensitive and **MUST** appear exactly as defined. +If present, `ignore_severities` and `ignore_types` values **MUST** use canonical identifiers defined by the severity and taxonomy specifications. + ### Example policy input (YAML) ```yaml @@ -102,7 +107,7 @@ Severity comparison **MUST** use this strict ranking: 4. `low` 5. `informational` -For threshold checks, a finding meets `fail_on` when its severity rank is equal to or higher than the threshold rank. +For threshold checks, a finding meets `fail_on` when its severity is the same as the threshold or appears to the left of the threshold in the ordered list above. Examples: From 793c1b73e7f01f615cb6cd4d868f997100ff51da Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 18:45:41 +1300 Subject: [PATCH 06/13] - updated the ambiguous invalid-ignore normative rule to require one deterministic outcome: if unsupported ignore_severities or ignore_types values are present, evaluators must stop with an invalid-policy result and produce no pass/fail decision. --- spec/ci-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 4736398..43b6582 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -51,7 +51,7 @@ The v0.1 CI policy evaluation semantics are designed to be: - Policies **MAY** define `ignore_types`. - If present, `ignore_severities` values **MUST** be severity levels defined in the OpenPAKT severity model. - If present, `ignore_types` values **MUST** be canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification. -- Evaluators **MUST** treat policies containing unsupported `ignore_severities` or `ignore_types` values as invalid input and **MUST NOT** continue evaluation with those values. +- Evaluators **MUST** treat policies containing unsupported `ignore_severities` or `ignore_types` values as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). - Evaluators **MUST** exclude ignored findings from fail/pass evaluation. - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. From cc7927c0e47732ff78711d2a46bd6841ea3b5c05 Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 18:57:01 +1300 Subject: [PATCH 07/13] =?UTF-8?q?-=20added=20a=20deterministic=20normative?= =?UTF-8?q?=20rule=20that=20missing=20or=20unsupported=20fail=5Fon=20must?= =?UTF-8?q?=20produce=20an=20invalid-policy=20result=20and=20stop=20evalua?= =?UTF-8?q?tion=20(no=20pass/fail=20decision),=20resolving=20the=20reviewe?= =?UTF-8?q?r=E2=80=99s=20ambiguity=20concern=20without=20changing=20the=20?= =?UTF-8?q?policy=20model=20or=20document=20structure.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- spec/ci-policy.md | 1 + 1 file changed, 1 insertion(+) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 43b6582..9514a3d 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -47,6 +47,7 @@ The v0.1 CI policy evaluation semantics are designed to be: - CI policy evaluation **MUST** operate on normalized OpenPAKT findings. - Evaluators **MUST** apply the severity ordering defined in the OpenPAKT severity model and referenced in this document. - Policies **MUST** define `fail_on`, and the value **MUST** be one of the severity levels defined in the OpenPAKT severity model. +- Evaluators **MUST** treat policies with a missing `fail_on` key or unsupported `fail_on` value as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. - If present, `ignore_severities` values **MUST** be severity levels defined in the OpenPAKT severity model. From 11d84fd11f30449f0de7da8cbdd852c82a189608 Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 19:11:46 +1300 Subject: [PATCH 08/13] - added explicit support for invalid-policy in the machine-readable decision guidance so evaluation outputs consistently represent malformed policy outcomes alongside pass and fail. - defined deterministic CI mapping for invalid-policy in the CI compatibility section by requiring exit-status mapping for all decisions, including invalid-policy -> failed job/stage. This closes the prior ambiguity noted in review. - updated external reporting compatibility guidance to preserve invalid-policy as a valid final decision value in exported outputs. --- spec/ci-policy.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 9514a3d..35699ae 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -57,7 +57,7 @@ The v0.1 CI policy evaluation semantics are designed to be: - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. - Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. -- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`), `fail_on`, and matched finding identifiers. +- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`), `fail_on`, and matched finding identifiers. ## Policy input model (v0.1) @@ -138,17 +138,18 @@ Ignored findings: Implementations in CI systems (for example GitHub Actions, GitLab CI, and Azure Pipelines) **SHOULD** preserve the normative evaluation order and pass/fail rules in this document. -The CI platform exit status **SHOULD** be derived directly from the policy decision: +The CI platform exit status **MUST** be derived directly from the policy decision: - `pass` -> successful job/stage - `fail` -> failed job/stage +- `invalid-policy` -> failed job/stage ### External reporting compatibility When exporting results to external reporting formats, producers **SHOULD** preserve: - the original policy inputs used for evaluation -- the final decision (`pass`/`fail`) +- the final decision (`pass`/`fail`/`invalid-policy`) - the set of matching non-ignored finding identifiers Export behavior **MUST NOT** redefine OpenPAKT evaluation semantics. From 1dd628dfc615dfc84bae5fb3a594619ff62b02ce Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 19:32:28 +1300 Subject: [PATCH 09/13] - added explicit normative handling for unknown policy keys: v0.1 policies must not include unknown top-level keys, and such policies are now explicitly invalid-policy inputs that must stop evaluation deterministically. - added MUST-level type constraints for ignore fields: if present, ignore_severities and ignore_types must be arrays of strings, with canonical-value requirements retained, and non-array/unsupported cases now explicitly map to invalid-policy behavior. - added deterministic guidance for matched finding identifier output: when duplicate IDs exist, evaluators must preserve duplicates in original finding order in machine-readable results. - aligned policy input model prose with the normative rules by explicitly stating unknown-key invalidity and array-of-string requirements for ignore fields. --- spec/ci-policy.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 35699ae..5f450ac 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -50,14 +50,16 @@ The v0.1 CI policy evaluation semantics are designed to be: - Evaluators **MUST** treat policies with a missing `fail_on` key or unsupported `fail_on` value as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. -- If present, `ignore_severities` values **MUST** be severity levels defined in the OpenPAKT severity model. -- If present, `ignore_types` values **MUST** be canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification. -- Evaluators **MUST** treat policies containing unsupported `ignore_severities` or `ignore_types` values as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). +- Policies **MUST NOT** include unknown top-level policy keys. +- If present, `ignore_severities` **MUST** be an array of strings, and values **MUST** be severity levels defined in the OpenPAKT severity model. +- If present, `ignore_types` **MUST** be an array of strings, and values **MUST** be canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification. +- Evaluators **MUST** treat policies with unknown top-level keys, non-array `ignore_severities`/`ignore_types`, or unsupported `ignore_severities`/`ignore_types` values as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). - Evaluators **MUST** exclude ignored findings from fail/pass evaluation. - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. - Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. - Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`), `fail_on`, and matched finding identifiers. +- If matched finding identifiers contain duplicates, evaluators **MUST** preserve duplicates in the original finding order in the machine-readable result. ## Policy input model (v0.1) @@ -67,9 +69,9 @@ A v0.1 policy input uses three concepts: - `ignore_severities` (optional): list of severities to exclude - `ignore_types` (optional): list of finding `type` values to exclude -Policy keys are case-sensitive and **MUST** appear exactly as defined. +Policy keys are case-sensitive and **MUST** appear exactly as defined. Unknown top-level keys are invalid in v0.1 policies. -If present, `ignore_severities` and `ignore_types` values **MUST** use canonical identifiers defined by the severity and taxonomy specifications. +If present, `ignore_severities` and `ignore_types` **MUST** be arrays of strings and values **MUST** use canonical identifiers defined by the severity and taxonomy specifications. ### Example policy input (YAML) From a61b60b95f6bf104f05d44025553b7e68ab31155 Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 19:48:28 +1300 Subject: [PATCH 10/13] =?UTF-8?q?-=20added=20explicit=20invalid-policy=20o?= =?UTF-8?q?utput-shape=20semantics=20for=20machine-readable=20results:=20w?= =?UTF-8?q?hen=20decision=20is=20invalid-policy,=20fail=5Fon=20must=20be?= =?UTF-8?q?=20null=20and=20matched=20finding=20identifiers=20must=20be=20a?= =?UTF-8?q?n=20empty=20array,=20removing=20ambiguity=20for=20malformed-pol?= =?UTF-8?q?icy=20outputs.=20-=20kept=20and=20aligned=20duplicate-ID=20beha?= =?UTF-8?q?vior=20by=20preserving=20duplicates=20in=20original=20finding?= =?UTF-8?q?=20order,=20then=20updated=20external=20reporting=20wording=20f?= =?UTF-8?q?rom=20=E2=80=9Cset=E2=80=9D=20semantics=20to=20an=20=E2=80=9Cor?= =?UTF-8?q?dered=20list=E2=80=9D=20that=20explicitly=20preserves=20duplica?= =?UTF-8?q?tes,=20eliminating=20the=20conflict=20noted=20in=20review.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- spec/ci-policy.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 5f450ac..5a4988d 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -60,6 +60,7 @@ The v0.1 CI policy evaluation semantics are designed to be: - Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. - Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`), `fail_on`, and matched finding identifiers. - If matched finding identifiers contain duplicates, evaluators **MUST** preserve duplicates in the original finding order in the machine-readable result. +- For `invalid-policy` decisions, machine-readable results **MUST** set `fail_on` to `null` and matched finding identifiers to an empty array. ## Policy input model (v0.1) @@ -152,7 +153,7 @@ When exporting results to external reporting formats, producers **SHOULD** prese - the original policy inputs used for evaluation - the final decision (`pass`/`fail`/`invalid-policy`) -- the set of matching non-ignored finding identifiers +- the ordered list of matching non-ignored finding identifiers (preserving duplicates in original finding order) Export behavior **MUST NOT** redefine OpenPAKT evaluation semantics. From 0a121848949813a4846aad168b68911294cfa4fe Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 20:02:43 +1300 Subject: [PATCH 11/13] - added deterministic handling for malformed/non-normalized findings by requiring evaluators to stop with invalid-policy when findings input is malformed (e.g., missing required fields or unsupported severity/type values), preventing divergent skip/coerce behaviors. - relaxed unknown-key strict rejection to improve interoperability: unknown top-level policy keys are now explicitly allowed and must be ignored, while keeping strict type validation for ignore fields (non-array remains invalid-policy). - refined ignore entry handling: ignore_severities/ignore_types must be arrays of strings, and unsupported entries inside those arrays must be ignored instead of invalidating the whole policy, reducing brittle cross-version failures. - strengthened matched-ID determinism to require original finding-order emission for all matched identifiers (not only duplicates), with duplicate preservation still explicit. --- spec/ci-policy.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 5a4988d..ca0439c 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -45,21 +45,22 @@ The v0.1 CI policy evaluation semantics are designed to be: ## Normative guidance - CI policy evaluation **MUST** operate on normalized OpenPAKT findings. +- If input findings are malformed or not normalized (for example missing required finding fields or unsupported severity/type values), evaluators **MUST** stop evaluation with an `invalid-policy` result (no pass/fail decision is produced). - Evaluators **MUST** apply the severity ordering defined in the OpenPAKT severity model and referenced in this document. - Policies **MUST** define `fail_on`, and the value **MUST** be one of the severity levels defined in the OpenPAKT severity model. - Evaluators **MUST** treat policies with a missing `fail_on` key or unsupported `fail_on` value as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. -- Policies **MUST NOT** include unknown top-level policy keys. -- If present, `ignore_severities` **MUST** be an array of strings, and values **MUST** be severity levels defined in the OpenPAKT severity model. -- If present, `ignore_types` **MUST** be an array of strings, and values **MUST** be canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification. -- Evaluators **MUST** treat policies with unknown top-level keys, non-array `ignore_severities`/`ignore_types`, or unsupported `ignore_severities`/`ignore_types` values as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). +- Evaluators **MUST** ignore unknown top-level policy keys. +- If present, `ignore_severities` **MUST** be an array of strings; entries that are not severity levels defined in the OpenPAKT severity model **MUST** be ignored. +- If present, `ignore_types` **MUST** be an array of strings; entries that are not canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification **MUST** be ignored. +- Evaluators **MUST** treat non-array `ignore_severities`/`ignore_types` values as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). - Evaluators **MUST** exclude ignored findings from fail/pass evaluation. - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. - Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. - Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`), `fail_on`, and matched finding identifiers. -- If matched finding identifiers contain duplicates, evaluators **MUST** preserve duplicates in the original finding order in the machine-readable result. +- Evaluators **MUST** emit matched finding identifiers in the original finding order from the evaluated findings list and **MUST** preserve duplicates. - For `invalid-policy` decisions, machine-readable results **MUST** set `fail_on` to `null` and matched finding identifiers to an empty array. ## Policy input model (v0.1) @@ -70,9 +71,9 @@ A v0.1 policy input uses three concepts: - `ignore_severities` (optional): list of severities to exclude - `ignore_types` (optional): list of finding `type` values to exclude -Policy keys are case-sensitive and **MUST** appear exactly as defined. Unknown top-level keys are invalid in v0.1 policies. +Policy keys are case-sensitive and **MUST** appear exactly as defined. Unknown top-level keys are allowed and **MUST** be ignored. -If present, `ignore_severities` and `ignore_types` **MUST** be arrays of strings and values **MUST** use canonical identifiers defined by the severity and taxonomy specifications. +If present, `ignore_severities` and `ignore_types` **MUST** be arrays of strings. Entries that do not use canonical identifiers defined by the severity and taxonomy specifications **MUST** be ignored. ### Example policy input (YAML) From 46103b1d381be06f7c5d640a226133adca90b42b Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 20:37:41 +1300 Subject: [PATCH 12/13] - clarify CI policy evaluation input source as normalized findings from report.findings (or equivalent extracted normalized findings list). - explicitly scope CI policy evaluation to findings only, and state that scenario definitions/execution outcomes are not directly evaluated. - resolve invalid-result ambiguity by separating: - invalid-policy for invalid policy input - invalid-findings for malformed or non-normalized findings input - update normative evaluation flow to validate policy first, then findings, before threshold evaluation. - extend machine-readable result guidance to include invalid-findings, with deterministic field behavior for both invalid outcomes. - add optional implementation-specific output guidance for ignored finding identifiers and exclusion reasons, without affecting normative pass/fail behavior. - add a minimal deterministic invalid example showing expected invalid-findings output. - preserve v0.1 minimal scope, canonical severity ordering, snake_case field naming, and taxonomy-aligned finding types. --- spec/ci-policy.md | 50 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index ca0439c..4dc989c 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -13,6 +13,10 @@ The v0.1 model provides a tool-independent way to determine pass/fail outcomes f CI policy evaluation operates on findings that conform to the OpenPAKT report schema. +CI evaluation input is the normalized findings array from an OpenPAKT report (`report.findings`) or an equivalent extracted normalized findings list. + +OpenPAKT v0.1 CI policy evaluation applies to normalized findings and does **not** directly evaluate scenario definitions or scenario execution outcomes. + ## Scope This document defines: @@ -45,23 +49,24 @@ The v0.1 CI policy evaluation semantics are designed to be: ## Normative guidance - CI policy evaluation **MUST** operate on normalized OpenPAKT findings. -- If input findings are malformed or not normalized (for example missing required finding fields or unsupported severity/type values), evaluators **MUST** stop evaluation with an `invalid-policy` result (no pass/fail decision is produced). - Evaluators **MUST** apply the severity ordering defined in the OpenPAKT severity model and referenced in this document. - Policies **MUST** define `fail_on`, and the value **MUST** be one of the severity levels defined in the OpenPAKT severity model. -- Evaluators **MUST** treat policies with a missing `fail_on` key or unsupported `fail_on` value as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). +- Evaluators **MUST** treat policies with a missing `fail_on` key or unsupported `fail_on` value as invalid input and **MUST** stop evaluation with an `invalid-policy` result (no pass/fail decision is produced). - Policies **MAY** define `ignore_severities`. - Policies **MAY** define `ignore_types`. - Evaluators **MUST** ignore unknown top-level policy keys. - If present, `ignore_severities` **MUST** be an array of strings; entries that are not severity levels defined in the OpenPAKT severity model **MUST** be ignored. - If present, `ignore_types` **MUST** be an array of strings; entries that are not canonical taxonomy identifiers defined in the OpenPAKT taxonomy specification **MUST** be ignored. -- Evaluators **MUST** treat non-array `ignore_severities`/`ignore_types` values as invalid input and **MUST** stop evaluation with an invalid-policy result (no pass/fail decision is produced). +- Evaluators **MUST** treat non-array `ignore_severities`/`ignore_types` values as invalid policy input and **MUST** stop evaluation with an `invalid-policy` result (no pass/fail decision is produced). +- If evaluated findings input is malformed or not normalized (for example missing required finding fields or unsupported severity/type values), evaluators **MUST** stop evaluation with an `invalid-findings` result (no pass/fail decision is produced). - Evaluators **MUST** exclude ignored findings from fail/pass evaluation. - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. - Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. -- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`), `fail_on`, and matched finding identifiers. +- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`/`invalid-findings`), `fail_on`, and matched finding identifiers. - Evaluators **MUST** emit matched finding identifiers in the original finding order from the evaluated findings list and **MUST** preserve duplicates. - For `invalid-policy` decisions, machine-readable results **MUST** set `fail_on` to `null` and matched finding identifiers to an empty array. +- For `invalid-findings` decisions, machine-readable results **MUST** set `fail_on` to the validated policy threshold and matched finding identifiers to an empty array. ## Policy input model (v0.1) @@ -90,15 +95,17 @@ ignore_types: Given: - a policy `P` -- a normalized findings list `F` +- a findings list `F` sourced from `report.findings` or an equivalent extracted normalized findings list evaluation proceeds as follows: -1. Start with all findings in `F`. -2. Remove findings where `severity` is listed in `P.ignore_severities`. -3. Remove findings where `type` is listed in `P.ignore_types`. -4. From the remaining findings, select findings with `severity >= P.fail_on` according to the severity ordering defined in this document. -5. If one or more findings match step 4, decision is `fail`; otherwise decision is `pass`. +1. Validate `P` according to this document. If invalid, decision is `invalid-policy` and evaluation stops. +2. Validate `F` as normalized OpenPAKT findings. If invalid, decision is `invalid-findings` and evaluation stops. +3. Start with all findings in `F`. +4. Remove findings where `severity` is listed in `P.ignore_severities`. +5. Remove findings where `type` is listed in `P.ignore_types`. +6. From the remaining findings, select findings with `severity >= P.fail_on` according to the severity ordering defined in this document. +7. If one or more findings match step 6, decision is `fail`; otherwise decision is `pass`. If `ignore_severities` or `ignore_types` are omitted, evaluators **MUST** treat them as empty sets. @@ -134,6 +141,7 @@ Ignored findings: - **MUST NOT** contribute to threshold matching - **MAY** be reported as excluded in implementation-specific output +- **MAY** include ignored finding identifiers and exclusion reasons in implementation-specific output - **MUST NOT** change the normative pass/fail rule ## Compatibility guidance @@ -147,13 +155,14 @@ The CI platform exit status **MUST** be derived directly from the policy decisio - `pass` -> successful job/stage - `fail` -> failed job/stage - `invalid-policy` -> failed job/stage +- `invalid-findings` -> failed job/stage ### External reporting compatibility When exporting results to external reporting formats, producers **SHOULD** preserve: - the original policy inputs used for evaluation -- the final decision (`pass`/`fail`/`invalid-policy`) +- the final decision (`pass`/`fail`/`invalid-policy`/`invalid-findings`) - the ordered list of matching non-ignored finding identifiers (preserving duplicates in original finding order) Export behavior **MUST NOT** redefine OpenPAKT evaluation semantics. @@ -184,6 +193,25 @@ findings: | `fail_on: critical`, `ignore_severities: [informational]` | `f-001`, `f-002` | none | `pass` | | `fail_on: medium`, `ignore_severities: [high, medium]` | `f-003` | none | `pass` | +### Invalid input example + +```yaml +policy: + fail_on: high +findings: + - id: f-001 + type: tool_abuse_privilege_escalation + severity: severe +``` + +Expected machine-readable result: + +```yaml +decision: invalid-findings +fail_on: high +matched_finding_ids: [] +``` + ## Versioning and compatibility notes This document defines the minimal CI policy evaluation semantics for OpenPAKT v0.1. From d569920859af731a5493bc13650e878701ae748b Mon Sep 17 00:00:00 2001 From: Meisterware Admin Date: Sun, 15 Mar 2026 20:50:07 +1300 Subject: [PATCH 13/13] - introduce a full CI policy semantics document for OpenPAKT v0.1 with deterministic pass/fail behavior on normalized findings. - add normative validation rules for policy input and findings input, including explicit invalid-policy and invalid-findings outcomes. - define deterministic evaluation steps: validate inputs, apply ignore filters, compare severities against fail_on, emit final decision. - standardize machine-readable output requirements with decision, fail_on, and matched_finding_ids (ordered, duplicates preserved). - update severity doc threshold example to use fail_on for cross-spec consistency. --- spec/ci-policy.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/spec/ci-policy.md b/spec/ci-policy.md index 4dc989c..bfe620f 100644 --- a/spec/ci-policy.md +++ b/spec/ci-policy.md @@ -63,10 +63,10 @@ The v0.1 CI policy evaluation semantics are designed to be: - A build **MUST** fail if at least one non-ignored finding has severity at or above `fail_on`. - A build **MUST** pass if no non-ignored finding has severity at or above `fail_on`. - Evaluators **MUST NOT** use tool-specific extensions to alter the normative pass/fail outcome. -- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`/`invalid-findings`), `fail_on`, and matched finding identifiers. -- Evaluators **MUST** emit matched finding identifiers in the original finding order from the evaluated findings list and **MUST** preserve duplicates. -- For `invalid-policy` decisions, machine-readable results **MUST** set `fail_on` to `null` and matched finding identifiers to an empty array. -- For `invalid-findings` decisions, machine-readable results **MUST** set `fail_on` to the validated policy threshold and matched finding identifiers to an empty array. +- Evaluators **SHOULD** return a machine-readable evaluation result that includes at least: decision (`pass`/`fail`/`invalid-policy`/`invalid-findings`), `fail_on`, and `matched_finding_ids`. +- Evaluators **MUST** emit `matched_finding_ids` in the original finding order from the evaluated findings list and **MUST** preserve duplicates. +- For `invalid-policy` decisions, machine-readable results **MUST** set `fail_on` to `null` and `matched_finding_ids` to an empty array. +- For `invalid-findings` decisions, machine-readable results **MUST** set `fail_on` to the validated policy threshold and `matched_finding_ids` to an empty array. ## Policy input model (v0.1) @@ -163,7 +163,7 @@ When exporting results to external reporting formats, producers **SHOULD** prese - the original policy inputs used for evaluation - the final decision (`pass`/`fail`/`invalid-policy`/`invalid-findings`) -- the ordered list of matching non-ignored finding identifiers (preserving duplicates in original finding order) +- `matched_finding_ids` as the ordered list of matching non-ignored finding identifiers (preserving duplicates in original finding order) Export behavior **MUST NOT** redefine OpenPAKT evaluation semantics. @@ -196,8 +196,6 @@ findings: ### Invalid input example ```yaml -policy: - fail_on: high findings: - id: f-001 type: tool_abuse_privilege_escalation