Define OpenPAKT CI policy evaluation semantics v0.1#22
Conversation
meisterware-admin
changed the title
meisterware-admin
added
spec
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9961eea01c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ally update locally. - fixed the CI policy examples to use canonical taxonomy identifiers in normalized findings and ignore_types usage (prompt_injection, sensitive_data_exposure), resolving the non-conformant type ID issue while preserving evaluation behavior and structure. (Codex) - kept the policy model unchanged but aligned the example key usage in the severity document from fail-on to fail_on, removing the cross-spec naming inconsistency with the CI policy document’s strict key rule. (Codex)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40bd644f95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ust use canonical severity values and ignore_types must use canonical taxonomy identifiers, and specified deterministic invalid-input handling (unsupported values make the policy invalid and evaluation must stop). - kept the policy model unchanged but clarified key/value precision in the policy input section by explicitly requiring canonical identifiers for ignore-list values when present. - resolved threshold comparison ambiguity by replacing rank-direction phrasing with explicit ordered-list semantics (“same as threshold or to the left in the defined order”).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b9019a018
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…deterministic outcome: if unsupported ignore_severities or ignore_types values are present, evaluators must stop with an invalid-policy result and produce no pass/fail decision.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 793c1b73e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…il_on must produce an invalid-policy result and stop evaluation (no pass/fail decision), resolving the reviewer’s ambiguity concern without changing the policy model or document structure.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc7927c0e4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ecision guidance so evaluation outputs consistently represent malformed policy outcomes alongside pass and fail. - defined deterministic CI mapping for invalid-policy in the CI compatibility section by requiring exit-status mapping for all decisions, including invalid-policy -> failed job/stage. This closes the prior ambiguity noted in review. - updated external reporting compatibility guidance to preserve invalid-policy as a valid final decision value in exported outputs.
|
@codex review the entire repository for specification consistency.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11d84fd11f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…icies must not include unknown top-level keys, and such policies are now explicitly invalid-policy inputs that must stop evaluation deterministically. - added MUST-level type constraints for ignore fields: if present, ignore_severities and ignore_types must be arrays of strings, with canonical-value requirements retained, and non-array/unsupported cases now explicitly map to invalid-policy behavior. - added deterministic guidance for matched finding identifier output: when duplicate IDs exist, evaluators must preserve duplicates in original finding order in machine-readable results. - aligned policy input model prose with the normative rules by explicitly stating unknown-key invalidity and array-of-string requirements for ignore fields.
|
@codex review the entire repository for specification consistency. Focus on:
Return a full summary of all issues before inline comments. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1dd628dfc6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
I tried running "codex review" but looks like it could only run using your account @meisterware-admin |
😲 |
…adable results: when decision is invalid-policy, fail_on must be null and matched finding identifiers must be an empty array, removing ambiguity for malformed-policy outputs. - kept and aligned duplicate-ID behavior by preserving duplicates in original finding order, then updated external reporting wording from “set” semantics to an “ordered list” that explicitly preserves duplicates, eliminating the conflict noted in review.
|
@codex review the entire repository for specification consistency. Focus on:
Return a full summary of all issues before inline comments. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a61b60b95f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@meisterware-admin looks like this is going around n circles? do you want to ask Chat for guidance after this? |
|
yeah codex fixes one then tells another thing after rather than fixing everything at once. I will have chatgpt review after this... |
…by requiring evaluators to stop with invalid-policy when findings input is malformed (e.g., missing required fields or unsupported severity/type values), preventing divergent skip/coerce behaviors. - relaxed unknown-key strict rejection to improve interoperability: unknown top-level policy keys are now explicitly allowed and must be ignored, while keeping strict type validation for ignore fields (non-array remains invalid-policy). - refined ignore entry handling: ignore_severities/ignore_types must be arrays of strings, and unsupported entries inside those arrays must be ignored instead of invalidating the whole policy, reducing brittle cross-version failures. - strengthened matched-ID determinism to require original finding-order emission for all matched identifiers (not only duplicates), with duplicate preservation still explicit.
…om report.findings (or equivalent extracted normalized findings list). - explicitly scope CI policy evaluation to findings only, and state that scenario definitions/execution outcomes are not directly evaluated. - resolve invalid-result ambiguity by separating: - invalid-policy for invalid policy input - invalid-findings for malformed or non-normalized findings input - update normative evaluation flow to validate policy first, then findings, before threshold evaluation. - extend machine-readable result guidance to include invalid-findings, with deterministic field behavior for both invalid outcomes. - add optional implementation-specific output guidance for ignored finding identifiers and exclusion reasons, without affecting normative pass/fail behavior. - add a minimal deterministic invalid example showing expected invalid-findings output. - preserve v0.1 minimal scope, canonical severity ordering, snake_case field naming, and taxonomy-aligned finding types.
…h deterministic pass/fail behavior on normalized findings. - add normative validation rules for policy input and findings input, including explicit invalid-policy and invalid-findings outcomes. - define deterministic evaluation steps: validate inputs, apply ignore filters, compare severities against fail_on, emit final decision. - standardize machine-readable output requirements with decision, fail_on, and matched_finding_ids (ordered, duplicates preserved). - update severity doc threshold example to use fail_on for cross-spec consistency.
meisterware-admin
deleted the
spec/draft-ci-policy-evaluation-semantics-specification
branch
2 participants
Summary
This pull request adds the draft OpenPAKT v0.1 specification for CI policy evaluation semantics.
The document defines a minimal, deterministic, and tool-independent model for evaluating OpenPAKT findings in CI pipelines. It focuses on how normalized findings are interpreted for pass/fail decisions using stable severity ordering and a small set of policy concepts, without introducing unnecessary DSL complexity or implementation-specific workflow logic.
Type of change
Select all that apply:
Specification classification (if applicable)
Related issue
Link the related issue number.
Closes #5
Specification changes should normally be linked to a Proposal or Specification Change issue.
What changed
ci-policyspecification for OpenPAKT v0.1fail_onImpact
This change completes another core OpenPAKT v0.1 specification component and makes the CI evaluation model explicit and portable across tools and pipelines.
It improves specification clarity for scanner authors and CI implementers, supports deterministic build gating, and strengthens interoperability between OpenPAKT findings and CI policy engines.
Compatibility
Choose one:
If "Breaking change", explain migration considerations.
Checklist
Notes for reviewers
Reviewers should focus on whether the policy semantics remain intentionally minimal for v0.1 while still being deterministic enough for CI enforcement.
Particular attention should be given to: