Define Tool Compatibility Guidelines for OpenPAKT Implementations

[meisterware-admin]

Overview

Define tool compatibility guidelines for scanners, CI systems, and security tools that produce or consume OpenPAKT reports.

While the OpenPAKT specification defines the data structures and semantics for findings, scenarios, and CI policy evaluation, practical adoption requires guidance for tool implementers.

This proposal introduces non-normative guidelines that help tools integrate with OpenPAKT consistently while preserving interoperability across ecosystems.

The goal is to ensure that tools implementing OpenPAKT behave in predictable ways when producing reports, interpreting findings, and integrating with CI pipelines.

---

Motivation

The OpenPAKT specification provides a minimal interoperable model for representing AI agent security findings, including report schema, severity levels, scenarios, and CI policy semantics.

However, tools implementing OpenPAKT may differ significantly in their architecture and capabilities. For example:

• repository scanners
• runtime agent monitors
• CI policy engines
• security dashboards
• DevSecOps automation tools

Without compatibility guidance, different tools might:

• interpret OpenPAKT fields differently
• emit inconsistent report structures
• implement severity mappings inconsistently
• diverge in CI evaluation behavior

Providing compatibility guidance helps maintain interoperability across implementations while allowing flexibility for different types of tools.

---

Proposed Approach

Introduce a Tool Compatibility Guidelines document that provides recommendations for tools implementing OpenPAKT.

The document should remain non-normative and focus on practical implementation guidance rather than specification requirements.

Possible guideline categories include:

Report generation

Tools producing OpenPAKT reports should:

• generate reports that conform to the defined schema
• emit canonical severity values defined by the specification
• include consistent finding identifiers
• preserve evidence and component references when available

---

Finding normalization

Tools should normalize their internal detection results into the canonical OpenPAKT finding structure before exporting reports.

Example normalized finding fields:

• id
• type
• severity
• component
• description
• evidence

---

Severity mapping

Scanners using vendor-specific severity scales should map them to the canonical OpenPAKT severity levels:

• critical
• high
• medium
• low
• informational

The canonical severity levels are defined in the OpenPAKT severity model to ensure consistent CI evaluation behavior.

---

CI pipeline behavior

Tools integrating with CI pipelines should:

• preserve OpenPAKT findings when exporting to external formats
• maintain deterministic CI policy evaluation behavior
• ensure CI pass/fail results are derived from OpenPAKT severity thresholds

These expectations align with the OpenPAKT CI policy evaluation semantics defined in the specification.

---

Interoperability considerations

Tools implementing OpenPAKT should aim to:

• remain compatible with future specification versions
• avoid introducing incompatible extensions
• maintain stable identifiers for finding types
• preserve metadata required for cross-tool correlation

---

Alternatives Considered

No compatibility guidance

The specification could rely entirely on normative definitions without providing implementation guidance.

However, this approach increases the risk of inconsistent tool behavior across implementations.

---

Strict implementation requirements

Another option would be defining detailed mandatory implementation rules.

However, strict requirements could limit the flexibility needed for different tool architectures.

Providing recommended compatibility guidelines strikes a balance between interoperability and flexibility.

---

Risks and Trade-offs

Over-specification

Providing too many compatibility rules may constrain tool implementers unnecessarily.

The guidelines should remain lightweight and advisory.

---

Fragmentation risk

If tools diverge significantly from the guidelines, interoperability could suffer.

Clear documentation and examples can help encourage consistent implementation.

---

Maintenance overhead

Compatibility guidelines may require updates as the OpenPAKT ecosystem evolves.

However, maintaining guidance is important for long-term adoption.

---

Open Questions

• Should compatibility guidance remain non-normative, or should some recommendations become normative over time?
• Should the guidelines include minimum requirements for compliant tools?
• Should the OpenPAKT project provide reference implementation patterns for common tool types?
• Should compatibility guidance include recommended behavior for future specification version upgrades?

---

Examples

Example workflow of a compatible OpenPAKT toolchain:

Repository
   ↓
Security Scanner
   ↓
OpenPAKT Report
   ↓
CI Policy Evaluation
   ↓
Security Dashboard

In this workflow:

• the scanner produces an OpenPAKT report
• the CI system evaluates findings using the defined policy semantics
• results are displayed in dashboards or security monitoring systems

Each tool in the chain can interoperate because they follow the same OpenPAKT compatibility expectations.

---

Next Steps

If this proposal gains support:

1. Draft a Tool Compatibility Guidelines document for the OpenPAKT repository.
2. Provide practical examples of compliant tool implementations.
3. Document recommended practices for scanners, CI systems, and reporting tools.
4. Validate compatibility guidance against the Detektor reference implementation.
5. Iterate on the guidelines as the OpenPAKT ecosystem grows.