Design Discussion: Auditor Layer (Source Validation, Rule Gates, Dry-Run Parity)

[PrzemyslawKlys]

Context

As IntelligenceX evolves from read-first workflows toward broader automation, we should formalize a guardrail layer that is explicit, testable, and enforceable across connectors/tools.

Today we already have useful primitives (tool scoping, policy constraints, logs, and read-first defaults), but they are not yet packaged as a single auditor/control layer with deterministic behavior.

Problem

Without a formal auditor pipeline, we risk inconsistent behavior across actions/connectors:

• uneven source grounding requirements
• different validation depth per action
• drift in reasoning/output quality over long tasks
• partial or connector-specific dry-run behavior

Goal

Define an auditor architecture that provides:

1. Rule packs (allowed/required/prohibited patterns by scope/env)
2. Pre-execution gate (policy + intent + target validation)
3. Post-execution gate (result validation + consistency checks)
4. Source-attribution checks for knowledge-sensitive outputs
5. Dry-run parity for all write-capable actions
6. Clear fail/flag/escalate semantics

Option Space

Option A: Lightweight in-process validator

• Pros: fast to ship, low complexity
• Cons: hard to scale, weaker isolation

Option B: Middleware policy engine (recommended MVP+)

• Pros: central enforcement, reusable across tools, clearer telemetry
• Cons: moderate implementation effort

Option C: External policy decision point (OPA-like)

• Pros: strongest governance/compliance model
• Cons: highest complexity, operational overhead

Suggested Phased Plan

Phase 1 (MVP)

• action schema normalization (intent, actor, target, risk level)
• pre-flight rule checks
• standardized dry-run behavior for write actions
• structured decision logs (allow/deny/warn + rationale)

Phase 2

• post-exec validator
• source-grounding score + confidence tags
• connector capability matrix (supportsDryRun / supportsRollback / requiresApproval)

Phase 3

• policy packs by environment (dev/stage/prod)
• approval workflows for high-risk paths
• enterprise evidence exports

Open Questions

1. Should policy be hard-fail by default or warn-first during rollout?
2. What minimum evidence is required before allowing write actions?
3. How strict should source attribution be per action category?
4. Should dry-run be mandatory for all write actions, or policy-driven?
5. Which telemetry/metrics define "drift" in practice?

Success Criteria

• consistent guardrail behavior across connectors
• reduced unsafe/low-confidence outputs
• measurable dry-run adoption for write flows
• clear auditability of allow/deny decisions