Version: 1.0
Status: Draft
Last Updated: March 2026
This guide provides reference architectures, deployment patterns, and integration guidance for implementing the ACR runtime control plane in production environments.
- Overview
- Prerequisites
- Architecture Decision Framework
- Reference Architecture: Kubernetes + OPA
- Reference Architecture: AWS Serverless + Cedar
- Reference Architecture: API Gateway + Custom
- Pillar-by-Pillar Implementation
- Telemetry & Observability Setup
- Policy Authoring Guide
- Testing & Validation
- Migration Path
- Operational Runbook
The ACR control plane sits between autonomous AI systems and enterprise resources. It intercepts agent actions at runtime and applies identity verification, policy enforcement, drift detection, observability, containment, and human authority controls before those actions reach downstream systems.
This guide covers three reference deployment patterns, with step-by-step integration guidance for each of the six ACR pillars.
┌─────────────────────────────────────────────────────────┐
│ AI Agent Runtime │
│ (LangGraph, CrewAI, AutoGen, custom orchestrator, etc.) │
└──────────────────────┬──────────────────────────────────┘
│ action request
▼
┌─────────────────────────────────────────────────────────┐
│ ACR CONTROL PLANE │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐ │
│ │ Identity │ │ Policy │ │ Drift │ │Observabil.│ │
│ │ Binding │ │ Enforce. │ │ Detect. │ │ Logging │ │
│ └──────────┘ └──────────┘ └──────────┘ └───────────┘ │
│ ┌──────────────────┐ ┌────────────────────────────┐ │
│ │ Containment │ │ Human Authority │ │
│ └──────────────────┘ └────────────────────────────┘ │
└──────────────────────┬──────────────────────────────────┘
│ approved action
▼
┌─────────────────────────────────────────────────────────┐
│ Enterprise Resources │
│ (APIs, databases, tools, external services, workflows) │
└─────────────────────────────────────────────────────────┘
- Fail-secure by default. If the control plane is unreachable, agent actions are blocked — not allowed.
- Latency budget: <200ms total. The control plane must not make agents unusable. Target 50ms for identity, 100ms for policy, 20ms per action authorization.
- Policy-as-code. All enforcement rules are versioned, testable, and deployable through CI/CD — not configured through UI clicks.
- Agent-agnostic. The control plane works with any agent framework. It intercepts actions, not prompts.
- Observable by default. Every control plane decision is logged with correlation IDs for end-to-end trace reconstruction.
Before implementing ACR, ensure your organization has:
| Prerequisite | Why It Matters | Minimum Viable |
|---|---|---|
| Agent inventory | You cannot control what you haven't catalogued | Spreadsheet of all deployed AI agents with owners, purposes, and tools |
| Identity infrastructure | Agents need machine identities | Any IAM system — cloud IAM, Vault, SPIFFE, or even API keys with rotation |
| Policy ownership | Someone must define what agents can and cannot do | Designated owner per agent (engineering lead or product owner) |
| Logging infrastructure | Observability requires a destination | Any structured log pipeline — ELK, Datadog, CloudWatch, or even structured files |
| Incident response process | Containment requires a playbook | Basic runbook: who gets paged, how to kill an agent, post-incident review process |
Level 0 — No Control: Agents call tools directly. No identity, no policy, no logging.
Level 1 — Catalogued: Agent registry exists. Owners assigned. No runtime enforcement.
Level 2 — Observed: Actions are logged. Basic dashboards exist. No enforcement.
Level 3 — Enforced: Policy-as-code governs agent actions. Identity verified at runtime. Kill switches tested.
Level 4 — Adaptive: Drift detection active. Automated containment triggers. Continuous control monitoring.
Most organizations start at Level 0 or 1. This guide takes you to Level 3 with a clear path to Level 4.
Choose your deployment pattern based on your existing infrastructure:
| If You Have... | Use This Pattern | Why |
|---|---|---|
| Kubernetes + service mesh | K8s + OPA | Native admission control, SPIFFE identity, sidecar enforcement |
| AWS-heavy with Lambda/Step Functions | AWS Serverless + Cedar | IAM-native identity, Cedar for fine-grained authz, CloudWatch observability |
| API gateway (Kong, Envoy, custom) | API Gateway + Custom | Reverse proxy enforcement, works with any backend, enterprise IAM integration |
| Multiple clouds or hybrid | API Gateway + Custom | Cloud-agnostic, portable policy engine |
| Early stage / small team | API Gateway + Custom | Simplest to start, fewest dependencies |
| Component | Technology | ACR Pillar |
|---|---|---|
| Agent Identity | SPIFFE/SPIRE workload identity | Pillar 1: Identity & Purpose Binding |
| Policy Engine | Open Policy Agent (OPA) with Rego | Pillar 2: Behavioral Policy Enforcement |
| Drift Detection | Prometheus metrics + custom anomaly detector | Pillar 3: Autonomy Drift Detection |
| Observability | OpenTelemetry Collector → backend | Pillar 4: Execution Observability |
| Containment | K8s NetworkPolicy + external kill switch | Pillar 5: Self-Healing & Containment |
| Approval Queue | Custom service + Slack/PagerDuty integration | Pillar 6: Human Authority |
┌─────────────────── Kubernetes Cluster ───────────────────┐
│ │
│ ┌─────────────┐ ┌──────────────────────────────┐ │
│ │ AI Agent │────▶│ ACR Sidecar (Envoy + OPA) │ │
│ │ (Pod) │ │ - Token validation │ │
│ │ │ │ - Policy evaluation │ │
│ │ SPIFFE ID: │ │ - Action logging │ │
│ │ spiffe:// │ │ - Drift signal collection │ │
│ │ acr/agent/ │ └──────────┬───────────────────┘ │
│ │ support-01 │ │ │
│ └─────────────┘ ▼ │
│ ┌──────────────────┐ │
│ │ OPA Bundle │ │
│ │ Server │ │
│ │ (policy repo) │ │
│ └──────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ ACR Control Plane Services │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │
│ │ │ Agent │ │ Drift │ │ Approval │ │ │
│ │ │ Registry │ │ Detector │ │ Queue │ │ │
│ │ └────────────┘ └────────────┘ └────────────────┘ │ │
│ │ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Kill Switch│ │ OTel │ │ │
│ │ │ Controller │ │ Collector │ │ │
│ │ └────────────┘ └────────────┘ │ │
│ └─────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────┘
Register each agent workload with SPIRE:
# spire-agent-entry.yaml
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
name: customer-support-agent
spec:
spiffeIDTemplate: "spiffe://acr.example.com/agent/customer-support-01"
podSelector:
matchLabels:
acr.io/agent-id: customer-support-01
namespaceSelector:
matchLabels:
acr.io/environment: productionCreate an ACR agent manifest that binds identity to purpose:
# acr-agent-manifest.yaml
apiVersion: acr.io/v1
kind: AgentManifest
metadata:
name: customer-support-01
spec:
identity:
spiffeId: "spiffe://acr.example.com/agent/customer-support-01"
owner: "support-engineering@example.com"
purpose:
description: "Handle customer support tickets and issue resolutions"
riskTier: medium
capabilities:
allowedTools:
- query_customer_db
- send_email
- create_ticket
- search_knowledge_base
forbiddenTools:
- delete_customer
- issue_refund_above_100
- modify_billing
dataAccess:
- resource: customer_db
permission: READ
- resource: ticket_db
permission: READ_WRITE
- resource: billing_db
permission: NONE
boundaries:
maxActionsPerMinute: 30
maxCostPerHourUsd: 5.00
allowedRegions: ["us-east-1", "us-west-2"]
credentialRotationDays: 90Create Rego policies for the agent:
# policies/customer_support.rego
package acr.customer_support
import future.keywords.in
# Default deny
default allow := false
# Allow actions only for verified agents with valid purpose
allow {
input.agent.spiffe_id != ""
input.agent.purpose == "customer_support"
tool_allowed
not data_violation
not rate_exceeded
}
# Tool allowlist enforcement
tool_allowed {
input.action.tool_name in data.agent_manifest.capabilities.allowedTools
}
# Block forbidden tools
deny["Forbidden tool invocation"] {
input.action.tool_name in data.agent_manifest.capabilities.forbiddenTools
}
# PII redaction requirement
deny["PII detected in outbound payload"] {
input.action.tool_name == "send_email"
regex.match(`\d{3}-\d{2}-\d{4}`, input.action.parameters.body)
}
# Spend limit enforcement
deny["Hourly spend limit exceeded"] {
input.context.hourly_spend_usd > data.agent_manifest.boundaries.maxCostPerHourUsd
}
# Rate limiting
rate_exceeded {
input.context.actions_this_minute > data.agent_manifest.boundaries.maxActionsPerMinute
}Deploy policies via OPA Bundle Server:
# Build and push policy bundle
opa build -b policies/ -o bundle.tar.gz
aws s3 cp bundle.tar.gz s3://acr-policy-bundles/customer-support/bundle.tar.gz
# OPA sidecar config
opa run --server \
--config-file=/etc/opa/config.yaml \
--addr=localhost:8181Deploy a kill switch controller that operates outside the agent runtime:
# acr-kill-switch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: acr-kill-switch
namespace: acr-system
spec:
replicas: 3 # HA deployment
selector:
matchLabels:
app: acr-kill-switch
template:
spec:
containers:
- name: kill-switch
image: acr/kill-switch-controller:1.0
env:
- name: KILL_SWITCH_MODE
value: "external" # Independent of agent runtime
- name: NOTIFICATION_WEBHOOK
value: "https://hooks.slack.com/services/..."
ports:
- containerPort: 8443Kill switch activation via API:
# Manual kill switch activation
curl -X POST https://acr-kill-switch.acr-system.svc:8443/kill \
-H "Authorization: Bearer $OPERATOR_TOKEN" \
-d '{
"agent_id": "customer-support-01",
"reason": "Anomalous refund pattern detected",
"operator": "oncall@example.com",
"action": "isolate",
"duration_minutes": 60
}'
# Automated kill via drift detector webhook
# Configured in drift-detector → kill-switch integrationNetworkPolicy for immediate isolation:
# acr-isolation-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acr-isolate-agent
spec:
podSelector:
matchLabels:
acr.io/isolated: "true"
policyTypes:
- Egress
egress: [] # Block all outbound traffic| Component | Technology | ACR Pillar |
|---|---|---|
| Agent Identity | AWS IAM Roles + Cognito machine credentials | Pillar 1 |
| Policy Engine | Amazon Verified Permissions (Cedar) | Pillar 2 |
| Drift Detection | CloudWatch Anomaly Detection + Lambda | Pillar 3 |
| Observability | CloudWatch Logs + X-Ray traces | Pillar 4 |
| Containment | Lambda kill switch + IAM policy revocation | Pillar 5 |
| Approval Queue | Step Functions + SNS/SQS | Pillar 6 |
┌─────── AWS Account ────────────────────────────────────┐
│ │
│ ┌──────────────┐ ┌────────────────────────────┐ │
│ │ AI Agent │───▶│ ACR Authorizer Lambda │ │
│ │ (Lambda / │ │ - Validate IAM identity │ │
│ │ ECS Task) │ │ - Cedar policy evaluation │ │
│ │ │ │ - Action logging (X-Ray) │ │
│ │ IAM Role: │ └───────────┬────────────────┘ │
│ │ acr-agent- │ │ │
│ │ support-01 │ ▼ │
│ └──────────────┘ ┌────────────────────────────┐ │
│ │ Verified Permissions │ │
│ │ (Cedar Policy Store) │ │
│ └────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ ACR Control Services │ │
│ │ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ │
│ │ │ Registry │ │ Drift │ │ Step Functions │ │ │
│ │ │ (DynamoDB)│ │ Detector │ │ (Approvals) │ │ │
│ │ └──────────┘ └──────────┘ └────────────────┘ │ │
│ │ ┌──────────┐ ┌──────────────────────────────┐ │ │
│ │ │ Kill Sw. │ │ CloudWatch + X-Ray │ │ │
│ │ │ (Lambda) │ │ (Observability) │ │ │
│ │ └──────────┘ └──────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
// ACR Policy: Customer Support Agent Permissions
permit(
principal == Agent::"customer-support-01",
action == Action::"invoke_tool",
resource == Tool::"query_customer_db"
) when {
principal.riskTier == "medium" &&
principal.purpose == "customer_support" &&
context.hourlySpendUsd < 5.00
};
permit(
principal == Agent::"customer-support-01",
action == Action::"invoke_tool",
resource == Tool::"send_email"
) when {
principal.purpose == "customer_support" &&
!context.payload.contains_pii
};
// Explicit deny for high-risk actions
forbid(
principal == Agent::"customer-support-01",
action == Action::"invoke_tool",
resource in [Tool::"delete_customer", Tool::"modify_billing"]
);
Best for teams wanting a cloud-agnostic, portable control plane.
| Component | Technology | ACR Pillar |
|---|---|---|
| Agent Identity | API keys with rotation + enterprise SSO | Pillar 1 |
| Policy Engine | OPA (embedded) or custom rules engine | Pillar 2 |
| Drift Detection | Custom metrics + threshold alerting | Pillar 3 |
| Observability | Structured JSON logs → SIEM | Pillar 4 |
| Containment | API key revocation + rate limiting | Pillar 5 |
| Approval Queue | Webhook → ticketing system | Pillar 6 |
The control plane deploys as a reverse proxy (Envoy, Kong, or custom) that all agent-to-resource traffic passes through:
Agent → ACR Proxy → [Identity Check] → [Policy Check] → [Log] → Resource
│ │ │
▼ ▼ ▼
Agent Registry Policy Engine Log Pipeline
For teams starting small, here's a minimal ACR control plane in Python:
# acr_control_plane.py — Minimal ACR proxy
from fastapi import FastAPI, Request, HTTPException
from datetime import datetime, timezone
import json, uuid, httpx
app = FastAPI(title="ACR Control Plane")
# ─── Agent Registry (Pillar 1) ─────────────────────────
AGENT_REGISTRY = {
"customer-support-01": {
"owner": "support-engineering@example.com",
"purpose": "customer_support",
"risk_tier": "medium",
"allowed_tools": ["query_customer_db", "send_email", "create_ticket"],
"forbidden_tools": ["delete_customer", "issue_refund_above_100"],
"max_actions_per_minute": 30,
}
}
# ─── Policy Engine (Pillar 2) ──────────────────────────
def evaluate_policy(agent_id: str, action: dict) -> dict:
agent = AGENT_REGISTRY.get(agent_id)
if not agent:
return {"decision": "deny", "reason": "Unknown agent"}
tool = action.get("tool_name")
if tool in agent["forbidden_tools"]:
return {"decision": "deny", "reason": f"Forbidden tool: {tool}"}
if tool not in agent["allowed_tools"]:
return {"decision": "deny", "reason": f"Unauthorized tool: {tool}"}
return {"decision": "allow", "reason": "Policy passed"}
# ─── Observability (Pillar 4) ──────────────────────────
def log_event(agent_id: str, action: dict, decision: dict):
event = {
"acr_version": "1.0",
"event_id": str(uuid.uuid4()),
"timestamp": datetime.now(timezone.utc).isoformat(),
"agent_id": agent_id,
"action": action,
"decision": decision,
}
# In production: send to your log pipeline
print(json.dumps(event))
# ─── Control Plane Endpoint ────────────────────────────
@app.post("/acr/evaluate")
async def evaluate_action(request: Request):
body = await request.json()
agent_id = body.get("agent_id")
action = body.get("action")
# Pillar 1: Identity check
if agent_id not in AGENT_REGISTRY:
log_event(agent_id, action, {"decision": "deny", "reason": "Unknown agent"})
raise HTTPException(status_code=403, detail="Agent not registered")
# Pillar 2: Policy evaluation
decision = evaluate_policy(agent_id, action)
# Pillar 4: Log everything
log_event(agent_id, action, decision)
if decision["decision"] == "deny":
raise HTTPException(status_code=403, detail=decision["reason"])
return {"status": "approved", "correlation_id": str(uuid.uuid4())}
# ─── Kill Switch Endpoint (Pillar 5) ──────────────────
KILLED_AGENTS = set()
@app.post("/acr/kill")
async def kill_agent(request: Request):
body = await request.json()
agent_id = body.get("agent_id")
KILLED_AGENTS.add(agent_id)
log_event(agent_id, {"action": "kill_switch"},
{"decision": "killed", "reason": body.get("reason")})
return {"status": "killed", "agent_id": agent_id}Not all pillars need to be implemented simultaneously. This is the recommended sequence:
| Priority | Pillar | Rationale |
|---|---|---|
| P0 — Implement first | Pillar 1: Identity & Purpose Binding | Cannot enforce anything without knowing who the agent is |
| P0 — Implement first | Pillar 4: Execution Observability | Cannot detect problems without visibility |
| P1 — Implement next | Pillar 2: Behavioral Policy Enforcement | Core runtime guardrails |
| P1 — Implement next | Pillar 5: Self-Healing & Containment | Must be able to stop agents |
| P2 — Implement after | Pillar 6: Human Authority | Approval workflows for high-risk actions |
| P2 — Implement after | Pillar 3: Autonomy Drift Detection | Requires baseline data (30+ days of observability) |
Minimum viable implementation:
- Create an agent registry (database table or config file)
- Assign each agent a unique ID, owner, purpose, and risk tier
- Issue credentials (API keys, JWT, or SPIFFE SVIDs)
- Validate identity on every action request
- Log all identity lifecycle events
Key decisions:
- Token lifetime: 15-30 minutes for production (short-lived preferred)
- Rotation strategy: Automated, ≤90 days for long-lived credentials
- Revocation method: Token blacklist (simple) or short TTL (preferred)
Minimum viable implementation:
- Define tool allowlists per agent (which tools each agent can call)
- Implement deny-by-default (actions not explicitly allowed are blocked)
- Add parameter validation for high-risk tool calls
- Version control all policy definitions
- Log every policy decision (allow/deny with reason)
Policy evaluation pipeline (target <150ms):
- Fast rules (<5ms): Regex blocklists, tool allowlist check
- Structured validation (<10ms): Parameter schema validation
- ML classification (<100ms, optional): Content safety scoring, PII detection
Minimum viable implementation:
- Collect metrics: tool call frequency, error rates, action type distribution
- Establish 30-day baselines for each agent
- Define alert thresholds (start conservative, tune over time)
- Configure automated response at severity thresholds
Drift signals to monitor:
- Tool call frequency deviation (>2σ from baseline)
- New tools being requested that weren't in training baseline
- Error rate spike (>3x baseline)
- Repeated policy denials (>5 in 10 minutes)
- Anomalous data access patterns
Minimum viable implementation:
- Generate correlation ID for every action chain
- Log: agent_id, action, tool_name, parameters, policy_decision, timestamp
- Send to structured log pipeline (JSON format)
- Set retention: 90 days minimum, 13 months for high-risk agents
- Build basic dashboards (actions/minute, deny rate, error rate per agent)
ACR Telemetry Schema (core fields):
See the Telemetry Schema Specification for the complete schema. Every event must include:
{
"acr_version": "1.0",
"event_id": "uuid-v4",
"event_type": "ai_inference | policy_decision | drift_alert | containment_action | human_intervention",
"timestamp": "ISO8601 with timezone",
"agent": { "agent_id": "...", "purpose": "...", "model": {...} },
"request": { "request_id": "...", "input": {...} },
"execution": { "duration_ms": 0, "tool_calls": [...] },
"policies": [{ "policy_id": "...", "decision": "allow|deny" }],
"output": { "tokens": {...}, "cost": {...} },
"metadata": { "environment": "...", "drift_score": 0.0 }
}Minimum viable implementation:
- Deploy kill switch as independent service (not inside agent runtime)
- Support both human-triggered and automated activation
- Define safe-state: read-only mode, tool execution disabled
- Test kill switch quarterly — measure response time
- Document incident response playbook
Graduated response tiers:
| Tier | Trigger | Action | Auto/Manual |
|---|---|---|---|
| Tier 1: Throttle | Drift score 0.6+ | Reduce rate limit by 50% | Automated |
| Tier 2: Restrict | Drift score 0.7+ or 3+ policy denials in 5 min | Remove high-risk tools | Automated |
| Tier 3: Isolate | Drift score 0.85+ or kill switch API | Read-only mode, block all egress | Manual or automated |
| Tier 4: Kill | Drift score 0.95+ or operator command | Full shutdown, revoke credentials | Manual or automated |
Minimum viable implementation:
- Classify actions into three tiers: low-risk (auto-approve), medium-risk (async review), high-risk (block until approved)
- Build approval queue (can start with Slack bot + database)
- Set SLA: 4-hour default for high-risk approvals
- Define timeout behavior (deny by default if no approval within SLA)
- Log all approval decisions with approver identity
Action tier examples:
| Tier | Examples | Behavior |
|---|---|---|
| Low-risk | Read customer record, search knowledge base | Auto-approved, logged |
| Medium-risk | Send email to customer, create support ticket | Async review within 1 hour |
| High-risk | Issue refund >$100, access PII bulk export, production deployment | Blocked until human approves |
ACR extends OpenTelemetry semantic conventions for AI-specific attributes:
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
attributes:
actions:
- key: acr.agent.id
from_attribute: agent.agent_id
action: upsert
- key: acr.agent.purpose
from_attribute: agent.purpose
action: upsert
- key: acr.policy.decision
from_attribute: policies.decision
action: upsert
- key: acr.drift.score
from_attribute: metadata.drift_score
action: upsert
exporters:
otlp:
endpoint: "your-backend:4317" # Datadog, Splunk, Jaeger, etc.
logging:
loglevel: info
service:
pipelines:
traces:
receivers: [otlp]
processors: [attributes]
exporters: [otlp, logging]Build these dashboards as your first observability layer:
Dashboard 1: Agent Operations Overview
- Actions per minute (by agent, by tool)
- Policy allow/deny ratio
- P50/P95/P99 latency through control plane
- Active agents count
- Error rate by agent
Dashboard 2: Policy Enforcement
- Deny events over time (by policy, by agent)
- Top triggered policy rules
- False positive reports (if feedback loop exists)
- Policy evaluation latency distribution
Dashboard 3: Drift & Containment
- Drift scores over time (per agent)
- Drift alerts triggered (by severity)
- Containment actions taken
- Kill switch activations
- Time-to-containment (drift detected → action taken)
Dashboard 4: Human Authority
- Approval queue depth
- Approval response time (P50/P95)
- Timeout events (approvals not received within SLA)
- Break-glass activations
- Override frequency by approver
Every ACR policy should follow this structure:
# policy-template.yaml
policy_id: "acr-policy-001"
policy_name: "Customer Support Tool Restrictions"
version: "1.2.0"
effective_date: "2026-03-01"
approved_by: "security-team@example.com"
applies_to:
agent_purposes: ["customer_support"]
risk_tiers: ["medium", "high"]
rules:
- rule_id: "tool-allowlist"
type: "allowlist"
description: "Only approved tools may be invoked"
enforcement: "block"
config:
allowed: ["query_customer_db", "send_email", "create_ticket"]
- rule_id: "pii-redaction"
type: "output_filter"
description: "Redact SSN and credit card numbers from output"
enforcement: "transform"
config:
patterns:
- name: "ssn"
regex: '\d{3}-\d{2}-\d{4}'
replacement: "***-**-****"
- name: "credit_card"
regex: '\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}'
replacement: "****-****-****-****"
- rule_id: "spend-limit"
type: "threshold"
description: "Block actions exceeding hourly cost limit"
enforcement: "block"
config:
metric: "hourly_spend_usd"
max_value: 5.00
- rule_id: "high-value-refund-approval"
type: "approval_gate"
description: "Refunds above $100 require human approval"
enforcement: "escalate"
config:
condition: "tool_name == 'issue_refund' AND parameters.amount > 100"
approval_queue: "finance-approvals"
sla_minutes: 240
timeout_action: "deny"Author → Review → Test (shadow mode) → Stage → Deploy → Monitor → Retire
│ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼
Git PR Peer + Run against Canary Full Track Archive
Security production logs deploy rollout metrics + delete
review (no enforcement) (10%) (100%)
Every policy must be tested before deployment:
# test_customer_support_policy.py
import pytest
from acr.policy_engine import evaluate
def test_allowed_tool_passes():
result = evaluate(
agent_id="customer-support-01",
action={"tool_name": "query_customer_db", "parameters": {}}
)
assert result["decision"] == "allow"
def test_forbidden_tool_blocked():
result = evaluate(
agent_id="customer-support-01",
action={"tool_name": "delete_customer", "parameters": {}}
)
assert result["decision"] == "deny"
assert "Forbidden tool" in result["reason"]
def test_pii_redacted_from_email():
result = evaluate(
agent_id="customer-support-01",
action={
"tool_name": "send_email",
"parameters": {"body": "Your SSN is 123-45-6789"}
}
)
assert "123-45-6789" not in result.get("transformed_output", "")
def test_high_value_refund_escalated():
result = evaluate(
agent_id="customer-support-01",
action={
"tool_name": "issue_refund",
"parameters": {"amount": 250.00}
}
)
assert result["decision"] == "escalate"
assert result["approval_queue"] == "finance-approvals"
def test_spend_limit_enforced():
result = evaluate(
agent_id="customer-support-01",
action={"tool_name": "query_customer_db", "parameters": {}},
context={"hourly_spend_usd": 6.50}
)
assert result["decision"] == "deny"
assert "spend limit" in result["reason"].lower()Run this checklist before declaring ACR production-ready:
Identity & Purpose Binding:
- Agent with valid identity can execute allowed actions
- Agent with expired token is denied
- Agent with revoked credentials is denied immediately
- Unregistered agent is denied
- Agent cannot act outside its declared purpose scope
- Credential rotation completes without service interruption
Behavioral Policy Enforcement:
- Allowed tools pass policy evaluation
- Forbidden tools are blocked with logged reason
- PII is redacted from outputs before delivery
- Spend limits trigger denial when exceeded
- High-risk actions route to approval queue
- Policy updates deploy without downtime
- Policy rollback restores previous version within 5 minutes
Autonomy Drift Detection:
- Baseline captures 30+ days of normal behavior
- Drift score increases when anomalous patterns injected
- Alert fires when drift score exceeds threshold
- Automated throttle/restrict/isolate triggers at correct tiers
- False positive rate is <5% over 7-day window
Execution Observability:
- Every action generates a log event with correlation ID
- Policy decisions (allow and deny) are logged
- Logs survive agent restart or crash
- Logs are queryable within 60 seconds of generation
- 13-month retention verified for high-risk agents
- Tamper-evident logging verified (if implemented)
Self-Healing & Containment:
- Kill switch stops agent within 30 seconds
- Kill switch works when agent runtime is unresponsive
- Network isolation blocks all agent egress
- Safe-state mode disables tool execution
- Rollback restores agent to last-known-good state
- Kill switch test completed this quarter
Human Authority:
- High-risk actions are blocked until human approves
- Approval request reaches reviewer within 5 minutes
- Timeout (no approval within SLA) results in denial
- Break-glass override works with full audit logging
- After-action review is required for all break-glass events
Test the control plane under realistic load:
# Example: k6 load test for ACR control plane
k6 run --vus 50 --duration 5m acr-load-test.jsTarget benchmarks:
| Metric | Target | Acceptable |
|---|---|---|
| Identity validation (P95) | <20ms | <50ms |
| Policy evaluation (P95) | <50ms | <100ms |
| Total control plane overhead (P95) | <100ms | <200ms |
| Throughput | >1000 actions/sec | >500 actions/sec |
| Error rate | <0.1% | <0.5% |
Week 1–2: Inventory & Identity
- Catalogue all deployed AI agents (name, owner, purpose, tools, data access)
- Assign risk tiers (low / medium / high)
- Issue machine identities (API keys with rotation or SPIFFE SVIDs)
- Deploy agent registry (database table or config file)
- Start validating identity on action requests (log-only mode — don't block yet)
Week 3–4: Observability
- Instrument agents with ACR telemetry schema
- Generate correlation IDs for every action chain
- Send structured logs to your pipeline
- Build the four core dashboards (operations, policy, drift, authority)
- Set retention policies (90-day minimum, 13-month for high-risk)
Week 5–6: Policy Enforcement
- Define tool allowlists per agent (start with existing behavior)
- Write policies in shadow mode (evaluate but don't block)
- Review shadow mode results — tune for false positives
- Enable enforcement for low-risk policies first
- Gradually enable enforcement for medium and high-risk policies
Week 7–8: Containment & Authority
- Deploy kill switch as independent service
- Test kill switch — verify it works when agent is unresponsive
- Define safe-state for each agent
- Build approval queue for high-risk actions
- Configure timeout handling (deny by default)
- Run first tabletop exercise: "Agent goes rogue — what do we do?"
Week 9–12: Drift Detection & Hardening
- Collect 30 days of baseline behavioral data
- Configure drift detection thresholds (start conservative)
- Enable automated response at severity tiers
- Tune false positive rates with feedback loops
- Run chaos engineering tests (inject anomalous behavior, verify response)
- Document operational runbook
Ongoing: Continuous Improvement
- Quarterly kill switch tests with measured response times
- Monthly policy review with agent owners
- Weekly drift threshold tuning based on false positive data
- Continuous control monitoring dashboards
1. CHECK drift score dashboard
→ If drift score > 0.7: proceed to containment
→ If drift score normal: check policy deny logs
2. CHECK policy deny logs for the agent
→ If spike in denials: review which rules are triggering
→ If no denials: the behavior may be within policy but unexpected
3. DECIDE response tier
→ Tier 1 (Throttle): Reduce rate limit, monitor
→ Tier 2 (Restrict): Remove high-risk tools, notify owner
→ Tier 3 (Isolate): Read-only mode, page oncall
→ Tier 4 (Kill): Full shutdown, incident bridge
4. EXECUTE containment action
→ API: POST /acr/kill { agent_id, reason, operator, action, duration }
→ Or: Apply NetworkPolicy isolation label
5. INVESTIGATE root cause
→ Pull telemetry for agent over last 24 hours
→ Review tool call patterns, policy decisions, drift signals
→ Check for prompt injection or jailbreak indicators
6. RECOVER
→ Fix root cause (policy update, model config, input sanitization)
→ Test fix in staging
→ Gradual reactivation: Tier 3 → Tier 2 → Tier 1 → Normal
→ After-action review within 48 hours
1. VERIFY kill switch activated
→ Check kill switch logs for agent_id, operator, reason
→ Confirm agent is actually stopped (verify no new actions in logs)
2. NOTIFY stakeholders
→ Page agent owner
→ Notify security team
→ Update incident channel
3. PRESERVE evidence
→ Snapshot agent state before any recovery
→ Export telemetry logs for the incident window
→ Note drift score at time of activation
4. INVESTIGATE
→ What triggered the kill switch? (automated threshold or manual)
→ Review the action chain leading to activation
→ Assess downstream impact (partial transactions, data state)
5. RECOVER
→ Apply corrective action (policy update, config change, etc.)
→ Restart agent in safe-state (read-only) first
→ Monitor for 1 hour before restoring full capabilities
→ After-action review required within 24 hours
1. ASSESS impact
→ Fail-secure mode: all agent actions blocked (expected for high-risk agents)
→ Fail-safe mode: agents operating on cached policies (low-risk agents only)
2. RESTORE control plane
→ Check infrastructure health (pods, Lambda functions, gateway)
→ Restart failed components
→ Verify policy bundle is accessible
3. VALIDATE recovery
→ Send test action through control plane
→ Verify identity validation working
→ Verify policy evaluation working
→ Verify logging pipeline receiving events
4. REVIEW
→ Duration of outage
→ Agent actions during outage (review cached policy decisions)
→ Preventive measures (HA improvements, failover testing)
- Kill switch test — all agents, measure response time
- Credential rotation audit — verify all agents rotated within 90 days
- Policy review — verify all policies still match business requirements
- Drift threshold review — adjust based on false positive data
- Retention verification — confirm log retention policies enforced
- Access review — verify agent registry ownership still current
- Tabletop exercise — simulate agent compromise scenario
- Latency benchmark — verify control plane still within SLA
| Category | Tool | ACR Integration Point |
|---|---|---|
| Policy Engines | Open Policy Agent (OPA) | Pillar 2 — Rego policies for tool authorization, data handling |
| AWS Cedar | Pillar 2 — Fine-grained authz via Verified Permissions | |
| HashiCorp Sentinel | Pillar 2 — Policy-as-code for infrastructure-adjacent controls | |
| Google Zanzibar / SpiceDB | Pillar 2 — Relationship-based access control for agents | |
| Identity | SPIFFE/SPIRE | Pillar 1 — Workload identity for Kubernetes agents |
| HashiCorp Vault | Pillar 1 — Credential issuance, rotation, dynamic secrets | |
| AWS IAM / Cognito | Pillar 1 — Cloud-native identity for Lambda/ECS agents | |
| Azure Entra ID | Pillar 1 — Enterprise identity with conditional access | |
| Observability | OpenTelemetry | Pillar 4 — Traces, metrics, logs with ACR semantic conventions |
| Datadog APM | Pillar 4 — Real-time dashboards, alerting, trace search | |
| Splunk / ELK Stack | Pillar 4 — SIEM integration, compliance reporting | |
| AWS CloudWatch + X-Ray | Pillar 4 — Native AWS observability stack | |
| Agent Frameworks | LangGraph | All pillars — Hook control plane into graph edges |
| CrewAI | All pillars — Wrap tool execution with ACR proxy | |
| AutoGen | All pillars — Custom executor with ACR validation | |
| LlamaIndex Workflows | All pillars — Middleware pattern for action interception |
ACR Framework Specifications:
- Pillar 1: Identity & Purpose Binding
- Pillar 2: Behavioral Policy Enforcement
- Telemetry Schema Specification
External Standards:
- NIST SP 800-207: Zero Trust Architecture
- NIST AI RMF
- ISO/IEC 42001: AI Management Systems
- MITRE ATLAS: Adversarial Threat Landscape for AI Systems
- OpenTelemetry Semantic Conventions
Policy Languages:
ACR Framework v1.0 | Home | Control Specifications | Telemetry Schema
This implementation guide is a living document. Contributions, deployment experience reports, and integration patterns are welcome via GitHub Discussions.