Skip to content

README.md

Latest commit

 

History

History
384 lines (302 loc) · 22.4 KB

README.md

File metadata and controls

384 lines (302 loc) · 22.4 KB

CI PyPI License: MIT Python 3.10+ Release Open in GitHub Codespaces

Σ OVERWATCH

Current release: v2.1.0 See: docs/release/RELEASE_NOTES_v2.1.0.md

Repo Radar KPI (latest)

Repo KPI Badge Roadmap Badge

Completed Track (v2.1.0 — Decision Infrastructure Hardening)

  • All 8 KPIs >= 7.0 PASS
  • SSI >= 55 PASS (60.02)
  • 22 issues closed across 7 stability releases (v2.0.10–v2.0.16)
  • Authority ledger chain verification + replay detection
  • Evidence source binding + economic cost ledger
  • Intent mutation detection + schema version enforcement
  • Enterprise release checklist + operator runbook

Next Track (v2.2.0 — Institutional Expansion)

  • Enterprise Connectors
  • Multi-provider DISR abstraction
  • Authority-bound contract enforcement expansion
  • Integration schema finalization

Domain Modes (v2.1.2)

  • IntelOps — 12 function handlers: claim ingest, validate, drift detect, patch, MG update, canon promote, authority/evidence check, triage, supersede, half-life, confidence recalc
  • FranOps — 12 function handlers: canon propose/bless/enforce, retcon assess/execute/propagate, inflation monitor, expire, supersede, scope check, drift detect, rollback
  • ReflectionOps — 19 function handlers: episode begin/seal/archive, gate evaluate/degrade/killswitch, audit non-coercion, severity score, coherence check, reflection ingest, IRIS resolve, replay + institutional memory (precedent ingest, pattern fingerprint, precedent match, knowledge consolidate, temporal recall, knowledge decay, IRIS precedent resolve)
  • AuthorityOps — 19 function handlers: action intake, actor/resource resolve, policy load, DLR presence, assumption validate, half-life check, blast radius threshold, kill-switch, decision gate, audit emit, delegation chain validate, authority drift detect, blast radius simulate/propagate/seal, drift history query, assumption sweep, cross-domain drift correlate
  • ParadoxOps — 12 function handlers: tension set create, pole manage, dimension attach/shift, pressure compute, imbalance compute, threshold evaluate, drift promote, interdimensional drift detect, seal snapshot, patch issue, lifecycle transition
  • ActionOps — 19 function handlers: commitment intake/validate, deliverable track, deadline check, compliance evaluate, risk assess, breach detect, escalation, remediation, adjust, complete, report + decision accounting (cost record, time-to-decision, value assess, debt detect, ROI compute, budget enforce, accounting report)
  • Cascade Engine — 27 cross-domain rules with depth-limited propagation
  • Event Contracts — routing table mapping 79 functions + 91 events
  • Money Demo v2 — 10-step end-to-end pipeline (make demo-money)
  • JRM — log-agnostic refinement engine: 3 adapters, 5-stage pipeline, JRM-X packet builder, enterprise federation
  • Drift Radar — cross-domain drift intelligence surface: correlation, trending, forecasting, remediation prioritization
  • DecisionSurface — portable Coherence Ops runtime with 3 adapters (Notebook, CLI, Vantage stub), shared claim-event engine, 54 tests
  • 1861+ total tests passing

EDGE Modules

Exportable single-file HTML apps — zero dependencies, work offline, governance-ready.

Module Version Description
EDGE Unified v1.0.0 8-tab suite: Suite, Hiring, Bid, Compliance, BOE, IRIS, Delegation, Utility
Coherence Dashboard v2.0.0 4-tab coherence overview with claims, drift, and analysis
JRM EDGE v1.0.8 9-stage JRM pipeline explorer: events table, packets, health ring, test lab, stream mode, So What panel, policy drawer, Consistency Gate (sheaf-residue engine)
RFP Co-Pilot v1.0.0 8-tab RFP extraction workflow: Co-Pilot prompt, JSON spec, Power Query M scripts, role action packets
RFP Co-Pilot Exec Brief v1.0.0 1-page executive summary with Print/PDF
Hiring UI v1.0.0 Staffing intake
BidNoBid UI v1.0.0 Opportunity evaluation
Compliance Matrix v1.0.0 Requirements mapping
BOE Pricing v1.0.0 Rate calculations
Award Staffing v1.2.0 Cost estimation
Suite ReadOnly v1.0.0 Telemetry + rollup
Domino Delegation Encryption v1.0.0 4-of-7 Shamir threshold encryption ceremony with domino chain co-presence proof, TTL-gated keywords, AES-256-GCM
Domino Delegation Verifier v1.0.0 Read-only ceremony record verification (core edition)
Coherence Ops v1.0.0 7-tab human-friendly guide: Overview, How It Works, The 4 Domains, The 4 Artifacts, Drift → Patch, Quick Start, FAQ
SplitKey MaxStrength v2 Maximum-strength Shamir split-key ceremony with enhanced entropy and validation

JRM EDGE v1.0.8 features: Consistency Gate module (graph-based coherence scoring, basic + sheaf modes), CRS gauge, edge builder, DS auto-emission on threshold breach, patch suggestions (EXPIRE_CLAIM, REQUEST_EVIDENCE, DOWNWEIGHT_AUTHORITY, SUPERSEDE_CANON, SPLIT_CLAIM), Test Lab full harness (6 fixtures + 5 property tests). Prior: So What panel, Analyzer vs Deep Sigma toggle, packet chain timeline + diff, live stream mode + Freeze & Seal, policy drawer (guarded) + regression rerun.

RFP Co-Pilot: Parse Once. Execute Everywhere. — AI-assisted RFP extraction into structured JSON, loaded via Excel Power Query, with role-based action packets for 6 proposal team roles.

Competitive Gap Closure (v2.1.0)

  • Tool-call + LLM span tracing (OTel)
  • Connector auto-instrumentation + W3C trace context
  • RuntimeGate + SLO circuit breaker
  • Compliance export encryption-at-rest + scheduled export
  • Fairness drift types + external tool adapter (AIF360, Fairlearn)

Positioning Shift

Institutional Decision Infrastructure

Trust layer for agentic AI: verify before act, seal what happened, detect drift, ship patches.

What It Does

Organizations make thousands of decisions. Almost none are structurally recorded with their reasoning, evidence, or assumptions. When leaders leave, conditions change, or AI accelerates decisions 100x — governance fails silently.

Σ OVERWATCH fills this gap with three primitives:

  • Truth — Decision Ledger Records (DLR) capture what was decided, by whom, with what evidence
  • Reasoning — Reasoning Scaffolds (RS) capture why — claims, counter-claims, weights
  • Memory — Decision Scaffolds + Memory Graphs (DS + MG) make institutional knowledge queryable

When assumptions decay, Drift fires. When drift exceeds tolerance, a Patch corrects it. This is the Drift → Patch loop — continuous self-correction.

Quick Start

pip install deepsigma

# One-command Money Demo (recommended first run)
make demo

# Health check
deepsigma doctor

# Score coherence (0–100, A–F)
python -m core score ./core/examples/sample_episodes.json --json

# Drift → Patch canonical entrypoint
python -m core.examples.drift_patch_cycle

# Full 7-step Golden Path (no credentials needed)
deepsigma golden-path sharepoint \
  --fixture src/demos/golden_path/fixtures/sharepoint_small --clean

# Retention lifecycle sweep (cron-friendly)
deepsigma retention sweep --tenant tenant-alpha

Golden-Path Proof Artifacts

Credibility Index dashboard proof Drift detection and patch proof

# Golden Path run (ingest -> drift -> patch -> recall)
PYTHONPATH=src python -m tools.golden_path_cli golden-path sharepoint \
  --fixture src/demos/golden_path/fixtures/sharepoint_small \
  --output golden_path_output --clean

# Trust Scorecard (includes WHY retrieval SLO check)
PYTHONPATH=src python -m tools.trust_scorecard \
  --input golden_path_output \
  --output golden_path_output/trust_scorecard.json
============================================================
  GOLDEN PATH
============================================================
  [1] CONNECT              PASS
  [2] NORMALIZE            PASS
  [3] EXTRACT              PASS
  [4] SEAL                 PASS
  [5] DRIFT                PASS
  [6] PATCH                PASS
  [7] RECALL               PASS
...
  IRIS:       WHY=RESOLVED, WHAT_CHANGED=RESOLVED, STATUS=RESOLVED
  Drift:      6 events
  Patch:      applied
============================================================
Trust Scorecard written to golden_path_output/trust_scorecard.json
  SLOs:    ALL PASS

Trust Scorecard highlights from the same run:

  • iris_why_latency_ms: 1.4 (<= 60000 target, retrieval <= 60s)
  • patch_applied: true
  • drift_events_detected: 6
  • all_steps_passed: true

Court-Grade Proof (60 seconds)

# Seal + sign + authority bind + transparency log + pack
python src/tools/reconstruct/seal_and_prove.py \
    --decision-id DEC-001 --clock 2026-02-21T00:00:00Z \
    --sign-algo hmac --sign-key-id ds-dev --sign-key "$KEY" \
    --auto-authority --pack-dir /tmp/pack

# Verify everything in one command:
python src/tools/reconstruct/verify_pack.py --pack /tmp/pack --key "$KEY"

Key Capabilities

Capability Description Docs
Coherence Ops CLI Score, audit, query, reconcile decision artifacts CLI Reference
Golden Path 7-step end-to-end proof loop Golden Path
Credibility Engine Institutional-scale claim lattice with formal scoring Engine Docs
Trust Scorecard Measurable SLOs from every Golden Path run Spec
Excel-first BOOT Govern decisions in a shared workbook — no code required BOOT Protocol
MDPT Multi-Dimensional Prompt Toolkit for governed prompt ops MDPT Docs
Court-Grade Admissibility Seal-and-prove pipeline: Merkle commitments, transparency log, multi-sig witness, hardware key hooks Admissibility Levels
MCP Server Model Context Protocol server with auth + rate limiting MCP Adapter
API Reference Auto-generated OpenAPI + static Redoc docs API Docs
RDF/SPARQL Semantic lattice queries via in-process SPARQL 1.1 SPARQL Service
Dashboard React dashboard with Trust Scorecard + Zustand store Dashboard
Runtime Gate Composable pre-execution policy constraints with SLO circuit breaker src/engine/runtime_gate.py
OTel Span Tracing Tool-call + LLM completion spans, connector auto-instrumentation, W3C context propagation src/adapters/otel/
Compliance Export SOC 2 evidence packages with encryption-at-rest + scheduled auto-export CLI: compliance export
Fairness Adapter Hybrid fairness monitoring — ingest AIF360/Fairlearn reports as drift signals src/adapters/fairness/
Domain Modes IntelOps (12), FranOps (12), ReflectionOps (19), AuthorityOps (19), ParadoxOps (12), ActionOps (19) — 93 function handlers with deterministic replay src/core/modes/
ParadoxOps Paradox tension sets — pair/triple/higher-order pole detection, multi-dimensional scoring, inter-dimensional drift, full lifecycle src/core/paradox_ops/
DecisionSurface Portable Coherence Ops runtime — generic adapter layer for claim/event evaluation across environments (Notebook, CLI, Vantage) src/core/decision_surface/
Cascade Engine Cross-domain event propagation with 27 declarative rules and depth-limited cascading src/core/modes/cascade.py
Event Contracts Routing table: 79 functions + 91 events mapped to FEEDS topics, subtypes, and handlers src/core/feeds/contracts/
Money Demo v2 10-step pipeline exercising all 6 domain modes with drift, retcon, and cascade src/demos/money_demo/
Institutional Memory Precedent registry, pattern fingerprinting, knowledge consolidation, temporal decay — RE-F13→F19 src/core/institutional_memory/
Drift Radar Cross-domain drift correlation, trending, forecasting, remediation prioritization src/core/drift_radar/
Decision Accounting Cost tracking, value scoring, debt engine, ROI computation, budget enforcement — ACTION-F13→F19 src/core/decision_accounting/
FEEDS Event Surface 9-module pub/sub event backbone: envelope + schemas, file-bus + DLQ, manifest-first ingest, authority gate, evidence check, drift triage, canon store, claim validator, MG writer src/core/feeds/
JRM Pipeline Log-agnostic Judgment Refinement Module — 3 adapters (Suricata, Snort, Copilot), 5-stage coherence pipeline, JRM-X packet zips src/core/jrm/
JRM Federation Cross-environment gate, hub, advisory engine, HMAC-SHA256 packet signing enterprise/src/deepsigma/jrm_ext/

Connectors

All connectors conform to the Connector Contract v1.0.

Connector Transport Docs
SharePoint Graph API docs
Power Platform Dataverse Web API docs
AskSage REST API docs
Snowflake Cortex + SQL API docs
LangGraph LangChain Callback docs
OpenClaw WASM Sandbox docs
Local LLM llama.cpp / OpenAI-compatible docs

Security And Data Handling

  • Data Boundaries — data at rest, storage locations, retention, redaction, tenancy isolation, connector flow, secrets policy, and network boundaries.
  • Connector SDK — ConnectorV1 contract and safety expectations for custom adapters.
  • DISR Security Model — Breakable -> Detectable -> Rotatable -> Recoverable posture for pilot security credibility.
  • Key Lifecycle — key versioning, TTL, and rotation cadence.
  • Recovery Runbook — compromise response and re-encryption recovery sequence.
  • 10-Minute Security Demo — reproducible DISR drill (make security-gate + make security-demo).
  • DISR Re-encrypt Benchmark — CI-eligible benchmark (make benchmark) with output in release_kpis/scalability_metrics.json.
  • Scalability Regression Gate — prevents throughput regressions (make scalability-gate).
  • Benchmark Trend — historical throughput visualization (make benchmark-trend).

Note: --ci-mode runs produce deterministic, KPI-eligible benchmark evidence. Use --real-workload for full AES-256-GCM crypto operations with production keys.

Monitoring

  • Prometheus metrics endpoint: GET /metrics
  • Grafana dashboard: ops/grafana/deepsigma.json
  • Local monitoring stack: docker-compose.monitoring.yml

Start monitoring stack:

docker compose -f docker-compose.monitoring.yml up --build

Endpoints:

  • DeepSigma API: http://localhost:8000
  • Prometheus: http://localhost:9090
  • Grafana: http://localhost:3001 (admin / admin)

Repo Structure

DeepSigma/
├── src/                 # 12 Python packages (all source code)
│   ├── core/   #   Core library + CLI
│   ├── engine/          #   Compression, degrade ladder, supervisor
│   ├── adapters/        #   MCP, SharePoint, Snowflake, LangGraph, OpenClaw, AskSage
│   ├── deepsigma/       #   Unified product CLI
│   ├── demos/           #   Golden Path, Excel-first Money Demo
│   ├── mdpt/            #   MDPT tools + Power App starter kit
│   └── ...              #   credibility_engine, services, mesh, governance, tenancy, verifiers, tools
├── tests/               # 2150+ tests, fixtures, datasets
├── docs/                # Documentation + examples (canonical, mermaid, lattices, etc.)
├── dashboard/           # React dashboard + API server
├── schemas/             # JSON schemas (core engine + Prompt OS)
├── artifacts/           # Workbooks, templates, sealed runs, sample data
├── prompts/             # Canonical prompts + Prompt OS control prompts
└── .github/             # CI/CD workflows

Documentation
QUICKSTART.md 5-minute starter path (deepsigma init + make demo)
START_HERE.md Front door
HERO_DEMO.md 5-minute hands-on walkthrough
NAV.md Full navigation index
ABOUT.md Reality Await Layer (RAL)
OPS_RUNBOOK.md Operations + incident playbooks
HELM.md Helm install/test runbook (kind/minikube)
STATELESS_API_SCALE_GUIDE.md 3-replica benchmark + sizing guidance
STABILITY.md Versioning policy + stability guarantees
docs/99-docs-map.md Complete docs map

Excel Prompt OS v2

Structured cognition workbook for institutional decision-making — no code required.

Prompts

Contributing

See CONTRIBUTING.md.

License

MIT

Σ OVERWATCH We don't sell agents. We sell the ability to trust them.