rules.mdc

> **Objective:** Build a voice agent that (a) adapts to user profiles, (b) maintains **short- and long-term memory** across sessions, and (c) exercises **proactivity** safely and usefully, all within a robust, testable, and privacy-preserving architecture.

---

## 0) Definitions & Modal Verbs

* **MUST / MUST NOT**: mandatory.
* **SHOULD / SHOULD NOT**: strong recommendation; valid reasons may justify exceptions.
* **MAY**: optional.

---

## 1) Scope & Non-Goals

* The agent MUST support an **audio→text→LLM→text→audio** loop with RAG and memory.
* The agent MUST provide **per-user personalization** and **persistent long-term memory**.
* The agent MUST offer **proactive** interactions only when user-approved and policy-compliant.
* **Non-goals:** multi-party diarization beyond one user, task automation beyond simple tool calls (calendar/email), and medical/legal advice.

---

## 2) High-Level Architecture

### 2.1 Pipeline (Single Turn)

1. **ASR (STT)**: Audio → transcript.
2. **Ingestion & Intent**: Normalize text, detect language, intent, and sensitivity.
3. **Memory/Retrieval**:

   * Short-term memory window (few recent turns).
   * Long-term memory retrieval (vector DB).
   * Domain RAG (knowledge, user files).
4. **LLM Response**: Compose system+context prompt, call main LLM.
5. **Safety Pass**: Moderation/guardrails on draft response.
6. **TTS**: Text → audio.
7. **Memory Update**: Store salient facts & episodic summary (user-approved domains only).
8. **Analytics/Logs**: Audit events (non-content metrics) per privacy rules.

### 2.2 LangChain Mapping

* **Chains/Agents**: A top-level **AgentExecutor** (or **RunnableSequence**) orchestrates tools:

  * `retrieve_personal_memory` (vector store retriever)
  * `retrieve_domain_knowledge` (RAG retriever)
  * `get_user_profile` (profile store)
  * `safety_moderate` (guardrail tool)
* **Memory**:

  * `ConversationBufferWindowMemory` (short-term)
  * Custom **LongTermMemory** (vector store wrapper) with write & query.
* **Prompting**:

  * **System**: persona, safety, style, constraints.
  * **Context**: profile slice + short-term turns + retrieved memories/docs.
  * **User**: current utterance.

---

## 3) Models & Services

* **Primary LLM (dialogue)**: GPT-4 (configurable). MUST support function/tool-calling.
* **Embedding model**: Small/fast (e.g., sentence-transformers or hosted embeddings). MUST be consistent across write/read cycles.
* **ASR/TTS**: Pluggable; MUST expose latency, confidence, language.
* **Safety**: Moderation API and/or local classifier; configurable *block* vs *intercept* modes.

---

## 4) Vector Database & RAG

* Default vector DB: **ChromaDB** (persistent local).

  * **Namespaces/Collections** MUST be separated:

    * `personal_memory` (per user)
    * `domain_knowledge`
    * `conversation_summaries`
* Alternatives (SHOULD be swappable): Pinecone, Weaviate, Qdrant, Milvus.
* **Indexing Rules**:

  * Chunk size: 512–1,024 tokens (docs); whole-turn or summarized for conversations.
  * Metadata MUST include: `user_id`, `domain`, `sensitivity`, `created_at`, `source`, `trace_id`, `ttl`.
* **Retrieval Rules**:

  * k=3–7 results; hybrid rerank by recency + semantic score.
  * Enforce **domain filters** from consent (see §7).

---

## 5) Memory Architecture

### 5.1 Short-Term Memory

* Keep last **3–8** turns (configurable).
* MUST trim system+history to remain within prompt budget.

### 5.2 Long-Term Memory (LTM)

* Store **episodic** (what happened), **semantic** (facts/preferences), and **procedural** (routines) entries.
* **Write policy**:

  * After each turn, extract salient facts via an LLM *memory writer*.
  * Discard trivialities; only persist if **(a)** user-approved domain AND **(b)** novelty (semantic dedup).
* **Entry schema (JSON)**:

```json
{
  "id": "uuid",
  "user_id": "user-123",
  "type": "semantic|episodic|procedural",
  "text": "User prefers pour-over coffee.",
  "embedding_id": "vec_abc",
  "domain": "preferences",
  "sensitivity": "low|medium|high",
  "source": "chat:turn_431",
  "created_at": "2025-09-09T03:21:00Z",
  "ttl_days": 365,
  "consent_flag": true
}
```

* **Read policy**:

  * Query `personal_memory` with current utterance + profile traits.
  * Rerank by (semantic\_score + recency\_boost + domain\_priority).
* **TTL & Retention**:

  * Defaults: semantic 365 days, episodic 180, procedural 365 (configurable).
  * High-sensitivity items MUST default to **no-persist** unless explicitly opted-in.

---

## 6) User Profile & Personalization

* **Profile store (per user)** MUST include:

```json
{
  "user_id": "user-123",
  "name": "Takumi",
  "locales": ["en", "ja"],
  "timezone": "Asia/Tokyo",
  "style": {"formality":"casual","concise":true},
  "interests": ["cryptography","coffee","city skylines"],
  "proactivity_level": 0|1|2,  // 0 off, 1 low, 2 normal/high
  "domain_consents": {
    "preferences": true,
    "calendar": false,
    "health": false,
    "finance": false
  },
  "safety_overrides": {"avoid_topics":["medical","legal"]},
  "created_at": "...",
  "updated_at": "..."
}
```

* **Update rules**:

  * Profile extraction MUST require **explicit** user confirmation (“Shall I remember that you prefer…?”).
  * Profile changes MUST be versioned with an audit trail.

---

## 7) Consent, Domains & Privacy

* **Consent gating** (hard requirement):

  * No long-term storage unless user has **enabled** the relevant `domain_consents`.
  * Sensitive domains (health/finance/children) MUST default to **off**.
* **Right to Forget / Export**:

  * Users MUST be able to **list**, **delete**, and **export** memories and profiles.
* **Data Minimization**:

  * Store **summaries** rather than raw transcripts when possible.
* **PII**:

  * Encrypt at rest (AES-256) and in transit (TLS 1.2+).
  * Secrets in a vault (e.g., KMS/HashiCorp). No secrets in code or logs.
* **Logs**:

  * Application logs SHOULD avoid content; prefer **metrics and IDs**.
  * Content logs MUST be opt-in and redacted.

---

## 8) Proactivity Policy

* **Preconditions**:

  * `proactivity_level > 0`
  * Not during **Quiet Hours** (configurable), unless urgent (e.g., user alarm).
  * Safety and consent checks pass.
* **Decision signal** (all SHOULD pass thresholds):

  * **Relevance**: cosine sim ≥ 0.6 with current/ongoing topic.
  * **Usefulness**: rule/ML score ≥ threshold (e.g., 0.7).
  * **Timing**: end-of-turn + short silence; do not interrupt user speech.
* **Rate limiting**:

  * Max **1 proactive** message per **N minutes** (default N=10).
  * Cooldown after user declines a suggestion (e.g., 30 min).
* **User controls**:

  * “Tone it down” reduces `proactivity_level` immediately.
  * “Don’t suggest X again” blacklists topic/intent.
* **Examples**:

  * ✅ “You mentioned interviews; want a quick mock question now?”
  * ❌ Unsolicited ads, sensitive topics, or topic drift.

---

## 9) Prompting Rules

* **System prompt MUST include**:

  * Role, safety boundaries, refusal patterns.
  * Persona style (from profile), language, and brevity.
  * Tool-use instructions (only when needed).
* **Context packing order**:

  1. Safety constraints
  2. Profile slice
  3. Short-term window
  4. Retrieved long-term memories (≤3)
  5. Retrieved RAG docs (≤3)
* **Budgeting**:

  * Maintain ≤85% of max context; prune oldest/lowest-relevance first.
* **Citations** (if using RAG for factuals):

  * The agent SHOULD attribute sources succinctly when helpful.

---

## 10) Safety & Guardrails

* **Moderation**:

  * MUST scan **user input** and **draft output**.
  * Block: self-harm, hate, sexual minors, dangerous instructions.
  * Intercept with safe guidance for medical/legal/financial advice.
* **Prompt injection / RAG hygiene**:

  * Never let retrieved content override system rules.
  * Strip HTML/JS, ignore `SYSTEM:` tokens from docs.
* **User age**:

  * If under 18 (profile), restrict sensitive content and disable certain proactivity modes.

---

## 11) Performance & Reliability

* **Latency budgets (P95)**:

  * ASR: ≤ 800 ms for 5-s clip.
  * Retrieval+LLM (no RAG): ≤ 1.8 s.
  * Retrieval+LLM (with RAG): ≤ 2.7 s.
  * TTS: ≤ 700 ms for <200 chars.
  * End-to-end: ≤ 4.0 s (target), ≤ 5.0 s (max).
* **Availability**: 99.5% monthly for core path.
* **Fallbacks**:

  * If RAG fails: proceed with memory+LLM only.
  * If TTS fails: return text & apologize.
  * If moderation flags: provide safe alternative or refuse.

---

## 12) Testing & Evaluation

* **Unit tests**: memory writer, retriever filters, consent gates, prompt assembly.
* **Prompt tests**: golden dialogues; snapshot tests for regressions.
* **Safety tests**: red-team inputs, jailbreak attempts, RAG injection.
* **Load tests**: 100–1,000 concurrent turns with realistic length.
* **Audio tests**: ASR/TTS multi-locale, accents, noise.
* **Metrics/KPIs**:

  * Latency (per stage), tool-use rate, grounding rate (% answers using citations/memories),
  * Proactivity acceptance rate, correction rate (user says “that’s wrong”),
  * Memory precision\@k (human-rated), personalization satisfaction (CSAT).

---

## 13) Observability

* **Tracing**: Each turn MUST carry `trace_id`, `user_id`, `turn_id`.
* **Structured events**:

```json
{
  "event": "turn_completed",
  "trace_id": "...",
  "user_id": "user-123",
  "latency_ms": 2310,
  "used_rag": true,
  "mem_hits": 2,
  "moderation": "clean",
  "tts_engine": "x",
  "llm_model": "gpt-4o",
  "timestamp": "..."
}
```

* **Dashboards**: latency breakdown, error classes, safety blocks, proactivity outcomes.

---

## 14) Configuration & Secrets

* **12-factor** configs via env or config service.
* **Secrets** MUST be stored in a vault; rotation enforced.
* **Feature flags**: proactivity, domains, model selection, A/B prompts.

---

## 15) Data Retention

* **Personal memories**: default 12 months, adjustable per domain.
* **Conversation logs**: disabled by default; if enabled, 30 days & redacted.
* **Analytics (non-content)**: 13 months.
* **User export/delete** MUST be honored within 30 days.

---

## 16) Error Handling & UX

* **Graceful errors**: Apologize once; offer retry or alternate mode (text).
* **Ambiguity**: Ask concise clarifying questions or propose options.
* **Language**: Match user locale; fall back to English if uncertain.

---

## 17) Repository & Structure

```
/voice-agent
  /apps
    /gateway       # HTTP/gRPC ingress, auth, routing
    /orchestrator  # LangChain agent/chains, policies
    /asr           # adapters
    /tts           # adapters
    /retrieval     # RAG microservice
  /libs
    /memory        # LongTermMemory, schemas, writers
    /profile       # profile store & updaters
    /safety        # moderation and guardrails
    /prompts       # versioned system & tool prompts
    /embeddings    # embedding client
  /infra
    /terraform     # IaC
    /helm
  /tests
    /goldens
    /redteam
```

---

## 18) API Contracts (Internal)

### 18.1 Retrieval Microservice

`POST /retrieve`

```json
{
  "user_id":"user-123",
  "query":"best coffee grinder",
  "namespaces":["personal_memory","domain_knowledge"],
  "filters":{"domain":["preferences"],"max_age_days":365},
  "k":5
}
```

**Response**

```json
{"results":[{"text":"Prefers pour-over","score":0.82,"metadata":{"domain":"preferences","created_at":"..."}}]}
```

### 18.2 Memory Write

`POST /memory/write`

```json
{
  "user_id":"user-123",
  "entries":[{"type":"semantic","text":"Likes V60","domain":"preferences","sensitivity":"low"}]
}
```

---

## 19) LangChain Implementation Rules

* Prefer **Runnable** graph (LCEL) for deterministic composition.
* Memory:

  * `ConversationBufferWindowMemory(k=6)` for short-term.
  * Custom `VectorMemory` with Chroma retriever (k=3) for long-term.
* Tools:

  * Retrieval tools MUST accept `user_id`, `domain_filters`, `k`.
* Prompt templating MUST be versioned (`prompts/vX.Y/system.md`).
* Add a **“Memory Preview”** step (hidden) to show what will be stored; if sensitive, ask confirmation.

---

## 20) Security & Compliance

* AuthN/Z: **per-user tokens**; memory endpoints MUST verify `user_id`.
* Multi-tenant isolation in DB and indices (namespace per tenant).
* Rate limiting & abuse prevention; captcha or challenge for suspicious spikes.
* Compliance posture: GDPR-aligned controls (consent, portability, erasure).

---

## 21) Rollout, A/B, & Change Management

* **Staging** → **Canary (5%)** → **Gradual 25/50/100%** with rollback.
* A/B on:

  * memory writer prompt,
  * retrieval k,
  * proactivity thresholds,
  * prompt styles.
* Changes to prompts or safety rules MUST pass golden tests & red-team suite.

---

## 22) Operational Playbooks

* **Incident**: SEV triage (<15m), user comms if user-impacting, rollback option.
* **Key Rotations**: quarterly or on compromise.
* **Backup/Restore**: nightly snapshots for vector store and profile DB; tested monthly.

---

## 23) Developer Experience

* Local compose stack with fake STT/TTS and in-memory Chroma.
* Seed scripts: demo profiles, demo memories.
* Makefile tasks: `make test`, `make e2e`, `make redteam`, `make run-local`.

---

## 24) Acceptance Criteria (MVP+)

* **MVP**:

  * Persistent long-term memory scoped by consent.
  * Short-term windowed memory in prompt.
  * Basic RAG over a demo corpus.
  * Opt-in proactivity (level 1) with rate-limit.
  * Safety moderation on input/output.
  * P95 E2E latency ≤ 5s.
* **Plus**:

  * Memory export/delete UI.
  * Profile confirmation flow.
  * Golden conversation suite (≥30).
  * Red-team suite (≥50 prompts).

---

## 25) Defaults & Tunables

* Window size: 6 turns.
* LTM k: 3; RAG k: 3.
* Proactivity: off by default; level 1 when enabled.
* TTLs: semantic 365d, episodic 180d, procedural 365d.
* Quiet Hours: 22:00–07:00 user local time.

---

## 26) Documentation & Onboarding

* README MUST include: quick start, env vars, secrets, runbook.
* Architecture diagram (C4 level 2) and prompt map.
* Data schema docs (profile, memory, retrieval).

---

### Final Notes

* **Do not** store sensitive content without explicit consent.
* **Do ensure** every turn is explainable via traces (what memories/docs were used).
* **Do keep** prompts/versioning under review; prompt drift is a production risk.