diff --git a/.gitignore b/.gitignore index 659c46f..2e3da59 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,13 @@ temp/ .smriti/CLAUDE.md .smriti/knowledge/ .smriti/index.json + +# Personal writing / local-only notes +docs/writing/ + +# Letta Code agent state +.letta/ + +# Zsh plugins (should not be in project repo) +zsh-autosuggestions/ +zsh-syntax-highlighting/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 93fbf7b..a52b986 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -7,7 +7,7 @@ repos: name: Gitleaks - Detect secrets entry: gitleaks detect --source . -c .gitleaks.toml language: system - stages: [commit] + stages: [pre-commit] pass_filenames: false always_run: true diff --git a/README.md b/README.md index ba9e970..ad3ee91 100644 --- a/README.md +++ b/README.md @@ -2,19 +2,40 @@ Smriti — Shared memory for AI-powered engineering teams

-Built on top of [QMD](https://github.com/tobi/qmd) by Tobi Lütke. +

+ An exploration of memory in the agentic world +

--- -## The Problem +The agentic world is moving fast. Every team is shipping with AI — Claude Code, +Cursor, Codex, Cline. The agents are getting better. The tooling is maturing. + +But there's a gap nobody has fully closed: + +> **Agents don't remember.** + +Not from yesterday. Not from each other. Not from your teammates. Every session +starts from zero, no matter how much your team has already figured out. + +This isn't just a developer experience problem. It's a foundational gap in how +agents work. As they get more capable and longer-running, memory becomes a +prerequisite — not a feature. Without it, knowledge stays buried in chat +histories. Teams re-discover what they've already figured out. Decisions get +made twice. + +The answer, I think, mirrors how our own memory works: -Your team ships code with AI agents every day — Claude Code, Cursor, Codex. But -every agent has a blind spot: +> **Ingest → Categorize → Recall → Search** -> **They don't remember anything.** Not from yesterday. Not from each other. Not -> from your teammates. +That's the brain. That's what **Smriti** (Sanskrit: _memory_) is building +toward. -Here's what that looks like: +--- + +## The Problem, Up Close + +Here's what the gap looks like in practice: | Monday | Tuesday | | ------------------------------------------------------------- | --------------------------------------------------- | @@ -31,16 +52,17 @@ The result: - **Zero continuity** — each session starts from scratch, no matter how much your team has already figured out -The agents are brilliant. But they're amnesic. **This is the biggest gap in -AI-assisted development today.** +The agents are brilliant. But they're amnesic. **This is the biggest unsolved +gap in agentic AI today.** + +--- ## What Smriti Does -**Smriti** (Sanskrit: _memory_) is a shared memory layer that sits underneath -all your AI agents. +Smriti is a shared memory layer that sits underneath your AI agents. -Every conversation → automatically captured → indexed → -searchable. One command to recall what matters. +Every conversation → automatically captured → indexed → searchable. One command +to recall what matters. ```bash # What did we figure out about the auth migration? @@ -53,12 +75,67 @@ smriti list --project myapp smriti search "rate limiting strategy" --project api-service ``` -> **20,000 tokens** of past conversations → **500 tokens** of relevant -> context. Your agents get what they need without blowing up your token budget. +> **20,000 tokens** of past conversations → **500 tokens** of relevant context. +> Your agents get what they need without blowing up your token budget. -## The Workflow +Built on top of [QMD](https://github.com/tobi/qmd) by Tobi Lütke. Everything +runs locally — no cloud, no accounts, no telemetry. + +--- + +## Install + +**macOS / Linux:** + +```bash +curl -fsSL https://raw.githubusercontent.com/zero8dotdev/smriti/main/install.sh | bash +``` + +**Windows** (PowerShell): + +```powershell +irm https://raw.githubusercontent.com/zero8dotdev/smriti/main/install.ps1 | iex +``` + +Both installers will: + +- Install [Bun](https://bun.sh) if you don't have it +- Clone Smriti to `~/.smriti` +- Set up the `smriti` CLI on your PATH +- Configure the Claude Code auto-save hook + +**Requirements:** macOS, Linux, or Windows 10+ · Git · Bun ≥ 1.1 +(auto-installed) · Ollama (optional, for synthesis) + +```bash +smriti upgrade # update to latest +``` + +--- + +## Quick Start + +```bash +# 1. Ingest your recent Claude Code sessions +smriti ingest claude + +# 2. Search what your team has discussed +smriti search "database connection pooling" + +# 3. Recall with synthesis into one coherent summary (requires Ollama) +smriti recall "how did we handle rate limiting" --synthesize + +# 4. Share knowledge with your team through git +smriti share --project myapp +git add .smriti && git commit -m "chore: share session knowledge" + +# 5. Teammates pull it in +smriti sync --project myapp +``` + +--- -Here's what changes when your team runs Smriti: +## The Workflow **1. Conversations are captured automatically** @@ -101,49 +178,11 @@ teammates pull it and import it into their local memory. No cloud service, no account, no sync infrastructure — just git. ```bash -# Share what you've learned smriti share --project myapp --category decision - -# Pull in what others have shared smriti sync --project myapp ``` -## Install - -**macOS / Linux:** - -```bash -curl -fsSL https://raw.githubusercontent.com/zero8dotdev/smriti/main/install.sh | bash -``` - -**Windows** (PowerShell): - -```powershell -irm https://raw.githubusercontent.com/zero8dotdev/smriti/main/install.ps1 | iex -``` - -Both installers will: - -- Install [Bun](https://bun.sh) if you don't have it -- Clone Smriti to `~/.smriti` -- Set up the `smriti` CLI on your PATH -- Configure the Claude Code auto-save hook - -### Requirements - -- **macOS, Linux, or Windows 10+** -- **Git** -- **Bun** >= 1.1 (installed automatically) -- **Ollama** (optional — for local summarization and synthesis) - -### Upgrade - -```bash -smriti upgrade -``` - -Pulls the latest version from GitHub and reinstalls dependencies. Equivalent to -re-running the install script. +--- ## Commands @@ -175,8 +214,8 @@ smriti categorize # Auto-categorize sessions smriti projects # List all tracked projects smriti upgrade # Update smriti to the latest version -# Context and comparison -smriti context # Generate project context for .smriti/CLAUDE.md +# Context injection +smriti context # Generate project context → .smriti/CLAUDE.md smriti context --dry-run # Preview without writing smriti compare --last # Compare last 2 sessions (tokens, tools, files) smriti compare # Compare specific sessions @@ -187,6 +226,8 @@ smriti sync # Import teammates' shared knowledge smriti team # View team contributions ``` +--- + ## How It Works ``` @@ -221,308 +262,7 @@ Claude Code Cursor Codex Other Agents Everything runs locally. Your conversations never leave your machine. The SQLite database, the embeddings, the search indexes — all on disk, all yours. -## Ingest Architecture - -Smriti ingest uses a layered pipeline: - -1. `parsers/*` extract agent transcripts into normalized messages (no DB writes). -2. `session-resolver` derives project/session state, including incremental offsets. -3. `store-gateway` persists messages, sidecars, session meta, and costs. -4. `ingest/index.ts` orchestrates the flow with per-session error isolation. - -This keeps parser logic, resolution logic, and persistence logic separated and testable. -See `INGEST_ARCHITECTURE.md` and `src/ingest/README.md` for implementation details. - -## Tagging & Categories - -Sessions and messages are automatically tagged into a hierarchical category -tree. Tags flow through every command — search, recall, list, and share — so you -can slice your team's knowledge by topic. - -### Default Category Tree - -Smriti ships with 7 top-level categories and 21 subcategories: - -| Category | Subcategories | -| -------------- | ----------------------------------------------------------------------- | -| `code` | `code/implementation`, `code/pattern`, `code/review`, `code/snippet` | -| `architecture` | `architecture/design`, `architecture/decision`, `architecture/tradeoff` | -| `bug` | `bug/report`, `bug/fix`, `bug/investigation` | -| `feature` | `feature/requirement`, `feature/design`, `feature/implementation` | -| `project` | `project/setup`, `project/config`, `project/dependency` | -| `decision` | `decision/technical`, `decision/process`, `decision/tooling` | -| `topic` | `topic/learning`, `topic/explanation`, `topic/comparison` | - -### Auto-Classification - -Smriti uses a two-stage pipeline to classify messages: - -1. **Rule-based** — 24 keyword patterns with weighted confidence scoring. Each - pattern targets a specific subcategory (e.g., words like "crash", - "stacktrace", "panic" map to `bug/report`). Confidence is calculated from - keyword density and rule weight. -2. **LLM fallback** — When rule confidence falls below the threshold (default - `0.5`, configurable via `SMRITI_CLASSIFY_THRESHOLD`), Ollama classifies the - message. Only activated when you pass `--llm`. - -The most frequent category across a session's messages becomes the session-level -tag. - -```bash -# Auto-categorize all uncategorized sessions (rule-based) -smriti categorize - -# Include LLM fallback for ambiguous sessions -smriti categorize --llm - -# Categorize a specific session -smriti categorize --session -``` - -### Manual Tagging - -Override or supplement auto-classification with manual tags: - -```bash -smriti tag - -# Examples -smriti tag abc123 decision/technical -smriti tag abc123 bug/fix -``` - -Manual tags are stored with confidence `1.0` and source `"manual"`. - -### Custom Categories - -Add your own categories to extend the default tree: - -```bash -# List the full category tree -smriti categories - -# Add a top-level category -smriti categories add ops --name "Operations" - -# Add a nested category under an existing parent -smriti categories add ops/incident --name "Incident Response" --parent ops - -# Include a description -smriti categories add ops/runbook --name "Runbooks" --parent ops --description "Operational runbook sessions" -``` - -### How Tags Filter Commands - -The `--category` flag works across search, recall, list, and share: - -| Command | Effect of `--category` | -| --------------- | ------------------------------------------------------------------------------- | -| `smriti list` | Shows categories column; filters sessions to matching category | -| `smriti search` | Filters full-text search results to matching category | -| `smriti recall` | Filters recall context; works with `--synthesize` | -| `smriti share` | Controls which sessions are exported; files organized into `.smriti/knowledge/` | -| `smriti status` | Shows session count per category (no filter flag — always shows all) | - -**Hierarchical filtering** — Filtering by a parent category automatically -includes all its children. `--category decision` matches `decision/technical`, -`decision/process`, and `decision/tooling`. - -### Categories in Share & Sync - -**Categories survive the share/sync roundtrip exactly.** What gets serialized -during `smriti share` is exactly what gets deserialized during `smriti sync` — -the same category ID goes in, the same category ID comes out. No -reclassification, no transformation, no loss. The category a session was tagged -with on one machine is the category it will be indexed under on every other -machine that syncs it. - -When you share sessions, the category is embedded in YAML frontmatter inside -each exported markdown file: - -```yaml ---- -id: 2e5f420a-e376-4ad4-8b35-ad94838cbc42 -category: project -project: smriti -agent: claude-code -author: zero8 -shared_at: 2026-02-10T11:29:44.501Z -tags: ["project", "project/dependency"] --- -``` - -When a teammate runs `smriti sync`, the frontmatter is parsed and the category -is restored into their local `smriti_session_tags` table — indexed as `project`, -searchable as `project`, filterable as `project`. The serialization and -deserialization are symmetric: `share` writes `category: project` → `sync` reads -`category: project` → `tagSession(db, sessionId, "project", 1.0, "team")`. No -intermediate step reinterprets the value. - -Files are organized into subdirectories by primary category (e.g., -`.smriti/knowledge/project/`, `.smriti/knowledge/decision/`), but sync reads the -category from frontmatter, not the directory path. - -> **Note:** Currently only the primary `category` field is restored on sync. -> Secondary tags in the `tags` array are serialized in the frontmatter but not -> yet imported. If a session had multiple tags (e.g., `project` + -> `decision/tooling`), only the primary tag survives the roundtrip. - -```bash -# Share decisions — category metadata travels with the files -smriti share --project myapp --category decision - -# Teammate syncs — categories restored exactly from frontmatter -smriti sync --project myapp -``` - -### Examples - -```bash -# All architectural decisions -smriti search "database" --category architecture - -# Recall only bug-related context -smriti recall "connection timeout" --category bug --synthesize - -# List feature sessions for a specific project -smriti list --category feature --project myapp - -# Share only decision sessions -smriti share --project myapp --category decision -``` - -## Context: Token Reduction (North Star) - -Every new Claude Code session starts from zero — no awareness of what happened -yesterday, which files were touched, what decisions were made. `smriti context` -generates a compact project summary (~200-300 tokens) and injects it into -`.smriti/CLAUDE.md`, which Claude Code auto-discovers. - -```bash -smriti context # auto-detect project, write .smriti/CLAUDE.md -smriti context --dry-run # preview without writing -smriti context --project myapp # explicit project -smriti context --days 14 # 14-day lookback (default: 7) -``` - -The output looks like this: - -```markdown -## Project Context - -> Auto-generated by `smriti context` on 2026-02-11. Do not edit manually. - -### Recent Sessions (last 7 days) - -- **2h ago** Enriched ingestion pipeline (12 turns) [code] -- **1d ago** Search & recall pipeline (8 turns) [feature] - -### Hot Files - -`src/db.ts` (14 ops), `src/ingest/claude.ts` (11 ops), `src/search/index.ts` (8 -ops) - -### Git Activity - -- commit `main`: "Fix auth token refresh" (2026-02-10) - -### Usage - -5 sessions, 48 turns, ~125K input / ~35K output tokens -``` - -No Ollama, no network calls, no model loading. Pure SQL queries against sidecar -tables, rendered as markdown. Runs in < 100ms. - -### Measuring the Impact - -Does this actually save tokens? Honestly — we don't know yet. We built the tools -to measure it, ran A/B tests, and the results so far are... humbling. Claude is -annoyingly good at finding the right files even without help. - -But this is the north star, not the destination. We believe context injection -will matter most on large codebases without detailed docs, ambiguous tasks that -require exploration, and multi-session continuity. We just need the data to -prove it (or disprove it and try something else). - -So we're shipping the measurement tools and asking you to help. Run A/B tests on -your projects, paste the results in -[Issue #13](https://github.com/zero8dotdev/smriti/issues/13), and let's figure -this out together. - -#### A/B Testing Guide - -```bash -# Step 1: Baseline session (no context) -mv .smriti/CLAUDE.md .smriti/CLAUDE.md.bak -# Start a Claude Code session, give it a task, let it finish, exit - -# Step 2: Context session -mv .smriti/CLAUDE.md.bak .smriti/CLAUDE.md -smriti context -# Start a new session, give the EXACT same task, let it finish, exit - -# Step 3: Ingest and compare -smriti ingest claude -smriti compare --last --project myapp -``` - -#### Compare Command - -```bash -smriti compare # by session ID (supports partial IDs) -smriti compare --last # last 2 sessions for current project -smriti compare --last --project myapp # last 2 sessions for specific project -smriti compare --last --json # machine-readable output -``` - -Output: - -``` -Session A: Fix auth bug (no context) -Session B: Fix auth bug (with context) - -Metric A B Diff ----------------------------------------------------------------- -Turns 12 8 -4 (-33%) -Total tokens 45K 32K -13000 (-29%) -Tool calls 18 11 -7 (-39%) -File reads 10 4 -6 (-60%) - -Tool breakdown: - Bash 4 3 - Glob 3 0 - Read 10 4 - Write 1 4 -``` - -#### What We've Tested So Far - -| Task Type | Context Impact | Notes | -| ----------------------------------------- | -------------- | ---------------------------------------------------------------------- | -| Knowledge questions ("how does X work?") | Minimal | Both sessions found the right files immediately from project CLAUDE.md | -| Implementation tasks ("add --since flag") | Minimal | Small, well-scoped tasks don't need exploration | -| Ambiguous/exploration tasks | Untested | Expected sweet spot — hot files guide Claude to the right area | -| Large codebases (no project CLAUDE.md) | Untested | Expected sweet spot — context replaces missing documentation | - -**We need your help.** If you run A/B tests on your projects, please share your -results in [GitHub Issues](https://github.com/zero8dotdev/smriti/issues). -Include the `smriti compare` output and a description of the task. This data -will help us understand where context injection actually matters. - -### Token Savings (Search & Recall) - -Separate from context injection, Smriti's search and recall pipeline compresses -past conversations: - -| Scenario | Raw Conversations | Via Smriti | Reduction | -| ----------------------------------- | ----------------- | ----------- | --------- | -| Relevant context from past sessions | ~20,000 tokens | ~500 tokens | **40x** | -| Multi-session recall + synthesis | ~10,000 tokens | ~200 tokens | **50x** | -| Full project conversation history | 50,000+ tokens | ~500 tokens | **100x** | - -Lower token spend, faster responses, more room for the actual work in your -context window. ## Privacy @@ -534,68 +274,87 @@ Smriti is local-first by design. No cloud, no telemetry, no accounts. - Synthesis via local [Ollama](https://ollama.ai) (optional) - Team sharing happens through git — you control what gets committed +--- + ## FAQ -**When does knowledge get captured?** Automatically. Smriti hooks into your AI -coding tool (Claude Code, Cursor, etc.) and captures every session without any -manual step. You just code normally and `smriti ingest` pulls in the -conversations. +**When does knowledge get captured?** Automatically. Smriti hooks into Claude +Code and captures every session without any manual step. For other agents, run +`smriti ingest all` to pull in conversations on demand. **Who has access to my data?** Only you. Everything lives in a local SQLite -database (`~/.cache/qmd/index.sqlite`). There's no cloud, no accounts, no -telemetry. Team sharing is explicit — you run `smriti share` to export, commit -the `.smriti/` folder to git, and teammates run `smriti sync` to import. +database. There's no cloud, no accounts, no telemetry. Team sharing is +explicit — you run `smriti share`, commit the `.smriti/` folder, and teammates +run `smriti sync`. **Can AI agents query the knowledge base?** Yes. `smriti recall "query"` returns -relevant past context that agents can use. When you run `smriti share`, it -generates a `.smriti/CLAUDE.md` index so Claude Code automatically discovers -shared knowledge. Agents can search, grep, and recall from the full knowledge -base. +relevant past context. `smriti share` generates a `.smriti/CLAUDE.md` so Claude +Code automatically discovers shared knowledge at the start of every session. **How do multiple projects stay separate?** Each project gets its own `.smriti/` -folder in its repo root. Sessions are tagged with project IDs in the central -database. Search works cross-project by default, but you can scope to a single -project with `--project `. Knowledge shared via git stays within that -project's repo. +folder. Sessions are tagged with project IDs in the central database. Search +works cross-project by default, scoped with `--project `. **Does this work with Jira or other issue trackers?** Not yet — Smriti is -git-native today. Issue tracker integrations are on the roadmap. If you have -ideas, open a discussion in -[GitHub Issues](https://github.com/zero8dotdev/smriti/issues). +git-native today. Issue tracker integrations are on the roadmap. -**How does this help preserve existing features during changes?** The reasoning -behind each code change is captured and searchable. When an AI agent starts a -new session, it can recall _why_ something was built a certain way — reducing -the chance of accidentally breaking existing behavior. +**Further reading:** See [docs/cli.md](./docs/cli.md) for the full command +reference, [INGEST_ARCHITECTURE.md](./INGEST_ARCHITECTURE.md) for the ingestion +pipeline, and [CLAUDE.md](./CLAUDE.md) for the database schema and +architecture. -## Uninstall +--- -```bash -curl -fsSL https://raw.githubusercontent.com/zero8dotdev/smriti/main/uninstall.sh | bash -``` +## About -To also remove hook state, prepend `SMRITI_PURGE=1` to the command. +I've been coding with AI agents for about 8 months. At some point the +frustration became impossible to ignore — every new session, you start from +zero. Explaining the same context, the same decisions, the same constraints. +That's not a great developer experience. + +I started small: custom prompts to export Claude sessions. That grew old fast. I +needed categorization. Found QMD. Started building on top of it. Dogfooded it. +Hit walls. Solved one piece at a time. + +At some point it worked well enough that I shared it with some friends. Some +used it, some ignored it — fair, the AI tooling space is noisy. But I kept +exploring, and found others building toward the same problem: Claude-mem, Letta, +a growing community of people who believe memory is the next foundational layer +for AI. -## Documentation +That's what Smriti is, really. An exploration. The developer tool is one layer. +But the deeper question is: what does memory for autonomous agents actually need +to look like? The answer probably mirrors how our own brain works — **Ingest → +Categorize → Recall → Search**. We're figuring that out, one piece at a time. -See [CLAUDE.md](./CLAUDE.md) for the full reference — API docs, database schema, -architecture details, and troubleshooting. +I come from the developer tooling space. Bad tooling bothers me. There's always +a better way. This is that project. + +--- ## Special Thanks Smriti is built on top of [QMD](https://github.com/tobi/qmd) — a beautifully designed local search engine for markdown files created by -[Tobi Lütke](https://github.com/tobi), CEO of Shopify. +[Tobi Lütke](https://github.com/tobi), CEO of Shopify. QMD gave us fast, +local-first SQLite with full-text search, vector embeddings, and +content-addressable hashing — all on your machine, zero cloud dependencies. +Instead of rebuilding that infrastructure from scratch, we focused entirely on +the memory layer, multi-agent ingestion, and team sharing. -QMD gave us the foundation we needed: a fast, local-first SQLite store with -full-text search, vector embeddings, and content-addressable hashing — all -running on your machine with zero cloud dependencies. Instead of rebuilding that -infrastructure from scratch, we were able to focus entirely on the memory layer, -multi-agent ingestion, and team sharing that makes Smriti useful. +Thank you, Tobi, for open-sourcing it. + +--- -Thank you, Tobi, for open-sourcing QMD. It's a reminder that the best tools are -often the ones that quietly do the hard work so others can build something new -on top. +## Uninstall + +```bash +curl -fsSL https://raw.githubusercontent.com/zero8dotdev/smriti/main/uninstall.sh | bash +``` + +To also remove hook state, prepend `SMRITI_PURGE=1` to the command. + +--- ## License diff --git a/docs/architecture.md b/docs/architecture.md index 5ec1d33..1740953 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,149 +1,139 @@ # Architecture -## Overview +Smriti's architecture follows the same pattern as memory in your brain: +**Ingest → Categorize → Recall → Search**. -``` - Claude Code Cursor Codex Other Agents - | | | | - v v v v - ┌──────────────────────────────────────────┐ - │ Smriti Ingestion Layer │ - │ │ - │ src/ingest/claude.ts (JSONL parser) │ - │ src/ingest/codex.ts (JSONL parser) │ - │ src/ingest/cursor.ts (JSON parser) │ - │ src/ingest/generic.ts (file import) │ - └──────────────────┬───────────────────────┘ - │ - v - ┌──────────────────────────────────────────┐ - │ QMD Core (via src/qmd.ts) │ - │ │ - │ addMessage() content-addressed │ - │ searchMemoryFTS() BM25 full-text │ - │ searchMemoryVec() vector similarity │ - │ recallMemories() dedup + synthesis │ - └──────────────────┬───────────────────────┘ - │ - v - ┌──────────────────────────────────────────┐ - │ SQLite Database │ - │ ~/.cache/qmd/index.sqlite │ - │ │ - │ QMD tables: │ - │ memory_sessions memory_messages │ - │ memory_fts content_vectors │ - │ │ - │ Smriti tables: │ - │ smriti_session_meta (agent, project) │ - │ smriti_projects (registry) │ - │ smriti_categories (taxonomy) │ - │ smriti_session_tags (categorization) │ - │ smriti_message_tags (categorization) │ - │ smriti_shares (team dedup) │ - └──────────────────────────────────────────┘ -``` +Every layer has one job. Parsers extract conversations. The resolver maps +them to projects. The store persists them. Search retrieves them. Nothing +crosses those boundaries. -## QMD Integration +--- -Smriti builds on top of [QMD](https://github.com/tobi/qmd), a local-first search engine. QMD provides: - -- **Content-addressable storage** — Messages are SHA256-hashed, no duplicates -- **FTS5 full-text search** — BM25 ranking with Porter stemming -- **Vector embeddings** — 384-dim vectors via embeddinggemma (node-llama-cpp) -- **Reciprocal Rank Fusion** — Combines FTS and vector results +## System Overview -All QMD imports go through a single re-export hub at `src/qmd.ts`: - -```ts -// Every file imports from here, never from qmd directly -import { addMessage, searchMemoryFTS, recallMemories } from "./qmd"; -import { hashContent } from "./qmd"; -import { ollamaRecall } from "./qmd"; +``` +Claude Code Cursor Codex Cline Copilot + | | | | | + v v v v v +┌──────────────────────────────────────────────┐ +│ Smriti Ingestion Layer │ +│ │ +│ parsers/claude.ts (JSONL) │ +│ parsers/codex.ts (JSONL) │ +│ parsers/cursor.ts (JSON) │ +│ parsers/cline.ts (task files) │ +│ parsers/copilot.ts (VS Code storage) │ +│ parsers/generic.ts (file import) │ +│ │ +│ session-resolver.ts (project detection) │ +│ store-gateway.ts (persistence) │ +└──────────────────┬───────────────────────────┘ + │ + v +┌──────────────────────────────────────────────┐ +│ QMD Core (via src/qmd.ts) │ +│ │ +│ addMessage() content-addressed │ +│ searchMemoryFTS() BM25 full-text │ +│ searchMemoryVec() vector similarity │ +│ recallMemories() dedup + synthesis │ +└──────────────────┬───────────────────────────┘ + │ + v +┌──────────────────────────────────────────────┐ +│ SQLite (~/.cache/qmd/index.sqlite) │ +│ │ +│ QMD tables: │ +│ memory_sessions memory_messages │ +│ memory_fts (BM25) content_vectors │ +│ │ +│ Smriti tables: │ +│ smriti_session_meta (agent, project) │ +│ smriti_projects (registry) │ +│ smriti_categories (taxonomy) │ +│ smriti_session_tags (categorization) │ +│ smriti_message_tags (categorization) │ +│ smriti_shares (team dedup) │ +└──────────────────────────────────────────────┘ ``` -This creates a clean boundary — if QMD's API changes, only `src/qmd.ts` needs updating. +Everything runs locally. Nothing leaves your machine. -## Ingestion Pipeline +--- -Each agent has a dedicated parser. The flow: +## Built on QMD -1. **Discover** — Glob for session files in agent-specific log directories -2. **Deduplicate** — Check `smriti_session_meta` for already-ingested session IDs -3. **Parse** — Agent-specific parsing into a common `ParsedMessage[]` format -4. **Store** — Save via QMD's `addMessage()` (content-addressed, SHA256 hashed) -5. **Annotate** — Attach Smriti metadata (agent ID, project ID) to `smriti_session_meta` +Smriti builds on [QMD](https://github.com/tobi/qmd) — a local-first search +engine for markdown files by Tobi Lütke. QMD handles the hard parts: -### Project Detection (Claude Code) +- **Content-addressable storage** — messages are SHA256-hashed, no duplicates +- **FTS5 full-text search** — BM25 ranking with Porter stemming +- **Vector embeddings** — 384-dim via EmbeddingGemma (node-llama-cpp), + computed entirely on-device +- **Reciprocal Rank Fusion** — combines FTS and vector results -Claude Code stores sessions in `~/.claude/projects//`. The directory name encodes the filesystem path with `-` replacing `/`: +All QMD imports go through a single re-export hub at `src/qmd.ts`. No file +in the codebase imports from QMD directly — only through this hub. If QMD's +API changes, one file needs updating. +```ts +import { addMessage, searchMemoryFTS, recallMemories } from "./qmd"; +import { hashContent, ollamaRecall } from "./qmd"; ``` --Users-zero8-zero8.dev-openfga → /Users/zero8/zero8.dev/openfga -``` - -Since folder names can also contain dashes, `deriveProjectPath()` uses greedy `existsSync()` matching: it tries candidate paths from left to right, picking the longest existing directory at each step. -`deriveProjectId()` then strips the configured `PROJECTS_ROOT` (default `~/zero8.dev`) to produce a clean project name like `openfga` or `avkash/regulation-hub`. +--- -## Search Architecture - -### Filtered Search +## Ingestion Pipeline -`searchFiltered()` in `src/search/index.ts` extends QMD's FTS5 search with JOINs to Smriti's metadata tables: +Ingestion is a four-stage pipeline with clean separation between stages: -```sql -FROM memory_fts mf -JOIN memory_messages mm ON mm.rowid = mf.rowid -JOIN memory_sessions ms ON ms.id = mm.session_id -LEFT JOIN smriti_session_meta sm ON sm.session_id = mm.session_id -WHERE mf.content MATCH ? - AND sm.project_id = ? -- project filter - AND sm.agent_id = ? -- agent filter - AND EXISTS (...) -- category filter via smriti_message_tags -``` +1. **Parse** — agent-specific parsers extract conversations into a normalized + `ParsedMessage[]` format. No DB writes, no side effects. Pure functions. +2. **Resolve** — `session-resolver.ts` maps sessions to projects, handles + incremental ingestion (picks up where it left off), derives clean project + IDs from agent-specific path formats. +3. **Store** — `store-gateway.ts` persists messages, session metadata, + sidecars, and cost data. All writes go through here. +4. **Orchestrate** — `ingest/index.ts` drives the flow with per-session error + isolation. One broken session doesn't stop the rest. -### Recall +### Project Detection -`recall()` in `src/search/recall.ts` wraps search with: +Claude Code encodes project paths into directory names like +`-Users-zero8-zero8.dev-openfga` (slashes become dashes). Since folder +names can also contain real dashes, `deriveProjectPath()` uses greedy +`existsSync()` matching — trying candidate paths left to right, picking the +longest valid directory at each step. -1. **Session deduplication** — Keep only the best-scoring result per session -2. **Optional synthesis** — Sends results to Ollama's `ollamaRecall()` for a coherent summary +`deriveProjectId()` then strips `SMRITI_PROJECTS_ROOT` to produce a clean +name: `openfga`, `avkash/regulation-hub`. -When no filters are specified, it delegates directly to QMD's native `recallMemories()`. +--- -## Team Sharing +## Search -### Export (`smriti share`) +Smriti adds a metadata filter layer on top of QMD's native search: -Sessions are exported as markdown files with YAML frontmatter: - -``` -.smriti/ -├── config.json -├── index.json # Manifest of all shared files -└── knowledge/ - ├── decision/ - │ └── 2026-02-10_auth-migration-approach.md - └── bug/ - └── 2026-02-09_connection-pool-fix.md -``` +**`smriti search`** — FTS5 full-text with JOINs to Smriti's metadata tables. +Filters by project, agent, and category without touching the vector index. +Fast, synchronous, no model loading. -Each file contains: -- YAML frontmatter (session ID, category, project, agent, author, tags) -- Session title as heading -- Summary (if available) -- Full conversation in `**role**: content` format +**`smriti recall`** — Two paths depending on whether filters are applied: -Content hashes prevent re-exporting the same content. +- *No filters* → delegates to QMD's native `recallMemories()`: FTS + vector + + Reciprocal Rank Fusion + session dedup. Full hybrid pipeline. +- *With filters* → filtered FTS search + session dedup. Vector search is + currently bypassed when filters are active. (This is a known gap — see + [search.md](./search.md) for details.) -### Import (`smriti sync`) +**`smriti embed`** — builds vector embeddings for all unembedded messages. +Required before vector search works. Runs locally via node-llama-cpp. -Reads markdown files from `.smriti/knowledge/`, parses frontmatter and conversation, and imports via `addMessage()`. Content hashing prevents duplicate imports. +--- ## Database Schema -### QMD Tables (not modified by Smriti) +### QMD Tables | Table | Purpose | |-------|---------| @@ -156,10 +146,26 @@ Reads markdown files from `.smriti/knowledge/`, parses frontmatter and conversat | Table | Purpose | |-------|---------| -| `smriti_agents` | Agent registry (claude-code, codex, cursor) | +| `smriti_agents` | Agent registry (claude-code, codex, cursor...) | | `smriti_projects` | Project registry (id, filesystem path) | | `smriti_session_meta` | Maps sessions to agents and projects | | `smriti_categories` | Hierarchical category taxonomy | -| `smriti_session_tags` | Category tags on sessions (with confidence) | -| `smriti_message_tags` | Category tags on messages (with confidence) | +| `smriti_session_tags` | Category tags on sessions (with confidence score) | +| `smriti_message_tags` | Category tags on messages (with confidence score) | | `smriti_shares` | Deduplication tracking for team sharing | + +--- + +## Team Sharing + +Export (`smriti share`) converts sessions to markdown with YAML frontmatter +and writes them to `.smriti/knowledge/`, organized by category. The YAML +carries session ID, category, project, agent, author, and tags — enough to +reconstruct the full metadata on import. + +Import (`smriti sync`) parses frontmatter, restores categories, and inserts +via `addMessage()`. Content hashing prevents duplicate imports. The +roundtrip is symmetric: what gets written during share is exactly what gets +read during sync. + +See [team-sharing.md](./team-sharing.md) for the workflow. diff --git a/docs/cli.md b/docs/cli.md index 25b5071..13f1116 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,40 +1,82 @@ -# CLI Reference +# Smriti CLI Reference + +Everything you can do with `smriti`. For the big picture, see the +[README](../README.md). + +--- + +## Global Flags + +```bash +smriti --version # Print version +smriti --help # Print command overview +smriti help # Same as --help +``` + +--- + +## Global Filters + +These flags work across `search`, `recall`, `list`, and `share`: + +| Flag | Description | +|------|-------------| +| `--category ` | Filter by category (e.g. `decision`, `bug/fix`) | +| `--project ` | Filter by project ID | +| `--agent ` | Filter by agent (`claude-code`, `codex`, `cursor`, `cline`, `copilot`) | +| `--limit ` | Max results returned | +| `--json` | Machine-readable JSON output | + +Hierarchical category filtering: `--category decision` matches `decision`, +`decision/technical`, `decision/process`, and `decision/tooling`. + +--- ## Ingestion ### `smriti ingest ` -Import conversations from an AI agent into Smriti's memory. +Pull conversations from an AI agent into Smriti's memory. -| Agent | Source | Format | -|-------|--------|--------| -| `claude` / `claude-code` | `~/.claude/projects/*/*.jsonl` | JSONL | -| `codex` | `~/.codex/**/*.jsonl` | JSONL | -| `cursor` | `.cursor/**/*.json` (requires `--project-path`) | JSON | -| `file` / `generic` | Any file path | Chat or JSONL | -| `all` | All known agents at once | — | +| Agent | Source | +|-------|--------| +| `claude` / `claude-code` | `~/.claude/projects/*/*.jsonl` | +| `codex` | `~/.codex/**/*.jsonl` | +| `cline` | `~/.cline/tasks/**` | +| `copilot` | VS Code `workspaceStorage` (auto-detected per OS) | +| `cursor` | `.cursor/**/*.json` (requires `--project-path`) | +| `file` / `generic` | Any file path | +| `all` | All known agents at once | ```bash smriti ingest claude smriti ingest codex +smriti ingest cline +smriti ingest copilot smriti ingest cursor --project-path /path/to/project smriti ingest file ~/transcript.txt --title "Planning Session" --format chat smriti ingest all ``` **Options:** -- `--project-path ` — Project directory (required for Cursor) -- `--file ` — File path (for generic ingest) -- `--format ` — File format (default: `chat`) -- `--title ` — Session title -- `--session ` — Custom session ID -- `--project ` — Assign to a project -## Search +| Flag | Description | +|------|-------------| +| `--project-path ` | Project directory (required for Cursor) | +| `--file ` | File path (alternative to positional arg for generic ingest) | +| `--format ` | File format (default: `chat`) | +| `--title ` | Session title override | +| `--session ` | Custom session ID | +| `--project ` | Assign ingested sessions to a specific project | + +--- + +## Search & Recall ### `smriti search ` -Hybrid search across all memory using BM25 full-text and vector similarity. +Hybrid full-text + vector search across all memory. Returns ranked results +with session and message context. ```bash smriti search "rate limiting" @@ -43,51 +85,59 @@ smriti search "deployment" --category decision --limit 10 smriti search "API design" --json ``` -**Options:** -- `--category ` — Filter by category -- `--project ` — Filter by project -- `--agent ` — Filter by agent (`claude-code`, `codex`, `cursor`) -- `--limit ` — Max results (default: 20) -- `--json` — JSON output +**Options:** All global filters apply. + +--- ### `smriti recall ` -Smart recall: searches, deduplicates by session, and optionally synthesizes results into a coherent summary. +Like search, but deduplicates results by session and optionally synthesizes +them into a single coherent summary via Ollama. ```bash smriti recall "how did we handle caching" smriti recall "database setup" --synthesize smriti recall "auth flow" --synthesize --model qwen3:0.5b --max-tokens 200 -smriti recall "deployment" --project api --json +smriti recall "deployment" --category decision --project api --json ``` **Options:** -- `--synthesize` — Synthesize results into one summary via Ollama -- `--model ` — Ollama model for synthesis (default: `qwen3:8b-tuned`) -- `--max-tokens ` — Max synthesis output tokens -- All filter options from `search` + +| Flag | Description | +|------|-------------| +| `--synthesize` | Synthesize results into one summary via Ollama (requires Ollama running) | +| `--model ` | Ollama model to use (default: `qwen3:8b-tuned`) | +| `--max-tokens ` | Max tokens for synthesized output | +| All global filters | `--category`, `--project`, `--agent`, `--limit`, `--json` | + +--- ## Sessions ### `smriti list` -List recent sessions with optional filtering. +List recent sessions with filtering. ```bash smriti list smriti list --project myapp --agent claude-code smriti list --category decision --limit 20 -smriti list --all --json +smriti list --all +smriti list --json ``` **Options:** -- `--all` — Include inactive sessions -- `--json` — JSON output -- All filter options from `search` + +| Flag | Description | +|------|-------------| +| `--all` | Include inactive/archived sessions | +| All global filters | `--category`, `--project`, `--agent`, `--limit`, `--json` | + +--- ### `smriti show ` -Display all messages in a session. +Display all messages in a session. Supports partial session IDs. ```bash smriti show abc12345 @@ -95,15 +145,27 @@ smriti show abc12345 --limit 10 smriti show abc12345 --json ``` +**Options:** + +| Flag | Description | +|------|-------------| +| `--limit ` | Max messages to display | +| `--json` | JSON output | + +--- + ### `smriti status` -Memory statistics: session counts, message counts, agent breakdowns, project breakdowns, category distribution. +Memory statistics: total sessions, messages, agent breakdown, project +breakdown, category distribution. ```bash smriti status smriti status --json ``` +--- + ### `smriti projects` List all registered projects. @@ -113,11 +175,28 @@ smriti projects smriti projects --json ``` +--- + +## Embeddings + +### `smriti embed` + +Build vector embeddings for all unembedded messages. Required for semantic +(vector) search to work. Runs locally via `node-llama-cpp` — no network +calls. + +```bash +smriti embed +``` + +--- + ## Categorization ### `smriti categorize` -Auto-categorize uncategorized sessions using rule-based matching and optional LLM classification. +Auto-categorize uncategorized sessions using rule-based keyword matching with +an optional LLM fallback for ambiguous cases. ```bash smriti categorize @@ -126,60 +205,159 @@ smriti categorize --llm ``` **Options:** -- `--session ` — Categorize a specific session only -- `--llm` — Use Ollama LLM for ambiguous classifications + +| Flag | Description | +|------|-------------| +| `--session ` | Categorize a specific session only | +| `--llm` | Enable Ollama LLM fallback for low-confidence classifications | + +--- ### `smriti tag ` -Manually tag a session with a category. +Manually assign a category to a session. Stored with confidence `1.0` and +source `"manual"`. ```bash smriti tag abc12345 decision/technical smriti tag abc12345 bug/fix ``` +--- + ### `smriti categories` -Show the category tree. +Display the full category tree. ```bash smriti categories ``` +**Default categories:** + +| Category | Subcategories | +|----------|---------------| +| `code` | `code/implementation`, `code/pattern`, `code/review`, `code/snippet` | +| `architecture` | `architecture/design`, `architecture/decision`, `architecture/tradeoff` | +| `bug` | `bug/report`, `bug/fix`, `bug/investigation` | +| `feature` | `feature/requirement`, `feature/design`, `feature/implementation` | +| `project` | `project/setup`, `project/config`, `project/dependency` | +| `decision` | `decision/technical`, `decision/process`, `decision/tooling` | +| `topic` | `topic/learning`, `topic/explanation`, `topic/comparison` | + +--- + ### `smriti categories add ` -Add a custom category. +Add a custom category to the tree. ```bash -smriti categories add infra/monitoring --name "Monitoring" --parent infra --description "Monitoring and observability" +smriti categories add ops --name "Operations" +smriti categories add ops/incident --name "Incident Response" --parent ops +smriti categories add ops/runbook --name "Runbooks" --parent ops --description "Operational runbook sessions" ``` -## Embeddings +**Options:** -### `smriti embed` +| Flag | Description | +|------|-------------| +| `--name ` | Display name (required) | +| `--parent ` | Parent category ID | +| `--description ` | Optional description | -Build vector embeddings for all unembedded messages. Required for semantic search. +--- + +## Context & Compare + +### `smriti context` + +Generate a compact project summary (~200–300 tokens) and write it to +`.smriti/CLAUDE.md`. Claude Code auto-discovers this file at session start. + +Runs entirely from SQL — no Ollama, no network, no model loading. Typically +completes in under 100ms. ```bash -smriti embed +smriti context +smriti context --dry-run +smriti context --project myapp +smriti context --days 14 +smriti context --json ``` +**Options:** + +| Flag | Description | +|------|-------------| +| `--project ` | Project to generate context for (auto-detected from `cwd` if omitted) | +| `--days ` | Lookback window in days (default: `7`) | +| `--dry-run` | Print output to stdout without writing the file | +| `--json` | JSON output | + +--- + +### `smriti compare ` + +Compare two sessions across turns, tokens, tool calls, and file reads. Useful +for A/B testing context injection impact. + +```bash +smriti compare abc123 def456 +smriti compare --last +smriti compare --last --project myapp +smriti compare --last --json +``` + +**Options:** + +| Flag | Description | +|------|-------------| +| `--last` | Compare the two most recent sessions (for current project) | +| `--project ` | Project scope for `--last` | +| `--json` | JSON output | + +Partial session IDs are supported (first 7+ characters). + +--- + ## Team Sharing ### `smriti share` -Export sessions as markdown files to a `.smriti/` directory for git-based sharing. +Export sessions as clean markdown files to `.smriti/knowledge/` for +git-based team sharing. Generates LLM reflections via Ollama by default. +Also writes `.smriti/CLAUDE.md` so Claude Code auto-discovers shared +knowledge. ```bash smriti share --project myapp smriti share --category decision smriti share --session abc12345 smriti share --output /custom/path +smriti share --no-reflect +smriti share --reflect-model llama3.2 +smriti share --segmented --min-relevance 7 ``` +**Options:** + +| Flag | Description | +|------|-------------| +| `--project ` | Export sessions for a specific project | +| `--category ` | Export only sessions with this category | +| `--session ` | Export a single session | +| `--output ` | Custom output directory (default: `.smriti/`) | +| `--no-reflect` | Skip LLM reflections (reflections are on by default) | +| `--reflect-model ` | Ollama model for reflections | +| `--segmented` | Use 3-stage segmentation pipeline — beta | +| `--min-relevance ` | Relevance threshold for segmented mode (default: `6`) | + +--- + ### `smriti sync` -Import team knowledge from a `.smriti/` directory. +Import team knowledge from a `.smriti/knowledge/` directory into local +memory. Deduplicates by content hash — same content won't import twice. ```bash smriti sync @@ -187,10 +365,32 @@ smriti sync --project myapp smriti sync --input /custom/path ``` +**Options:** + +| Flag | Description | +|------|-------------| +| `--project ` | Scope sync to a specific project | +| `--input ` | Custom input directory (default: `.smriti/`) | + +--- + ### `smriti team` -View team contributions (authors, counts, categories). +View team contributions: authors, session counts, and category breakdown. ```bash smriti team ``` + +--- + +## Maintenance + +### `smriti upgrade` + +Pull the latest version from GitHub and reinstall dependencies. Equivalent to +re-running the install script. + +```bash +smriti upgrade +``` diff --git a/docs/configuration.md b/docs/configuration.md index 713e8c9..0941099 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,81 +1,113 @@ # Configuration -Smriti uses environment variables for configuration. Bun auto-loads `.env` files, so you can set these in a `.env.local` file in the smriti directory. +Smriti uses environment variables for configuration. Bun auto-loads `.env` +files, so you can put these in `~/.smriti/.env` and they'll be picked up +automatically — no need to set them in your shell profile. + +Most people never need to touch these. The defaults work. The ones you're +most likely to change are `SMRITI_PROJECTS_ROOT` (to match where your +projects actually live) and `QMD_MEMORY_MODEL` (if you want a lighter Ollama +model). + +--- ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| -| `QMD_DB_PATH` | `~/.cache/qmd/index.sqlite` | Path to the shared SQLite database | -| `CLAUDE_LOGS_DIR` | `~/.claude/projects` | Claude Code session logs directory | -| `CODEX_LOGS_DIR` | `~/.codex` | Codex CLI session logs directory | -| `SMRITI_PROJECTS_ROOT` | `~/zero8.dev` | Root directory for project detection | +| `QMD_DB_PATH` | `~/.cache/qmd/index.sqlite` | SQLite database path | +| `CLAUDE_LOGS_DIR` | `~/.claude/projects` | Claude Code session logs | +| `CODEX_LOGS_DIR` | `~/.codex` | Codex CLI session logs | +| `CLINE_LOGS_DIR` | `~/.cline/tasks` | Cline CLI tasks | +| `COPILOT_STORAGE_DIR` | auto-detected per OS | VS Code workspaceStorage root | +| `SMRITI_PROJECTS_ROOT` | `~/zero8.dev` | Root for project ID derivation | | `OLLAMA_HOST` | `http://127.0.0.1:11434` | Ollama API endpoint | -| `QMD_MEMORY_MODEL` | `qwen3:8b-tuned` | Ollama model for synthesis/summarization | -| `SMRITI_CLASSIFY_THRESHOLD` | `0.5` | Confidence below which LLM classification triggers | +| `QMD_MEMORY_MODEL` | `qwen3:8b-tuned` | Ollama model for synthesis | +| `SMRITI_CLASSIFY_THRESHOLD` | `0.5` | LLM classification trigger threshold | | `SMRITI_AUTHOR` | `$USER` | Author name for team sharing | +| `SMRITI_DAEMON_DEBOUNCE_MS` | `30000` | File-stability wait before auto-ingest | + +--- ## Projects Root -The `SMRITI_PROJECTS_ROOT` variable controls how Smriti derives project IDs from Claude Code session paths. +`SMRITI_PROJECTS_ROOT` is the most commonly changed setting. It controls how +Smriti derives clean project IDs from Claude Code session paths. -Claude Code encodes project paths in directory names like `-Users-zero8-zero8.dev-openfga`. Smriti reconstructs the real path and strips the projects root prefix: +Claude Code encodes project paths into directory names like +`-Users-zero8-zero8.dev-openfga`. Smriti reconstructs the real filesystem +path and strips the projects root prefix to produce a readable ID: -| Claude Dir Name | Derived Project ID | -|----------------|-------------------| -| `-Users-zero8-zero8.dev-openfga` | `openfga` | -| `-Users-zero8-zero8.dev-avkash-regulation-hub` | `avkash/regulation-hub` | -| `-Users-zero8-zero8.dev` | `zero8.dev` | +| Claude dir name | Projects root | Derived ID | +|-----------------|---------------|------------| +| `-Users-zero8-zero8.dev-openfga` | `~/zero8.dev` | `openfga` | +| `-Users-zero8-zero8.dev-avkash-regulation-hub` | `~/zero8.dev` | `avkash/regulation-hub` | +| `-Users-alice-code-myapp` | `~/code` | `myapp` | -To change the projects root: +If your projects live under `~/code` instead of `~/zero8.dev`: ```bash -export SMRITI_PROJECTS_ROOT="$HOME/projects" +export SMRITI_PROJECTS_ROOT="$HOME/code" ``` +--- + ## Database Location -By default, Smriti shares QMD's database at `~/.cache/qmd/index.sqlite`. This means your QMD document search and Smriti memory search share the same vector index — no duplication. +By default, Smriti shares QMD's database at `~/.cache/qmd/index.sqlite`. +This means QMD document search and Smriti memory search share the same vector +index — one embedding store, no duplication. -To use a separate database: +To keep them separate: ```bash export QMD_DB_PATH="$HOME/.cache/smriti/memory.sqlite" ``` +--- + ## Ollama Setup -Ollama is optional. It's used for: -- `smriti recall --synthesize` — Synthesize recalled context into a summary -- `smriti categorize --llm` — LLM-assisted categorization +Ollama is optional. Everything core — ingestion, search, recall, sharing — +works without it. Ollama only powers the features that require a language +model: + +- `smriti recall --synthesize` — Compress recalled context into a summary +- `smriti share` — Generate session reflections (skip with `--no-reflect`) +- `smriti categorize --llm` — LLM fallback for ambiguous categorization -Install and start Ollama: +Install and start: ```bash -# Install (macOS) +# macOS brew install ollama - -# Start the server ollama serve # Pull the default model ollama pull qwen3:8b-tuned ``` -To use a different model: +The default model (`qwen3:8b-tuned`) is good but large (~4.7GB). For a +lighter option: ```bash -export QMD_MEMORY_MODEL="mistral:7b" +export QMD_MEMORY_MODEL="qwen3:0.5b" +ollama pull qwen3:0.5b ``` -## Claude Code Hook +To point at a remote Ollama instance: + +```bash +export OLLAMA_HOST="http://192.168.1.100:11434" +``` -The install script sets up an auto-save hook at `~/.claude/hooks/save-memory.sh`. This requires: +--- -- **jq** — for parsing the hook's JSON input -- **Claude Code** — must be installed with hooks support +## Claude Code Hook -The hook is configured in `~/.claude/settings.json`: +The install script creates `~/.claude/hooks/save-memory.sh` and registers it +in `~/.claude/settings.json`. This is what captures sessions automatically +when you end a Claude Code conversation. ```json { @@ -86,7 +118,7 @@ The hook is configured in `~/.claude/settings.json`: "hooks": [ { "type": "command", - "command": "/path/to/.claude/hooks/save-memory.sh", + "command": "/Users/you/.claude/hooks/save-memory.sh", "timeout": 30, "async": true } @@ -97,4 +129,8 @@ The hook is configured in `~/.claude/settings.json`: } ``` -To disable the hook, remove the entry from `settings.json` or set `SMRITI_NO_HOOK=1` during install. +**Requires `jq`** — the hook parses JSON input from Claude Code. Install with +`brew install jq` or `apt install jq`. + +To disable: remove the entry from `settings.json`. To skip hook setup during +install, set `SMRITI_NO_HOOK=1` before running the installer. diff --git a/docs/getting-started.md b/docs/getting-started.md index a4fd4bc..8d328d3 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,17 +1,30 @@ # Getting Started +You're about to give your AI agents memory. + +By the end of this guide, your Claude Code sessions will be automatically +saved, searchable, and shareable — across sessions, across days, across your +team. + +--- + ## Install +**macOS / Linux:** + ```bash curl -fsSL https://raw.githubusercontent.com/zero8dotdev/smriti/main/install.sh | bash ``` -The installer will: -1. Check for (and install) [Bun](https://bun.sh) -2. Clone Smriti to `~/.smriti` -3. Install dependencies -4. Create the `smriti` CLI at `~/.local/bin/smriti` -5. Set up the Claude Code auto-save hook +**Windows** (PowerShell): + +```powershell +irm https://raw.githubusercontent.com/zero8dotdev/smriti/main/install.ps1 | iex +``` + +The installer: checks for Bun (installs it if missing) → clones Smriti to +`~/.smriti` → creates the `smriti` CLI → sets up the Claude Code auto-save +hook. ### Verify @@ -22,63 +35,104 @@ smriti help If `smriti` is not found, add `~/.local/bin` to your PATH: ```bash -echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc -source ~/.zshrc +echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc ``` +--- + ## First Run -### 1. Ingest your Claude Code conversations +### 1. Pull in your Claude Code sessions ```bash smriti ingest claude ``` -This scans `~/.claude/projects/` for all session transcripts and imports them. +Scans `~/.claude/projects/` and imports every conversation. On first run this +might take a moment if you've been coding with Claude for a while. -### 2. Check what was imported +### 2. See what you have ```bash smriti status ``` -Output shows session count, message count, and per-agent/per-project breakdowns. +Session counts, message counts, breakdown by project and agent. This is your +memory — everything Smriti knows about your past work. -### 3. Search your memory +### 3. Search it ```bash smriti search "authentication" ``` +Keyword search across every session. Try something you remember working +through in a past conversation. + ### 4. Recall with context ```bash smriti recall "how did we set up the database" ``` -This searches, deduplicates by session, and returns the most relevant snippets. +Like search, but smarter — deduplicates by session and surfaces the most +relevant snippets. Add `--synthesize` to compress results into a single +coherent summary (requires Ollama). -### 5. Build embeddings for semantic search +### 5. Turn on semantic search ```bash smriti embed ``` -After embedding, searches find semantically similar content — not just keyword matches. +Builds vector embeddings locally. After this, searches find semantically +similar content — not just keyword matches. "auth flow" starts surfacing +results that talk about "login mechanism." -## Auto-Save (Claude Code) +--- -If the installer set up the hook, every Claude Code conversation is saved automatically. No action needed — just code as usual. +## Auto-Save -To verify the hook is active: +If the install completed cleanly, you're done — every Claude Code session is +saved automatically when you end it. No manual step, no copy-pasting. + +Verify the hook is active: ```bash cat ~/.claude/settings.json | grep save-memory ``` +If the hook isn't there, re-run the installer or set it up manually — see +[Configuration](./configuration.md#claude-code-hook). + +--- + +## Share with Your Team + +Once you've built up memory, share the useful parts through git: + +```bash +smriti share --project myapp --category decision +cd ~/projects/myapp +git add .smriti/ && git commit -m "Share auth migration decisions" +git push +``` + +A teammate imports it: + +```bash +git pull && smriti sync --project myapp +smriti recall "auth migration" --project myapp +``` + +Their agent now has your context. See [Team Sharing](./team-sharing.md) for +the full guide. + +--- + ## Next Steps -- [CLI Reference](./cli.md) — All commands and options -- [Team Sharing](./team-sharing.md) — Share knowledge via git -- [Configuration](./configuration.md) — Environment variables and customization +- [CLI Reference](./cli.md) — Every command and option +- [Team Sharing](./team-sharing.md) — Share knowledge through git +- [Configuration](./configuration.md) — Customize paths, models, and behavior - [Architecture](./architecture.md) — How Smriti works under the hood diff --git a/docs/CI_HARDENING_EXECUTION_PLAN.md b/docs/internal/ci-hardening.md similarity index 100% rename from docs/CI_HARDENING_EXECUTION_PLAN.md rename to docs/internal/ci-hardening.md diff --git a/docs/DESIGN.md b/docs/internal/design.md similarity index 100% rename from docs/DESIGN.md rename to docs/internal/design.md diff --git a/docs/e2e-dev-release-flow-test.md b/docs/internal/e2e-release-flow.md similarity index 100% rename from docs/e2e-dev-release-flow-test.md rename to docs/internal/e2e-release-flow.md diff --git a/docs/search-recall-architecture.md b/docs/internal/search-analysis.md similarity index 100% rename from docs/search-recall-architecture.md rename to docs/internal/search-analysis.md diff --git a/docs/website.md b/docs/internal/website.md similarity index 100% rename from docs/website.md rename to docs/internal/website.md diff --git a/docs/WORKFLOW_AUTOMATION.md b/docs/internal/workflow-automation.md similarity index 100% rename from docs/WORKFLOW_AUTOMATION.md rename to docs/internal/workflow-automation.md diff --git a/docs/search.md b/docs/search.md new file mode 100644 index 0000000..d09204c --- /dev/null +++ b/docs/search.md @@ -0,0 +1,134 @@ +# Search & Recall + +Smriti has two ways to retrieve memory: `search` and `recall`. They use +different retrieval strategies and are optimized for different situations. + +--- + +## search vs recall + +| | `smriti search` | `smriti recall` | +|--|-----------------|-----------------| +| **Retrieval** | Full-text (BM25) | Full-text + vector (hybrid) | +| **Deduplication** | None — all matching messages | One best result per session | +| **Synthesis** | No | Yes, with `--synthesize` | +| **Best for** | Finding specific text, scanning results | Getting context before starting work | + +Use **search** when you know roughly what you're looking for and want to scan +results. Use **recall** when you want the most relevant context from your +history, deduplicated and optionally compressed. + +--- + +## How Search Works + +`smriti search` runs a BM25 full-text query against every ingested message. +It's fast, synchronous, and returns ranked results immediately — no model +loading. + +```bash +smriti search "rate limiting" +smriti search "auth" --project myapp --agent claude-code +smriti search "deployment" --category decision --limit 10 +``` + +Filters (`--project`, `--category`, `--agent`) narrow results with SQL JOINs +against Smriti's metadata tables. They compose — all filters apply together. + +--- + +## How Recall Works + +`smriti recall` goes further. It runs full-text search, deduplicates results +so you get at most one snippet per session (the highest-scoring one), and +optionally synthesizes everything into a single coherent summary. + +```bash +smriti recall "how did we handle rate limiting" +smriti recall "database setup" --synthesize +smriti recall "auth flow" --synthesize --model qwen3:0.5b +``` + +**Without filters:** recall uses QMD's full hybrid pipeline — BM25 + +vector embeddings + Reciprocal Rank Fusion. Semantic matches work here: "auth +flow" can surface results that talk about "login mechanism." + +**With filters:** recall currently uses full-text search only. The hybrid +pipeline is bypassed when `--project`, `--category`, or `--agent` is applied. +This is a known limitation — filtered recall loses semantic matching. It's +on the roadmap to fix. + +--- + +## Synthesis + +`--synthesize` sends the recalled context to Ollama and asks it to produce a +single coherent summary. This is the difference between getting 10 raw +snippets and getting a paragraph that distills what matters. + +```bash +smriti recall "connection pooling decisions" --synthesize +``` + +Requires Ollama running locally. See [Configuration](./configuration.md#ollama-setup) +for setup. Use `--model` to pick a lighter model if the default is too slow. + +--- + +## Vector Search + +Vector search finds semantically similar content — results that mean the same +thing even if they don't share the same words. It requires embeddings to be +built first: + +```bash +smriti embed +``` + +This runs locally via node-llama-cpp and EmbeddingGemma. It can take a few +minutes on a large history, but only processes new messages — subsequent runs +are fast. + +Once embeddings exist, unfiltered `smriti recall` automatically uses the full +hybrid pipeline (BM25 + vector + RRF). Filtered recall and `smriti search` +currently use BM25 only. + +--- + +## Filtering + +All filters compose and work across both commands: + +```bash +# Scope to a project +smriti recall "auth" --project myapp + +# Scope to a specific agent +smriti search "deployment" --agent cursor + +# Scope to a category +smriti recall "why did we choose postgres" --category decision + +# Combine them +smriti search "migration" --project api --category decision --limit 5 +``` + +Category filtering is hierarchical — `--category decision` matches +`decision`, `decision/technical`, `decision/process`, and +`decision/tooling`. + +--- + +## Token Compression + +The point of recall isn't just finding relevant content — it's making that +content usable in a new session without blowing up the context window. + +| Scenario | Raw | Via Smriti | Reduction | +|----------|-----|------------|-----------| +| Relevant context from past sessions | ~20,000 tokens | ~500 tokens | **40x** | +| Multi-session recall + synthesis | ~10,000 tokens | ~200 tokens | **50x** | +| Full project conversation history | 50,000+ tokens | ~500 tokens | **100x** | + +That's what `--synthesize` is for — not a summary for you to read, but +compressed context for your next agent session to start with. diff --git a/docs/team-sharing.md b/docs/team-sharing.md index 22c81ba..e9228fa 100644 --- a/docs/team-sharing.md +++ b/docs/team-sharing.md @@ -1,60 +1,107 @@ # Team Sharing -Smriti's team sharing works through git — no cloud service, no accounts, no sync infrastructure. +When you work through something hard with an AI agent — a tricky migration, +an architectural decision, a bug that took three hours to trace — that +knowledge shouldn't stay locked in your chat history. Smriti lets you export +it, commit it to git, and make it available to every agent your team uses. -## How It Works +No cloud service. No accounts. No sync infrastructure. Just git. -1. **Export** knowledge from your local memory to a `.smriti/` directory -2. **Commit** the `.smriti/` directory to your project repo -3. **Teammates pull** and import the shared knowledge into their local memory +--- + +## The Flow + +1. **Export** — `smriti share` converts your sessions into clean markdown + files and writes them to `.smriti/knowledge/` in your project directory +2. **Commit** — you push `.smriti/` to your project repo like any other file +3. **Import** — teammates run `smriti sync` to pull the knowledge into their + local memory +4. **Recall** — any agent on the team can now recall that context + +--- + +## End-to-End Example + +**Alice finishes a productive session on auth:** + +```bash +smriti share --project myapp --category decision -The `.smriti/` directory lives inside your project repo alongside your code. +cd ~/projects/myapp +git add .smriti/ +git commit -m "Share auth migration decisions" +git push +``` + +**Bob starts a new session the next morning:** + +```bash +cd ~/projects/myapp +git pull +smriti sync --project myapp + +smriti recall "auth migration" --project myapp +``` + +Bob's agent now has Alice's full context — the decisions made, the approaches +considered and rejected, the trade-offs. Alice didn't have to explain +anything. Bob didn't have to ask. + +--- ## Exporting Knowledge -### Share by project +### By project ```bash smriti share --project myapp ``` -This exports all sessions tagged with project `myapp` to the project's `.smriti/knowledge/` directory. +Exports all sessions tagged to that project. -### Share by category +### By category ```bash smriti share --category decision smriti share --category architecture/design ``` -### Share a specific session +Export only what matters. Decision sessions tend to have the highest +signal — they capture the *why* behind code choices, not just the *what*. + +### A single session ```bash smriti share --session abc12345 ``` -### Custom output directory +### Options -```bash -smriti share --project myapp --output /path/to/.smriti -``` +| Flag | Description | +|------|-------------| +| `--no-reflect` | Skip LLM session reflections (on by default — requires Ollama) | +| `--reflect-model ` | Ollama model for reflections | +| `--output ` | Custom output directory | +| `--segmented` | 3-stage segmentation pipeline (beta) | + +--- -## Output Format +## What Gets Exported ``` .smriti/ -├── config.json # Sharing configuration -├── index.json # Manifest of all shared files +├── config.json +├── index.json └── knowledge/ ├── decision/ │ └── 2026-02-10_auth-migration-approach.md - ├── bug-fix/ + ├── bug/ │ └── 2026-02-09_connection-pool-fix.md └── uncategorized/ └── 2026-02-08_initial-setup.md ``` -Each knowledge file is markdown with YAML frontmatter: +Each file is clean markdown with YAML frontmatter: ```markdown --- @@ -69,31 +116,36 @@ tags: ["decision", "decision/technical"] # Auth migration approach -> Summary of the session if available +> Generated reflection: Alice and the agent decided on a phased migration +> approach, starting with read-path only to reduce risk... **user**: How should we handle the auth migration? **assistant**: I'd recommend a phased approach... ``` -## Importing Knowledge +The reflection at the top is generated by Ollama — a short synthesis of what +was decided and why. Use `--no-reflect` to skip it. -When a teammate has shared knowledge: +--- + +## Importing Knowledge ```bash -git pull # Get the latest .smriti/ files -smriti sync --project myapp # Import into local memory +git pull +smriti sync --project myapp ``` -Or import from a specific directory: +Content is hashed before import — the same session imported twice creates no +duplicates. Run `smriti sync` as often as you like. + +Import from a specific directory: ```bash smriti sync --input /path/to/.smriti ``` -### Deduplication - -Content is hashed before import. If the same knowledge has already been imported, it's skipped automatically. You can safely run `smriti sync` repeatedly. +--- ## Viewing Contributions @@ -101,61 +153,31 @@ Content is hashed before import. If the same knowledge has already been imported smriti team ``` -Shows who has shared what: - ``` Author Shared Categories Latest alice 12 decision, bug/fix 2026-02-10 bob 8 architecture, code 2026-02-09 ``` -## Git Integration - -Add `.smriti/` to your repo: - -```bash -cd /path/to/myapp -git add .smriti/ -git commit -m "Share auth migration knowledge" -git push -``` - -### `.gitignore` Recommendations - -The `config.json` and `index.json` should be committed. If you want to be selective: - -```gitignore -# Commit everything in .smriti/ -!.smriti/ -``` - -## Workflow Example +--- -### Alice (shares knowledge) +## Claude Code Auto-Discovery -```bash -# Alice had a productive session about auth -smriti share --project myapp --category decision +When you run `smriti share`, it writes a `.smriti/CLAUDE.md` index file. +Claude Code auto-discovers this at the start of every session — giving it +immediate awareness of your team's shared knowledge without any manual +prompting. -# Commit to the project repo -cd ~/projects/myapp -git add .smriti/ -git commit -m "Share auth migration decisions" -git push -``` - -### Bob (imports knowledge) +--- -```bash -# Bob pulls the latest -cd ~/projects/myapp -git pull +## Notes -# Import Alice's shared knowledge -smriti sync --project myapp +**Categories survive the roundtrip.** The category a session was tagged with +on one machine is the category it's indexed under on every machine that syncs +it — no reclassification, no loss. -# Now Bob can recall Alice's context -smriti recall "auth migration" --project myapp -``` +**Only the primary category is restored on sync.** If a session had multiple +tags, only the primary one survives. Known limitation. -Bob's AI agent now has access to Alice's decisions without Alice needing to explain anything. +**You control what gets shared.** Nothing is exported unless you explicitly +run `smriti share`. Your local memory stays local until you decide otherwise.