Add deja-local: SQLite-backed vector memory for AI agents#9
Conversation
…nsistency Local vector brain for agents that need instant recall. No network, no external services, no waiting for indexes to sync. - Built-in character n-gram embedder (384 dims, ~0.1ms, zero deps) - Pluggable embed function (swap in OpenAI/Anthropic/local model) - Same learn/inject/query API shape as deja-client - Optional JSON file persistence - Brute-force cosine similarity (fast for <100k vectors) - 14 tests passing https://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw
Default embedder is now all-MiniLM-L6-v2 running locally via ONNX. ~23MB model cached after first download, ~5ms per embed after that. No network needed for recall. Zero eventual consistency. - embed: 'ngram' for zero-dep fallback - embed: yourFn for custom (OpenAI, etc) - model: option to swap ONNX model - Extracted shared search() helper, removed duplication https://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw
… else Nuked all speculative complexity: trigger/learning split, scopes, confidence, reason, source, recallCount, lastRecalledAt, Stats, query vs inject distinction, LearnOptions, InjectOptions, QueryOptions. What's left: - learn(text) → store a memory - recall(context) → get relevant memories back - forget(id), list(), clear(), save(), load() - Real embeddings via transformers.js (default) - N-gram fallback for zero-dep mode - Custom embed function support API surface: createMemory() returns a MemoryStore. https://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw
…gs only This is a complete rewrite optimizing for reliability over simplicity: - SQLite storage (bun:sqlite) — ACID durable, WAL mode - Every learn() hits disk before returning - Every recall() is logged to an audit table (context + matched IDs + scores) - Deduplication: near-identical memories are not stored twice - N-gram fallback removed — real embeddings only (transformers.js) - In-memory vector index rebuilt from DB on startup - path is required — no in-memory-only mode Trust guarantees verified by 19 tests across 4 categories: durability, consistency, deduplication, auditability https://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw
- Rename learn() → remember(), add backward-compat alias - Add confirm()/reject() for confidence scoring (the ratchet) - Memories start at 0.5 confidence, confirm boosts +0.1, reject drops -0.15 - Recall ranks by blended score: 70% relevance + 30% confidence - Conflict resolution: memories with 0.6-0.95 similarity are detected as conflicts — new memory supersedes old, old memory's confidence drops to 30% - Recall decomposition: complex queries are split into keyword pairs and individual terms, each embedded independently for broader recall - Schema migration for existing databases (adds confidence + supersedes columns) - 33 tests covering all new features + backward compat - Add README with full API documentation https://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a34d2533a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| * ``` | ||
| */ | ||
|
|
||
| import { Database } from 'bun:sqlite' |
There was a problem hiding this comment.
bun:sqlite import from the published entrypoint
This package is advertised as npm install deja-local and ships standard import/require exports, but the first line of the runtime entrypoint imports bun:sqlite. Plain Node cannot resolve the bun: scheme, so any non-Bun consumer will fail at module load time before createMemory() can run. Unless this package is explicitly marked Bun-only, the published entrypoint needs a runtime-neutral SQLite binding.
Useful? React with 👍 / 👎.
| // Open database, enable WAL mode, create schema | ||
| const db = new Database(opts.path) | ||
| db.exec('PRAGMA journal_mode = WAL') | ||
| db.exec('PRAGMA synchronous = NORMAL') |
There was a problem hiding this comment.
journal_mode=WAL combined with synchronous = NORMAL can still lose the most recent committed transactions if the host crashes or loses power immediately after remember(), confirm(), or reject() returns. For a library whose core promise is durable cross-session memory, that means users can silently lose the latest memories in the exact restart scenario they are trying to survive.
Useful? React with 👍 / 👎.
1. Mark package as Bun-only (engines field + description) since bun:sqlite cannot be resolved by Node.js consumers. 2. Upgrade synchronous pragma from NORMAL to FULL to prevent data loss on host crash with WAL mode — honoring the durability promise. https://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw
2 participants
Summary
Introduces
deja-local, a new package providing durable, local vector memory for AI agents. The implementation includes a complete SQLite-backed memory store with semantic search, confidence scoring, conflict resolution, and audit logging.Key Changes
Core memory store (
src/index.ts): ImplementscreateMemory()function that provides:remember()- Store memories with automatic deduplication and conflict resolutionrecall()- Semantic search with query decomposition for complex queriesconfirm()/reject()- Feedback mechanism to boost/demote memory confidenceforget()- Permanent memory deletionlist()/recallLog()- Inspection of stored memories and audit trailEmbedding support:
Intelligent memory management:
confirm()(+0.1) and decayed byreject()(-0.15), clamped to [0.01, 1.0]Query decomposition: Complex queries are automatically split into keyword pairs and individual terms for broader recall coverage
Durability & audit:
Comprehensive test suite (
test/index.test.ts): 544 lines covering:Documentation (
README.md): Complete API reference with examples and configuration guideNotable Implementation Details
learn()alias and schema migrationshttps://claude.ai/code/session_01Fb57JFi8VM36x8pCM1BFTw