Advanced Episodic Memory via Local Vector Stores

[galic1987]

Summary

Integrate a lightweight, embedded vector database for semantic retrieval of past debugging sessions and architectural decisions across projects.

Problem

Currently, episodic memory tracks approaches and project patterns but lacks semantic similarity search. When working on a new project, the agent can't retrieve relevant debugging sessions or architectural decisions from past work on entirely different codebases.

Proposal

Integrate an embedded vector store (e.g., sqlite-vss or a Rust-native implementation like qdrant in embedded mode) to enable:

• Cross-project memory: Retrieve relevant past experiences when encountering similar problems in new projects
• Semantic search over debug sessions: Find past debugging sessions by describing the current problem in natural language
• Architectural decision retrieval: Query past architectural choices and their outcomes
• Pattern learning: Build a growing knowledge base of coding patterns, anti-patterns, and project-specific conventions

Implementation Ideas

• Embed debug session summaries, tool call sequences, and user corrections as vectors
• Use the existing hnsw_rs dependency for approximate nearest-neighbor search
• Store vectors alongside episodic memory in ~/.selfware/memory/
• Expose via /memory search <query> interactive command
• Auto-index successful task completions and user corrections

Relevant Code

• src/cognitive/episodic.rs — current episodic memory
• src/memory/ — vector memory, RAG
• src/analysis/vector_store.rs — existing vector store infrastructure