Skip to content

Memory System

Jump to bottom
Alessio Rocchi edited this page Jan 27, 2026 · 1 revision

Memory System

Complete guide to aistack's persistent memory with SQLite, FTS5, and vector search.

Overview

The memory system provides persistent storage for:

  • Knowledge and patterns
  • Session context
  • Task history
  • Agent metadata

Storage: SQLite database with FTS5 full-text search and optional vector embeddings.

Location: Configured via memory.path (default: ./data/aistack.db)

Core Concepts

Memory Entry

interface MemoryEntry {
  id: string;                      // UUID v4
  key: string;                     // User-defined key
  namespace: string;               // Logical grouping
  content: string;                 // Stored content
  embedding?: Float32Array;        // Optional vector
  metadata?: Record<string, any>;  // JSON metadata
  createdAt: Date;
  updatedAt: Date;
}

Namespaces

Namespaces provide logical separation:

// Store in different namespaces
await memory.store('pattern:singleton', 'Use singleton for config', {
  namespace: 'architecture'
});

await memory.store('api:user-create', 'POST /api/users', {
  namespace: 'api-docs'
});

// Search within namespace
await memory.search('singleton', {
  namespace: 'architecture'
});

Benefits:

  • Organize data by feature/module
  • Faster searches (smaller dataset)
  • Prevent key collisions
  • Easier cleanup

Storing Data

Basic Store

// Via MCP
{
  "tool": "memory_store",
  "arguments": {
    "key": "pattern:di",
    "content": "Use dependency injection for loose coupling",
    "namespace": "best-practices"
  }
}

// Via TypeScript API
import { getMemoryManager } from '@blackms/aistack';

const memory = getMemoryManager();

await memory.store(
  'pattern:di',
  'Use dependency injection for loose coupling',
  {
    namespace: 'best-practices',
    metadata: {
      tags: ['architecture', 'patterns'],
      source: 'code-review'
    }
  }
);

With Metadata

await memory.store('api:user-create', 'POST /api/users creates a user', {
  namespace: 'api-docs',
  metadata: {
    method: 'POST',
    path: '/api/users',
    auth: 'required',
    tags: ['user', 'crud']
  }
});

With Vector Embedding

await memory.store('concept:jwt', 'JWT tokens provide stateless authentication', {
  namespace: 'security',
  generateEmbedding: true  // Requires vector search enabled
});

Configuration:

{
  "memory": {
    "vectorSearch": {
      "enabled": true,
      "provider": "openai",
      "model": "text-embedding-3-small"
    }
  }
}

Update Existing Entry

// Same key = update
await memory.store('pattern:di', 'UPDATED: Use dependency injection...', {
  namespace: 'best-practices'
});

// updatedAt timestamp changes
// id remains the same

Searching Data

Full-Text Search (FTS5)

// Via MCP
{
  "tool": "memory_search",
  "arguments": {
    "query": "dependency injection",
    "namespace": "best-practices",
    "limit": 10
  }
}

// Via TypeScript API
const results = await memory.search('dependency injection', {
  namespace: 'best-practices',
  limit: 10
});

console.log(results);
// {
//   count: 1,
//   results: [
//     {
//       entry: { key: 'pattern:di', content: '...', ... },
//       score: 0.95,
//       matchType: 'fts'
//     }
//   ]
// }

Vector Search

Semantic search using embeddings:

const results = await memory.search('how to authenticate users', {
  namespace: 'security',
  useVector: true,
  threshold: 0.7,  // Similarity threshold (0-1)
  limit: 5
});

// Returns semantically similar entries
// Even if exact keywords don't match

Hybrid Search

Combines FTS and vector search:

const results = await memory.search('authentication patterns', {
  useVector: true,    // Try vector first
  threshold: 0.7,     // If similarity >= 0.7
  limit: 10
});

// If vector search finds results → use vector results
// Otherwise → fall back to FTS search

Advanced FTS Queries

// Exact phrase
await memory.search('"dependency injection"');

// AND operator
await memory.search('dependency AND injection');

// OR operator
await memory.search('dependency OR coupling');

// Prefix match
await memory.search('depend*');

// Exclude term
await memory.search('dependency -coupling');

Retrieving Data

Get by Key

// Via MCP
{
  "tool": "memory_get",
  "arguments": {
    "key": "pattern:di",
    "namespace": "best-practices"
  }
}

// Via TypeScript API
const entry = memory.get('pattern:di', 'best-practices');

if (entry) {
  console.log(entry.content);
}

List All Entries

// Via MCP
{
  "tool": "memory_list",
  "arguments": {
    "namespace": "best-practices",
    "limit": 20,
    "offset": 0
  }
}

// Via TypeScript API
const entries = await memory.list({
  namespace: 'best-practices',
  limit: 20,
  offset: 0
});

console.log(`Found ${entries.length} entries`);

List All Namespaces

const namespaces = memory.listNamespaces();
console.log(namespaces);
// ['default', 'best-practices', 'api-docs', 'security']

Deleting Data

Delete Entry

// Via MCP
{
  "tool": "memory_delete",
  "arguments": {
    "key": "pattern:di",
    "namespace": "best-practices"
  }
}

// Via TypeScript API
await memory.delete('pattern:di', 'best-practices');

Delete Namespace

// Delete all entries in namespace
const entries = await memory.list({ namespace: 'old-namespace' });
for (const entry of entries) {
  await memory.delete(entry.key, 'old-namespace');
}

Session Management

Sessions provide temporary context for agent workflows.

Create Session

// Via MCP
{
  "tool": "session_start",
  "arguments": {
    "metadata": {
      "project": "user-auth",
      "goal": "Implement authentication"
    }
  }
}

// Via TypeScript API
const session = await memory.createSession({
  project: 'user-auth',
  goal: 'Implement authentication'
});

console.log('Session ID:', session.id);

End Session

// Via MCP
{
  "tool": "session_end",
  "arguments": {
    "sessionId": "550e8400-e29b-41d4-a716-446655440000"
  }
}

// Via TypeScript API
await memory.endSession(session.id);

Get Session Status

const sessionInfo = memory.getSession(session.id);

console.log(sessionInfo);
// {
//   id: '550e8400-...',
//   status: 'active',
//   startedAt: Date,
//   metadata: { project: 'user-auth', ... }
// }

List Active Sessions

// Via MCP
{
  "tool": "session_active",
  "arguments": {}
}

// Via TypeScript API
const sessions = memory.listSessions({ status: 'active' });

Task Tracking

Track agent tasks in the database.

Create Task

// Via MCP
{
  "tool": "task_create",
  "arguments": {
    "agentType": "coder",
    "input": "Implement JWT validation",
    "sessionId": "550e8400-..."
  }
}

// Via TypeScript API
const task = await memory.createTask(
  'coder',
  'Implement JWT validation',
  session.id
);

Update Task Status

await memory.updateTaskStatus(
  task.id,
  'completed',
  'JWT validation implemented successfully'
);

List Tasks

// All tasks in session
const tasks = memory.listTasks({ sessionId: session.id });

// By status
const pending = memory.listTasks({ status: 'pending' });
const completed = memory.listTasks({ status: 'completed' });

Database Schema

Memory Table

CREATE TABLE memory (
  id TEXT PRIMARY KEY,           -- UUID v4
  key TEXT NOT NULL,
  namespace TEXT DEFAULT 'default',
  content TEXT NOT NULL,
  embedding BLOB,                -- Float32Array as bytes
  metadata TEXT,                 -- JSON
  created_at INTEGER NOT NULL,   -- Unix timestamp (ms)
  updated_at INTEGER NOT NULL,
  UNIQUE(namespace, key)
);

CREATE INDEX idx_memory_namespace ON memory(namespace);
CREATE INDEX idx_memory_key ON memory(key);
CREATE INDEX idx_memory_updated ON memory(updated_at DESC);

FTS5 Virtual Table

CREATE VIRTUAL TABLE memory_fts USING fts5(
  key,
  content,
  namespace,
  content=memory,
  content_rowid=rowid,
  tokenize='porter unicode61'
);

Sessions Table

CREATE TABLE sessions (
  id TEXT PRIMARY KEY,
  status TEXT NOT NULL,          -- 'active' | 'ended' | 'error'
  started_at INTEGER NOT NULL,
  ended_at INTEGER,
  metadata TEXT
);

Tasks Table

CREATE TABLE tasks (
  id TEXT PRIMARY KEY,
  session_id TEXT,
  agent_type TEXT NOT NULL,
  status TEXT NOT NULL,          -- 'pending' | 'running' | 'completed' | 'failed'
  input TEXT,
  output TEXT,
  created_at INTEGER NOT NULL,
  completed_at INTEGER,
  FOREIGN KEY (session_id) REFERENCES sessions(id)
);

Vector Search Deep Dive

How It Works

  1. Generate Embedding

    • Text content → Vector (Float32Array)
    • OpenAI: 1536 dimensions
    • Ollama (nomic): 768 dimensions

  2. Store Embedding

    • Stored as BLOB in SQLite

  3. Search

    • Query → Generate query embedding
    • Compare with all stored embeddings
    • Cosine similarity calculation
    • Return results above threshold

Configuration

{
  "memory": {
    "vectorSearch": {
      "enabled": true,
      "provider": "openai",
      "model": "text-embedding-3-small"
    }
  },
  "providers": {
    "openai": {
      "apiKey": "${OPENAI_API_KEY}"
    }
  }
}

Provider Options

OpenAI:

{
  "vectorSearch": {
    "provider": "openai",
    "model": "text-embedding-3-small"  // or text-embedding-3-large
  }
}

Ollama (Local):

{
  "vectorSearch": {
    "provider": "ollama",
    "model": "nomic-embed-text"
  }
}

When to Use Vector Search

Use Vector Search for:

  • Semantic similarity (concepts, not keywords)
  • Cross-language searches
  • Fuzzy matching
  • Large datasets

Use FTS5 for:

  • Exact keyword matches
  • Structured queries
  • Small datasets
  • Fast searches (no API calls)

Best Practices

Key Naming

✓ GOOD:
  "pattern:singleton"
  "api:user-create"
  "concept:jwt"

✗ BAD:
  "temp123"
  "data"
  "stuff"

Use structured keys: category:identifier

Namespace Organization

✓ GOOD:
  namespace: "user-auth"
  namespace: "api-docs"
  namespace: "architecture"

✗ BAD:
  namespace: "default" (for everything)
  namespace: "misc"

Organize by feature or module.

Metadata Usage

✓ GOOD:
await memory.store('api:user-create', 'POST /api/users', {
  metadata: {
    method: 'POST',
    path: '/api/users',
    auth: 'required',
    tags: ['user', 'crud']
  }
});

✗ BAD:
await memory.store('api', 'some api stuff');

Use metadata for filtering and context.

Search Optimization

// 1. Use namespace to limit search space
await memory.search('query', { namespace: 'specific-namespace' });

// 2. Use appropriate limit
await memory.search('query', { limit: 10 });  // Not 1000

// 3. Use vector search for semantic queries
await memory.search('how to do X', { useVector: true });

// 4. Use FTS for keyword searches
await memory.search('exact keyword');

Session Cleanup

// Always end sessions when done
await memory.endSession(session.id);

// Periodically clean old sessions
const oldSessions = memory.listSessions({
  status: 'ended',
  endedBefore: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000) // 7 days
});

for (const session of oldSessions) {
  // Archive or delete
}

Performance Tips

1. Use Indexes

Built-in indexes on:

  • namespace
  • key
  • updated_at

Create custom indexes if needed:

CREATE INDEX idx_custom ON memory(json_extract(metadata, '$.tag'));

2. Vacuum Database

sqlite3 ./data/aistack.db "VACUUM;"

Run periodically to reclaim space.

3. Limit Result Sets

// Don't load everything
await memory.list({ limit: 100 });

// Use pagination
await memory.list({ limit: 20, offset: 20 });

4. Use Transactions

For bulk operations:

const db = memory.getDatabase();
const insertMany = db.transaction((entries) => {
  for (const entry of entries) {
    memory.store(entry.key, entry.content);
  }
});

insertMany(largeDataset);

Advanced Usage

Custom Queries

import { getMemoryManager } from '@blackms/aistack';

const memory = getMemoryManager();
const db = memory.getDatabase();

// Custom SQL query
const results = db.prepare(`
  SELECT * FROM memory
  WHERE namespace = ?
  AND json_extract(metadata, '$.priority') = 'high'
  ORDER BY created_at DESC
  LIMIT 10
`).all('user-auth');

Export Data

// Export all memory entries
const entries = await memory.list({ limit: 10000 });

fs.writeFileSync(
  'memory-export.json',
  JSON.stringify(entries, null, 2)
);

Import Data

const data = JSON.parse(fs.readFileSync('memory-export.json'));

for (const entry of data) {
  await memory.store(entry.key, entry.content, {
    namespace: entry.namespace,
    metadata: entry.metadata
  });
}

Troubleshooting

Database Locked

Error: SQLITE_BUSY: database is locked

Solution:

# Check for other processes
lsof ./data/aistack.db

# Increase timeout
{
  "memory": {
    "options": {
      "timeout": 10000
    }
  }
}

FTS5 Not Available

Error: no such module: fts5

Solution:

# Check SQLite version
sqlite3 --version
# Should be 3.9.0+ with FTS5

# macOS
brew upgrade sqlite3

# Linux
sudo apt-get install sqlite3 libsqlite3-dev

Vector Search Not Working

Problem: Falls back to FTS

Solution:

// 1. Enable vector search
{
  "memory": {
    "vectorSearch": {
      "enabled": true,
      "provider": "openai"
    }
  }
}

// 2. Configure provider
{
  "providers": {
    "openai": {
      "apiKey": "${OPENAI_API_KEY}"
    }
  }
}

// 3. Generate embeddings when storing
await memory.store('key', 'content', {
  generateEmbedding: true
});

Next Steps

Related:

Home

Getting Started

Core Concepts

Agent Guides

MCP Tools

Recipes

Advanced

Reference

Clone this wiki locally