diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index c0da4588..7798bba0 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "axon", - "version": "0.35.0", + "version": "0.35.1", "description": "Self-hosted web crawling and RAG pipeline with MCP tooling for scrape, crawl, ingest, embed, query, and ask workflows.", "homepage": "https://github.com/jmagar/axon_rust", "repository": "https://github.com/jmagar/axon_rust", diff --git a/CHANGELOG.md b/CHANGELOG.md index aa305791..a12c3347 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.35.1] - 2026-04-04 + +### Changed +- **Structured documentation**: Reorganized project documentation for clarity and consistency. + ## [0.35.0] - 2026-04-03 ### Fixed diff --git a/Cargo.toml b/Cargo.toml index 188a4742..d5f04d79 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "axon" -version = "0.35.0" +version = "0.35.1" edition = "2024" rust-version = "1.94.0" description = "Web crawl, scrape, embed, and query CLI backed by a self-hosted RAG stack" diff --git a/docs/CHECKLIST.md b/docs/CHECKLIST.md new file mode 100644 index 00000000..6e7a64b1 --- /dev/null +++ b/docs/CHECKLIST.md @@ -0,0 +1,68 @@ +# Release Checklist -- Axon + +Pre-release quality checklist. Complete all items before tagging a release. + +## Version and metadata + +- [ ] All version-bearing files in sync: `Cargo.toml`, `apps/web/package.json`, `CHANGELOG.md` +- [ ] `CHANGELOG.md` has an entry for the new version +- [ ] README version badge is correct (if present) + +## Configuration + +- [ ] `.env.example` documents every environment variable the binary reads +- [ ] `.env.example` has no actual secrets -- only placeholders +- [ ] `.env` is in `.gitignore` and `.dockerignore` +- [ ] `services.env` is in `.gitignore` and `.dockerignore` + +## Build and test + +- [ ] `just verify` passes (fmt-check + clippy + check + test) +- [ ] `just precommit` passes (monolith check + verify) +- [ ] `just docker-build` succeeds +- [ ] Web UI builds: `cd apps/web && pnpm build` +- [ ] Doctor reports all services healthy: `axon doctor` +- [ ] MCP smoke test passes: `just mcp-smoke` + +## Security + +- [ ] No credentials in code, docs, or git history +- [ ] `.gitignore` includes `.env`, `services.env`, `*.secret` +- [ ] `.dockerignore` includes `.env`, `services.env`, `.git/` +- [ ] Docker containers run as non-root (s6-setuidgid, UID 1001) +- [ ] No baked environment variables in Docker images +- [ ] `AXON_WEB_API_TOKEN` is not exposed as `NEXT_PUBLIC_*` + +## Infrastructure + +- [ ] `docker-compose.services.yaml` starts cleanly +- [ ] `docker-compose.yaml` builds and starts cleanly +- [ ] All 6 worker types start (crawl, embed, extract, ingest, refresh, graph) +- [ ] PostgreSQL schema auto-creates via `ensure_schema()` +- [ ] RabbitMQ queues are created on first worker start + +## Documentation + +- [ ] Root `CLAUDE.md` matches current architecture +- [ ] CLI command table in `CLAUDE.md` is up to date +- [ ] Environment variable section covers new variables +- [ ] `docs/MCP-TOOL-SCHEMA.md` regenerated: `just gen-mcp-schema` + +## Monolith policy + +- [ ] No `.rs` files exceed 500 lines (except allowlisted) +- [ ] No functions exceed 120 lines +- [ ] `scripts/enforce_monoliths.py` passes on staged files + +## Database + +- [ ] Migrations in `migrations/` are sequential +- [ ] Schema changes documented in `docs/SCHEMA.md` +- [ ] `ensure_schema()` handles upgrade path from previous version + +## Web UI + +- [ ] `apps/web` builds without errors +- [ ] Biome lint passes: `cd apps/web && pnpm lint` +- [ ] No `NEXT_PUBLIC_*` variables leak server-side secrets +- [ ] WebSocket auth token handling is correct (two-tier model) diff --git a/docs/CONFIG.md b/docs/CONFIG.md new file mode 100644 index 00000000..1382ac9a --- /dev/null +++ b/docs/CONFIG.md @@ -0,0 +1,250 @@ +# Configuration Reference -- Axon + +Axon is configured through three layers: environment variables, the `axon.json` file, and CLI flags. + +## Precedence (highest to lowest) + +1. CLI flags (`--pg-url`, `--collection`, etc.) +2. Environment variables (`AXON_PG_URL`, `AXON_COLLECTION`, etc.) +3. `axon.json` configuration file +4. Built-in defaults + +## Environment files + +Two env files are used: + +| File | Purpose | Loaded by | +|------|---------|-----------| +| `.env` | App runtime + shared Docker Compose interpolation | Docker Compose (automatic), `dotenvy` in binary | +| `services.env` | Infrastructure container credentials | Docker Compose `env_file:` directive | + +```bash +cp .env.example .env +chmod 600 .env +cp .env.example services.env +chmod 600 services.env +``` + +## axon.json + +The `axon.json` file provides structured configuration with schema validation (`axon.schema.json`). Key sections: + +| Section | Keys | Purpose | +|---------|------|---------| +| `services` | `qdrant_url`, `tei_url`, `chrome_remote_url`, `neo4j_url`, `backend_url` | Service endpoint URLs | +| `llm` | `base_url`, `model` | LLM provider settings | +| `tei` | `max_retries`, `request_timeout_ms`, `max_client_batch_size`, `embedding_model`, `pooling` | TEI embedding configuration | +| `search` | `hybrid_enabled`, `hybrid_candidates`, `hnsw_ef` | Vector search tuning | +| `ask` | `max_context_chars`, `candidate_limit`, `chunk_limit`, `min_relevance_score` | RAG answer pipeline | +| `embed` | `collection`, `doc_concurrency`, `doc_timeout_secs`, `strict_predelete` | Embedding pipeline | +| `queues` | `crawl`, `extract`, `embed`, `ingest`, `refresh`, `graph` | AMQP queue names | +| `workers` | `ingest_lanes`, `max_pending_crawl_jobs`, `job_stale_timeout_secs` | Worker tuning | +| `graph` | `concurrency`, `llm_model`, `similarity_threshold` | Neo4j graph RAG | +| `acp` | `adapter_cmd`, `prewarm`, `auto_approve`, `max_concurrent_sessions` | ACP orchestration | +| `web` | `allowed_origins`, `allow_insecure_dev`, `docker_socket_path` | Web UI settings | +| `mcp` | `transport`, `http_host`, `http_port`, `artifact_dir` | MCP server config | +| `serve` | `host`, `port` | Backend bridge config | +| `chrome` | `diagnostics`, `proxy`, `user_agent` | Chrome browser settings | +| `logging` | `file`, `max_bytes`, `max_files`, `no_color` | Log output config | +| `output` | `dir`, `extract_est_cost_per_1k_tokens` | Output directory config | +| `ingest` | `github_max_issues`, `github_max_prs`, `download_max_bytes` | Ingest limits | +| `oauth` | `auth_url`, `token_url`, `redirect_uri`, `scopes` | MCP OAuth broker | + +## Environment variables by category + +### Core runtime (required) + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_PG_URL` | -- | PostgreSQL connection DSN | +| `AXON_REDIS_URL` | -- | Redis connection DSN | +| `AXON_AMQP_URL` | -- | RabbitMQ AMQP connection DSN | +| `QDRANT_URL` | -- | Qdrant vector database URL | +| `TEI_URL` | -- | Text Embeddings Inference URL | + +### Docker Compose credentials (required for local stack) + +| Variable | Default | Description | +|----------|---------|-------------| +| `POSTGRES_USER` | `axon` | PostgreSQL username | +| `POSTGRES_PASSWORD` | -- | PostgreSQL password | +| `POSTGRES_DB` | `axon` | PostgreSQL database name | +| `REDIS_PASSWORD` | -- | Redis auth password | +| `RABBITMQ_USER` | `axon` | RabbitMQ username | +| `RABBITMQ_PASS` | -- | RabbitMQ password | + +### Host paths + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_DATA_DIR` | `./data` | Root directory for all persistent data | +| `HOST_HOME` | -- | Host user home (for session ingestion bind mount) | +| `AXON_WORKSPACE` | -- | Host workspace dir mounted into axon-web | +| `HOST_WORKSPACE` | -- | Host path to axon_rust repo | +| `AXON_BIN` | -- | Path to pre-built axon binary inside container | + +### Server ports + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_SERVE_HOST` | `127.0.0.1` | Backend bridge bind address | +| `AXON_SERVE_PORT` | `49000` | Backend bridge port | +| `AXON_WEB_DEV_PORT` | `49010` | Next.js dev server port | +| `SHELL_SERVER_PORT` | `49011` | Shell WebSocket server port | +| `AXON_MCP_HTTP_PORT` | `8001` | MCP HTTP server port | +| `AXON_MCP_HTTP_HOST` | `0.0.0.0` | MCP HTTP server bind address | +| `AXON_MCP_TRANSPORT` | `http` | MCP transport: `http`, `stdio`, or `both` | + +### Lite mode + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_LITE` | -- | Set to `1` to enable lite mode (no Postgres/Redis/RabbitMQ) | +| `AXON_SQLITE_PATH` | `~/.local/share/axon/jobs.db` | SQLite path for lite mode | + +### TEI embedding + +| Variable | Default | Description | +|----------|---------|-------------| +| `TEI_MAX_RETRIES` | `5` | Max retry attempts per request | +| `TEI_REQUEST_TIMEOUT_MS` | `30000` | Per-attempt timeout (clamped 100-600000) | +| `TEI_MAX_CLIENT_BATCH_SIZE` | `128` | Default batch size (auto-splits on 413) | +| `TEI_HTTP_PORT` | `52000` | Host port for TEI container | +| `TEI_EMBEDDING_MODEL` | `Qwen/Qwen3-Embedding-0.6B` | HuggingFace embedding model | +| `TEI_MAX_CONCURRENT_REQUESTS` | `80` | Max concurrent TEI requests | +| `TEI_MAX_BATCH_TOKENS` | `163840` | Max batch tokens | +| `TEI_MAX_BATCH_REQUESTS` | `80` | Max batch requests | +| `TEI_POOLING` | `last-token` | Pooling strategy | +| `TEI_TOKENIZATION_WORKERS` | `8` | Tokenization workers | +| `HF_TOKEN` | -- | HuggingFace token for gated models | + +### LLM / ACP + +| Variable | Default | Description | +|----------|---------|-------------| +| `OPENAI_BASE_URL` | -- | OpenAI-compatible base URL (legacy) | +| `OPENAI_API_KEY` | -- | API key for LLM provider | +| `OPENAI_MODEL` | -- | Model override for ACP-backed completions | +| `AXON_ASK_AGENT` | `claude` | Which ACP agent handles ask/research | +| `AXON_ACP_ADAPTER_CMD` | -- | Global ACP adapter override | +| `AXON_ACP_ADAPTER_ARGS` | -- | Global ACP adapter args (pipe-delimited) | +| `AXON_ACP_AUTO_APPROVE` | `true` | Auto-approve agent tool permissions | +| `AXON_ACP_MAX_CONCURRENT_SESSIONS` | `8` | Max concurrent ACP sessions | +| `AXON_ACP_TURN_TIMEOUT_MS` | `300000` | Per-turn timeout for Pulse Chat | +| `AXON_ACP_PREWARM` | `true` | Prewarm adapter on startup | +| `AXON_ACP_WS_URL` | -- | Remote ACP WebSocket URL | +| `AXON_ACP_WS_TOKEN` | -- | Remote ACP WebSocket token | + +### Queues and collections + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_COLLECTION` | `cortex` | Qdrant collection name | +| `AXON_CRAWL_QUEUE` | `axon.crawl.jobs` | Crawl job queue | +| `AXON_EXTRACT_QUEUE` | `axon.extract.jobs` | Extract job queue | +| `AXON_EMBED_QUEUE` | `axon.embed.jobs` | Embed job queue | +| `AXON_INGEST_QUEUE` | `axon.ingest.jobs` | Ingest job queue | +| `AXON_GRAPH_QUEUE` | `axon.graph.jobs` | Graph job queue | +| `AXON_INGEST_LANES` | `2` | Parallel ingest worker lanes | + +### Search and research + +| Variable | Default | Description | +|----------|---------|-------------| +| `TAVILY_API_KEY` | -- | Tavily AI Search API key | + +### Ingest credentials + +| Variable | Default | Description | +|----------|---------|-------------| +| `GITHUB_TOKEN` | -- | GitHub PAT for private repos and rate limits | +| `REDDIT_CLIENT_ID` | -- | Reddit OAuth2 client ID | +| `REDDIT_CLIENT_SECRET` | -- | Reddit OAuth2 client secret | + +### Chrome browser + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_CHROME_REMOTE_URL` | `http://axon-chrome:6000` | CDP management endpoint | +| `CHROME_URL` | `http://127.0.0.1:6000` | Spider-rs native CDP var | +| `AXON_CHROME_DIAGNOSTICS` | `false` | Enable browser diagnostics | +| `AXON_CHROME_PROXY` | -- | Proxy URL for Chrome requests | +| `AXON_CHROME_USER_AGENT` | -- | Custom User-Agent | + +### Hybrid search + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_HYBRID_SEARCH` | `true` | Enable BM42 sparse + dense RRF fusion | +| `AXON_HYBRID_CANDIDATES` | `100` | Candidates per prefetch arm (10-500) | +| `AXON_ASK_HYBRID_CANDIDATES` | `150` | Ask pipeline hybrid window | +| `AXON_HNSW_EF_SEARCH` | `128` | HNSW ef for named-mode search (32-512) | +| `AXON_HNSW_EF_SEARCH_LEGACY` | `64` | HNSW ef for legacy unnamed-mode | + +### Worker tuning + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_EMBED_DOC_CONCURRENCY` | CPU count | Max concurrent embed docs | +| `AXON_EMBED_DOC_TIMEOUT_SECS` | `300` | Per-document embed timeout | +| `AXON_EMBED_STRICT_PREDELETE` | `true` | Require pre-delete before upsert | +| `AXON_MAX_PENDING_CRAWL_JOBS` | `100` | Crawl queue cap (0 = unlimited) | +| `AXON_CRAWL_SIZE_WARN_THRESHOLD` | `10000` | Warn above N pages | +| `AXON_JOB_STALE_TIMEOUT_SECS` | `300` | Stale job detection | +| `AXON_JOB_STALE_CONFIRM_SECS` | `60` | Stale confirmation grace period | + +### Web app + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_BACKEND_URL` | `http://axon-workers:49000` | Backend URL for Next.js rewrites | +| `AXON_WEB_API_TOKEN` | -- | Primary API/WS auth token (server-only) | +| `AXON_WEB_BROWSER_API_TOKEN` | -- | Second-tier /api/* token (browser) | +| `NEXT_PUBLIC_AXON_API_TOKEN` | -- | Browser-exposed API token | +| `AXON_WEB_ALLOWED_ORIGINS` | -- | Comma-separated allowed origins | +| `AXON_WEB_ALLOW_INSECURE_DEV` | `false` | Allow localhost without auth | +| `AXON_SHELL_WS_TOKEN` | -- | Shell WebSocket auth token | +| `AXON_ALLOWED_CLAUDE_BETAS` | `interleaved-thinking` | Allowed Claude betas for Pulse | + +### Neo4j / GraphRAG + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_NEO4J_URL` | -- | Neo4j HTTP URL (empty = disabled) | +| `AXON_NEO4J_USER` | `neo4j` | Neo4j username | +| `AXON_NEO4J_PASSWORD` | -- | Neo4j password | +| `AXON_GRAPH_CONCURRENCY` | `4` | Parallel extraction jobs | +| `AXON_GRAPH_LLM_URL` | `http://localhost:11434` | Ollama/OpenAI URL for extraction | +| `AXON_GRAPH_LLM_MODEL` | `qwen3.5:4b` | Graph extraction model | +| `AXON_GRAPH_SIMILARITY_THRESHOLD` | `0.75` | Cross-document edge threshold | +| `AXON_GRAPH_SIMILARITY_LIMIT` | `20` | Max similar URLs for edges | +| `AXON_GRAPH_CONTEXT_MAX_CHARS` | `2000` | Graph context chars for `ask --graph` | + +### Logging + +| Variable | Default | Description | +|----------|---------|-------------| +| `RUST_LOG` | `info` | Rust tracing filter | +| `AXON_LOG_FILE` | -- | Structured log file path | +| `AXON_LOG_MAX_BYTES` | `10485760` | Log rotation size (10 MB) | +| `AXON_LOG_MAX_FILES` | `3` | Rotated log files retained | +| `AXON_NO_COLOR` | -- | Disable ANSI color output | + +### Miscellaneous + +| Variable | Default | Description | +|----------|---------|-------------| +| `AXON_MCP_ARTIFACT_DIR` | `$AXON_DATA_DIR/axon/artifacts` | MCP response artifact directory | +| `AXON_INLINE_BYTES_THRESHOLD` | `8192` | Auto-inline payload threshold | +| `AXON_OUTPUT_DIR` | `$AXON_DATA_DIR/axon/output` | Output directory for file-writing commands | +| `AXON_GIT_SHA` | `dev` | Git SHA baked into Docker labels | +| `AXON_NO_WIPE` | -- | Prevent destructive cache wipes | + +## Dev vs container URL resolution + +The CLI auto-detects its runtime environment: + +- **Inside Docker** (`/.dockerenv` exists): uses container DNS (`axon-postgres:5432`) +- **Outside Docker** (local dev): rewrites to localhost with mapped ports (`127.0.0.1:53432`) + +This means `.env` can use container DNS names -- `normalize_local_service_url()` in `config.rs` handles translation transparently. diff --git a/docs/GUARDRAILS.md b/docs/GUARDRAILS.md new file mode 100644 index 00000000..cb90c62e --- /dev/null +++ b/docs/GUARDRAILS.md @@ -0,0 +1,115 @@ +# Security Guardrails -- Axon + +Safety and security patterns enforced across the Axon stack. + +## Credential management + +### Storage + +- All credentials live in `.env` and `services.env` with `chmod 600` permissions +- Never commit `.env` or `services.env` +- Use `.env.example` as a tracked template with placeholder values only +- Docker Compose reads `.env` automatically for `${VAR}` interpolation; service containers receive credentials via `env_file: services.env` + +### Ignore files + +`.gitignore` and `.dockerignore` must include: + +``` +.env +services.env +*.secret +*.pem +*.key +``` + +### Pre-commit enforcement + +Lefthook hooks verify security invariants: + +| Hook | Purpose | +|------|---------| +| `check_env_staged.sh` | Blocks commits that include `.env` files | +| `check_dockerignore_guards.sh` | Verifies `.dockerignore` contains required patterns | + +## Web app token model + +Axon uses a two-tier token architecture for the web UI: + +| Token | Scope | Browser-visible | +|-------|-------|-----------------| +| `AXON_WEB_API_TOKEN` | Primary -- gates `/api/*` and `/ws` | No | +| `AXON_WEB_BROWSER_API_TOKEN` | Second-tier -- gates `/api/*` only | Yes (via `NEXT_PUBLIC_AXON_API_TOKEN`) | +| `NEXT_PUBLIC_AXON_API_TOKEN` | Client-side -- sent as `x-api-key` and `?token=` | Yes | + +Rules: +- `AXON_WEB_API_TOKEN` is server-only -- never set as a `NEXT_PUBLIC_*` variable +- When `AXON_WEB_BROWSER_API_TOKEN` is set, `NEXT_PUBLIC_AXON_API_TOKEN` must match it +- When `AXON_WEB_BROWSER_API_TOKEN` is unset, `NEXT_PUBLIC_AXON_API_TOKEN` must match `AXON_WEB_API_TOKEN` +- The `?token=` query param on WebSocket URLs is a necessary limitation (upgrade requests cannot carry custom headers) + +## MCP OAuth + +MCP OAuth (`atk_` tokens) is a separate auth system for MCP HTTP clients. It does not interact with the web UI token model. + +## Docker security + +### Non-root execution + +Containers use s6-overlay with PID 1 running as root (required by s6). Worker processes drop privileges via `s6-setuidgid axon` (UID 1001): + +```sh +exec s6-setuidgid axon /usr/local/bin/axon crawl worker +``` + +### No baked environment + +Docker images must not contain credentials at build time: +- No `ENV AXON_PG_URL=...` in Dockerfiles +- No `COPY .env` in Dockerfiles +- Credentials are injected at runtime via `env_file:` or container `environment:` + +### Image verification + +```bash +# Check for baked secrets +docker inspect axon:local | jq '.[0].Config.Env' + +# Check container revision matches git SHA +./scripts/check-container-revisions.sh +``` + +## Network security + +### HTTPS in production + +- All service URLs should use `https://` in production +- HTTP is acceptable for local development and Docker-internal networking +- The Chrome CDP endpoint is HTTP-only by design (internal network) + +### URL validation + +`validate_url()` in `crates/core/http.rs` enforces: +- No private/loopback IPs (SSRF protection) +- No file:// or other non-HTTP schemes +- Blocked malware/phishing domains (via Spider `firewall` feature) + +## Input handling + +### Crawl safety + +- Default `--max-pages 0` is uncapped -- `AXON_CRAWL_SIZE_WARN_THRESHOLD` warns when exceeded +- `AXON_MAX_PENDING_CRAWL_JOBS` caps the queue to prevent runaway crawls +- Auto path-prefix scoping limits crawl scope on deep URLs +- `--respect-robots` defaults to `false` -- legal implications acknowledged + +### Text chunking + +`chunk_text()` splits at 2000 chars with 200-char overlap. Very long pages produce many Qdrant points -- monitor collection size after large crawls. + +## Logging + +- Never log credentials, tokens, or API keys +- CLI outputs JSON data to stdout and progress/logs to stderr +- Log rotation: 10 MB max, 3 backups (`AXON_LOG_MAX_BYTES`, `AXON_LOG_MAX_FILES`) +- ANSI codes are stripped from web UI output via `console::strip_ansi_codes()` diff --git a/docs/INVENTORY.md b/docs/INVENTORY.md new file mode 100644 index 00000000..4db1a584 --- /dev/null +++ b/docs/INVENTORY.md @@ -0,0 +1,154 @@ +# Component Inventory -- Axon + +Complete listing of all Axon components. + +## MCP tools + +| Tool | Description | +|------|-------------| +| `axon` | Single entry point with `action`/`subaction` routing for all operations | + +Axon exposes one MCP tool with the full operation space routed via the `action` parameter. + +### Direct actions (no subaction required) + +| Action | Description | +|--------|-------------| +| `ask` | RAG: semantic search + LLM answer synthesis | +| `export` | Export full index manifest to JSON | +| `map` | Discover all URLs at a domain without scraping | +| `query` | Semantic vector search | +| `research` | Web research via Tavily with LLM synthesis | +| `retrieve` | Fetch stored document chunks from Qdrant | +| `scrape` | Scrape URLs to markdown | +| `screenshot` | Capture page screenshot via Chrome | +| `search` | Web search via Tavily, auto-queues crawl jobs | + +### Lifecycle action families (subaction required) + +| Action | Subactions | Description | +|--------|-----------|-------------| +| `crawl` | `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover` | Full site crawling | +| `extract` | `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover` | LLM-powered structured extraction | +| `embed` | `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover` | Vector embedding into Qdrant | +| `ingest` | `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover` | External source ingestion (GitHub, Reddit, YouTube, sessions) | +| `refresh` | `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover`, `schedule` | Periodic URL re-indexing | +| `graph` | `build`, `status`, `explore`, `stats` | Neo4j knowledge graph operations | +| `artifacts` | `head`, `grep`, `wc`, `read`, `list`, `delete`, `clean`, `search` | MCP artifact file management | + +### Info actions + +| Action | Description | +|--------|-------------| +| `doctor` | Diagnose service connectivity | +| `domains` | List indexed domains + stats | +| `help` | Return action reference | +| `sources` | List all indexed URLs + chunk counts | +| `stats` | Qdrant collection statistics | +| `status` | Async job queue status | + +## MCP resources + +| URI | Description | +|-----|-------------| +| `axon://schema/mcp-tool` | Tool schema definition | + +## CLI commands + +All MCP actions are also available as CLI commands: + +| Command | Async | Description | +|---------|-------|-------------| +| `scrape ...` | No | Scrape URLs to markdown | +| `crawl ...` | Yes | Full site crawl | +| `map ` | No | URL discovery without scraping | +| `extract ` | Yes | LLM-powered structured extraction | +| `search ` | No | Web search via Tavily | +| `research ` | No | Web research with LLM synthesis | +| `embed [input]` | Yes | Embed into Qdrant | +| `export` | No | Export index manifest | +| `query ` | No | Semantic vector search | +| `retrieve ` | No | Fetch stored chunks | +| `ask ` | No | RAG search + answer | +| `evaluate ` | No | RAG vs baseline comparison | +| `suggest [focus]` | No | Suggest new URLs to crawl | +| `ingest ` | Yes | Ingest GitHub, Reddit, YouTube | +| `sessions [format]` | No | Ingest AI session exports | +| `sources` | No | List indexed URLs | +| `domains` | No | List indexed domains | +| `stats` | No | Qdrant collection stats | +| `status` | No | Job queue status | +| `doctor` | No | Service connectivity check | +| `debug` | No | Doctor + LLM troubleshooting | +| `mcp` | No | Start MCP stdio server | +| `refresh ` | Yes | Periodic re-indexing | +| `graph ` | Depends | Knowledge graph operations | +| `serve` | No | Start web UI supervisor | +| `watch ` | Depends | Scheduled task management | +| `migrate` | No | Collection upgrade (unnamed to named vectors) | + +## Infrastructure services + +| Service | Image | Exposed Port | Purpose | +|---------|-------|-------------|---------| +| `axon-postgres` | postgres:17-alpine | 53432 | Job persistence | +| `axon-redis` | redis:8.2-alpine | 53379 | Queue state and caching | +| `axon-rabbitmq` | rabbitmq:4.0-management | 45535 | AMQP job queue | +| `axon-qdrant` | qdrant/qdrant:v1.13.1 | 53333, 53334 (gRPC) | Vector store | +| `axon-tei` | ghcr.io/huggingface/text-embeddings-inference | 52000 | Embedding generation | +| `axon-chrome` | docker/chrome/Dockerfile | 6000, 9222 (CDP) | Headless browser | + +## App services + +| Service | Image | Exposed Port | Purpose | +|---------|-------|-------------|---------| +| `axon-workers` | docker/Dockerfile | 49000, 8001 | Workers + serve + MCP HTTP | +| `axon-web` | docker/web/Dockerfile | 49010 | Next.js dashboard | + +## Worker types + +| Worker | Queue | Description | +|--------|-------|-------------| +| Crawl | `axon.crawl.jobs` | Full site crawling with sitemap backfill | +| Extract | `axon.extract.jobs` | LLM-powered structured data extraction | +| Embed | `axon.embed.jobs` | TEI embedding + Qdrant upsert | +| Ingest | `axon.ingest.jobs` | GitHub/Reddit/YouTube source ingestion | +| Refresh | `axon.refresh.jobs` | Periodic URL re-indexing | +| Graph | `axon.graph.jobs` | Neo4j entity extraction and graph building | + +## Workspace crates + +| Crate | Path | Purpose | +|-------|------|---------| +| `cli` | `crates/cli/` | Command handlers for all CLI subcommands | +| `core` | `crates/core/` | Config, HTTP client, content processing | +| `crawl` | `crates/crawl/` | Spider-based crawl engine | +| `ingest` | `crates/ingest/` | GitHub, Reddit, YouTube ingest adapters | +| `jobs` | `crates/jobs/` | Async job framework (AMQP + SQLite backends) | +| `mcp` | `crates/mcp/` | MCP server schema and handlers | +| `services` | `crates/services/` | Typed service layer (consumed by CLI, MCP, web) | +| `vector` | `crates/vector/` | Qdrant ops, TEI embedding, hybrid search | +| `web` | `crates/web/` | WebSocket execution bridge | + +## Database tables + +| Table | Purpose | +|-------|---------| +| `axon_crawl_jobs` | Crawl job metadata and results | +| `axon_extract_jobs` | Extract job metadata and results | +| `axon_embed_jobs` | Embed job metadata and results | +| `axon_ingest_jobs` | Ingest job metadata and results | + +## Scripts + +| Script | Purpose | +|--------|---------| +| `scripts/axon` | Wrapper script (auto-sources .env) | +| `scripts/dev-setup.sh` | Bootstrap development environment | +| `scripts/rebuild-fresh.sh` | Build + start Docker containers | +| `scripts/check-container-revisions.sh` | Verify container git SHA matches | +| `scripts/check_dockerignore_guards.sh` | Verify .dockerignore patterns | +| `scripts/enforce_monoliths.py` | Enforce file/function size limits | +| `scripts/generate_mcp_schema_doc.py` | Regenerate MCP-TOOL-SCHEMA.md | +| `scripts/live-test-all-commands.sh` | Integration test all CLI commands | +| `scripts/test-mcp-tools-mcporter.sh` | MCP smoke test suite | diff --git a/docs/README.md b/docs/README.md index 904bc792..01a22234 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,39 +1,105 @@ -# docs/ -Last Modified: 2026-03-20 - -Documentation index for Axon. - -## Core Docs -- [Architecture](./ARCHITECTURE.md) -- [Export Backup Contract](./EXPORT.md) -- [Restore Guide](./RESTORE.md) -- [Graph Features](./GRAPH.md) -- [Feature Delivery Framework](./FEATURE-DELIVERY-FRAMEWORK.md) -- [API](./API.md) -- [Deployment](./DEPLOYMENT.md) -- [Operations](./OPERATIONS.md) -- [Performance](./PERFORMANCE.md) -- [Security](./SECURITY.md) -- [Schema](./SCHEMA.md) -- [Job Lifecycle](./JOB-LIFECYCLE.md) -- [Serve Runtime](./SERVE.md) -- [Shell Completions](./SHELL-COMPLETIONS.md) -- [Live Test Scripts](./LIVE-TEST-SCRIPTS.md) -- [Testing Guide](./TESTING.md) - -## MCP Docs -- [MCP Runtime Guide](./MCP.md) -- [MCP Tool Schema](./MCP-TOOL-SCHEMA.md) - -## Guide Directories -- [Command Reference](./commands/README.md) -- [`ingest/`](./ingest) - -## Working Directories -- [`plans/`](./plans) -- [`reports/`](./reports) -- [`sessions/`](./sessions) - -## Related -- [Repository README](../README.md) -- [Module README Index](../README.md#module-readmes) +# Axon Documentation + +Web crawl, scrape, extract, embed, and query -- all in one binary backed by a self-hosted RAG stack. + +## What is Axon + +Axon is a trimodal application: + +| Mode | Entry point | Port | Purpose | +|------|-------------|------|---------| +| CLI | `axon ` | -- | Interactive command-line tool for crawl, scrape, embed, query, ask | +| MCP server | `axon mcp` | 8001 | Single-tool MCP server exposing all CLI operations to AI agents | +| Web UI | `axon serve` | 49000 (backend), 49010 (Next.js) | Supervisor that runs backend bridge, MCP HTTP, workers, shell server, and Next.js dashboard | + +All three modes share the same Rust binary, the same services layer, and the same infrastructure stack. + +## Structured documentation + +### Root + +| File | Description | +|------|-------------| +| [README.md](README.md) | This file -- documentation index | +| [SETUP.md](SETUP.md) | Step-by-step setup for local dev and Docker | +| [CONFIG.md](CONFIG.md) | Configuration reference -- axon.json and environment variables | +| [CHECKLIST.md](CHECKLIST.md) | Pre-release quality checklist | +| [GUARDRAILS.md](GUARDRAILS.md) | Security guardrails and safety patterns | +| [INVENTORY.md](INVENTORY.md) | Complete component inventory | + +### mcp/ + +| File | Description | +|------|-------------| +| [mcp/CLAUDE.md](mcp/CLAUDE.md) | Index for MCP docs | +| [mcp/TOOLS.md](mcp/TOOLS.md) | Tool actions, subactions, parameters, examples | +| [mcp/ENV.md](mcp/ENV.md) | MCP-specific environment variables | +| [mcp/TRANSPORT.md](mcp/TRANSPORT.md) | stdio, HTTP, streamable-http transport config | +| [mcp/DEPLOY.md](mcp/DEPLOY.md) | Deployment patterns -- local, Docker, lite mode | +| [mcp/CONNECT.md](mcp/CONNECT.md) | Connect from Claude Code, Codex, Gemini | +| [mcp/DEV.md](mcp/DEV.md) | MCP development workflow | +| [mcp/PATTERNS.md](mcp/PATTERNS.md) | Code patterns -- dispatch, artifacts, error handling | + +### repo/ + +| File | Description | +|------|-------------| +| [repo/CLAUDE.md](repo/CLAUDE.md) | Index for repo docs | +| [repo/REPO.md](repo/REPO.md) | Directory tree, workspace crates, root files | +| [repo/RECIPES.md](repo/RECIPES.md) | Justfile recipes reference | +| [repo/SCRIPTS.md](repo/SCRIPTS.md) | Scripts directory reference | +| [repo/RULES.md](repo/RULES.md) | Coding rules, git workflow, versioning | +| [repo/MEMORY.md](repo/MEMORY.md) | Memory and knowledge persistence | + +### stack/ + +| File | Description | +|------|-------------| +| [stack/CLAUDE.md](stack/CLAUDE.md) | Index for stack docs | +| [stack/ARCH.md](stack/ARCH.md) | Trimodal architecture, worker topology, data flow | +| [stack/TECH.md](stack/TECH.md) | Technology choices -- Rust, Spider, Qdrant, hybrid search | +| [stack/PRE-REQS.md](stack/PRE-REQS.md) | Prerequisites and dependency installation | + +## Existing documentation + +### Core docs + +- [Architecture](./ARCHITECTURE.md) -- detailed system architecture +- [API](./API.md) -- HTTP API reference +- [Deployment](./DEPLOYMENT.md) -- production deployment guide +- [Operations](./OPERATIONS.md) -- operational runbooks +- [Performance](./PERFORMANCE.md) -- tuning and benchmarks +- [Security](./SECURITY.md) -- security model +- [Schema](./SCHEMA.md) -- database schema +- [Job Lifecycle](./JOB-LIFECYCLE.md) -- async job state machine +- [Serve Runtime](./SERVE.md) -- `axon serve` supervisor details +- [Export/Backup](./EXPORT.md) -- export contract +- [Restore](./RESTORE.md) -- restore guide +- [Graph Features](./GRAPH.md) -- Neo4j graph RAG +- [Testing](./TESTING.md) -- test patterns and coverage + +### MCP docs + +- [MCP Runtime Guide](./MCP.md) -- MCP server internals +- [MCP Tool Schema](./MCP-TOOL-SCHEMA.md) -- wire contract (source of truth) + +### Guide directories + +- [commands/](./commands/) -- CLI command deep-dives +- [ingest/](./ingest/) -- ingest pipeline docs +- [auth/](./auth/) -- authentication patterns +- [services/](./services/) -- service layer docs +- [sessions/](./sessions/) -- session ingestion +- [superpowers/](./superpowers/) -- advanced workflows + +### Working directories + +- [plans/](./plans/) -- feature plans and proposals +- [reports/](./reports/) -- analysis reports + +## Quick links + +- **First time?** Start with [SETUP.md](SETUP.md), then [stack/ARCH.md](stack/ARCH.md) +- **MCP integration?** [mcp/CONNECT.md](mcp/CONNECT.md) then [mcp/TOOLS.md](mcp/TOOLS.md) +- **Contributing?** [repo/RULES.md](repo/RULES.md) then [repo/RECIPES.md](repo/RECIPES.md) +- **Deploying?** [mcp/DEPLOY.md](mcp/DEPLOY.md) then [CONFIG.md](CONFIG.md) diff --git a/docs/SETUP.md b/docs/SETUP.md new file mode 100644 index 00000000..59692853 --- /dev/null +++ b/docs/SETUP.md @@ -0,0 +1,156 @@ +# Setup Guide -- Axon + +Step-by-step instructions to get Axon running locally or via Docker. + +## Prerequisites + +| Dependency | Version | Purpose | +|------------|---------|---------| +| Rust | 1.94+ | Compiler and toolchain (see `rust-toolchain.toml`) | +| Docker | 24+ | Infrastructure services | +| Docker Compose | v2+ | Service orchestration | +| just | latest | Task runner | +| Node.js | 22+ | Web UI (Next.js) | +| pnpm | 9+ | Web UI package manager | + +Optional but recommended: + +| Tool | Purpose | +|------|---------| +| sccache | Compilation cache (auto-detected by Justfile) | +| mold | Fast linker (auto-detected by Justfile) | +| cargo-nextest | Faster parallel test runner | + +See [stack/PRE-REQS.md](stack/PRE-REQS.md) for detailed installation instructions. + +## 1. Clone the repository + +```bash +git clone https://github.com/jmagar/axon.git ~/workspace/axon_rust +cd ~/workspace/axon_rust +``` + +## 2. Run the setup script + +```bash +just setup +``` + +This installs Rust toolchain components, cargo tools, pnpm, and verifies all prerequisites. If `just` is not installed, run `./scripts/dev-setup.sh` directly -- it installs `just` for you. + +## 3. Configure environment + +```bash +cp .env.example .env +chmod 600 .env +cp .env.example services.env +chmod 600 services.env +``` + +Edit `.env` and set required values: + +```bash +# Postgres, Redis, RabbitMQ credentials +POSTGRES_PASSWORD=your_secure_password +REDIS_PASSWORD=your_secure_password +RABBITMQ_PASS=your_secure_password + +# Connection URLs (update passwords to match) +AXON_PG_URL=postgresql://axon:your_secure_password@axon-postgres:5432/axon +AXON_REDIS_URL=redis://:your_secure_password@axon-redis:6379 +AXON_AMQP_URL=amqp://axon:your_secure_password@axon-rabbitmq:5672 + +# Host paths +AXON_DATA_DIR=/path/to/persistent/data +HOST_HOME=/home/yourname +``` + +See [CONFIG.md](CONFIG.md) for the full variable reference. + +## 4. Start infrastructure + +```bash +just services-up +``` + +This starts PostgreSQL, Redis, RabbitMQ, Qdrant, TEI, and Chrome via `docker-compose.services.yaml`. + +For GPU-accelerated embeddings (NVIDIA): + +```bash +docker compose -f docker-compose.services.yaml -f docker-compose.gpu.yaml up -d +``` + +## 5. Run the local app stack + +```bash +just dev +``` + +This builds the binary, starts infrastructure (if not already running), and launches `axon serve` which supervises: +- Backend bridge (port 49000) +- MCP HTTP server (port 8001) +- All 6 worker types (crawl, embed, extract, ingest, refresh, graph) +- Shell WebSocket server +- Next.js dev server (port 49010) + +## 6. Verify + +```bash +# Check service connectivity +./scripts/axon doctor + +# Test a scrape +./scripts/axon scrape https://example.com --wait true + +# Check the web UI +# Open http://localhost:49010 +``` + +## Lite mode (zero infrastructure) + +For quick testing without Postgres, Redis, or RabbitMQ: + +```bash +AXON_LITE=1 ./scripts/axon scrape https://example.com --wait true +``` + +Lite mode uses SQLite for job storage and runs workers in-process. Qdrant and TEI are still required for embeddings. See the `AXON_LITE` section in [CONFIG.md](CONFIG.md). + +## Docker deployment + +For production or containerized deployment: + +```bash +# Start infrastructure +just services-up + +# Build and start app containers (workers + web) +just up +``` + +See [mcp/DEPLOY.md](mcp/DEPLOY.md) for detailed Docker deployment patterns. + +## Troubleshooting + +### "doctor" reports service unreachable + +- Confirm infrastructure is running: `docker compose -f docker-compose.services.yaml ps` +- Check that `.env` credentials match `services.env` +- For local dev, URLs auto-normalize to localhost ports (e.g., `axon-postgres:5432` becomes `127.0.0.1:53432`) + +### Build fails with spider_agent path error + +`Cargo.toml` references a local `spider_agent` path for development. In CI or fresh environments, switch to the crates.io version. See the `spider_agent` gotcha in the root `CLAUDE.md`. + +### TEI container exits immediately + +- TEI requires a GPU with NVIDIA drivers for the default image +- CPU-only hosts: use `docker-compose.services.yaml` without the GPU overlay +- Check model download: `docker compose -f docker-compose.services.yaml logs axon-tei` + +### Web UI shows "connection refused" + +- Verify `axon serve` is running (it starts Next.js as a supervised child) +- Check port 49010 is not in use: `lsof -i :49010` +- For Docker: ensure `axon-web` container is healthy diff --git a/docs/mcp/AGENTS.md b/docs/mcp/AGENTS.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/mcp/AGENTS.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/mcp/CLAUDE.md b/docs/mcp/CLAUDE.md new file mode 100644 index 00000000..21eab328 --- /dev/null +++ b/docs/mcp/CLAUDE.md @@ -0,0 +1,36 @@ +# MCP Server Documentation -- Axon + +Documentation for the Axon MCP server (`axon mcp`). + +## Files + +| File | Description | +|------|-------------| +| [TOOLS.md](TOOLS.md) | Tool actions, subactions, parameters, and response format | +| [ENV.md](ENV.md) | MCP-specific environment variables | +| [TRANSPORT.md](TRANSPORT.md) | stdio, HTTP, and streamable-http transport configuration | +| [DEPLOY.md](DEPLOY.md) | Deployment patterns -- local dev, Docker, lite mode | +| [CONNECT.md](CONNECT.md) | Connect from Claude Code, Codex CLI, Gemini CLI | +| [DEV.md](DEV.md) | MCP development workflow and adding new actions | +| [PATTERNS.md](PATTERNS.md) | Code patterns -- dispatch, artifacts, error handling | + +## Reading order + +**New to the Axon MCP server:** +1. ENV.md -- understand required configuration +2. TRANSPORT.md -- choose stdio or HTTP +3. CONNECT.md -- wire up your MCP client +4. TOOLS.md -- learn the action/subaction API surface + +**Adding or modifying MCP actions:** +1. PATTERNS.md -- dispatch and artifact patterns +2. DEV.md -- step-by-step workflow +3. TOOLS.md -- existing API surface + +## Cross-references + +- [../CONFIG.md](../CONFIG.md) -- full environment variable reference +- [../stack/ARCH.md](../stack/ARCH.md) -- trimodal architecture overview +- [../repo/REPO.md](../repo/REPO.md) -- repository structure +- [../MCP.md](../MCP.md) -- MCP runtime internals +- [../MCP-TOOL-SCHEMA.md](../MCP-TOOL-SCHEMA.md) -- wire contract (source of truth) diff --git a/docs/mcp/CONNECT.md b/docs/mcp/CONNECT.md new file mode 100644 index 00000000..cc4bcf34 --- /dev/null +++ b/docs/mcp/CONNECT.md @@ -0,0 +1,163 @@ +# Connect to Axon MCP + +How to connect to the Axon MCP server from supported clients. + +## Claude Code CLI + +### stdio + +```bash +claude mcp add axon -- /path/to/axon mcp +``` + +Or with environment variables: + +```bash +claude mcp add axon -- env QDRANT_URL=http://127.0.0.1:53333 TEI_URL=http://127.0.0.1:52000 /path/to/axon mcp +``` + +### HTTP + +When `axon serve` or `axon mcp` is running with HTTP transport (default): + +```bash +claude mcp add --transport http axon http://localhost:8001/mcp +``` + +### Scopes + +| Flag | Scope | Config file | +|------|-------|-------------| +| `--scope project` | Current project only | `.claude/settings.local.json` | +| `--scope user` | All projects | `~/.claude/settings.json` | +| (none) | Project default | `.claude/settings.local.json` | + +## Codex CLI + +### stdio + +`.codex/mcp.json` (project) or `~/.codex/mcp.json` (global): + +```json +{ + "mcpServers": { + "axon": { + "command": "/path/to/axon", + "args": ["mcp"], + "env": { + "QDRANT_URL": "http://127.0.0.1:53333", + "TEI_URL": "http://127.0.0.1:52000", + "AXON_PG_URL": "postgresql://axon:pass@127.0.0.1:53432/axon", + "AXON_REDIS_URL": "redis://:pass@127.0.0.1:53379", + "AXON_AMQP_URL": "amqp://axon:pass@127.0.0.1:45535" + } + } + } +} +``` + +### HTTP + +```json +{ + "mcpServers": { + "axon": { + "type": "http", + "url": "http://localhost:8001/mcp" + } + } +} +``` + +## Gemini CLI + +### stdio + +`gemini-extension.json` (project root or `~/.gemini/`): + +```json +{ + "mcpServers": { + "axon": { + "command": "/path/to/axon", + "args": ["mcp"], + "env": { + "QDRANT_URL": "http://127.0.0.1:53333", + "TEI_URL": "http://127.0.0.1:52000" + } + } + } +} +``` + +### HTTP + +```json +{ + "mcpServers": { + "axon": { + "type": "http", + "url": "http://localhost:8001/mcp" + } + } +} +``` + +## Manual configuration reference + +### Config file locations + +| Client | Scope | File | +|--------|-------|------| +| Claude Code | Project | `.claude/settings.local.json` | +| Claude Code | User | `~/.claude/settings.json` | +| Codex CLI | Project | `.codex/mcp.json` | +| Codex CLI | User | `~/.codex/mcp.json` | +| Gemini CLI | Project | `gemini-extension.json` | +| Gemini CLI | Global | `~/.gemini/gemini-extension.json` | + +## Lite mode connection + +For zero-infrastructure MCP (no Postgres/Redis/RabbitMQ): + +```json +{ + "mcpServers": { + "axon": { + "command": "/path/to/axon", + "args": ["mcp"], + "env": { + "AXON_LITE": "1", + "QDRANT_URL": "http://127.0.0.1:53333", + "TEI_URL": "http://127.0.0.1:52000" + } + } + } +} +``` + +## Verifying connection + +```bash +# HTTP health check +curl -s http://localhost:8001/health + +# Test via doctor +axon doctor + +# Test a tool call via Claude Code +claude "call axon with action=doctor" +``` + +If connection fails: + +1. Verify the server is running (`just dev` or `axon mcp`) +2. Check port 8001 is not blocked +3. For stdio: confirm the `axon` binary path is correct and all env vars are set +4. Run `axon doctor` to check infrastructure connectivity + +## See also + +- [TRANSPORT.md](TRANSPORT.md) -- transport configuration details +- [ENV.md](ENV.md) -- environment variables +- [DEPLOY.md](DEPLOY.md) -- deployment patterns diff --git a/docs/mcp/DEPLOY.md b/docs/mcp/DEPLOY.md new file mode 100644 index 00000000..0c4c89c4 --- /dev/null +++ b/docs/mcp/DEPLOY.md @@ -0,0 +1,142 @@ +# Deployment Guide -- Axon MCP + +Deployment patterns for the Axon MCP server. Choose the method that fits your environment. + +## Local development + +### Full stack (recommended) + +```bash +just dev +``` + +This starts infrastructure services, builds the binary, and launches `axon serve` which supervises the MCP HTTP server (port 8001) alongside the backend, workers, and web UI. + +### MCP server only + +```bash +# Start infrastructure +just services-up + +# Run MCP server standalone (stdio) +axon mcp + +# Or HTTP transport +AXON_MCP_TRANSPORT=http axon mcp +``` + +### Lite mode (zero infrastructure) + +```bash +AXON_LITE=1 axon mcp +``` + +Runs with SQLite for job storage. Requires Qdrant and TEI for embedding/search. Does not support graph, refresh, or watch operations. + +## Docker + +### Infrastructure + app containers + +```bash +# Start infrastructure (Postgres, Redis, RabbitMQ, Qdrant, TEI, Chrome) +just services-up + +# Build and start app containers (workers + web) +just up +``` + +The `axon-workers` container exposes: +- Port 49000: backend bridge + WebSocket +- Port 8001: MCP HTTP server + +The `axon-web` container exposes: +- Port 49010: Next.js dashboard + +### Docker Compose split + +| File | Contents | Env file | +|------|----------|----------| +| `docker-compose.services.yaml` | Infrastructure (Postgres, Redis, RabbitMQ, Qdrant, TEI, Chrome) | `services.env` | +| `docker-compose.yaml` | App containers (workers, web) | `.env` | +| `docker-compose.gpu.yaml` | GPU override for TEI and Ollama | -- | + +Both compose files share the `axon` bridge network and read `.env` for `${VAR}` interpolation. + +### GPU acceleration + +For NVIDIA hosts with GPU-accelerated TEI: + +```bash +docker compose -f docker-compose.services.yaml -f docker-compose.gpu.yaml up -d +``` + +CPU-only hosts use `docker-compose.services.yaml` alone. + +### Container architecture + +The `axon-workers` container uses s6-overlay for process supervision: + +| s6 service | Binary command | Purpose | +|------------|---------------|---------| +| `web-server` | `axon serve` | Backend bridge + MCP HTTP | +| `crawl-worker` | `axon crawl worker` | Crawl job processor | +| `embed-worker` | `axon embed worker` | Embedding pipeline | +| `extract-worker` | `axon extract worker` | LLM extraction | +| `ingest-worker` | `axon ingest worker` | Source ingestion | +| `graph-worker` | `axon graph worker` | Neo4j graph building | + +All worker processes run as the `axon` user (UID 1001) via `s6-setuidgid`. + +The `axon-web` container uses s6-overlay for: +- `pnpm-dev`: Next.js dev server +- `pnpm-watcher`: Polls lockfile for changes +- `claude-session`: Persistent Claude Code session +- `claude-watcher`: Hot-reload trigger + +### Build + +```bash +# Build workers image +docker build -f docker/Dockerfile -t axon:local . + +# Build web image +docker build -f docker/web/Dockerfile -t axon-web:local apps/web + +# Build Chrome image +docker build -f docker/chrome/Dockerfile -t axon-chrome:local docker/chrome +``` + +Build context must be the repo root (both compose files set `context: .`). + +### Health checks + +```bash +# Infrastructure +docker compose -f docker-compose.services.yaml ps + +# App containers +docker compose ps + +# Service connectivity +docker exec axon-workers axon doctor +``` + +## Data volumes + +All persistent data uses `${AXON_DATA_DIR:-./data}/axon/...`: + +| Volume | Content | +|--------|---------| +| `$AXON_DATA_DIR/axon/postgres` | PostgreSQL data | +| `$AXON_DATA_DIR/axon/redis` | Redis persistence | +| `$AXON_DATA_DIR/axon/rabbitmq` | RabbitMQ data | +| `$AXON_DATA_DIR/axon/qdrant` | Qdrant vector storage | +| `$AXON_DATA_DIR/axon/tei-data` | TEI model cache | +| `$AXON_DATA_DIR/axon/artifacts` | MCP response artifacts | +| `$AXON_DATA_DIR/axon/output` | CLI output files | + +## See also + +- [TRANSPORT.md](TRANSPORT.md) -- transport configuration +- [CONNECT.md](CONNECT.md) -- client connection methods +- [ENV.md](ENV.md) -- environment variables diff --git a/docs/mcp/DEV.md b/docs/mcp/DEV.md new file mode 100644 index 00000000..821d5e42 --- /dev/null +++ b/docs/mcp/DEV.md @@ -0,0 +1,174 @@ +# MCP Development Workflow -- Axon + +Day-to-day development guide for the Axon MCP server. + +## Quick start + +```bash +git clone https://github.com/jmagar/axon.git +cd axon_rust +cp .env.example .env && chmod 600 .env +# Edit .env with your credentials + +just dev # Starts infrastructure + axon serve (includes MCP HTTP on port 8001) +``` + +## MCP source structure + +``` +crates/ +├── mcp.rs # Module root — re-exports schema + server +├── mcp/ +│ ├── schema.rs # Tool input schema, action/subaction enums, serde parsing +│ └── server.rs # AxonMcpServer, handler dispatch, HTTP/stdio transport setup +├── services.rs # Module root — service layer exports +├── services/ +│ ├── context.rs # ServiceContext — shared state for all handlers +│ ├── types/ +│ │ └── service.rs # Typed result structs returned by all service functions +│ └── ... # Per-domain service functions (query, ask, sources, etc.) +``` + +The MCP server calls the services layer, which is the same layer used by CLI handlers and web routes. MCP handlers map typed service results to MCP wire format. + +## Development cycle + +1. **Edit source** -- modify handlers in `crates/mcp/server.rs` or service functions in `crates/services/` +2. **Build** -- `cargo check` or `just check` for fast feedback +3. **Test** -- `cargo test` or `just test` +4. **Run** -- `just dev` starts the full stack including MCP HTTP +5. **Verify** -- `just mcp-smoke` runs the MCP smoke test suite + +## Adding a new action + +### Direct action (no subaction) + +1. **Add enum variant** in `crates/mcp/schema.rs`: + - Add to the action enum + - Define required and optional parameters + +2. **Implement service function** in `crates/services/`: + - Create a function that returns a typed result struct + - Define the result struct in `crates/services/types/service.rs` + +3. **Add CLI handler** in `crates/cli/commands/`: + - Create a new command file + - Call the service function, format output for stdout + +4. **Wire MCP dispatch** in `crates/mcp/server.rs`: + - Add match arm for the new action + - Call the service function + - Map the typed result to MCP response format + +5. **Regenerate schema doc**: + ```bash + just gen-mcp-schema + ``` + +6. **Add tests**: + - Unit test for the service function + - Add to `scripts/test-mcp-tools-mcporter.sh` smoke test + +### Lifecycle action (with subactions) + +Lifecycle actions (crawl, extract, embed, ingest, refresh) follow a common pattern with `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover` subactions. + +1. Use the existing `crates/jobs/` framework +2. Implement a `Processor` trait for the new job type +3. Add a worker binary path +4. Wire into MCP dispatch with subaction routing + +## Testing + +### Unit tests + +```bash +cargo test # All tests +cargo test mcp # MCP-specific tests +``` + +### MCP smoke tests + +```bash +just mcp-smoke +# or directly: +./scripts/test-mcp-tools-mcporter.sh +``` + +The smoke test starts an MCP server, sends tool calls, and verifies responses. + +### curl testing (HTTP transport) + +```bash +# Health check +curl -s http://localhost:8001/health + +# Tool call +curl -X POST http://localhost:8001/mcp \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"axon","arguments":{"action":"doctor"}}}' + +# List tools +curl -X POST http://localhost:8001/mcp \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' +``` + +### MCP Inspector + +```bash +npx @modelcontextprotocol/inspector +``` + +Connect to `http://localhost:8001/mcp`. + +## Code style + +| Tool | Command | Purpose | +|------|---------|---------| +| clippy | `cargo clippy --all-targets` | Lint | +| rustfmt | `cargo fmt` | Format | +| cargo check | `cargo check` | Type check | +| cargo test | `cargo test` | Run tests | + +Run all checks: + +```bash +just verify # fmt-check + clippy + check + test +just fix # auto-fix: fmt + clippy --fix +``` + +## Debugging + +### Log levels + +Set `RUST_LOG` for MCP-specific filtering: + +```bash +RUST_LOG=info,axon::crates::mcp=debug axon mcp +``` + +### Response artifacts + +When `response_mode=path` (default), responses are written to `$AXON_MCP_ARTIFACT_DIR`. Inspect artifacts: + +```bash +ls -la .cache/axon-mcp/ +cat .cache/axon-mcp/latest-response.json | jq . +``` + +### Service context + +The `ServiceContext` in `crates/services/context.rs` carries a `ServiceCapabilities` struct that gates operations based on the runtime mode (full vs lite). Check capabilities before executing: + +```rust +if !ctx.capabilities.jobs.supported { + return Err("Operation requires full mode (Postgres + RabbitMQ)".into()); +} +``` + +## See also + +- [TOOLS.md](TOOLS.md) -- action/subaction reference +- [PATTERNS.md](PATTERNS.md) -- code patterns +- [../repo/RECIPES.md](../repo/RECIPES.md) -- Justfile recipes diff --git a/docs/mcp/ENV.md b/docs/mcp/ENV.md new file mode 100644 index 00000000..2dd96377 --- /dev/null +++ b/docs/mcp/ENV.md @@ -0,0 +1,72 @@ +# MCP Environment Variables -- Axon + +Environment variables specific to the Axon MCP server. The MCP server inherits all Axon stack variables (Postgres, Redis, RabbitMQ, Qdrant, TEI, LLM). This page covers MCP-specific configuration. + +## MCP server + +| Variable | Required | Default | Description | Sensitive | +|----------|----------|---------|-------------|-----------| +| `AXON_MCP_HTTP_HOST` | no | `0.0.0.0` | Bind address for HTTP transport | no | +| `AXON_MCP_HTTP_PORT` | no | `8001` | Listen port for HTTP transport | no | +| `AXON_MCP_TRANSPORT` | no | `http` | Transport mode: `http`, `stdio`, or `both` | no | +| `AXON_MCP_ARTIFACT_DIR` | no | `$AXON_DATA_DIR/axon/artifacts` | Directory for response artifacts | no | +| `AXON_INLINE_BYTES_THRESHOLD` | no | `8192` | Auto-inline payload size threshold (bytes) | no | + +## OAuth broker (optional) + +MCP OAuth is an optional auth system for MCP HTTP clients. It uses `atk_` tokens and is separate from the web UI token model. + +| Variable | Required | Default | Description | Sensitive | +|----------|----------|---------|-------------|-----------| +| `GOOGLE_OAUTH_CLIENT_ID` | no | -- | Google OAuth client ID | no | +| `GOOGLE_OAUTH_CLIENT_SECRET` | no | -- | Google OAuth client secret | **yes** | +| `GOOGLE_OAUTH_AUTH_URL` | no | -- | Authorization endpoint | no | +| `GOOGLE_OAUTH_TOKEN_URL` | no | -- | Token endpoint | no | +| `GOOGLE_OAUTH_REDIRECT_URI` | no | -- | Full redirect URI | no | +| `GOOGLE_OAUTH_REDIRECT_HOST` | no | -- | Redirect hostname | no | +| `GOOGLE_OAUTH_REDIRECT_PATH` | no | -- | Redirect path | no | +| `GOOGLE_OAUTH_REDIRECT_POLICY` | no | -- | Redirect policy | no | +| `GOOGLE_OAUTH_SCOPES` | no | -- | OAuth scopes | no | +| `GOOGLE_OAUTH_BROKER_ISSUER` | no | -- | Token issuer | no | +| `GOOGLE_OAUTH_REDIS_URL` | no | -- | Redis URL for token cache | no | +| `GOOGLE_OAUTH_REDIS_PREFIX` | no | -- | Redis key prefix | no | + +## Stack variables consumed by MCP + +The MCP server reads existing Axon stack variables at startup: + +| Variable | Purpose | +|----------|---------| +| `AXON_PG_URL` | Job persistence (full mode) | +| `AXON_REDIS_URL` | Queue state and cancel flags (full mode) | +| `AXON_AMQP_URL` | Job queue dispatch (full mode) | +| `QDRANT_URL` | Vector search and retrieval | +| `TEI_URL` | Embedding generation | +| `OPENAI_BASE_URL` | LLM provider (legacy path) | +| `OPENAI_API_KEY` | LLM auth | +| `OPENAI_MODEL` | Model override for ACP completions | +| `TAVILY_API_KEY` | Web search and research | +| `AXON_LITE` | Run without Postgres/Redis/RabbitMQ | +| `AXON_COLLECTION` | Default Qdrant collection | + +## Lite mode + +When `AXON_LITE=1`, the MCP server runs without Postgres, Redis, or RabbitMQ. Jobs use SQLite and run in-process. Some operations are unavailable: + +| Operation | Available in lite | +|-----------|-------------------| +| scrape, query, ask, search | Yes | +| crawl (sync), embed, ingest | Yes | +| graph, refresh, watch | No | +| export | No | + +## Precedence + +1. CLI flags override environment variables +2. Environment variables override `axon.json` settings +3. `axon.json` overrides built-in defaults + +## See also + +- [TRANSPORT.md](TRANSPORT.md) -- transport-specific configuration +- [../CONFIG.md](../CONFIG.md) -- full environment variable reference diff --git a/docs/mcp/GEMINI.md b/docs/mcp/GEMINI.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/mcp/GEMINI.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/mcp/PATTERNS.md b/docs/mcp/PATTERNS.md new file mode 100644 index 00000000..50bf8390 --- /dev/null +++ b/docs/mcp/PATTERNS.md @@ -0,0 +1,211 @@ +# MCP Code Patterns -- Axon + +Reusable patterns in the Axon MCP server implementation. + +## Single-tool dispatch + +Axon uses a single MCP tool (`axon`) with `action`/`subaction` routing. All operations go through one entry point. + +### Schema parsing + +Input is parsed strictly with serde in `crates/mcp/schema.rs`: + +```rust +#[derive(Deserialize)] +pub struct McpInput { + pub action: Action, + pub subaction: Option, + pub url: Option, + pub urls: Option>, + pub query: Option, + pub response_mode: Option, + // ... per-action fields +} +``` + +Rules: +- `action` is required and must match canonical enum names +- `subaction` is required for lifecycle families (crawl, extract, embed, ingest, refresh, graph, artifacts) +- No fallback fields, no token normalization, no case folding +- Invalid input returns MCP `invalid_params` error + +### Handler dispatch + +In `crates/mcp/server.rs`, the main handler matches on action: + +```rust +match input.action { + Action::Scrape => handle_scrape(ctx, input).await, + Action::Crawl => match input.subaction { + Some(CrawlSubaction::Start) => handle_crawl_start(ctx, input).await, + Some(CrawlSubaction::Status) => handle_crawl_status(ctx, input).await, + // ... + }, + Action::Ask => handle_ask(ctx, input).await, + // ... +} +``` + +Each handler calls the services layer and maps the typed result to MCP wire format. + +## Services layer + +All MCP handlers call through the services layer (`crates/services/`), never directly to infrastructure: + +``` +MCP handler -> services::query() -> vector::ops::search() -> Qdrant +MCP handler -> services::ask() -> vector::ops::ask() -> Qdrant + ACP +CLI handler -> services::query() -> (same path) +Web route -> services::query() -> (same path) +``` + +Each service function returns a typed result struct defined in `crates/services/types/service.rs`. No raw JSON printing or stdout side-effects in the service layer. + +## Artifact response pattern + +Heavy operations write results to artifact files instead of returning inline: + +``` +1. Handler executes operation +2. Result serialized to JSON +3. Written to $AXON_MCP_ARTIFACT_DIR/.json +4. MCP response returns compact metadata: + - path: artifact file path + - bytes: file size + - line_count: lines in artifact + - sha256: content hash + - preview: first N lines + - preview_truncated: boolean +``` + +### Response mode selection + +```rust +enum ResponseMode { + Path, // Default — artifact only, return metadata + Inline, // Return full result inline (capped/truncated) + Both, // Write artifact AND return inline + AutoInline, // Inline if below threshold, else artifact +} +``` + +`auto_inline` checks `AXON_INLINE_BYTES_THRESHOLD` (default 8192 bytes). Payloads at or below the threshold are returned inline without requiring a separate artifact read. + +## Error handling + +### Structured errors + +```rust +// Service layer returns typed errors +pub enum AxonError { + ServiceUnavailable(String), // Infrastructure not reachable + InvalidInput(String), // Bad parameters + NotFound(String), // Resource not found + Internal(String), // Unexpected failure +} +``` + +### MCP error mapping + +| Source | MCP error code | +|--------|---------------| +| Invalid action/subaction | `invalid_params` | +| Missing required field | `invalid_params` | +| Service unreachable | `internal_error` | +| Job not found | `internal_error` | + +### Canonical error envelope + +```json +{ + "ok": false, + "action": "crawl", + "subaction": "status", + "error": "Job abc-123 not found" +} +``` + +## ServiceContext + +The `ServiceContext` is constructed at startup and shared across all handlers: + +```rust +pub struct ServiceContext { + pub config: Config, + pub capabilities: ServiceCapabilities, + pub pg_pool: Option, + pub redis: Option, + pub amqp: Option, + // ... +} +``` + +`ServiceCapabilities` gates operations based on runtime mode: + +```rust +pub struct ServiceCapabilities { + pub jobs: CapabilityGate, // Requires Postgres + RabbitMQ + pub graph: CapabilityGate, // Requires Neo4j + pub search: CapabilityGate, // Requires Tavily API key + // ... +} +``` + +MCP handlers check capabilities before executing: + +```rust +if !ctx.capabilities.jobs.supported { + return Err(McpError::new("Operation requires full mode")); +} +``` + +## Lifecycle job pattern + +Lifecycle actions (crawl, extract, embed, ingest, refresh) share a common pattern: + +1. `start` -- enqueue job to AMQP queue, return job ID +2. `status` -- query Postgres for job state +3. `cancel` -- set cancel flag in Redis +4. `list` -- list recent jobs from Postgres +5. `cleanup` -- remove completed/failed jobs +6. `clear` -- remove all jobs +7. `recover` -- reclaim stale/interrupted jobs + +Each job type has: +- A `Processor` trait implementation in `crates/jobs//` +- A queue name in `axon.json` (e.g., `axon.crawl.jobs`) +- A database table (e.g., `axon_crawl_jobs`) +- A worker binary path (e.g., `axon crawl worker`) + +## Hybrid search pattern + +Vector search uses Reciprocal Rank Fusion (RRF) with two prefetch arms: + +``` +Query + ├── Dense vector (TEI embedding) → HNSW search + ├── BM42 sparse vector (keyword) → Sparse index search + └── RRF fusion → merged, re-ranked results +``` + +Named-mode collections (new) support hybrid search. Legacy unnamed-mode collections fall back to dense-only. The `VectorMode` is cached per-process -- restart workers after collection migration. + +## ACP completion pattern + +Operations requiring LLM synthesis (`ask`, `evaluate`, `suggest`, `research`, `extract` fallback, `debug`) use the Agent Client Protocol (ACP): + +``` +1. Service function prepares prompt + context +2. ACP adapter spawned as subprocess (configured via AXON_ACP_ADAPTER_CMD) +3. Adapter communicates with LLM provider +4. Response streamed back through ACP protocol +5. Service function parses and returns typed result +``` + +Pre-warming (`AXON_ACP_PREWARM=true`) eliminates cold-start latency by spawning the adapter at server startup. + +## See also + +- [TOOLS.md](TOOLS.md) -- action/subaction reference +- [DEV.md](DEV.md) -- development workflow +- [../stack/ARCH.md](../stack/ARCH.md) -- architecture overview diff --git a/docs/mcp/TOOLS.md b/docs/mcp/TOOLS.md new file mode 100644 index 00000000..f669f53e --- /dev/null +++ b/docs/mcp/TOOLS.md @@ -0,0 +1,283 @@ +# MCP Tools Reference -- Axon + +## Design + +Axon exposes exactly one MCP tool: + +| Tool | Purpose | Primary parameter | +|------|---------|-------------------| +| `axon` | Unified action router for all crawl/scrape/embed/query/RAG operations | `action` | + +This single-tool pattern routes all operations through `action` + optional `subaction`, keeping the MCP surface minimal while supporting 50+ operations. + +## Tool: `axon` + +### Input schema + +```json +{ + "action": "", + "subaction": "", + "url": "", + "urls": ["", ""], + "query": "", + "response_mode": "path|inline|both|auto_inline" +} +``` + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `action` | enum | yes | Operation to perform | +| `subaction` | enum | for lifecycle families | Sub-operation (start, status, cancel, etc.) | +| `response_mode` | enum | no | Response format: `path` (default), `inline`, `both`, `auto_inline` | + +Additional parameters vary by action. See sections below. + +### Response format + +All responses use the canonical envelope: + +```json +{ + "ok": true, + "action": "scrape", + "subaction": null, + "data": { ... } +} +``` + +Error responses: + +```json +{ + "ok": false, + "error": "description of the failure" +} +``` + +### Response modes + +| Mode | Behavior | +|------|----------| +| `path` (default) | Write result to `.cache/axon-mcp/` artifact file, return metadata (path, bytes, sha256, preview) | +| `inline` | Return full result inline (capped/truncated) | +| `both` | Write artifact and return inline content | +| `auto_inline` | Inline if payload is below `AXON_INLINE_BYTES_THRESHOLD` (default 8192), otherwise artifact | + +## Direct actions + +These actions do not require `subaction`. + +### scrape + +Scrape one or more URLs to markdown. + +```json +{ "action": "scrape", "url": "https://example.com" } +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `url` | string | -- | URL to scrape | +| `render_mode` | enum | `auto_switch` | `http`, `chrome`, `auto_switch` | +| `format` | enum | `markdown` | `markdown`, `html`, `rawHtml`, `json` | +| `embed` | bool | `true` | Auto-embed content into Qdrant | +| `root_selector` | string | -- | CSS selector for content root | +| `exclude_selector` | string | -- | CSS selector for elements to exclude | + +### query + +Semantic vector search against the Qdrant collection. + +```json +{ "action": "query", "query": "embedding pipeline architecture" } +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `query` | string | -- | Search text | +| `limit` | usize | 10 | Max results | +| `offset` | usize | 0 | Skip N results | +| `collection` | string | `cortex` | Qdrant collection | +| `since` | string | -- | Filter: only docs after this date | +| `before` | string | -- | Filter: only docs before this date | + +### ask + +RAG: semantic search + LLM answer synthesis with citations. + +```json +{ "action": "ask", "query": "How does hybrid search work?" } +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `query` | string | -- | Question to answer | +| `graph` | bool | `false` | Inject Neo4j graph context | +| `diagnostics` | bool | `false` | Include retrieval diagnostics | +| `collection` | string | `cortex` | Qdrant collection | +| `since` | string | -- | Date filter (after) | +| `before` | string | -- | Date filter (before) | + +### search + +Web search via Tavily, auto-queues crawl jobs for results. + +```json +{ "action": "search", "query": "rust async patterns" } +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `query` | string | -- | Search query | +| `limit` | usize | 10 | Max results | +| `search_time_range` | enum | -- | `day`, `week`, `month`, `year` | + +### research + +Web research via Tavily AI search with LLM synthesis. + +```json +{ "action": "research", "query": "vector database comparison 2025" } +``` + +Parameters are the same as `search`. + +### retrieve + +Fetch stored document chunks from Qdrant by URL. + +```json +{ "action": "retrieve", "url": "https://docs.example.com/guide" } +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `url` | string | -- | URL to retrieve chunks for | +| `max_points` | usize | -- | Limit returned chunks | + +### map + +Discover all URLs at a domain without scraping content. + +```json +{ "action": "map", "url": "https://example.com" } +``` + +### screenshot + +Capture a page screenshot via headless Chrome. + +```json +{ "action": "screenshot", "url": "https://example.com" } +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `url` | string | -- | URL to capture | +| `full_page` | bool | `false` | Full page vs viewport | +| `viewport` | string | -- | Viewport dimensions | +| `output` | string | -- | Output file path | + +### export + +Export full index manifest (jobs + ingest targets + refresh schedules + Qdrant summary). + +```json +{ "action": "export" } +``` + +## Lifecycle action families + +These actions require a `subaction`. All support: `start`, `status`, `cancel`, `list`, `cleanup`, `clear`, `recover`. + +### crawl + +```json +{ "action": "crawl", "subaction": "start", "urls": ["https://example.com"] } +``` + +Start parameters: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `urls` | string[] | -- | Seed URLs (required) | +| `max_pages` | u32 | 0 (uncapped) | Page limit | +| `max_depth` | usize | 5 | Max crawl depth | +| `include_subdomains` | bool | `false` | Include subdomains | +| `respect_robots` | bool | `false` | Honor robots.txt | +| `discover_sitemaps` | bool | `true` | Run sitemap backfill | +| `render_mode` | enum | `auto_switch` | `http`, `chrome`, `auto_switch` | +| `delay_ms` | u64 | 0 | Per-request delay | + +### extract + +```json +{ "action": "extract", "subaction": "start", "urls": ["https://example.com/pricing"] } +``` + +### embed + +```json +{ "action": "embed", "subaction": "start", "input": "path/to/file.md" } +``` + +### ingest + +```json +{ "action": "ingest", "subaction": "start", "source_type": "github", "target": "owner/repo" } +``` + +Source types: `github`, `reddit`, `youtube`, `sessions`. + +### refresh + +```json +{ "action": "refresh", "subaction": "start", "url": "https://docs.example.com" } +``` + +Refresh also supports scheduling: +```json +{ "action": "refresh", "subaction": "schedule", "schedule_subaction": "list" } +``` + +### graph + +```json +{ "action": "graph", "subaction": "build", "url": "https://example.com" } +``` + +Subactions: `build`, `status`, `explore`, `stats`. + +### artifacts + +MCP artifact file management: +```json +{ "action": "artifacts", "subaction": "list" } +``` + +Subactions: `head`, `grep`, `wc`, `read`, `list`, `delete`, `clean`, `search`. + +## Info actions + +| Action | Description | +|--------|-------------| +| `doctor` | Check connectivity to all infrastructure services | +| `domains` | List all indexed domains with stats | +| `help` | Return full action reference as markdown | +| `sources` | List all indexed URLs with chunk counts | +| `stats` | Qdrant collection statistics | +| `status` | Async job queue status | + +## Error semantics + +| Error type | MCP code | Cause | +|------------|----------|-------| +| Input/shape failure | `invalid_params` | Missing required field, invalid enum value | +| Runtime failure | `internal_error` | Service unreachable, database error | + +## See also + +- [ENV.md](ENV.md) -- MCP environment variables +- [PATTERNS.md](PATTERNS.md) -- dispatch and artifact patterns +- [../MCP-TOOL-SCHEMA.md](../MCP-TOOL-SCHEMA.md) -- wire contract (source of truth) diff --git a/docs/mcp/TRANSPORT.md b/docs/mcp/TRANSPORT.md new file mode 100644 index 00000000..00ee4a0b --- /dev/null +++ b/docs/mcp/TRANSPORT.md @@ -0,0 +1,161 @@ +# Transport Methods -- Axon MCP + +## Overview + +Axon MCP supports three transport configurations: + +| Transport | Auth | Use case | Config value | +|-----------|------|----------|--------------| +| stdio | None (process isolation) | Claude Desktop, Codex CLI, local tools | `stdio` | +| HTTP | OAuth / bearer token | Docker, remote servers, shared access | `http` | +| Both | Mixed | Serve HTTP while also accepting stdio | `both` | + +Set the transport via environment variable: + +```bash +AXON_MCP_TRANSPORT=http # default +``` + +Or via `axon.json`: + +```json +{ + "mcp": { + "transport": "http", + "http_host": "0.0.0.0", + "http_port": 8001 + } +} +``` + +## stdio + +JSON-RPC messages over stdin/stdout. No network listener, no auth required. + +```bash +AXON_MCP_TRANSPORT=stdio +axon mcp +``` + +### Claude Desktop configuration + +`~/.claude/claude_desktop_config.json`: + +```json +{ + "mcpServers": { + "axon": { + "command": "/path/to/axon", + "args": ["mcp"], + "env": { + "QDRANT_URL": "http://127.0.0.1:53333", + "TEI_URL": "http://127.0.0.1:52000", + "AXON_PG_URL": "postgresql://axon:pass@127.0.0.1:53432/axon", + "AXON_REDIS_URL": "redis://:pass@127.0.0.1:53379", + "AXON_AMQP_URL": "amqp://axon:pass@127.0.0.1:45535" + } + } + } +} +``` + +### Codex CLI configuration + +`.codex/mcp.json`: + +```json +{ + "mcpServers": { + "axon": { + "command": "/path/to/axon", + "args": ["mcp"], + "env": { + "QDRANT_URL": "http://127.0.0.1:53333", + "TEI_URL": "http://127.0.0.1:52000" + } + } + } +} +``` + +### When to use + +- Local development with Claude Desktop or Codex CLI +- Single-user setups where the binary runs as a child process +- Lite mode (`AXON_LITE=1`) for zero-infrastructure operation + +## HTTP + +Streamable-HTTP transport with MCP protocol support. Uses the `rmcp` crate with `transport-streamable-http-server` feature. + +```bash +AXON_MCP_TRANSPORT=http +AXON_MCP_HTTP_HOST=0.0.0.0 +AXON_MCP_HTTP_PORT=8001 +axon mcp +``` + +### Endpoints + +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/mcp` | POST | MCP JSON-RPC with streamable-http | +| `/health` | GET | Health check | + +### Claude Code configuration + +`.claude/settings.local.json`: + +```json +{ + "mcpServers": { + "axon": { + "type": "http", + "url": "http://localhost:8001/mcp" + } + } +} +``` + +### When to use + +- Docker deployments (axon-workers exposes port 8001) +- Remote/shared MCP server +- Multiple clients connecting to one server +- `axon serve` (automatically starts MCP HTTP on port 8001) + +## Both + +Run HTTP transport while also accepting stdio connections. + +```bash +AXON_MCP_TRANSPORT=both +axon mcp +``` + +This is useful for development: serve HTTP for web clients while also allowing local stdio connections. + +## Transport in `axon serve` + +When running `axon serve`, the MCP HTTP server starts automatically as part of the supervised process tree. No separate `axon mcp` invocation is needed. + +| Supervisor child | Port | Transport | +|-----------------|------|-----------| +| MCP HTTP server | 8001 | streamable-http | +| Backend bridge | 49000 | HTTP/WebSocket | +| Next.js | 49010 | HTTP | + +## Port assignments + +| Service | Default port | Env var | +|---------|-------------|---------| +| MCP HTTP | 8001 | `AXON_MCP_HTTP_PORT` | +| Backend bridge | 49000 | `AXON_SERVE_PORT` | +| Next.js web UI | 49010 | `AXON_WEB_DEV_PORT` | +| Shell WebSocket | 49011 | `SHELL_SERVER_PORT` | + +## See also + +- [CONNECT.md](CONNECT.md) -- client connection instructions +- [ENV.md](ENV.md) -- MCP environment variables +- [DEPLOY.md](DEPLOY.md) -- deployment patterns diff --git a/docs/repo/AGENTS.md b/docs/repo/AGENTS.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/repo/AGENTS.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/repo/CLAUDE.md b/docs/repo/CLAUDE.md new file mode 100644 index 00000000..dd83dfb4 --- /dev/null +++ b/docs/repo/CLAUDE.md @@ -0,0 +1,22 @@ +# Repository Documentation -- Axon + +Reference documentation for the repository structure, conventions, and tooling. + +## File index + +| File | Purpose | +|------|---------| +| [CLAUDE.md](CLAUDE.md) | This file -- index for repo/ documentation | +| [REPO.md](REPO.md) | Repository structure -- directory tree, workspace crates, root files | +| [RECIPES.md](RECIPES.md) | Justfile recipes -- dev, docker, quality, web | +| [SCRIPTS.md](SCRIPTS.md) | Scripts reference -- maintenance, hooks, testing | +| [RULES.md](RULES.md) | Coding rules -- git workflow, versioning, monolith policy | +| [MEMORY.md](MEMORY.md) | Memory files -- beads issue tracker and knowledge persistence | + +## Cross-references + +- [../SETUP.md](../SETUP.md) -- step-by-step setup guide +- [../CONFIG.md](../CONFIG.md) -- environment variable reference +- [../GUARDRAILS.md](../GUARDRAILS.md) -- security guardrails +- [../stack/TECH.md](../stack/TECH.md) -- technology stack +- [../stack/ARCH.md](../stack/ARCH.md) -- architecture overview diff --git a/docs/repo/GEMINI.md b/docs/repo/GEMINI.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/repo/GEMINI.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/repo/MEMORY.md b/docs/repo/MEMORY.md new file mode 100644 index 00000000..88b19b35 --- /dev/null +++ b/docs/repo/MEMORY.md @@ -0,0 +1,55 @@ +# Knowledge Persistence -- Axon + +Axon uses `bd` (beads) for issue tracking and persistent knowledge, not markdown TODO lists or MEMORY.md files. + +## Beads (bd) + +All task tracking and knowledge persistence uses the `bd` CLI: + +```bash +bd prime # Full workflow context and commands +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work +bd close # Complete work +bd remember # Save persistent knowledge +``` + +### Rules + +- Use `bd` for ALL task tracking -- do NOT use TodoWrite, TaskCreate, or markdown TODO lists +- Run `bd prime` for detailed command reference and session close protocol +- Use `bd remember` for persistent knowledge -- do NOT create MEMORY.md files + +## Session completion protocol + +When ending a work session, complete ALL steps: + +1. File issues for remaining work (`bd` create) +2. Run quality gates if code changed (`just verify`) +3. Update issue status (`bd close` / `bd update`) +4. Push to remote: + ```bash + git pull --rebase + bd dolt push + git push + git status # Must show "up to date with origin" + ``` +5. Clean up stashes and remote branches +6. Provide context for next session + +Work is NOT complete until `git push` succeeds. + +## What NOT to persist + +- Code patterns visible in the codebase (read the code) +- Git history facts (use `git log`) +- Debugging sessions (ephemeral) +- Temporary state +- Information already in `CLAUDE.md` or documentation files + +## Related + +- Root `CLAUDE.md` -- primary project instructions +- `docs/` -- structured documentation +- `bd prime` -- full beads workflow reference diff --git a/docs/repo/RECIPES.md b/docs/repo/RECIPES.md new file mode 100644 index 00000000..b48a82e0 --- /dev/null +++ b/docs/repo/RECIPES.md @@ -0,0 +1,100 @@ +# Justfile Recipes -- Axon + +Run `just --list` to see all available recipes. + +## Development + +| Recipe | Purpose | +|--------|---------| +| `just setup` | Bootstrap development environment (installs all dependencies) | +| `just check` | Fast type check (`cargo check`) | +| `just check-tests` | Type check including test code | +| `just test` | Run all tests (prefers cargo-nextest, falls back to cargo test) | +| `just test-fast` | Run lib tests only (no integration tests) | +| `just test-infra` | Run infrastructure integration tests (requires services) | +| `just test-all` | Run all targets with all features | +| `just fmt` | Format all Rust code | +| `just fmt-check` | Check formatting without modifying | +| `just clippy` | Run clippy lints | +| `just fix` | Auto-fix: format + clippy --fix | +| `just fix-all` | Fix Rust + web (pnpm format) | +| `just clean` | Remove build artifacts | + +## Quality gates + +| Recipe | Purpose | +|--------|---------| +| `just verify` | Full CI gate: dockerignore check + fmt + clippy + check + test | +| `just ci` | Alias for verify | +| `just precommit` | Full pre-commit: monolith check + verify | +| `just lint-all` | Rust fmt-check + clippy + web lint | + +## Docker + +| Recipe | Purpose | +|--------|---------| +| `just docker-build` | Build workers Docker image (`axon:local`) | +| `just services-up` | Start infrastructure (Postgres, Redis, RabbitMQ, Qdrant, TEI, Chrome) | +| `just services-down` | Stop infrastructure | +| `just up` | Build and start app containers (workers + web) | +| `just down` | Stop app containers | +| `just down-all` | Stop everything (app + infrastructure) | +| `just rebuild-fresh` | Full rebuild: check + build + start containers | +| `just rebuild` | check + test + docker-build | + +## Local stack + +| Recipe | Purpose | +|--------|---------| +| `just dev` | Full local dev: stop existing, start infra, build, run `axon serve` | +| `just serve [port]` | Run `axon serve` (debug build, default port 49000) | +| `just serve-release [port]` | Run `axon serve` (release build) | +| `just workers` | Start all 6 worker types as background processes | +| `just stop` | Kill running axon serve, workers, and Next.js processes | + +## Web UI + +| Recipe | Purpose | +|--------|---------| +| `just web-dev` | Start Next.js dev server standalone | +| `just web-build` | Build Next.js for production | +| `just web-lint` | Run Biome linter on web code | +| `just web-format` | Format web code with Biome | + +## Testing + +| Recipe | Purpose | +|--------|---------| +| `just mcp-smoke` | Run MCP tool smoke tests | +| `just test-infra-up` | Start test infrastructure containers | +| `just test-infra-down` | Stop test infrastructure | + +## Build tools + +| Recipe | Purpose | +|--------|---------| +| `just build` | Release build (`cargo build --release`) | +| `just install` | Build release + symlink to `~/.local/bin/axon` | +| `just nextest-install` | Install cargo-nextest | +| `just llvm-cov-install` | Install cargo-llvm-cov for coverage | +| `just coverage-branch` | Generate lcov coverage report | + +## Maintenance + +| Recipe | Purpose | +|--------|---------| +| `just gen-mcp-schema` | Regenerate MCP-TOOL-SCHEMA.md from source | +| `just cache-status` | Check build cache status | +| `just cache-prune` | Prune build cache | +| `just docker-context-probe` | Check Docker build context size | +| `just check-container-revisions` | Verify container git SHA matches | +| `just watch-check` | cargo-watch: check + test on every save | + +## Chaining + +```bash +just fmt clippy check test # Run quality checks in sequence +just verify # Same as above, single command +``` + +The `verify` recipe is the standard pre-PR gate. diff --git a/docs/repo/REPO.md b/docs/repo/REPO.md new file mode 100644 index 00000000..4de42453 --- /dev/null +++ b/docs/repo/REPO.md @@ -0,0 +1,154 @@ +# Repository Structure -- Axon + +## Directory tree + +``` +axon_rust/ +├── apps/ +│ └── web/ # Next.js dashboard (Pulse workspace, omnibox) +│ ├── app/ # Next.js app router pages +│ ├── components/ # React components +│ ├── hooks/ # React hooks (WebSocket, API) +│ ├── lib/ # Shared utilities +│ ├── proxy.ts # API proxy with token auth +│ ├── shell-server.mjs # Shell WebSocket server +│ ├── biome.json # Linter/formatter config +│ ├── package.json # Node dependencies +│ └── CLAUDE.md # Web-specific development instructions +│ +├── crates/ # Rust workspace crates (module roots) +│ ├── cli.rs # CLI module root +│ ├── cli/ # Command handlers +│ │ └── commands/ # Per-command handlers (scrape, crawl, ask, etc.) +│ ├── core.rs # Core module root +│ ├── core/ # Config, HTTP client, content processing +│ │ └── config/ # CLI flags, env parsing, runtime config +│ ├── crawl.rs # Crawl module root +│ ├── crawl/ # Spider-based crawl engine +│ ├── ingest.rs # Ingest module root +│ ├── ingest/ # GitHub, Reddit, YouTube adapters +│ ├── jobs.rs # Jobs module root +│ ├── jobs/ # Async job framework +│ │ ├── common/ # Shared job infrastructure +│ │ ├── crawl/ # Crawl job processor, worker, runtime +│ │ ├── extract/ # Extract job processor +│ │ ├── embed/ # Embed job processor +│ │ └── ingest.rs # Ingest job processor +│ ├── mcp.rs # MCP module root +│ ├── mcp/ # MCP schema and server +│ │ ├── schema.rs # Tool input schema, action enums +│ │ └── server.rs # Handler dispatch, transport setup +│ ├── services.rs # Services module root +│ ├── services/ # Typed service layer +│ │ ├── context.rs # ServiceContext and capabilities +│ │ ├── types/ # Result structs +│ │ ├── acp/ # ACP session lifecycle +│ │ └── acp_llm/ # ACP-backed LLM completions +│ ├── vector.rs # Vector module root +│ ├── vector/ # Qdrant ops, TEI, hybrid search +│ │ └── ops/ # TEI embed, Qdrant upsert/search, ask +│ ├── web.rs # WebSocket execution bridge +│ └── web/ # Web-specific handlers +│ +├── docker/ # Container builds and s6 supervision +│ ├── Dockerfile # Multi-stage: cargo-chef -> build -> runtime +│ ├── chrome/ # Headless Chrome + CDP proxy +│ ├── web/ # Next.js + s6-overlay +│ │ ├── Dockerfile +│ │ ├── cont-init.d/ # Container init scripts +│ │ └── s6-rc.d/ # s6 service definitions +│ ├── rabbitmq/ # Preconfigured RabbitMQ +│ ├── s6/ # Worker s6 service definitions +│ │ ├── cont-init.d/ +│ │ └── s6-rc.d/ +│ └── CLAUDE.md # Docker build instructions +│ +├── docs/ # Documentation (this directory) +├── migrations/ # SQL migrations +├── scripts/ # Maintenance, hooks, testing scripts +├── tests/ # Integration tests +├── config/ # Additional config files +├── specs/ # Specifications +│ +├── main.rs # Binary entry point +├── lib.rs # Library root (run/run_once, command dispatch) +├── crates.rs # Workspace crate re-exports +├── Cargo.toml # Rust package manifest +├── Cargo.lock # Dependency lock file +├── Justfile # Task runner recipes +├── axon.json # Runtime configuration (158 keys) +├── axon.schema.json # JSON Schema for axon.json +├── lefthook.yml # Git hooks +├── deny.toml # cargo-deny config +├── renovate.json # Dependency update bot +├── rust-toolchain.toml # Rust 1.94.0 pinned toolchain +│ +├── docker-compose.yaml # App containers (workers + web) +├── docker-compose.services.yaml # Infrastructure services +├── docker-compose.gpu.yaml # GPU override for TEI/Ollama +├── docker-compose.test.yaml # Test infrastructure +├── .env.example # Environment variable template +├── services.env # Infrastructure container credentials +│ +├── CLAUDE.md # Project instructions for Claude Code +├── AGENTS.md -> CLAUDE.md # Codex agent alias +├── GEMINI.md -> CLAUDE.md # Gemini agent alias +├── README.md # User-facing documentation +└── CHANGELOG.md # Version history +``` + +## Workspace crates + +Axon uses a flat module layout rooted at `crates/`. Each crate has a module root file (`crates/.rs`) and a subdirectory (`crates//`). + +| Crate | Module root | Purpose | +|-------|------------|---------| +| cli | `crates/cli.rs` | CLI command handlers -- one file per subcommand | +| core | `crates/core.rs` | Config parsing, HTTP client, content/markdown processing | +| crawl | `crates/crawl.rs` | Spider-based crawl engine, render mode switching | +| ingest | `crates/ingest.rs` | Source adapters (GitHub, Reddit, YouTube) | +| jobs | `crates/jobs.rs` | Async job framework with AMQP and SQLite backends | +| mcp | `crates/mcp.rs` | MCP server schema definition and handler dispatch | +| services | `crates/services.rs` | Typed service layer consumed by CLI, MCP, and web | +| vector | `crates/vector.rs` | Qdrant operations, TEI embedding, hybrid search | +| web | `crates/web.rs` | WebSocket execution bridge for web UI | + +### Module layout convention (enforced) + +Rust 2018+ file-per-module layout. `mod.rs` is forbidden: + +``` +# Correct +foo.rs <- module root +foo/ + bar.rs <- submodule + +# Wrong +foo/ + mod.rs <- forbidden +``` + +Enforced by `scripts/check_no_mod_rs.sh`. + +## Root files + +| File | Required | Purpose | +|------|----------|---------| +| `CLAUDE.md` | Yes | Project instructions (37K, comprehensive) | +| `README.md` | Yes | User-facing documentation (55K) | +| `CHANGELOG.md` | Yes | Version history | +| `.env.example` | Yes | Environment template (150+ variables) | +| `Justfile` | Yes | Task runner (30+ recipes) | +| `axon.json` | Yes | Runtime configuration (158 keys, schema-validated) | +| `Cargo.toml` | Yes | Rust package manifest | +| `main.rs` | Yes | Binary entry point | +| `lib.rs` | Yes | Library root with command dispatch | + +## Docker compose files + +| File | Contents | Network | +|------|----------|---------| +| `docker-compose.services.yaml` | PostgreSQL, Redis, RabbitMQ, Qdrant, TEI, Chrome | `axon` bridge | +| `docker-compose.yaml` | axon-workers, axon-web | `axon` bridge | +| `docker-compose.gpu.yaml` | NVIDIA GPU reservations overlay | -- | +| `docker-compose.test.yaml` | Test infrastructure | -- | diff --git a/docs/repo/RULES.md b/docs/repo/RULES.md new file mode 100644 index 00000000..872c22c9 --- /dev/null +++ b/docs/repo/RULES.md @@ -0,0 +1,130 @@ +# Coding Rules -- Axon + +Standards and conventions enforced across the Axon codebase. + +## Git workflow + +### Conventional commits + +| Prefix | Purpose | Example | +|--------|---------|---------| +| `feat:` | New feature | `feat(mcp): add screenshot action` | +| `fix:` | Bug fix | `fix(crawl): handle timeout on sitemap backfill` | +| `chore:` | Maintenance | `chore: update spider to 2.47` | +| `refactor:` | Code restructure | `refactor(services): extract ask pipeline` | +| `test:` | Tests | `test(vector): add hybrid search tests` | +| `docs:` | Documentation | `docs: update CONFIG reference` | +| `ci:` | CI/CD changes | `ci: add nextest to verify` | + +### Branch strategy + +- `main` is production-ready +- Feature branches for development +- PR required before merge + +### Never commit + +- `.env` or `services.env` files +- API keys, tokens, or passwords +- Large binary files +- `target/`, `node_modules/`, `.next/` + +## Version bumping + +### Bump type rules + +| Commit prefix | Bump | Example | +|---------------|------|---------| +| `feat!:` or `BREAKING CHANGE` | Major | `0.35.0` -> `1.0.0` | +| `feat:` or `feat(...):` | Minor | `0.35.0` -> `0.36.0` | +| Everything else | Patch | `0.35.0` -> `0.35.1` | + +### Version-bearing files + +All must have the same version: + +| File | Field | +|------|-------| +| `Cargo.toml` | `version = "X.Y.Z"` in `[package]` | +| `apps/web/package.json` | `"version": "X.Y.Z"` | +| `CHANGELOG.md` | New entry under `## X.Y.Z` | + +## Monolith policy (enforced) + +Changed `.rs` files are checked at CI and via lefthook pre-commit: + +| Metric | Warn | Fail | +|--------|------|------| +| File size | -- | 500 lines | +| Function size | 80 lines | 120 lines | + +Exempt: `tests/**`, `benches/**`, `config/**`, `**/config.rs`. + +Exceptions: add to `.monolith-allowlist`. + +Enforcement: `scripts/enforce_monoliths.py` runs on staged files. + +## Module layout (enforced) + +Rust 2018+ file-per-module layout. `mod.rs` is forbidden: + +``` +foo.rs <- module root (declarations: mod bar; mod baz;) +foo/ + bar.rs <- submodule + baz.rs <- submodule +``` + +Enforcement: `scripts/check_no_mod_rs.sh`. + +## Rust code standards + +- `cargo fmt` before committing +- `cargo clippy` clean (all warnings are errors in CI) +- `unsafe` code is denied (`#[deny(unsafe_code)]` in `Cargo.toml`) +- Errors: `Box` at command boundaries, typed errors internally +- Logging: `log_info` / `log_warn` (not `println!` in library code) +- `--json` flag enables machine-readable output on all result-printing commands +- Structured log output via `tracing` with `env-filter` + +### Services layer contract + +- CLI commands, MCP handlers, and web routes all call through `crates/services/` +- Each service function returns a typed result struct (no raw JSON, no stdout side-effects) +- Service result types live in `crates/services/types/service.rs` + +## TypeScript code standards (web UI) + +- ESM modules, `import` syntax +- Biome for linting and formatting +- No `any` types +- Strict mode in `tsconfig.json` +- Next.js App Router conventions + +## Pre-commit hooks (lefthook) + +| Hook | Purpose | +|------|---------| +| `enforce_monoliths.py` | File and function size limits | +| `enforce_no_legacy_symbols.py` | Block deprecated names | +| `check_dockerignore_guards.sh` | Docker ignore patterns | +| `check_env_staged.sh` | Block .env commits | +| `check_no_mod_rs.sh` | No mod.rs files | +| `warn_new_unwraps.sh` | Flag new .unwrap() calls | + +Install hooks: + +```bash +./scripts/install-git-hooks.sh +``` + +## Performance profiles + +Concurrency is tuned relative to available CPU cores: + +| Profile | Crawl | Sitemap | Backfill | Timeout | Retries | +|---------|-------|---------|----------|---------|---------| +| `high-stable` (default) | CPUs x 8 | CPUs x 12 | CPUs x 6 | 20s | 2 | +| `balanced` | CPUs x 4 | CPUs x 6 | CPUs x 3 | 30s | 2 | +| `extreme` | CPUs x 16 | CPUs x 20 | CPUs x 10 | 15s | 1 | +| `max` | CPUs x 24 | CPUs x 32 | CPUs x 20 | 12s | 1 | diff --git a/docs/repo/SCRIPTS.md b/docs/repo/SCRIPTS.md new file mode 100644 index 00000000..21ab0107 --- /dev/null +++ b/docs/repo/SCRIPTS.md @@ -0,0 +1,109 @@ +# Scripts Reference -- Axon + +Scripts in the `scripts/` directory for maintenance, testing, and development. + +## Wrapper script + +| Script | Purpose | +|--------|---------| +| `scripts/axon` | Shell wrapper that auto-sources `.env` before running the binary. Use this instead of the bare binary for local dev. | + +Usage: +```bash +./scripts/axon doctor +./scripts/axon scrape https://example.com --wait true +``` + +## Development setup + +| Script | Purpose | +|--------|---------| +| `dev-setup.sh` | Bootstrap development environment: installs Rust, just, pnpm, cargo tools | + +## Quality checks + +| Script | Purpose | +|--------|---------| +| `enforce_monoliths.py` | Enforce file size (500 lines) and function size (120 lines) limits on `.rs` files | +| `enforce_no_legacy_symbols.py` | Block deprecated function/type names | +| `check_dockerignore_guards.sh` | Verify `.dockerignore` contains required patterns | +| `check_env_staged.sh` | Block commits that include `.env` files | +| `check_no_mod_rs.sh` | Enforce no `mod.rs` files (Rust 2018+ convention) | +| `check_no_next_middleware.sh` | Block Next.js middleware files | +| `check_pg_advisory_lock.sh` | Verify advisory lock usage | +| `check_shell_completions.sh` | Verify shell completion generation | +| `check_mcp_http_only.sh` | Verify MCP transport configuration | + +## Docker and deployment + +| Script | Purpose | +|--------|---------| +| `rebuild-fresh.sh` | Build Docker images and start containers | +| `check-container-revisions.sh` | Verify container git SHA matches local HEAD | +| `check_docker_context_size.sh` | Audit Docker build context for large files | +| `audit_compose_images.py` | Audit Docker Compose image versions | +| `cache-guard.sh` | Build cache management (status/prune) | + +## Testing + +| Script | Purpose | +|--------|---------| +| `test-mcp-tools-mcporter.sh` | MCP smoke test suite (50+ tool calls) | +| `live-test-all-commands.sh` | Integration test all CLI commands against live services | +| `test-ask-quality-regressions.sh` | RAG answer quality regression tests | +| `test-mcp-oauth-protection.sh` | MCP OAuth endpoint security tests | +| `test_qdrant_quality.py` | Qdrant data quality analysis | + +## Code generation + +| Script | Purpose | +|--------|---------| +| `generate_mcp_schema_doc.py` | Regenerate `docs/MCP-TOOL-SCHEMA.md` from source | +| `mcp_doc_renderer.py` | MCP schema documentation renderer | +| `mcp_schema_parser.py` | MCP schema parser | +| `mcp_schema_models.py` | MCP schema model definitions | + +## Data management + +| Script | Purpose | +|--------|---------| +| `reingest.py` | Re-ingest sources from export manifest | +| `migrate_legacy_source_urls.py` | Migrate legacy source URL formats | +| `extract-base-urls.sh` | Extract base URLs from indexed sources | +| `list-all-domains.sh` | List all indexed domains | + +## Analysis + +| Script | Purpose | +|--------|---------| +| `qdrant-quality.py` | Entry point for Qdrant quality analysis | +| `qdrant_quality_analysis.py` | Quality analysis implementation | +| `qdrant_quality_client.py` | Qdrant client for analysis | +| `qdrant_quality_impl.py` | Quality metrics implementation | +| `qdrant_quality_models.py` | Quality model definitions | +| `qdrant_quality_reporting.py` | Quality reporting | +| `qdrant_quality_runtime.py` | Quality runtime | +| `qdrant_quality_settings.py` | Quality settings | +| `qdrant_quality_ui.py` | Quality UI | + +## Miscellaneous + +| Script | Purpose | +|--------|---------| +| `cleanup-claude.sh` | Clean up Claude Code artifacts | +| `install-git-hooks.sh` | Install lefthook git hooks | +| `install-agent-skill.sh` | Install agent skill symlinks | +| `validate_skills_ref.sh` | Validate skill references | +| `warn_new_unwraps.sh` | Warn about new `.unwrap()` calls in staged code | +| `hook_deny_audit_sync.py` | Hook: verify cargo-deny audit | +| `hook_justfile_lefthook_sync.py` | Hook: verify Justfile/lefthook sync | + +## Conventions + +All scripts follow these rules: + +- Bash scripts use `set -euo pipefail` (strict mode) +- Python scripts use type hints +- All variables are quoted (`"$var"`) +- Scripts are executable (`chmod +x`) +- Exit code 0 = success, 1 = failure, 2 = usage error diff --git a/docs/stack/AGENTS.md b/docs/stack/AGENTS.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/stack/AGENTS.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/stack/ARCH.md b/docs/stack/ARCH.md new file mode 100644 index 00000000..60ba2fbf --- /dev/null +++ b/docs/stack/ARCH.md @@ -0,0 +1,213 @@ +# Architecture Overview -- Axon + +## Trimodal design + +Axon is a single Rust binary that operates in three modes: + +``` + +-----------+ + | axon.rs | (single binary) + +-----+-----+ + | + +---------------+---------------+ + | | | + +-----+-----+ +-----+-----+ +------+------+ + | CLI mode | | MCP mode | | Serve mode | + | axon | | axon mcp | | axon serve | + +-----+-----+ +-----+-----+ +------+------+ + | | | + +-------+-------+ +-------+-------+ + | | | + +-----+-----+ +-----+-----+ +------+------+ + | Services | | Backend | | Next.js | + | Layer | | Bridge | | Web UI | + +-----+-----+ | (port | | (port | + | | 49000) | | 49010) | + | +-----+-----+ +------+------+ + | | | + +-------+-------+------+ | + | | | + +-----+-----+ +-----+-----+ | + | Jobs | | Vector | +-----------+ | + | Framework | | Ops | | MCP | | + | | | | | HTTP | | + +-----+-----+ +-----+-----+ | (port | | + | | | 8001) | | + | | +----+----+ | + | | | | + +-----+-----+ +-----+-----+ | | + | Postgres | | Qdrant | +------+------+ + | Redis | | (vector | | Workers | + | RabbitMQ | | store) | | (6 types) | + +-----------+ +-----------+ +-------------+ +``` + +All three modes share the same services layer (`crates/services/`), ensuring consistent behavior across CLI, MCP, and web interfaces. + +## Services layer + +The services layer is the API boundary between all consumers (CLI, MCP, web) and the underlying infrastructure: + +``` +CLI handlers ─┐ +MCP handlers ─┼── services::{query, ask, sources, ...} ── vector/ops, jobs, ... +Web routes ─┘ +``` + +Each service function: +- Takes typed input parameters +- Returns a typed result struct (defined in `crates/services/types/service.rs`) +- Has no stdout side-effects +- Can be called from any entry point + +## Worker topology + +Six worker types process jobs from RabbitMQ queues: + +| Worker | Queue | Processing | +|--------|-------|------------| +| Crawl | `axon.crawl.jobs` | Spider-based site crawling with render mode switching | +| Extract | `axon.extract.jobs` | LLM-powered structured data extraction | +| Embed | `axon.embed.jobs` | TEI embedding + Qdrant upsert | +| Ingest | `axon.ingest.jobs` | Source ingestion (GitHub, Reddit, YouTube) | +| Refresh | `axon.refresh.jobs` | Periodic URL re-indexing with schedule support | +| Graph | `axon.graph.jobs` | Neo4j entity extraction and graph building | + +### Worker deployment + +**Docker (production):** Workers run as s6-supervised services inside `axon-workers` container. Each drops to UID 1001 via `s6-setuidgid`. + +**Local dev:** `axon serve` supervises all workers in-process. Or run individually: `axon crawl worker`. + +**Lite mode:** Workers run in the same tokio runtime as the CLI. No separate processes. + +## Async job lifecycle + +``` +Submitted ─> Pending ─> Running ─> Completed + │ │ + │ ├─> Failed + │ └─> Cancelled + └─> Stale (watchdog reclaim) +``` + +Jobs are persisted in PostgreSQL (full mode) or SQLite (lite mode). The `JobBackend` trait abstracts the storage backend. + +Key behaviors: +- `--wait false` (default): fire-and-forget, returns job ID immediately +- `--wait true`: blocks until completion +- Stale detection: `AXON_JOB_STALE_TIMEOUT_SECS` (300s) + confirmation grace period +- Cancel: sets flag in Redis, worker checks on next iteration + +## Data flow: crawl to RAG + +``` +1. axon crawl https://docs.example.com + ├── Spider crawls pages (HTTP or Chrome) + ├── Auto-switch: HTTP first, Chrome if >60% thin pages + ├── Sitemap backfill discovers missed URLs + └── Pages saved as markdown + +2. axon embed (automatic after crawl) + ├── chunk_text(): 2000 chars, 200 overlap + ├── TEI generates dense embeddings + ├── BM42 sparse vectors computed locally + └── Qdrant upsert (named-mode: dense + sparse) + +3. axon ask "How does X work?" + ├── Hybrid search: dense + BM42 with RRF fusion + ├── Candidate selection and re-ranking + ├── Context assembly (120K char limit) + ├── ACP adapter generates answer with citations + └── Optional: Neo4j graph context injection +``` + +## MCP request flow + +``` +MCP Client (Claude Code / Codex / Gemini) + │ + ▼ +Transport (stdio / streamable-http) + │ + ▼ +rmcp framework (JSON-RPC handling) + │ + ▼ +AxonMcpServer::call_tool() + │ + ▼ +Schema parser (serde strict parsing) + │ + ▼ +Action dispatcher (match on action enum) + │ + ▼ +Service function (typed result) + │ + ▼ +Response formatter (artifact or inline) + │ + ▼ +MCP response (canonical envelope) +``` + +## Web UI architecture + +The web UI consists of: + +| Component | Technology | Purpose | +|-----------|-----------|---------| +| Next.js app | React + App Router | Dashboard, omnibox, Pulse workspace | +| Backend bridge | Axum + WebSocket | Command execution, Docker stats | +| Shell server | Node.js WebSocket | Terminal emulation | +| Proxy | Next.js API routes | Auth + rewrite to backend | + +The web UI communicates with the backend via: +- HTTP API routes (proxied through Next.js `/api/*`) +- WebSocket connections for real-time command output +- Shell WebSocket for terminal sessions + +### Auth model + +Two-tier token system: +1. `AXON_WEB_API_TOKEN` -- server-only, gates `/api/*` and `/ws` +2. `AXON_WEB_BROWSER_API_TOKEN` -- browser-visible, gates `/api/*` only + +## Serve supervisor + +`axon serve` acts as a process supervisor, managing: + +| Child process | Default port | Purpose | +|--------------|-------------|---------| +| Backend bridge | 49000 | HTTP/WS API server | +| MCP HTTP | 8001 | MCP streamable-http server | +| Workers (6) | -- | Job processors | +| Shell server | 49011 | Terminal WebSocket | +| Next.js | 49010 | Web UI dev server | + +All children are supervised: if one exits, the supervisor restarts it. + +## Configuration resolution + +``` +CLI flags (highest precedence) + │ + ▼ +Environment variables ($AXON_*) + │ + ▼ +axon.json (158 keys, schema-validated) + │ + ▼ +Built-in defaults (lowest precedence) +``` + +The `Config` struct in `crates/core/config.rs` merges all sources at startup. + +## See also + +- [TECH.md](TECH.md) -- technology choices +- [PRE-REQS.md](PRE-REQS.md) -- prerequisites +- [../mcp/PATTERNS.md](../mcp/PATTERNS.md) -- MCP code patterns +- [../ARCHITECTURE.md](../ARCHITECTURE.md) -- detailed architecture doc diff --git a/docs/stack/CLAUDE.md b/docs/stack/CLAUDE.md new file mode 100644 index 00000000..62805006 --- /dev/null +++ b/docs/stack/CLAUDE.md @@ -0,0 +1,19 @@ +# Technology Stack Documentation -- Axon + +Reference documentation for the technology choices, architecture, and prerequisites. + +## File index + +| File | Purpose | +|------|---------| +| [CLAUDE.md](CLAUDE.md) | This file -- index for stack/ documentation | +| [ARCH.md](ARCH.md) | Architecture overview -- trimodal design, worker topology, data flow | +| [TECH.md](TECH.md) | Technology choices -- Rust, Spider, Qdrant, hybrid search, ACP | +| [PRE-REQS.md](PRE-REQS.md) | Prerequisites -- tools and versions for development | + +## Cross-references + +- [../repo/REPO.md](../repo/REPO.md) -- repository structure +- [../repo/RECIPES.md](../repo/RECIPES.md) -- Justfile recipes +- [../CONFIG.md](../CONFIG.md) -- configuration reference +- [../mcp/DEPLOY.md](../mcp/DEPLOY.md) -- deployment patterns diff --git a/docs/stack/GEMINI.md b/docs/stack/GEMINI.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/stack/GEMINI.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/stack/PRE-REQS.md b/docs/stack/PRE-REQS.md new file mode 100644 index 00000000..ddd1c731 --- /dev/null +++ b/docs/stack/PRE-REQS.md @@ -0,0 +1,109 @@ +# Prerequisites -- Axon + +Required tools and versions before developing or deploying. + +## Required tools + +| Tool | Version | Install | Purpose | +|------|---------|---------|---------| +| Rust | 1.94+ | [rustup.rs](https://rustup.rs/) | Compiler (pinned via `rust-toolchain.toml`) | +| Docker | 24+ | [docs.docker.com](https://docs.docker.com/get-docker/) | Infrastructure services | +| Docker Compose | v2+ | Bundled with Docker | Service orchestration | +| just | latest | `cargo install just` | Task runner | +| Node.js | 22+ | [nodejs.org](https://nodejs.org/) | Web UI runtime | +| pnpm | 9+ | `corepack enable` | Web UI package manager | +| jq | 1.6+ | System package manager | JSON parsing in scripts | +| curl | any | System package manager | HTTP testing | +| Python | 3.10+ | System package manager | Scripts and analysis tools | + +### Verify + +```bash +rustc --version # rustc 1.94.x +cargo --version # cargo 1.94.x +docker --version # Docker 24+ +docker compose version # Docker Compose v2+ +just --version # just x.y.z +node --version # v22.x.x +pnpm --version # 9.x.x +jq --version # jq-1.6+ +python3 --version # Python 3.10+ +``` + +## Automated setup + +The setup script installs everything: + +```bash +just setup +# or directly: +./scripts/dev-setup.sh +``` + +This installs: Rust toolchain components, cargo tools, pnpm, and verifies all prerequisites. If `just` is not yet installed, `dev-setup.sh` installs it. + +## Optional tools (recommended) + +| Tool | Install | Purpose | +|------|---------|---------| +| sccache | `cargo install sccache` | Compilation cache (auto-detected by Justfile) | +| mold | System package manager | Fast linker (auto-detected by Justfile) | +| cargo-nextest | `cargo install cargo-nextest` | Faster parallel test runner | +| cargo-deny | `cargo install cargo-deny` | Dependency auditing | +| cargo-llvm-cov | `cargo install cargo-llvm-cov` | Code coverage | +| lefthook | `cargo install lefthook` | Git hooks | +| gh | [cli.github.com](https://cli.github.com/) | GitHub CLI for PRs and issues | + +## Infrastructure services + +Infrastructure runs via Docker Compose. No manual installation needed: + +| Service | Docker image | Host port | Purpose | +|---------|-------------|-----------|---------| +| PostgreSQL | postgres:17-alpine | 53432 | Job persistence | +| Redis | redis:8.2-alpine | 53379 | Queue state, caching | +| RabbitMQ | rabbitmq:4.0-management | 45535 | AMQP job queue | +| Qdrant | qdrant/qdrant:v1.13.1 | 53333 | Vector store | +| TEI | ghcr.io/huggingface/text-embeddings-inference | 52000 | Embedding generation | +| Chrome | Custom (docker/chrome/) | 6000 | Headless browser | + +Start all infrastructure: + +```bash +just services-up +``` + +### GPU requirements (optional) + +TEI benefits from GPU acceleration for embedding generation: +- NVIDIA GPU with CUDA drivers +- `docker compose -f docker-compose.services.yaml -f docker-compose.gpu.yaml up -d` +- CPU-only: works without GPU overlay, but slower embedding + +### System resources + +Recommended minimums for local development: + +| Resource | Minimum | Recommended | +|----------|---------|-------------| +| RAM | 8 GB | 16+ GB | +| Disk | 20 GB | 50+ GB (for Qdrant data and TEI models) | +| CPU | 4 cores | 8+ cores (concurrency scales with cores) | + +## Quick start + +```bash +git clone https://github.com/jmagar/axon.git ~/workspace/axon_rust +cd ~/workspace/axon_rust +just setup # Install all tools +cp .env.example .env && chmod 600 .env +# Edit .env with credentials +just services-up # Start infrastructure +just dev # Build + run full stack +``` + +## See also + +- [../SETUP.md](../SETUP.md) -- detailed setup guide +- [TECH.md](TECH.md) -- technology stack details +- [../repo/RECIPES.md](../repo/RECIPES.md) -- Justfile recipes diff --git a/docs/stack/TECH.md b/docs/stack/TECH.md new file mode 100644 index 00000000..5c01af77 --- /dev/null +++ b/docs/stack/TECH.md @@ -0,0 +1,139 @@ +# Technology Choices -- Axon + +## Language and runtime + +| Component | Technology | Version | Purpose | +|-----------|-----------|---------|---------| +| Core binary | Rust | 1.94+ (edition 2024) | CLI, MCP server, workers, backend bridge | +| Web UI | TypeScript + Next.js | Node 22+ | Dashboard, Pulse workspace | +| Scripts | Bash + Python | -- | Maintenance, testing, analysis | + +## Key dependencies + +### Rust crates + +| Crate | Version | Purpose | +|-------|---------|---------| +| `spider` | 2.x | Web crawling engine (HTTP + Chrome rendering) | +| `spider_agent` | 2.47+ | Tavily search integration | +| `spider_transformations` | 2.x | Content transformation (markdown, readability) | +| `rmcp` | 1.1+ | MCP server framework (stdio + streamable-http) | +| `axum` | 0.8 | HTTP server + WebSocket for backend bridge | +| `tokio` | 1.x | Async runtime (multi-threaded) | +| `sqlx` | 0.8 | PostgreSQL + SQLite async driver | +| `lapin` | 4.x | RabbitMQ AMQP client | +| `redis` | 1.0 | Redis client (tokio-comp) | +| `reqwest` | 0.13 | HTTP client (rustls, streaming) | +| `clap` | 4.x | CLI argument parsing | +| `serde` / `serde_json` | 1.x | Serialization | +| `text-splitter` | 0.29 | Semantic text chunking (code + markdown) | +| `tree-sitter-*` | various | AST-based code chunking (Rust, Python, JS, TS, Go, Bash) | +| `octocrab` | 0.49 | GitHub API client | +| `bollard` | 0.20 | Docker API client | +| `agent-client-protocol` | 0.10+ | ACP adapter protocol | + +### Infrastructure + +| Service | Image/Version | Purpose | +|---------|--------------|---------| +| PostgreSQL | 17-alpine | Job persistence, metadata storage | +| Redis | 8.2-alpine | Queue state, cancel flags, caching | +| RabbitMQ | 4.0-management | AMQP job queue with 6 queues | +| Qdrant | v1.13.1 | Vector database (dense + sparse search) | +| TEI | HuggingFace latest | Text embedding generation | +| Chrome | Custom Dockerfile | Headless browser for JavaScript rendering | + +### Web UI (Next.js) + +| Package | Purpose | +|---------|---------| +| Next.js (App Router) | React framework | +| Biome | Linter and formatter | +| shadcn/ui | Component library | +| pnpm | Package manager | + +## Embedding pipeline + +### TEI (Text Embeddings Inference) + +- Default model: `Qwen/Qwen3-Embedding-0.6B` +- Pooling: `last-token` +- Batch size: up to 128 (auto-splits on 413 Payload Too Large) +- Retry: 5 attempts with exponential backoff (1s, 2s, 4s, 8s + jitter) +- GPU acceleration via NVIDIA (optional; CPU fallback available) + +### Text chunking + +- `chunk_text()`: 2000 characters with 200-character overlap +- Code files: tree-sitter AST-based chunking (preserves function boundaries) +- Each chunk becomes one Qdrant point with `chunk_text` payload field + +## Hybrid vector search + +New Qdrant collections use named vectors with two search paths: + +| Vector | Type | Source | Purpose | +|--------|------|--------|---------| +| `dense` | Float (dimension matches model) | TEI embedding | Semantic similarity | +| `bm42` | Sparse | Computed locally from chunk text | Keyword matching | + +Search uses Reciprocal Rank Fusion (RRF) via Qdrant `/query` API: +1. Dense prefetch: HNSW search (`hnsw_ef=128`) +2. Sparse prefetch: BM42 index search +3. RRF fusion: merge and re-rank results + +Legacy unnamed-mode collections fall back to dense-only search. Use `axon migrate` to upgrade. + +### Tuning + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `AXON_HYBRID_SEARCH` | `true` | Enable hybrid search | +| `AXON_HYBRID_CANDIDATES` | `100` | Prefetch candidates per arm | +| `AXON_ASK_HYBRID_CANDIDATES` | `150` | Ask pipeline window (higher for reranking) | +| `AXON_HNSW_EF_SEARCH` | `128` | HNSW ef for named-mode (32-512) | + +## Crawl engine + +Spider-based crawling with three render modes: + +| Mode | Description | +|------|-------------| +| `http` | Pure HTTP fetch (fastest, no JS) | +| `chrome` | Headless Chrome rendering (JS-heavy sites) | +| `auto-switch` (default) | HTTP first; if >60% thin pages, retry with Chrome | + +Key Spider features enabled: `basic`, `chrome`, `regex`, `sitemap`, `adblock`, `chrome_stealth`, `firewall`, `ua_generator`. + +Features explicitly NOT enabled: +- `balance`: silently throttles with zero logging +- `glob`: causes budget-aware `is_allowed()` to reject first URL with `with_limit(1)` + +## ACP (Agent Client Protocol) + +LLM synthesis operations (`ask`, `evaluate`, `suggest`, `research`, `extract` fallback, `debug`) use ACP: + +- Adapter subprocess spawned via `AXON_ACP_ADAPTER_CMD` (e.g., `claude`, `codex`, `gemini`) +- Pre-warming eliminates cold-start latency (~45s) +- `OPENAI_MODEL` controls the model used for completions +- Max concurrent sessions: 8 (configurable) +- Per-turn timeout: 5 minutes (configurable) +- Remote ACP: route completions through a remote `axon serve` WebSocket + +## Build tooling + +| Tool | Purpose | +|------|---------| +| just | Task runner (30+ recipes) | +| lefthook | Git hooks | +| sccache | Compilation cache (auto-detected) | +| mold | Fast linker (auto-detected) | +| cargo-nextest | Parallel test runner | +| cargo-deny | Dependency auditing | +| cargo-llvm-cov | Code coverage | + +## See also + +- [ARCH.md](ARCH.md) -- architecture patterns +- [PRE-REQS.md](PRE-REQS.md) -- prerequisites +- [../repo/RECIPES.md](../repo/RECIPES.md) -- Justfile recipes