Shift data aggregation from Ollama pipeline to MCP host (Claude)

.env.example

@@ -8,25 +8,18 @@
 # Get your key at platform.openai.com
 # OPENAI_API_KEY=sk-...
 
-# ── Ollama ──────────────────────────────────────────────────────────
-# Model used by the 3-agent preprocessing pipeline.
-OLLAMA_MODEL=llama3.1:8b
-# Timeout (seconds) for Ollama LLM calls — covers cold-start model loading.
-OLLAMA_TIMEOUT=300
-# Context window size — increase for large pentest outputs.
-OLLAMA_NUM_CTX=8192
-# Keep model in memory between sequential agent calls (prevents repeated cold starts).
-OLLAMA_KEEP_ALIVE=10m
-# Retry count for transient Ollama failures (per agent).
-OLLAMA_RETRIES=2
-
-# ── Ollama MCP Orchestrator ────────────────────────────────────────
-# Timeout (seconds) for HTTP calls from the MCP orchestrator to agent containers.
-# Must exceed OLLAMA_TIMEOUT * (1 + OLLAMA_RETRIES) to prevent premature timeout
-# while agents are still retrying internally.  Default: 1200s (20 min).
-AGENT_TIMEOUT=1200
-# Retry count for failed agent HTTP calls.
-AGENT_RETRIES=2
+# ── Ollama (OPTIONAL — legacy local pipeline) ──────────────────────
+# The MCP host (Claude) now handles data aggregation directly.
+# These settings are only needed if you enable the Ollama pipeline:
+#   docker compose --profile ollama up -d
+#
+# OLLAMA_MODEL=llama3.1:8b
+# OLLAMA_TIMEOUT=300
+# OLLAMA_NUM_CTX=8192
+# OLLAMA_KEEP_ALIVE=10m
+# OLLAMA_RETRIES=2
+# AGENT_TIMEOUT=1200
+# AGENT_RETRIES=2
 
 # ── Kali MCP Server ──────────────────────────────────────────────
 # Port exposed on the host for direct SSE connections (http://localhost:9001/sse).

CLAUDE.md

@@ -16,15 +16,15 @@ Read the following before writing a single line:
 - `CLAUDE.md` (this file), `README.md`
 - `docker-compose.yml`, `Makefile`, `.env.example`
 - `blhackbox/mcp/server.py` — blhackbox stdio MCP server (Claude Code Web path)
-- `mcp_servers/ollama_mcp_server.py` — Ollama MCP orchestrator
+- `mcp_servers/ollama_mcp_server.py` — Ollama MCP orchestrator (optional, `--profile ollama`)
 - Every file directly relevant to the task: the relevant `Dockerfile`, `*_server.py`, `*_agent.py`, agent prompts in `blhackbox/prompts/agents/` — whatever applies
 - Do not rely on memory from previous sessions. Read the actual current files.
 
 **Phase 3: Understand Before Acting**
 Before writing code, answer these internally:
 1. What is the root cause — not the symptom, the actual root cause?
 2. Does the fix conflict with anything else in the codebase?
-3. Does it break the Ollama pipeline contract? (`AggregatedPayload` schema must stay stable across Ingestion → Processing → Synthesis)
+3. Does it break the `AggregatedPayload` schema contract? (Must stay stable for `aggregate_results`, report generation, and the optional Ollama pipeline)
 4. Does it violate the `shell=False` rule?
 5. Am I touching agent prompts in `blhackbox/prompts/agents/`? If so — do I need a rebuild, or can I use a volume mount override?
 6. Is there a simpler fix that achieves the same result?
@@ -36,16 +36,19 @@ Only after answering all six — write the fix.
 ## Project Purpose
 BLHACKBOX is an MCP-based autonomous pentesting framework. The AI client (Claude Code,
 Claude Desktop, or ChatGPT) IS the orchestrator — it decides which tools to call,
-collects raw outputs, and sends them to the Ollama pipeline for preprocessing before
-writing the final pentest report.
+collects raw outputs, and structures them directly into an `AggregatedPayload` via
+the `aggregate_results` MCP tool before writing the final pentest report.
+
+The Ollama preprocessing pipeline (3 agents) is now optional (`--profile ollama`)
+for local-only / offline processing. By default, the MCP host handles aggregation.
 
 ## Code Standards
 - All Python code must be type-annotated
 - All MCP tool inputs must be validated with Pydantic
 - All subprocess calls must use `subprocess.run(args_list, shell=False)`
 - Never use `shell=True` in subprocess calls
 - Never log API keys or secrets
-- `AggregatedPayload` schema (`blhackbox/models/aggregated_payload.py`) is the contract between the pipeline and the AI — do not break it without updating all three agents
+- `AggregatedPayload` schema (`blhackbox/models/aggregated_payload.py`) is the contract between the MCP host and the reporting tools — do not break it without updating all consumers
 
 ## Adding a New MCP Server
 1. Create `new-mcp/` directory with your server code
@@ -57,8 +60,8 @@ writing the final pentest report.
 7. Document tools in README.md components table
 8. Add unit tests
 
-## Adding or Tuning an Agent Prompt
-Agent prompts are in `blhackbox/prompts/agents/`:
+## Adding or Tuning an Agent Prompt (Optional Ollama Pipeline)
+Agent prompts are in `blhackbox/prompts/agents/` (only relevant if using `--profile ollama`):
 - `ingestionagent.md` — Ingestion Agent system prompt
 - `processingagent.md` — Processing Agent system prompt
 - `synthesisagent.md` — Synthesis Agent system prompt

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help up up-full up-gateway down logs test test-local lint format clean nuke \
+.PHONY: help up up-full up-ollama up-gateway down logs test test-local lint format clean nuke \
        pull status health portainer gateway-logs ollama-pull ollama-shell \
        claude-code \
        neo4j-browser logs-ollama-mcp logs-kali \
@@ -18,20 +18,23 @@ help: ## Show this help
 pull: ## Pull all pre-built images from Docker Hub
 	$(COMPOSE) pull
 
-up: ## Start core stack (10 containers — no gateway)
+up: ## Start core stack (4 containers — no Ollama, no gateway)
 	$(COMPOSE) up -d
 
+up-ollama: ## Start with Ollama pipeline (9 containers — legacy local processing)
+	$(COMPOSE) --profile ollama up -d
+
 down: ## Stop all services (all profiles)
-	$(COMPOSE) --profile gateway --profile neo4j --profile claude-code down
+	$(COMPOSE) --profile gateway --profile neo4j --profile claude-code --profile ollama down
 
 logs: ## Tail logs from all services
 	$(COMPOSE) logs -f
 
 # ── Stack variations ─────────────────────────────────────────────
-up-full: ## Start full stack: core + Neo4j (11 containers)
+up-full: ## Start full stack: core + Neo4j (5 containers)
 	$(COMPOSE) --profile neo4j up -d
 
-up-gateway: ## Start core + MCP Gateway for Claude Desktop / ChatGPT (11 containers)
+up-gateway: ## Start core + MCP Gateway for Claude Desktop / ChatGPT (5 containers)
 	$(COMPOSE) --profile gateway up -d
 
 # ── Testing & Code Quality ─────────────────────────────────────
@@ -48,15 +51,15 @@ format: ## Auto-format code
 	ruff format blhackbox/ tests/
 
 clean: ## Remove containers, volumes, networks, and build artifacts (keeps images)
-	$(COMPOSE) --profile gateway --profile neo4j --profile claude-code down -v --remove-orphans
+	$(COMPOSE) --profile gateway --profile neo4j --profile claude-code --profile ollama down -v --remove-orphans
 	find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
 	rm -rf dist/ build/ *.egg-info
 
 nuke: ## Full cleanup: containers + volumes + ALL images (frees max disk space)
 	@echo "\033[1;33m  WARNING: This will remove ALL blhackbox containers, volumes, AND images.\033[0m"
 	@echo "\033[2m  You will need to 'docker compose pull' or 'docker compose build' again.\033[0m"
 	@echo ""
-	$(COMPOSE) --profile gateway --profile neo4j --profile claude-code down -v --remove-orphans --rmi all
+	$(COMPOSE) --profile gateway --profile neo4j --profile claude-code --profile ollama down -v --remove-orphans --rmi all
 	@echo ""
 	@echo "\033[2m  Pruning dangling images and build cache...\033[0m"
 	docker image prune -f
@@ -91,7 +94,7 @@ status: ## Health status of all containers
 	@echo ""
 	@echo "\033[1m  blhackbox Container Status\033[0m"
 	@echo "\033[2m  ──────────────────────────────────────\033[0m"
-	@$(COMPOSE) --profile gateway --profile neo4j --profile claude-code ps --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || $(COMPOSE) ps
+	@$(COMPOSE) --profile gateway --profile neo4j --profile claude-code --profile ollama ps --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || $(COMPOSE) ps
 	@echo ""
 
 health: ## Quick health check of all MCP servers
@@ -108,20 +111,20 @@ health: ## Quick health check of all MCP servers
 		docker exec blhackbox-screenshot-mcp python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9004/health')" > /dev/null 2>&1 \
 		&& echo "\033[32m[OK]\033[0m" || echo "\033[31m[FAIL]\033[0m"
 	@printf "  %-22s " "Ollama MCP (9000)"; \
-		docker exec blhackbox-ollama-mcp python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9000/sse')" > /dev/null 2>&1 \
-		&& echo "\033[32m[OK]\033[0m" || echo "\033[31m[FAIL]\033[0m"
+		docker inspect --format='{{.State.Running}}' blhackbox-ollama-mcp 2>/dev/null | grep -q "true" \
+		&& echo "\033[32m[OK]\033[0m" || echo "\033[33m[OFF]\033[0m  (optional — enable with: make up-ollama)"
 	@printf "  %-22s " "Ollama (11434)"; \
-		docker exec blhackbox-ollama ollama list > /dev/null 2>&1 \
-		&& echo "\033[32m[OK]\033[0m" || echo "\033[31m[FAIL]\033[0m"
+		docker inspect --format='{{.State.Running}}' blhackbox-ollama 2>/dev/null | grep -q "true" \
+		&& echo "\033[32m[OK]\033[0m" || echo "\033[33m[OFF]\033[0m  (optional — enable with: make up-ollama)"
 	@printf "  %-22s " "Agent Ingestion"; \
-		docker exec blhackbox-agent-ingestion python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8001/health')" > /dev/null 2>&1 \
-		&& echo "\033[32m[OK]\033[0m" || echo "\033[31m[FAIL]\033[0m"
+		docker inspect --format='{{.State.Running}}' blhackbox-agent-ingestion 2>/dev/null | grep -q "true" \
+		&& echo "\033[32m[OK]\033[0m" || echo "\033[33m[OFF]\033[0m  (optional)"
 	@printf "  %-22s " "Agent Processing"; \
-		docker exec blhackbox-agent-processing python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8002/health')" > /dev/null 2>&1 \
-		&& echo "\033[32m[OK]\033[0m" || echo "\033[31m[FAIL]\033[0m"
+		docker inspect --format='{{.State.Running}}' blhackbox-agent-processing 2>/dev/null | grep -q "true" \
+		&& echo "\033[32m[OK]\033[0m" || echo "\033[33m[OFF]\033[0m  (optional)"
 	@printf "  %-22s " "Agent Synthesis"; \
-		docker exec blhackbox-agent-synthesis python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:8003/health')" > /dev/null 2>&1 \
-		&& echo "\033[32m[OK]\033[0m" || echo "\033[31m[FAIL]\033[0m"
+		docker inspect --format='{{.State.Running}}' blhackbox-agent-synthesis 2>/dev/null | grep -q "true" \
+		&& echo "\033[32m[OK]\033[0m" || echo "\033[33m[OFF]\033[0m  (optional)"
 	@printf "  %-22s " "MCP Gateway (8080)"; \
 		docker inspect --format='{{.State.Running}}' blhackbox-mcp-gateway 2>/dev/null | grep -q "true" \
 		&& echo "\033[32m[OK]\033[0m" || echo "\033[33m[OFF]\033[0m  (optional — enable with: make up-gateway)"