Shift data aggregation from Ollama pipeline to MCP host (Claude)

[valITino]

Summary

This PR fundamentally restructures how blhackbox processes pentest data. Instead of sending raw tool outputs to a 3-agent Ollama preprocessing pipeline (Ingestion → Processing → Synthesis), the MCP host (Claude Code, Claude Desktop, or ChatGPT) now handles parsing, deduplication, correlation, and structuring directly. The Ollama pipeline is moved to an optional legacy fallback (--profile ollama).

Key Changes

Core Architecture

• Reduced default stack from 9 containers to 4 (kali-mcp, wire-mcp, screenshot-mcp, portainer)
• Ollama pipeline (ollama-mcp, agent-ingestion, agent-processing, agent-synthesis, ollama) now optional via --profile ollama
• Reduced RAM requirement from 16 GB to 8 GB for core stack; 16 GB+ only needed if using Ollama pipeline

MCP Server Changes (blhackbox/mcp/server.py)

• Added two new tools:
  • aggregate_results(payload) — validates and persists the AggregatedPayload JSON, stores to disk, optionally syncs to Neo4j
  • get_payload_schema() — returns the AggregatedPayload JSON schema so Claude knows the expected structure
• Updated get_template description to reflect that Claude handles aggregation directly
• Removed references to process_scan_results() (Ollama pipeline method)

Data Model Updates (blhackbox/models/aggregated_payload.py)

• Updated docstring: AggregatedPayload is now produced by the MCP host, not the Ollama pipeline
• Added model field to AggregatedMetadata to track which model performed aggregation (Claude, Ollama, etc.)
• Kept ollama_model field for backward compatibility but marked as deprecated

Workflow Documentation

• Updated claude_playbook.md: Phase 4 renamed from "Process" to "Aggregate"; Claude now does the work directly
• Updated all prompt templates (full-pentest, api-security, bug-bounty, etc.): removed references to process_scan_results(), replaced with get_payload_schema() + aggregate_results() pattern
• Updated README.md: clarified that core stack is 4 containers, Ollama is optional, reduced RAM requirements

Docker & Deployment

• docker-compose.yml: Ollama services moved to --profile ollama; MCP Gateway now only depends on core 3 MCP servers (removed ollama-mcp dependency)
• Makefile: added up-ollama target; updated container counts in help text; updated down and clean to include --profile ollama
• .env.example: commented out all Ollama-related settings; noted they're only needed for --profile ollama
• blhackbox-mcp-catalog.yaml: commented out ollama-mcp registry entry (can be uncommented if using legacy pipeline)
• docker/claude-code-entrypoint.sh: Ollama Pipeline check now warns instead of failing if not running

Entrypoint & Startup

• Claude Code startup script now shows Ollama Pipeline as optional (WARN status if not running, rather than failure)
• Health check output updated to show 3 core services instead of 4

Implementation Details

• aggregate_results() validates the payload against the Pydantic schema, persists to results_dir/session-{session_id}.json, and attempts Neo4j storage (best-effort, doesn't fail if Neo4j unavailable)
• The tool returns a summary with host/vulnerability/endpoint counts and a hint for report generation
• get_payload_schema() returns the full JSON schema from AggregatedPayload.model_json_schema() for Claude to reference
• All prompt templates now follow the same pattern: collect raw outputs → call get_payload_schema() → parse/deduplicate/correlate → call aggregate_results() → call generate_report()
• Backward compatibility maintained: Ollama pipeline still

https://claude.ai/code/session_01MXWTGUUSheo3EkgHrzRmjy