[Agentic Phase 3] Internal context agent for orchestrated search/response

[bashandbone]

Phase 3: Context Agent Tooling

Parent Epic: #123
Depends On: #124 (Agent Integration)
Target: v0.3
Risk Level: Medium

Build internal "context agent" system for orchestrated multi-step reasoning over code context, enabling complex analysis workflows.

Goals

• Multi-step reasoning over code
• Intelligent result synthesis
• Context-aware decision making
• Transparent reasoning traces

Vision

A context agent that can:

• Break down complex queries into sub-tasks
• Execute multi-step analysis workflows
• Maintain state across reasoning steps
• Synthesize comprehensive answers
• Explain its reasoning process

Implementation Checklist

Context Agent Architecture

• Define context agent interface
• Implement conversation/state management
• Build reasoning trace system
• Create step orchestration logic

Core Capabilities

• Multi-file analysis
  • Dependency tracking
  • Cross-reference resolution
  • Impact analysis
• Iterative refinement
  • Query decomposition
  • Progressive search
  • Result aggregation
• Reasoning explanations
  • Step-by-step traces
  • Confidence scoring
  • Alternative paths considered

Tool Integration

• Search tools (from [Agentic Phase 1] Integrate agents into find_code pipeline #124)
• Data provider tools (from [Agentic Phase 2] Integrate data providers into pipeline #125)
• Code analysis tools
  • AST navigation
  • Symbol resolution
  • Pattern matching
• Documentation tools
  • Inline comments
  • Docstrings
  • README files

Orchestration Patterns

• Sequential reasoning (step 1 → 2 → 3)
• Parallel exploration (multiple hypotheses)
• Iterative refinement (search → analyze → refine)
• Conditional branching (if-then reasoning)

Testing

• Unit tests for agent components
• Integration tests for workflows
• Reasoning quality evaluation
• Performance benchmarks
• Edge case handling

Configuration

• Agent personality/behavior settings
• Reasoning depth limits
• Timeout configuration
• Cost controls
• Debug/trace mode

Success Criteria

• Context agent handles complex queries
• Multi-step reasoning produces correct results
• Reasoning traces are comprehensible
• Performance acceptable (< 10s for complex queries)
• Tests passing
• Documentation complete

Example Workflows

1. Architecture analysis: "How does authentication flow work?"

   • Step 1: Find entry points (login endpoints)
   • Step 2: Trace authentication logic
   • Step 3: Identify token handling
   • Step 4: Map session management
   • Synthesis: Complete flow diagram

2. Refactoring impact: "What would break if I change this function signature?"

   • Step 1: Find all call sites
   • Step 2: Analyze argument usage
   • Step 3: Check type compatibility
   • Step 4: Identify tests
   • Synthesis: Impact report

3. Bug investigation: "Why is this feature failing?"

   • Step 1: Locate feature code
   • Step 2: Find related tests
   • Step 3: Check recent changes
   • Step 4: Analyze error patterns
   • Synthesis: Diagnosis and fix suggestions

Reference

• Related: [Agentic Phase 3] Internal context agent for orchestrated search/response #126 (pydantic-graph will structure these workflows)
• Related: [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127 (pydantic-eval will measure quality)

[github-actions]

Claude finished @bashandbone's task —— View job

---

Analysis: Issue #126 - Context Agent Tooling (Phase 3)

I've completed a comprehensive analysis of this Phase 3 issue for implementing internal context agent capabilities. Here's my assessment:

---

Summary

This issue proposes building an internal "context agent" system to orchestrate multi-step reasoning over code context. This is a sophisticated enhancement that would enable CodeWeaver to handle complex analytical queries through decomposition, progressive search, and intelligent synthesis.

Key Dependencies:

• Issue [EPIC] Agentic enhancements: intelligent search, orchestration, and evaluation #123: Parent Epic for Agentic Enhancements
• Issue [Agentic Phase 1] Integrate agents into find_code pipeline #124: Phase 1 - Agent Integration (must be completed first)
• Issue [Agentic Phase 2] Integrate data providers into pipeline #125: Phase 2 - Data Provider Integration
• Issue [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127: Phase 4 - pydantic-graph orchestration (referenced for workflow structuring)

---

Related Issues & Context

I found the following related issues in the project:

1. [EPIC] Agentic enhancements: intelligent search, orchestration, and evaluation #123 (OPEN): [EPIC] Agentic enhancements - Parent epic covering the full agentic vision
2. [Agentic Phase 1] Integrate agents into find_code pipeline #124 (OPEN): [Phase 1] Integrate agents into find_code pipeline - Foundation work
3. [Agentic Phase 2] Integrate data providers into pipeline #125 (OPEN): [Phase 2] Integrate data providers into pipeline - Data source expansion
4. [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127 (OPEN): [Phase 4] Pipeline orchestration with pydantic-graph - Graph-based workflows

Labels Applied: enhancement, pipeline, orchestration, pydantic-ai, agents

---

Current Codebase State

✅ Strong Foundation Already In Place

The codebase has substantial infrastructure that will support context agent implementation:

1. Agent Provider Integration (src/codeweaver/providers/agent/):

   • Comprehensive pydantic-ai provider re-exports covering 20+ agent providers
   • Support for OpenAI, Anthropic, Google, Bedrock, Groq, and many others
   • Toolset abstractions ready for agent workflows
   • Model settings and inference capabilities

2. Semantic Classification System (src/codeweaver/semantic/):

   • Multi-dimensional importance scoring for code segments
   • Task-aware context ranking (discovery, comprehension, modification, debugging, documentation)
   • AST-based semantic metadata tracking
   • 26 languages with full AST support + 170+ with heuristic fallbacks

3. Span-Based Architecture (src/codeweaver/core/spans.py):

   • Precise code location tracking with line/column accuracy
   • Set operations (union, intersection, difference) for span composition
   • Thread-safe immutable operations
   • Critical for multi-step reasoning workflows

4. Existing Agent API (src/codeweaver/agent_api/):

   • find_code tool interface defined (currently stubbed during refactoring)
   • Intent classification system for task-aware searches
   • Response models with execution metadata

⚠️ Current Gaps

1. No Context Agent Implementation Yet: No existing agent orchestration or multi-step reasoning
2. find_code Tool Incomplete: Currently returning stub responses during refactoring (src/codeweaver/agent_api/find_code.py:48-68)
3. pydantic-graph Not Integrated: Dependencies declared but no workflow orchestration implemented
4. No Reasoning Trace System: No infrastructure for recording agent decision paths
5. Limited Tool Ecosystem: Context agent will need specialized tools beyond find_code

---

Implementation Analysis

Architectural Approach

The issue proposes a context agent that can:

• Decompose complex queries into sub-tasks
• Execute multi-step analysis workflows
• Maintain state across reasoning steps
• Synthesize comprehensive answers
• Provide transparent reasoning traces

Constitutional Compliance Check:

✅ AI-First Context (Principle I): Directly aligned - enhances AI agent understanding through orchestrated context delivery

✅ Proven Patterns (Principle II): Leverages pydantic-ai ecosystem and established agent patterns

⚠️ Evidence-Based Development (Principle III): Implementation must avoid placeholders and be backed by tests

✅ Testing Philosophy (Principle IV): Focus on realistic integration scenarios over unit tests

✅ Simplicity Through Architecture (Principle V): Clear orchestration patterns, avoid unnecessary complexity

Core Components Needed

1. Context Agent Interface

   • Agent initialization with model configuration
   • Conversation/state management across reasoning steps
   • Tool binding for specialized context operations
   • Graceful error handling and fallbacks

2. Reasoning Trace System

   • Step-by-step decision recording
   • Confidence scoring for each step
   • Alternative paths considered
   • Transparent explanations for users

3. Multi-Step Orchestration

   • Sequential reasoning (step 1 → 2 → 3)
   • Parallel exploration (multiple hypotheses)
   • Iterative refinement (search → analyze → refine)
   • Conditional branching (if-then logic)

4. Tool Integration

   • Search Tools: find_code, semantic search, keyword search
   • Code Analysis: AST navigation, symbol resolution, pattern matching
   • Dependency Tracking: Import trees, call graphs, cross-references
   • Documentation Tools: Inline comments, docstrings, README files
   • Data Provider Tools: External sources (from [Agentic Phase 2] Integrate data providers into pipeline #125)

5. State Management

   • Conversation history persistence
   • Intermediate result caching
   • Context budget tracking (token limits)
   • Partial result recovery on failure

Integration with pydantic-graph (#127)

Per the issue, pydantic-graph will structure these workflows. The codebase already includes pydantic-graph>=1.4.0 as a dependency. Key integration points:

• Graph Nodes: Individual reasoning steps (search, analyze, synthesize)
• Edges: Data flow between steps with validation
• Conditional Logic: Branch based on intermediate results
• Parallel Execution: Explore multiple hypotheses simultaneously
• Error Recovery: Graceful degradation when steps fail

---

Example Workflows Analysis

The issue provides three compelling examples:

1. Architecture Analysis: "How does authentication flow work?"

• Step 1: Find entry points → Use find_code with intent="implementation"
• Step 2: Trace authentication logic → Follow function calls and imports
• Step 3: Identify token handling → Pattern matching for JWT/session code
• Step 4: Map session management → Find state management patterns
• Synthesis: Generate flow diagram with spans and explanations

2. Refactoring Impact: "What would break if I change this function signature?"

• Step 1: Find all call sites → Use AST grep for function references
• Step 2: Analyze argument usage → Check parameter types and usage patterns
• Step 3: Check type compatibility → Verify against proposed signature
• Step 4: Identify tests → Find test files covering this function
• Synthesis: Generate impact report with affected files and confidence scores

3. Bug Investigation: "Why is this feature failing?"

• Step 1: Locate feature code → find_code with intent="debugging"
• Step 2: Find related tests → Semantic search for test coverage
• Step 3: Check recent changes → Git integration for change analysis
• Step 4: Analyze error patterns → Pattern matching across logs
• Synthesis: Diagnosis with fix suggestions and confidence levels

---

Suggested Labels

• ✅ enhancement (already applied)
• ✅ pipeline (already applied)
• ✅ orchestration (already applied)
• ✅ pydantic-ai (already applied)
• ✅ agents (already applied)
• 💡 Consider: complex, phase-3, v0.3

---

Implementation Difficulty & Impact

Difficulty: High (7-8 weeks of focused development)

Why High:

1. Multi-step orchestration with state management is complex
2. Reasoning trace system needs careful design for transparency
3. Integration with pydantic-graph requires understanding graph semantics
4. Tool ecosystem must be extended significantly
5. Robust error handling across async workflows is challenging
6. Testing multi-step reasoning requires sophisticated test scenarios

Complexity Breakdown:

• Context Agent Core: 2-3 weeks (agent initialization, state management, tool binding)
• Reasoning Trace System: 1-2 weeks (recording, scoring, explanation generation)
• Orchestration Patterns: 2-3 weeks (sequential, parallel, iterative, conditional)
• Tool Integration: 1-2 weeks (extending tool ecosystem, binding to agent)
• Testing & Documentation: 1-2 weeks (comprehensive test scenarios, user guides)

Impact: Very High

Positive Impact:

• 🎯 Transforms CodeWeaver from a search tool to an intelligent context platform
• 🧠 Enables Complex Queries that current keyword/semantic search cannot handle
• 📊 Transparent Reasoning builds user trust through explainable decisions
• 🔗 Foundation for Future Features like automated refactoring suggestions
• 💡 Competitive Differentiation - few MCP servers offer orchestrated reasoning

Risks:

• ⚠️ Complexity Creep: Multi-step orchestration can become hard to debug
• ⚠️ Performance: Async workflows with multiple LLM calls may be slow
• ⚠️ Cost: Agent-driven context curation will increase API costs
• ⚠️ Dependencies: Requires [Agentic Phase 1] Integrate agents into find_code pipeline #124 and [Agentic Phase 2] Integrate data providers into pipeline #125 to be completed first
• ⚠️ Evidence-Based Development: Must avoid placeholder implementations (Constitutional Principle III)

---

Implementation Recommendations

Phase 3 Execution Plan

Prerequisites:

1. ✅ Complete Issue [Agentic Phase 1] Integrate agents into find_code pipeline #124 (Agent Integration) - Foundation required
2. ✅ Complete Issue [Agentic Phase 2] Integrate data providers into pipeline #125 (Data Provider Integration) - Expands tool capabilities
3. ✅ Restore find_code functionality from stub state

Implementation Sequence:

Week 1-2: Context Agent Foundation

• Design and implement context agent interface using pydantic-ai
• Build conversation/state management system
• Create basic reasoning trace infrastructure
• Implement simple sequential orchestration (proof of concept)

Week 3-4: Tool Ecosystem Expansion

• Implement specialized context tools (get_semantic_neighbors, get_call_graph, etc.)
• Integrate with existing semantic classification system
• Add AST navigation and symbol resolution tools
• Create documentation extraction tools

Week 5-6: Advanced Orchestration

• Integrate pydantic-graph for workflow structuring
• Implement parallel exploration pattern
• Add iterative refinement capabilities
• Build conditional branching logic

Week 7-8: Polish & Testing

• Comprehensive integration testing with example workflows
• Performance optimization (caching, batching)
• Reasoning quality evaluation (consider pydantic-eval from [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127)
• Documentation and usage examples

Success Metrics

Per the issue's success criteria, measure:

• ✅ Complex query handling (architecture analysis, impact assessment, bug investigation)
• ✅ Multi-step reasoning correctness (validated against ground truth)
• ✅ Reasoning trace comprehensibility (user feedback)
• ✅ Performance < 10s for complex queries (95th percentile)
• ✅ Test coverage for critical paths (integration > unit)
• ✅ Documentation completeness (API docs, examples, guides)

---

Potential Implementation Challenges

1. State Management Complexity: Maintaining context across async agent calls while respecting token budgets

   • Mitigation: Use pydantic models for strict state validation, implement context pruning

2. Error Propagation: Handling failures gracefully in multi-step workflows

   • Mitigation: Design fallback strategies at each orchestration level, record partial results

3. Performance Bottlenecks: Multiple LLM API calls in sequence may be slow

   • Mitigation: Parallel execution where possible, aggressive caching, streaming responses

4. Testing Complexity: Validating multi-step reasoning requires sophisticated test scenarios

   • Mitigation: Record/replay pattern for deterministic tests, integration focus over mocks

5. Constitutional Compliance: Avoiding placeholder implementations while building incrementally

   • Mitigation: Each component must be fully functional before moving to next, document limitations clearly

---

Related Code References

Key files to review during implementation:

• Agent Providers: src/codeweaver/providers/agent/agent_providers.py - pydantic-ai integration
• Agent API: src/codeweaver/agent_api/find_code.py - Tool interface (needs restoration)
• Semantic System: src/codeweaver/semantic/classifications.py - Task-aware importance scoring
• Spans: src/codeweaver/core/spans.py - Precise location tracking
• Registry: src/codeweaver/common/registry.py - Provider and tool registration
• Implementation Plan: plans/IMPLEMENTATION_PLAN.md - Phase 3 architecture details

---

Conclusion

This is an ambitious and well-thought-out feature that will significantly enhance CodeWeaver's capabilities. The issue provides clear goals, comprehensive checklists, and concrete examples. The codebase has strong foundations that will support this work, but the implementation is non-trivial and requires careful attention to the constitutional principles.

Key Blockers:

• Issue [Agentic Phase 1] Integrate agents into find_code pipeline #124 must be completed first
• Issue [Agentic Phase 2] Integrate data providers into pipeline #125 should be completed for expanded tool capabilities
• find_code tool must be restored from stub state

Recommendation: This is a Phase 3 feature that should not be started until Phase 1 and Phase 2 are complete. The 7-8 week timeline is realistic for high-quality implementation that respects evidence-based development principles. Consider breaking this into smaller sub-issues for more manageable incremental delivery.

Next Steps:

1. Complete blockers ([Agentic Phase 1] Integrate agents into find_code pipeline #124, [Agentic Phase 2] Integrate data providers into pipeline #125)
2. Design context agent interface (collaborate with team)
3. Prototype simple sequential orchestration
4. Validate approach before full implementation

---

| Branch: claude/issue-126-20251119-2123