Add support for Mistral batch jobs API

[bashandbone]

Description

Add support for Mistral batch jobs API for more efficient and cost-effective embedding generation.

Current State

• File: mistral.py:33
• Only real-time API supported
• No batch job support

Benefits of Batch Jobs

• Cost savings: Batch pricing typically lower
• Efficiency: Process large volumes efficiently
• Rate limiting: Avoid rate limit issues
• Resource optimization: Better utilization

Challenges

• State management: Track batch job status
• Async nature: Jobs complete asynchronously
• Error handling: Batch failures need special handling
• Polling: Need to check job completion

Use Cases

Ideal for:

• Initial indexing of large codebases
• Periodic re-indexing
• Bulk updates
• Non-time-critical operations

Less ideal for:

• Real-time search
• Interactive operations
• Time-sensitive updates

Requirements

• Implement batch job submission
• Add job status polling
• Handle batch results
• Graceful fallback to real-time API
• Configuration for batch vs real-time

Source

• File: mistral.py:33
• Branch: 003-our-aim-to

[github-actions]

Claude finished @bashandbone's task —— View job

---

Analysis: Mistral Batch Jobs API Support

I've completed a comprehensive review of this feature request. Here's my analysis:

---

Summary

This issue proposes adding support for Mistral's batch jobs API to complement the existing real-time embedding API in src/codeweaver/providers/embedding/providers/mistral.py:31. The batch API offers significant benefits for non-time-critical operations like initial codebase indexing and periodic re-indexing.

---

Current Implementation

File: src/codeweaver/providers/embedding/providers/mistral.py:31-33

The current implementation:

• Uses real-time mistral.embeddings.create_async() calls
• Processes embeddings synchronously with immediate results
• Already has a TODO comment acknowledging this limitation
• Follows the standard EmbeddingProvider base class pattern

Architecture Alignment: The implementation properly follows CodeWeaver's provider pattern:

• Extends EmbeddingProvider[Mistral] base class
• Uses EmbeddingModelCapabilities for configuration
• Implements _embed_documents() and _embed_query() abstract methods
• Handles token counting and error management

---

Analysis & Recommendations

1. Constitutional Compliance Check

This feature aligns with CodeWeaver's constitutional principles:

• ✅ Evidence-Based Development: Batch APIs demonstrably reduce costs and improve efficiency for bulk operations
• ✅ Proven Patterns: Batch processing is a well-established pattern (OpenAI, Anthropic, others use similar approaches)
• ✅ Simplicity Through Architecture: Implementation can be additive without breaking existing interfaces
• ⚠️ AI-First Context: Must ensure batch mode doesn't degrade context delivery quality or timeliness

2. Implementation Complexity: HIGH

New Components Needed:

1. State Management Infrastructure

   • Job submission tracking (job IDs, submission timestamps)
   • Status polling mechanism with exponential backoff
   • Result retrieval and caching
   • Persistent storage for long-running jobs (possibly filesystem or database)

2. Async Job Orchestration

   • Background task runner for polling job status
   • Queue management for multiple concurrent batch jobs
   • Timeout and retry logic for failed jobs
   • Graceful degradation to real-time API on failure

3. Configuration Layer

   • User-facing settings for batch vs. real-time selection
   • Thresholds for automatic batch mode (e.g., > 1000 documents)
   • Polling interval configuration
   • Timeout settings for batch completion

4. API Integration

   • Mistral batch job submission endpoint integration
   • Status checking endpoint integration
   • Result retrieval endpoint integration
   • Error handling for batch-specific failures

Architectural Challenges:

• State Persistence: Current EmbeddingProvider is stateless; batch jobs require state management across sessions
• Async Nature: Batch jobs complete asynchronously, requiring either:
  • Blocking wait with polling (simpler but blocks execution)
  • Background task execution with callback/webhook (complex but non-blocking)
  • Hybrid approach: short jobs poll, long jobs use background tasks
• Provider Registry: May need updates to support batch-capable providers differently
• Testing Complexity: Integration tests require mocking async job lifecycle

3. Impact Assessment

Benefits:

• Cost Savings: Batch pricing typically 50-70% lower than real-time
• Efficiency: Better throughput for bulk operations (initial indexing, re-indexing)
• Rate Limiting: Avoids rate limit issues during large batch operations
• Resource Optimization: More efficient API utilization

Risks & Trade-offs:

• Latency: Batch jobs introduce significant delay (minutes to hours vs. seconds)
• Complexity: Adds substantial state management overhead
• Maintenance: More moving parts to test, debug, and maintain
• User Experience: Requires clear communication about job status and progress

Affected Components:

• MistralEmbeddingProvider (direct changes)
• EmbeddingProvider base class (possible interface extensions)
• Configuration system (new batch-related settings)
• Potentially: chunking service, indexing pipelines
• Testing infrastructure (new integration test requirements)

4. Use Case Validation

Ideal For (as stated in issue):

• ✅ Initial indexing of large codebases (10,000+ files)
• ✅ Periodic full re-indexing (nightly/weekly jobs)
• ✅ Bulk updates after major refactors
• ✅ Non-time-critical operations

Not Ideal For:

• ❌ Real-time search queries
• ❌ Interactive developer workflows
• ❌ Time-sensitive context retrieval
• ❌ Small document batches (< 100 documents)

Recommendation: Implement automatic mode selection based on:

• Document count thresholds (e.g., > 1000 documents → batch)
• User explicit preference via configuration
• Operation type (indexing vs. query)

---

Implementation Approach Recommendations

Option 1: Minimal Viable Implementation (Recommended for Phase 1)

Scope: Add batch support with blocking wait and basic polling

Pros:

• Simpler implementation (2-3 week effort)
• Leverages existing provider patterns
• Addresses 80% of use cases (initial indexing)
• Lower risk of bugs

Cons:

• Blocks execution during batch processing
• Less efficient for very large jobs
• Manual configuration required

Option 2: Full-Featured Implementation

Scope: Complete async job orchestration with background tasks

Pros:

• Non-blocking execution
• Better user experience
• Handles all use cases optimally
• Production-ready for enterprise

Cons:

• Significantly more complex (6-8 week effort)
• Requires persistent state storage
• More testing and edge cases
• Higher maintenance burden

Option 3: Hybrid Approach (Recommended for Phase 2)

Scope: Start with Option 1, evolve to Option 2 based on user feedback

Rationale:

• Aligns with constitutional principle: "Evidence-Based Development"
• Validates demand before major investment
• Allows iterative refinement
• Reduces technical debt risk

---

Related Issues & Context

Search Results:

• Issue Expand vector provider support and fix capabilities definition #104: "Expand vector provider support and fix capabilities definition" - Related provider expansion work
• No existing batch job infrastructure found in codebase
• This would be the first batch-capable provider in CodeWeaver

Similar Patterns:

• OpenAI provider (openai_factory.py) could benefit from similar batch support
• Pattern could be abstracted to BatchCapableEmbeddingProvider base class for reuse

---

Label Recommendations

Currently Applied (✅):

• enhancement - Correctly categorizes this as a new feature
• performance - Batch processing improves performance for bulk operations
• embeddings - Directly relates to embedding provider functionality

Suggested Additions:

• architecture - Requires significant architectural design work
• providers - Affects the provider system
• help-wanted - Community could contribute after design is established
• research-needed - Requires investigation of Mistral's batch API capabilities

Not Recommended:

• good-first-issue - Too complex for newcomers
• high-priority - Should be decided by maintainers based on roadmap

---

Recommendations

Immediate Actions:

1. Design Phase: Create an architecture design document for batch job support

   • State management approach
   • API interface design
   • Configuration schema
   • Error handling strategy

2. Spike Task: Investigate Mistral's batch API capabilities and limitations

   • Job size limits
   • Polling frequency recommendations
   • Result expiration policies
   • Pricing structure

3. Discussion: Engage with maintainers/community on approach (Option 1 vs. 2 vs. 3)

Implementation Priority: Medium-High

Justification:

• High Value: Significant cost savings for bulk operations
• Clear Use Case: Initial indexing is a common pain point
• Moderate Risk: Well-understood pattern, but complex implementation
• Strategic Fit: Aligns with CodeWeaver's focus on efficiency and cost optimization

Estimated Effort:

• Option 1 (Minimal): 2-3 weeks (1 developer)
• Option 2 (Full): 6-8 weeks (1 developer)
• Option 3 (Hybrid): 2-3 weeks Phase 1, 4-5 weeks Phase 2

---

Next Steps

If this feature is prioritized, I recommend:

1. Create design document in plans/ directory following the pattern in ARCHITECTURE.md

2. Add architecture decision record (ADR) documenting the batch approach choice

3. Create sub-issues for:

   • Batch job state management infrastructure
   • Mistral batch API integration
   • Configuration layer additions
   • Testing infrastructure
   • Documentation updates

4. Consider broader impact: Should other providers (OpenAI, Cohere, etc.) support batch modes?

---

Branch: claude/issue-107-20251119-2052

[lustsazeus-lab]

This seems very feasible as a two-path design:

1. keep current realtime path for interactive calls
2. add a batch backend for indexing/reindexing with job state persisted (submitted/running/done/failed)

A practical first cut is to abstract "embedding provider" behind one interface so batch and realtime share result parsing + retry policy.