Feature/litellm prompt caching implementation

[esafwan]

PR: Implement LiteLLM Prompt Caching

Overview

This PR integrates LiteLLM's prompt caching capabilities into HUF. This feature allows agents to cache and reuse prompt prefixes (system instructions, tool definitions, conversation history), significantly reducing inference costs and latency for supported models (e.g., Anthropic Claude 3.5 Sonnet, GPT-4o).

Changes

1. Data Model & Validation (feat(core))

• Agent DocType:
  • Added enable_prompt_caching field.
  • Added validation logic in Agent.validate() to check if the selected provider/model supports prompt caching using litellm.supports_prompt_caching(). warning is shown if unsupported.
• Agent Run DocType:
  • Added cached_tokens (Int) field to track token savings per run.

2. AI Provider Logic (feat(ai))

• Agent Integration (agent_integration.py):
  • Propagates the enable_prompt_caching setting from the Agent document to the execution context.
• LiteLLM Provider (litellm.py):
  • Updated run() and run_stream() to pass cached_messages=True to litellm.completion.
  • Implemented logic to extract prompt_tokens_details.cached_tokens from the LLM response usage data.
  • Correctly maps cached tokens to the Agent Run document.

3. Documentation (docs)

• Added PROMPT_CACHING_IMPLEMENTATION.md: A comprehensive guide covering:
  • Architecture explanation.
  • Supported providers (Anthropic, OpenAI, DeepSeek, etc.).
  • Configuration steps.
  • Verification and troubleshooting.

Benefits

• Cost Efficiency: Cached tokens are typically ~90% cheaper (e.g., Anthropic charges 10% of the input price for cached reads).
• Performance: Time-to-First-Token (TTFT) is reduced for requests hitting the cache.
• Visibility: Users can now see exactly how many tokens were cached in the Agent Run logs.

How to Test

Prerequisites

• Ensure litellm is updated to a version supporting prompt caching (e.g., >=1.40.0).
• Run bench migrate to apply schema changes.

Steps

1. Configure Agent:
   • Open an Agent document.
   • Select a model that supports caching (e.g., anthropic/claude-3-5-sonnet-20240620).
   • Check the new "Enable Prompt Caching" box.
   • Save. (Verify no warnings appear).
2. Run 1 (Cache Creation):
   • Send a message to the agent.
   • Check the resulting Agent Run.
   • Cached Tokens should be 0 (or greater if using a shared system prompt that was already cached globally).
3. Run 2 (Cache Read):
   • Send another message to the same agent (or same agent config).
   • Check the Agent Run.
   • Cached Tokens should be > 0.
4. Validation Check:
   • Switch model to one that doesn't support caching (e.g., openai/gpt-3.5-turbo older versions).
   • Try to save. You should see a warning.

Commits

• feat(core): Add prompt caching schema & validation
• feat(ai): Implement prompt caching execution logic
• docs: Add Prompt Caching implementation guide