[Bug]: LLMExtractionStrategy not applied when CrawlerRunConfig.cache_mode=ENABLED

[fraank]

crawl4ai version

0.7.4

Expected Behavior

When using cache_mode=CacheMode.ENABLED on CrawlerRunConfig and a previously crawled and cached URL the result.extracted_content is filled with freshly generated output of LLM.

Current Behavior

When using cache_mode=CacheMode.ENABLED on CrawlerRunConfig and a previously crawled and cached URL the result.extracted_content is empty.

Is this reproducible?

Yes

Code snippets

import os
import asyncio
import json
from pydantic import BaseModel, Field
from typing import List
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy

class Product(BaseModel):
    name: str
    price: str

async def main():
    # 1. Define the LLM extraction strategy
    llm_strategy = LLMExtractionStrategy(
        llm_config = LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv('OPENAI_API_KEY')),
        schema=Product.schema_json(), # Or use model_json_schema()
        extraction_type="schema",
        instruction="Extract all product objects with 'name' and 'price' from the content.",
        chunk_token_threshold=1000,
        overlap_rate=0.0,
        apply_chunking=True,
        input_format="markdown",   # or "html", "fit_markdown"
        extra_args={"temperature": 0.0, "max_tokens": 800}
    )

    # 2. Build the crawler config
    crawl_config = CrawlerRunConfig(
        extraction_strategy=llm_strategy,
        cache_mode=CacheMode.ENABLED # (!!) BYPASS is working
    )

    # 3. Create a browser config if needed
    browser_cfg = BrowserConfig(headless=True)

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        # 4. Let's say we want to crawl a single page
        result = await crawler.arun(
            url="https://example.com/products",
            config=crawl_config
        )

        if result.success:
            # 5. The extracted content is presumably JSON
            data = json.loads(result.extracted_content)
            print("Extracted items:", data)

            # 6. Show usage stats
            llm_strategy.show_usage()  # prints token usage
        else:
            print("Error:", result.error_message)

The Code Example is taken from the LLM Strategies Documentation

OS

macOS

Python version

3.11.4

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[SohamKukreti]

Hey, thanks for the raising this issue. I managed to reproduce it on crawl4ai v0.7.4.

Root cause Analysis

When cache_mode=CacheMode.ENABLED and a cache hit occurs, AsyncWebCrawler.arun returns the cached CrawlResult early and never invokes the processing pipeline (aprocess_html). The extraction pipeline (where LLMExtractionStrategy runs) only executes inside the fresh-fetch branch.

• Code path on cache hit (early return without processing):

else:
    self.logger.url_status(..., tag="COMPLETE")
    cached_result.success = bool(html)
    cached_result.session_id = getattr(config, "session_id", None)
    cached_result.redirected_url = cached_result.redirected_url or url
    return CrawlResultContainer(cached_result)

• Processing/extraction runs only on fresh fetch:

crawl_result: CrawlResult = await self.aprocess_html(
    url=url,
    html=html,
    extracted_content=extracted_content,
    config=config,
    ...
)
...
return CrawlResultContainer(crawl_result)

• Cache read/write includes extracted_content:
  • Read: aget_cached_url(...) hydrates CrawlResult (lines ~288+)
  • Write: acache_url(...) persists extracted_content (lines ~381–409), but only when we take the fresh path.

Impact: If the original cached entry was created without an extraction strategy (or extraction was empty/"[]"), subsequent runs with an extraction strategy still return the cached result unchanged, leaving result.extracted_content empty.

Proposed fixes

• Minimal/low risk: On cache hit, if config.extraction_strategy is present and cached extracted_content is empty (or strategy signature differs), call aprocess_html using the cached HTML, then update the cache if allowed. This avoids re-fetches and keeps current behavior otherwise.
• Opt-in flag: Add force_extraction_on_cache_hit to CrawlerRunConfig; when true, always run extraction on cached HTML and write back if allowed.
• Longer-term: Store a strategy/config signature alongside the cache entry and re-run extraction when signatures differ or output is missing.