Release v0.8.5

.claude/commands/c4ai-check.md

@@ -0,0 +1,89 @@
+---
+description: "Test current changes with adversarial tests, then run full regression suite"
+arguments:
+  - name: changes
+    description: "Description of what changed (e.g. 'fixed URL normalization to preserve trailing slashes')"
+    required: true
+---
+
+# Crawl4AI Change Verification (c4ai-check)
+
+You are verifying that recent code changes work correctly AND haven't broken anything else. This is a two-phase process.
+
+**Input:** $ARGUMENTS
+
+## PHASE 1: Adversarial Testing of Current Changes
+
+Based on the change description above:
+
+1. **Understand the change**: Read the relevant files that were modified. Use `git diff` to see exactly what changed.
+
+2. **Write targeted adversarial tests**: Create a temporary test file at `tests/regression/test_tmp_changes.py` that HEAVILY tests the specific changes:
+   - Normal cases (does it work as intended?)
+   - Edge cases (boundary values, empty inputs, None, huge inputs)
+   - Regression cases (does the OLD bug still occur? it shouldn't)
+   - Interaction cases (does it break anything it touches?)
+   - Adversarial cases (weird inputs that could expose issues)
+   - At least 10-15 focused tests per change area
+
+   Rules for the temp test file:
+   - Use `@pytest.mark.asyncio` for async tests
+   - Use real browser crawling where needed (`async with AsyncWebCrawler()`)
+   - Use the `local_server` fixture from conftest.py when needed
+   - NO mocking - test real behavior
+   - Each test must have a clear docstring explaining what it verifies
+
+3. **Run the targeted tests**:
+   ```bash
+   .venv/bin/python -m pytest tests/regression/test_tmp_changes.py -v --tb=short
+   ```
+
+4. **Report results**: Show pass/fail summary. If any fail, investigate and determine if it's a real bug in the changes or a test issue. Fix the tests if needed, fix the code if there's a real bug.
+
+## PHASE 2: Full Regression Suite
+
+After Phase 1 passes:
+
+1. **Run the full regression suite** (skip network tests for speed):
+   ```bash
+   .venv/bin/python -m pytest tests/regression/ -v -m "not network" --tb=short -q
+   ```
+
+2. **Analyze failures**: For any failures:
+   - Determine if the failure is caused by the current changes (REGRESSION) or pre-existing
+   - Regressions are blockers - report them clearly
+   - Pre-existing failures should be noted but don't block
+
+3. **Clean up**: Delete the temporary test file:
+   ```bash
+   rm tests/regression/test_tmp_changes.py
+   ```
+
+## PHASE 3: Report
+
+Present a clear summary:
+
+```
+## c4ai-check Results
+
+**Changes tested:** [brief description]
+
+### Phase 1: Targeted Tests
+- Tests written: X
+- Passed: X / Failed: X
+- [List any issues found]
+
+### Phase 2: Regression Suite
+- Total: X passed, X failed, X skipped
+- Regressions caused by changes: [None / list]
+- Pre-existing issues: [None / list]
+
+### Verdict: PASS / FAIL
+[If FAIL, explain what needs fixing]
+```
+
+IMPORTANT:
+- Always delete `test_tmp_changes.py` when done, even if tests fail
+- A PASS verdict means: all targeted tests pass AND no new regressions in the suite
+- A FAIL verdict means: either targeted tests found bugs OR changes caused regressions
+- Be honest about failures - don't hide issues

.context/PR-TODOLIST.md

@@ -0,0 +1,151 @@
+# PR Review Todolist
+
+> Last updated: 2026-03-07 | Total open PRs: 6
+
+---
+
+## Remaining Open PRs (6)
+
+### Bug Fixes (2)
+
+| PR | Author | Description | Notes |
+|----|--------|-------------|-------|
+| #1207 | moncapitaine | Fix streaming error handling | Old PR, likely needs rebase |
+| #462 | jtanningbed | Fix: Add newline before pre codeblock start in html2text. 1-line fix | Very old, may still apply |
+
+### Docs/Maintenance (2)
+
+| PR | Author | Description | Notes |
+|----|--------|-------------|-------|
+| #1756 | VasiliyRad | Added AG2 community integration example and Quickstart pointer | Community example |
+| #1533 | unclecode | Add Claude Code GitHub Workflow | Owner's PR, CI |
+
+### Skipped (owner PRs)
+
+| PR | Author | Description |
+|----|--------|-------------|
+| #1533 | unclecode | Add Claude Code GitHub Workflow |
+| #1124 | unclecode | Add VNC streaming support |
+
+---
+
+## Previously Closed PRs (won't merge)
+
+| PR | Author | Description | Reason |
+|----|--------|-------------|--------|
+| #999 | loliw | Regex-based filters for deep crawling | URLPatternFilter already supports regex |
+| #1180 | kunalmanelkar | CallbackURLFilter for deep crawling | Breaks sync apply() interface |
+| #1425 | denrusio | OpenRouter API support | litellm handles openrouter/ natively |
+| #1702 | YxmMyth | CSS background image extraction | Too invasive for niche feature |
+| #1707 | dillonledoux | Crawl-delay from robots.txt | Too complex for non-standard directive |
+| #1729 | hoi | External Redis support | Docker infra - maintainer territory |
+| #1592 | Ahmed-Tawfik94 | CDP page leaks and race conditions | Superseded by develop page lifecycle system |
+
+## Previously Closed PRs (from old todolist)
+
+| PR | Author | Original Description | What happened |
+|----|--------|---------------------|---------------|
+| #1572 | Ahmed-Tawfik94 | Fix CDP setting with managed browser | Closed |
+| #1234 | AdarsHH30 | Fix TypeError when keep_data_attributes=False | Closed |
+| #1211 | Praneeth1-O-1 | Fix: safely create new page if no page exists | Closed |
+| #1200 | fischerdr | Bugfix browser manager session handling | Closed |
+| #1106 | devxpain | Fix: Adapt to CrawlerMonitor constructor change | Closed |
+| #1081 | Joorrit | Fix deep crawl scorer logic was inverted | Closed |
+| #1065 | mccullya | Fix: Update deprecated Groq models | Closed |
+| #1059 | Aaron2516 | Fix wrong proxy config type in proxy demo example | Closed |
+| #1058 | Aaron2516 | Fix dict-type proxy_config not handled properly | Closed |
+| #983 | umerkhan95 | Fix memory leak and empty responses in streaming mode | Closed |
+| #948 | GeorgeVince | Fix summarize_page.py example | Closed |
+| #1689 | mzyfree | Docker: optimize concurrency performance | Closed (contributor acknowledged) |
+| #1706 | vikas-gits-good | Fix arun_many not working with DeepCrawlStrategy | Closed |
+| #1683 | Vaccarini-Lorenzo | Implement double config for AdaptiveCrawler | Closed |
+| #1674 | blentz | Add output pagination/control for MCP endpoints | Closed |
+| #1650 | KennyStryker | Add support for Vertex AI in LLM Extraction Strategy | Closed |
+| #1580 | arpagon | Add Azure OpenAI configuration support | Closed |
+| #1417 | NickMandylas | Add CDP headers support for remote browser auth | Closed |
+| #1255 | itsskofficial | Fix JsonCssSelector to handle adjacent sibling CSS selectors | Closed |
+| #1245 | mukul-atomicwork | Feature: GitHub releases integration | Closed |
+| #1238 | yerik515 | Fix ManagedBrowser constructor and Windows encoding issues | Closed |
+| #1220 | dcieslak19973 | Allow OPENAI_BASE_URL for LLM base_url | Closed |
+| #901 | gbe3hunna | CrawlResult model: add pydantic fields and descriptions | Closed |
+| #800 | atomlong | ensure_ascii=False for json.dumps | Closed |
+| #799 | atomlong | Allow setting base_url for LLM extraction strategy in CLI | Closed |
+| #741 | atomlong | Add config option to control Content-Security-Policy header | Closed |
+| #723 | alexandreolives | Optional close page after screenshot | Closed |
+| #681 | ksallee | JS execution should happen after waiting | Closed |
+| #416 | dar0xt | Add keep-aria-label-attribute option | Closed |
+| #332 | nelzomal | Add remove_invisible_texts method to crawler strategy | Closed |
+| #312 | AndreaFrancis | Add save to HuggingFace support | Closed |
+| #1488 | AkosLukacs | Fix syntax error in README JSON example | Closed |
+| #1483 | NiclasLindqvist | Update README.md with latest docker image | Closed |
+| #1416 | adityaagre | Fix missing bracket in README code block | Closed |
+| #1272 | zhenjunMa | Fix get title bug in amazon example | Closed |
+| #1263 | vvanglro | Fix: consistent with sdk behavior | Closed |
+| #1225 | albertkim | Fix docker deployment guide URL | Closed |
+| #1223 | dowithless | Docs: add links to other language versions of README | Closed |
+| #1159 | lbeziaud | Fix cleanup warning when no process on debug port | Closed |
+| #1098 | B-X-Y | Docs: fix outdated links to Docker guide | Closed |
+| #1093 | Aaron2516 | Docs: Fixed incorrect elapsed calculation | Closed |
+| #967 | prajjwalnag | Update README.md | Closed |
+| #671 | SteveAlphaVantage | Update README.md | Closed |
+| #605 | mochamadsatria | Fix typo in docker-deployment.md filename | Closed |
+| #335 | amanagarwal042 | Add Documentation for Monitoring with OpenTelemetry | Closed |
+| #1722 | YuriNachos | Add missing docstring to MCP md endpoint | Merged directly |
+
+---
+
+## Resolved This Session (batch 5)
+
+| PR | Author | Description | Date |
+|----|--------|-------------|------|
+| #1622 | Ahmed-Tawfik94 | fix: verify redirect targets in URL seeder | 2026-03-07 |
+| #1786 | Br1an67 | fix: wire mean_delay/max_range into dispatcher | 2026-03-07 |
+| #1796 | Br1an67 | fix: DOMParser in process_iframes | 2026-03-07 |
+| #1795 | Br1an67 | fix: require api_token for /token endpoint | 2026-03-07 |
+| #1798 | SohamKukreti | fix: deep-crawl streaming mirrors Python library | 2026-03-07 |
+| #1734 | pgoslatara | chore: update GitHub Actions versions | 2026-03-07 |
+| #1290 | 130347665 | feat: type-list pipeline in JSON extraction | 2026-03-07 |
+| #1668 | microHoffman | feat: --json-ensure-ascii CLI flag | 2026-03-07 |
+
+## Resolved (batch 4)
+
+| PR | Author | Description | Date |
+|----|--------|-------------|------|
+| #1494 | AkosLukacs | docs: fix docstring param name crawler_config -> config | 2026-03-07 |
+| #1715 | YuriNachos | docs: add missing CacheMode import in quickstart | 2026-03-07 |
+| #1716 | YuriNachos | docs: fix return types to RunManyReturn | 2026-03-07 |
+| #1308 | dominicx | docs: fix css_selector type from list to string | 2026-03-07 |
+| #1789 | Br1an67 | fix: UTF-8 encoding for CLI file output | 2026-03-07 |
+| #1793 | Br1an67 | fix: configurable link_preview_timeout in AdaptiveConfig | 2026-03-07 |
+| #1792 | Br1an67 | fix: wait_for_images on screenshot endpoint | 2026-03-07 |
+| #1794 | Br1an67 | fix: cross-platform terminal input in CrawlerMonitor | 2026-03-07 |
+| #1784 | Br1an67 | fix: UnicodeEncodeError in URL seeder + zero-width chars | 2026-03-07 |
+| #1730 | hoi | fix: add TTL expiry for Redis task data | 2026-03-07 |
+
+## Previously Resolved (batches 1-3)
+
+| PR | Author | Description | Date |
+|----|--------|-------------|------|
+| #1805 | nightcityblade | fix: prevent AdaptiveCrawler from crawling external domains | 2026-03-07 |
+| #1763 | Otman404 | fix: return in finally block silently suppressing exceptions | 2026-03-07 |
+| #1803 | SohamKukreti | fix: from_serializable_dict ignoring plain data dicts | 2026-03-07 |
+| #1804 | nightcityblade | feat: add score_threshold to BestFirstCrawlingStrategy | 2026-03-07 |
+| #1790 | Br1an67 | fix: handle nested brackets in LINK_PATTERN regex | 2026-03-07 |
+| #1787 | Br1an67 | fix: strip markdown fences in LLM JSON responses | 2026-03-07 |
+| #1782 | Br1an67 | fix: preserve class/id in cleaned_html | 2026-03-07 |
+| #1788 | Br1an67 | fix: guard against None LLM content | 2026-03-07 |
+| #1783 | Br1an67 | fix: strip port from domain in is_external_url | 2026-03-07 |
+| #1179 | phamngocquy | fix: raw HTML URL token leak | 2026-03-07 |
+| #1694 | theredrad | feat: add force viewport screenshot | 2026-02-01 |
+| #1746 | ChiragBellara | fix: avoid Common Crawl calls for sitemap-only seeding | 2026-02-01 |
+| #1714 | YuriNachos | fix: replace tf-playwright-stealth with playwright-stealth | 2026-02-01 |
+| #1721 | YuriNachos | fix: respect base tag for relative link resolution | 2026-02-01 |
+| #1719 | YuriNachos | fix: include GoogleSearchCrawler script.js in package | 2026-02-01 |
+| #1717 | YuriNachos | fix: allow local embeddings by removing OpenAI fallback | 2026-02-01 |
+| #1667 | christian-oudard | fix: deep-crawl CLI outputting only first page | 2026-02-01 |
+| #1296 | vladmandic | fix: VersionManager ignoring CRAWL4_AI_BASE_DIRECTORY | 2026-02-01 |
+| #1364 | nnxiong | fix: script tag removal losing adjacent text | 2026-02-01 |
+| #1077 | RoyLeviLangware | fix: bs4 deprecation warning (text -> string) | 2026-02-01 |
+| #1281 | garyluky | fix: proxy auth ERR_INVALID_AUTH_CREDENTIALS | 2026-02-01 |
+| #1463 | TristanDonze | feat: device_scale_factor for screenshot quality | 2026-02-06 |
+| #1435 | charlaie | feat: redirected_status_code in CrawlResult | 2026-02-06 |

.github/workflows/docker-release.yml

@@ -31,7 +31,7 @@ jobs:
           df -h
 
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@v6
 
       - name: Extract version from release or tag
         id: get_version
@@ -58,16 +58,16 @@ jobs:
           echo "Semantic versions - Major: $MAJOR, Minor: $MINOR"
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@v4
 
       - name: Log in to Docker Hub
-        uses: docker/login-action@v3
+        uses: docker/login-action@v4
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_TOKEN }}
 
       - name: Build and push Docker images
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           push: true

.github/workflows/release.yml

@@ -13,10 +13,10 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@v6
 
       - name: Set up Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: '3.12'
 

.gitignore

@@ -286,6 +286,7 @@ docs/apps/linkdin/samples/insights/*
 
 scripts/
 !scripts/gen-sbom.sh
+!scripts/update_stats.py
 
 
 # Databse files
@@ -298,3 +299,6 @@ scripts/
 *.rdb
 *.ldb
 MEMORY.md
+
+# Handoff files
+HANDOFF-*.md