diff --git a/.claude/commands/c4ai-check.md b/.claude/commands/c4ai-check.md
new file mode 100644
index 000000000..f2f80009d
--- /dev/null
+++ b/.claude/commands/c4ai-check.md
@@ -0,0 +1,89 @@
+---
+description: "Test current changes with adversarial tests, then run full regression suite"
+arguments:
+ - name: changes
+ description: "Description of what changed (e.g. 'fixed URL normalization to preserve trailing slashes')"
+ required: true
+---
+
+# Crawl4AI Change Verification (c4ai-check)
+
+You are verifying that recent code changes work correctly AND haven't broken anything else. This is a two-phase process.
+
+**Input:** $ARGUMENTS
+
+## PHASE 1: Adversarial Testing of Current Changes
+
+Based on the change description above:
+
+1. **Understand the change**: Read the relevant files that were modified. Use `git diff` to see exactly what changed.
+
+2. **Write targeted adversarial tests**: Create a temporary test file at `tests/regression/test_tmp_changes.py` that HEAVILY tests the specific changes:
+ - Normal cases (does it work as intended?)
+ - Edge cases (boundary values, empty inputs, None, huge inputs)
+ - Regression cases (does the OLD bug still occur? it shouldn't)
+ - Interaction cases (does it break anything it touches?)
+ - Adversarial cases (weird inputs that could expose issues)
+ - At least 10-15 focused tests per change area
+
+ Rules for the temp test file:
+ - Use `@pytest.mark.asyncio` for async tests
+ - Use real browser crawling where needed (`async with AsyncWebCrawler()`)
+ - Use the `local_server` fixture from conftest.py when needed
+ - NO mocking - test real behavior
+ - Each test must have a clear docstring explaining what it verifies
+
+3. **Run the targeted tests**:
+ ```bash
+ .venv/bin/python -m pytest tests/regression/test_tmp_changes.py -v --tb=short
+ ```
+
+4. **Report results**: Show pass/fail summary. If any fail, investigate and determine if it's a real bug in the changes or a test issue. Fix the tests if needed, fix the code if there's a real bug.
+
+## PHASE 2: Full Regression Suite
+
+After Phase 1 passes:
+
+1. **Run the full regression suite** (skip network tests for speed):
+ ```bash
+ .venv/bin/python -m pytest tests/regression/ -v -m "not network" --tb=short -q
+ ```
+
+2. **Analyze failures**: For any failures:
+ - Determine if the failure is caused by the current changes (REGRESSION) or pre-existing
+ - Regressions are blockers - report them clearly
+ - Pre-existing failures should be noted but don't block
+
+3. **Clean up**: Delete the temporary test file:
+ ```bash
+ rm tests/regression/test_tmp_changes.py
+ ```
+
+## PHASE 3: Report
+
+Present a clear summary:
+
+```
+## c4ai-check Results
+
+**Changes tested:** [brief description]
+
+### Phase 1: Targeted Tests
+- Tests written: X
+- Passed: X / Failed: X
+- [List any issues found]
+
+### Phase 2: Regression Suite
+- Total: X passed, X failed, X skipped
+- Regressions caused by changes: [None / list]
+- Pre-existing issues: [None / list]
+
+### Verdict: PASS / FAIL
+[If FAIL, explain what needs fixing]
+```
+
+IMPORTANT:
+- Always delete `test_tmp_changes.py` when done, even if tests fail
+- A PASS verdict means: all targeted tests pass AND no new regressions in the suite
+- A FAIL verdict means: either targeted tests found bugs OR changes caused regressions
+- Be honest about failures - don't hide issues
diff --git a/.context/PR-TODOLIST.md b/.context/PR-TODOLIST.md
new file mode 100644
index 000000000..c75fa993d
--- /dev/null
+++ b/.context/PR-TODOLIST.md
@@ -0,0 +1,151 @@
+# PR Review Todolist
+
+> Last updated: 2026-03-07 | Total open PRs: 6
+
+---
+
+## Remaining Open PRs (6)
+
+### Bug Fixes (2)
+
+| PR | Author | Description | Notes |
+|----|--------|-------------|-------|
+| #1207 | moncapitaine | Fix streaming error handling | Old PR, likely needs rebase |
+| #462 | jtanningbed | Fix: Add newline before pre codeblock start in html2text. 1-line fix | Very old, may still apply |
+
+### Docs/Maintenance (2)
+
+| PR | Author | Description | Notes |
+|----|--------|-------------|-------|
+| #1756 | VasiliyRad | Added AG2 community integration example and Quickstart pointer | Community example |
+| #1533 | unclecode | Add Claude Code GitHub Workflow | Owner's PR, CI |
+
+### Skipped (owner PRs)
+
+| PR | Author | Description |
+|----|--------|-------------|
+| #1533 | unclecode | Add Claude Code GitHub Workflow |
+| #1124 | unclecode | Add VNC streaming support |
+
+---
+
+## Previously Closed PRs (won't merge)
+
+| PR | Author | Description | Reason |
+|----|--------|-------------|--------|
+| #999 | loliw | Regex-based filters for deep crawling | URLPatternFilter already supports regex |
+| #1180 | kunalmanelkar | CallbackURLFilter for deep crawling | Breaks sync apply() interface |
+| #1425 | denrusio | OpenRouter API support | litellm handles openrouter/ natively |
+| #1702 | YxmMyth | CSS background image extraction | Too invasive for niche feature |
+| #1707 | dillonledoux | Crawl-delay from robots.txt | Too complex for non-standard directive |
+| #1729 | hoi | External Redis support | Docker infra - maintainer territory |
+| #1592 | Ahmed-Tawfik94 | CDP page leaks and race conditions | Superseded by develop page lifecycle system |
+
+## Previously Closed PRs (from old todolist)
+
+| PR | Author | Original Description | What happened |
+|----|--------|---------------------|---------------|
+| #1572 | Ahmed-Tawfik94 | Fix CDP setting with managed browser | Closed |
+| #1234 | AdarsHH30 | Fix TypeError when keep_data_attributes=False | Closed |
+| #1211 | Praneeth1-O-1 | Fix: safely create new page if no page exists | Closed |
+| #1200 | fischerdr | Bugfix browser manager session handling | Closed |
+| #1106 | devxpain | Fix: Adapt to CrawlerMonitor constructor change | Closed |
+| #1081 | Joorrit | Fix deep crawl scorer logic was inverted | Closed |
+| #1065 | mccullya | Fix: Update deprecated Groq models | Closed |
+| #1059 | Aaron2516 | Fix wrong proxy config type in proxy demo example | Closed |
+| #1058 | Aaron2516 | Fix dict-type proxy_config not handled properly | Closed |
+| #983 | umerkhan95 | Fix memory leak and empty responses in streaming mode | Closed |
+| #948 | GeorgeVince | Fix summarize_page.py example | Closed |
+| #1689 | mzyfree | Docker: optimize concurrency performance | Closed (contributor acknowledged) |
+| #1706 | vikas-gits-good | Fix arun_many not working with DeepCrawlStrategy | Closed |
+| #1683 | Vaccarini-Lorenzo | Implement double config for AdaptiveCrawler | Closed |
+| #1674 | blentz | Add output pagination/control for MCP endpoints | Closed |
+| #1650 | KennyStryker | Add support for Vertex AI in LLM Extraction Strategy | Closed |
+| #1580 | arpagon | Add Azure OpenAI configuration support | Closed |
+| #1417 | NickMandylas | Add CDP headers support for remote browser auth | Closed |
+| #1255 | itsskofficial | Fix JsonCssSelector to handle adjacent sibling CSS selectors | Closed |
+| #1245 | mukul-atomicwork | Feature: GitHub releases integration | Closed |
+| #1238 | yerik515 | Fix ManagedBrowser constructor and Windows encoding issues | Closed |
+| #1220 | dcieslak19973 | Allow OPENAI_BASE_URL for LLM base_url | Closed |
+| #901 | gbe3hunna | CrawlResult model: add pydantic fields and descriptions | Closed |
+| #800 | atomlong | ensure_ascii=False for json.dumps | Closed |
+| #799 | atomlong | Allow setting base_url for LLM extraction strategy in CLI | Closed |
+| #741 | atomlong | Add config option to control Content-Security-Policy header | Closed |
+| #723 | alexandreolives | Optional close page after screenshot | Closed |
+| #681 | ksallee | JS execution should happen after waiting | Closed |
+| #416 | dar0xt | Add keep-aria-label-attribute option | Closed |
+| #332 | nelzomal | Add remove_invisible_texts method to crawler strategy | Closed |
+| #312 | AndreaFrancis | Add save to HuggingFace support | Closed |
+| #1488 | AkosLukacs | Fix syntax error in README JSON example | Closed |
+| #1483 | NiclasLindqvist | Update README.md with latest docker image | Closed |
+| #1416 | adityaagre | Fix missing bracket in README code block | Closed |
+| #1272 | zhenjunMa | Fix get title bug in amazon example | Closed |
+| #1263 | vvanglro | Fix: consistent with sdk behavior | Closed |
+| #1225 | albertkim | Fix docker deployment guide URL | Closed |
+| #1223 | dowithless | Docs: add links to other language versions of README | Closed |
+| #1159 | lbeziaud | Fix cleanup warning when no process on debug port | Closed |
+| #1098 | B-X-Y | Docs: fix outdated links to Docker guide | Closed |
+| #1093 | Aaron2516 | Docs: Fixed incorrect elapsed calculation | Closed |
+| #967 | prajjwalnag | Update README.md | Closed |
+| #671 | SteveAlphaVantage | Update README.md | Closed |
+| #605 | mochamadsatria | Fix typo in docker-deployment.md filename | Closed |
+| #335 | amanagarwal042 | Add Documentation for Monitoring with OpenTelemetry | Closed |
+| #1722 | YuriNachos | Add missing docstring to MCP md endpoint | Merged directly |
+
+---
+
+## Resolved This Session (batch 5)
+
+| PR | Author | Description | Date |
+|----|--------|-------------|------|
+| #1622 | Ahmed-Tawfik94 | fix: verify redirect targets in URL seeder | 2026-03-07 |
+| #1786 | Br1an67 | fix: wire mean_delay/max_range into dispatcher | 2026-03-07 |
+| #1796 | Br1an67 | fix: DOMParser in process_iframes | 2026-03-07 |
+| #1795 | Br1an67 | fix: require api_token for /token endpoint | 2026-03-07 |
+| #1798 | SohamKukreti | fix: deep-crawl streaming mirrors Python library | 2026-03-07 |
+| #1734 | pgoslatara | chore: update GitHub Actions versions | 2026-03-07 |
+| #1290 | 130347665 | feat: type-list pipeline in JSON extraction | 2026-03-07 |
+| #1668 | microHoffman | feat: --json-ensure-ascii CLI flag | 2026-03-07 |
+
+## Resolved (batch 4)
+
+| PR | Author | Description | Date |
+|----|--------|-------------|------|
+| #1494 | AkosLukacs | docs: fix docstring param name crawler_config -> config | 2026-03-07 |
+| #1715 | YuriNachos | docs: add missing CacheMode import in quickstart | 2026-03-07 |
+| #1716 | YuriNachos | docs: fix return types to RunManyReturn | 2026-03-07 |
+| #1308 | dominicx | docs: fix css_selector type from list to string | 2026-03-07 |
+| #1789 | Br1an67 | fix: UTF-8 encoding for CLI file output | 2026-03-07 |
+| #1793 | Br1an67 | fix: configurable link_preview_timeout in AdaptiveConfig | 2026-03-07 |
+| #1792 | Br1an67 | fix: wait_for_images on screenshot endpoint | 2026-03-07 |
+| #1794 | Br1an67 | fix: cross-platform terminal input in CrawlerMonitor | 2026-03-07 |
+| #1784 | Br1an67 | fix: UnicodeEncodeError in URL seeder + zero-width chars | 2026-03-07 |
+| #1730 | hoi | fix: add TTL expiry for Redis task data | 2026-03-07 |
+
+## Previously Resolved (batches 1-3)
+
+| PR | Author | Description | Date |
+|----|--------|-------------|------|
+| #1805 | nightcityblade | fix: prevent AdaptiveCrawler from crawling external domains | 2026-03-07 |
+| #1763 | Otman404 | fix: return in finally block silently suppressing exceptions | 2026-03-07 |
+| #1803 | SohamKukreti | fix: from_serializable_dict ignoring plain data dicts | 2026-03-07 |
+| #1804 | nightcityblade | feat: add score_threshold to BestFirstCrawlingStrategy | 2026-03-07 |
+| #1790 | Br1an67 | fix: handle nested brackets in LINK_PATTERN regex | 2026-03-07 |
+| #1787 | Br1an67 | fix: strip markdown fences in LLM JSON responses | 2026-03-07 |
+| #1782 | Br1an67 | fix: preserve class/id in cleaned_html | 2026-03-07 |
+| #1788 | Br1an67 | fix: guard against None LLM content | 2026-03-07 |
+| #1783 | Br1an67 | fix: strip port from domain in is_external_url | 2026-03-07 |
+| #1179 | phamngocquy | fix: raw HTML URL token leak | 2026-03-07 |
+| #1694 | theredrad | feat: add force viewport screenshot | 2026-02-01 |
+| #1746 | ChiragBellara | fix: avoid Common Crawl calls for sitemap-only seeding | 2026-02-01 |
+| #1714 | YuriNachos | fix: replace tf-playwright-stealth with playwright-stealth | 2026-02-01 |
+| #1721 | YuriNachos | fix: respect base tag for relative link resolution | 2026-02-01 |
+| #1719 | YuriNachos | fix: include GoogleSearchCrawler script.js in package | 2026-02-01 |
+| #1717 | YuriNachos | fix: allow local embeddings by removing OpenAI fallback | 2026-02-01 |
+| #1667 | christian-oudard | fix: deep-crawl CLI outputting only first page | 2026-02-01 |
+| #1296 | vladmandic | fix: VersionManager ignoring CRAWL4_AI_BASE_DIRECTORY | 2026-02-01 |
+| #1364 | nnxiong | fix: script tag removal losing adjacent text | 2026-02-01 |
+| #1077 | RoyLeviLangware | fix: bs4 deprecation warning (text -> string) | 2026-02-01 |
+| #1281 | garyluky | fix: proxy auth ERR_INVALID_AUTH_CREDENTIALS | 2026-02-01 |
+| #1463 | TristanDonze | feat: device_scale_factor for screenshot quality | 2026-02-06 |
+| #1435 | charlaie | feat: redirected_status_code in CrawlResult | 2026-02-06 |
diff --git a/.github/workflows/docker-release.yml b/.github/workflows/docker-release.yml
index fe64bdaac..23c7760dc 100644
--- a/.github/workflows/docker-release.yml
+++ b/.github/workflows/docker-release.yml
@@ -31,7 +31,7 @@ jobs:
df -h
- name: Checkout code
- uses: actions/checkout@v4
+ uses: actions/checkout@v6
- name: Extract version from release or tag
id: get_version
@@ -58,16 +58,16 @@ jobs:
echo "Semantic versions - Major: $MAJOR, Minor: $MINOR"
- name: Set up Docker Buildx
- uses: docker/setup-buildx-action@v3
+ uses: docker/setup-buildx-action@v4
- name: Log in to Docker Hub
- uses: docker/login-action@v3
+ uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
- name: Build and push Docker images
- uses: docker/build-push-action@v5
+ uses: docker/build-push-action@v6
with:
context: .
push: true
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index bf1ad7dcd..91e84ffb4 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -13,10 +13,10 @@ jobs:
steps:
- name: Checkout code
- uses: actions/checkout@v4
+ uses: actions/checkout@v6
- name: Set up Python
- uses: actions/setup-python@v5
+ uses: actions/setup-python@v6
with:
python-version: '3.12'
diff --git a/.gitignore b/.gitignore
index 2f27e49f0..c5478815a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -286,6 +286,7 @@ docs/apps/linkdin/samples/insights/*
scripts/
!scripts/gen-sbom.sh
+!scripts/update_stats.py
# Databse files
@@ -298,3 +299,6 @@ scripts/
*.rdb
*.ldb
MEMORY.md
+
+# Handoff files
+HANDOFF-*.md
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 000000000..38d22c2cf
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,102 @@
+# Contributing to Crawl4AI
+
+Welcome to the Crawl4AI project! As an open-source library for web crawling and AI integration, we value contributions from the community. This guide explains our branching strategy, how to contribute effectively, and the overall release process. Our goal is to maintain a stable, collaborative environment where bug fixes, features, and improvements can be integrated smoothly while allowing for experimental development.
+
+We follow a GitFlow-inspired workflow to ensure predictability and quality. Releases occur approximately every two weeks, with a focus on semantic versioning, comprehensive documentation, and user-friendly updates.
+
+## Core Branches
+
+- **main**: The stable branch containing production-ready code. It's always identical to the latest released version and is tagged for releases. Do not submit PRs directly here.
+- **develop**: The primary integration branch for ongoing development. This is where all contributions (bug fixes, minor features, documentation updates) are merged. Submit your pull requests targeting this branch.
+- **next**: Reserved for the lead maintainer (Unclecode) to experiment with major features, refactors, or cutting-edge changes. These are merged into `develop` when ready.
+- **release/vX.Y.Z**: Temporary branches created from `develop` for final release preparations (e.g., version bumps, demos, release notes). These are short-lived and deleted after the release.
+
+## Contributor Workflow
+
+We encourage contributions of all kinds: bug fixes, new features, documentation improvements, tests, or even Docker enhancements. Follow these steps to contribute:
+
+1. **Fork the Repository**: Create your own fork on GitHub.
+2. **Create a Branch**: Base your work on the `develop` branch.
+
+ ```
+ git checkout develop
+ git checkout -b feature/your-feature-name # Or bugfix/your-bugfix-name
+
+ ```
+
+3. **Make Changes**:
+ - Implement your feature or fix.
+ - If updating documentation (e.g., README.md, mkdocs.yml, or docs/blog/), ensure version references are consistent (e.g., update site_name in mkdocs.yml to reflect the upcoming version if relevant).
+ - For Docker-related changes (e.g., Dockerfile, docker-compose.yml, or docs/md_v2/core/docker-deployment.md), test locally and include build instructions in your PR description.
+ - Add tests if applicable (run `pytest` to verify).
+ - Follow code style guidelines (use black for formatting).
+4. **Commit and Push**:
+ - Use descriptive commit messages (e.g., "Fix: Resolve issue with async crawling").
+ - Push to your fork: `git push origin feature/your-feature-name`.
+5. **Submit a Pull Request**:
+ - Target the `develop` branch.
+ - Provide a clear description: What does it do? Link to any issues. Include screenshots or code examples if helpful.
+ - If your change affects documentation or Docker, mention how it aligns with the version (e.g., "Updates Docker docs for v0.7.0 compatibility").
+ - We'll review and merge approved PRs into `develop`.
+6. **Discuss Large Changes**: For major features or experimental ideas, open an issue first to align with the project's direction.
+
+If your PR involves breaking changes, include a migration guide in the description.
+
+## Lead Maintainer's Workflow (For Reference)
+
+- The lead maintainer (Unclecode) uses the `next` branch for isolated experimental work.
+- Features from `next` are periodically merged into `develop` (via rebase and merge) to keep everything in sync.
+- This isolation ensures your contributions aren't disrupted by ongoing major changes.
+
+## Release Process (High-Level Overview)
+
+Releases happen bi-weekly to ship improvements regularly. As a contributor, your merged changes in `develop` will be included in the next release unless specified otherwise. Here's a summary of what happens:
+
+- **Preparation**: A temporary `release/vX.Y.Z` branch is created from `develop`. Any ready features from `next` are merged here.
+- **Final Updates**:
+ - Version bump in code (e.g., `__version__.py`).
+ - Creation of a demo script in `examples/` to showcase new features.
+ - Writing release notes in `docs/blog/` (personal "I" voice from Unclecode, with code examples, impacts, and migration guides if needed).
+ - Documentation updates: README.md (highlights, version refs), mkdocs.yml (site_name with version), docs/blog/index.md (add new release), and copying notes to `docs/md_v2/blog/releases/`.
+ - Docker updates: Dockerfile (version arg), docker-compose.yml, deploy/docker/README.md, and docs/md_v2/core/docker-deployment.md. A release candidate image (e.g., `X.Y.Z-r1`) is built and tested.
+- **Testing and Merge**: Full tests run; changes committed and merged to `main` with a tag.
+- **Publication**: Tagged release on GitHub (with notes), publish to PyPI, and push Docker images (stable and `latest` after testing).
+- **Sync**: Back-merge to `develop` and reset `next` for the next cycle.
+
+Semantic versioning is used: MAJOR for breaking changes, MINOR for features, PATCH for fixes. Pre-releases (e.g., `-rc1`) may be used for testing.
+
+If your contribution requires Docker testing or affects docs, it may be part of this stepβfeel free to suggest updates in your PR.
+
+## Benefits of This Approach
+
+- **Stability**: `main` is always reliable for users.
+- **Collaboration**: Fixed PR target (`develop`) makes contributing straightforward.
+- **Isolation**: Experimental work in `next` doesn't block team progress.
+- **User-Focused**: Releases include demos, detailed notes, and updated docs/Docker for easy adoption.
+- **Predictability**: Bi-weekly cadence keeps the project active.
+
+## Checklist for Contributors
+
+Before submitting a PR:
+
+- [ ] Based on and targeting `develop`.
+- [ ] Tests pass (`pytest`).
+- [ ] Docs updated if needed (e.g., version refs in mkdocs.yml, Docker files).
+- [ ] No breaking changes without a migration guide.
+- [ ] Descriptive title and description.
+
+## Common Issues
+
+- **Merge Conflicts**: Rebase your branch on latest `develop` before PR.
+- **Docker Builds**: Test multi-arch (amd64/arm64) locally if changing Dockerfile.
+- **Version Consistency**: Ensure any version mentions match semantic rules.
+
+## Communication
+
+- Open issues for discussions or bugs.
+- Join our Discord (link in README) for real-time help.
+- After releases, announcements go to GitHub, Discord, and social media.
+
+Thanks for contributing to Crawl4AI β we appreciate your help in making it better!
+
+*Last Updated: Feb 3, 2026*
diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md
index 6068c6184..a424ed7c0 100644
--- a/CONTRIBUTORS.md
+++ b/CONTRIBUTORS.md
@@ -23,6 +23,39 @@ We would like to thank the following people for their contributions to Crawl4AI:
- [HamzaFarhan](https://github.com/HamzaFarhan) - Handled the cases where markdown_with_citations, references_markdown, and filtered_html might not be defined [#293](https://github.com/unclecode/crawl4ai/pull/293)
- [NanmiCoder](https://github.com/NanmiCoder) - fix: crawler strategy exception handling and fixes [#271](https://github.com/unclecode/crawl4ai/pull/271)
- [paulokuong](https://github.com/paulokuong) - fix: RAWL4_AI_BASE_DIRECTORY should be Path object instead of string [#298](https://github.com/unclecode/crawl4ai/pull/298)
+- [TheRedRad](https://github.com/theredrad) - feat: add force viewport screenshot option [#1694](https://github.com/unclecode/crawl4ai/pull/1694)
+- [ChiragBellara](https://github.com/ChiragBellara) - fix: avoid Common Crawl calls for sitemap-only URL seeding [#1746](https://github.com/unclecode/crawl4ai/pull/1746)
+- [YuriNachos](https://github.com/YuriNachos) - fix: replace tf-playwright-stealth with playwright-stealth [#1714](https://github.com/unclecode/crawl4ai/pull/1714), fix: respect `` tag for relative link resolution [#1721](https://github.com/unclecode/crawl4ai/pull/1721), fix: include GoogleSearchCrawler script.js in package [#1719](https://github.com/unclecode/crawl4ai/pull/1719), fix: allow local embeddings by removing OpenAI fallback [#1717](https://github.com/unclecode/crawl4ai/pull/1717), docs: add missing CacheMode import [#1715](https://github.com/unclecode/crawl4ai/pull/1715), docs: fix return types to RunManyReturn [#1716](https://github.com/unclecode/crawl4ai/pull/1716)
+- [christian-oudard](https://github.com/christian-oudard) - fix: deep-crawl CLI outputting only the first page [#1667](https://github.com/unclecode/crawl4ai/pull/1667)
+- [vladmandic](https://github.com/vladmandic) - fix: VersionManager ignoring CRAWL4_AI_BASE_DIRECTORY env var [#1296](https://github.com/unclecode/crawl4ai/pull/1296)
+- [nnxiong](https://github.com/nnxiong) - fix: script tag removal losing adjacent text in cleaned_html [#1364](https://github.com/unclecode/crawl4ai/pull/1364)
+- [RoyLeviLangware](https://github.com/RoyLeviLangware) - fix: bs4 deprecation warning (text -> string) [#1077](https://github.com/unclecode/crawl4ai/pull/1077)
+- [garyluky](https://github.com/garyluky) - fix: proxy auth ERR_INVALID_AUTH_CREDENTIALS [#1281](https://github.com/unclecode/crawl4ai/pull/1281)
+- [Martichou](https://github.com/Martichou) - investigation: browser context memory leak under continuous load [#1640](https://github.com/unclecode/crawl4ai/pull/1640), [#943](https://github.com/unclecode/crawl4ai/issues/943)
+- [danyQe](https://github.com/danyQe) - identified: temperature typo in async_configs.py [#973](https://github.com/unclecode/crawl4ai/pull/973)
+- [saipavanmeruga7797](https://github.com/saipavanmeruga7797) - identified: local HTML file crawling bug with capture_console_messages [#1073](https://github.com/unclecode/crawl4ai/pull/1073)
+- [stevenaldinger](https://github.com/stevenaldinger) - identified: duplicate PROMPT_EXTRACT_BLOCKS dead code in prompts.py [#931](https://github.com/unclecode/crawl4ai/pull/931)
+- [chrizzly2309](https://github.com/chrizzly2309) - identified: JWT auth bypass when no credentials provided [#1133](https://github.com/unclecode/crawl4ai/pull/1133)
+- [complete-dope](https://github.com/complete-dope) - identified: console logging error attribute issue [#729](https://github.com/unclecode/crawl4ai/pull/729)
+- [TristanDonze](https://github.com/TristanDonze) - feat: add configurable device_scale_factor for screenshot quality [#1463](https://github.com/unclecode/crawl4ai/pull/1463)
+- [charlaie](https://github.com/charlaie) - feat: add redirected_status_code to CrawlResult [#1435](https://github.com/unclecode/crawl4ai/pull/1435)
+- [mzyfree](https://github.com/mzyfree) - investigation: Docker concurrency performance and pool resource management [#1689](https://github.com/unclecode/crawl4ai/pull/1689)
+- [nightcityblade](https://github.com/nightcityblade) - fix: prevent AdaptiveCrawler from crawling external domains [#1805](https://github.com/unclecode/crawl4ai/pull/1805)
+- [Otman404](https://github.com/Otman404) - fix: return in finally block silently suppressing exceptions in dispatcher [#1763](https://github.com/unclecode/crawl4ai/pull/1763)
+- [SohamKukreti](https://github.com/SohamKukreti) - fix: from_serializable_dict ignoring plain data dicts with "type" key [#1803](https://github.com/unclecode/crawl4ai/pull/1803), fix: deep-crawl streaming mirrors Python library behavior [#1798](https://github.com/unclecode/crawl4ai/pull/1798)
+- [Br1an67](https://github.com/Br1an67) - fix: handle nested brackets and parentheses in LINK_PATTERN regex [#1790](https://github.com/unclecode/crawl4ai/pull/1790), identified: strip markdown fences in LLM JSON responses [#1787](https://github.com/unclecode/crawl4ai/pull/1787), fix: preserve class/id in cleaned_html [#1782](https://github.com/unclecode/crawl4ai/pull/1782), fix: guard against None LLM content [#1788](https://github.com/unclecode/crawl4ai/pull/1788), fix: strip port from domain in is_external_url [#1783](https://github.com/unclecode/crawl4ai/pull/1783), fix: UTF-8 encoding for CLI output [#1789](https://github.com/unclecode/crawl4ai/pull/1789), fix: configurable link_preview_timeout [#1793](https://github.com/unclecode/crawl4ai/pull/1793), fix: wait_for_images on screenshot endpoint [#1792](https://github.com/unclecode/crawl4ai/pull/1792), fix: cross-platform terminal input in CrawlerMonitor [#1794](https://github.com/unclecode/crawl4ai/pull/1794), fix: UnicodeEncodeError in URL seeder [#1784](https://github.com/unclecode/crawl4ai/pull/1784), fix: wire mean_delay/max_range into dispatcher [#1786](https://github.com/unclecode/crawl4ai/pull/1786), fix: DOMParser in process_iframes [#1796](https://github.com/unclecode/crawl4ai/pull/1796), fix: require api_token for /token endpoint [#1795](https://github.com/unclecode/crawl4ai/pull/1795)
+- [nightcityblade](https://github.com/nightcityblade) - feat: add score_threshold to BestFirstCrawlingStrategy [#1804](https://github.com/unclecode/crawl4ai/pull/1804)
+- [phamngocquy](https://github.com/phamngocquy) - identified: raw HTML URL token leak [#1179](https://github.com/unclecode/crawl4ai/pull/1179)
+- [AkosLukacs](https://github.com/AkosLukacs) - docs: fix docstring param name crawler_config -> config [#1494](https://github.com/unclecode/crawl4ai/pull/1494)
+- [dominicx](https://github.com/dominicx) - docs: fix css_selector type from list to string [#1308](https://github.com/unclecode/crawl4ai/pull/1308)
+- [hoi](https://github.com/hoi) - fix: add TTL expiry for Redis task data [#1730](https://github.com/unclecode/crawl4ai/pull/1730)
+- [maksimzayats](https://github.com/maksimzayats) - docs: modernize deprecated API usage across 25 files [#1770](https://github.com/unclecode/crawl4ai/pull/1770)
+- [jtanningbed](https://github.com/jtanningbed) - fix: add newline before opening code fence in html2text [#462](https://github.com/unclecode/crawl4ai/pull/462)
+- [Ahmed-Tawfik94](https://github.com/Ahmed-Tawfik94) - identified: redirect target verification in URL seeder [#1622](https://github.com/unclecode/crawl4ai/pull/1622)
+- [hafezparast](https://github.com/hafezparast) - identified: PDFContentScrapingStrategy deserialization fix [#1815](https://github.com/unclecode/crawl4ai/issues/1815); fix: screenshot distortion, deep crawl timeout/arun_many, CLI encoding [#1829](https://github.com/unclecode/crawl4ai/pull/1829)
+- [pgoslatara](https://github.com/pgoslatara) - chore: update GitHub Actions to latest versions [#1734](https://github.com/unclecode/crawl4ai/pull/1734)
+- [130347665](https://github.com/130347665) - feat: type-list pipeline in JsonCssExtractionStrategy [#1290](https://github.com/unclecode/crawl4ai/pull/1290)
+- [microHoffman](https://github.com/microHoffman) - feat: add --json-ensure-ascii CLI flag for Unicode handling [#1668](https://github.com/unclecode/crawl4ai/pull/1668)
#### Feb-Alpha-1
- [sufianuddin](https://github.com/sufianuddin) - fix: [Documentation for JsonCssExtractionStrategy](https://github.com/unclecode/crawl4ai/issues/651)
diff --git a/Dockerfile b/Dockerfile
index 7cfdcf0ef..e5c6877e1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,7 +1,7 @@
FROM python:3.12-slim-bookworm AS build
# C4ai version
-ARG C4AI_VER=0.8.0
+ARG C4AI_VER=0.8.5
ENV C4AI_VERSION=$C4AI_VER
LABEL c4ai.version=$C4AI_VER
@@ -27,10 +27,20 @@ ARG INSTALL_TYPE=default
ARG ENABLE_GPU=false
ARG TARGETARCH
+# Redis version β pinned to a CVE-patched release by default.
+# Override with --build-arg REDIS_VERSION="" for latest, or
+# --build-arg REDIS_VERSION="6:7.2.7-1rl1~bookworm1" for a specific version.
+ARG REDIS_VERSION="6:7.2.7-1rl1~bookworm1"
+
LABEL maintainer="unclecode"
LABEL description="π₯π·οΈ Crawl4AI: Open-source LLM Friendly Web Crawler & scraper"
LABEL version="1.0"
+# Add official Redis repository for security-patched versions
+RUN curl -fsSL https://packages.redis.io/gpg | gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg \
+ && echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb bookworm main" \
+ > /etc/apt/sources.list.d/redis.list
+
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
curl \
@@ -41,7 +51,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
pkg-config \
python3-dev \
libjpeg-dev \
- redis-server \
+ redis-server${REDIS_VERSION:+=$REDIS_VERSION} \
supervisor \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/README.md b/README.md
index ac18d3d18..914db05f0 100644
--- a/README.md
+++ b/README.md
@@ -37,11 +37,13 @@ Limited slots._
Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
-[β¨ Check out latest update v0.8.0](#-recent-updates)
+[β¨ Check out latest update v0.8.5](#-recent-updates)
-β¨ **New in v0.8.0**: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with `resume_state` and `on_state_change` callbacks for long-running crawls. New `prefetch=True` mode for 5-10x faster URL discovery. Critical security fixes for Docker API (hooks disabled by default, file:// URLs blocked). [Release notes β](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.0.md)
+β¨ **New in v0.8.5**: Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes! Automatic 3-tier anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, config defaults API, consent popup removal, and critical security patches. [Release notes β](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.5.md)
-β¨ Recent v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. [Release notes β](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.8.md)
+β¨ Recent v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with `resume_state` and `on_state_change` callbacks for long-running crawls. New `prefetch=True` mode for 5-10x faster URL discovery. [Release notes β](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.0.md)
+
+β¨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. [Release notes β](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.8.md)
β¨ Previous v0.7.7: Complete Self-Hosting Platform with Real-time Monitoring! Enterprise-grade monitoring dashboard, comprehensive REST API, WebSocket streaming, and smart browser pool management. [Release notes β](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.7.md)
@@ -805,21 +807,21 @@ This release focuses on stability with 11 bug fixes addressing issues reported b
```
- **π¨ Multi-URL Configuration**: Different strategies for different URL patterns in one batch:
- ```python
- from crawl4ai import CrawlerRunConfig, MatchMode
+```python
+from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode
configs = [
# Documentation sites - aggressive caching
CrawlerRunConfig(
url_matcher=["*docs*", "*documentation*"],
- cache_mode="write",
+ cache_mode=CacheMode.WRITE_ONLY,
markdown_generator_options={"include_links": True}
),
# News/blog sites - fresh content
CrawlerRunConfig(
url_matcher=lambda url: 'blog' in url or 'news' in url,
- cache_mode="bypass"
+ cache_mode=CacheMode.BYPASS
),
# Fallback for everything else
diff --git a/SECURITY.md b/SECURITY.md
index 92e3d60e6..3245b9f57 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -98,10 +98,21 @@ When using Crawl4AI as a Python library:
| CVE-pending-1 | CRITICAL | RCE via hooks `__import__` | Removed from allowed builtins |
| CVE-pending-2 | HIGH | LFI via `file://` URLs | URL scheme validation added |
+### Fixed in v0.8.1
+
+| ID | Severity | Description | Fix |
+|----|----------|-------------|-----|
+| CVE-pending-3 | CRITICAL | RCE via deserialization + `eval()` in `/crawl` endpoint | Allowlisted deserializable types; AST-validated computed field expressions |
+
See [Security Advisory](https://github.com/unclecode/crawl4ai/security/advisories) for details.
## Security Features
+### v0.8.1+
+
+- **Deserialization Allowlist**: Only known-safe types can be instantiated via API config
+- **Safe Expression Evaluation**: Computed fields use AST validation (no `__import__`, no dunder access)
+
### v0.8.0+
- **URL Scheme Validation**: Blocks `file://`, `javascript:`, `data:` URLs on API
@@ -115,7 +126,8 @@ See [Security Advisory](https://github.com/unclecode/crawl4ai/security/advisorie
We thank the following security researchers for responsibly disclosing vulnerabilities:
-- **[Neo by ProjectDiscovery](https://projectdiscovery.io/blog/introducing-neo)** - RCE and LFI vulnerabilities (December 2025)
+- **Alec M** β RCE via deserialization in `/crawl` endpoint (January 2026)
+- **[Neo by ProjectDiscovery](https://projectdiscovery.io/blog/introducing-neo)** β RCE and LFI vulnerabilities (December 2025)
---
diff --git a/crawl4ai/__init__.py b/crawl4ai/__init__.py
index af35e6a0e..03e734deb 100644
--- a/crawl4ai/__init__.py
+++ b/crawl4ai/__init__.py
@@ -10,6 +10,7 @@
LXMLWebScrapingStrategy,
WebScrapingStrategy, # Backward compatibility alias
)
+from .processors.pdf import PDFContentScrapingStrategy
from .async_logger import (
AsyncLoggerBase,
AsyncLogger,
diff --git a/crawl4ai/__version__.py b/crawl4ai/__version__.py
index 1a1e87615..55ab44a19 100644
--- a/crawl4ai/__version__.py
+++ b/crawl4ai/__version__.py
@@ -1,7 +1,7 @@
# crawl4ai/__version__.py
# This is the version that will be used for stable releases
-__version__ = "0.8.0"
+__version__ = "0.8.5"
# For nightly builds, this gets set during build process
__nightly_version__ = None
diff --git a/crawl4ai/adaptive_crawler.py b/crawl4ai/adaptive_crawler.py
index b7c649b00..6aa1d3c20 100644
--- a/crawl4ai/adaptive_crawler.py
+++ b/crawl4ai/adaptive_crawler.py
@@ -179,6 +179,7 @@ class AdaptiveConfig:
# Embedding strategy parameters
embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2"
embedding_llm_config: Optional[Union[LLMConfig, Dict]] = None # Separate config for embeddings
+ query_llm_config: Optional[Union[LLMConfig, Dict]] = None # Config for query expansion (chat completion)
n_query_variations: int = 10
coverage_threshold: float = 0.85
alpha_shape_alpha: float = 0.5
@@ -206,6 +207,9 @@ class AdaptiveConfig:
# Example: Links with >0.85 similarity to existing KB get penalized to avoid redundancy
# Lower = more aggressive deduplication, Higher = allow more similar content
+ # Link preview timeout (seconds)
+ link_preview_timeout: float = 5.0
+
# Embedding stopping criteria parameters
embedding_min_relative_improvement: float = 0.1 # Minimum relative improvement to continue
# Example: If confidence is 0.6, need improvement > 0.06 per batch to continue crawling
@@ -256,24 +260,18 @@ def _embedding_llm_config_dict(self) -> Optional[Dict]:
"""Convert LLMConfig to dict format for backward compatibility."""
if self.embedding_llm_config is None:
return None
-
if isinstance(self.embedding_llm_config, dict):
- # Already a dict - return as-is for backward compatibility
return self.embedding_llm_config
-
- # Convert LLMConfig object to dict format
- return {
- 'provider': self.embedding_llm_config.provider,
- 'api_token': self.embedding_llm_config.api_token,
- 'base_url': getattr(self.embedding_llm_config, 'base_url', None),
- 'temperature': getattr(self.embedding_llm_config, 'temperature', None),
- 'max_tokens': getattr(self.embedding_llm_config, 'max_tokens', None),
- 'top_p': getattr(self.embedding_llm_config, 'top_p', None),
- 'frequency_penalty': getattr(self.embedding_llm_config, 'frequency_penalty', None),
- 'presence_penalty': getattr(self.embedding_llm_config, 'presence_penalty', None),
- 'stop': getattr(self.embedding_llm_config, 'stop', None),
- 'n': getattr(self.embedding_llm_config, 'n', None),
- }
+ return self.embedding_llm_config.to_dict()
+
+ @property
+ def _query_llm_config_dict(self) -> Optional[Dict]:
+ """Convert query LLM config to dict format."""
+ if self.query_llm_config is None:
+ return None
+ if isinstance(self.query_llm_config, dict):
+ return self.query_llm_config
+ return self.query_llm_config.to_dict()
class CrawlStrategy(ABC):
@@ -617,9 +615,10 @@ def _get_document_terms(self, crawl_result: CrawlResult) -> List[str]:
class EmbeddingStrategy(CrawlStrategy):
"""Embedding-based adaptive crawling using semantic space coverage"""
- def __init__(self, embedding_model: str = None, llm_config: Union[LLMConfig, Dict] = None):
+ def __init__(self, embedding_model: str = None, llm_config: Union[LLMConfig, Dict] = None, query_llm_config: Union[LLMConfig, Dict] = None):
self.embedding_model = embedding_model or "sentence-transformers/all-MiniLM-L6-v2"
self.llm_config = llm_config
+ self.query_llm_config = query_llm_config
self._embedding_cache = {}
self._link_embedding_cache = {} # Cache for link embeddings
self._validation_passed = False # Track if validation passed
@@ -630,19 +629,46 @@ def __init__(self, embedding_model: str = None, llm_config: Union[LLMConfig, Dic
self._validation_embeddings_cache = None # Cache validation query embeddings
self._kb_similarity_threshold = 0.95 # Threshold for deduplication
- def _get_embedding_llm_config_dict(self) -> Dict:
- """Get embedding LLM config as dict with fallback to default."""
+ def _get_embedding_llm_config_dict(self) -> Optional[Dict]:
+ """Get embedding LLM config as dict, or None for local embeddings."""
if hasattr(self, 'config') and self.config:
config_dict = self.config._embedding_llm_config_dict
if config_dict:
return config_dict
-
- # Fallback to default if no config provided
- return {
- 'provider': 'openai/text-embedding-3-small',
- 'api_token': os.getenv('OPENAI_API_KEY')
- }
-
+
+ # Return None to use local sentence-transformers embeddings
+ return None
+
+ def _get_query_llm_config_dict(self) -> Optional[Dict]:
+ """Get query LLM config as dict for chat completion calls.
+
+ Fallback chain:
+ 1. self.query_llm_config (explicit query config on strategy)
+ 2. self.config._query_llm_config_dict (from AdaptiveConfig)
+ 3. self.llm_config (legacy: single config for both)
+ 4. None (caller uses hardcoded defaults)
+ """
+ # 1. Explicit query config on strategy instance
+ if self.query_llm_config is not None:
+ if isinstance(self.query_llm_config, dict):
+ return self.query_llm_config
+ return self.query_llm_config.to_dict()
+
+ # 2. From AdaptiveConfig
+ if hasattr(self, 'config') and self.config:
+ config_dict = self.config._query_llm_config_dict
+ if config_dict:
+ return config_dict
+
+ # 3. Legacy fallback: use embedding/shared llm_config for backward compat
+ if self.llm_config is not None:
+ if isinstance(self.llm_config, dict):
+ return self.llm_config
+ return self.llm_config.to_dict()
+
+ # 4. None β caller applies hardcoded defaults
+ return None
+
async def _get_embeddings(self, texts: List[str]) -> Any:
"""Get embeddings using configured method"""
from .utils import get_text_embeddings
@@ -712,27 +738,22 @@ async def map_query_semantic_space(self, query: str, n_synthetic: int = 10) -> A
Return as a JSON array of strings."""
- # Use the LLM for query generation
- # Convert LLMConfig to dict if needed
- llm_config_dict = None
- if self.llm_config:
- if isinstance(self.llm_config, dict):
- llm_config_dict = self.llm_config
- else:
- # Convert LLMConfig object to dict
- llm_config_dict = {
- 'provider': self.llm_config.provider,
- 'api_token': self.llm_config.api_token
- }
-
+ # Use a chat completion model for query generation
+ llm_config_dict = self._get_query_llm_config_dict()
+
provider = llm_config_dict.get('provider', 'openai/gpt-4o-mini') if llm_config_dict else 'openai/gpt-4o-mini'
api_token = llm_config_dict.get('api_token') if llm_config_dict else None
-
+ base_url = llm_config_dict.get('base_url') if llm_config_dict else None
+
response = perform_completion_with_backoff(
provider=provider,
prompt_with_variables=prompt,
api_token=api_token,
- json_response=True
+ json_response=True,
+ base_url=base_url,
+ base_delay=llm_config_dict.get('backoff_base_delay', 2) if llm_config_dict else 2,
+ max_attempts=llm_config_dict.get('backoff_max_attempts', 3) if llm_config_dict else 3,
+ exponential_factor=llm_config_dict.get('backoff_exponential_factor', 2) if llm_config_dict else 2,
)
variations = json.loads(response.choices[0].message.content)
@@ -1298,7 +1319,8 @@ def _create_strategy(self, strategy_name: str) -> CrawlStrategy:
elif strategy_name == "embedding":
strategy = EmbeddingStrategy(
embedding_model=self.config.embedding_model,
- llm_config=self.config.embedding_llm_config
+ llm_config=self.config.embedding_llm_config,
+ query_llm_config=self.config.query_llm_config,
)
strategy.config = self.config # Pass config to strategy
return strategy
@@ -1353,11 +1375,10 @@ async def digest(self,
if isinstance(result.links, dict):
# Extract internal and external links from dict
internal_links = [Link(**link) for link in result.links.get('internal', [])]
- external_links = [Link(**link) for link in result.links.get('external', [])]
- self.state.pending_links.extend(internal_links + external_links)
+ self.state.pending_links.extend(internal_links)
else:
# Handle Links object
- self.state.pending_links.extend(result.links.internal + result.links.external)
+ self.state.pending_links.extend(result.links.internal)
# Update state
await self.strategy.update_state(self.state, [result])
@@ -1407,11 +1428,10 @@ async def digest(self,
if isinstance(result.links, dict):
# Extract internal and external links from dict
internal_links = [Link(**link_data) for link_data in result.links.get('internal', [])]
- external_links = [Link(**link_data) for link_data in result.links.get('external', [])]
- new_links = internal_links + external_links
+ new_links = internal_links
else:
# Handle Links object
- new_links = result.links.internal + result.links.external
+ new_links = result.links.internal
# Add new links to pending
for new_link in new_links:
@@ -1459,7 +1479,7 @@ async def _crawl_with_preview(self, url: str, query: str) -> Optional[CrawlResul
include_external=False,
query=query, # For BM25 scoring
concurrency=5,
- timeout=5,
+ timeout=self.config.link_preview_timeout,
max_links=50, # Reasonable limit
verbose=False
),
@@ -1822,7 +1842,7 @@ def _crawl_result_to_export_dict(self, result) -> Dict[str, Any]:
return export_dict
- def import_knowledge_base(self, filepath: Union[str, Path], format: str = "jsonl") -> None:
+ async def import_knowledge_base(self, filepath: Union[str, Path], format: str = "jsonl") -> None:
"""Import a knowledge base from a file
Args:
@@ -1851,7 +1871,7 @@ def import_knowledge_base(self, filepath: Union[str, Path], format: str = "jsonl
self.state.knowledge_base.extend(imported_results)
# Update state with imported data
- asyncio.run(self.strategy.update_state(self.state, imported_results))
+ await self.strategy.update_state(self.state, imported_results)
print(f"Imported {len(imported_results)} documents from {filepath}")
else:
diff --git a/crawl4ai/antibot_detector.py b/crawl4ai/antibot_detector.py
new file mode 100644
index 000000000..228c1b258
--- /dev/null
+++ b/crawl4ai/antibot_detector.py
@@ -0,0 +1,281 @@
+"""
+Anti-bot detection heuristics for crawl results.
+
+Examines HTTP status codes and HTML content patterns to determine
+if a crawl was blocked by anti-bot protection.
+
+Detection philosophy: false positives are cheap (the fallback mechanism
+rescues them), false negatives are catastrophic (user gets garbage).
+Err on the side of detection.
+
+Detection is layered:
+- HTTP 403/503 with HTML content β always blocked (these are never desired content)
+- Tier 1 patterns (structural markers) trigger on any page size
+- Tier 2 patterns (generic terms) trigger on short pages or any error status
+- Tier 3 structural integrity catches silent blocks and empty shells
+"""
+
+import re
+from typing import Optional, Tuple
+
+
+# ---------------------------------------------------------------------------
+# Tier 1: High-confidence structural markers (single signal sufficient)
+# These are unique to block pages and virtually never appear in real content.
+# ---------------------------------------------------------------------------
+_TIER1_PATTERNS = [
+ # Akamai β full reference pattern: Reference #18.2d351ab8.1557333295.a4e16ab
+ (re.compile(r"Reference\s*#\s*[\d]+\.[0-9a-f]+\.\d+\.[0-9a-f]+", re.IGNORECASE),
+ "Akamai block (Reference #)"),
+ # Akamai β "Pardon Our Interruption" challenge page
+ (re.compile(r"Pardon\s+Our\s+Interruption", re.IGNORECASE),
+ "Akamai challenge (Pardon Our Interruption)"),
+ # Cloudflare β challenge form with anti-bot token
+ (re.compile(r'challenge-form.*?__cf_chl_f_tk=', re.IGNORECASE | re.DOTALL),
+ "Cloudflare challenge form"),
+ # Cloudflare β error code spans (1020 Access Denied, 1010, 1012, 1015)
+ (re.compile(r'\d{4}', re.IGNORECASE),
+ "Cloudflare firewall block"),
+ # Cloudflare β IUAM challenge script
+ (re.compile(r'/cdn-cgi/challenge-platform/\S+orchestrate', re.IGNORECASE),
+ "Cloudflare JS challenge"),
+ # PerimeterX / HUMAN β block page with app ID assignment (not prose mentions)
+ (re.compile(r"window\._pxAppId\s*=", re.IGNORECASE),
+ "PerimeterX block"),
+ # PerimeterX β captcha CDN
+ (re.compile(r"captcha\.px-cdn\.net", re.IGNORECASE),
+ "PerimeterX captcha"),
+ # DataDome β captcha delivery domain (structural, not the word "datadome")
+ (re.compile(r"captcha-delivery\.com", re.IGNORECASE),
+ "DataDome captcha"),
+ # Imperva/Incapsula β resource iframe
+ (re.compile(r"_Incapsula_Resource", re.IGNORECASE),
+ "Imperva/Incapsula block"),
+ # Imperva/Incapsula β incident ID
+ (re.compile(r"Incapsula\s+incident\s+ID", re.IGNORECASE),
+ "Imperva/Incapsula incident"),
+ # Sucuri firewall
+ (re.compile(r"Sucuri\s+WebSite\s+Firewall", re.IGNORECASE),
+ "Sucuri firewall block"),
+ # Kasada
+ (re.compile(r"KPSDK\.scriptStart\s*=\s*KPSDK\.now\(\)", re.IGNORECASE),
+ "Kasada challenge"),
+ # Network security block β Reddit and other platforms serve large SPA shells
+ # with this message buried under 100KB+ of CSS/JS
+ (re.compile(r"blocked\s+by\s+network\s+security", re.IGNORECASE),
+ "Network security block"),
+]
+
+# ---------------------------------------------------------------------------
+# Tier 2: Medium-confidence patterns β only match on SHORT pages (< 10KB)
+# These terms appear in real content (articles, login forms, security blogs)
+# so we require the page to be small to avoid false positives.
+# ---------------------------------------------------------------------------
+_TIER2_PATTERNS = [
+ # Akamai / generic β "Access Denied" (extremely common on legit 403s too)
+ (re.compile(r"Access\s+Denied", re.IGNORECASE),
+ "Access Denied on short page"),
+ # Cloudflare β "Just a moment" / "Checking your browser"
+ (re.compile(r"Checking\s+your\s+browser", re.IGNORECASE),
+ "Cloudflare browser check"),
+ (re.compile(r"\s*Just\s+a\s+moment", re.IGNORECASE),
+ "Cloudflare interstitial"),
+ # CAPTCHA on a block page (not a login form β login forms are big pages)
+ (re.compile(r'class=["\']g-recaptcha["\']', re.IGNORECASE),
+ "reCAPTCHA on block page"),
+ (re.compile(r'class=["\']h-captcha["\']', re.IGNORECASE),
+ "hCaptcha on block page"),
+ # PerimeterX block page title
+ (re.compile(r"Access\s+to\s+This\s+Page\s+Has\s+Been\s+Blocked", re.IGNORECASE),
+ "PerimeterX block page"),
+ # Generic block phrases (only on short pages to avoid matching articles)
+ (re.compile(r"blocked\s+by\s+security", re.IGNORECASE),
+ "Blocked by security"),
+ (re.compile(r"Request\s+unsuccessful", re.IGNORECASE),
+ "Request unsuccessful (Imperva)"),
+]
+
+_TIER2_MAX_SIZE = 10000 # Only check tier 2 patterns on pages under 10KB
+
+# ---------------------------------------------------------------------------
+# Tier 3: Structural integrity β catches silent blocks, anti-bot redirects,
+# incomplete renders that pass pattern detection but are structurally broken
+# ---------------------------------------------------------------------------
+_STRUCTURAL_MAX_SIZE = 50000 # Only check pages under 50KB
+_CONTENT_ELEMENTS_RE = re.compile(
+ r'<(?:p|h[1-6]|article|section|li|td|a|pre)\b', re.IGNORECASE
+)
+_SCRIPT_TAG_RE = re.compile(r'<script\b', re.IGNORECASE)
+_STYLE_TAG_RE = re.compile(r'<style\b[\s\S]*?</style>', re.IGNORECASE)
+_SCRIPT_BLOCK_RE = re.compile(r'<script\b[\s\S]*?</script>', re.IGNORECASE)
+_TAG_RE = re.compile(r'<[^>]+>')
+_BODY_RE = re.compile(r'<body\b', re.IGNORECASE)
+
+# ---------------------------------------------------------------------------
+# Thresholds
+# ---------------------------------------------------------------------------
+_BLOCK_PAGE_MAX_SIZE = 5000 # 403 + short page = likely block
+_EMPTY_CONTENT_THRESHOLD = 100 # 200 + near-empty = JS-blocked render
+
+
+def _looks_like_data(html: str) -> bool:
+ """Check if content looks like a JSON/XML API response (not an HTML block page)."""
+ stripped = html.strip()
+ if not stripped:
+ return False
+ # Raw JSON/XML (not wrapped in HTML)
+ if stripped[0] in ('{', '['):
+ return True
+ # Browser-rendered JSON: browsers wrap raw JSON in <html><body><pre>{...}</pre>
+ if stripped[:10].lower().startswith(('<html', '<!')):
+ if re.search(r'<body[^>]*>\s*<pre[^>]*>\s*[{\[]', stripped[:500], re.IGNORECASE):
+ return True
+ return False
+ # Other XML-like content
+ return stripped[0] == '<'
+
+
+def _structural_integrity_check(html: str) -> Tuple[bool, str]:
+ """
+ Tier 3: Structural integrity check for pages that pass pattern detection
+ but are structurally broken β incomplete renders, anti-bot redirects, empty shells.
+
+ Only applies to pages < 50KB that aren't JSON/XML.
+
+ Returns:
+ Tuple of (is_blocked, reason).
+ """
+ html_len = len(html)
+
+ # Skip large pages (unlikely to be block pages) and data responses
+ if html_len > _STRUCTURAL_MAX_SIZE or _looks_like_data(html):
+ return False, ""
+
+ signals = []
+
+ # Signal 1: No <body> tag β definitive structural failure
+ if not _BODY_RE.search(html):
+ return True, f"Structural: no <body> tag ({html_len} bytes)"
+
+ # Signal 2: Minimal visible text after stripping scripts/styles/tags
+ body_match = re.search(r'<body\b[^>]*>([\s\S]*)</body>', html, re.IGNORECASE)
+ body_content = body_match.group(1) if body_match else html
+ stripped = _SCRIPT_BLOCK_RE.sub('', body_content)
+ stripped = _STYLE_TAG_RE.sub('', stripped)
+ visible_text = _TAG_RE.sub('', stripped).strip()
+ visible_len = len(visible_text)
+ if visible_len < 50:
+ signals.append("minimal_text")
+
+ # Signal 3: No content elements (semantic HTML)
+ content_elements = len(_CONTENT_ELEMENTS_RE.findall(html))
+ if content_elements == 0:
+ signals.append("no_content_elements")
+
+ # Signal 4: Script-heavy shell β scripts present but no content
+ script_count = len(_SCRIPT_TAG_RE.findall(html))
+ if script_count > 0 and content_elements == 0 and visible_len < 100:
+ signals.append("script_heavy_shell")
+
+ # Scoring
+ signal_count = len(signals)
+ if signal_count >= 2:
+ return True, f"Structural: {', '.join(signals)} ({html_len} bytes, {visible_len} chars visible)"
+
+ if signal_count == 1 and html_len < 5000:
+ return True, f"Structural: {signals[0]} on small page ({html_len} bytes, {visible_len} chars visible)"
+
+ return False, ""
+
+
+def is_blocked(
+ status_code: Optional[int],
+ html: str,
+ error_message: Optional[str] = None,
+) -> Tuple[bool, str]:
+ """
+ Detect if a crawl result indicates anti-bot blocking.
+
+ Uses layered detection to maximize coverage while minimizing false positives:
+ - Tier 1 patterns (structural markers) trigger on any page size
+ - Tier 2 patterns (generic terms) only trigger on short pages (< 10KB)
+ - Tier 3 structural integrity catches silent blocks and empty shells
+ - Status-code checks require corroborating content signals
+
+ Args:
+ status_code: HTTP status code from the response.
+ html: Raw HTML content from the response.
+ error_message: Error message from the crawl result, if any.
+
+ Returns:
+ Tuple of (is_blocked, reason). reason is empty string when not blocked.
+ """
+ html = html or ""
+ html_len = len(html)
+
+ # --- HTTP 429 is always rate limiting ---
+ if status_code == 429:
+ return True, "HTTP 429 Too Many Requests"
+
+ # --- Check for tier 1 patterns (high confidence, any page size) ---
+ # First check the raw start of the page (fast path for small pages).
+ # Then, for large pages, also check a stripped version (scripts/styles
+ # removed) because modern block pages bury text under 100KB+ of CSS/JS.
+ snippet = html[:15000]
+ if snippet:
+ for pattern, reason in _TIER1_PATTERNS:
+ if pattern.search(snippet):
+ return True, reason
+
+ # Large-page deep scan: strip scripts/styles and re-check tier 1
+ if html_len > 15000:
+ _stripped_for_t1 = _SCRIPT_BLOCK_RE.sub('', html[:500000])
+ _stripped_for_t1 = _STYLE_TAG_RE.sub('', _stripped_for_t1)
+ _deep_snippet = _stripped_for_t1[:30000]
+ for pattern, reason in _TIER1_PATTERNS:
+ if pattern.search(_deep_snippet):
+ return True, reason
+
+ # --- HTTP 403/503 β always blocked for non-data HTML responses ---
+ # Rationale: 403/503 are never the content the user wants. Modern block pages
+ # (Reddit, LinkedIn, etc.) serve full SPA shells that exceed 100KB, so
+ # size-based filtering misses them. Even for a legitimate auth error, the
+ # fallback (Web Unlocker) will also get 403 and we correctly report failure.
+ # False positives are cheap β the fallback mechanism rescues them.
+ if status_code in (403, 503) and not _looks_like_data(html):
+ if html_len < _EMPTY_CONTENT_THRESHOLD:
+ return True, f"HTTP {status_code} with near-empty response ({html_len} bytes)"
+ # For large pages, strip scripts/styles to find block text in the
+ # actual content (Reddit hides it under 180KB of inline CSS).
+ # Check tier 2 patterns regardless of page size.
+ if html_len > _TIER2_MAX_SIZE:
+ _stripped = _SCRIPT_BLOCK_RE.sub('', html[:500000])
+ _stripped = _STYLE_TAG_RE.sub('', _stripped)
+ _check_snippet = _stripped[:30000]
+ else:
+ _check_snippet = snippet
+ for pattern, reason in _TIER2_PATTERNS:
+ if pattern.search(_check_snippet):
+ return True, f"{reason} (HTTP {status_code}, {html_len} bytes)"
+ # Even without a pattern match, a non-data 403/503 HTML page is
+ # almost certainly a block. Flag it so the fallback gets a chance.
+ return True, f"HTTP {status_code} with HTML content ({html_len} bytes)"
+
+ # --- Tier 2 patterns on other 4xx/5xx + short page ---
+ if status_code and status_code >= 400 and html_len < _TIER2_MAX_SIZE:
+ for pattern, reason in _TIER2_PATTERNS:
+ if pattern.search(snippet):
+ return True, f"{reason} (HTTP {status_code}, {html_len} bytes)"
+
+ # --- HTTP 200 + near-empty content (JS-rendered empty page) ---
+ if status_code == 200:
+ stripped = html.strip()
+ if len(stripped) < _EMPTY_CONTENT_THRESHOLD and not _looks_like_data(html):
+ return True, f"Near-empty content ({len(stripped)} bytes) with HTTP 200"
+
+ # --- Tier 3: Structural integrity (catches silent blocks, redirects, incomplete renders) ---
+ _blocked, _reason = _structural_integrity_check(html)
+ if _blocked:
+ return True, _reason
+
+ return False, ""
diff --git a/crawl4ai/async_configs.py b/crawl4ai/async_configs.py
index aa5745fb0..d7171559b 100644
--- a/crawl4ai/async_configs.py
+++ b/crawl4ai/async_configs.py
@@ -1,3 +1,5 @@
+import copy
+import functools
import importlib
import os
import warnings
@@ -28,19 +30,131 @@
from .proxy_strategy import ProxyRotationStrategy
import inspect
-from typing import Any, Callable, Dict, List, Optional, Union
+from typing import Any, Awaitable, Callable, Dict, List, Optional, Union
from enum import Enum
# Type alias for URL matching
UrlMatcher = Union[str, Callable[[str], bool], List[Union[str, Callable[[str], bool]]]]
+def _with_defaults(cls):
+ """Class decorator: adds set_defaults/get_defaults/reset_defaults classmethods.
+
+ After decorating, every new instance resolves parameters as:
+ explicit arg > class-level user defaults > hardcoded default
+
+ Usage::
+
+ BrowserConfig.set_defaults(headless=False, viewport_width=1920)
+ cfg = BrowserConfig() # headless=False, viewport_width=1920
+ cfg = BrowserConfig(headless=True) # explicit wins β headless=True
+ """
+ original_init = cls.__init__
+ sig = inspect.signature(original_init)
+ param_names = [p for p in sig.parameters if p != "self"]
+ valid_params = frozenset(param_names)
+
+ @functools.wraps(original_init)
+ def wrapped_init(self, *args, **kwargs):
+ user_defaults = type(self)._user_defaults
+ if user_defaults:
+ # Determine which params the caller passed explicitly
+ explicit = set(kwargs.keys())
+ for i in range(len(args)):
+ if i < len(param_names):
+ explicit.add(param_names[i])
+ # Inject user defaults for non-explicit params
+ for key, value in user_defaults.items():
+ if key not in explicit:
+ kwargs[key] = copy.deepcopy(value)
+ original_init(self, *args, **kwargs)
+
+ cls.__init__ = wrapped_init
+ cls._user_defaults = {}
+
+ @classmethod
+ def set_defaults(klass, **kwargs):
+ """Set class-level default overrides for new instances.
+
+ Args:
+ **kwargs: Parameter names and their default values.
+
+ Raises:
+ ValueError: If any key is not a valid ``__init__`` parameter.
+ """
+ invalid = set(kwargs) - valid_params
+ if invalid:
+ raise ValueError(
+ f"Invalid parameter(s) for {klass.__name__}: {invalid}"
+ )
+ for k, v in kwargs.items():
+ klass._user_defaults[k] = copy.deepcopy(v)
+
+ @classmethod
+ def get_defaults(klass):
+ """Return a deep copy of the current class-level defaults."""
+ return copy.deepcopy(klass._user_defaults)
+
+ @classmethod
+ def reset_defaults(klass, *names):
+ """Clear class-level defaults.
+
+ With no arguments, removes all overrides.
+ With arguments, removes only the named overrides.
+ """
+ if names:
+ for n in names:
+ klass._user_defaults.pop(n, None)
+ else:
+ klass._user_defaults.clear()
+
+ cls.set_defaults = set_defaults
+ cls.get_defaults = get_defaults
+ cls.reset_defaults = reset_defaults
+ return cls
+
+
class MatchMode(Enum):
OR = "or"
AND = "and"
# from .proxy_strategy import ProxyConfig
+# Allowlist of types that can be deserialized via from_serializable_dict().
+# This prevents arbitrary class instantiation from untrusted input (e.g. API requests).
+ALLOWED_DESERIALIZE_TYPES = {
+ # Config classes
+ "BrowserConfig", "CrawlerRunConfig", "HTTPCrawlerConfig",
+ "LLMConfig", "ProxyConfig", "GeolocationConfig",
+ "SeedingConfig", "VirtualScrollConfig", "LinkPreviewConfig",
+ # Extraction strategies
+ "JsonCssExtractionStrategy", "JsonXPathExtractionStrategy",
+ "JsonLxmlExtractionStrategy", "LLMExtractionStrategy",
+ "CosineStrategy", "RegexExtractionStrategy",
+ # Markdown / content
+ "DefaultMarkdownGenerator",
+ "PruningContentFilter", "BM25ContentFilter", "LLMContentFilter",
+ # Scraping
+ "LXMLWebScrapingStrategy", "PDFContentScrapingStrategy",
+ # Chunking
+ "RegexChunking",
+ # Deep crawl
+ "BFSDeepCrawlStrategy", "DFSDeepCrawlStrategy", "BestFirstCrawlingStrategy",
+ # Filters & scorers
+ "FilterChain", "URLPatternFilter", "DomainFilter",
+ "ContentTypeFilter", "URLFilter", "SEOFilter", "ContentRelevanceFilter",
+ "KeywordRelevanceScorer", "URLScorer", "CompositeScorer",
+ "DomainAuthorityScorer", "FreshnessScorer", "PathDepthScorer",
+ # Enums
+ "CacheMode", "MatchMode", "DisplayMode",
+ # Dispatchers
+ "MemoryAdaptiveDispatcher", "SemaphoreDispatcher",
+ # Table extraction
+ "DefaultTableExtraction", "NoTableExtraction",
+ # Proxy
+ "RoundRobinProxyStrategy",
+}
+
def to_serializable_dict(obj: Any, ignore_default_value : bool = False):
"""
@@ -128,21 +242,37 @@ def from_serializable_dict(data: Any) -> Any:
if isinstance(data, (str, int, float, bool)):
return data
- # Handle typed data
- if isinstance(data, dict) and "type" in data:
+ # Handle typed data.
+ # Only enter the typed-object path for dicts that match the shapes produced
+ # by to_serializable_dict(): {"type": "<ClassName>", "params": {...}} or
+ # {"type": "dict", "value": {...}}. Plain business dicts that happen to
+ # carry a "type" key (e.g. JSON-Schema fragments, JsonCss field specs like
+ # {"type": "text", "name": "..."}) have neither "params" nor "value" and
+ # must fall through to the raw-dict path below so they are passed as data.
+ if (
+ isinstance(data, dict)
+ and "type" in data
+ and ("params" in data or (data["type"] == "dict" and "value" in data))
+ ):
# Handle plain dictionaries
if data["type"] == "dict" and "value" in data:
return {k: from_serializable_dict(v) for k, v in data["value"].items()}
+ # Security: only allow known-safe types to be deserialized
+ type_name = data["type"]
+ if type_name not in ALLOWED_DESERIALIZE_TYPES:
+ raise ValueError(
+ f"Deserialization of type '{type_name}' is not allowed. "
+ f"Only allowlisted configuration and strategy types can be deserialized."
+ )
+
cls = None
- # If you are receiving an error while trying to convert a dict to an object:
- # Either add a module to `modules_paths` list, or add the `data["type"]` to the crawl4ai __init__.py file
module_paths = ["crawl4ai"]
for module_path in module_paths:
try:
mod = importlib.import_module(module_path)
- if hasattr(mod, data["type"]):
- cls = getattr(mod, data["type"])
+ if hasattr(mod, type_name):
+ cls = getattr(mod, type_name)
break
except (ImportError, AttributeError):
continue
@@ -227,6 +357,8 @@ def clone(self, **kwargs) -> "GeolocationConfig":
return GeolocationConfig.from_dict(config_dict)
class ProxyConfig:
+ DIRECT = "direct" # Sentinel: use in proxy_config list to mean "no proxy"
+
def __init__(
self,
server: str,
@@ -235,7 +367,7 @@ def __init__(
ip: Optional[str] = None,
):
"""Configuration class for a single proxy.
-
+
Args:
server: Proxy server URL (e.g., "http://127.0.0.1:8080")
username: Optional username for proxy authentication
@@ -245,7 +377,7 @@ def __init__(
self.server = server
self.username = username
self.password = password
-
+
# Extract IP from server if not explicitly provided
self.ip = ip or self._extract_ip_from_server()
@@ -305,7 +437,7 @@ def from_dict(proxy_dict: Dict) -> "ProxyConfig":
server=proxy_dict.get("server"),
username=proxy_dict.get("username"),
password=proxy_dict.get("password"),
- ip=proxy_dict.get("ip")
+ ip=proxy_dict.get("ip"),
)
@staticmethod
@@ -335,7 +467,7 @@ def to_dict(self) -> Dict:
"server": self.server,
"username": self.username,
"password": self.password,
- "ip": self.ip
+ "ip": self.ip,
}
def clone(self, **kwargs) -> "ProxyConfig":
@@ -351,6 +483,7 @@ def clone(self, **kwargs) -> "ProxyConfig":
config_dict.update(kwargs)
return ProxyConfig.from_dict(config_dict)
+@_with_defaults
class BrowserConfig:
"""
Configuration class for setting up a browser instance and its context in AsyncPlaywrightCrawlerStrategy.
@@ -384,6 +517,15 @@ class BrowserConfig:
the local Playwright client resources. Useful for cloud/server scenarios
where you don't own the remote browser but need to prevent memory leaks
from accumulated Playwright instances. Default: False.
+ cdp_close_delay (float): Seconds to wait after disconnecting a CDP WebSocket before stopping the
+ Playwright subprocess. Gives the connection time to fully release. Set to
+ 0 to skip the delay entirely. Only applies when cdp_cleanup_on_close=True.
+ Default: 1.0.
+ cache_cdp_connection (bool): When True and using cdp_url, the Playwright subprocess and CDP WebSocket
+ are cached at the class level and shared across multiple BrowserManager
+ instances connecting to the same cdp_url. Reference-counted; the connection
+ is only closed when the last user releases it. Eliminates the overhead of
+ repeated Playwright/CDP setup and teardown. Default: False.
create_isolated_context (bool): When True and using cdp_url, forces creation of a new browser context
instead of reusing the default context. Essential for concurrent crawls
on the same browser to prevent navigation conflicts. Default: False.
@@ -404,6 +546,13 @@ class BrowserConfig:
viewport_height (int): Default viewport height for pages. Default: 600.
viewport (dict): Default viewport dimensions for pages. If set, overrides viewport_width and viewport_height.
Default: None.
+ device_scale_factor (float): The device pixel ratio used for rendering pages. Controls how many
+ physical pixels map to one CSS pixel, allowing simulation of HiDPI
+ or Retina displays. For example, a viewport of 1920x1080 with a
+ device_scale_factor of 2.0 produces screenshots at 3840x2160 resolution.
+ Increasing this value improves screenshot quality but may increase
+ memory usage and rendering time.
+ Default: 1.0.
verbose (bool): Enable verbose logging.
Default: True.
accept_downloads (bool): Whether to allow file downloads. If True, requires a downloads_path.
@@ -432,6 +581,19 @@ class BrowserConfig:
Default: [].
enable_stealth (bool): If True, applies playwright-stealth to bypass basic bot detection.
Cannot be used with use_undetected browser mode. Default: False.
+ memory_saving_mode (bool): If True, adds aggressive cache discard and V8 heap cap flags
+ to reduce Chromium memory growth. Recommended for high-volume
+ crawling (1000+ pages). May slightly reduce performance due to
+ cache eviction. Default: False.
+ max_pages_before_recycle (int): Number of pages to crawl before recycling the browser
+ process to reclaim leaked memory. 0 = disabled.
+ Recommended: 500-1000 for long-running crawlers.
+ Default: 0.
+ avoid_ads (bool): If True, blocks ad-related and tracker network requests at the
+ browser context level using a curated blocklist of top ad/tracker
+ domains. Default: False.
+ avoid_css (bool): If True, blocks loading of CSS files (css, less, scss, sass) to
+ reduce resource usage and speed up crawling. Default: False.
"""
def __init__(
@@ -444,6 +606,8 @@ def __init__(
browser_context_id: str = None,
target_id: str = None,
cdp_cleanup_on_close: bool = False,
+ cdp_close_delay: float = 1.0,
+ cache_cdp_connection: bool = False,
create_isolated_context: bool = False,
use_persistent_context: bool = False,
user_data_dir: str = None,
@@ -454,6 +618,7 @@ def __init__(
viewport_width: int = 1080,
viewport_height: int = 600,
viewport: dict = None,
+ device_scale_factor: float = 1.0,
accept_downloads: bool = False,
downloads_path: str = None,
storage_state: Union[str, dict, None] = None,
@@ -477,7 +642,11 @@ def __init__(
debugging_port: int = 9222,
host: str = "localhost",
enable_stealth: bool = False,
+ avoid_ads: bool = False,
+ avoid_css: bool = False,
init_scripts: List[str] = None,
+ memory_saving_mode: bool = False,
+ max_pages_before_recycle: int = 0,
):
self.browser_type = browser_type
@@ -488,6 +657,8 @@ def __init__(
self.browser_context_id = browser_context_id
self.target_id = target_id
self.cdp_cleanup_on_close = cdp_cleanup_on_close
+ self.cdp_close_delay = cdp_close_delay
+ self.cache_cdp_connection = cache_cdp_connection
self.create_isolated_context = create_isolated_context
self.use_persistent_context = use_persistent_context
self.user_data_dir = user_data_dir
@@ -519,6 +690,7 @@ def __init__(
if self.viewport is not None:
self.viewport_width = self.viewport.get("width", 1080)
self.viewport_height = self.viewport.get("height", 600)
+ self.device_scale_factor = device_scale_factor
self.accept_downloads = accept_downloads
self.downloads_path = downloads_path
self.storage_state = storage_state
@@ -537,7 +709,11 @@ def __init__(
self.debugging_port = debugging_port
self.host = host
self.enable_stealth = enable_stealth
+ self.avoid_ads = avoid_ads
+ self.avoid_css = avoid_css
self.init_scripts = init_scripts if init_scripts is not None else []
+ self.memory_saving_mode = memory_saving_mode
+ self.max_pages_before_recycle = max_pages_before_recycle
fa_user_agenr_generator = ValidUAGenerator()
if self.user_agent_mode == "random":
@@ -579,46 +755,16 @@ def __init__(
@staticmethod
def from_kwargs(kwargs: dict) -> "BrowserConfig":
- return BrowserConfig(
- browser_type=kwargs.get("browser_type", "chromium"),
- headless=kwargs.get("headless", True),
- browser_mode=kwargs.get("browser_mode", "dedicated"),
- use_managed_browser=kwargs.get("use_managed_browser", False),
- cdp_url=kwargs.get("cdp_url"),
- browser_context_id=kwargs.get("browser_context_id"),
- target_id=kwargs.get("target_id"),
- cdp_cleanup_on_close=kwargs.get("cdp_cleanup_on_close", False),
- create_isolated_context=kwargs.get("create_isolated_context", False),
- use_persistent_context=kwargs.get("use_persistent_context", False),
- user_data_dir=kwargs.get("user_data_dir"),
- chrome_channel=kwargs.get("chrome_channel", "chromium"),
- channel=kwargs.get("channel", "chromium"),
- proxy=kwargs.get("proxy"),
- proxy_config=kwargs.get("proxy_config", None),
- viewport_width=kwargs.get("viewport_width", 1080),
- viewport_height=kwargs.get("viewport_height", 600),
- accept_downloads=kwargs.get("accept_downloads", False),
- downloads_path=kwargs.get("downloads_path"),
- storage_state=kwargs.get("storage_state"),
- ignore_https_errors=kwargs.get("ignore_https_errors", True),
- java_script_enabled=kwargs.get("java_script_enabled", True),
- cookies=kwargs.get("cookies", []),
- headers=kwargs.get("headers", {}),
- user_agent=kwargs.get(
- "user_agent",
- "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
- "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
- ),
- user_agent_mode=kwargs.get("user_agent_mode"),
- user_agent_generator_config=kwargs.get("user_agent_generator_config"),
- text_mode=kwargs.get("text_mode", False),
- light_mode=kwargs.get("light_mode", False),
- extra_args=kwargs.get("extra_args", []),
- debugging_port=kwargs.get("debugging_port", 9222),
- host=kwargs.get("host", "localhost"),
- enable_stealth=kwargs.get("enable_stealth", False),
- init_scripts=kwargs.get("init_scripts", []),
- )
+ # Auto-deserialize any dict values that use the {"type": ..., "params": ...}
+ # serialization format (e.g. from JSON API requests or dump()/load() roundtrips).
+ kwargs = {
+ k: from_serializable_dict(v) if isinstance(v, dict) and "type" in v else v
+ for k, v in kwargs.items()
+ }
+ # Only pass keys present in kwargs so that __init__ defaults (and
+ # set_defaults() overrides) are respected for missing keys.
+ valid = inspect.signature(BrowserConfig.__init__).parameters.keys() - {"self"}
+ return BrowserConfig(**{k: v for k, v in kwargs.items() if k in valid})
def to_dict(self):
result = {
@@ -636,9 +782,10 @@ def to_dict(self):
"chrome_channel": self.chrome_channel,
"channel": self.channel,
"proxy": self.proxy,
- "proxy_config": self.proxy_config.to_dict() if self.proxy_config else None,
+ "proxy_config": self.proxy_config.to_dict() if hasattr(self.proxy_config, 'to_dict') else self.proxy_config,
"viewport_width": self.viewport_width,
"viewport_height": self.viewport_height,
+ "device_scale_factor": self.device_scale_factor,
"accept_downloads": self.accept_downloads,
"downloads_path": self.downloads_path,
"storage_state": self.storage_state,
@@ -657,7 +804,11 @@ def to_dict(self):
"debugging_port": self.debugging_port,
"host": self.host,
"enable_stealth": self.enable_stealth,
+ "avoid_ads": self.avoid_ads,
+ "avoid_css": self.avoid_css,
"init_scripts": self.init_scripts,
+ "memory_saving_mode": self.memory_saving_mode,
+ "max_pages_before_recycle": self.max_pages_before_recycle,
}
@@ -978,6 +1129,7 @@ def load(data: dict) -> "HTTPCrawlerConfig":
return config
return HTTPCrawlerConfig.from_kwargs(config)
+@_with_defaults
class CrawlerRunConfig():
"""
@@ -1104,7 +1256,11 @@ class CrawlerRunConfig():
Default: 5.
# Page Interaction Parameters
- js_code (str or list of str or None): JavaScript code/snippets to run on the page.
+ js_code (str or list of str or None): JavaScript code/snippets to run on the page
+ after wait_for and delay_before_return_html.
+ Default: None.
+ js_code_before_wait (str or list of str or None): JavaScript to run BEFORE wait_for.
+ Use for triggering loading that wait_for then checks.
Default: None.
js_only (bool): If True, indicates subsequent calls are JS-driven updates, not full page loads.
Default: False.
@@ -1118,8 +1274,16 @@ class CrawlerRunConfig():
If None, scrolls until the entire page is loaded. Default: None.
process_iframes (bool): If True, attempts to process and inline iframe content.
Default: False.
+ flatten_shadow_dom (bool): If True, flatten shadow DOM content into the light DOM
+ before HTML capture so page.content() includes it.
+ Also injects an init script to force-open closed shadow roots.
+ Default: False.
remove_overlay_elements (bool): If True, remove overlays/popups before extracting HTML.
Default: False.
+ remove_consent_popups (bool): If True, remove GDPR/cookie consent popups (IAB TCF/CMP)
+ before extracting HTML. Targets known CMP providers like
+ OneTrust, Cookiebot, TrustArc, Quantcast, Didomi, etc.
+ Default: False.
simulate_user (bool): If True, simulate user interactions (mouse moves, clicks) for anti-bot measures.
Default: False.
override_navigator (bool): If True, overrides navigator properties for more human-like behavior.
@@ -1136,6 +1300,9 @@ class CrawlerRunConfig():
Default: None.
screenshot_height_threshold (int): Threshold for page height to decide screenshot strategy.
Default: SCREENSHOT_HEIGHT_TRESHOLD (from config, e.g. 20000).
+ force_viewport_screenshot (bool): If True, always take viewport-only screenshots regardless of page height.
+ When False, uses automatic decision (viewport for short pages, full-page for long pages).
+ Default: False.
pdf (bool): Whether to generate a PDF of the page.
Default: False.
image_description_min_word_threshold (int): Minimum words for image description extraction.
@@ -1237,7 +1404,7 @@ def __init__(
prettiify: bool = False,
parser_type: str = "lxml",
scraping_strategy: ContentScrapingStrategy = None,
- proxy_config: Union[ProxyConfig, dict, None] = None,
+ proxy_config: Union["ProxyConfig", List["ProxyConfig"], dict, str, None] = None,
proxy_rotation_strategy: Optional[ProxyRotationStrategy] = None,
# Sticky Proxy Session Parameters
proxy_session_id: Optional[str] = None,
@@ -1272,6 +1439,7 @@ def __init__(
semaphore_count: int = 5,
# Page Interaction Parameters
js_code: Union[str, List[str]] = None,
+ js_code_before_wait: Union[str, List[str]] = None,
c4a_script: Union[str, List[str]] = None,
js_only: bool = False,
ignore_body_visibility: bool = True,
@@ -1279,7 +1447,9 @@ def __init__(
scroll_delay: float = 0.2,
max_scroll_steps: Optional[int] = None,
process_iframes: bool = False,
+ flatten_shadow_dom: bool = False,
remove_overlay_elements: bool = False,
+ remove_consent_popups: bool = False,
simulate_user: bool = False,
override_navigator: bool = False,
magic: bool = False,
@@ -1288,6 +1458,7 @@ def __init__(
screenshot: bool = False,
screenshot_wait_for: float = None,
screenshot_height_threshold: int = SCREENSHOT_HEIGHT_TRESHOLD,
+ force_viewport_screenshot: bool = False,
pdf: bool = False,
capture_mhtml: bool = False,
image_description_min_word_threshold: int = IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD,
@@ -1332,6 +1503,9 @@ def __init__(
match_mode: MatchMode = MatchMode.OR,
# Experimental Parameters
experimental: Dict[str, Any] = None,
+ # Anti-Bot Retry Parameters
+ max_retries: int = 0,
+ fallback_fetch_function: Optional[Callable[[str], Awaitable[str]]] = None,
):
# TODO: Planning to set properties dynamically based on the __init__ signature
self.url = url
@@ -1353,11 +1527,7 @@ def __init__(
self.prettiify = prettiify
self.parser_type = parser_type
self.scraping_strategy = scraping_strategy or LXMLWebScrapingStrategy()
- self.proxy_config = proxy_config
- if isinstance(proxy_config, dict):
- self.proxy_config = ProxyConfig.from_dict(proxy_config)
- if isinstance(proxy_config, str):
- self.proxy_config = ProxyConfig.from_string(proxy_config)
+ self.proxy_config = proxy_config # runs through property setter
self.proxy_rotation_strategy = proxy_rotation_strategy
@@ -1399,6 +1569,7 @@ def __init__(
# Page Interaction Parameters
self.js_code = js_code
+ self.js_code_before_wait = js_code_before_wait
self.c4a_script = c4a_script
self.js_only = js_only
self.ignore_body_visibility = ignore_body_visibility
@@ -1406,7 +1577,9 @@ def __init__(
self.scroll_delay = scroll_delay
self.max_scroll_steps = max_scroll_steps
self.process_iframes = process_iframes
+ self.flatten_shadow_dom = flatten_shadow_dom
self.remove_overlay_elements = remove_overlay_elements
+ self.remove_consent_popups = remove_consent_popups
self.simulate_user = simulate_user
self.override_navigator = override_navigator
self.magic = magic
@@ -1416,6 +1589,7 @@ def __init__(
self.screenshot = screenshot
self.screenshot_wait_for = screenshot_wait_for
self.screenshot_height_threshold = screenshot_height_threshold
+ self.force_viewport_screenshot = force_viewport_screenshot
self.pdf = pdf
self.capture_mhtml = capture_mhtml
self.image_description_min_word_threshold = image_description_min_word_threshold
@@ -1512,12 +1686,55 @@ def __init__(
# Experimental Parameters
self.experimental = experimental or {}
-
+
+ # Anti-Bot Retry Parameters
+ self.max_retries = max_retries
+ self.fallback_fetch_function = fallback_fetch_function
+
# Compile C4A scripts if provided
if self.c4a_script and not self.js_code:
self._compile_c4a_script()
+ @staticmethod
+ def _normalize_proxy_config(value):
+ """Normalize proxy_config to ProxyConfig, list of ProxyConfig/None, or None."""
+ if isinstance(value, list):
+ normalized = []
+ for p in value:
+ if p is None or p == "direct":
+ normalized.append(None)
+ elif isinstance(p, dict):
+ normalized.append(ProxyConfig.from_dict(p))
+ elif isinstance(p, str):
+ normalized.append(ProxyConfig.from_string(p))
+ else:
+ normalized.append(p)
+ return normalized
+ elif isinstance(value, dict):
+ return ProxyConfig.from_dict(value)
+ elif isinstance(value, str):
+ if value == "direct":
+ return None
+ return ProxyConfig.from_string(value)
+ return value # ProxyConfig or None
+
+ @property
+ def proxy_config(self):
+ return self._proxy_config
+
+ @proxy_config.setter
+ def proxy_config(self, value):
+ self._proxy_config = CrawlerRunConfig._normalize_proxy_config(value)
+
+ def _get_proxy_list(self) -> list:
+ """Normalize proxy_config to a list for the retry loop."""
+ if self.proxy_config is None:
+ return [None]
+ if isinstance(self.proxy_config, list):
+ return self.proxy_config if self.proxy_config else [None]
+ return [self.proxy_config]
+
def _compile_c4a_script(self):
"""Compile C4A script to JavaScript"""
try:
@@ -1631,122 +1848,17 @@ def __setattr__(self, name, value):
@staticmethod
def from_kwargs(kwargs: dict) -> "CrawlerRunConfig":
- return CrawlerRunConfig(
- # Content Processing Parameters
- word_count_threshold=kwargs.get("word_count_threshold", 200),
- extraction_strategy=kwargs.get("extraction_strategy"),
- chunking_strategy=kwargs.get("chunking_strategy", RegexChunking()),
- markdown_generator=kwargs.get("markdown_generator"),
- only_text=kwargs.get("only_text", False),
- css_selector=kwargs.get("css_selector"),
- target_elements=kwargs.get("target_elements", []),
- excluded_tags=kwargs.get("excluded_tags", []),
- excluded_selector=kwargs.get("excluded_selector", ""),
- keep_data_attributes=kwargs.get("keep_data_attributes", False),
- keep_attrs=kwargs.get("keep_attrs", []),
- remove_forms=kwargs.get("remove_forms", False),
- prettiify=kwargs.get("prettiify", False),
- parser_type=kwargs.get("parser_type", "lxml"),
- scraping_strategy=kwargs.get("scraping_strategy"),
- proxy_config=kwargs.get("proxy_config"),
- proxy_rotation_strategy=kwargs.get("proxy_rotation_strategy"),
- # Sticky Proxy Session Parameters
- proxy_session_id=kwargs.get("proxy_session_id"),
- proxy_session_ttl=kwargs.get("proxy_session_ttl"),
- proxy_session_auto_release=kwargs.get("proxy_session_auto_release", False),
- # Browser Location and Identity Parameters
- locale=kwargs.get("locale", None),
- timezone_id=kwargs.get("timezone_id", None),
- geolocation=kwargs.get("geolocation", None),
- # SSL Parameters
- fetch_ssl_certificate=kwargs.get("fetch_ssl_certificate", False),
- # Caching Parameters
- cache_mode=kwargs.get("cache_mode", CacheMode.BYPASS),
- session_id=kwargs.get("session_id"),
- bypass_cache=kwargs.get("bypass_cache", False),
- disable_cache=kwargs.get("disable_cache", False),
- no_cache_read=kwargs.get("no_cache_read", False),
- no_cache_write=kwargs.get("no_cache_write", False),
- shared_data=kwargs.get("shared_data", None),
- # Page Navigation and Timing Parameters
- wait_until=kwargs.get("wait_until", "domcontentloaded"),
- page_timeout=kwargs.get("page_timeout", 60000),
- wait_for=kwargs.get("wait_for"),
- wait_for_timeout=kwargs.get("wait_for_timeout"),
- wait_for_images=kwargs.get("wait_for_images", False),
- delay_before_return_html=kwargs.get("delay_before_return_html", 0.1),
- mean_delay=kwargs.get("mean_delay", 0.1),
- max_range=kwargs.get("max_range", 0.3),
- semaphore_count=kwargs.get("semaphore_count", 5),
- # Page Interaction Parameters
- js_code=kwargs.get("js_code"),
- js_only=kwargs.get("js_only", False),
- ignore_body_visibility=kwargs.get("ignore_body_visibility", True),
- scan_full_page=kwargs.get("scan_full_page", False),
- scroll_delay=kwargs.get("scroll_delay", 0.2),
- max_scroll_steps=kwargs.get("max_scroll_steps"),
- process_iframes=kwargs.get("process_iframes", False),
- remove_overlay_elements=kwargs.get("remove_overlay_elements", False),
- simulate_user=kwargs.get("simulate_user", False),
- override_navigator=kwargs.get("override_navigator", False),
- magic=kwargs.get("magic", False),
- adjust_viewport_to_content=kwargs.get("adjust_viewport_to_content", False),
- # Media Handling Parameters
- screenshot=kwargs.get("screenshot", False),
- screenshot_wait_for=kwargs.get("screenshot_wait_for"),
- screenshot_height_threshold=kwargs.get(
- "screenshot_height_threshold", SCREENSHOT_HEIGHT_TRESHOLD
- ),
- pdf=kwargs.get("pdf", False),
- capture_mhtml=kwargs.get("capture_mhtml", False),
- image_description_min_word_threshold=kwargs.get(
- "image_description_min_word_threshold",
- IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD,
- ),
- image_score_threshold=kwargs.get(
- "image_score_threshold", IMAGE_SCORE_THRESHOLD
- ),
- table_score_threshold=kwargs.get("table_score_threshold", 7),
- table_extraction=kwargs.get("table_extraction", None),
- exclude_all_images=kwargs.get("exclude_all_images", False),
- exclude_external_images=kwargs.get("exclude_external_images", False),
- # Link and Domain Handling Parameters
- exclude_social_media_domains=kwargs.get(
- "exclude_social_media_domains", SOCIAL_MEDIA_DOMAINS
- ),
- exclude_external_links=kwargs.get("exclude_external_links", False),
- exclude_social_media_links=kwargs.get("exclude_social_media_links", False),
- exclude_domains=kwargs.get("exclude_domains", []),
- exclude_internal_links=kwargs.get("exclude_internal_links", False),
- score_links=kwargs.get("score_links", False),
- preserve_https_for_internal_links=kwargs.get("preserve_https_for_internal_links", False),
- # Debugging and Logging Parameters
- verbose=kwargs.get("verbose", True),
- log_console=kwargs.get("log_console", False),
- # Network and Console Capturing Parameters
- capture_network_requests=kwargs.get("capture_network_requests", False),
- capture_console_messages=kwargs.get("capture_console_messages", False),
- # Connection Parameters
- method=kwargs.get("method", "GET"),
- stream=kwargs.get("stream", False),
- prefetch=kwargs.get("prefetch", False),
- process_in_browser=kwargs.get("process_in_browser", False),
- check_robots_txt=kwargs.get("check_robots_txt", False),
- user_agent=kwargs.get("user_agent"),
- user_agent_mode=kwargs.get("user_agent_mode"),
- user_agent_generator_config=kwargs.get("user_agent_generator_config", {}),
- # Deep Crawl Parameters
- deep_crawl_strategy=kwargs.get("deep_crawl_strategy"),
- # Link Extraction Parameters
- link_preview_config=kwargs.get("link_preview_config"),
- url=kwargs.get("url"),
- base_url=kwargs.get("base_url"),
- # URL Matching Parameters
- url_matcher=kwargs.get("url_matcher"),
- match_mode=kwargs.get("match_mode", MatchMode.OR),
- # Experimental Parameters
- experimental=kwargs.get("experimental"),
- )
+ # Auto-deserialize any dict values that use the {"type": ..., "params": ...}
+ # serialization format (e.g. from JSON API requests or dump()/load() roundtrips).
+ # This covers markdown_generator, extraction_strategy, content_filter, etc.
+ kwargs = {
+ k: from_serializable_dict(v) if isinstance(v, dict) and "type" in v else v
+ for k, v in kwargs.items()
+ }
+ # Only pass keys present in kwargs so that __init__ defaults (and
+ # set_defaults() overrides) are respected for missing keys.
+ valid = inspect.signature(CrawlerRunConfig.__init__).parameters.keys() - {"self"}
+ return CrawlerRunConfig(**{k: v for k, v in kwargs.items() if k in valid})
# Create a funciton returns dict of the object
def dump(self) -> dict:
@@ -1778,7 +1890,11 @@ def to_dict(self):
"prettiify": self.prettiify,
"parser_type": self.parser_type,
"scraping_strategy": self.scraping_strategy,
- "proxy_config": self.proxy_config,
+ "proxy_config": (
+ [p.to_dict() if hasattr(p, 'to_dict') else p for p in self.proxy_config]
+ if isinstance(self.proxy_config, list)
+ else (self.proxy_config.to_dict() if hasattr(self.proxy_config, 'to_dict') else self.proxy_config)
+ ),
"proxy_rotation_strategy": self.proxy_rotation_strategy,
"proxy_session_id": self.proxy_session_id,
"proxy_session_ttl": self.proxy_session_ttl,
@@ -1804,13 +1920,16 @@ def to_dict(self):
"max_range": self.max_range,
"semaphore_count": self.semaphore_count,
"js_code": self.js_code,
+ "js_code_before_wait": self.js_code_before_wait,
"js_only": self.js_only,
"ignore_body_visibility": self.ignore_body_visibility,
"scan_full_page": self.scan_full_page,
"scroll_delay": self.scroll_delay,
"max_scroll_steps": self.max_scroll_steps,
"process_iframes": self.process_iframes,
+ "flatten_shadow_dom": self.flatten_shadow_dom,
"remove_overlay_elements": self.remove_overlay_elements,
+ "remove_consent_popups": self.remove_consent_popups,
"simulate_user": self.simulate_user,
"override_navigator": self.override_navigator,
"magic": self.magic,
@@ -1851,6 +1970,7 @@ def to_dict(self):
"url_matcher": self.url_matcher,
"match_mode": self.match_mode,
"experimental": self.experimental,
+ "max_retries": self.max_retries,
}
def clone(self, **kwargs):
diff --git a/crawl4ai/async_crawler_strategy.py b/crawl4ai/async_crawler_strategy.py
index 121a38614..04434b583 100644
--- a/crawl4ai/async_crawler_strategy.py
+++ b/crawl4ai/async_crawler_strategy.py
@@ -12,6 +12,7 @@
from io import BytesIO
from PIL import Image, ImageDraw, ImageFont
import hashlib
+import random
import uuid
from .js_snippet import load_js_script
from .models import AsyncCrawlResponse
@@ -19,7 +20,7 @@
from .async_configs import BrowserConfig, CrawlerRunConfig, HTTPCrawlerConfig
from .async_logger import AsyncLogger
from .ssl_certificate import SSLCertificate
-from .user_agent_generator import ValidUAGenerator
+from .user_agent_generator import ValidUAGenerator, UAGen
from .browser_manager import BrowserManager
from .browser_adapter import BrowserAdapter, PlaywrightAdapter, UndetectedAdapter
@@ -87,7 +88,8 @@ def __init__(
"""
# Initialize browser config, either from provided object or kwargs
self.browser_config = browser_config or BrowserConfig.from_kwargs(kwargs)
- self.logger = logger
+ # Initialize with default logger if none provided to prevent NoneType errors
+ self.logger = logger if logger is not None else AsyncLogger(verbose=False)
# Initialize browser adapter
self.adapter = browser_adapter or PlaywrightAdapter()
@@ -138,8 +140,9 @@ async def close(self):
Close the browser and clean up resources.
"""
await self.browser_manager.close()
- # Explicitly reset the static Playwright instance
- BrowserManager._playwright_instance = None
+ # Explicitly reset the static Playwright instance (skip if using cached CDP)
+ if not self.browser_manager._using_cached_cdp:
+ BrowserManager._playwright_instance = None
async def kill_session(self, session_id: str):
"""
@@ -380,7 +383,11 @@ async def process_iframes(self, page):
() => {{
const iframe = document.getElementById('iframe-{i}');
const div = document.createElement('div');
- div.innerHTML = `{_iframe}`;
+ const parser = new DOMParser();
+ const doc = parser.parseFromString(`{_iframe}`, 'text/html');
+ while (doc.body.firstChild) {{
+ div.appendChild(doc.body.firstChild);
+ }}
div.className = '{class_name}';
iframe.replaceWith(div);
}}
@@ -463,6 +470,7 @@ async def crawl(
config.wait_for or
config.scan_full_page or
config.remove_overlay_elements or
+ config.remove_consent_popups or
config.simulate_user or
config.magic or
config.process_iframes or
@@ -495,6 +503,8 @@ async def crawl(
pdf_data=None,
mhtml_data=None,
get_delayed_content=None,
+ # For raw:/file:// URLs, use base_url if provided; don't fall back to the raw content
+ redirected_url=config.base_url,
)
else:
raise ValueError(
@@ -519,7 +529,8 @@ async def _crawl_web(
response_headers = {}
execution_result = None
status_code = None
- redirected_url = url
+ redirected_url = url
+ redirected_status_code = None
# Reset downloaded files list for new crawl
self._downloaded_files = []
@@ -528,126 +539,173 @@ async def _crawl_web(
captured_requests = []
captured_console = []
- # Handle user agent with magic mode
- user_agent_to_override = config.user_agent
- if user_agent_to_override:
- self.browser_config.user_agent = user_agent_to_override
- elif config.magic or config.user_agent_mode == "random":
- self.browser_config.user_agent = ValidUAGenerator().generate(
- **(config.user_agent_generator_config or {})
+ # Handle user agent with magic mode.
+ # For persistent contexts the UA is locked at browser launch time
+ # (launch_persistent_context bakes it into the protocol layer), so
+ # changing it here would only desync browser_config from reality.
+ # Users should set user_agent or user_agent_mode on BrowserConfig.
+ ua_changed = False
+ if not self.browser_config.use_persistent_context:
+ user_agent_to_override = config.user_agent
+ if user_agent_to_override:
+ self.browser_config.user_agent = user_agent_to_override
+ ua_changed = True
+ elif config.magic or config.user_agent_mode == "random":
+ self.browser_config.user_agent = ValidUAGenerator().generate(
+ **(config.user_agent_generator_config or {})
+ )
+ ua_changed = True
+
+ # Keep sec-ch-ua in sync whenever the UA changed
+ if ua_changed:
+ self.browser_config.browser_hint = UAGen.generate_client_hints(
+ self.browser_config.user_agent
)
+ self.browser_config.headers["sec-ch-ua"] = self.browser_config.browser_hint
# Get page for session
page, context = await self.browser_manager.get_page(crawlerRunConfig=config)
- # await page.goto(URL)
-
- # Add default cookie
- # await context.add_cookies(
- # [{"name": "cookiesEnabled", "value": "true", "url": url}]
- # )
-
- # Handle navigator overrides
- if config.override_navigator or config.simulate_user or config.magic:
- await context.add_init_script(load_js_script("navigator_overrider"))
-
- # Call hook after page creation
- await self.execute_hook("on_page_context_created", page, context=context, config=config)
+ # When reusing a session page, abort any pending loads from the
+ # previous navigation to prevent timeouts on the next goto().
+ if config.session_id:
+ try:
+ await page.evaluate("window.stop()")
+ except Exception:
+ pass
- # Network Request Capturing
- if config.capture_network_requests:
- async def handle_request_capture(request):
- try:
- post_data_str = None
+ try:
+ # Push updated UA + sec-ch-ua to the page so the server sees them
+ if ua_changed:
+ combined_headers = {
+ "User-Agent": self.browser_config.user_agent,
+ "sec-ch-ua": self.browser_config.browser_hint,
+ }
+ combined_headers.update(self.browser_config.headers)
+ await page.set_extra_http_headers(combined_headers)
+
+ # await page.goto(URL)
+
+ # Add default cookie
+ # await context.add_cookies(
+ # [{"name": "cookiesEnabled", "value": "true", "url": url}]
+ # )
+
+ # Handle navigator overrides β only inject if not already done
+ # at context level by setup_context(). This fallback covers
+ # managed-browser / persistent / CDP paths where setup_context()
+ # is called without a crawlerRunConfig.
+ if config.override_navigator or config.simulate_user or config.magic:
+ if not getattr(context, '_crawl4ai_nav_overrider_injected', False):
+ await context.add_init_script(load_js_script("navigator_overrider"))
+ context._crawl4ai_nav_overrider_injected = True
+
+ # Force-open closed shadow roots β same guard against duplication
+ if config.flatten_shadow_dom:
+ if not getattr(context, '_crawl4ai_shadow_dom_injected', False):
+ await context.add_init_script("""
+ const _origAttachShadow = Element.prototype.attachShadow;
+ Element.prototype.attachShadow = function(init) {
+ return _origAttachShadow.call(this, {...init, mode: 'open'});
+ };
+ """)
+ context._crawl4ai_shadow_dom_injected = True
+
+ # Call hook after page creation
+ await self.execute_hook("on_page_context_created", page, context=context, config=config)
+
+ # Network Request Capturing
+ if config.capture_network_requests:
+ async def handle_request_capture(request):
try:
- # Be cautious with large post data
- post_data = request.post_data_buffer
- if post_data:
- # Attempt to decode, fallback to base64 or size indication
- try:
- post_data_str = post_data.decode('utf-8', errors='replace')
- except UnicodeDecodeError:
- post_data_str = f"[Binary data: {len(post_data)} bytes]"
- except Exception:
- post_data_str = "[Error retrieving post data]"
-
- captured_requests.append({
- "event_type": "request",
- "url": request.url,
- "method": request.method,
- "headers": dict(request.headers), # Convert Header dict
- "post_data": post_data_str,
- "resource_type": request.resource_type,
- "is_navigation_request": request.is_navigation_request(),
- "timestamp": time.time()
- })
- except Exception as e:
- if self.logger:
- self.logger.warning(f"Error capturing request details for {request.url}: {e}", tag="CAPTURE")
- captured_requests.append({"event_type": "request_capture_error", "url": request.url, "error": str(e), "timestamp": time.time()})
+ post_data_str = None
+ try:
+ # Be cautious with large post data
+ post_data = request.post_data_buffer
+ if post_data:
+ # Attempt to decode, fallback to base64 or size indication
+ try:
+ post_data_str = post_data.decode('utf-8', errors='replace')
+ except UnicodeDecodeError:
+ post_data_str = f"[Binary data: {len(post_data)} bytes]"
+ except Exception:
+ post_data_str = "[Error retrieving post data]"
+
+ captured_requests.append({
+ "event_type": "request",
+ "url": request.url,
+ "method": request.method,
+ "headers": dict(request.headers), # Convert Header dict
+ "post_data": post_data_str,
+ "resource_type": request.resource_type,
+ "is_navigation_request": request.is_navigation_request(),
+ "timestamp": time.time()
+ })
+ except Exception as e:
+ if self.logger:
+ self.logger.warning(f"Error capturing request details for {request.url}: {e}", tag="CAPTURE")
+ captured_requests.append({"event_type": "request_capture_error", "url": request.url, "error": str(e), "timestamp": time.time()})
- async def handle_response_capture(response):
- try:
+ async def handle_response_capture(response):
try:
- # body = await response.body()
- # json_body = await response.json()
- text_body = await response.text()
+ try:
+ # body = await response.body()
+ # json_body = await response.json()
+ text_body = await response.text()
+ except Exception as e:
+ body = None
+ # json_body = None
+ # text_body = None
+ captured_requests.append({
+ "event_type": "response",
+ "url": response.url,
+ "status": response.status,
+ "status_text": response.status_text,
+ "headers": dict(response.headers), # Convert Header dict
+ "from_service_worker": response.from_service_worker,
+ "request_timing": response.request.timing, # Detailed timing info
+ "timestamp": time.time(),
+ "body" : {
+ # "raw": body,
+ # "json": json_body,
+ "text": text_body
+ }
+ })
except Exception as e:
- body = None
- # json_body = None
- # text_body = None
- captured_requests.append({
- "event_type": "response",
- "url": response.url,
- "status": response.status,
- "status_text": response.status_text,
- "headers": dict(response.headers), # Convert Header dict
- "from_service_worker": response.from_service_worker,
- "request_timing": response.request.timing, # Detailed timing info
- "timestamp": time.time(),
- "body" : {
- # "raw": body,
- # "json": json_body,
- "text": text_body
- }
- })
- except Exception as e:
- if self.logger:
- self.logger.warning(f"Error capturing response details for {response.url}: {e}", tag="CAPTURE")
- captured_requests.append({"event_type": "response_capture_error", "url": response.url, "error": str(e), "timestamp": time.time()})
-
- async def handle_request_failed_capture(request):
- try:
- captured_requests.append({
- "event_type": "request_failed",
- "url": request.url,
- "method": request.method,
- "resource_type": request.resource_type,
- "failure_text": str(request.failure) if request.failure else "Unknown failure",
- "timestamp": time.time()
- })
- except Exception as e:
- if self.logger:
- self.logger.warning(f"Error capturing request failed details for {request.url}: {e}", tag="CAPTURE")
- captured_requests.append({"event_type": "request_failed_capture_error", "url": request.url, "error": str(e), "timestamp": time.time()})
-
- page.on("request", handle_request_capture)
- page.on("response", handle_response_capture)
- page.on("requestfailed", handle_request_failed_capture)
-
- # Console Message Capturing
- handle_console = None
- handle_error = None
- if config.capture_console_messages:
- # Set up console capture using adapter
- handle_console = await self.adapter.setup_console_capture(page, captured_console)
- handle_error = await self.adapter.setup_error_capture(page, captured_console)
-
- # Set up console logging if requested
- # Note: For undetected browsers, console logging won't work directly
- # but captured messages can still be logged after retrieval
-
- try:
+ if self.logger:
+ self.logger.warning(f"Error capturing response details for {response.url}: {e}", tag="CAPTURE")
+ captured_requests.append({"event_type": "response_capture_error", "url": response.url, "error": str(e), "timestamp": time.time()})
+
+ async def handle_request_failed_capture(request):
+ try:
+ captured_requests.append({
+ "event_type": "request_failed",
+ "url": request.url,
+ "method": request.method,
+ "resource_type": request.resource_type,
+ "failure_text": str(request.failure) if request.failure else "Unknown failure",
+ "timestamp": time.time()
+ })
+ except Exception as e:
+ if self.logger:
+ self.logger.warning(f"Error capturing request failed details for {request.url}: {e}", tag="CAPTURE")
+ captured_requests.append({"event_type": "request_failed_capture_error", "url": request.url, "error": str(e), "timestamp": time.time()})
+
+ page.on("request", handle_request_capture)
+ page.on("response", handle_response_capture)
+ page.on("requestfailed", handle_request_failed_capture)
+
+ # Console Message Capturing
+ handle_console = None
+ handle_error = None
+ if config.capture_console_messages:
+ # Set up console capture using adapter
+ handle_console = await self.adapter.setup_console_capture(page, captured_console)
+ handle_error = await self.adapter.setup_error_capture(page, captured_console)
+
+ # Set up console logging if requested
+ # Note: For undetected browsers, console logging won't work directly
+ # but captured messages can still be logged after retrieval
# Get SSL certificate information if requested and URL is HTTPS
ssl_cert = None
if config.fetch_ssl_certificate:
@@ -683,7 +741,8 @@ async def handle_request_failed_capture(request):
await page.set_content(html_content, wait_until=config.wait_until)
response = None
- redirected_url = config.base_url or url
+ # For raw: URLs, only use base_url if provided; don't fall back to the raw HTML string
+ redirected_url = config.base_url
status_code = 200
response_headers = {}
else:
@@ -704,6 +763,7 @@ async def handle_request_failed_capture(request):
url, wait_until=config.wait_until, timeout=config.page_timeout
)
redirected_url = page.url
+ redirected_status_code = response.status if response else None
except Error as e:
# Allow navigation to be aborted when downloading files
# This is expected behavior for downloads in some browser engines
@@ -876,6 +936,7 @@ async def handle_request_failed_capture(request):
"scale": scale,
},
)
+ await cdp.detach()
except Exception as e:
self.logger.warning(
message="Failed to adjust viewport to content: {error}",
@@ -888,20 +949,52 @@ async def handle_request_failed_capture(request):
# await self._handle_full_page_scan(page, config.scroll_delay)
await self._handle_full_page_scan(page, config.scroll_delay, config.max_scroll_steps)
- # Handle virtual scroll if configured
+ # --- Phase 1: Pre-wait JS and interaction ---
+
+ # Execute js_code_before_wait (for triggering loading that wait_for checks)
+ if config.js_code_before_wait:
+ bw_result = await self.robust_execute_user_script(
+ page, config.js_code_before_wait
+ )
+ if not bw_result["success"]:
+ self.logger.warning(
+ message="js_code_before_wait had issues: {error}",
+ tag="JS_EXEC",
+ params={"error": bw_result.get("error")},
+ )
+
+ # Handle user simulation β generate mouse movement and scroll
+ # signals that anti-bot systems look for, without firing keyboard
+ # events (ArrowDown triggers JS framework navigation) or clicking
+ # at fixed positions (may hit buttons/links and navigate away).
+ if config.simulate_user or config.magic:
+ await page.mouse.move(random.randint(100, 300), random.randint(150, 300))
+ await page.mouse.move(random.randint(300, 600), random.randint(200, 400))
+ await page.mouse.wheel(0, random.randint(200, 400))
+
+ # --- Phase 2: Wait for page readiness ---
+
+ if config.wait_for:
+ try:
+ timeout = config.wait_for_timeout if config.wait_for_timeout is not None else config.page_timeout
+ await self.smart_wait(
+ page, config.wait_for, timeout=timeout
+ )
+ except Exception as e:
+ raise RuntimeError(f"Wait condition failed: {str(e)}")
+
+ # Handle virtual scroll if configured (after wait_for so container exists)
if config.virtual_scroll_config:
await self._handle_virtual_scroll(page, config.virtual_scroll_config)
- # Execute JavaScript if provided
- # if config.js_code:
- # if isinstance(config.js_code, str):
- # await page.evaluate(config.js_code)
- # elif isinstance(config.js_code, list):
- # for js in config.js_code:
- # await page.evaluate(js)
+ # Pre-content retrieval hooks and delay
+ await self.execute_hook("before_retrieve_html", page, context=context, config=config)
+ if config.delay_before_return_html:
+ await asyncio.sleep(config.delay_before_return_html)
+
+ # --- Phase 3: Post-wait JS (runs on fully-loaded page) ---
if config.js_code:
- # execution_result = await self.execute_user_script(page, config.js_code)
execution_result = await self.robust_execute_user_script(
page, config.js_code
)
@@ -916,28 +1009,7 @@ async def handle_request_failed_capture(request):
await self.execute_hook("on_execution_started", page, context=context, config=config)
await self.execute_hook("on_execution_ended", page, context=context, config=config, result=execution_result)
- # Handle user simulation
- if config.simulate_user or config.magic:
- await page.mouse.move(100, 100)
- await page.mouse.down()
- await page.mouse.up()
- await page.keyboard.press("ArrowDown")
-
- # Handle wait_for condition
- # Todo: Decide how to handle this
- if not config.wait_for and config.css_selector and False:
- # if not config.wait_for and config.css_selector:
- config.wait_for = f"css:{config.css_selector}"
-
- if config.wait_for:
- try:
- # Use wait_for_timeout if specified, otherwise fall back to page_timeout
- timeout = config.wait_for_timeout if config.wait_for_timeout is not None else config.page_timeout
- await self.smart_wait(
- page, config.wait_for, timeout=timeout
- )
- except Exception as e:
- raise RuntimeError(f"Wait condition failed: {str(e)}")
+ # --- Phase 4: DOM processing before HTML capture ---
# Update image dimensions if needed
if not self.browser_config.text_mode:
@@ -959,21 +1031,32 @@ async def handle_request_failed_capture(request):
if config.process_iframes:
page = await self.process_iframes(page)
- # Pre-content retrieval hooks and delay
- await self.execute_hook("before_retrieve_html", page, context=context, config=config)
- if config.delay_before_return_html:
- await asyncio.sleep(config.delay_before_return_html)
+ # Handle CMP/consent popup removal (before generic overlay removal)
+ if config.remove_consent_popups:
+ await self.remove_consent_popups(page)
# Handle overlay removal
if config.remove_overlay_elements:
await self.remove_overlay_elements(page)
- if config.css_selector:
+ # --- Phase 5: HTML capture ---
+
+ if config.flatten_shadow_dom:
+ # Use JS to serialize the full DOM including shadow roots
+ flatten_js = load_js_script("flatten_shadow_dom")
+ html = await self.adapter.evaluate(page, flatten_js)
+ if not html or not isinstance(html, str):
+ # Fallback to normal capture if JS returned nothing
+ self.logger.warning(
+ message="Shadow DOM flattening returned no content, falling back to page.content()",
+ tag="SCRAPE",
+ )
+ html = await page.content()
+ elif config.css_selector:
try:
- # Handle comma-separated selectors by splitting them
selectors = [s.strip() for s in config.css_selector.split(',')]
html_parts = []
-
+
for selector in selectors:
try:
content = await self.adapter.evaluate(page,
@@ -984,16 +1067,13 @@ async def handle_request_failed_capture(request):
html_parts.append(content)
except Error as e:
print(f"Warning: Could not get content for selector '{selector}': {str(e)}")
-
- # Wrap in a div to create a valid HTML structure
- html = f"<div class='crawl4ai-result'>\n" + "\n".join(html_parts) + "\n</div>"
+
+ html = f"<div class='crawl4ai-result'>\n" + "\n".join(html_parts) + "\n</div>"
except Error as e:
raise RuntimeError(f"Failed to extract HTML content: {str(e)}")
else:
html = await page.content()
-
- # # Get final HTML content
- # html = await page.content()
+
await self.execute_hook(
"before_return_html", page=page, html=html, context=context, config=config
)
@@ -1014,7 +1094,11 @@ async def handle_request_failed_capture(request):
if config.screenshot_wait_for:
await asyncio.sleep(config.screenshot_wait_for)
screenshot_data = await self.take_screenshot(
- page, screenshot_height_threshold=config.screenshot_height_threshold
+ page,
+ screenshot_height_threshold=config.screenshot_height_threshold,
+ force_viewport_screenshot=config.force_viewport_screenshot,
+ scan_full_page=config.scan_full_page,
+ scroll_delay=config.scroll_delay
)
if screenshot_data or pdf_data or mhtml_data:
@@ -1040,10 +1124,14 @@ async def get_delayed_content(delay: float = 5.0) -> str:
captured_console.extend(final_messages)
###
- # This ensures we capture the current page URL at the time we return the response,
+ # This ensures we capture the current page URL at the time we return the response,
# which correctly reflects any JavaScript navigation that occurred.
+ # For raw:/file:// URLs, preserve the earlier redirected_url (config.base_url or None)
+ # instead of using page.url which would be "about:blank".
###
- redirected_url = page.url # Use current page URL to capture JS redirects
+ is_local_content = url.startswith("file://") or url.startswith("raw://") or url.startswith("raw:")
+ if not is_local_content:
+ redirected_url = page.url # Use current page URL to capture JS redirects
# Return complete response
return AsyncCrawlResponse(
@@ -1060,6 +1148,7 @@ async def get_delayed_content(delay: float = 5.0) -> str:
self._downloaded_files if self._downloaded_files else None
),
redirected_url=redirected_url,
+ redirected_status_code=redirected_status_code,
# Include captured data if enabled
network_requests=captured_requests if config.capture_network_requests else None,
console_messages=captured_console if config.capture_console_messages else None,
@@ -1069,30 +1158,37 @@ async def get_delayed_content(delay: float = 5.0) -> str:
raise e
finally:
- # If no session_id is given we should close the page
- all_contexts = page.context.browser.contexts
- total_pages = sum(len(context.pages) for context in all_contexts)
- if config.session_id:
- pass
- elif total_pages <= 1 and (self.browser_config.use_managed_browser or self.browser_config.headless):
- pass
- else:
- # Detach listeners before closing to prevent potential errors during close
+ # Always clean up event listeners to prevent accumulation
+ # across reuses (even for session pages).
+ try:
if config.capture_network_requests:
page.remove_listener("request", handle_request_capture)
page.remove_listener("response", handle_response_capture)
page.remove_listener("requestfailed", handle_request_failed_capture)
if config.capture_console_messages:
- # Retrieve any final console messages for undetected browsers
if hasattr(self.adapter, 'retrieve_console_messages'):
final_messages = await self.adapter.retrieve_console_messages(page)
captured_console.extend(final_messages)
-
- # Clean up console capture
await self.adapter.cleanup_console_capture(page, handle_console, handle_error)
-
- # Close the page
- await page.close()
+ except Exception:
+ pass
+
+ if not config.session_id:
+ # ALWAYS decrement refcount first β must succeed even if
+ # the browser crashed or the page is in a bad state.
+ try:
+ await self.browser_manager.release_page_with_context(page)
+ except Exception:
+ pass
+
+ # Close the page unless it's the last one in a headless/managed browser
+ try:
+ all_contexts = page.context.browser.contexts
+ total_pages = sum(len(context.pages) for context in all_contexts)
+ if not (total_pages <= 1 and (self.browser_config.use_managed_browser or self.browser_config.headless)):
+ await page.close()
+ except Exception:
+ pass
# async def _handle_full_page_scan(self, page: Page, scroll_delay: float = 0.1):
async def _handle_full_page_scan(self, page: Page, scroll_delay: float = 0.1, max_scroll_steps: Optional[int] = None):
@@ -1428,6 +1524,50 @@ async def remove_overlay_elements(self, page: Page) -> None:
params={"error": str(e)},
)
+ async def remove_consent_popups(self, page: Page) -> None:
+ """
+ Removes GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot,
+ TrustArc, Quantcast, Didomi, Usercentrics, Sourcepoint, Klaro, Osano, Iubenda,
+ Complianz, CookieYes, ConsentManager, LiveRamp/Fides, etc.).
+
+ Strategy:
+ 1. Try clicking "Accept All" buttons (cleanest dismissal, sets cookies)
+ 2. Try IAB TCF / CMP JavaScript APIs
+ 3. Remove known CMP containers by selector
+ 4. Handle iframe-based CMPs
+ 5. Restore body scroll
+
+ Args:
+ page (Page): The Playwright page instance
+ """
+ remove_consent_js = load_js_script("remove_consent_popups")
+
+ try:
+ await self.adapter.evaluate(page,
+ f"""
+ (async () => {{
+ try {{
+ const removeConsent = {remove_consent_js};
+ await removeConsent();
+ return {{ success: true }};
+ }} catch (error) {{
+ return {{
+ success: false,
+ error: error.toString(),
+ stack: error.stack
+ }};
+ }}
+ }})()
+ """
+ )
+ await page.wait_for_timeout(500) # Wait for any animations to complete
+ except Exception as e:
+ self.logger.warning(
+ message="Failed to remove consent popups: {error}",
+ tag="SCRAPE",
+ params={"error": str(e)},
+ )
+
async def export_pdf(self, page: Page) -> bytes:
"""
Exports the current page as a PDF.
@@ -1582,7 +1722,10 @@ async def _generate_media_from_html(
await asyncio.sleep(config.screenshot_wait_for)
screenshot_height_threshold = getattr(config, 'screenshot_height_threshold', None)
screenshot_data = await self.take_screenshot(
- page, screenshot_height_threshold=screenshot_height_threshold
+ page,
+ screenshot_height_threshold=screenshot_height_threshold,
+ scan_full_page=getattr(config, 'scan_full_page', True),
+ scroll_delay=config.scroll_delay if config else 0.2
)
return screenshot_data, pdf_data, mhtml_data
@@ -1608,6 +1751,7 @@ async def _generate_media_from_html(
# Clean up the page
if page:
try:
+ await self.browser_manager.release_page_with_context(page)
await page.close()
except Exception:
pass
@@ -1623,6 +1767,14 @@ async def take_screenshot(self, page, **kwargs) -> str:
Returns:
str: The base64-encoded screenshot data
"""
+ # Check if viewport-only screenshot is forced
+ force_viewport = kwargs.get('force_viewport_screenshot', False)
+ scan_full_page = kwargs.get('scan_full_page', True)
+
+ if force_viewport or not scan_full_page:
+ # Use viewport-only screenshot
+ return await self.take_screenshot_naive(page)
+
need_scroll = await self.page_need_scroll(page)
if not need_scroll:
@@ -1684,12 +1836,26 @@ async def take_screenshot_scroller(self, page: Page, **kwargs) -> str:
str: The base64-encoded screenshot data
"""
try:
+ # Save original viewport so we can restore it after capture
+ original_viewport = page.viewport_size
+
# Get page height
dimensions = await self.get_page_dimensions(page)
page_width = dimensions["width"]
page_height = dimensions["height"]
- # page_height = await page.evaluate("document.documentElement.scrollHeight")
- # page_width = await page.evaluate("document.documentElement.scrollWidth")
+
+ # Freeze element dimensions before viewport change to prevent
+ # responsive CSS from rescaling images (fixes Elementor distortion)
+ await page.evaluate("""
+ document.querySelectorAll('img, video, picture, svg, canvas').forEach(el => {
+ const rect = el.getBoundingClientRect();
+ if (rect.width > 0 && rect.height > 0) {
+ el.style.setProperty('width', rect.width + 'px', 'important');
+ el.style.setProperty('height', rect.height + 'px', 'important');
+ el.dataset.crawl4aiFrozen = '1';
+ }
+ });
+ """)
# Set a large viewport
large_viewport_height = min(
@@ -1701,6 +1867,7 @@ async def take_screenshot_scroller(self, page: Page, **kwargs) -> str:
)
# Page still too long, segment approach
+ scroll_delay = kwargs.get("scroll_delay", 0.2)
segments = []
viewport_size = page.viewport_size
viewport_height = viewport_size["height"]
@@ -1711,28 +1878,33 @@ async def take_screenshot_scroller(self, page: Page, **kwargs) -> str:
# Special handling for the last segment
if i == num_segments - 1:
last_part_height = page_height % viewport_height
-
+
# If page_height is an exact multiple of viewport_height,
# we don't need an extra segment
if last_part_height == 0:
# Skip last segment if page height is exact multiple of viewport
break
-
+
# Adjust viewport to exactly match the remaining content height
await page.set_viewport_size({"width": page_width, "height": last_part_height})
-
+
await page.evaluate(f"window.scrollTo(0, {y_offset})")
- await asyncio.sleep(0.01) # wait for render
-
+ await asyncio.sleep(scroll_delay) # wait for render (respects scroll_delay config)
+
# Capture the current segment
- # Note: Using compression options (format, quality) would go here
seg_shot = await page.screenshot(full_page=False, type="jpeg", quality=85)
- # seg_shot = await page.screenshot(full_page=False)
img = Image.open(BytesIO(seg_shot)).convert("RGB")
segments.append(img)
- # Reset viewport to original size after capturing segments
- await page.set_viewport_size({"width": page_width, "height": viewport_height})
+ # Unfreeze element dimensions and restore original viewport
+ await page.evaluate("""
+ document.querySelectorAll('[data-crawl4ai-frozen]').forEach(el => {
+ el.style.removeProperty('width');
+ el.style.removeProperty('height');
+ delete el.dataset.crawl4aiFrozen;
+ });
+ """)
+ await page.set_viewport_size(original_viewport)
total_height = sum(img.height for img in segments)
stitched = Image.new("RGB", (segments[0].width, total_height))
@@ -1744,7 +1916,7 @@ async def take_screenshot_scroller(self, page: Page, **kwargs) -> str:
buffered = BytesIO()
stitched = stitched.convert("RGB")
- stitched.save(buffered, format="BMP", quality=85)
+ stitched.save(buffered, format="PNG")
encoded = base64.b64encode(buffered.getvalue()).decode("utf-8")
return encoded
@@ -2372,11 +2544,13 @@ async def _handle_file(self, path: str) -> AsyncCrawlResponse:
status_code=200
)
- async def _handle_raw(self, content: str) -> AsyncCrawlResponse:
+ async def _handle_raw(self, content: str, base_url: str = None) -> AsyncCrawlResponse:
return AsyncCrawlResponse(
html=content,
response_headers={},
- status_code=200
+ status_code=200,
+ # For raw: URLs, use base_url if provided; don't fall back to the raw content
+ redirected_url=base_url
)
@@ -2448,7 +2622,8 @@ async def _handle_http(
encoding = response.charset
if not encoding:
- encoding = chardet.detect(content.tobytes())['encoding'] or 'utf-8'
+ detection_result = await asyncio.to_thread(chardet.detect, content.tobytes())
+ encoding = detection_result['encoding'] or 'utf-8'
result = AsyncCrawlResponse(
html=content.tobytes().decode(encoding, errors='replace'),
@@ -2501,7 +2676,7 @@ async def crawl(
# Don't use parsed.path - urlparse truncates at '#' which is common in CSS
# Strip prefix directly: "raw://" (6 chars) or "raw:" (4 chars)
raw_content = url[6:] if url.startswith("raw://") else url[4:]
- return await self._handle_raw(raw_content)
+ return await self._handle_raw(raw_content, base_url=config.base_url)
else: # http or https
return await self._handle_http(url, config)
diff --git a/crawl4ai/async_dispatcher.py b/crawl4ai/async_dispatcher.py
index bd44557c7..1d6a236b7 100644
--- a/crawl4ai/async_dispatcher.py
+++ b/crawl4ai/async_dispatcher.py
@@ -458,14 +458,15 @@ async def run_urls(
except Exception as e:
if self.monitor:
- self.monitor.update_memory_status(f"QUEUE_ERROR: {str(e)}")
+ self.monitor.update_memory_status(f"QUEUE_ERROR: {str(e)}")
+ raise
finally:
# Clean up
memory_monitor.cancel()
if self.monitor:
self.monitor.stop()
- return results
+ return results
async def _update_queue_priorities(self):
"""Periodically update priorities of items in the queue to prevent starvation"""
diff --git a/crawl4ai/async_url_seeder.py b/crawl4ai/async_url_seeder.py
index 29fb4b50c..22fa8f630 100644
--- a/crawl4ai/async_url_seeder.py
+++ b/crawl4ai/async_url_seeder.py
@@ -400,18 +400,20 @@ async def urls(self,
if self.logger and hasattr(self.logger, 'verbose') and config.verbose is not None:
self.logger.verbose = config.verbose
- # ensure we have the latest CC collection id
- if self.index_id is None:
- self.index_id = await self._latest_index()
-
# Parse source parameter - split by '+' to get list of sources
- sources = source.split('+')
+ sources = [s.strip().lower() for s in source.split("+") if s.strip()]
+
valid_sources = {"cc", "sitemap"}
for s in sources:
if s not in valid_sources:
raise ValueError(
f"Invalid source '{s}'. Valid sources are: {', '.join(valid_sources)}")
+ # ensure we have the latest CC collection id when the source is cc
+ if s == "cc" and self.index_id is None:
+ self.index_id = await self._latest_index()
+
+
if hits_per_sec:
if hits_per_sec <= 0:
self._log(
@@ -448,16 +450,20 @@ async def gen():
async def producer():
try:
async for u in gen():
- if u in seen:
- self._log("debug", "Skipping duplicate URL: {url}",
- params={"url": u}, tag="URL_SEED")
+ try:
+ if u in seen:
+ self._log("debug", "Skipping duplicate URL: {url}",
+ params={"url": u}, tag="URL_SEED")
+ continue
+ if stop_event.is_set():
+ self._log(
+ "info", "Producer stopping due to max_urls limit.", tag="URL_SEED")
+ break
+ seen.add(u)
+ await queue.put(u) # Will block if queue is full, providing backpressure
+ except UnicodeEncodeError:
+ # Skip URLs that cause encoding errors (e.g. on Windows)
continue
- if stop_event.is_set():
- self._log(
- "info", "Producer stopping due to max_urls limit.", tag="URL_SEED")
- break
- seen.add(u)
- await queue.put(u) # Will block if queue is full, providing backpressure
except Exception as e:
self._log("error", "Producer encountered an error: {error}", params={
"error": str(e)}, tag="URL_SEED")
@@ -783,7 +789,8 @@ async def _resolve_head(self, url: str) -> Optional[str]:
Returns:
* the same URL if it answers 2xx,
- * the absolute redirect target if it answers 3xx,
+ * the verified absolute redirect target if it answers 3xx
+ and the target also answers 2xx,
* None on any other status or network error.
"""
try:
@@ -793,11 +800,23 @@ async def _resolve_head(self, url: str) -> Optional[str]:
if 200 <= r.status_code < 300:
return str(r.url)
- # single level redirect
+ # single level redirect β verify target is alive
if r.status_code in (301, 302, 303, 307, 308):
loc = r.headers.get("location")
if loc:
- return urljoin(url, loc)
+ target = urljoin(url, loc)
+ # Guard against self-redirects
+ if target == url:
+ return None
+ try:
+ r2 = await self.client.head(
+ target, timeout=10, follow_redirects=False
+ )
+ if 200 <= r2.status_code < 300:
+ return str(r2.url)
+ except Exception:
+ pass
+ return None
return None
@@ -985,7 +1004,8 @@ async def _iter_sitemap_content(self, url: str, content: bytes):
def _normalize_loc(raw: Optional[str]) -> Optional[str]:
if not raw:
return None
- normalized = urljoin(base_url, raw.strip())
+ cleaned = raw.strip().replace("\u200b", "").replace("\ufeff", "")
+ normalized = urljoin(base_url, cleaned)
if not normalized:
return None
return normalized
@@ -1105,7 +1125,8 @@ async def _iter_sitemap(self, url: str):
def _normalize_loc(raw: Optional[str]) -> Optional[str]:
if not raw:
return None
- normalized = urljoin(base_url, raw.strip())
+ cleaned = raw.strip().replace("\u200b", "").replace("\ufeff", "")
+ normalized = urljoin(base_url, cleaned)
if not normalized:
return None
return normalized
@@ -1290,6 +1311,7 @@ async def _validate(self, url: str, res_list: List[Dict[str, Any]], live: bool,
head_data = await asyncio.to_thread(_parse_head, html) if ok else {}
entry = {
"url": final or url,
+ "original_url": url,
"status": status,
"head_data": head_data,
}
diff --git a/crawl4ai/async_webcrawler.py b/crawl4ai/async_webcrawler.py
index ef03cb74b..34a7502ae 100644
--- a/crawl4ai/async_webcrawler.py
+++ b/crawl4ai/async_webcrawler.py
@@ -1,5 +1,6 @@
from .__version__ import __version__ as crawl4ai_version
import os
+import re
import sys
import time
from pathlib import Path
@@ -50,6 +51,7 @@
compute_head_fingerprint,
)
from .cache_validator import CacheValidator, CacheValidationResult
+from .antibot_detector import is_blocked
class AsyncWebCrawler:
@@ -227,11 +229,11 @@ async def arun(
screenshot=True,
...
)
- result = await crawler.arun(url="https://example.com", crawler_config=config)
+ result = await crawler.arun(url="https://example.com", config=config)
Args:
url: The URL to crawl (http://, https://, file://, or raw:)
- crawler_config: Configuration object controlling crawl behavior
+ config: Configuration object controlling crawl behavior
[other parameters maintained for backwards compatibility]
Returns:
@@ -372,13 +374,9 @@ async def arun(
# Fetch fresh content if needed
if not cached_result or not html:
- t1 = time.perf_counter()
-
- if config.user_agent:
- self.crawler_strategy.update_user_agent(
- config.user_agent)
+ from urllib.parse import urlparse
- # Check robots.txt if enabled
+ # Check robots.txt if enabled (once, before any attempts)
if config and config.check_robots_txt:
if not await self.robots_parser.can_fetch(
url, self.browser_config.user_agent
@@ -394,71 +392,250 @@ async def arun(
},
)
- ##############################
- # Call CrawlerStrategy.crawl #
- ##############################
- async_response = await self.crawler_strategy.crawl(
- url,
- config=config, # Pass the entire config object
- )
+ # --- Anti-bot retry setup ---
+ # raw: URLs contain caller-provided HTML (e.g. from cache),
+ # not content fetched from a web server. Anti-bot detection,
+ # proxy retries, and fallback fetching are meaningless here.
+ _is_raw_url = url.startswith("raw:") or url.startswith("raw://")
+
+ _max_attempts = 1 + getattr(config, "max_retries", 0)
+ _proxy_list = config._get_proxy_list()
+ _original_proxy_config = config.proxy_config
+ _block_reason = ""
+ _done = False
+ crawl_result = None
+ _crawl_stats = {
+ "attempts": 0,
+ "retries": 0,
+ "proxies_used": [],
+ "fallback_fetch_used": False,
+ "resolved_by": None,
+ }
+
+ for _attempt in range(_max_attempts):
+ if _done:
+ break
+
+ if _attempt > 0:
+ _crawl_stats["retries"] = _attempt
+ self.logger.warning(
+ message="Anti-bot retry {attempt}/{max_retries} for {url} β {reason}",
+ tag="ANTIBOT",
+ params={
+ "attempt": _attempt,
+ "max_retries": config.max_retries,
+ "url": url[:80],
+ "reason": _block_reason,
+ },
+ )
- html = sanitize_input_encode(async_response.html)
- screenshot_data = async_response.screenshot
- pdf_data = async_response.pdf_data
- js_execution_result = async_response.js_execution_result
+ for _p_idx, _proxy in enumerate(_proxy_list):
+ if _p_idx > 0 or _attempt > 0:
+ self.logger.info(
+ message="Trying proxy {idx}/{total}: {proxy}",
+ tag="ANTIBOT",
+ params={
+ "idx": _p_idx + 1,
+ "total": len(_proxy_list),
+ "proxy": _proxy.server if _proxy else "direct",
+ },
+ )
- t2 = time.perf_counter()
- self.logger.url_status(
- url=cache_context.display_url,
- success=bool(html),
- timing=t2 - t1,
- tag="FETCH",
- )
+ # Set the active proxy for this attempt
+ config.proxy_config = _proxy
+ _crawl_stats["attempts"] += 1
- ###############################################################
- # Process the HTML content, Call CrawlerStrategy.process_html #
- ###############################################################
- from urllib.parse import urlparse
- crawl_result: CrawlResult = await self.aprocess_html(
- url=url,
- html=html,
- extracted_content=extracted_content,
- config=config, # Pass the config object instead of individual parameters
- screenshot_data=screenshot_data,
- pdf_data=pdf_data,
- verbose=config.verbose,
- is_raw_html=True if url.startswith("raw:") else False,
- redirected_url=async_response.redirected_url,
- original_scheme=urlparse(url).scheme,
- **kwargs,
- )
+ try:
+ t1 = time.perf_counter()
- crawl_result.status_code = async_response.status_code
- crawl_result.redirected_url = async_response.redirected_url or url
- crawl_result.response_headers = async_response.response_headers
- crawl_result.downloaded_files = async_response.downloaded_files
- crawl_result.js_execution_result = js_execution_result
- crawl_result.mhtml = async_response.mhtml_data
- crawl_result.ssl_certificate = async_response.ssl_certificate
- # Add captured network and console data if available
- crawl_result.network_requests = async_response.network_requests
- crawl_result.console_messages = async_response.console_messages
-
- crawl_result.success = bool(html)
- crawl_result.session_id = getattr(
- config, "session_id", None)
- crawl_result.cache_status = "miss"
+ if config.user_agent:
+ self.crawler_strategy.update_user_agent(
+ config.user_agent)
+
+ async_response = await self.crawler_strategy.crawl(
+ url, config=config)
+
+ html = sanitize_input_encode(async_response.html)
+ screenshot_data = async_response.screenshot
+ pdf_data = async_response.pdf_data
+ js_execution_result = async_response.js_execution_result
+
+ self.logger.url_status(
+ url=cache_context.display_url,
+ success=bool(html),
+ timing=time.perf_counter() - t1,
+ tag="FETCH",
+ )
+
+ crawl_result = await self.aprocess_html(
+ url=url, html=html,
+ extracted_content=extracted_content,
+ config=config,
+ screenshot_data=screenshot_data,
+ pdf_data=pdf_data,
+ verbose=config.verbose,
+ is_raw_html=True if url.startswith("raw:") else False,
+ redirected_url=async_response.redirected_url,
+ original_scheme=urlparse(url).scheme,
+ **kwargs,
+ )
+
+ crawl_result.status_code = async_response.status_code
+ is_raw_url = url.startswith("raw:") or url.startswith("raw://")
+ crawl_result.redirected_url = async_response.redirected_url or (None if is_raw_url else url)
+ crawl_result.redirected_status_code = async_response.redirected_status_code
+ crawl_result.response_headers = async_response.response_headers
+ crawl_result.downloaded_files = async_response.downloaded_files
+ crawl_result.js_execution_result = js_execution_result
+ crawl_result.mhtml = async_response.mhtml_data
+ crawl_result.ssl_certificate = async_response.ssl_certificate
+ crawl_result.network_requests = async_response.network_requests
+ crawl_result.console_messages = async_response.console_messages
+ crawl_result.success = bool(html)
+ crawl_result.session_id = getattr(config, "session_id", None)
+ crawl_result.cache_status = "miss"
+
+ # Check if blocked (skip for raw: URLs β
+ # caller-provided content, anti-bot N/A)
+ if _is_raw_url:
+ _blocked = False
+ _block_reason = ""
+ else:
+ _blocked, _block_reason = is_blocked(
+ async_response.status_code, html)
+
+ _crawl_stats["proxies_used"].append({
+ "proxy": _proxy.server if _proxy else None,
+ "status_code": async_response.status_code,
+ "blocked": _blocked,
+ "reason": _block_reason if _blocked else "",
+ })
+
+ if not _blocked:
+ _crawl_stats["resolved_by"] = "proxy" if _proxy else "direct"
+ _done = True
+ break # Success β exit proxy loop
+
+ except Exception as _crawl_err:
+ _crawl_stats["proxies_used"].append({
+ "proxy": _proxy.server if _proxy else None,
+ "status_code": None,
+ "blocked": True,
+ "reason": str(_crawl_err),
+ })
+ self.logger.error_status(
+ url=url,
+ error=f"Proxy {_proxy.server if _proxy else 'direct'} failed: {_crawl_err}",
+ tag="ANTIBOT",
+ )
+ _block_reason = str(_crawl_err)
+ # If this is the only proxy and only attempt, re-raise
+ # so the caller gets the real error (not a silent swallow).
+ # But if there are more proxies or retries to try, continue.
+ if len(_proxy_list) <= 1 and _max_attempts <= 1:
+ raise
+
+ # Restore original proxy_config
+ config.proxy_config = _original_proxy_config
+
+ # --- Fallback fetch function (last resort after all retries+proxies exhausted) ---
+ # Invoke fallback when: (a) crawl_result exists but is blocked, OR
+ # (b) crawl_result is None because all proxies threw exceptions (browser crash, timeout).
+ # Skip for raw: URLs β fallback expects a real URL, not raw HTML content.
+ _fallback_fn = getattr(config, "fallback_fetch_function", None)
+ if _fallback_fn and not _done and not _is_raw_url:
+ _needs_fallback = (
+ crawl_result is None # All proxies threw exceptions
+ or is_blocked(crawl_result.status_code, crawl_result.html or "")[0]
+ )
+ if _needs_fallback:
+ self.logger.warning(
+ message="All retries exhausted, invoking fallback_fetch_function for {url}",
+ tag="ANTIBOT",
+ params={"url": url[:80]},
+ )
+ _crawl_stats["fallback_fetch_used"] = True
+ try:
+ _fallback_html = await _fallback_fn(url)
+ if _fallback_html:
+ _sanitized_html = sanitize_input_encode(_fallback_html)
+ try:
+ crawl_result = await self.aprocess_html(
+ url=url,
+ html=_sanitized_html,
+ extracted_content=extracted_content,
+ config=config,
+ screenshot_data=None,
+ pdf_data=None,
+ verbose=config.verbose,
+ is_raw_html=True,
+ redirected_url=url,
+ original_scheme=urlparse(url).scheme,
+ **kwargs,
+ )
+ except Exception as _proc_err:
+ # aprocess_html may fail if browser is dead (e.g.,
+ # consent popup removal needs Page.evaluate).
+ # Fall back to a minimal result with raw HTML.
+ self.logger.warning(
+ message="Fallback HTML processing failed ({err}), using raw HTML",
+ tag="ANTIBOT",
+ params={"err": str(_proc_err)[:100]},
+ )
+ crawl_result = CrawlResult(
+ url=url,
+ html=_sanitized_html,
+ success=True,
+ status_code=200,
+ )
+ crawl_result.success = True
+ crawl_result.status_code = 200
+ crawl_result.session_id = getattr(config, "session_id", None)
+ crawl_result.cache_status = "miss"
+ _crawl_stats["resolved_by"] = "fallback_fetch"
+ except Exception as _fallback_err:
+ self.logger.error_status(
+ url=url,
+ error=f"Fallback fetch failed: {_fallback_err}",
+ tag="ANTIBOT",
+ )
+
+ # --- Mark blocked results as failed ---
+ # Skip re-check when fallback was used β the fallback result is
+ # authoritative. Real pages may contain anti-bot script markers
+ # (e.g. PerimeterX JS on Walmart) that trigger false positives.
+ # Also skip for raw: URLs β caller-provided content, anti-bot N/A.
+ if crawl_result:
+ if not _crawl_stats.get("fallback_fetch_used") and not _is_raw_url:
+ _blocked, _block_reason = is_blocked(
+ crawl_result.status_code, crawl_result.html or "")
+ if _blocked:
+ crawl_result.success = False
+ crawl_result.error_message = f"Blocked by anti-bot protection: {_block_reason}"
+ crawl_result.crawl_stats = _crawl_stats
+ else:
+ # All proxies threw exceptions and fallback either wasn't
+ # configured or also failed. Build a minimal result so the
+ # caller gets crawl_stats instead of None.
+ crawl_result = CrawlResult(
+ url=url,
+ html="",
+ success=False,
+ status_code=None,
+ error_message=f"All proxies failed: {_block_reason}" if _block_reason else "All proxies failed",
+ )
+ crawl_result.crawl_stats = _crawl_stats
# Compute head fingerprint for cache validation
- if html:
- head_end = html.lower().find('</head>')
+ if crawl_result and crawl_result.html:
+ head_end = crawl_result.html.lower().find('</head>')
if head_end != -1:
- head_html = html[:head_end + 7]
+ head_html = crawl_result.html[:head_end + 7]
crawl_result.head_fingerprint = compute_head_fingerprint(head_html)
self.logger.url_status(
url=cache_context.display_url,
- success=crawl_result.success,
+ success=crawl_result.success if crawl_result else False,
timing=time.perf_counter() - start_time,
tag="COMPLETE",
)
@@ -479,7 +656,9 @@ async def arun(
cached_result.success = bool(html)
cached_result.session_id = getattr(
config, "session_id", None)
- cached_result.redirected_url = cached_result.redirected_url or url
+ # For raw: URLs, don't fall back to the raw HTML string as redirected_url
+ is_raw_url = url.startswith("raw:") or url.startswith("raw://")
+ cached_result.redirected_url = cached_result.redirected_url or (None if is_raw_url else url)
return CrawlResultContainer(cached_result)
except Exception as e:
@@ -653,11 +832,17 @@ async def aprocess_html(
# if not config.content_filter and not markdown_generator.content_filter:
# markdown_generator.content_filter = PruningContentFilter()
+ # Extract <base href> from raw HTML before it gets stripped by cleaning.
+ # This ensures relative URLs resolve correctly even with cleaned_html.
+ base_url = params.get("base_url") or params.get("redirected_url") or url
+ base_tag_match = re.search(r'<base\s[^>]*href\s*=\s*["\']([^"\']+)["\']', html, re.IGNORECASE)
+ if base_tag_match:
+ base_url = base_tag_match.group(1)
+
markdown_result: MarkdownGenerationResult = (
markdown_generator.generate_markdown(
input_html=markdown_input_html,
- # Use explicit base_url if provided (for raw: HTML), otherwise redirected_url, then url
- base_url=params.get("base_url") or params.get("redirected_url") or url
+ base_url=base_url
# html2text_options=kwargs.get('html2text', {})
)
)
@@ -808,25 +993,44 @@ async def arun_many(
print(f"Processed {result.url}: {len(result.markdown)} chars")
"""
config = config or CrawlerRunConfig()
- # if config is None:
- # config = CrawlerRunConfig(
- # word_count_threshold=word_count_threshold,
- # extraction_strategy=extraction_strategy,
- # chunking_strategy=chunking_strategy,
- # content_filter=content_filter,
- # cache_mode=cache_mode,
- # bypass_cache=bypass_cache,
- # css_selector=css_selector,
- # screenshot=screenshot,
- # pdf=pdf,
- # verbose=verbose,
- # **kwargs,
- # )
+
+ # When deep_crawl_strategy is set, bypass the dispatcher and call
+ # arun() directly for each URL. The DeepCrawlDecorator on arun()
+ # will invoke the strategy and return List[CrawlResult]. The
+ # dispatcher cannot handle that return type (it expects a single
+ # CrawlResult), so we must handle it here.
+ primary_cfg = config[0] if isinstance(config, list) else config
+ if getattr(primary_cfg, "deep_crawl_strategy", None):
+ if primary_cfg.stream:
+ async def _deep_crawl_stream():
+ for url in urls:
+ result = await self.arun(url, config=primary_cfg)
+ if isinstance(result, list):
+ for r in result:
+ yield r
+ else:
+ async for r in result:
+ yield r
+ return _deep_crawl_stream()
+ else:
+ all_results = []
+ for url in urls:
+ result = await self.arun(url, config=primary_cfg)
+ if isinstance(result, list):
+ all_results.extend(result)
+ else:
+ all_results.append(result)
+ return all_results
if dispatcher is None:
+ primary_cfg = config[0] if isinstance(config, list) else config
+ mean_delay = getattr(primary_cfg, "mean_delay", 0.1)
+ max_range = getattr(primary_cfg, "max_range", 0.3)
dispatcher = MemoryAdaptiveDispatcher(
rate_limiter=RateLimiter(
- base_delay=(1.0, 3.0), max_delay=60.0, max_retries=3
+ base_delay=(mean_delay, mean_delay + max_range),
+ max_delay=60.0,
+ max_retries=3,
),
)
diff --git a/crawl4ai/browser_manager.py b/crawl4ai/browser_manager.py
index fedc974f9..7ada68349 100644
--- a/crawl4ai/browser_manager.py
+++ b/crawl4ai/browser_manager.py
@@ -1,6 +1,6 @@
import asyncio
import time
-from typing import List, Optional
+from typing import Dict, List, Optional, Tuple
import os
import sys
import shutil
@@ -70,9 +70,6 @@ class ManagedBrowser:
def build_browser_flags(config: BrowserConfig) -> List[str]:
"""Common CLI flags for launching Chromium"""
flags = [
- "--disable-gpu",
- "--disable-gpu-compositing",
- "--disable-software-rasterizer",
"--no-sandbox",
"--disable-dev-shm-usage",
"--no-first-run",
@@ -88,7 +85,24 @@ def build_browser_flags(config: BrowserConfig) -> List[str]:
"--force-color-profile=srgb",
"--mute-audio",
"--disable-background-timer-throttling",
+ # Memory-saving flags: disable unused Chrome features
+ "--disable-features=OptimizationHints,MediaRouter,DialMediaRouteProvider",
+ "--disable-component-update",
+ "--disable-domain-reliability",
]
+ # GPU flags disable WebGL which anti-bot sensors detect as headless.
+ # Keep WebGL working (via SwiftShader) when stealth mode is active.
+ if not config.enable_stealth:
+ flags.extend([
+ "--disable-gpu",
+ "--disable-gpu-compositing",
+ "--disable-software-rasterizer",
+ ])
+ if config.memory_saving_mode:
+ flags.extend([
+ "--aggressive-cache-discard",
+ '--js-flags=--max-old-space-size=512',
+ ])
if config.light_mode:
flags.extend(BROWSER_DISABLE_OPTIONS)
if config.text_mode:
@@ -100,14 +114,13 @@ def build_browser_flags(config: BrowserConfig) -> List[str]:
"--disable-software-rasterizer",
"--disable-dev-shm-usage",
])
- # proxy support
+ # proxy support β only pass server URL, never credentials.
+ # Chromium's --proxy-server flag silently ignores inline user:pass@.
+ # Auth credentials are handled at the Playwright context level instead.
if config.proxy:
flags.append(f"--proxy-server={config.proxy}")
elif config.proxy_config:
- creds = ""
- if config.proxy_config.username and config.proxy_config.password:
- creds = f"{config.proxy_config.username}:{config.proxy_config.password}@"
- flags.append(f"--proxy-server={creds}{config.proxy_config.server}")
+ flags.append(f"--proxy-server={config.proxy_config.server}")
# dedupe
return list(dict.fromkeys(flags))
@@ -199,12 +212,18 @@ async def start(self) -> str:
p.wait(timeout=5)
else: # macOS / Linux
# kill any process listening on the same debugging port
- pids = (
- subprocess.check_output(shlex.split(f"lsof -t -i:{self.debugging_port}"))
- .decode()
- .strip()
- .splitlines()
- )
+ try:
+ pids = (
+ subprocess.check_output(
+ shlex.split(f"lsof -t -i:{self.debugging_port}"),
+ stderr=subprocess.DEVNULL,
+ )
+ .decode()
+ .strip()
+ .splitlines()
+ )
+ except (FileNotFoundError, subprocess.CalledProcessError):
+ pids = []
for pid in pids:
try:
os.kill(int(pid), signal.SIGTERM)
@@ -408,13 +427,17 @@ async def cleanup(self):
# Force kill if still running
if self.browser_process.poll() is None:
if sys.platform == "win32":
- # On Windows we might need taskkill for detached processes
+ # On Windows, use taskkill /T to kill the entire process tree
try:
- subprocess.run(["taskkill", "/F", "/PID", str(self.browser_process.pid)])
+ subprocess.run(["taskkill", "/F", "/T", "/PID", str(self.browser_process.pid)])
except Exception:
self.browser_process.kill()
else:
- self.browser_process.kill()
+ # On Unix, kill entire process group to reap child processes
+ try:
+ os.killpg(os.getpgid(self.browser_process.pid), signal.SIGKILL)
+ except (ProcessLookupError, OSError):
+ pass
await asyncio.sleep(0.1) # Brief wait for kill to take effect
except Exception as e:
@@ -559,6 +582,91 @@ async def clone_runtime_state(
+class _CDPConnectionCache:
+ """
+ Class-level cache for Playwright + CDP browser connections.
+
+ When enabled via BrowserConfig(cache_cdp_connection=True), multiple
+ BrowserManager instances connecting to the same cdp_url will share
+ a single Playwright subprocess and CDP WebSocket. Reference-counted;
+ the connection is closed when the last user releases it.
+ """
+
+ _cache: Dict[str, Tuple] = {} # cdp_url -> (playwright, browser, ref_count)
+ _lock: Optional[asyncio.Lock] = None # lazy-init to avoid event loop issues
+ _lock_loop: Optional[asyncio.AbstractEventLoop] = None
+
+ @classmethod
+ def _get_lock(cls) -> asyncio.Lock:
+ loop = asyncio.get_running_loop()
+ if cls._lock is None or cls._lock_loop is not loop:
+ cls._lock = asyncio.Lock()
+ cls._lock_loop = loop
+ return cls._lock
+
+ @classmethod
+ async def acquire(cls, cdp_url: str, use_undetected: bool = False):
+ """Get or create a cached (playwright, browser) for this cdp_url."""
+ async with cls._get_lock():
+ if cdp_url in cls._cache:
+ pw, browser, count = cls._cache[cdp_url]
+ if browser.is_connected():
+ cls._cache[cdp_url] = (pw, browser, count + 1)
+ return pw, browser
+ # Stale connection β clean up and fall through to create new
+ try:
+ await pw.stop()
+ except Exception:
+ pass
+ del cls._cache[cdp_url]
+
+ # Create new connection
+ if use_undetected:
+ from patchright.async_api import async_playwright
+ else:
+ from playwright.async_api import async_playwright
+ pw = await async_playwright().start()
+ browser = await pw.chromium.connect_over_cdp(cdp_url)
+ cls._cache[cdp_url] = (pw, browser, 1)
+ return pw, browser
+
+ @classmethod
+ async def release(cls, cdp_url: str):
+ """Decrement ref count; close connection when last user releases."""
+ async with cls._get_lock():
+ if cdp_url not in cls._cache:
+ return
+ pw, browser, count = cls._cache[cdp_url]
+ if count <= 1:
+ try:
+ await browser.close()
+ except Exception:
+ pass
+ try:
+ await pw.stop()
+ except Exception:
+ pass
+ del cls._cache[cdp_url]
+ else:
+ cls._cache[cdp_url] = (pw, browser, count - 1)
+
+ @classmethod
+ async def close_all(cls):
+ """Force-close all cached connections. Call on application shutdown."""
+ async with cls._get_lock():
+ for cdp_url in list(cls._cache.keys()):
+ pw, browser, _ = cls._cache[cdp_url]
+ try:
+ await browser.close()
+ except Exception:
+ pass
+ try:
+ await pw.stop()
+ except Exception:
+ pass
+ cls._cache.clear()
+
+
class BrowserManager:
"""
Manages the browser instance and context.
@@ -575,7 +683,20 @@ class BrowserManager:
"""
_playwright_instance = None
-
+
+ # Class-level tracking of pages in use, keyed by browser endpoint (CDP URL or instance id)
+ # This ensures multiple BrowserManager instances connecting to the same browser
+ # share the same page tracking, preventing race conditions.
+ _global_pages_in_use: dict = {} # endpoint_key -> set of pages
+ _global_pages_lock: asyncio.Lock = None # Initialized lazily
+
+ @classmethod
+ def _get_global_lock(cls) -> asyncio.Lock:
+ """Get or create the global pages lock (lazy initialization for async context)."""
+ if cls._global_pages_lock is None:
+ cls._global_pages_lock = asyncio.Lock()
+ return cls._global_pages_lock
+
@classmethod
async def get_playwright(cls, use_undetected: bool = False):
if use_undetected:
@@ -603,6 +724,8 @@ def __init__(self, browser_config: BrowserConfig, logger=None, use_undetected: b
self.default_context = None
self.managed_browser = None
self.playwright = None
+ self._using_cached_cdp = False
+ self._launched_persistent = False # True when using launch_persistent_context
# Session management
self.sessions = {}
@@ -611,12 +734,30 @@ def __init__(self, browser_config: BrowserConfig, logger=None, use_undetected: b
# Keep track of contexts by a "config signature," so each unique config reuses a single context
self.contexts_by_config = {}
self._contexts_lock = asyncio.Lock()
-
+
+ # Context lifecycle tracking for LRU eviction
+ self._context_refcounts = {} # sig -> int (active crawls using this context)
+ self._context_last_used = {} # sig -> float (monotonic timestamp for LRU)
+ self._page_to_sig = {} # page -> sig (for decrement lookup on release)
+ self._max_contexts = 20 # LRU eviction threshold
+
# Serialize context.new_page() across concurrent tasks to avoid races
# when using a shared persistent context (context.pages may be empty
# for all racers). Prevents 'Target page/context closed' errors.
self._page_lock = asyncio.Lock()
-
+
+ # Browser endpoint key for global page tracking (set after browser starts)
+ self._browser_endpoint_key: Optional[str] = None
+
+ # Browser recycling state (version-based approach)
+ self._pages_served = 0
+ self._browser_version = 1 # included in signature, bump to create new browser
+ self._pending_cleanup = {} # old_sig -> {"browser": browser, "contexts": [...], "done": Event}
+ self._pending_cleanup_lock = asyncio.Lock()
+ self._max_pending_browsers = 3 # safety cap β block if too many draining
+ self._cleanup_slot_available = asyncio.Event()
+ self._cleanup_slot_available.set() # starts open
+
# Stealth adapter for stealth mode
self._stealth_adapter = None
if self.config.enable_stealth and not self.use_undetected:
@@ -649,24 +790,106 @@ async def start(self):
"""
if self.playwright is not None:
await self.close()
-
- if self.use_undetected:
- from patchright.async_api import async_playwright
+
+ # Use cached CDP connection if enabled and cdp_url is set
+ if self.config.cache_cdp_connection and self.config.cdp_url:
+ self._using_cached_cdp = True
+ self.config.use_managed_browser = True
+ self.playwright, self.browser = await _CDPConnectionCache.acquire(
+ self.config.cdp_url, self.use_undetected
+ )
else:
- from playwright.async_api import async_playwright
+ self._using_cached_cdp = False
+ if self.use_undetected:
+ from patchright.async_api import async_playwright
+ else:
+ from playwright.async_api import async_playwright
+
+ # Initialize playwright
+ self.playwright = await async_playwright().start()
+
+ # ββ Persistent context via Playwright's native API ββββββββββββββ
+ # When use_persistent_context is set and we're not connecting to an
+ # external CDP endpoint, use launch_persistent_context() instead of
+ # subprocess + CDP. This properly supports proxy authentication
+ # (server + username + password) which the --proxy-server CLI flag
+ # cannot handle.
+ if (
+ self.config.use_persistent_context
+ and not self.config.cdp_url
+ and not self._using_cached_cdp
+ ):
+ # Collect stealth / optimization CLI flags, excluding ones that
+ # launch_persistent_context handles via keyword arguments.
+ _skip_prefixes = (
+ "--proxy-server",
+ "--remote-debugging-port",
+ "--user-data-dir",
+ "--headless",
+ "--window-size",
+ )
+ cli_args = [
+ flag
+ for flag in ManagedBrowser.build_browser_flags(self.config)
+ if not flag.startswith(_skip_prefixes)
+ ]
+ if self.config.extra_args:
+ cli_args.extend(self.config.extra_args)
+
+ launch_kwargs = {
+ "headless": self.config.headless,
+ "args": list(dict.fromkeys(cli_args)), # dedupe
+ "viewport": {
+ "width": self.config.viewport_width,
+ "height": self.config.viewport_height,
+ },
+ "user_agent": self.config.user_agent or None,
+ "ignore_https_errors": self.config.ignore_https_errors,
+ "accept_downloads": self.config.accept_downloads,
+ }
+
+ if self.config.proxy_config:
+ launch_kwargs["proxy"] = {
+ "server": self.config.proxy_config.server,
+ "username": self.config.proxy_config.username,
+ "password": self.config.proxy_config.password,
+ }
+
+ if self.config.storage_state:
+ launch_kwargs["storage_state"] = self.config.storage_state
+
+ user_data_dir = self.config.user_data_dir or tempfile.mkdtemp(
+ prefix="crawl4ai-persistent-"
+ )
- # Initialize playwright
- self.playwright = await async_playwright().start()
+ self.default_context = (
+ await self.playwright.chromium.launch_persistent_context(
+ user_data_dir, **launch_kwargs
+ )
+ )
+ self.browser = None # persistent context has no separate Browser
+ self._launched_persistent = True
+
+ await self.setup_context(self.default_context)
+
+ # Set the browser endpoint key for global page tracking
+ self._browser_endpoint_key = self._compute_browser_endpoint_key()
+ if self._browser_endpoint_key not in BrowserManager._global_pages_in_use:
+ BrowserManager._global_pages_in_use[self._browser_endpoint_key] = set()
+ return
if self.config.cdp_url or self.config.use_managed_browser:
self.config.use_managed_browser = True
- cdp_url = await self.managed_browser.start() if not self.config.cdp_url else self.config.cdp_url
- # Add CDP endpoint verification before connecting
- if not await self._verify_cdp_ready(cdp_url):
- raise Exception(f"CDP endpoint at {cdp_url} is not ready after startup")
+ if not self._using_cached_cdp:
+ cdp_url = await self.managed_browser.start() if not self.config.cdp_url else self.config.cdp_url
+
+ # Add CDP endpoint verification before connecting
+ if not await self._verify_cdp_ready(cdp_url):
+ raise Exception(f"CDP endpoint at {cdp_url} is not ready after startup")
+
+ self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)
- self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)
contexts = self.browser.contexts
# If browser_context_id is provided, we're using a pre-created context
@@ -716,6 +939,77 @@ async def start(self):
self.default_context = self.browser
+ # Set the browser endpoint key for global page tracking
+ self._browser_endpoint_key = self._compute_browser_endpoint_key()
+ # Initialize global tracking set for this endpoint if needed
+ if self._browser_endpoint_key not in BrowserManager._global_pages_in_use:
+ BrowserManager._global_pages_in_use[self._browser_endpoint_key] = set()
+
+ def _compute_browser_endpoint_key(self) -> str:
+ """
+ Compute a unique key identifying this browser connection.
+
+ For CDP connections, uses the normalized CDP URL so all BrowserManager
+ instances connecting to the same browser share page tracking.
+ For standalone browsers, uses instance id since each is independent.
+
+ Returns:
+ str: Unique identifier for this browser connection
+ """
+ # For CDP connections, use the CDP URL as the key (normalized)
+ if self.config.cdp_url:
+ return self._normalize_cdp_url(self.config.cdp_url)
+
+ # For managed browsers, use the CDP URL/port that was assigned
+ if self.managed_browser:
+ # Use debugging port as the key since it uniquely identifies the browser
+ port = getattr(self.managed_browser, 'debugging_port', None)
+ host = getattr(self.managed_browser, 'host', 'localhost')
+ if port:
+ return f"cdp:http://{host}:{port}"
+
+ # For standalone browsers, use instance id (no sharing needed)
+ return f"instance:{id(self)}"
+
+ def _normalize_cdp_url(self, cdp_url: str) -> str:
+ """
+ Normalize a CDP URL to a canonical form for consistent tracking.
+
+ Handles various formats:
+ - http://localhost:9222
+ - ws://localhost:9222/devtools/browser/xxx
+ - http://localhost:9222?browser_id=xxx
+
+ Returns:
+ str: Normalized CDP key in format "cdp:http://host:port"
+ """
+ from urllib.parse import urlparse
+
+ parsed = urlparse(cdp_url)
+ host = parsed.hostname or 'localhost'
+ port = parsed.port or 9222
+
+ return f"cdp:http://{host}:{port}"
+
+ def _get_pages_in_use(self) -> set:
+ """Get the set of pages currently in use for this browser."""
+ if self._browser_endpoint_key and self._browser_endpoint_key in BrowserManager._global_pages_in_use:
+ return BrowserManager._global_pages_in_use[self._browser_endpoint_key]
+ # Fallback: shouldn't happen, but return empty set
+ return set()
+
+ def _mark_page_in_use(self, page) -> None:
+ """Mark a page as in use."""
+ if self._browser_endpoint_key:
+ if self._browser_endpoint_key not in BrowserManager._global_pages_in_use:
+ BrowserManager._global_pages_in_use[self._browser_endpoint_key] = set()
+ BrowserManager._global_pages_in_use[self._browser_endpoint_key].add(page)
+
+ def _release_page_from_use(self, page) -> None:
+ """Release a page from the in-use tracking."""
+ if self._browser_endpoint_key and self._browser_endpoint_key in BrowserManager._global_pages_in_use:
+ BrowserManager._global_pages_in_use[self._browser_endpoint_key].discard(page)
+
async def _verify_cdp_ready(self, cdp_url: str) -> bool:
"""Verify CDP endpoint is ready with exponential backoff.
@@ -781,10 +1075,20 @@ def _build_browser_args(self) -> dict:
"--force-color-profile=srgb",
"--mute-audio",
"--disable-background-timer-throttling",
+ # Memory-saving flags: disable unused Chrome features
+ "--disable-features=OptimizationHints,MediaRouter,DialMediaRouteProvider",
+ "--disable-component-update",
+ "--disable-domain-reliability",
# "--single-process",
f"--window-size={self.config.viewport_width},{self.config.viewport_height}",
]
+ if self.config.memory_saving_mode:
+ args.extend([
+ "--aggressive-cache-discard",
+ '--js-flags=--max-old-space-size=512',
+ ])
+
if self.config.light_mode:
args.extend(BROWSER_DISABLE_OPTIONS)
@@ -925,6 +1229,17 @@ async def setup_context(
or crawlerRunConfig.magic
):
await context.add_init_script(load_js_script("navigator_overrider"))
+ context._crawl4ai_nav_overrider_injected = True
+
+ # Force-open closed shadow roots when flatten_shadow_dom is enabled
+ if crawlerRunConfig and crawlerRunConfig.flatten_shadow_dom:
+ await context.add_init_script("""
+ const _origAttachShadow = Element.prototype.attachShadow;
+ Element.prototype.attachShadow = function(init) {
+ return _origAttachShadow.call(this, {...init, mode: 'open'});
+ };
+ """)
+ context._crawl4ai_shadow_dom_injected = True
# Apply custom init_scripts from BrowserConfig (for stealth evasions, etc.)
if self.config.init_scripts:
@@ -939,6 +1254,12 @@ async def create_browser_context(self, crawlerRunConfig: CrawlerRunConfig = None
Returns:
Context: Browser context object with the specified configurations
"""
+ if self.browser is None:
+ raise RuntimeError(
+ "Cannot create new browser contexts when using "
+ "use_persistent_context=True. Persistent context uses a "
+ "single shared context."
+ )
# Base settings
user_agent = self.config.headers.get("User-Agent", self.config.user_agent)
viewport_settings = {
@@ -947,59 +1268,47 @@ async def create_browser_context(self, crawlerRunConfig: CrawlerRunConfig = None
}
proxy_settings = {"server": self.config.proxy} if self.config.proxy else None
- blocked_extensions = [
+ # CSS extensions (blocked separately via avoid_css flag)
+ css_extensions = ["css", "less", "scss", "sass"]
+
+ # Static resource extensions (blocked when text_mode is enabled)
+ static_extensions = [
# Images
- "jpg",
- "jpeg",
- "png",
- "gif",
- "webp",
- "svg",
- "ico",
- "bmp",
- "tiff",
- "psd",
+ "jpg", "jpeg", "png", "gif", "webp", "svg", "ico", "bmp", "tiff", "psd",
# Fonts
- "woff",
- "woff2",
- "ttf",
- "otf",
- "eot",
- # Styles
- # 'css', 'less', 'scss', 'sass',
+ "woff", "woff2", "ttf", "otf", "eot",
# Media
- "mp4",
- "webm",
- "ogg",
- "avi",
- "mov",
- "wmv",
- "flv",
- "m4v",
- "mp3",
- "wav",
- "aac",
- "m4a",
- "opus",
- "flac",
+ "mp4", "webm", "ogg", "avi", "mov", "wmv", "flv", "m4v",
+ "mp3", "wav", "aac", "m4a", "opus", "flac",
# Documents
- "pdf",
- "doc",
- "docx",
- "xls",
- "xlsx",
- "ppt",
- "pptx",
+ "pdf", "doc", "docx", "xls", "xlsx", "ppt", "pptx",
# Archives
- "zip",
- "rar",
- "7z",
- "tar",
- "gz",
+ "zip", "rar", "7z", "tar", "gz",
# Scripts and data
- "xml",
- "swf",
- "wasm",
+ "xml", "swf", "wasm",
+ ]
+
+ # Ad and tracker domain patterns (curated from uBlock/EasyList sources)
+ ad_tracker_patterns = [
+ "**/google-analytics.com/**",
+ "**/googletagmanager.com/**",
+ "**/googlesyndication.com/**",
+ "**/doubleclick.net/**",
+ "**/adservice.google.com/**",
+ "**/adsystem.com/**",
+ "**/adzerk.net/**",
+ "**/adnxs.com/**",
+ "**/ads.linkedin.com/**",
+ "**/facebook.net/**",
+ "**/analytics.twitter.com/**",
+ "**/ads-twitter.com/**",
+ "**/hotjar.com/**",
+ "**/clarity.ms/**",
+ "**/scorecardresearch.com/**",
+ "**/pixel.wp.com/**",
+ "**/amazon-adsystem.com/**",
+ "**/mixpanel.com/**",
+ "**/segment.com/**",
]
# Common context settings
@@ -1010,21 +1319,19 @@ async def create_browser_context(self, crawlerRunConfig: CrawlerRunConfig = None
"accept_downloads": self.config.accept_downloads,
"storage_state": self.config.storage_state,
"ignore_https_errors": self.config.ignore_https_errors,
- "device_scale_factor": 1.0,
+ "device_scale_factor": self.config.device_scale_factor,
"java_script_enabled": self.config.java_script_enabled,
}
if crawlerRunConfig:
# Check if there is value for crawlerRunConfig.proxy_config set add that to context
if crawlerRunConfig.proxy_config:
- proxy_settings = {
- "server": crawlerRunConfig.proxy_config.server,
- }
- if crawlerRunConfig.proxy_config.username:
- proxy_settings.update({
- "username": crawlerRunConfig.proxy_config.username,
- "password": crawlerRunConfig.proxy_config.password,
- })
+ from playwright.async_api import ProxySettings
+ proxy_settings = ProxySettings(
+ server=crawlerRunConfig.proxy_config.server,
+ username=crawlerRunConfig.proxy_config.username,
+ password=crawlerRunConfig.proxy_config.password,
+ )
context_settings["proxy"] = proxy_settings
if self.config.text_mode:
@@ -1055,48 +1362,103 @@ async def create_browser_context(self, crawlerRunConfig: CrawlerRunConfig = None
# Create and return the context with all settings
context = await self.browser.new_context(**context_settings)
- # Apply text mode settings if enabled
+ # Build dynamic blocking list based on config flags
+ to_block = []
+ if self.config.avoid_css:
+ to_block.extend(css_extensions)
if self.config.text_mode:
- # Create and apply route patterns for each extension
- for ext in blocked_extensions:
+ to_block.extend(static_extensions)
+
+ if to_block:
+ for ext in to_block:
await context.route(f"**/*.{ext}", lambda route: route.abort())
+
+ if self.config.avoid_ads:
+ for pattern in ad_tracker_patterns:
+ await context.route(pattern, lambda route: route.abort())
+
return context
def _make_config_signature(self, crawlerRunConfig: CrawlerRunConfig) -> str:
"""
- Converts the crawlerRunConfig into a dict, excludes ephemeral fields,
- then returns a hash of the sorted JSON. This yields a stable signature
- that identifies configurations requiring a unique browser context.
+ Hash ONLY the CrawlerRunConfig fields that affect browser context
+ creation (create_browser_context) or context setup (setup_context).
+
+ Whitelist approach: fields like css_selector, word_count_threshold,
+ screenshot, verbose, etc. do NOT cause a new context to be created.
"""
import json
- config_dict = crawlerRunConfig.__dict__.copy()
- # Exclude items that do not affect browser-level setup.
- # Expand or adjust as needed, e.g. chunking_strategy is purely for data extraction, not for browser config.
- ephemeral_keys = [
- "session_id",
- "js_code",
- "scraping_strategy",
- "extraction_strategy",
- "chunking_strategy",
- "cache_mode",
- "content_filter",
- "semaphore_count",
- "url"
- ]
-
- # Do NOT exclude locale, timezone_id, or geolocation as these DO affect browser context
- # and should cause a new context to be created if they change
-
- for key in ephemeral_keys:
- if key in config_dict:
- del config_dict[key]
- # Convert to canonical JSON string
- signature_json = json.dumps(config_dict, sort_keys=True, default=str)
+ sig_dict = {}
- # Hash the JSON so we get a compact, unique string
- signature_hash = hashlib.sha256(signature_json.encode("utf-8")).hexdigest()
- return signature_hash
+ # Fields that flow into create_browser_context()
+ pc = crawlerRunConfig.proxy_config
+ if pc is not None:
+ sig_dict["proxy_config"] = {
+ "server": getattr(pc, "server", None),
+ "username": getattr(pc, "username", None),
+ "password": getattr(pc, "password", None),
+ }
+ else:
+ sig_dict["proxy_config"] = None
+
+ sig_dict["locale"] = crawlerRunConfig.locale
+ sig_dict["timezone_id"] = crawlerRunConfig.timezone_id
+
+ geo = crawlerRunConfig.geolocation
+ if geo is not None:
+ sig_dict["geolocation"] = {
+ "latitude": geo.latitude,
+ "longitude": geo.longitude,
+ "accuracy": geo.accuracy,
+ }
+ else:
+ sig_dict["geolocation"] = None
+
+ # Fields that flow into setup_context() as init scripts
+ sig_dict["override_navigator"] = crawlerRunConfig.override_navigator
+ sig_dict["simulate_user"] = crawlerRunConfig.simulate_user
+ sig_dict["magic"] = crawlerRunConfig.magic
+
+ # Browser version β bumped on recycle to force new browser instance
+ sig_dict["_browser_version"] = self._browser_version
+
+ signature_json = json.dumps(sig_dict, sort_keys=True, default=str)
+ return hashlib.sha256(signature_json.encode("utf-8")).hexdigest()
+
+ def _evict_lru_context_locked(self):
+ """
+ If contexts exceed the limit, find the least-recently-used context
+ with zero active crawls and remove it from all tracking dicts.
+
+ MUST be called while holding self._contexts_lock.
+
+ Returns the BrowserContext to close (caller closes it OUTSIDE the
+ lock), or None if no eviction is needed or possible.
+ """
+ if len(self.contexts_by_config) <= self._max_contexts:
+ return None
+
+ # Sort candidates by last-used timestamp (oldest first)
+ candidates = sorted(
+ self._context_last_used.items(),
+ key=lambda item: item[1],
+ )
+ for evict_sig, _ in candidates:
+ if self._context_refcounts.get(evict_sig, 0) == 0:
+ ctx = self.contexts_by_config.pop(evict_sig, None)
+ self._context_refcounts.pop(evict_sig, None)
+ self._context_last_used.pop(evict_sig, None)
+ # Clean up stale page->sig mappings for evicted context
+ stale_pages = [
+ p for p, s in self._page_to_sig.items() if s == evict_sig
+ ]
+ for p in stale_pages:
+ del self._page_to_sig[p]
+ return ctx
+
+ # All contexts are in active use β cannot evict
+ return None
async def _apply_stealth_to_page(self, page):
"""Apply stealth to a page if stealth mode is enabled"""
@@ -1194,6 +1556,7 @@ async def get_page(self, crawlerRunConfig: CrawlerRunConfig):
# context reuse for multiple URLs with the same config (e.g., batch/deep crawls).
if self.config.create_isolated_context:
config_signature = self._make_config_signature(crawlerRunConfig)
+ to_close = None
async with self._contexts_lock:
if config_signature in self.contexts_by_config:
@@ -1202,14 +1565,44 @@ async def get_page(self, crawlerRunConfig: CrawlerRunConfig):
context = await self.create_browser_context(crawlerRunConfig)
await self.setup_context(context, crawlerRunConfig)
self.contexts_by_config[config_signature] = context
+ self._context_refcounts[config_signature] = 0
+ to_close = self._evict_lru_context_locked()
+
+ # Increment refcount INSIDE lock before releasing
+ self._context_refcounts[config_signature] = (
+ self._context_refcounts.get(config_signature, 0) + 1
+ )
+ self._context_last_used[config_signature] = time.monotonic()
+
+ # Close evicted context OUTSIDE lock
+ if to_close is not None:
+ try:
+ await to_close.close()
+ except Exception:
+ pass
# Always create a new page for each crawl (isolation for navigation)
- page = await context.new_page()
+ try:
+ page = await context.new_page()
+ except Exception:
+ async with self._contexts_lock:
+ if config_signature in self._context_refcounts:
+ self._context_refcounts[config_signature] = max(
+ 0, self._context_refcounts[config_signature] - 1
+ )
+ raise
await self._apply_stealth_to_page(page)
+ self._page_to_sig[page] = config_signature
elif self.config.storage_state:
- context = await self.create_browser_context(crawlerRunConfig)
+ tmp_context = await self.create_browser_context(crawlerRunConfig)
ctx = self.default_context # default context, one window only
- ctx = await clone_runtime_state(context, ctx, crawlerRunConfig, self.config)
+ ctx = await clone_runtime_state(tmp_context, ctx, crawlerRunConfig, self.config)
+ # Close the temporary context β only needed as a clone source
+ try:
+ await tmp_context.close()
+ except Exception:
+ pass
+ context = ctx # so (page, context) return value is correct
# Avoid concurrent new_page on shared persistent context
# See GH-1198: context.pages can be empty under races
async with self._page_lock:
@@ -1217,32 +1610,52 @@ async def get_page(self, crawlerRunConfig: CrawlerRunConfig):
await self._apply_stealth_to_page(page)
else:
context = self.default_context
- pages = context.pages
- page = next((p for p in pages if p.url == crawlerRunConfig.url), None)
- if not page:
- if pages:
- page = pages[0]
+
+ # Handle pre-existing target case (for reconnecting to specific CDP targets)
+ if self.config.browser_context_id and self.config.target_id:
+ page = await self._get_page_by_target_id(context, self.config.target_id)
+ if not page:
+ async with self._page_lock:
+ page = await context.new_page()
+ self._mark_page_in_use(page)
+ await self._apply_stealth_to_page(page)
else:
- # Double-check under lock to avoid TOCTOU and ensure only
- # one task calls new_page when pages=[] concurrently
+ # Mark pre-existing target as in use
+ self._mark_page_in_use(page)
+ else:
+ # For CDP connections (external browser), multiple Playwright connections
+ # create separate browser/context objects. Page reuse across connections
+ # isn't reliable because each connection sees different page objects.
+ # Always create new pages for CDP to avoid cross-connection race conditions.
+ if self.config.cdp_url and not self.config.use_managed_browser:
async with self._page_lock:
+ page = await context.new_page()
+ self._mark_page_in_use(page)
+ await self._apply_stealth_to_page(page)
+ else:
+ # For managed browsers (single process), page reuse is safe.
+ # Use lock to safely check for available pages and track usage.
+ # This prevents race conditions when multiple crawls run concurrently.
+ async with BrowserManager._get_global_lock():
pages = context.pages
- if pages:
- page = pages[0]
- elif self.config.browser_context_id and self.config.target_id:
- # Pre-existing context/target provided - use CDP to get the page
- # This handles the case where Playwright doesn't see the target yet
- page = await self._get_page_by_target_id(context, self.config.target_id)
- if not page:
- # Fallback: create new page in existing context
- page = await context.new_page()
- await self._apply_stealth_to_page(page)
+ pages_in_use = self._get_pages_in_use()
+ # Find first available page (exists and not currently in use)
+ available_page = next(
+ (p for p in pages if p not in pages_in_use),
+ None
+ )
+ if available_page:
+ page = available_page
else:
+ # No available pages - create a new one
page = await context.new_page()
await self._apply_stealth_to_page(page)
+ # Mark page as in use (global tracking)
+ self._mark_page_in_use(page)
else:
# Otherwise, check if we have an existing context for this config
config_signature = self._make_config_signature(crawlerRunConfig)
+ to_close = None
async with self._contexts_lock:
if config_signature in self.contexts_by_config:
@@ -1252,15 +1665,45 @@ async def get_page(self, crawlerRunConfig: CrawlerRunConfig):
context = await self.create_browser_context(crawlerRunConfig)
await self.setup_context(context, crawlerRunConfig)
self.contexts_by_config[config_signature] = context
+ self._context_refcounts[config_signature] = 0
+ to_close = self._evict_lru_context_locked()
+
+ # Increment refcount INSIDE lock before releasing
+ self._context_refcounts[config_signature] = (
+ self._context_refcounts.get(config_signature, 0) + 1
+ )
+ self._context_last_used[config_signature] = time.monotonic()
+
+ # Close evicted context OUTSIDE lock
+ if to_close is not None:
+ try:
+ await to_close.close()
+ except Exception:
+ pass
# Create a new page from the chosen context
- page = await context.new_page()
+ try:
+ page = await context.new_page()
+ except Exception:
+ async with self._contexts_lock:
+ if config_signature in self._context_refcounts:
+ self._context_refcounts[config_signature] = max(
+ 0, self._context_refcounts[config_signature] - 1
+ )
+ raise
await self._apply_stealth_to_page(page)
+ self._page_to_sig[page] = config_signature
# If a session_id is specified, store this session so we can reuse later
if crawlerRunConfig.session_id:
self.sessions[crawlerRunConfig.session_id] = (context, page, time.time())
+ self._pages_served += 1
+
+ # Check if browser recycle threshold is hit β bump version for next requests
+ # This happens AFTER incrementing counter so concurrent requests see correct count
+ await self._maybe_bump_browser_version()
+
return page, context
async def kill_session(self, session_id: str):
@@ -1272,11 +1715,235 @@ async def kill_session(self, session_id: str):
"""
if session_id in self.sessions:
context, page, _ = self.sessions[session_id]
+ self._release_page_from_use(page)
+ # Decrement context refcount for the session's page
+ should_close_context = False
+ async with self._contexts_lock:
+ sig = self._page_to_sig.pop(page, None)
+ if sig is not None and sig in self._context_refcounts:
+ self._context_refcounts[sig] = max(
+ 0, self._context_refcounts[sig] - 1
+ )
+ # Only close the context if no other pages are using it
+ # (refcount dropped to 0) AND we own the context (not managed)
+ if not self.config.use_managed_browser:
+ if self._context_refcounts.get(sig, 0) == 0:
+ self.contexts_by_config.pop(sig, None)
+ self._context_refcounts.pop(sig, None)
+ self._context_last_used.pop(sig, None)
+ should_close_context = True
await page.close()
- if not self.config.use_managed_browser:
+ if should_close_context:
await context.close()
del self.sessions[session_id]
+ def release_page(self, page):
+ """
+ Release a page from the in-use tracking set (global tracking).
+ Sync variant β does NOT decrement context refcount.
+ """
+ self._release_page_from_use(page)
+
+ async def release_page_with_context(self, page):
+ """
+ Release a page and decrement its context's refcount under the lock.
+
+ Should be called from the async crawl finally block instead of
+ release_page() so the context lifecycle is properly tracked.
+ """
+ self._release_page_from_use(page)
+ sig = None
+ refcount = -1
+ async with self._contexts_lock:
+ sig = self._page_to_sig.pop(page, None)
+ if sig is not None and sig in self._context_refcounts:
+ self._context_refcounts[sig] = max(
+ 0, self._context_refcounts[sig] - 1
+ )
+ refcount = self._context_refcounts[sig]
+
+ # Check if this signature belongs to an old browser waiting to be cleaned up
+ if sig is not None and refcount == 0:
+ await self._maybe_cleanup_old_browser(sig)
+
+ def _should_recycle(self) -> bool:
+ """Check if page threshold reached for browser recycling."""
+ limit = self.config.max_pages_before_recycle
+ if limit <= 0:
+ return False
+ return self._pages_served >= limit
+
+ async def _maybe_bump_browser_version(self):
+ """Bump browser version if threshold reached, moving old browser to pending cleanup.
+
+ New requests automatically get a new browser (via new signature).
+ Old browser drains naturally and gets cleaned up when refcount hits 0.
+ """
+ if not self._should_recycle():
+ return
+
+ # Safety cap: wait if too many old browsers are draining
+ while True:
+ async with self._pending_cleanup_lock:
+ # Re-check threshold under lock (another request may have bumped already)
+ if not self._should_recycle():
+ return
+
+ # Check safety cap
+ if len(self._pending_cleanup) >= self._max_pending_browsers:
+ if self.logger:
+ self.logger.debug(
+ message="Waiting for old browser to drain (pending: {count})",
+ tag="BROWSER",
+ params={"count": len(self._pending_cleanup)},
+ )
+ self._cleanup_slot_available.clear()
+ # Release lock and wait
+ else:
+ # We have a slot β do the bump inside this lock hold
+ old_version = self._browser_version
+ active_sigs = []
+ idle_sigs = []
+ async with self._contexts_lock:
+ for sig in list(self._context_refcounts.keys()):
+ if self._context_refcounts.get(sig, 0) > 0:
+ active_sigs.append(sig)
+ else:
+ idle_sigs.append(sig)
+
+ if self.logger:
+ self.logger.info(
+ message="Bumping browser version {old} -> {new} after {count} pages ({active} active, {idle} idle sigs)",
+ tag="BROWSER",
+ params={
+ "old": old_version,
+ "new": old_version + 1,
+ "count": self._pages_served,
+ "active": len(active_sigs),
+ "idle": len(idle_sigs),
+ },
+ )
+
+ # Only add sigs with active crawls to pending cleanup.
+ # Sigs with refcount 0 are cleaned up immediately below
+ # to avoid them being stuck in _pending_cleanup forever
+ # (no future release would trigger their cleanup).
+ done_event = asyncio.Event()
+ for sig in active_sigs:
+ self._pending_cleanup[sig] = {
+ "version": old_version,
+ "done": done_event,
+ }
+
+ # Bump version β new get_page() calls will create new contexts
+ self._browser_version += 1
+ self._pages_served = 0
+
+ # Clean up idle sigs immediately (outside pending_cleanup_lock below)
+ break # exit while loop to do cleanup outside locks
+
+ # Safety cap path: wait for a cleanup slot, then retry.
+ # Timeout prevents permanent deadlock if stuck entries never drain.
+ try:
+ await asyncio.wait_for(
+ self._cleanup_slot_available.wait(), timeout=30.0
+ )
+ except asyncio.TimeoutError:
+ # Force-clean any pending entries that have refcount 0
+ # (they're stuck and will never drain naturally)
+ async with self._pending_cleanup_lock:
+ stuck_sigs = [
+ s for s in list(self._pending_cleanup.keys())
+ if self._context_refcounts.get(s, 0) == 0
+ ]
+ for sig in stuck_sigs:
+ self._pending_cleanup.pop(sig, None)
+ if stuck_sigs:
+ if self.logger:
+ self.logger.warning(
+ message="Force-cleaned {count} stuck pending entries after timeout",
+ tag="BROWSER",
+ params={"count": len(stuck_sigs)},
+ )
+ # Clean up the stuck contexts
+ for sig in stuck_sigs:
+ async with self._contexts_lock:
+ context = self.contexts_by_config.pop(sig, None)
+ self._context_refcounts.pop(sig, None)
+ self._context_last_used.pop(sig, None)
+ if context is not None:
+ try:
+ await context.close()
+ except Exception:
+ pass
+ if len(self._pending_cleanup) < self._max_pending_browsers:
+ self._cleanup_slot_available.set()
+
+ # Reached via break β clean up idle sigs immediately (outside locks)
+ for sig in idle_sigs:
+ async with self._contexts_lock:
+ context = self.contexts_by_config.pop(sig, None)
+ self._context_refcounts.pop(sig, None)
+ self._context_last_used.pop(sig, None)
+ if context is not None:
+ try:
+ await context.close()
+ except Exception:
+ pass
+ if idle_sigs and self.logger:
+ self.logger.debug(
+ message="Immediately cleaned up {count} idle contexts from version {version}",
+ tag="BROWSER",
+ params={"count": len(idle_sigs), "version": old_version},
+ )
+
+ async def _maybe_cleanup_old_browser(self, sig: str):
+ """Clean up an old browser's context if its refcount hit 0 and it's pending cleanup."""
+ async with self._pending_cleanup_lock:
+ if sig not in self._pending_cleanup:
+ return # Not an old browser signature
+
+ cleanup_info = self._pending_cleanup.pop(sig)
+ old_version = cleanup_info["version"]
+
+ if self.logger:
+ self.logger.debug(
+ message="Cleaning up context from browser version {version} (sig: {sig})",
+ tag="BROWSER",
+ params={"version": old_version, "sig": sig[:12]},
+ )
+
+ # Remove context from tracking
+ async with self._contexts_lock:
+ context = self.contexts_by_config.pop(sig, None)
+ self._context_refcounts.pop(sig, None)
+ self._context_last_used.pop(sig, None)
+
+ # Close context outside locks
+ if context is not None:
+ try:
+ await context.close()
+ except Exception:
+ pass
+
+ # Check if any signatures from this old version remain
+ remaining_old = [
+ s for s, info in self._pending_cleanup.items()
+ if info["version"] == old_version
+ ]
+
+ if not remaining_old:
+ if self.logger:
+ self.logger.info(
+ message="All contexts from browser version {version} cleaned up",
+ tag="BROWSER",
+ params={"version": old_version},
+ )
+
+ # Open a cleanup slot if we're below the cap
+ if len(self._pending_cleanup) < self._max_pending_browsers:
+ self._cleanup_slot_available.set()
+
def _cleanup_expired_sessions(self):
"""Clean up expired sessions based on TTL."""
current_time = time.time()
@@ -1290,6 +1957,27 @@ def _cleanup_expired_sessions(self):
async def close(self):
"""Close all browser resources and clean up."""
+ # Cached CDP path: only clean up this instance's sessions/contexts,
+ # then release the shared connection reference.
+ if self._using_cached_cdp:
+ session_ids = list(self.sessions.keys())
+ for session_id in session_ids:
+ await self.kill_session(session_id)
+ for ctx in self.contexts_by_config.values():
+ try:
+ await ctx.close()
+ except Exception:
+ pass
+ self.contexts_by_config.clear()
+ self._context_refcounts.clear()
+ self._context_last_used.clear()
+ self._page_to_sig.clear()
+ await _CDPConnectionCache.release(self.config.cdp_url)
+ self.browser = None
+ self.playwright = None
+ self._using_cached_cdp = False
+ return
+
if self.config.cdp_url:
# When using external CDP, we don't own the browser process.
# If cdp_cleanup_on_close is True, properly disconnect from the browser
@@ -1307,6 +1995,9 @@ async def close(self):
except Exception:
pass
self.contexts_by_config.clear()
+ self._context_refcounts.clear()
+ self._context_last_used.clear()
+ self._page_to_sig.clear()
# Disconnect from browser (doesn't terminate it, just releases connection)
if self.browser:
@@ -1321,7 +2012,8 @@ async def close(self):
)
self.browser = None
# Allow time for CDP connection to fully release before another client connects
- await asyncio.sleep(1.0)
+ if self.config.cdp_close_delay > 0:
+ await asyncio.sleep(self.config.cdp_close_delay)
# Stop Playwright instance to prevent memory leaks
if self.playwright:
@@ -1329,6 +2021,35 @@ async def close(self):
self.playwright = None
return
+ # ββ Persistent context launched via launch_persistent_context ββ
+ if self._launched_persistent:
+ session_ids = list(self.sessions.keys())
+ for session_id in session_ids:
+ await self.kill_session(session_id)
+ for ctx in self.contexts_by_config.values():
+ try:
+ await ctx.close()
+ except Exception:
+ pass
+ self.contexts_by_config.clear()
+ self._context_refcounts.clear()
+ self._context_last_used.clear()
+ self._page_to_sig.clear()
+
+ # Closing the persistent context also terminates the browser
+ if self.default_context:
+ try:
+ await self.default_context.close()
+ except Exception:
+ pass
+ self.default_context = None
+
+ if self.playwright:
+ await self.playwright.stop()
+ self.playwright = None
+ self._launched_persistent = False
+ return
+
if self.config.sleep_on_close:
await asyncio.sleep(0.5)
@@ -1347,6 +2068,9 @@ async def close(self):
params={"error": str(e)}
)
self.contexts_by_config.clear()
+ self._context_refcounts.clear()
+ self._context_last_used.clear()
+ self._page_to_sig.clear()
if self.browser:
await self.browser.close()
diff --git a/crawl4ai/browser_profiler.py b/crawl4ai/browser_profiler.py
index 1a961e037..c944a596c 100644
--- a/crawl4ai/browser_profiler.py
+++ b/crawl4ai/browser_profiler.py
@@ -15,7 +15,9 @@
import json
import subprocess
import time
-from typing import List, Dict, Optional, Any
+from enum import Enum
+from pathlib import Path
+from typing import List, Dict, Optional, Any, Set
from rich.console import Console
from .async_configs import BrowserConfig
@@ -24,6 +26,111 @@
from .utils import get_home_folder
+class ShrinkLevel(str, Enum):
+ """Profile shrink aggressiveness levels."""
+ NONE = "none" # Keep everything
+ LIGHT = "light" # Remove caches only
+ MEDIUM = "medium" # Caches + history/favicons
+ AGGRESSIVE = "aggressive" # Auth only (recommended)
+ MINIMAL = "minimal" # Cookies + localStorage only
+
+
+# Whitelist: what to KEEP at each level (everything else gets deleted)
+# Note: "Cookies" can be at root (older Chrome) or in Network/ (Chrome 96+)
+# storage_state.json is Playwright's portable cookie format (unencrypted)
+# It MUST be kept in all levels for cross-machine profile portability
+KEEP_PATTERNS: Dict[ShrinkLevel, Set[str]] = {
+ ShrinkLevel.NONE: {"*"},
+ ShrinkLevel.LIGHT: {
+ "Network", "Cookies", "Local Storage", "Session Storage", "IndexedDB",
+ "Preferences", "Secure Preferences", "Login Data", "Login Data For Account",
+ "Web Data", "History", "History-journal", "Visited Links", "Bookmarks",
+ "TransportSecurity", "Trust Tokens", "storage_state.json",
+ },
+ ShrinkLevel.MEDIUM: {
+ "Network", "Cookies", "Local Storage", "Session Storage", "IndexedDB",
+ "Preferences", "Secure Preferences", "Login Data", "Login Data For Account",
+ "Web Data", "TransportSecurity", "storage_state.json",
+ },
+ ShrinkLevel.AGGRESSIVE: {"Network", "Cookies", "Local Storage", "IndexedDB", "Preferences", "storage_state.json"},
+ ShrinkLevel.MINIMAL: {"Network", "Cookies", "Local Storage", "storage_state.json"},
+}
+
+
+def _get_size(path: Path) -> int:
+ """Get total size in bytes. Works for files and directories."""
+ if path.is_file():
+ return path.stat().st_size
+ total = 0
+ try:
+ for f in path.rglob("*"):
+ if f.is_file():
+ total += f.stat().st_size
+ except (PermissionError, OSError):
+ pass
+ return total
+
+
+def _format_size(n: int) -> str:
+ """Format bytes as human-readable string."""
+ for u in ("B", "KB", "MB", "GB"):
+ if n < 1024:
+ return f"{n:.1f} {u}"
+ n /= 1024
+ return f"{n:.1f} TB"
+
+
+def shrink_profile(
+ profile_path: str,
+ level: ShrinkLevel = ShrinkLevel.AGGRESSIVE,
+ dry_run: bool = False
+) -> Dict[str, Any]:
+ """
+ Shrink a Chrome profile to reduce storage while preserving auth data.
+
+ Args:
+ profile_path: Path to profile directory
+ level: How aggressively to shrink (LIGHT/MEDIUM/AGGRESSIVE/MINIMAL)
+ dry_run: If True, only report what would be removed
+
+ Returns:
+ Dict with 'removed', 'kept', 'bytes_freed', 'size_before', 'size_after', 'errors'
+ """
+ if level == ShrinkLevel.NONE:
+ return {"removed": [], "kept": [], "bytes_freed": 0, "errors": []}
+
+ profile = Path(profile_path)
+ if not profile.exists() or not profile.is_dir():
+ raise ValueError(f"Profile not found: {profile_path}")
+
+ # Chrome profiles may have data in Default/ subdirectory
+ target = profile / "Default" if (profile / "Default").is_dir() else profile
+
+ keep = KEEP_PATTERNS[level]
+ result = {"removed": [], "kept": [], "bytes_freed": 0, "errors": [], "size_before": _get_size(profile)}
+
+ for item in target.iterdir():
+ name = item.name
+ # Check if item matches any keep pattern
+ if any(name == p or name.startswith(p) for p in keep):
+ result["kept"].append(name)
+ else:
+ size = _get_size(item)
+ if not dry_run:
+ try:
+ shutil.rmtree(item) if item.is_dir() else item.unlink()
+ result["removed"].append(name)
+ result["bytes_freed"] += size
+ except Exception as e:
+ result["errors"].append(f"{name}: {e}")
+ else:
+ result["removed"].append(name)
+ result["bytes_freed"] += size
+
+ result["size_after"] = _get_size(profile) if not dry_run else None
+ return result
+
+
class BrowserProfiler:
"""
A dedicated class for managing browser profiles for Crawl4AI.
@@ -272,9 +379,12 @@ def input_thread():
self.logger.error(f"Fallback listener failed: {e}", tag=tag)
user_done_event.set()
- async def create_profile(self,
- profile_name: Optional[str] = None,
- browser_config: Optional[BrowserConfig] = None) -> Optional[str]:
+ async def create_profile(
+ self,
+ profile_name: Optional[str] = None,
+ browser_config: Optional[BrowserConfig] = None,
+ shrink_level: ShrinkLevel = ShrinkLevel.NONE,
+ ) -> Optional[str]:
"""
Creates a browser profile by launching a browser for interactive user setup
and waits until the user closes it. The profile is stored in a directory that
@@ -285,6 +395,8 @@ async def create_profile(self,
If None, a name is generated based on timestamp.
browser_config (BrowserConfig, optional): Configuration for the browser.
If None, a default configuration is used with headless=False.
+ shrink_level (ShrinkLevel, optional): Optionally shrink profile after creation.
+ Default is NONE (no shrinking).
Returns:
str: Path to the created profile directory, or None if creation failed
@@ -311,16 +423,30 @@ async def create_profile(self,
```
"""
# Create default browser config if none provided
+ # IMPORTANT: We disable cookie encryption so profiles can be transferred
+ # between machines (e.g., local -> cloud). Without this, Chrome encrypts
+ # cookies with OS keychain which isn't portable.
+ portable_profile_args = [
+ "--password-store=basic", # Linux: use basic store, not gnome-keyring
+ "--use-mock-keychain", # macOS: use mock keychain, not real one
+ ]
+
if browser_config is None:
from .async_configs import BrowserConfig
browser_config = BrowserConfig(
browser_type="chromium",
headless=False, # Must be visible for user interaction
- verbose=True
+ verbose=True,
+ extra_args=portable_profile_args,
)
else:
# Ensure headless is False for user interaction
browser_config.headless = False
+ # Add portable profile args
+ if browser_config.extra_args:
+ browser_config.extra_args.extend(portable_profile_args)
+ else:
+ browser_config.extra_args = portable_profile_args
# Generate profile name if not provided
if not profile_name:
@@ -487,8 +613,12 @@ async def check_browser_process():
# Make sure browser is fully cleaned up
await managed_browser.cleanup()
-
- # Return the profile path
+
+ # Shrink profile if requested
+ if shrink_level != ShrinkLevel.NONE and profile_path:
+ self.logger.info(f"Shrinking profile with level: {shrink_level.value}", tag="PROFILE")
+ self.shrink(profile_path, shrink_level)
+
return profile_path
def list_profiles(self) -> List[Dict[str, Any]]:
@@ -646,7 +776,44 @@ def delete_profile(self, profile_name_or_path: str) -> bool:
return True
except Exception:
return False
-
+
+ def shrink(
+ self,
+ profile_name_or_path: str,
+ level: ShrinkLevel = ShrinkLevel.AGGRESSIVE,
+ dry_run: bool = False
+ ) -> Dict[str, Any]:
+ """
+ Shrink a profile to reduce storage while preserving authentication data.
+
+ Args:
+ profile_name_or_path: Profile name or full path
+ level: LIGHT, MEDIUM, AGGRESSIVE (default), or MINIMAL
+ dry_run: If True, only preview what would be removed
+
+ Returns:
+ Dict with 'removed', 'kept', 'bytes_freed', 'size_before', 'size_after', 'errors'
+ """
+ # Resolve path
+ if os.path.isabs(profile_name_or_path):
+ profile_path = profile_name_or_path
+ else:
+ profile_path = os.path.join(self.profiles_dir, profile_name_or_path)
+
+ if not os.path.isdir(profile_path):
+ raise ValueError(f"Profile not found: {profile_name_or_path}")
+
+ result = shrink_profile(profile_path, level, dry_run)
+
+ action = "Would free" if dry_run else "Freed"
+ self.logger.info(
+ f"{action} {_format_size(result['bytes_freed'])} "
+ f"({len(result['removed'])} items removed, {len(result['kept'])} kept)",
+ tag="SHRINK"
+ )
+
+ return result
+
async def interactive_manager(self, crawl_callback=None):
"""
Launch an interactive profile management console.
diff --git a/crawl4ai/cli.py b/crawl4ai/cli.py
index 51b535002..02b67155e 100644
--- a/crawl4ai/cli.py
+++ b/crawl4ai/cli.py
@@ -15,15 +15,15 @@
from crawl4ai import (
CacheMode,
- AsyncWebCrawler,
+ AsyncWebCrawler,
CrawlResult,
- BrowserConfig,
+ BrowserConfig,
CrawlerRunConfig,
- LLMExtractionStrategy,
+ LLMExtractionStrategy,
LXMLWebScrapingStrategy,
JsonCssExtractionStrategy,
JsonXPathExtractionStrategy,
- BM25ContentFilter,
+ BM25ContentFilter,
PruningContentFilter,
BrowserProfiler,
DefaultMarkdownGenerator,
@@ -32,7 +32,9 @@
DFSDeepCrawlStrategy,
BestFirstCrawlingStrategy,
)
+from crawl4ai.browser_profiler import ShrinkLevel, _format_size
from crawl4ai.config import USER_SETTINGS
+from crawl4ai.cloud import cloud_cmd
from litellm import completion
from pathlib import Path
@@ -53,7 +55,7 @@ def get_global_config() -> dict:
def save_global_config(config: dict):
config_file = Path.home() / ".crawl4ai" / "global.yml"
- with open(config_file, "w") as f:
+ with open(config_file, "w", encoding="utf-8") as f:
yaml.dump(config, f)
def setup_llm_config() -> tuple[str, str]:
@@ -521,11 +523,15 @@ async def crawl_with_profile_cli(profile_path, url):
# Run the crawler
result = await run_crawler(url, browser_cfg, crawler_cfg, True)
+ # Get JSON output config
+ config = get_global_config()
+ ensure_ascii = config.get("JSON_ENSURE_ASCII", USER_SETTINGS["JSON_ENSURE_ASCII"]["default"])
+
# Handle output
if output_format == "all":
- console.print(json.dumps(result.model_dump(), indent=2))
+ console.print(json.dumps(result.model_dump(), indent=2, ensure_ascii=ensure_ascii))
elif output_format == "json":
- console.print(json.dumps(json.loads(result.extracted_content), indent=2))
+ console.print(json.dumps(json.loads(result.extracted_content), indent=2, ensure_ascii=ensure_ascii))
elif output_format in ["markdown", "md"]:
console.print(result.markdown.raw_markdown)
elif output_format == "title":
@@ -624,6 +630,9 @@ def cli():
"""Crawl4AI CLI - Web content extraction and browser profile management tool"""
pass
+# Add cloud command group
+cli.add_command(cloud_cmd)
+
@cli.group("browser")
def browser_cmd():
@@ -1019,11 +1028,12 @@ def cdp_cmd(user_data_dir: Optional[str], port: int, browser_type: str, headless
@click.option("--profile", "-p", help="Use a specific browser profile (by name)")
@click.option("--deep-crawl", type=click.Choice(["bfs", "dfs", "best-first"]), help="Enable deep crawling with specified strategy (bfs, dfs, or best-first)")
@click.option("--max-pages", type=int, default=10, help="Maximum number of pages to crawl in deep crawl mode")
-def crawl_cmd(url: str, browser_config: str, crawler_config: str, filter_config: str,
+@click.option("--json-ensure-ascii/--no-json-ensure-ascii", default=None, help="Escape non-ASCII characters in JSON output (default: from global config)")
+def crawl_cmd(url: str, browser_config: str, crawler_config: str, filter_config: str,
extraction_config: str, json_extract: str, schema: str, browser: Dict, crawler: Dict,
- output: str, output_file: str, bypass_cache: bool, question: str, verbose: bool, profile: str, deep_crawl: str, max_pages: int):
+ output: str, output_file: str, bypass_cache: bool, question: str, verbose: bool, profile: str, deep_crawl: str, max_pages: int, json_ensure_ascii: Optional[bool]):
"""Crawl a website and extract content
-
+
Simple Usage:
crwl crawl https://example.com
"""
@@ -1186,7 +1196,13 @@ def crawl_cmd(url: str, browser_config: str, crawler_config: str, filter_config:
browser_cfg.verbose = config.get("VERBOSE", False)
crawler_cfg.verbose = config.get("VERBOSE", False)
-
+
+ # Get JSON output config (priority: CLI flag > global config)
+ if json_ensure_ascii is not None:
+ ensure_ascii = json_ensure_ascii
+ else:
+ ensure_ascii = config.get("JSON_ENSURE_ASCII", USER_SETTINGS["JSON_ENSURE_ASCII"]["default"])
+
# Run crawler
result : CrawlResult = anyio.run(
run_crawler,
@@ -1221,35 +1237,59 @@ def crawl_cmd(url: str, browser_config: str, crawler_config: str, filter_config:
if output == "all":
if isinstance(result, list):
output_data = [r.model_dump() for r in all_results]
- click.echo(json.dumps(output_data, indent=2))
+ click.echo(json.dumps(output_data, indent=2, ensure_ascii=ensure_ascii))
else:
- click.echo(json.dumps(main_result.model_dump(), indent=2))
+ click.echo(json.dumps(main_result.model_dump(), indent=2, ensure_ascii=ensure_ascii))
elif output == "json":
print(main_result.extracted_content)
extracted_items = json.loads(main_result.extracted_content)
- click.echo(json.dumps(extracted_items, indent=2))
-
+ click.echo(json.dumps(extracted_items, indent=2, ensure_ascii=ensure_ascii))
+
elif output in ["markdown", "md"]:
- click.echo(main_result.markdown.raw_markdown)
+ if isinstance(result, list):
+ # Combine markdown from all crawled pages for deep crawl
+ for r in all_results:
+ click.echo(f"\n\n{'='*60}\n# {r.url}\n{'='*60}\n\n")
+ click.echo(r.markdown.raw_markdown)
+ else:
+ click.echo(main_result.markdown.raw_markdown)
elif output in ["markdown-fit", "md-fit"]:
- click.echo(main_result.markdown.fit_markdown)
+ if isinstance(result, list):
+ # Combine fit markdown from all crawled pages for deep crawl
+ for r in all_results:
+ click.echo(f"\n\n{'='*60}\n# {r.url}\n{'='*60}\n\n")
+ click.echo(r.markdown.fit_markdown)
+ else:
+ click.echo(main_result.markdown.fit_markdown)
else:
if output == "all":
- with open(output_file, "w") as f:
+ with open(output_file, "w", encoding="utf-8") as f:
if isinstance(result, list):
output_data = [r.model_dump() for r in all_results]
- f.write(json.dumps(output_data, indent=2))
+ f.write(json.dumps(output_data, indent=2, ensure_ascii=ensure_ascii))
else:
- f.write(json.dumps(main_result.model_dump(), indent=2))
+ f.write(json.dumps(main_result.model_dump(), indent=2, ensure_ascii=ensure_ascii))
elif output == "json":
- with open(output_file, "w") as f:
+ with open(output_file, "w", encoding="utf-8") as f:
f.write(main_result.extracted_content)
elif output in ["markdown", "md"]:
- with open(output_file, "w") as f:
- f.write(main_result.markdown.raw_markdown)
+ with open(output_file, "w", encoding="utf-8") as f:
+ if isinstance(result, list):
+ # Combine markdown from all crawled pages for deep crawl
+ for r in all_results:
+ f.write(f"\n\n{'='*60}\n# {r.url}\n{'='*60}\n\n")
+ f.write(r.markdown.raw_markdown)
+ else:
+ f.write(main_result.markdown.raw_markdown)
elif output in ["markdown-fit", "md-fit"]:
- with open(output_file, "w") as f:
- f.write(main_result.markdown.fit_markdown)
+ with open(output_file, "w", encoding="utf-8") as f:
+ if isinstance(result, list):
+ # Combine fit markdown from all crawled pages for deep crawl
+ for r in all_results:
+ f.write(f"\n\n{'='*60}\n# {r.url}\n{'='*60}\n\n")
+ f.write(r.markdown.fit_markdown)
+ else:
+ f.write(main_result.markdown.fit_markdown)
except Exception as e:
raise click.ClickException(str(e))
@@ -1373,17 +1413,159 @@ def config_set_cmd(key: str, value: str):
console.print(f"[green]Successfully set[/green] [cyan]{key}[/cyan] = [green]{display_value}[/green]")
-@cli.command("profiles")
-def profiles_cmd():
- """Manage browser profiles interactively
-
+@cli.group("profiles", invoke_without_command=True)
+@click.pass_context
+def profiles_cmd(ctx):
+ """Manage browser profiles for authenticated crawling
+
Launch an interactive browser profile manager where you can:
- List all existing profiles
- Create new profiles for authenticated browsing
- Delete unused profiles
+
+ Subcommands:
+ crwl profiles create <name> - Create a new profile
+ crwl profiles list - List all profiles
+ crwl profiles delete <name> - Delete a profile
+
+ Or run without subcommand for interactive menu:
+ crwl profiles
+ """
+ # If no subcommand provided, run interactive manager
+ if ctx.invoked_subcommand is None:
+ anyio.run(manage_profiles)
+
+
+@profiles_cmd.command("create")
+@click.argument("name")
+def profiles_create_cmd(name: str):
+ """Create a new browser profile
+
+ Opens a browser window for you to log in and set up your identity.
+ Press 'q' in the terminal when finished to save the profile.
+
+ Example:
+ crwl profiles create github-auth
+ """
+ profiler = BrowserProfiler()
+ console.print(Panel(f"[bold cyan]Creating Profile: {name}[/bold cyan]\n"
+ "A browser window will open for you to set up your identity.\n"
+ "Log in to sites, adjust settings, then press 'q' to save.",
+ border_style="cyan"))
+
+ async def _create():
+ try:
+ profile_path = await profiler.create_profile(name)
+ if profile_path:
+ console.print(f"[green]Profile successfully created at:[/green] {profile_path}")
+ else:
+ console.print("[red]Failed to create profile.[/red]")
+ sys.exit(1)
+ except Exception as e:
+ console.print(f"[red]Error creating profile: {str(e)}[/red]")
+ sys.exit(1)
+
+ anyio.run(_create)
+
+
+@profiles_cmd.command("list")
+def profiles_list_cmd():
+ """List all browser profiles
+
+ Example:
+ crwl profiles list
+ """
+ profiler = BrowserProfiler()
+ profiles = profiler.list_profiles()
+ display_profiles_table(profiles)
+
+
+@profiles_cmd.command("delete")
+@click.argument("name")
+@click.option("--force", "-f", is_flag=True, help="Skip confirmation")
+def profiles_delete_cmd(name: str, force: bool):
+ """Delete a browser profile
+
+ Example:
+ crwl profiles delete old-profile
+ crwl profiles delete old-profile --force
+ """
+ profiler = BrowserProfiler()
+
+ # Find profile by name
+ profiles = profiler.list_profiles()
+ profile = next((p for p in profiles if p["name"] == name), None)
+
+ if not profile:
+ console.print(f"[red]Profile not found:[/red] {name}")
+ sys.exit(1)
+
+ if not force:
+ if not Confirm.ask(f"[yellow]Delete profile '{name}'?[/yellow]"):
+ console.print("[cyan]Cancelled.[/cyan]")
+ return
+
+ try:
+ profiler.delete_profile(name)
+ console.print(f"[green]Profile '{name}' deleted successfully.[/green]")
+ except Exception as e:
+ console.print(f"[red]Error deleting profile: {str(e)}[/red]")
+ sys.exit(1)
+
+
+@cli.command("shrink")
+@click.argument("profile_name")
+@click.option(
+ "--level", "-l",
+ type=click.Choice(["light", "medium", "aggressive", "minimal"]),
+ default="aggressive",
+ help="Shrink level (default: aggressive)"
+)
+@click.option("--dry-run", "-n", is_flag=True, help="Preview without removing files")
+def shrink_cmd(profile_name: str, level: str, dry_run: bool):
+ """Shrink a browser profile to reduce storage.
+
+ Removes cache, history, and other non-essential data while preserving
+ authentication (cookies, localStorage, IndexedDB).
+
+ Shrink levels:
+ light - Remove caches only
+ medium - Remove caches + history
+ aggressive - Keep only auth data (recommended)
+ minimal - Keep only cookies + localStorage
+
+ Examples:
+ crwl shrink my_profile
+ crwl shrink my_profile --level minimal
+ crwl shrink my_profile --dry-run
"""
- # Run interactive profile manager
- anyio.run(manage_profiles)
+ profiler = BrowserProfiler()
+
+ try:
+ result = profiler.shrink(profile_name, ShrinkLevel(level), dry_run)
+ except ValueError as e:
+ console.print(f"[red]Error:[/red] {e}")
+ sys.exit(1)
+
+ # Display results
+ action = "Would remove" if dry_run else "Removed"
+ console.print(f"\n[cyan]Shrink Results ({level.upper()}):[/cyan]")
+ console.print(f" {action}: {len(result['removed'])} items")
+ console.print(f" Kept: {len(result['kept'])} items")
+ console.print(f" Space freed: {_format_size(result['bytes_freed'])}")
+
+ if result.get("size_before"):
+ console.print(f" Size before: {_format_size(result['size_before'])}")
+ if result.get("size_after"):
+ console.print(f" Size after: {_format_size(result['size_after'])}")
+
+ if result["errors"]:
+ console.print(f"\n[red]Errors ({len(result['errors'])}):[/red]")
+ for err in result["errors"]:
+ console.print(f" - {err}")
+
+ if dry_run:
+ console.print("\n[yellow]Dry run - no files were actually removed.[/yellow]")
@cli.command(name="")
@click.argument("url", required=False)
@@ -1403,9 +1585,10 @@ def profiles_cmd():
@click.option("--profile", "-p", help="Use a specific browser profile (by name)")
@click.option("--deep-crawl", type=click.Choice(["bfs", "dfs", "best-first"]), help="Enable deep crawling with specified strategy")
@click.option("--max-pages", type=int, default=10, help="Maximum number of pages to crawl in deep crawl mode")
-def default(url: str, example: bool, browser_config: str, crawler_config: str, filter_config: str,
+@click.option("--json-ensure-ascii/--no-json-ensure-ascii", default=None, help="Escape non-ASCII characters in JSON output (default: from global config)")
+def default(url: str, example: bool, browser_config: str, crawler_config: str, filter_config: str,
extraction_config: str, json_extract: str, schema: str, browser: Dict, crawler: Dict,
- output: str, bypass_cache: bool, question: str, verbose: bool, profile: str, deep_crawl: str, max_pages: int):
+ output: str, bypass_cache: bool, question: str, verbose: bool, profile: str, deep_crawl: str, max_pages: int, json_ensure_ascii: Optional[bool]):
"""Crawl4AI CLI - Web content extraction tool
Simple Usage:
@@ -1457,7 +1640,8 @@ def default(url: str, example: bool, browser_config: str, crawler_config: str, f
verbose=verbose,
profile=profile,
deep_crawl=deep_crawl,
- max_pages=max_pages
+ max_pages=max_pages,
+ json_ensure_ascii=json_ensure_ascii
)
def main():
diff --git a/crawl4ai/cloud/__init__.py b/crawl4ai/cloud/__init__.py
new file mode 100644
index 000000000..74f7548d2
--- /dev/null
+++ b/crawl4ai/cloud/__init__.py
@@ -0,0 +1,16 @@
+"""
+Crawl4AI Cloud Module - Integration with Crawl4AI Cloud service.
+
+This module provides:
+- CLI commands for cloud profile management
+- API client for cloud operations (future)
+- Cloud configuration utilities
+"""
+
+from .cli import cloud_cmd, get_cloud_config, require_auth
+
+__all__ = [
+ "cloud_cmd",
+ "get_cloud_config",
+ "require_auth",
+]
diff --git a/crawl4ai/cloud/cli.py b/crawl4ai/cloud/cli.py
new file mode 100644
index 000000000..355b48f98
--- /dev/null
+++ b/crawl4ai/cloud/cli.py
@@ -0,0 +1,473 @@
+"""
+Crawl4AI Cloud CLI - Commands for interacting with Crawl4AI Cloud service.
+
+Commands:
+ crwl cloud auth - Authenticate with API key
+ crwl cloud profiles upload - Upload a profile to cloud
+ crwl cloud profiles list - List cloud profiles
+ crwl cloud profiles delete - Delete a cloud profile
+"""
+
+import click
+import httpx
+import os
+import shutil
+import sys
+import tarfile
+import tempfile
+from pathlib import Path
+
+import yaml
+from rich.console import Console
+from rich.panel import Panel
+from rich.table import Table
+
+from crawl4ai import BrowserProfiler
+from crawl4ai.browser_profiler import ShrinkLevel, _format_size
+
+console = Console()
+
+# Default cloud API URL
+DEFAULT_CLOUD_API_URL = "https://api.crawl4ai.com"
+
+
+def get_global_config() -> dict:
+ """Load global config from ~/.crawl4ai/global.yml"""
+ config_file = Path.home() / ".crawl4ai" / "global.yml"
+ if not config_file.exists():
+ return {}
+ with open(config_file) as f:
+ return yaml.safe_load(f) or {}
+
+
+def save_global_config(config: dict):
+ """Save global config to ~/.crawl4ai/global.yml"""
+ config_dir = Path.home() / ".crawl4ai"
+ config_dir.mkdir(parents=True, exist_ok=True)
+ config_file = config_dir / "global.yml"
+ with open(config_file, "w") as f:
+ yaml.dump(config, f)
+
+
+def get_cloud_config() -> tuple[str, str]:
+ """Get cloud API key and URL from config."""
+ config = get_global_config()
+ api_key = config.get("CLOUD_API_KEY")
+ api_url = config.get("CLOUD_API_URL", DEFAULT_CLOUD_API_URL)
+ return api_key, api_url
+
+
+def require_auth() -> tuple[str, str]:
+ """Require authentication, exit if not configured."""
+ api_key, api_url = get_cloud_config()
+ if not api_key:
+ console.print("[red]Not authenticated with Crawl4AI Cloud.[/red]")
+ console.print("\nRun [cyan]crwl cloud auth[/cyan] to authenticate.")
+ sys.exit(1)
+ return api_key, api_url
+
+
+# ==================== Cloud Command Group ====================
+
+@click.group("cloud")
+def cloud_cmd():
+ """Crawl4AI Cloud commands - manage cloud profiles and authentication.
+
+ Use browser profiles for authenticated crawling in the cloud.
+
+ Getting started:
+ 1. Get an API key at https://api.crawl4ai.com/dashboard
+ 2. Run: crwl cloud auth
+ 3. Create a local profile: crwl profiles
+ 4. Upload to cloud: crwl cloud profiles upload my_profile
+ """
+ pass
+
+
+# ==================== Auth Commands ====================
+
+@cloud_cmd.command("auth")
+@click.option("--api-key", "-k", help="API key (will prompt if not provided)")
+@click.option("--api-url", "-u", help=f"API URL (default: {DEFAULT_CLOUD_API_URL})")
+@click.option("--logout", is_flag=True, help="Remove saved credentials")
+@click.option("--status", is_flag=True, help="Show current auth status")
+def auth_cmd(api_key: str, api_url: str, logout: bool, status: bool):
+ """Authenticate with Crawl4AI Cloud.
+
+ Your API key is saved locally in ~/.crawl4ai/global.yml
+
+ To get an API key:
+ 1. Go to https://api.crawl4ai.com/dashboard
+ 2. Sign in or create an account
+ 3. Navigate to API Keys section
+ 4. Create a new key and copy it
+
+ Examples:
+ crwl cloud auth # Interactive authentication
+ crwl cloud auth --api-key sk_... # Provide key directly
+ crwl cloud auth --status # Check current status
+ crwl cloud auth --logout # Remove saved credentials
+ """
+ config = get_global_config()
+
+ if status:
+ current_key = config.get("CLOUD_API_KEY")
+ current_url = config.get("CLOUD_API_URL", DEFAULT_CLOUD_API_URL)
+
+ if current_key:
+ # Mask the key for display
+ masked = current_key[:8] + "..." + current_key[-4:] if len(current_key) > 12 else "***"
+ console.print(Panel(
+ f"[green]Authenticated[/green]\n\n"
+ f"API Key: [cyan]{masked}[/cyan]\n"
+ f"API URL: [blue]{current_url}[/blue]",
+ title="Cloud Auth Status",
+ border_style="green"
+ ))
+ else:
+ console.print(Panel(
+ "[yellow]Not authenticated[/yellow]\n\n"
+ "Run [cyan]crwl cloud auth[/cyan] to authenticate.\n\n"
+ "Get your API key at:\n"
+ "[blue]https://api.crawl4ai.com/dashboard[/blue]",
+ title="Cloud Auth Status",
+ border_style="yellow"
+ ))
+ return
+
+ if logout:
+ if "CLOUD_API_KEY" in config:
+ del config["CLOUD_API_KEY"]
+ save_global_config(config)
+ console.print("[green]Logged out successfully.[/green]")
+ else:
+ console.print("[yellow]Not currently authenticated.[/yellow]")
+ return
+
+ # Interactive auth
+ if not api_key:
+ console.print(Panel(
+ "[cyan]Crawl4AI Cloud Authentication[/cyan]\n\n"
+ "To get your API key:\n"
+ " 1. Go to [blue]https://api.crawl4ai.com/dashboard[/blue]\n"
+ " 2. Sign in or create an account\n"
+ " 3. Navigate to API Keys section\n"
+ " 4. Create a new key and paste it below",
+ title="Setup",
+ border_style="cyan"
+ ))
+ api_key = click.prompt("\nEnter your API key", hide_input=True)
+
+ if not api_key:
+ console.print("[red]API key is required.[/red]")
+ sys.exit(1)
+
+ # Validate the key by making a test request
+ test_url = api_url or config.get("CLOUD_API_URL", DEFAULT_CLOUD_API_URL)
+
+ console.print("\n[dim]Validating API key...[/dim]")
+
+ try:
+ response = httpx.get(
+ f"{test_url}/v1/profiles",
+ headers={"X-API-Key": api_key},
+ timeout=10.0
+ )
+
+ if response.status_code == 401:
+ console.print("[red]Invalid API key.[/red]")
+ sys.exit(1)
+ elif response.status_code != 200:
+ console.print(f"[red]Error validating key: {response.status_code}[/red]")
+ sys.exit(1)
+
+ except httpx.RequestError as e:
+ console.print(f"[red]Connection error: {e}[/red]")
+ sys.exit(1)
+
+ # Save to config
+ config["CLOUD_API_KEY"] = api_key
+ if api_url:
+ config["CLOUD_API_URL"] = api_url
+ save_global_config(config)
+
+ console.print("[green]Authentication successful![/green]")
+ console.print(f"Credentials saved to [cyan]~/.crawl4ai/global.yml[/cyan]")
+
+
+# ==================== Profiles Command Group ====================
+
+@cloud_cmd.group("profiles")
+def profiles_cmd():
+ """Manage cloud browser profiles.
+
+ Upload local browser profiles to Crawl4AI Cloud for authenticated crawling.
+
+ Workflow:
+ 1. Create a local profile: crwl profiles
+ 2. Shrink it (optional): crwl shrink my_profile
+ 3. Upload to cloud: crwl cloud profiles upload my_profile
+ 4. Use in API: {"browser_config": {"profile_id": "..."}}
+ """
+ pass
+
+
+@profiles_cmd.command("upload")
+@click.argument("profile_name")
+@click.option("--name", "-n", help="Cloud profile name (defaults to local name)")
+@click.option("--level", "-l",
+ type=click.Choice(["light", "medium", "aggressive", "minimal"]),
+ default="aggressive",
+ help="Shrink level before upload (default: aggressive)")
+@click.option("--no-shrink", is_flag=True, help="Skip shrinking (upload as-is)")
+def upload_cmd(profile_name: str, name: str, level: str, no_shrink: bool):
+ """Upload a browser profile to Crawl4AI Cloud.
+
+ The profile will be shrunk to remove caches before uploading.
+ Use --no-shrink to upload the profile as-is.
+
+ Examples:
+ crwl cloud profiles upload my_profile
+ crwl cloud profiles upload my_profile --name github-auth
+ crwl cloud profiles upload my_profile --level minimal
+ crwl cloud profiles upload my_profile --no-shrink
+ """
+ api_key, api_url = require_auth()
+
+ # Find the profile
+ profiler = BrowserProfiler()
+ profile_path = profiler.get_profile_path(profile_name)
+
+ if not profile_path:
+ console.print(f"[red]Profile not found: {profile_name}[/red]")
+ console.print("\nAvailable profiles:")
+ for p in profiler.list_profiles():
+ console.print(f" - {p['name']}")
+ sys.exit(1)
+
+ cloud_name = name or profile_name
+
+ console.print(f"\n[cyan]Uploading profile:[/cyan] {profile_name}")
+ console.print(f"[cyan]Cloud name:[/cyan] {cloud_name}")
+
+ # Step 1: Shrink (unless --no-shrink)
+ if not no_shrink:
+ console.print(f"\n[dim][1/4] Shrinking profile ({level})...[/dim]")
+ try:
+ result = profiler.shrink(profile_name, ShrinkLevel(level), dry_run=False)
+ console.print(f" Freed: {_format_size(result['bytes_freed'])}")
+ if result.get("size_after"):
+ console.print(f" Size: {_format_size(result['size_after'])}")
+ except Exception as e:
+ console.print(f"[yellow]Warning: Could not shrink profile: {e}[/yellow]")
+ else:
+ console.print("\n[dim][1/4] Skipping shrink...[/dim]")
+
+ # Step 2: Package as tar.gz
+ console.print("[dim][2/4] Packaging profile...[/dim]")
+
+ temp_dir = Path(tempfile.mkdtemp(prefix="crawl4ai_upload_"))
+ tar_path = temp_dir / f"{cloud_name}.tar.gz"
+
+ try:
+ with tarfile.open(tar_path, "w:gz") as tar:
+ # Add profile contents (not the directory itself)
+ for item in Path(profile_path).iterdir():
+ tar.add(item, arcname=item.name)
+
+ size_bytes = tar_path.stat().st_size
+ console.print(f" Created: {tar_path.name} ({_format_size(size_bytes)})")
+
+ # Step 3: Upload
+ console.print("[dim][3/4] Uploading to cloud...[/dim]")
+
+ with open(tar_path, "rb") as f:
+ response = httpx.post(
+ f"{api_url}/v1/profiles",
+ headers={"X-API-Key": api_key},
+ files={"file": (f"{cloud_name}.tar.gz", f, "application/gzip")},
+ data={"name": cloud_name},
+ timeout=120.0
+ )
+
+ if response.status_code == 409:
+ console.print(f"[red]Profile '{cloud_name}' already exists in cloud.[/red]")
+ console.print("Use --name to specify a different name, or delete the existing profile first.")
+ sys.exit(1)
+ elif response.status_code == 400:
+ error = response.json().get("detail", "Unknown error")
+ console.print(f"[red]Upload rejected: {error}[/red]")
+ sys.exit(1)
+ elif response.status_code != 200:
+ console.print(f"[red]Upload failed: {response.status_code}[/red]")
+ console.print(response.text)
+ sys.exit(1)
+
+ result = response.json()
+ profile_id = result["id"]
+
+ console.print("[dim][4/4] Done![/dim]")
+
+ # Success output
+ console.print(Panel(
+ f"[green]Profile uploaded successfully![/green]\n\n"
+ f"Profile ID: [cyan]{profile_id}[/cyan]\n"
+ f"Name: [blue]{cloud_name}[/blue]\n"
+ f"Size: {_format_size(size_bytes)}\n\n"
+ f"[dim]Use in API:[/dim]\n"
+ f' {{"browser_config": {{"profile_id": "{profile_id}"}}}}',
+ title="Upload Complete",
+ border_style="green"
+ ))
+
+ if result.get("scan_warnings"):
+ console.print("\n[yellow]Scan warnings:[/yellow]")
+ for warning in result["scan_warnings"]:
+ console.print(f" - {warning}")
+
+ finally:
+ # Cleanup temp directory
+ shutil.rmtree(temp_dir, ignore_errors=True)
+
+
+@profiles_cmd.command("list")
+def list_cmd():
+ """List all cloud profiles.
+
+ Shows all profiles uploaded to your Crawl4AI Cloud account.
+ """
+ api_key, api_url = require_auth()
+
+ console.print("\n[dim]Fetching profiles...[/dim]")
+
+ try:
+ response = httpx.get(
+ f"{api_url}/v1/profiles",
+ headers={"X-API-Key": api_key},
+ timeout=30.0
+ )
+
+ if response.status_code != 200:
+ console.print(f"[red]Error: {response.status_code}[/red]")
+ console.print(response.text)
+ sys.exit(1)
+
+ data = response.json()
+ profiles = data.get("profiles", [])
+
+ if not profiles:
+ console.print(Panel(
+ "[yellow]No cloud profiles found.[/yellow]\n\n"
+ "Upload a profile with:\n"
+ " [cyan]crwl cloud profiles upload <profile_name>[/cyan]",
+ title="Cloud Profiles",
+ border_style="yellow"
+ ))
+ return
+
+ # Create table
+ table = Table(title="Cloud Profiles")
+ table.add_column("Name", style="cyan")
+ table.add_column("Profile ID", style="dim")
+ table.add_column("Size", justify="right")
+ table.add_column("Created", style="green")
+ table.add_column("Last Used", style="blue")
+
+ for p in profiles:
+ size = _format_size(p.get("size_bytes", 0)) if p.get("size_bytes") else "-"
+ created = p.get("created_at", "-")[:10] if p.get("created_at") else "-"
+ last_used = p.get("last_used_at", "-")[:10] if p.get("last_used_at") else "Never"
+
+ table.add_row(
+ p["name"],
+ p["id"][:8] + "...",
+ size,
+ created,
+ last_used
+ )
+
+ console.print(table)
+ console.print(f"\nTotal: {len(profiles)} profile(s)")
+
+ except httpx.RequestError as e:
+ console.print(f"[red]Connection error: {e}[/red]")
+ sys.exit(1)
+
+
+@profiles_cmd.command("delete")
+@click.argument("profile_name_or_id")
+@click.option("--yes", "-y", is_flag=True, help="Skip confirmation")
+def delete_cmd(profile_name_or_id: str, yes: bool):
+ """Delete a cloud profile.
+
+ You can specify either the profile name or ID.
+
+ Examples:
+ crwl cloud profiles delete my_profile
+ crwl cloud profiles delete abc123...
+ crwl cloud profiles delete my_profile --yes
+ """
+ api_key, api_url = require_auth()
+
+ # First, try to find the profile
+ console.print("\n[dim]Finding profile...[/dim]")
+
+ try:
+ # List profiles to find by name
+ response = httpx.get(
+ f"{api_url}/v1/profiles",
+ headers={"X-API-Key": api_key},
+ timeout=30.0
+ )
+
+ if response.status_code != 200:
+ console.print(f"[red]Error: {response.status_code}[/red]")
+ sys.exit(1)
+
+ profiles = response.json().get("profiles", [])
+
+ # Find matching profile
+ profile = None
+ for p in profiles:
+ if p["name"] == profile_name_or_id or p["id"] == profile_name_or_id or p["id"].startswith(profile_name_or_id):
+ profile = p
+ break
+
+ if not profile:
+ console.print(f"[red]Profile not found: {profile_name_or_id}[/red]")
+ console.print("\nAvailable profiles:")
+ for p in profiles:
+ console.print(f" - {p['name']} ({p['id'][:8]}...)")
+ sys.exit(1)
+
+ # Confirm deletion
+ console.print(f"\nProfile: [cyan]{profile['name']}[/cyan]")
+ console.print(f"ID: [dim]{profile['id']}[/dim]")
+
+ if not yes:
+ if not click.confirm("\nAre you sure you want to delete this profile?"):
+ console.print("[yellow]Cancelled.[/yellow]")
+ return
+
+ # Delete
+ console.print("\n[dim]Deleting...[/dim]")
+
+ response = httpx.delete(
+ f"{api_url}/v1/profiles/{profile['id']}",
+ headers={"X-API-Key": api_key},
+ timeout=30.0
+ )
+
+ if response.status_code == 404:
+ console.print("[red]Profile not found (may have been already deleted).[/red]")
+ sys.exit(1)
+ elif response.status_code != 200:
+ console.print(f"[red]Error: {response.status_code}[/red]")
+ console.print(response.text)
+ sys.exit(1)
+
+ console.print(f"[green]Profile '{profile['name']}' deleted successfully.[/green]")
+
+ except httpx.RequestError as e:
+ console.print(f"[red]Connection error: {e}[/red]")
+ sys.exit(1)
diff --git a/crawl4ai/components/crawler_monitor.py b/crawl4ai/components/crawler_monitor.py
index 49bf9a150..2a5e7e9c0 100644
--- a/crawl4ai/components/crawler_monitor.py
+++ b/crawl4ai/components/crawler_monitor.py
@@ -60,24 +60,34 @@ def stop(self):
def _ui_loop(self):
"""Main UI rendering loop."""
+ import os
+ import sys
+
+ if os.name == 'nt':
+ self._ui_loop_windows()
+ else:
+ self._ui_loop_unix()
+
+ def _ui_loop_unix(self):
+ """UI loop for Unix/macOS using termios."""
import sys
import select
import termios
import tty
-
+
# Setup terminal for non-blocking input
old_settings = termios.tcgetattr(sys.stdin)
try:
tty.setcbreak(sys.stdin.fileno())
-
+
# Use Live display to render the UI
with Live(self.layout, refresh_per_second=1/self.refresh_rate, screen=True) as live:
self.live = live # Store the live display for updates
-
+
# Main UI loop
while not self.stop_event.is_set():
self._update_display()
-
+
# Check for key press (non-blocking)
if select.select([sys.stdin], [], [], 0)[0]:
key = sys.stdin.read(1)
@@ -88,15 +98,37 @@ def _ui_loop(self):
self.stop_event.set()
self.monitor.is_running = False
break
-
+
time.sleep(self.refresh_rate)
-
+
# Just check if the monitor was stopped
if not self.monitor.is_running:
break
finally:
# Restore terminal settings
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
+
+ def _ui_loop_windows(self):
+ """UI loop for Windows using msvcrt."""
+ import msvcrt
+
+ with Live(self.layout, refresh_per_second=1/self.refresh_rate, screen=True) as live:
+ self.live = live
+
+ while not self.stop_event.is_set():
+ self._update_display()
+
+ if msvcrt.kbhit():
+ key = msvcrt.getch().decode("utf-8", errors="ignore")
+ if key == 'q':
+ self.stop_event.set()
+ self.monitor.is_running = False
+ break
+
+ time.sleep(self.refresh_rate)
+
+ if not self.monitor.is_running:
+ break
def _update_display(self):
"""Update the terminal display with current statistics."""
diff --git a/crawl4ai/config.py b/crawl4ai/config.py
index 08f56b832..9cd02f971 100644
--- a/crawl4ai/config.py
+++ b/crawl4ai/config.py
@@ -47,7 +47,7 @@
MIN_WORD_THRESHOLD = 1
IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD = 1
-IMPORTANT_ATTRS = ["src", "href", "alt", "title", "width", "height"]
+IMPORTANT_ATTRS = ["src", "href", "alt", "title", "width", "height", "class", "id"]
ONLY_TEXT_ELIGIBLE_TAGS = [
"b",
"i",
@@ -102,6 +102,9 @@
PAGE_TIMEOUT = 60000
DOWNLOAD_PAGE_TIMEOUT = 60000
+# Delimiter for concatenating multiple HTML examples in schema generation
+HTML_EXAMPLE_DELIMITER = "=== HTML EXAMPLE {index} ==="
+
# Global user settings with descriptions and default values
USER_SETTINGS = {
"DEFAULT_LLM_PROVIDER": {
@@ -142,5 +145,10 @@
"description": "Default user agent mode (default, random, or mobile)",
"type": "string",
"options": ["default", "random", "mobile"]
+ },
+ "JSON_ENSURE_ASCII": {
+ "default": True,
+ "description": "Whether to escape non-ASCII characters in JSON output (False preserves Unicode like 'Ε‘', True escapes as '\\u0161')",
+ "type": "boolean"
}
}
diff --git a/crawl4ai/content_filter_strategy.py b/crawl4ai/content_filter_strategy.py
index 50baed276..0909be33d 100644
--- a/crawl4ai/content_filter_strategy.py
+++ b/crawl4ai/content_filter_strategy.py
@@ -527,7 +527,15 @@ def filter_content(self, html: str, min_word_threshold: int = None) -> List[str]
# Sort selected candidates by original document order
selected_candidates.sort(key=lambda x: x[0])
- return [self.clean_element(tag) for _, _, tag in selected_candidates]
+ # Deduplicate by chunk text, keeping first occurrence (lowest index)
+ seen_texts = set()
+ unique_candidates = []
+ for index, chunk, tag in selected_candidates:
+ if chunk not in seen_texts:
+ seen_texts.add(chunk)
+ unique_candidates.append((index, chunk, tag))
+
+ return [self.clean_element(tag) for _, _, tag in unique_candidates]
class PruningContentFilter(RelevantContentFilter):
@@ -665,7 +673,7 @@ def filter_content(self, html: str, min_word_threshold: int = None) -> List[str]
def _remove_comments(self, soup):
"""Removes HTML comments"""
- for element in soup(text=lambda text: isinstance(text, Comment)):
+ for element in soup(string=lambda string: isinstance(string, Comment)):
element.extract()
def _remove_unwanted_tags(self, soup):
diff --git a/crawl4ai/content_scraping_strategy.py b/crawl4ai/content_scraping_strategy.py
index e915ff5bf..ade19aa11 100644
--- a/crawl4ai/content_scraping_strategy.py
+++ b/crawl4ai/content_scraping_strategy.py
@@ -695,24 +695,57 @@ def _scrap(
meta = {}
content_element = None
+ if css_selector:
+ try:
+ selected = body.cssselect(css_selector)
+ if selected:
+ content_element = lhtml.Element("div")
+ content_element.extend(copy.deepcopy(selected))
+ else:
+ content_element = body
+ except Exception as e:
+ self._log("error", f"Error with css_selector: {str(e)}", "SCRAPE")
+ content_element = body
+
if target_elements:
try:
+ source = content_element if content_element is not None else body
for_content_targeted_element = []
for target_element in target_elements:
- for_content_targeted_element.extend(body.cssselect(target_element))
+ for_content_targeted_element.extend(source.cssselect(target_element))
content_element = lhtml.Element("div")
content_element.extend(copy.deepcopy(for_content_targeted_element))
except Exception as e:
self._log("error", f"Error with target element detection: {str(e)}", "SCRAPE")
return None
- else:
+ elif content_element is None:
content_element = body
# Remove script and style tags
- for tag in ["script", "style", "link", "meta", "noscript"]:
+ for tag in ["style", "link", "meta", "noscript"]:
for element in body.xpath(f".//{tag}"):
if element.getparent() is not None:
element.getparent().remove(element)
+
+ # Handle script separately
+ for element in body.xpath(f".//script"):
+ parent = element.getparent()
+ if parent is not None:
+ tail = element.tail # Get the tail text
+ if tail:
+ prev = element.getprevious() # Get the previous sibling node
+ if prev is not None:
+ if prev.tail:
+ prev.tail += tail
+ else:
+ prev.tail = tail
+ else:
+ if parent.text:
+ parent.text += tail
+ else:
+ parent.text = tail
+ parent.remove(element) # Delete the element
+
# Handle social media and domain exclusions
kwargs["exclude_domains"] = set(kwargs.get("exclude_domains", []))
diff --git a/crawl4ai/deep_crawling/bff_strategy.py b/crawl4ai/deep_crawling/bff_strategy.py
index fdb962485..072ec2f9a 100644
--- a/crawl4ai/deep_crawling/bff_strategy.py
+++ b/crawl4ai/deep_crawling/bff_strategy.py
@@ -2,7 +2,7 @@
import asyncio
import logging
from datetime import datetime
-from typing import AsyncGenerator, Optional, Set, Dict, List, Tuple, Any, Callable, Awaitable
+from typing import AsyncGenerator, Optional, Set, Dict, List, Tuple, Any, Callable, Awaitable, Union
from urllib.parse import urlparse
from ..models import TraversalStats
@@ -39,16 +39,20 @@ def __init__(
filter_chain: FilterChain = FilterChain(),
url_scorer: Optional[URLScorer] = None,
include_external: bool = False,
+ score_threshold: float = -infinity,
max_pages: int = infinity,
logger: Optional[logging.Logger] = None,
# Optional resume/callback parameters for crash recovery
resume_state: Optional[Dict[str, Any]] = None,
on_state_change: Optional[Callable[[Dict[str, Any]], Awaitable[None]]] = None,
+ # Optional cancellation callback - checked before each URL is processed
+ should_cancel: Optional[Callable[[], Union[bool, Awaitable[bool]]]] = None,
):
self.max_depth = max_depth
self.filter_chain = filter_chain
self.url_scorer = url_scorer
self.include_external = include_external
+ self.score_threshold = score_threshold
self.max_pages = max_pages
# self.logger = logger or logging.getLogger(__name__)
# Ensure logger is always a Logger instance, not a dict from serialization
@@ -63,6 +67,7 @@ def __init__(
# Store for use in arun methods
self._resume_state = resume_state
self._on_state_change = on_state_change
+ self._should_cancel = should_cancel
self._last_state: Optional[Dict[str, Any]] = None
# Shadow list for queue items (only used when on_state_change is set)
self._queue_shadow: Optional[List[Tuple[float, int, str, Optional[str]]]] = None
@@ -89,6 +94,55 @@ async def can_process_url(self, url: str, depth: int) -> bool:
return True
+ def cancel(self) -> None:
+ """
+ Cancel the crawl. Thread-safe, can be called from any context.
+
+ The crawl will stop before processing the next URL. The current URL
+ being processed (if any) will complete before the crawl stops.
+ """
+ self._cancel_event.set()
+
+ @property
+ def cancelled(self) -> bool:
+ """
+ Check if the crawl was/is cancelled. Thread-safe.
+
+ Returns:
+ True if the crawl has been cancelled, False otherwise.
+ """
+ return self._cancel_event.is_set()
+
+ async def _check_cancellation(self) -> bool:
+ """
+ Check if crawl should be cancelled.
+
+ Handles both internal cancel flag and external should_cancel callback.
+ Supports both sync and async callbacks.
+
+ Returns:
+ True if crawl should be cancelled, False otherwise.
+ """
+ if self._cancel_event.is_set():
+ return True
+
+ if self._should_cancel:
+ try:
+ # Handle both sync and async callbacks
+ result = self._should_cancel()
+ if asyncio.iscoroutine(result):
+ result = await result
+
+ if result:
+ self._cancel_event.set()
+ self.stats.end_time = datetime.now()
+ return True
+ except Exception as e:
+ # Fail-open: log warning and continue crawling
+ self.logger.warning(f"should_cancel callback error: {e}")
+
+ return False
+
async def link_discovery(
self,
result: CrawlResult,
@@ -125,7 +179,7 @@ async def link_discovery(
base_url = normalize_url_for_deep_crawl(url, source_url)
if base_url in visited:
continue
- if not await self.can_process_url(url, new_depth):
+ if not await self.can_process_url(base_url, new_depth):
self.stats.urls_skipped += 1
continue
@@ -148,6 +202,9 @@ async def _arun_best_first(
The queue items are tuples of (score, depth, url, parent_url). Lower scores
are treated as higher priority. URLs are processed in batches for efficiency.
"""
+ # Reset cancel event for strategy reuse
+ self._cancel_event = asyncio.Event()
+
queue: asyncio.PriorityQueue = asyncio.PriorityQueue()
# Conditional state initialization for resume support
@@ -180,7 +237,12 @@ async def _arun_best_first(
if self._pages_crawled >= self.max_pages:
self.logger.info(f"Max pages limit ({self.max_pages}) reached, stopping crawl")
break
-
+
+ # Check external cancellation callback before processing this batch
+ if await self._check_cancellation():
+ self.logger.info("Crawl cancelled by user")
+ break
+
# Calculate how many more URLs we can process in this batch
remaining = self.max_pages - self._pages_crawled
batch_size = min(BATCH_SIZE, remaining)
@@ -245,6 +307,13 @@ async def _arun_best_first(
for new_url, new_parent in new_links:
new_depth = depths.get(new_url, depth + 1)
new_score = self.url_scorer.score(new_url) if self.url_scorer else 0
+ # Skip URLs with scores below the threshold
+ if new_score < self.score_threshold:
+ self.logger.debug(
+ f"URL {new_url} skipped: score {new_score} below threshold {self.score_threshold}"
+ )
+ self.stats.urls_skipped += 1
+ continue
queue_item = (-new_score, new_depth, new_url, new_parent)
await queue.put(queue_item)
# Add to shadow list if tracking
@@ -262,11 +331,26 @@ async def _arun_best_first(
],
"depths": depths,
"pages_crawled": self._pages_crawled,
+ "cancelled": self._cancel_event.is_set(),
}
self._last_state = state
await self._on_state_change(state)
- # End of crawl.
+ # Final state update if cancelled
+ if self._cancel_event.is_set() and self._on_state_change and self._queue_shadow is not None:
+ state = {
+ "strategy_type": "best_first",
+ "visited": list(visited),
+ "queue_items": [
+ {"score": s, "depth": d, "url": u, "parent_url": p}
+ for s, d, u, p in self._queue_shadow
+ ],
+ "depths": depths,
+ "pages_crawled": self._pages_crawled,
+ "cancelled": True,
+ }
+ self._last_state = state
+ await self._on_state_change(state)
async def _arun_batch(
self,
diff --git a/crawl4ai/deep_crawling/bfs_strategy.py b/crawl4ai/deep_crawling/bfs_strategy.py
index 35b669394..dfb759272 100644
--- a/crawl4ai/deep_crawling/bfs_strategy.py
+++ b/crawl4ai/deep_crawling/bfs_strategy.py
@@ -2,7 +2,7 @@
import asyncio
import logging
from datetime import datetime
-from typing import AsyncGenerator, Optional, Set, Dict, List, Tuple, Any, Callable, Awaitable
+from typing import AsyncGenerator, Optional, Set, Dict, List, Tuple, Any, Callable, Awaitable, Union
from urllib.parse import urlparse
from ..models import TraversalStats
@@ -34,6 +34,8 @@ def __init__(
# Optional resume/callback parameters for crash recovery
resume_state: Optional[Dict[str, Any]] = None,
on_state_change: Optional[Callable[[Dict[str, Any]], Awaitable[None]]] = None,
+ # Optional cancellation callback - checked before each URL is processed
+ should_cancel: Optional[Callable[[], Union[bool, Awaitable[bool]]]] = None,
):
self.max_depth = max_depth
self.filter_chain = filter_chain
@@ -54,6 +56,7 @@ def __init__(
# Store for use in arun methods
self._resume_state = resume_state
self._on_state_change = on_state_change
+ self._should_cancel = should_cancel
self._last_state: Optional[Dict[str, Any]] = None
async def can_process_url(self, url: str, depth: int) -> bool:
@@ -78,6 +81,55 @@ async def can_process_url(self, url: str, depth: int) -> bool:
return True
+ def cancel(self) -> None:
+ """
+ Cancel the crawl. Thread-safe, can be called from any context.
+
+ The crawl will stop before processing the next URL. The current URL
+ being processed (if any) will complete before the crawl stops.
+ """
+ self._cancel_event.set()
+
+ @property
+ def cancelled(self) -> bool:
+ """
+ Check if the crawl was/is cancelled. Thread-safe.
+
+ Returns:
+ True if the crawl has been cancelled, False otherwise.
+ """
+ return self._cancel_event.is_set()
+
+ async def _check_cancellation(self) -> bool:
+ """
+ Check if crawl should be cancelled.
+
+ Handles both internal cancel flag and external should_cancel callback.
+ Supports both sync and async callbacks.
+
+ Returns:
+ True if crawl should be cancelled, False otherwise.
+ """
+ if self._cancel_event.is_set():
+ return True
+
+ if self._should_cancel:
+ try:
+ # Handle both sync and async callbacks
+ result = self._should_cancel()
+ if asyncio.iscoroutine(result):
+ result = await result
+
+ if result:
+ self._cancel_event.set()
+ self.stats.end_time = datetime.now()
+ return True
+ except Exception as e:
+ # Fail-open: log warning and continue crawling
+ self.logger.warning(f"should_cancel callback error: {e}")
+
+ return False
+
async def link_discovery(
self,
result: CrawlResult,
@@ -118,7 +170,7 @@ async def link_discovery(
base_url = normalize_url_for_deep_crawl(url, source_url)
if base_url in visited:
continue
- if not await self.can_process_url(url, next_depth):
+ if not await self.can_process_url(base_url, next_depth):
self.stats.urls_skipped += 1
continue
@@ -162,6 +214,9 @@ async def _arun_batch(
Batch (non-streaming) mode:
Processes one BFS level at a time, then yields all the results.
"""
+ # Reset cancel event for strategy reuse
+ self._cancel_event = asyncio.Event()
+
# Conditional state initialization for resume support
if self._resume_state:
visited = set(self._resume_state.get("visited", []))
@@ -185,7 +240,12 @@ async def _arun_batch(
if self._pages_crawled >= self.max_pages:
self.logger.info(f"Max pages limit ({self.max_pages}) reached, stopping crawl")
break
-
+
+ # Check external cancellation callback before processing this level
+ if await self._check_cancellation():
+ self.logger.info("Crawl cancelled by user")
+ break
+
next_level: List[Tuple[str, Optional[str]]] = []
urls = [url for url, _ in current_level]
@@ -218,12 +278,26 @@ async def _arun_batch(
"pending": [{"url": u, "parent_url": p} for u, p in next_level],
"depths": depths,
"pages_crawled": self._pages_crawled,
+ "cancelled": self._cancel_event.is_set(),
}
self._last_state = state
await self._on_state_change(state)
current_level = next_level
+ # Final state update if cancelled
+ if self._cancel_event.is_set() and self._on_state_change:
+ state = {
+ "strategy_type": "bfs",
+ "visited": list(visited),
+ "pending": [{"url": u, "parent_url": p} for u, p in current_level],
+ "depths": depths,
+ "pages_crawled": self._pages_crawled,
+ "cancelled": True,
+ }
+ self._last_state = state
+ await self._on_state_change(state)
+
return results
async def _arun_stream(
@@ -236,6 +310,9 @@ async def _arun_stream(
Streaming mode:
Processes one BFS level at a time and yields results immediately as they arrive.
"""
+ # Reset cancel event for strategy reuse
+ self._cancel_event = asyncio.Event()
+
# Conditional state initialization for resume support
if self._resume_state:
visited = set(self._resume_state.get("visited", []))
@@ -252,6 +329,11 @@ async def _arun_stream(
depths: Dict[str, int] = {start_url: 0}
while current_level and not self._cancel_event.is_set():
+ # Check external cancellation callback before processing this level
+ if await self._check_cancellation():
+ self.logger.info("Crawl cancelled by user")
+ break
+
next_level: List[Tuple[str, Optional[str]]] = []
urls = [url for url, _ in current_level]
visited.update(urls)
@@ -293,6 +375,7 @@ async def _arun_stream(
"pending": [{"url": u, "parent_url": p} for u, p in next_level],
"depths": depths,
"pages_crawled": self._pages_crawled,
+ "cancelled": self._cancel_event.is_set(),
}
self._last_state = state
await self._on_state_change(state)
@@ -301,9 +384,22 @@ async def _arun_stream(
# by considering these URLs as visited but not counting them toward the max_pages limit
if results_count == 0 and urls:
self.logger.warning(f"No results returned for {len(urls)} URLs, marking as visited")
-
+
current_level = next_level
+ # Final state update if cancelled
+ if self._cancel_event.is_set() and self._on_state_change:
+ state = {
+ "strategy_type": "bfs",
+ "visited": list(visited),
+ "pending": [{"url": u, "parent_url": p} for u, p in current_level],
+ "depths": depths,
+ "pages_crawled": self._pages_crawled,
+ "cancelled": True,
+ }
+ self._last_state = state
+ await self._on_state_change(state)
+
async def shutdown(self) -> None:
"""
Clean up resources and signal cancellation of the crawl.
diff --git a/crawl4ai/deep_crawling/dfs_strategy.py b/crawl4ai/deep_crawling/dfs_strategy.py
index d98d06a79..3e4987f25 100644
--- a/crawl4ai/deep_crawling/dfs_strategy.py
+++ b/crawl4ai/deep_crawling/dfs_strategy.py
@@ -1,4 +1,5 @@
# dfs_deep_crawl_strategy.py
+import asyncio
from typing import AsyncGenerator, Optional, Set, Dict, List, Tuple
from ..models import CrawlResult
@@ -38,6 +39,9 @@ async def _arun_batch(
in control of traversal. Every successful page bumps ``_pages_crawled`` and
seeds new stack items discovered via :meth:`link_discovery`.
"""
+ # Reset cancel event for strategy reuse
+ self._cancel_event = asyncio.Event()
+
# Conditional state initialization for resume support
if self._resume_state:
visited = set(self._resume_state.get("visited", []))
@@ -59,6 +63,11 @@ async def _arun_batch(
self._reset_seen(start_url)
while stack and not self._cancel_event.is_set():
+ # Check external cancellation callback before processing this URL
+ if await self._check_cancellation():
+ self.logger.info("Crawl cancelled by user")
+ break
+
url, parent, depth = stack.pop()
if url in visited or depth > self.max_depth:
continue
@@ -105,9 +114,28 @@ async def _arun_batch(
"depths": depths,
"pages_crawled": self._pages_crawled,
"dfs_seen": list(self._dfs_seen),
+ "cancelled": self._cancel_event.is_set(),
}
self._last_state = state
await self._on_state_change(state)
+
+ # Final state update if cancelled
+ if self._cancel_event.is_set() and self._on_state_change:
+ state = {
+ "strategy_type": "dfs",
+ "visited": list(visited),
+ "stack": [
+ {"url": u, "parent_url": p, "depth": d}
+ for u, p, d in stack
+ ],
+ "depths": depths,
+ "pages_crawled": self._pages_crawled,
+ "dfs_seen": list(self._dfs_seen),
+ "cancelled": True,
+ }
+ self._last_state = state
+ await self._on_state_change(state)
+
return results
async def _arun_stream(
@@ -123,6 +151,9 @@ async def _arun_stream(
yielded before we even look at the next stack entry. Successful crawls
still feed :meth:`link_discovery`, keeping DFS order intact.
"""
+ # Reset cancel event for strategy reuse
+ self._cancel_event = asyncio.Event()
+
# Conditional state initialization for resume support
if self._resume_state:
visited = set(self._resume_state.get("visited", []))
@@ -141,6 +172,11 @@ async def _arun_stream(
self._reset_seen(start_url)
while stack and not self._cancel_event.is_set():
+ # Check external cancellation callback before processing this URL
+ if await self._check_cancellation():
+ self.logger.info("Crawl cancelled by user")
+ break
+
url, parent, depth = stack.pop()
if url in visited or depth > self.max_depth:
continue
@@ -183,10 +219,28 @@ async def _arun_stream(
"depths": depths,
"pages_crawled": self._pages_crawled,
"dfs_seen": list(self._dfs_seen),
+ "cancelled": self._cancel_event.is_set(),
}
self._last_state = state
await self._on_state_change(state)
+ # Final state update if cancelled
+ if self._cancel_event.is_set() and self._on_state_change:
+ state = {
+ "strategy_type": "dfs",
+ "visited": list(visited),
+ "stack": [
+ {"url": u, "parent_url": p, "depth": d}
+ for u, p, d in stack
+ ],
+ "depths": depths,
+ "pages_crawled": self._pages_crawled,
+ "dfs_seen": list(self._dfs_seen),
+ "cancelled": True,
+ }
+ self._last_state = state
+ await self._on_state_change(state)
+
async def link_discovery(
self,
result: CrawlResult,
@@ -246,7 +300,7 @@ async def link_discovery(
if not normalized_url or normalized_url in seen:
continue
- if not await self.can_process_url(raw_url, next_depth):
+ if not await self.can_process_url(normalized_url, next_depth):
self.stats.urls_skipped += 1
continue
diff --git a/crawl4ai/deep_crawling/filters.py b/crawl4ai/deep_crawling/filters.py
index c075cb7d5..2fb819ecc 100644
--- a/crawl4ai/deep_crawling/filters.py
+++ b/crawl4ai/deep_crawling/filters.py
@@ -85,7 +85,7 @@ def logger(self):
def add_filter(self, filter_: URLFilter) -> "FilterChain":
"""Add a filter to the chain"""
- self.filters.append(filter_)
+ self.filters = self.filters + (filter_,)
return self # Enable method chaining
async def apply(self, url: str) -> bool:
@@ -216,10 +216,11 @@ def _add_pattern(self, pattern: str, pattern_type: int):
@lru_cache(maxsize=10000)
def apply(self, url: str) -> bool:
+ url_path = urlparse(url).path
+
# Quick suffix check (*.html)
if self._simple_suffixes:
- path = url.split("?")[0]
- if path.split("/")[-1].split(".")[-1] in self._simple_suffixes:
+ if url_path.split("/")[-1].split(".")[-1] in self._simple_suffixes:
result = True
self._update_stats(result)
return not result if self._reverse else result
@@ -232,21 +233,13 @@ def apply(self, url: str) -> bool:
self._update_stats(result)
return not result if self._reverse else result
- # Prefix check (/foo/*)
+ # Prefix check (/foo/* or https://domain/foo/*)
if self._simple_prefixes:
- path = url.split("?")[0]
- # if any(path.startswith(p) for p in self._simple_prefixes):
- # result = True
- # self._update_stats(result)
- # return not result if self._reverse else result
- ####
- # Modified the prefix matching logic to ensure path boundary checking:
- # - Check if the matched prefix is followed by a path separator (`/`), query parameter (`?`), fragment (`#`), or is at the end of the path
- # - This ensures `/api/` only matches complete path segments, not substrings like `/apiv2/`
- ####
for prefix in self._simple_prefixes:
- if path.startswith(prefix):
- if len(path) == len(prefix) or path[len(prefix)] in ['/', '?', '#']:
+ # Use url_path for path-only prefixes, full URL for absolute prefixes
+ match_against = url if '://' in prefix else url_path
+ if match_against.startswith(prefix):
+ if len(match_against) == len(prefix) or match_against[len(prefix)] in ['/', '?', '#']:
result = True
self._update_stats(result)
return not result if self._reverse else result
diff --git a/crawl4ai/extraction_strategy.py b/crawl4ai/extraction_strategy.py
index 6be1c7c7b..a31560160 100644
--- a/crawl4ai/extraction_strategy.py
+++ b/crawl4ai/extraction_strategy.py
@@ -1,4 +1,5 @@
from abc import ABC, abstractmethod
+import ast
import inspect
from typing import Any, List, Dict, Optional, Tuple, Pattern, Union
from concurrent.futures import ThreadPoolExecutor, as_completed
@@ -13,6 +14,7 @@
CHUNK_TOKEN_THRESHOLD,
OVERLAP_RATE,
WORD_TOKEN_RATE,
+ HTML_EXAMPLE_DELIMITER,
)
from .utils import * # noqa: F403
@@ -46,6 +48,42 @@
from lxml import html, etree
+def _strip_markdown_fences(text: str) -> str:
+ """Strip markdown code fences (e.g. ```json ... ```) from LLM responses."""
+ text = text.strip()
+ return re.sub(
+ r"^```(?:[a-zA-Z0-9_-]+)?\s*|```$", "", text, flags=re.MULTILINE
+ ).strip()
+
+
+def _get_top_level_structure(html_content: str, max_depth: int = 3) -> str:
+ """Return a compact tag outline of the HTML body up to a given depth.
+
+ Used in schema validation feedback when baseSelector matches 0 elements,
+ so the LLM can see what top-level tags actually exist.
+ """
+ try:
+ tree = html.fromstring(html_content)
+ except Exception:
+ return ""
+ body = tree.xpath("//body")
+ root = body[0] if body else tree
+ lines = []
+
+ def _walk(el, depth):
+ if depth > max_depth or not isinstance(el.tag, str):
+ return
+ classes = el.get("class", "").split()
+ cls_str = "." + ".".join(classes) if classes else ""
+ id_str = f"#{el.get('id')}" if el.get("id") else ""
+ lines.append(" " * depth + f"<{el.tag}{id_str}{cls_str}>")
+ for child in el:
+ _walk(child, depth + 1)
+
+ _walk(root, 0)
+ return "\n".join(lines[:60])
+
+
class ExtractionStrategy(ABC):
"""
Abstract base class for all extraction strategies.
@@ -258,7 +296,7 @@ def filter_documents_embeddings(
return documents
if len(documents) < at_least_k:
- at_least_k = len(documents) // 2
+ at_least_k = max(1, len(documents) // 2)
from sklearn.metrics.pairwise import cosine_similarity
@@ -413,7 +451,10 @@ def extract(self, url: str, html: str, *q, **kwargs) -> List[Dict[str, Any]]:
"""
# Assume `html` is a list of text chunks for this strategy
t = time.time()
- text_chunks = html.split(self.DEL) # Split by lines or paragraphs as needed
+ # Split by delimiter; fall back to double-newline splitting for raw text
+ text_chunks = html.split(self.DEL)
+ if len(text_chunks) == 1:
+ text_chunks = [chunk.strip() for chunk in html.split("\n\n") if chunk.strip()]
# Pre-filter documents using embeddings and semantic_filter
text_chunks = self.filter_documents_embeddings(
@@ -676,8 +717,12 @@ def extract(self, url: str, ix: int, html: str) -> List[Dict[str, Any]]:
content = response.choices[0].message.content
blocks = None
- if self.force_json_response:
- blocks = json.loads(content)
+ if not content:
+ finish_reason = getattr(response.choices[0], "finish_reason", "unknown")
+ blocks = [{"index": 0, "error": True, "tags": ["error"],
+ "content": f"LLM returned no content (finish_reason: {finish_reason})"}]
+ elif self.force_json_response:
+ blocks = json.loads(_strip_markdown_fences(content))
if isinstance(blocks, dict):
# If it has only one key which calue is list then assign that to blocks, exampled: {"news": [..]}
if len(blocks) == 1 and isinstance(list(blocks.values())[0], list):
@@ -696,9 +741,8 @@ def extract(self, url: str, ix: int, html: str) -> List[Dict[str, Any]]:
for block in blocks:
block["error"] = False
except Exception:
- parsed, unparsed = split_and_parse_json_objects(
- response.choices[0].message.content
- )
+ raw_content = response.choices[0].message.content or ""
+ parsed, unparsed = split_and_parse_json_objects(raw_content)
blocks = parsed
if unparsed:
blocks.append(
@@ -876,8 +920,12 @@ async def aextract(self, url: str, ix: int, html: str) -> List[Dict[str, Any]]:
content = response.choices[0].message.content
blocks = None
- if self.force_json_response:
- blocks = json.loads(content)
+ if not content:
+ finish_reason = getattr(response.choices[0], "finish_reason", "unknown")
+ blocks = [{"index": 0, "error": True, "tags": ["error"],
+ "content": f"LLM returned no content (finish_reason: {finish_reason})"}]
+ elif self.force_json_response:
+ blocks = json.loads(_strip_markdown_fences(content))
if isinstance(blocks, dict):
if len(blocks) == 1 and isinstance(list(blocks.values())[0], list):
blocks = list(blocks.values())[0]
@@ -892,9 +940,8 @@ async def aextract(self, url: str, ix: int, html: str) -> List[Dict[str, Any]]:
for block in blocks:
block["error"] = False
except Exception:
- parsed, unparsed = split_and_parse_json_objects(
- response.choices[0].message.content
- )
+ raw_content = response.choices[0].message.content or ""
+ parsed, unparsed = split_and_parse_json_objects(raw_content)
blocks = parsed
if unparsed:
blocks.append(
@@ -992,6 +1039,69 @@ def show_usage(self) -> None:
#######################################################
# New extraction strategies for JSON-based extraction #
#######################################################
+
+# Safe builtins allowed in computed field expressions
+_SAFE_EVAL_BUILTINS = {
+ "str": str, "int": int, "float": float, "bool": bool,
+ "len": len, "round": round, "abs": abs, "min": min, "max": max,
+ "sum": sum, "sorted": sorted, "reversed": reversed,
+ "list": list, "dict": dict, "tuple": tuple, "set": set,
+ "enumerate": enumerate, "zip": zip, "map": map, "filter": filter,
+ "any": any, "all": all, "range": range,
+ "True": True, "False": False, "None": None,
+ "isinstance": isinstance, "type": type,
+}
+
+
+def _safe_eval_expression(expression: str, local_vars: dict) -> Any:
+ """
+ Evaluate a computed field expression safely using AST validation.
+
+ Allows simple transforms (math, string methods, attribute access on data)
+ while blocking dangerous operations (__import__, dunder access, etc.).
+
+ Args:
+ expression: The Python expression string to evaluate.
+ local_vars: The local variables (extracted item fields) available to the expression.
+
+ Returns:
+ The result of evaluating the expression.
+
+ Raises:
+ ValueError: If the expression contains disallowed constructs.
+ """
+ try:
+ tree = ast.parse(expression, mode="eval")
+ except SyntaxError as e:
+ raise ValueError(f"Invalid expression syntax: {e}")
+
+ for node in ast.walk(tree):
+ # Block import statements
+ if isinstance(node, (ast.Import, ast.ImportFrom)):
+ raise ValueError("Import statements are not allowed in expressions")
+
+ # Block attribute access to dunder attributes (e.g., __class__, __globals__)
+ if isinstance(node, ast.Attribute) and node.attr.startswith("_"):
+ raise ValueError(
+ f"Access to private/dunder attribute '{node.attr}' is not allowed"
+ )
+
+ # Block calls to __import__ or any name starting with _
+ if isinstance(node, ast.Call):
+ func = node.func
+ if isinstance(func, ast.Name) and func.id.startswith("_"):
+ raise ValueError(
+ f"Calling '{func.id}' is not allowed in expressions"
+ )
+ if isinstance(func, ast.Attribute) and func.attr.startswith("_"):
+ raise ValueError(
+ f"Calling '{func.attr}' is not allowed in expressions"
+ )
+
+ safe_globals = {"__builtins__": _SAFE_EVAL_BUILTINS}
+ return eval(compile(tree, "<expression>", "eval"), safe_globals, local_vars)
+
+
class JsonElementExtractionStrategy(ExtractionStrategy):
"""
Abstract base class for extracting structured JSON from HTML content.
@@ -1099,6 +1209,11 @@ def _get_elements(self, element, selector: str):
def _extract_field(self, element, field):
try:
+ if "source" in field:
+ element = self._resolve_source(element, field["source"])
+ if element is None:
+ return field.get("default")
+
if field["type"] == "nested":
nested_elements = self._get_elements(element, field["selector"])
nested_element = nested_elements[0] if nested_elements else None
@@ -1147,17 +1262,30 @@ def _extract_single_field(self, element, field):
else:
selected = element
- value = None
- if field["type"] == "text":
- value = self._get_element_text(selected)
- elif field["type"] == "attribute":
- value = self._get_element_attribute(selected, field["attribute"])
- elif field["type"] == "html":
- value = self._get_element_html(selected)
- elif field["type"] == "regex":
- text = self._get_element_text(selected)
- match = re.search(field["pattern"], text)
- value = match.group(1) if match else None
+ type_pipeline = field["type"]
+ if not isinstance(type_pipeline, list):
+ type_pipeline = [type_pipeline]
+ value = selected
+ for step in type_pipeline:
+ if step == "text":
+ value = self._get_element_text(value)
+ elif step == "attribute":
+ value = self._get_element_attribute(value, field["attribute"])
+ elif step == "html":
+ value = self._get_element_html(value)
+ elif step == "regex":
+ pattern = field.get("pattern")
+ if pattern:
+ # If value is still an element, extract text first (backward compat)
+ if not isinstance(value, str):
+ value = self._get_element_text(value)
+ if isinstance(value, str):
+ match = re.search(pattern, value)
+ value = match.group(field.get("group", 1)) if match else None
+ else:
+ value = None
+ if value is None:
+ break
if "transform" in field:
value = self._apply_transform(value, field["transform"])
@@ -1227,7 +1355,7 @@ def _apply_transform(self, value, transform):
def _compute_field(self, item, field):
try:
if "expression" in field:
- return eval(field["expression"], {}, item)
+ return _safe_eval_expression(field["expression"], item)
elif "function" in field:
return field["function"](item)
except Exception as e:
@@ -1271,6 +1399,282 @@ def _get_element_attribute(self, element, attribute: str):
"""Get attribute value from element"""
pass
+ @abstractmethod
+ def _resolve_source(self, element, source: str):
+ """Navigate to a sibling element relative to the base element.
+
+ Used when a field's data lives in a sibling of the base element
+ rather than a descendant. For example, Hacker News splits each
+ submission across two sibling <tr> rows.
+
+ Args:
+ element: The current base element.
+ source: A sibling selector string. Currently supports the
+ ``"+ <selector>"`` syntax which navigates to the next
+ sibling matching ``<selector>``.
+
+ Returns:
+ The resolved sibling element, or ``None`` if not found.
+ """
+ pass
+
+ @staticmethod
+ def _validate_schema(
+ schema: dict,
+ html_content: str,
+ schema_type: str = "CSS",
+ expected_fields: Optional[List[str]] = None,
+ ) -> dict:
+ """Run the generated schema against HTML and return a diagnostic result.
+
+ Args:
+ schema: The extraction schema to validate.
+ html_content: The HTML to validate against.
+ schema_type: "CSS" or "XPATH".
+ expected_fields: When provided, enables strict mode β success
+ requires ALL expected fields to be present and populated.
+ When None, uses fuzzy mode (populated_fields > 0).
+
+ Returns a dict with keys: success, base_elements_found, total_fields,
+ populated_fields, field_coverage, field_details, issues,
+ sample_base_html, top_level_structure.
+ """
+ result = {
+ "success": False,
+ "base_elements_found": 0,
+ "total_fields": 0,
+ "populated_fields": 0,
+ "field_coverage": 0.0,
+ "field_details": [],
+ "issues": [],
+ "sample_base_html": "",
+ "top_level_structure": "",
+ }
+
+ try:
+ StrategyClass = (
+ JsonCssExtractionStrategy
+ if schema_type.upper() == "CSS"
+ else JsonXPathExtractionStrategy
+ )
+ strategy = StrategyClass(schema=schema)
+ items = strategy.extract(url="", html_content=html_content)
+ except Exception as e:
+ result["issues"].append(f"Extraction crashed: {e}")
+ return result
+
+ # Count base elements directly
+ try:
+ parsed = strategy._parse_html(html_content)
+ base_elements = strategy._get_base_elements(parsed, schema["baseSelector"])
+ result["base_elements_found"] = len(base_elements)
+
+ # Grab sample innerHTML of first base element (truncated)
+ if base_elements:
+ sample = strategy._get_element_html(base_elements[0])
+ result["sample_base_html"] = sample[:2000]
+ except Exception:
+ pass
+
+ if result["base_elements_found"] == 0:
+ result["issues"].append(
+ f"baseSelector '{schema.get('baseSelector', '')}' matched 0 elements"
+ )
+ result["top_level_structure"] = _get_top_level_structure(html_content)
+ return result
+
+ # Analyze field coverage
+ all_fields = schema.get("fields", [])
+ field_names = [f["name"] for f in all_fields]
+ result["total_fields"] = len(field_names)
+
+ for fname in field_names:
+ values = [item.get(fname) for item in items]
+ populated_count = sum(1 for v in values if v is not None and v != "")
+ sample_val = next((v for v in values if v is not None and v != ""), None)
+ if sample_val is not None:
+ sample_val = str(sample_val)[:120]
+ result["field_details"].append({
+ "name": fname,
+ "populated_count": populated_count,
+ "total_count": len(items),
+ "sample_value": sample_val,
+ })
+
+ result["populated_fields"] = sum(
+ 1 for fd in result["field_details"] if fd["populated_count"] > 0
+ )
+ if result["total_fields"] > 0:
+ result["field_coverage"] = result["populated_fields"] / result["total_fields"]
+
+ # Build issues
+ if result["populated_fields"] == 0:
+ result["issues"].append(
+ "All fields returned None/empty β selectors likely wrong"
+ )
+ else:
+ empty_fields = [
+ fd["name"]
+ for fd in result["field_details"]
+ if fd["populated_count"] == 0
+ ]
+ if empty_fields:
+ result["issues"].append(
+ f"Fields always empty: {', '.join(empty_fields)}"
+ )
+
+ # Check for missing expected fields (strict mode)
+ if expected_fields:
+ schema_field_names = {f["name"] for f in schema.get("fields", [])}
+ missing = [f for f in expected_fields if f not in schema_field_names]
+ if missing:
+ result["issues"].append(
+ f"Expected fields missing from schema: {', '.join(missing)}"
+ )
+
+ # Success criteria
+ if expected_fields:
+ # Strict: all expected fields must exist in schema AND be populated
+ schema_field_names = {f["name"] for f in schema.get("fields", [])}
+ populated_names = {
+ fd["name"] for fd in result["field_details"] if fd["populated_count"] > 0
+ }
+ result["success"] = (
+ result["base_elements_found"] > 0
+ and all(f in populated_names for f in expected_fields)
+ )
+ else:
+ # Fuzzy: at least something extracted
+ result["success"] = (
+ result["base_elements_found"] > 0 and result["populated_fields"] > 0
+ )
+ return result
+
+ @staticmethod
+ def _build_feedback_message(
+ validation_result: dict,
+ schema: dict,
+ attempt: int,
+ is_repeated: bool,
+ ) -> str:
+ """Build a structured feedback message from a validation result."""
+ vr = validation_result
+ parts = []
+
+ parts.append(f"## Schema Validation β Attempt {attempt}")
+
+ # Base selector
+ if vr["base_elements_found"] == 0:
+ parts.append(
+ f"**CRITICAL:** baseSelector `{schema.get('baseSelector', '')}` "
+ f"matched **0 elements**. The schema cannot extract anything."
+ )
+ if vr["top_level_structure"]:
+ parts.append(
+ "Here is the top-level HTML structure so you can pick a valid selector:\n```\n"
+ + vr["top_level_structure"]
+ + "\n```"
+ )
+ else:
+ parts.append(
+ f"baseSelector matched **{vr['base_elements_found']}** element(s)."
+ )
+
+ # Field coverage table
+ if vr["field_details"]:
+ parts.append(
+ f"\n**Field coverage:** {vr['populated_fields']}/{vr['total_fields']} fields have data\n"
+ )
+ parts.append("| Field | Populated | Sample |")
+ parts.append("|-------|-----------|--------|")
+ for fd in vr["field_details"]:
+ sample = fd["sample_value"] or "*(empty)*"
+ parts.append(
+ f"| {fd['name']} | {fd['populated_count']}/{fd['total_count']} | {sample} |"
+ )
+
+ # Issues
+ if vr["issues"]:
+ parts.append("\n**Issues:**")
+ for issue in vr["issues"]:
+ parts.append(f"- {issue}")
+
+ # Sample base HTML when all fields empty
+ if vr["populated_fields"] == 0 and vr["sample_base_html"]:
+ parts.append(
+ "\nHere is the innerHTML of the first base element β "
+ "use it to find correct child selectors:\n```html\n"
+ + vr["sample_base_html"]
+ + "\n```"
+ )
+
+ # Repeated schema warning
+ if is_repeated:
+ parts.append(
+ "\n**WARNING:** You returned the exact same schema as before. "
+ "You MUST change the selectors to fix the issues above."
+ )
+
+ parts.append(
+ "\nPlease fix the schema and return ONLY valid JSON, nothing else."
+ )
+ return "\n".join(parts)
+
+ @staticmethod
+ async def _infer_target_json(query: str, html_snippet: str, llm_config, url: str = None, usage: 'TokenUsage' = None) -> Optional[dict]:
+ """Infer a target JSON example from a query and HTML snippet via a quick LLM call.
+
+ Args:
+ usage: Optional TokenUsage accumulator. If provided, token counts from
+ this LLM call are added to it in-place.
+
+ Returns the parsed dict, or None if inference fails.
+ """
+ from .utils import aperform_completion_with_backoff
+
+ url_line = f"URL: {url}\n" if url else ""
+ prompt = (
+ "You are given a data extraction request and a snippet of HTML from a webpage.\n"
+ "Your job is to produce a single example JSON object representing ONE item "
+ "that the user wants to extract.\n\n"
+ "Rules:\n"
+ "- Return ONLY a valid JSON object β one flat object, NOT wrapped in an array or outer key.\n"
+ "- The object represents a single repeated item (e.g., one product, one article, one row).\n"
+ "- Use clean snake_case field names matching the user's description.\n"
+ "- If the item has nested repeated sub-items, represent those as an array with one example inside.\n"
+ "- Fill values with realistic examples from the HTML so the meaning is clear.\n\n"
+ 'Example β if the request is "extract product name, price, and reviews":\n'
+ '{"name": "Widget Pro", "price": "$29.99", "reviews": [{"author": "Jane", "text": "Great product"}]}\n\n'
+ f"{url_line}"
+ f"Extraction request: {query}\n\n"
+ f"HTML snippet:\n```html\n{html_snippet[:2000]}\n```\n\n"
+ "Return ONLY the JSON object for ONE item:"
+ )
+
+ try:
+ response = await aperform_completion_with_backoff(
+ provider=llm_config.provider,
+ prompt_with_variables=prompt,
+ json_response=True,
+ api_token=llm_config.api_token,
+ base_url=llm_config.base_url,
+ )
+ if usage is not None:
+ usage.completion_tokens += response.usage.completion_tokens
+ usage.prompt_tokens += response.usage.prompt_tokens
+ usage.total_tokens += response.usage.total_tokens
+ raw = response.choices[0].message.content
+ if not raw or not raw.strip():
+ return None
+ return json.loads(_strip_markdown_fences(raw))
+ except Exception:
+ return None
+
+ @staticmethod
+ def _extract_expected_fields(target_json: dict) -> List[str]:
+ """Extract top-level field names from a target JSON example."""
+ return list(target_json.keys())
+
_GENERATE_SCHEMA_UNWANTED_PROPS = {
'provider': 'Instead, use llm_config=LLMConfig(provider="...")',
'api_token': 'Instead, use llm_config=LlMConfig(api_token="...")',
@@ -1342,96 +1746,301 @@ def _build_schema_prompt(html: str, schema_type: str, query: str = None, target_
@staticmethod
def generate_schema(
- html: str,
+ html: str = None,
schema_type: str = "CSS",
query: str = None,
target_json_example: str = None,
llm_config: 'LLMConfig' = create_llm_config(),
provider: str = None,
api_token: str = None,
+ url: Union[str, List[str]] = None,
+ validate: bool = True,
+ max_refinements: int = 3,
+ usage: 'TokenUsage' = None,
**kwargs
) -> dict:
"""
- Generate extraction schema from HTML content and optional query (sync version).
+ Generate extraction schema from HTML content or URL(s) (sync version).
Args:
- html (str): The HTML content to analyze
- query (str, optional): Natural language description of what data to extract
- provider (str): Legacy Parameter. LLM provider to use
- api_token (str): Legacy Parameter. API token for LLM provider
- llm_config (LLMConfig): LLM configuration object
- **kwargs: Additional args passed to LLM processor
+ html (str, optional): The HTML content to analyze. If not provided, url must be set.
+ schema_type (str): "CSS" or "XPATH". Defaults to "CSS".
+ query (str, optional): Natural language description of what data to extract.
+ target_json_example (str, optional): Example of desired JSON output.
+ llm_config (LLMConfig): LLM configuration object.
+ provider (str): Legacy Parameter. LLM provider to use.
+ api_token (str): Legacy Parameter. API token for LLM provider.
+ url (str or List[str], optional): URL(s) to fetch HTML from. If provided, html parameter is ignored.
+ When multiple URLs are provided, HTMLs are fetched in parallel and concatenated.
+ validate (bool): If True, validate the schema against the HTML and
+ refine via LLM feedback loop. Defaults to False (zero overhead).
+ max_refinements (int): Max refinement rounds when validate=True. Defaults to 3.
+ usage (TokenUsage, optional): Token usage accumulator. If provided,
+ token counts from all LLM calls (including inference and
+ validation retries) are added to it in-place.
+ **kwargs: Additional args passed to LLM processor.
Returns:
- dict: Generated schema following the JsonElementExtractionStrategy format
- """
- from .utils import perform_completion_with_backoff
-
- for name, message in JsonElementExtractionStrategy._GENERATE_SCHEMA_UNWANTED_PROPS.items():
- if locals()[name] is not None:
- raise AttributeError(f"Setting '{name}' is deprecated. {message}")
+ dict: Generated schema following the JsonElementExtractionStrategy format.
- prompt = JsonElementExtractionStrategy._build_schema_prompt(html, schema_type, query, target_json_example)
+ Raises:
+ ValueError: If neither html nor url is provided.
+ """
+ import asyncio
try:
- response = perform_completion_with_backoff(
- provider=llm_config.provider,
- prompt_with_variables=prompt,
- json_response=True,
- api_token=llm_config.api_token,
- base_url=llm_config.base_url,
- extra_args=kwargs
- )
- return json.loads(response.choices[0].message.content)
- except Exception as e:
- raise Exception(f"Failed to generate schema: {str(e)}")
+ loop = asyncio.get_running_loop()
+ except RuntimeError:
+ loop = None
+
+ coro = JsonElementExtractionStrategy.agenerate_schema(
+ html=html,
+ schema_type=schema_type,
+ query=query,
+ target_json_example=target_json_example,
+ llm_config=llm_config,
+ provider=provider,
+ api_token=api_token,
+ url=url,
+ validate=validate,
+ max_refinements=max_refinements,
+ usage=usage,
+ **kwargs
+ )
+
+ if loop is None:
+ return asyncio.run(coro)
+ else:
+ import concurrent.futures
+ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
+ future = executor.submit(asyncio.run, coro)
+ return future.result()
@staticmethod
async def agenerate_schema(
- html: str,
+ html: str = None,
schema_type: str = "CSS",
query: str = None,
target_json_example: str = None,
llm_config: 'LLMConfig' = None,
+ provider: str = None,
+ api_token: str = None,
+ url: Union[str, List[str]] = None,
+ validate: bool = True,
+ max_refinements: int = 3,
+ usage: 'TokenUsage' = None,
**kwargs
) -> dict:
"""
- Generate extraction schema from HTML content (async version).
+ Generate extraction schema from HTML content or URL(s) (async version).
Use this method when calling from async contexts (e.g., FastAPI) to avoid
issues with certain LLM providers (e.g., Gemini/Vertex AI) that require
async execution.
Args:
- html (str): The HTML content to analyze
- schema_type (str): "CSS" or "XPATH"
- query (str, optional): Natural language description of what data to extract
- target_json_example (str, optional): Example of desired JSON output
- llm_config (LLMConfig): LLM configuration object
- **kwargs: Additional args passed to LLM processor
+ html (str, optional): The HTML content to analyze. If not provided, url must be set.
+ schema_type (str): "CSS" or "XPATH". Defaults to "CSS".
+ query (str, optional): Natural language description of what data to extract.
+ target_json_example (str, optional): Example of desired JSON output.
+ llm_config (LLMConfig): LLM configuration object.
+ provider (str): Legacy Parameter. LLM provider to use.
+ api_token (str): Legacy Parameter. API token for LLM provider.
+ url (str or List[str], optional): URL(s) to fetch HTML from. If provided, html parameter is ignored.
+ When multiple URLs are provided, HTMLs are fetched in parallel and concatenated.
+ validate (bool): If True, validate the schema against the HTML and
+ refine via LLM feedback loop. Defaults to False (zero overhead).
+ max_refinements (int): Max refinement rounds when validate=True. Defaults to 3.
+ usage (TokenUsage, optional): Token usage accumulator. If provided,
+ token counts from all LLM calls (including inference and
+ validation retries) are added to it in-place.
+ **kwargs: Additional args passed to LLM processor.
Returns:
- dict: Generated schema following the JsonElementExtractionStrategy format
+ dict: Generated schema following the JsonElementExtractionStrategy format.
+
+ Raises:
+ ValueError: If neither html nor url is provided.
"""
- from .utils import aperform_completion_with_backoff
+ from .utils import aperform_completion_with_backoff, preprocess_html_for_schema
+
+ # Validate inputs
+ if html is None and (url is None or (isinstance(url, list) and len(url) == 0)):
+ raise ValueError("Either 'html' or 'url' must be provided")
+
+ # Check deprecated parameters
+ for name, message in JsonElementExtractionStrategy._GENERATE_SCHEMA_UNWANTED_PROPS.items():
+ if locals()[name] is not None:
+ raise AttributeError(f"Setting '{name}' is deprecated. {message}")
if llm_config is None:
llm_config = create_llm_config()
+ # Save original HTML(s) before preprocessing (for validation against real HTML)
+ original_htmls = []
+
+ # Fetch HTML from URL(s) if provided
+ if url is not None:
+ from .async_webcrawler import AsyncWebCrawler
+ from .async_configs import BrowserConfig, CrawlerRunConfig, CacheMode
+
+ browser_config = BrowserConfig(
+ headless=True,
+ text_mode=True,
+ light_mode=True,
+ )
+ crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
+
+ # Normalize to list
+ urls = [url] if isinstance(url, str) else url
+
+ async with AsyncWebCrawler(config=browser_config) as crawler:
+ if len(urls) == 1:
+ result = await crawler.arun(url=urls[0], config=crawler_config)
+ if not result.success:
+ raise Exception(f"Failed to fetch URL '{urls[0]}': {result.error_message}")
+ if result.status_code >= 400:
+ raise Exception(f"HTTP {result.status_code} error for URL '{urls[0]}'")
+ html = result.html
+ original_htmls = [result.html]
+ else:
+ results = await crawler.arun_many(urls=urls, config=crawler_config)
+ html_parts = []
+ for i, result in enumerate(results, 1):
+ if not result.success:
+ raise Exception(f"Failed to fetch URL '{result.url}': {result.error_message}")
+ if result.status_code >= 400:
+ raise Exception(f"HTTP {result.status_code} error for URL '{result.url}'")
+ original_htmls.append(result.html)
+ cleaned = preprocess_html_for_schema(
+ html_content=result.html,
+ text_threshold=2000,
+ attr_value_threshold=500,
+ max_size=500_000
+ )
+ header = HTML_EXAMPLE_DELIMITER.format(index=i)
+ html_parts.append(f"{header}\n{cleaned}")
+ html = "\n\n".join(html_parts)
+ else:
+ original_htmls = [html]
+
+ # Preprocess HTML for schema generation (skip if already preprocessed from multiple URLs)
+ if url is None or isinstance(url, str):
+ html = preprocess_html_for_schema(
+ html_content=html,
+ text_threshold=2000,
+ attr_value_threshold=500,
+ max_size=500_000
+ )
+
+ # --- Resolve expected fields for strict validation ---
+ expected_fields = None
+ if validate:
+ if target_json_example:
+ # User provided target JSON β extract field names from it
+ try:
+ if isinstance(target_json_example, str):
+ target_obj = json.loads(target_json_example)
+ else:
+ target_obj = target_json_example
+ expected_fields = JsonElementExtractionStrategy._extract_expected_fields(target_obj)
+ except (json.JSONDecodeError, TypeError):
+ pass
+ elif query:
+ # No target JSON but query describes fields β infer via quick LLM call
+ first_url = None
+ if url is not None:
+ first_url = url if isinstance(url, str) else url[0]
+ inferred = await JsonElementExtractionStrategy._infer_target_json(
+ query=query, html_snippet=html, llm_config=llm_config, url=first_url, usage=usage
+ )
+ if inferred:
+ expected_fields = JsonElementExtractionStrategy._extract_expected_fields(inferred)
+ # Also inject as target_json_example for the schema prompt
+ if not target_json_example:
+ target_json_example = json.dumps(inferred, indent=2)
+
prompt = JsonElementExtractionStrategy._build_schema_prompt(html, schema_type, query, target_json_example)
+ messages = [{"role": "user", "content": prompt}]
- try:
- response = await aperform_completion_with_backoff(
- provider=llm_config.provider,
- prompt_with_variables=prompt,
- json_response=True,
- api_token=llm_config.api_token,
- base_url=llm_config.base_url,
- extra_args=kwargs
+ prev_schema_json = None
+ last_schema = None
+ max_attempts = 1 + (max_refinements if validate else 0)
+
+ for attempt in range(max_attempts):
+ try:
+ response = await aperform_completion_with_backoff(
+ provider=llm_config.provider,
+ prompt_with_variables=prompt,
+ json_response=True,
+ api_token=llm_config.api_token,
+ base_url=llm_config.base_url,
+ messages=messages,
+ extra_args=kwargs,
+ )
+ if usage is not None:
+ usage.completion_tokens += response.usage.completion_tokens
+ usage.prompt_tokens += response.usage.prompt_tokens
+ usage.total_tokens += response.usage.total_tokens
+ raw = response.choices[0].message.content
+ if not raw or not raw.strip():
+ raise ValueError("LLM returned an empty response")
+
+ schema = json.loads(_strip_markdown_fences(raw))
+ last_schema = schema
+ except json.JSONDecodeError as e:
+ # JSON parse failure β ask LLM to fix it
+ if not validate or attempt >= max_attempts - 1:
+ raise Exception(f"Failed to parse schema JSON: {str(e)}")
+ messages.append({"role": "assistant", "content": raw})
+ messages.append({"role": "user", "content": (
+ f"Your response was not valid JSON. Parse error: {e}\n"
+ "Please return ONLY valid JSON, nothing else."
+ )})
+ continue
+ except Exception as e:
+ raise Exception(f"Failed to generate schema: {str(e)}")
+
+ # If validation is off, return immediately (zero overhead path)
+ if not validate:
+ return schema
+
+ # --- Validation feedback loop ---
+ # Validate against original HTML(s); success if works on at least one
+ best_result = None
+ for orig_html in original_htmls:
+ vr = JsonElementExtractionStrategy._validate_schema(
+ schema, orig_html, schema_type,
+ expected_fields=expected_fields,
+ )
+ if best_result is None or vr["populated_fields"] > best_result["populated_fields"]:
+ best_result = vr
+ if vr["success"]:
+ break
+
+ if best_result["success"]:
+ return schema
+
+ # Last attempt β return best-effort
+ if attempt >= max_attempts - 1:
+ return schema
+
+ # Detect repeated schema
+ current_json = json.dumps(schema, sort_keys=True)
+ is_repeated = current_json == prev_schema_json
+ prev_schema_json = current_json
+
+ # Build feedback and extend conversation
+ feedback = JsonElementExtractionStrategy._build_feedback_message(
+ best_result, schema, attempt + 1, is_repeated
)
- return json.loads(response.choices[0].message.content)
- except Exception as e:
- raise Exception(f"Failed to generate schema: {str(e)}")
+ messages.append({"role": "assistant", "content": raw})
+ messages.append({"role": "user", "content": feedback})
+
+ # Should not reach here, but return last schema as safety net
+ if last_schema is not None:
+ return last_schema
+ raise Exception("Failed to generate schema: no attempts succeeded")
class JsonCssExtractionStrategy(JsonElementExtractionStrategy):
"""
@@ -1480,6 +2089,21 @@ def _get_element_html(self, element) -> str:
def _get_element_attribute(self, element, attribute: str):
return element.get(attribute)
+ def _resolve_source(self, element, source: str):
+ source = source.strip()
+ if not source.startswith("+"):
+ return None
+ sel = source[1:].strip() # e.g. "tr", "tr.subtext", ".classname"
+ parts = sel.split(".")
+ tag = parts[0].strip() or None
+ classes = [p.strip() for p in parts[1:] if p.strip()]
+ kwargs = {}
+ if classes:
+ kwargs["class_"] = lambda c, _cls=classes: c and all(
+ cl in c for cl in _cls
+ )
+ return element.find_next_sibling(tag, **kwargs)
+
class JsonLxmlExtractionStrategy(JsonElementExtractionStrategy):
def __init__(self, schema: Dict[str, Any], **kwargs):
kwargs["input_format"] = "html"
@@ -1745,7 +2369,22 @@ def _get_element_attribute(self, element, attribute: str):
if self.verbose:
print(f"Error getting attribute '{attribute}': {e}")
return None
-
+
+ def _resolve_source(self, element, source: str):
+ source = source.strip()
+ if not source.startswith("+"):
+ return None
+ sel = source[1:].strip()
+ parts = sel.split(".")
+ tag = parts[0].strip() or "*"
+ classes = [p.strip() for p in parts[1:] if p.strip()]
+ xpath = f"./following-sibling::{tag}"
+ for cls in classes:
+ xpath += f"[contains(concat(' ',normalize-space(@class),' '),' {cls} ')]"
+ xpath += "[1]"
+ results = element.xpath(xpath)
+ return results[0] if results else None
+
def _clear_caches(self):
"""Clear caches to free memory"""
if self.use_caching:
@@ -1846,7 +2485,22 @@ def _get_element_html(self, element) -> str:
return etree.tostring(element, encoding='unicode')
def _get_element_attribute(self, element, attribute: str):
- return element.get(attribute)
+ return element.get(attribute)
+
+ def _resolve_source(self, element, source: str):
+ source = source.strip()
+ if not source.startswith("+"):
+ return None
+ sel = source[1:].strip()
+ parts = sel.split(".")
+ tag = parts[0].strip() or "*"
+ classes = [p.strip() for p in parts[1:] if p.strip()]
+ xpath = f"./following-sibling::{tag}"
+ for cls in classes:
+ xpath += f"[contains(concat(' ',normalize-space(@class),' '),' {cls} ')]"
+ xpath += "[1]"
+ results = element.xpath(xpath)
+ return results[0] if results else None
class JsonXPathExtractionStrategy(JsonElementExtractionStrategy):
"""
@@ -1912,6 +2566,21 @@ def _get_element_html(self, element) -> str:
def _get_element_attribute(self, element, attribute: str):
return element.get(attribute)
+ def _resolve_source(self, element, source: str):
+ source = source.strip()
+ if not source.startswith("+"):
+ return None
+ sel = source[1:].strip()
+ parts = sel.split(".")
+ tag = parts[0].strip() or "*"
+ classes = [p.strip() for p in parts[1:] if p.strip()]
+ xpath = f"./following-sibling::{tag}"
+ for cls in classes:
+ xpath += f"[contains(concat(' ',normalize-space(@class),' '),' {cls} ')]"
+ xpath += "[1]"
+ results = element.xpath(xpath)
+ return results[0] if results else None
+
"""
RegexExtractionStrategy
Fast, zero-LLM extraction of common entities via regular expressions.
diff --git a/crawl4ai/html2text/__init__.py b/crawl4ai/html2text/__init__.py
index ca15b4534..9f241bacd 100644
--- a/crawl4ai/html2text/__init__.py
+++ b/crawl4ai/html2text/__init__.py
@@ -316,6 +316,12 @@ def handle_tag(
if self.tag_callback(self, tag, attrs, start) is True:
return
+ # Handle <base> tag to update base URL for relative links
+ if tag == "base" and start:
+ href = attrs.get("href")
+ if href:
+ self.baseurl = href
+
# first thing inside the anchor tag is another tag
# that produces some output
if (
@@ -677,6 +683,11 @@ def link_url(self: HTML2Text, link: str, title: str = "") -> None:
self.o(str(li.num) + ". ")
self.start = True
+ if tag == "caption":
+ if not start:
+ # Ensure caption text ends on its own line before table rows
+ self.soft_br()
+
if tag in ["table", "tr", "td", "th"]:
if self.ignore_tables:
if tag == "tr":
@@ -708,6 +719,9 @@ def link_url(self: HTML2Text, link: str, title: str = "") -> None:
if self.pad_tables:
self.o("<" + config.TABLE_MARKER_FOR_PAD + ">")
self.o(" \n")
+ else:
+ # Ensure table starts on its own line (GFM requirement)
+ self.soft_br()
else:
if self.pad_tables:
# add break in case the table is empty or its 1 row table
@@ -715,18 +729,34 @@ def link_url(self: HTML2Text, link: str, title: str = "") -> None:
self.o("</" + config.TABLE_MARKER_FOR_PAD + ">")
self.o(" \n")
if tag in ["td", "th"] and start:
- if self.split_next_td:
- self.o("| ")
+ if self.pad_tables:
+ # pad_tables mode: keep upstream inter-cell delimiter only
+ # (pad post-processor adds leading/trailing pipes and alignment)
+ if self.split_next_td:
+ self.o("| ")
+ else:
+ # GFM mode: leading pipe on first cell, spaced pipes between cells
+ if self.split_next_td:
+ self.o(" | ")
+ else:
+ self.o("| ")
self.split_next_td = True
if tag == "tr" and start:
self.td_count = 0
if tag == "tr" and not start:
+ if not self.pad_tables:
+ # Add trailing pipe for GFM compliance
+ self.o(" |")
self.split_next_td = False
self.soft_br()
if tag == "tr" and not start and self.table_start:
- # Underline table header
- self.o("|".join(["---"] * self.td_count))
+ if self.pad_tables:
+ # pad_tables: plain separator (post-processor reformats)
+ self.o("|".join(["---"] * self.td_count))
+ else:
+ # GFM: separator with leading/trailing pipes
+ self.o("| " + " | ".join(["---"] * self.td_count) + " |")
self.soft_br()
self.table_start = False
if tag in ["td", "th"] and start:
@@ -1069,6 +1099,15 @@ def update_params(self, **kwargs):
setattr(self, key, value)
def handle_tag(self, tag, attrs, start):
+ # Handle <base> tag to update base URL for relative links
+ # Must be handled before preserved tags since <base> is in <head>
+ if tag == "base" and start:
+ href = attrs.get("href") if attrs else None
+ if href:
+ self.baseurl = href
+ # Also let parent class handle it
+ return super().handle_tag(tag, attrs, start)
+
# Handle preserved tags
if tag in self.preserve_tags:
if start:
@@ -1107,7 +1146,7 @@ def handle_tag(self, tag, attrs, start):
# Handle pre tags
if tag == "pre":
if start:
- self.o("```\n") # Markdown code block start
+ self.o("\n```\n") # Markdown code block start
self.inside_pre = True
else:
self.o("\n```\n") # Markdown code block end
diff --git a/crawl4ai/js_snippet/flatten_shadow_dom.js b/crawl4ai/js_snippet/flatten_shadow_dom.js
new file mode 100644
index 000000000..e13f3f319
--- /dev/null
+++ b/crawl4ai/js_snippet/flatten_shadow_dom.js
@@ -0,0 +1,104 @@
+/**
+ * Flatten all open shadow DOM trees into the light DOM so that
+ * page.content() / outerHTML can serialize the full composed view.
+ *
+ * Uses manual recursive serialization with proper slot resolution.
+ * Resolves slots via the live DOM API (assignedNodes), skips only
+ * shadow-scoped styles, and produces clean HTML with no regex hacks.
+ *
+ * Returns the full HTML string including shadow content.
+ */
+(() => {
+ const VOID = new Set([
+ 'area','base','br','col','embed','hr','img','input',
+ 'link','meta','param','source','track','wbr'
+ ]);
+
+ // Serialize a DOM node. When it has a shadow root, switch to
+ // shadow-aware serialization that resolves <slot> elements.
+ const serialize = (node) => {
+ if (node.nodeType === Node.TEXT_NODE) return node.textContent;
+ if (node.nodeType === Node.COMMENT_NODE) return '';
+ if (node.nodeType !== Node.ELEMENT_NODE) return '';
+
+ const tag = node.tagName.toLowerCase();
+ const attrs = serializeAttrs(node);
+ let inner = '';
+
+ if (node.shadowRoot) {
+ inner = serializeShadowRoot(node);
+ } else {
+ for (const child of node.childNodes) {
+ inner += serialize(child);
+ }
+ }
+
+ if (VOID.has(tag)) return `<${tag}${attrs}>`;
+ return `<${tag}${attrs}>${inner}</${tag}>`;
+ };
+
+ // Serialize a shadow root's children, resolving slots against
+ // the host's light DOM children.
+ const serializeShadowRoot = (host) => {
+ let result = '';
+ for (const child of host.shadowRoot.childNodes) {
+ result += serializeShadowChild(child, host);
+ }
+ return result;
+ };
+
+ // Serialize a node that lives inside a shadow root.
+ // <style> tags are skipped (scoped CSS, useless outside shadow).
+ // <slot> tags are replaced with their assigned (projected) nodes.
+ const serializeShadowChild = (node, host) => {
+ if (node.nodeType === Node.TEXT_NODE) return node.textContent;
+ if (node.nodeType === Node.COMMENT_NODE) return '';
+ if (node.nodeType !== Node.ELEMENT_NODE) return '';
+
+ const tag = node.tagName.toLowerCase();
+
+ // Skip shadow-scoped styles only
+ if (tag === 'style') return '';
+
+ // Resolve <slot>: replace with projected light DOM content
+ if (tag === 'slot') {
+ const assigned = node.assignedNodes({ flatten: true });
+ if (assigned.length > 0) {
+ let out = '';
+ for (const a of assigned) out += serialize(a);
+ return out;
+ }
+ // No assigned nodes β use the slot's fallback content
+ let fallback = '';
+ for (const child of node.childNodes) {
+ fallback += serializeShadowChild(child, host);
+ }
+ return fallback;
+ }
+
+ const attrs = serializeAttrs(node);
+ let inner = '';
+
+ if (node.shadowRoot) {
+ // Nested shadow root β recurse
+ inner = serializeShadowRoot(node);
+ } else {
+ for (const child of node.childNodes) {
+ inner += serializeShadowChild(child, host);
+ }
+ }
+
+ if (VOID.has(tag)) return `<${tag}${attrs}>`;
+ return `<${tag}${attrs}>${inner}</${tag}>`;
+ };
+
+ const serializeAttrs = (node) => {
+ let s = '';
+ for (const a of node.attributes || []) {
+ s += ` ${a.name}="${a.value.replace(/&/g, '&').replace(/"/g, '"')}"`;
+ }
+ return s;
+ };
+
+ return serialize(document.documentElement);
+})()
diff --git a/crawl4ai/js_snippet/remove_consent_popups.js b/crawl4ai/js_snippet/remove_consent_popups.js
new file mode 100644
index 000000000..9aac8d345
--- /dev/null
+++ b/crawl4ai/js_snippet/remove_consent_popups.js
@@ -0,0 +1,710 @@
+async () => {
+ // Helper: check if element is visible
+ const isVisible = (elem) => {
+ if (!elem) return false;
+ const style = window.getComputedStyle(elem);
+ return style.display !== "none" && style.visibility !== "hidden" && style.opacity !== "0";
+ };
+
+ // =========================================================================
+ // Phase 1: Click "Accept All" buttons (fastest, cleanest dismissal)
+ // =========================================================================
+
+ // CMP-specific accept button selectors (ordered by market share)
+ const cmpAcceptSelectors = [
+ // OneTrust / CookiePro
+ '#onetrust-accept-btn-handler',
+ '#accept-recommended-btn-handler',
+ // Cookiebot (Usercentrics/Cybot)
+ '#CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll',
+ '#CybotCookiebotDialogBodyButtonAccept',
+ '#CybotCookiebotDialogBodyLevelButtonAccept',
+ // Didomi
+ '#didomi-notice-agree-button',
+ '.didomi-button-highlight',
+ // Quantcast Choice
+ '.qc-cmp2-summary-buttons button[mode="primary"]',
+ // Sourcepoint
+ '.sp_choice_type_11',
+ '.sp_choice_type_ACCEPT_ALL',
+ // Google FundingChoices / FC
+ '.fc-button.fc-cta-consent.fc-primary-button',
+ '.fc-cta-consent',
+ '.fc-confirm-choices',
+ // TrustArc
+ '#truste-consent-button',
+ '.truste_popframe .pdynamicbutton .call',
+ // ConsentManager.net
+ '.cmpboxbtnyes',
+ '#cmpwelcomebtnyes',
+ '.cmpboxbtnyescustomchoices',
+ // Osano
+ '.osano-cm-accept-all',
+ '.osano-cm-accept',
+ // Iubenda
+ '#iubenda-cs-accept-btn',
+ '.iubenda-cs-accept-btn',
+ // Complianz (WordPress)
+ '.cmplz-btn.cmplz-accept',
+ // LiveRamp / Fides (Ethyca)
+ '.fides-accept-all-button',
+ '#fides-banner .fides-accept-all-button',
+ // CookieYes
+ '.cky-btn-accept',
+ '[data-cky-tag="accept-button"]',
+ // Klaro
+ '.klaro .cm-btn-accept-all',
+ '.klaro .cm-btn-success',
+ '.klaro .cookie-notice .cm-btn-success',
+ // Termly
+ '[data-tid="banner-accept"]',
+ // CookieFirst
+ 'button[data-cookiefirst-action="accept"]',
+ // CookieScript
+ '#cookiescript_accept',
+ // Borlabs Cookie (WordPress)
+ 'a[data-cookie-accept-all]',
+ '.brlbs-btn-accept-all',
+ // Civic Cookie Control
+ '#ccc-recommended-settings',
+ '#ccc-notify-accept',
+ '.ccc-accept-button',
+ // Cookie Information
+ '.coi-banner__accept',
+ // Evidon / Crownpeak
+ '#_evidon-accept-button',
+ // Axeptio
+ 'button#axeptio_btn_acceptAll',
+ // HubSpot
+ '#hs-eu-confirmation-button',
+ // Ketch
+ '#lanyard_root button[class*="confirmButton"]',
+ // Moove GDPR (WordPress)
+ '.moove-gdpr-infobar-allow-all',
+ // TermsFeed
+ '.cc-nb-okagree',
+ // tarteaucitron.js
+ '#tarteaucitronPersonalize2',
+ '.tarteaucitronAllow',
+ // CookieHub
+ '.ch2-allow-all-btn',
+ // Cookie Notice (WP plugin)
+ '#cn-accept-cookie',
+ // EU Cookie Compliance (Drupal)
+ '.eu-cookie-compliance-banner .agree-button',
+ '.eu-cookie-compliance-banner .accept-all',
+ // WordPress GDPR Cookie Consent
+ '#gdpr-cookie-consent-bar #cookie_action_accept',
+ // Cookie Law Info / WebToffee
+ '[data-cli_action="accept"]',
+ // Shopify Native
+ '#shopify-pc__banner__btn-accept',
+ // Wix Native
+ '[data-hook="ccsu-banner-accept"]',
+ // Finsweet (Webflow)
+ '[fs-consent-element="allow"]',
+ '[fs-cc="banner"] [fs-cc="allow"]',
+ // Pandectes (Shopify)
+ '#pandectes-banner .cc-allow',
+ // Clickio
+ '#cl-consent [data-role="b_agree"]',
+ // Snigel
+ '.snigel-cmp-framework #accept-choices',
+ // Cassie
+ '.cassie-accept-all',
+ // FastCMP
+ '.fast-cmp-home-accept button',
+ // Sibbo
+ '#acceptAllMain',
+ // PubTech
+ '#pt-accept-all',
+ // UniConsent
+ '#unic-agree',
+ // Ezoic
+ '#ez-accept-all',
+ // Transcend
+ '#transcend-consent-manager .inner-container button',
+ // Cloudflare Zaraz
+ '#cf_consent-buttons__accept-all',
+ // CookieConsent v2 (Insites/Osano OSS)
+ '#s-all-bn',
+ // CookieConsent v3 (orestbida)
+ '.cm__btn[data-role="all"]',
+ // Openli / Legal Monster
+ '#lm-accept-all',
+ // UK Cookie Consent (Catapult)
+ '#catapultCookie',
+ // Mediavine
+ '[data-name="mediavine-gdpr-cmp"] [format="primary"]',
+ // Consentmo (Shopify)
+ '.isense-cc-allow',
+ // AdOpt
+ '#adopt-accept-all-button',
+ // Truyo
+ 'button#acceptAllCookieButton',
+ // KConsent
+ '#kc-acceptAndHide',
+ // Gravito
+ '#modalConfirmBtn.gravitoCMP-button',
+ // CT Ultimate GDPR (WordPress)
+ '#ct-ultimate-gdpr-cookie-accept',
+ // Hu-manity
+ '[data-hu-action="cookies-notice-consent-choices-3"]',
+ // GDPR Legal Cookie (Shopify)
+ '.overlay_bc_banner *[data-cookie-accept-all]',
+ // Bing / Microsoft
+ '#bnp_btn_accept',
+ // Privado
+ '#cookie-consent-banner #accept-button',
+ // Cookie Alert
+ 'button[data-controller="cookie-alert/extended/button/accept"]',
+ // iWink / STARTER
+ 'body.cookies-request #cookie-bar .allow-cookies',
+ // Real Cookie Banner (devowl.io)
+ '.rcb-banner-cta-accept-all',
+ ];
+
+ // Generic accept button selectors (attribute-based)
+ const genericAcceptSelectors = [
+ 'button[id*="accept" i]',
+ 'button[class*="accept-all" i]',
+ 'button[class*="acceptAll" i]',
+ 'a[id*="accept" i]',
+ 'button[id*="agree" i]',
+ 'button[class*="agree" i]',
+ 'button[class*="allow-all" i]',
+ 'button[class*="allowAll" i]',
+ 'button[data-action="accept"]',
+ 'button[data-action="accept-all"]',
+ 'button[data-gdpr="accept"]',
+ 'button[data-consent="accept"]',
+ ];
+
+ // Try clicking a CMP-specific accept button
+ const clickButton = async (selectors) => {
+ for (const selector of selectors) {
+ try {
+ const btn = document.querySelector(selector);
+ if (btn && isVisible(btn)) {
+ btn.click();
+ await new Promise(r => setTimeout(r, 300));
+ return true;
+ }
+ } catch (e) { /* continue */ }
+ }
+ return false;
+ };
+
+ let accepted = await clickButton(cmpAcceptSelectors);
+ if (!accepted) accepted = await clickButton(genericAcceptSelectors);
+
+ // Text-content fallback: find buttons by visible text
+ if (!accepted) {
+ const acceptPatterns = [
+ /^accept\s*(all)?(\s*cookies)?$/i,
+ /^allow\s*(all)?(\s*cookies)?$/i,
+ /^i\s*agree$/i,
+ /^agree(\s*(and|&)\s*(close|continue))?$/i,
+ /^got\s*it[!]?$/i,
+ /^consent$/i,
+ /^(accept|agree)\s*&?\s*close$/i,
+ ];
+
+ const candidates = document.querySelectorAll(
+ 'button, a[role="button"], [role="button"], input[type="submit"], input[type="button"]'
+ );
+ for (const btn of candidates) {
+ const text = (btn.textContent || btn.value || '').trim();
+ if (text.length > 0 && text.length < 40) {
+ for (const pattern of acceptPatterns) {
+ if (pattern.test(text) && isVisible(btn)) {
+ btn.click();
+ accepted = true;
+ await new Promise(r => setTimeout(r, 300));
+ break;
+ }
+ }
+ if (accepted) break;
+ }
+ }
+ }
+
+ // Shadow DOM: Usercentrics, Axeptio
+ if (!accepted) {
+ const shadowRoots = [
+ { id: 'usercentrics-root', btn: 'button[data-testid="uc-accept-all-button"]' },
+ { cls: 'axeptio_mount', btn: 'button#axeptio_btn_acceptAll' },
+ ];
+ for (const cfg of shadowRoots) {
+ try {
+ const host = cfg.id
+ ? document.getElementById(cfg.id)
+ : document.querySelector('.' + cfg.cls);
+ if (host && host.shadowRoot) {
+ const btn = host.shadowRoot.querySelector(cfg.btn);
+ if (btn) {
+ btn.click();
+ accepted = true;
+ await new Promise(r => setTimeout(r, 300));
+ break;
+ }
+ }
+ } catch (e) { /* continue */ }
+ }
+ }
+
+ // Iframe-based CMPs (Sourcepoint, FastCMP, TrustArc, Privacy Manager)
+ if (!accepted) {
+ const iframeSelectors = [
+ 'iframe[id^="sp_message_iframe"]',
+ 'iframe#fast-cmp-iframe',
+ 'iframe[id*="consent" i]',
+ 'iframe[title*="consent" i]',
+ 'iframe[title*="cookie" i]',
+ 'iframe[title*="privacy" i]',
+ 'iframe[src*="privacymanager" i]',
+ 'iframe[src*="consent-tool" i]',
+ ];
+ for (const sel of iframeSelectors) {
+ try {
+ const iframe = document.querySelector(sel);
+ if (iframe && iframe.contentDocument) {
+ const iframeDoc = iframe.contentDocument;
+ const btns = iframeDoc.querySelectorAll(
+ 'button[title="Accept All" i], button[title="Accept" i], ' +
+ '.sp_choice_type_11, button.message-button, ' +
+ 'button[class*="accept" i], button[class*="agree" i]'
+ );
+ for (const btn of btns) {
+ if (btn.offsetParent !== null) {
+ btn.click();
+ accepted = true;
+ await new Promise(r => setTimeout(r, 300));
+ break;
+ }
+ }
+ if (accepted) break;
+ }
+ } catch (e) { /* cross-origin iframes will throw SecurityError, expected */ }
+ }
+ }
+
+ // =========================================================================
+ // Phase 2: Try CMP JavaScript APIs
+ // =========================================================================
+
+ // IAB TCF v2 API
+ if (typeof window.__tcfapi === 'function') {
+ try {
+ window.__tcfapi('addEventListener', 2, () => {});
+ } catch (e) { /* continue */ }
+ }
+
+ // Didomi API
+ if (typeof window.Didomi !== 'undefined') {
+ try {
+ window.Didomi.setUserAgreeToAll();
+ } catch (e) { /* continue */ }
+ }
+
+ // Cookiebot API
+ if (typeof window.Cookiebot !== 'undefined') {
+ try {
+ window.Cookiebot.submitCustomConsent(true, true, true);
+ } catch (e) { /* continue */ }
+ }
+
+ // Osano API
+ if (typeof window.Osano !== 'undefined') {
+ try {
+ window.Osano.cm.acceptAll();
+ } catch (e) { /* continue */ }
+ }
+
+ // Klaro API
+ if (typeof window.klaro !== 'undefined') {
+ try {
+ window.klaro.getManager().acceptAll();
+ } catch (e) { /* continue */ }
+ }
+
+ // Wait for CMP animations/transitions
+ await new Promise(r => setTimeout(r, 500));
+
+ // =========================================================================
+ // Phase 3: Remove known CMP containers by selector
+ // =========================================================================
+ const cmpContainerSelectors = [
+ // --- Tier 1: Enterprise CMPs ---
+
+ // OneTrust / CookiePro
+ '#onetrust-consent-sdk',
+ '#onetrust-banner-sdk',
+ '.onetrust-pc-dark-filter',
+ '#onetrust-pc-sdk',
+
+ // Cookiebot (Usercentrics/Cybot)
+ '#CybotCookiebotDialog',
+ '#CybotCookiebotDialogBodyUnderlay',
+ '#dtcookie-container',
+ '#cookiebanner',
+
+ // TrustArc
+ '#truste-consent-track',
+ '.truste_overlay',
+ '.truste_box_overlay',
+ '#truste-consent-content',
+ '#consent_blackbar',
+ '.trustarc-banner-container',
+
+ // Quantcast Choice
+ '.qc-cmp2-container',
+ '#qc-cmp2-main',
+ '.qc-cmp-cleanslate',
+
+ // Didomi
+ '#didomi-host',
+ '#didomi-popup',
+ '#didomi-notice',
+
+ // Usercentrics
+ '#usercentrics-root',
+ '#usercentrics-cmp-ui',
+
+ // Sourcepoint
+ 'div[id^="sp_message_container"]',
+ '.sp_message_container',
+
+ // Google FundingChoices / FC
+ '.fc-consent-root',
+ '.fc-dialog-overlay',
+ '.fc-dialog-container',
+
+ // --- Tier 2: Mid-Market CMPs ---
+
+ // Klaro
+ '.klaro',
+
+ // Osano
+ '.osano-cm-window',
+ '.osano-cm-dialog',
+
+ // Iubenda
+ '#iubenda-cs-banner',
+
+ // Complianz (WordPress)
+ '.cmplz-cookiebanner',
+ '#cmplz-cookiebanner-container',
+
+ // CookieYes
+ '.cky-consent-container',
+ '.cky-overlay',
+
+ // ConsentManager.net
+ '.cmpbox',
+ '#cmpbox',
+ '#cmpbox2',
+ '#cmpwrapper',
+
+ // LiveRamp / Fides (Ethyca)
+ '.fides-overlay',
+ '#fides-banner',
+ '#fides-overlay',
+ '#fides-overlay-wrapper',
+
+ // Termly
+ '#termly-code-snippet-support',
+
+ // CookieFirst
+ '#cookiefirst-root',
+ '.cookiefirst-root',
+
+ // CookieScript
+ '#cookiescript_injected',
+ '.cookiescript_fsd_main',
+
+ // Borlabs Cookie (WordPress)
+ '#BorlabsCookieBox',
+ '._brlbs-bar-wrap',
+ '._brlbs-box-wrap',
+
+ // Civic Cookie Control
+ '#ccc',
+ '#ccc-module',
+ '#ccc-overlay',
+
+ // Cookie Information
+ '#cookie-information-template-wrapper',
+ '#coiOverlay',
+
+ // Evidon / Crownpeak
+ '#_evidon_banner',
+ '#_evidon-background',
+ '#evidon-prefdiag-overlay',
+
+ // Axeptio
+ '.axeptio_widget',
+ '.axeptio_mount',
+
+ // HubSpot
+ '#hs-eu-cookie-confirmation',
+
+ // Ketch
+ '#lanyard_root',
+
+ // --- Tier 3: Regional / WordPress / Specialized CMPs ---
+
+ // tarteaucitron.js
+ '#tarteaucitronRoot',
+ '#tarteaucitronAlertBig',
+
+ // CookieHub
+ '.ch2-container',
+ '.ch2-dialog',
+
+ // Moove GDPR (WordPress)
+ '#moove_gdpr_cookie_info_bar',
+ '#moove_gdpr_cookie_modal',
+ '.gdpr_cookie_settings_popup_overlay',
+
+ // TermsFeed
+ '.termsfeed-com---nb',
+
+ // Cookie Notice (WP plugin)
+ '#cookie-notice',
+
+ // Cookie Law Info / WebToffee
+ '#cookie-law-info-bar',
+ '#cookie-law-bg',
+ '.cli-popupbar-overlay',
+
+ // EU Cookie Compliance (Drupal)
+ '.eu-cookie-compliance-banner',
+
+ // WordPress GDPR Cookie Consent
+ '#gdpr-cookie-consent-bar',
+
+ // Shopify Native
+ '#shopify-pc__banner',
+
+ // Wix Native
+ '[data-comp-type="cookie-banner-root-wix"]',
+ '[data-hook="ccsu-banner-wrapper"]',
+
+ // Finsweet (Webflow)
+ '[fs-consent-element="banner"]',
+ '.fs-cc-components',
+
+ // Pandectes (Shopify)
+ '#pandectes-banner',
+
+ // Clickio
+ '#cl-consent',
+
+ // Snigel
+ '.snigel-cmp-framework',
+
+ // Cassie
+ '.cassie-cookie-module',
+ '.cassie-pre-banner',
+
+ // FastCMP
+ '#fast-cmp-root',
+
+ // Sibbo
+ 'sibbo-cmp-layout',
+
+ // PubTech
+ '#pubtech-cmp',
+
+ // UniConsent
+ '.unic',
+
+ // Ezoic
+ '#ez-cookie-dialog-wrapper',
+
+ // Transcend
+ '#transcend-consent-manager',
+
+ // Cloudflare Zaraz
+ '.cf_modal_container',
+
+ // CookieConsent v2 (Insites/Osano OSS)
+ '#cc--main',
+
+ // CookieConsent v3 (orestbida)
+ '#cc-main',
+
+ // Openli / Legal Monster
+ '.legalmonster-cleanslate',
+
+ // UK Cookie Consent (Catapult)
+ '#catapult-cookie-bar',
+
+ // Sirdata
+ '#sd-cmp',
+
+ // Mediavine
+ '[data-name="mediavine-gdpr-cmp"]',
+
+ // Consentmo (Shopify)
+ '.isense-cc-window',
+
+ // AdOpt
+ '#cookie-banner',
+
+ // Truyo
+ '#truyo-consent-module',
+
+ // KConsent
+ '#kconsent',
+ '.kc-overlay',
+
+ // Gravito
+ '.gravitoCMP-background-overlay',
+
+ // CT Ultimate GDPR (WordPress)
+ '#ct-ultimate-gdpr-cookie-popup',
+
+ // Hu-manity
+ '#hu.hu-wrapper',
+
+ // GDPR Legal Cookie (Shopify)
+ '.overlay_bc_banner',
+
+ // Piwik PRO
+ '.PiwikPROConsentForm-container',
+
+ // Tealium
+ '#__tealiumGDPRecModal',
+ '#__tealiumImplicitmodal',
+ '#consent-layer',
+
+ // PMC (Penske Media)
+ '#pmc-pp-tou--notice',
+
+ // Privado
+ '#cookie-consent-banner',
+
+ // Real Cookie Banner (devowl.io)
+ '.rcb-banner',
+
+ // Bing / Microsoft
+ '#bnp_container',
+ '#bnp_cookie_banner',
+
+ // LinkedIn
+ '.artdeco-global-alert[type="COOKIE_CONSENT"]',
+
+ // Privacy Manager
+ '#gdpr-consent-tool-wrapper',
+
+ // --- Generic patterns (catch-all) ---
+ '[class*="cookie-consent" i]',
+ '[id*="cookie-consent" i]',
+ '[class*="cookie-banner" i]',
+ '[id*="cookie-banner" i]',
+ '[class*="consent-banner" i]',
+ '[id*="consent-banner" i]',
+ '[class*="consent-popup" i]',
+ '[id*="consent-popup" i]',
+ '[class*="gdpr-banner" i]',
+ '[id*="gdpr-banner" i]',
+ '[class*="cookie-notice" i]',
+ '[id*="cookie-notice" i]',
+ '[class*="cookie-law" i]',
+ '[id*="cookie-law" i]',
+ '[class*="cookie-popup" i]',
+ '[id*="cookie-popup" i]',
+ '[class*="cookie-overlay" i]',
+ '[id*="cookie-overlay" i]',
+ '.cc-banner',
+ '.cc-window',
+ ];
+
+ for (const selector of cmpContainerSelectors) {
+ try {
+ const elements = document.querySelectorAll(selector);
+ elements.forEach(el => el.remove());
+ } catch (e) { /* continue */ }
+ }
+
+ // =========================================================================
+ // Phase 4: Remove CMP iframes
+ // =========================================================================
+ const cmpIframeSelectors = [
+ 'iframe[id^="sp_message_iframe"]',
+ 'iframe#fast-cmp-iframe',
+ 'iframe[src*="consent" i]',
+ 'iframe[src*="cookie-cdn" i]',
+ 'iframe[src*="cookiebot" i]',
+ 'iframe[src*="trustarc" i]',
+ 'iframe[src*="consentmanager" i]',
+ 'iframe[src*="privacymanager" i]',
+ 'iframe[src*="cmp-consent-tool" i]',
+ 'iframe[title*="consent" i]',
+ 'iframe[title*="cookie" i]',
+ 'iframe[title*="gdpr" i]',
+ 'iframe[name="__tcfapiLocator"]',
+ ];
+
+ for (const selector of cmpIframeSelectors) {
+ try {
+ const iframes = document.querySelectorAll(selector);
+ iframes.forEach(iframe => {
+ // Also remove parent if it's a CMP wrapper (fixed/high-z)
+ const parent = iframe.parentElement;
+ if (parent && parent.children.length <= 2) {
+ const style = window.getComputedStyle(parent);
+ if (style.position === 'fixed' || style.position === 'absolute' ||
+ parseInt(style.zIndex) > 999) {
+ parent.remove();
+ return;
+ }
+ }
+ iframe.remove();
+ });
+ } catch (e) { /* continue */ }
+ }
+
+ // =========================================================================
+ // Phase 5: Restore body scroll and clean up CMP artifacts
+ // =========================================================================
+
+ // Reset overflow on body and html
+ document.body.style.overflow = '';
+ document.body.style.overflowY = '';
+ document.body.style.position = '';
+ document.body.style.marginRight = '';
+ document.body.style.paddingRight = '';
+ document.documentElement.style.overflow = '';
+ document.documentElement.style.overflowY = '';
+ document.documentElement.style.position = '';
+
+ // Remove known CMP body classes
+ const cmpBodyClasses = [
+ 'ot-overflow-hidden',
+ 'sp_message_open',
+ 'didomi-popup-open',
+ 'cmpbox-show',
+ 'cmplz-blocked',
+ 'qc-cmp2-no-scroll',
+ 'osano-cm-show',
+ 'cky-modal-open',
+ 'fides-overlay-modal-open',
+ 'cc-no-scroll',
+ 'gdpr-cookie-notice-center-loaded',
+ 'fc-consent-root-open',
+ 'cookie-notification-active',
+ 'consent-bar-push-large',
+ 'with-eu-cookie-guideline',
+ 'cookies-request',
+ 'eu-cookie-compliance-popup-open',
+ 'has-cookie-bar',
+ ];
+
+ for (const cls of cmpBodyClasses) {
+ document.body.classList.remove(cls);
+ document.documentElement.classList.remove(cls);
+ }
+};
diff --git a/crawl4ai/link_preview.py b/crawl4ai/link_preview.py
index 13d32d58f..3ed29666e 100644
--- a/crawl4ai/link_preview.py
+++ b/crawl4ai/link_preview.py
@@ -289,17 +289,21 @@ def _merge_head_data(
Returns:
Links object with head_data attached to matching links
"""
- # Create URL to head_data mapping
+ # Create URL to head_data mapping (key by both final and original URL for redirects)
url_to_head_data = {}
for result in head_results:
url = result.get("url")
+ head_info = {
+ "head_data": result.get("head_data", {}),
+ "status": result.get("status", "unknown"),
+ "error": result.get("error"),
+ "relevance_score": result.get("relevance_score")
+ }
if url:
- url_to_head_data[url] = {
- "head_data": result.get("head_data", {}),
- "status": result.get("status", "unknown"),
- "error": result.get("error"),
- "relevance_score": result.get("relevance_score")
- }
+ url_to_head_data[url] = head_info
+ original_url = result.get("original_url")
+ if original_url and original_url != url:
+ url_to_head_data[original_url] = head_info
# Update internal links
updated_internal = []
@@ -336,9 +340,15 @@ def _merge_head_data(
updated_internal.append(updated_link)
else:
- # Keep original link unchanged
+ # Calculate total_score even without head data, using intrinsic_score
+ link.total_score = calculate_total_score(
+ intrinsic_score=getattr(link, 'intrinsic_score', None),
+ contextual_score=None,
+ score_links_enabled=getattr(config, 'score_links', False),
+ query_provided=bool(config.link_preview_config.query)
+ )
updated_internal.append(link)
-
+
# Update external links
updated_external = []
for link in original_links.external:
@@ -374,9 +384,15 @@ def _merge_head_data(
updated_external.append(updated_link)
else:
- # Keep original link unchanged
+ # Calculate total_score even without head data, using intrinsic_score
+ link.total_score = calculate_total_score(
+ intrinsic_score=getattr(link, 'intrinsic_score', None),
+ contextual_score=None,
+ score_links_enabled=getattr(config, 'score_links', False),
+ query_provided=bool(config.link_preview_config.query)
+ )
updated_external.append(link)
-
+
# Sort links by relevance score if available
if any(hasattr(link, 'head_data') and link.head_data and 'relevance_score' in link.head_data
for link in updated_internal + updated_external):
diff --git a/crawl4ai/markdown_generation_strategy.py b/crawl4ai/markdown_generation_strategy.py
index 622cc8dad..8275eb43c 100644
--- a/crawl4ai/markdown_generation_strategy.py
+++ b/crawl4ai/markdown_generation_strategy.py
@@ -8,7 +8,7 @@
from urllib.parse import urljoin
# Pre-compile the regex pattern
-LINK_PATTERN = re.compile(r'!?\[([^\]]+)\]\(([^)]+?)(?:\s+"([^"]*)")?\)')
+LINK_PATTERN = re.compile(r'!?\[((?:[^\[\]]|\[(?:[^\[\]]|\[[^\]]*\])*\])*)\]\(((?:[^()\s]|\([^()]*\))*)(?:\s+"([^"]*)")?\)')
def fast_urljoin(base: str, url: str) -> str:
diff --git a/crawl4ai/models.py b/crawl4ai/models.py
index 930cda235..506538970 100644
--- a/crawl4ai/models.py
+++ b/crawl4ai/models.py
@@ -1,4 +1,5 @@
-from pydantic import BaseModel, HttpUrl, PrivateAttr, Field, ConfigDict
+from pydantic import BaseModel, HttpUrl, PrivateAttr, Field, ConfigDict, BeforeValidator
+from typing import Annotated
from typing import List, Dict, Optional, Callable, Awaitable, Union, Any
from typing import AsyncGenerator
from typing import Generic, TypeVar
@@ -149,6 +150,7 @@ class CrawlResult(BaseModel):
ssl_certificate: Optional[SSLCertificate] = None
dispatch_result: Optional[DispatchResult] = None
redirected_url: Optional[str] = None
+ redirected_status_code: Optional[int] = None
network_requests: Optional[List[Dict[str, Any]]] = None
console_messages: Optional[List[Dict[str, Any]]] = None
tables: List[Dict] = Field(default_factory=list) # NEW β [{headers,rows,caption,summary}]
@@ -156,6 +158,8 @@ class CrawlResult(BaseModel):
head_fingerprint: Optional[str] = None
cached_at: Optional[float] = None
cache_status: Optional[str] = None # "hit", "hit_validated", "hit_fallback", "miss"
+ # Anti-bot retry/proxy usage stats
+ crawl_stats: Optional[Dict[str, Any]] = None
model_config = ConfigDict(arbitrary_types_allowed=True)
@@ -294,6 +298,10 @@ def __init__(self, results: Union[CrawlResultT, List[CrawlResultT]]):
def __iter__(self):
return iter(self._results)
+ async def __aiter__(self):
+ for item in self._results:
+ yield item
+
def __getitem__(self, index):
return self._results[index]
@@ -332,6 +340,7 @@ class AsyncCrawlResponse(BaseModel):
downloaded_files: Optional[List[str]] = None
ssl_certificate: Optional[SSLCertificate] = None
redirected_url: Optional[str] = None
+ redirected_status_code: Optional[int] = None
network_requests: Optional[List[Dict[str, Any]]] = None
console_messages: Optional[List[Dict[str, Any]]] = None
@@ -340,6 +349,15 @@ class AsyncCrawlResponse(BaseModel):
###############################
# Scraping Models
###############################
+def _coerce_int(v):
+ """Coerce to int or return None for non-numeric values like '100%' or 'auto'."""
+ if v is None:
+ return None
+ try:
+ return int(v)
+ except (ValueError, TypeError):
+ return None
+
class MediaItem(BaseModel):
src: Optional[str] = ""
data: Optional[str] = ""
@@ -349,7 +367,7 @@ class MediaItem(BaseModel):
type: str = "image"
group_id: Optional[int] = 0
format: Optional[str] = None
- width: Optional[int] = None
+ width: Annotated[Optional[int], BeforeValidator(_coerce_int)] = None
class Link(BaseModel):
diff --git a/crawl4ai/processors/pdf/__init__.py b/crawl4ai/processors/pdf/__init__.py
index a6627f135..69a6f75a2 100644
--- a/crawl4ai/processors/pdf/__init__.py
+++ b/crawl4ai/processors/pdf/__init__.py
@@ -145,6 +145,7 @@ def _get_pdf_path(self, url: str) -> str:
# Create temp file with .pdf extension
temp_file = tempfile.NamedTemporaryFile(suffix='.pdf', delete=False)
+ temp_file.close() # Close handle immediately; file persists due to delete=False
self._temp_files.append(temp_file.name)
try:
diff --git a/crawl4ai/prompts.py b/crawl4ai/prompts.py
index c306308e5..773d3af38 100644
--- a/crawl4ai/prompts.py
+++ b/crawl4ai/prompts.py
@@ -8,67 +8,6 @@
Your task is to break down this HTML content into semantically relevant blocks, and for each block, generate a JSON object with the following keys:
-- index: an integer representing the index of the block in the content
-- tags: a list of semantic tags that are relevant to the content of the block
-- content: a list of strings containing the text content of the block
-- questions: a list of 3 questions that a user may ask about the content in this block
-
-To generate the JSON objects:
-
-1. Carefully read through the HTML content and identify logical breaks or shifts in the content that would warrant splitting it into separate blocks.
-
-2. For each block:
- a. Assign it an index based on its order in the content.
- b. Analyze the content and generate a list of relevant semantic tags that describe what the block is about.
- c. Extract the text content, clean it up if needed, and store it as a list of strings in the "content" field.
- d. Come up with 3 questions that a user might ask about this specific block of content, based on the tags and content. The questions should be relevant and answerable by the content in the block.
-
-3. Ensure that the order of the JSON objects matches the order of the blocks as they appear in the original HTML content.
-
-4. Double-check that each JSON object includes all required keys (index, tags, content, questions) and that the values are in the expected format (integer, list of strings, etc.).
-
-5. Make sure the generated JSON is complete and parsable, with no errors or omissions.
-
-6. Make sure to escape any special characters in the HTML content, and also single or double quote to avoid JSON parsing issues.
-
-Please provide your output within <blocks> tags, like this:
-
-<blocks>
-[{
- "index": 0,
- "tags": ["introduction", "overview"],
- "content": ["This is the first paragraph of the article, which provides an introduction and overview of the main topic."],
- "questions": [
- "What is the main topic of this article?",
- "What can I expect to learn from reading this article?",
- "Is this article suitable for beginners or experts in the field?"
- ]
-},
-{
- "index": 1,
- "tags": ["history", "background"],
- "content": ["This is the second paragraph, which delves into the history and background of the topic.",
- "It provides context and sets the stage for the rest of the article."],
- "questions": [
- "What historical events led to the development of this topic?",
- "How has the understanding of this topic evolved over time?",
- "What are some key milestones in the history of this topic?"
- ]
-}]
-</blocks>
-
-Remember, the output should be a complete, parsable JSON wrapped in <blocks> tags, with no omissions or errors. The JSON objects should semantically break down the content into relevant blocks, maintaining the original order."""
-
-PROMPT_EXTRACT_BLOCKS = """Here is the URL of the webpage:
-<url>{URL}</url>
-
-And here is the cleaned HTML content of that webpage:
-<html>
-{HTML}
-</html>
-
-Your task is to break down this HTML content into semantically relevant blocks, and for each block, generate a JSON object with the following keys:
-
- index: an integer representing the index of the block in the content
- content: a list of strings containing the text content of the block
@@ -359,6 +298,7 @@
"attribute": "attribute_name", // Optional
"transform": "transformation_type", // Optional
"pattern": "regex_pattern", // Optional
+ "source": "+ sibling_selector", // Optional β navigate to sibling element first
"fields": [] // For nested/list types
}
]
@@ -372,6 +312,27 @@
- nested: Object containing other fields
- list: Array of similar items
- regex: Pattern-based extraction
+
+Optional field keys:
+- source: Navigate to a sibling element before running the selector.
+ Syntax: "+ <css_selector>" β finds the next sibling matching the selector.
+ Example: "source": "+ tr" finds the next sibling <tr> of the base element.
+ Example: "source": "+ tr.subtext" finds the next sibling <tr> with class "subtext".
+ The field's selector then runs inside the resolved sibling element.
+ Use this when a logical item's data is split across sibling elements (e.g. table rows).
+
+CRITICAL - How selectors work at each level:
+- baseSelector runs against the FULL document and returns all matching elements.
+- Field selectors run INSIDE each base element (descendants only, not siblings).
+- This means a field selector will NEVER match sibling elements of the base element.
+- To reach sibling data, use the "source" key to navigate to the sibling first.
+- Therefore: NEVER use the same (or equivalent) selector as baseSelector in a field.
+ It would search for the element inside itself, which returns nothing for flat/sibling layouts.
+
+When repeating items are siblings (e.g. table rows, flat divs):
+- CORRECT: Use baseSelector to match each item, then use flat fields (text/attribute) to extract data directly from within each item.
+- WRONG: Using baseSelector as a "list" field selector inside itself β this produces empty arrays.
+- For data in sibling elements: Use "source" to navigate to the sibling, then extract from there.
</type_definitions>
<behavior_rules>
@@ -387,8 +348,9 @@
- Include prices, dates, titles, and other common data types
3. Always:
- - Use reliable CSS selectors
- - Handle dynamic class names appropriately
+ - Use reliable CSS selectors that will survive page rebuilds
+ - NEVER use auto-generated or hashed class names (e.g. `.styles_card__xK9r2`, `.css-1a2b3c`, `.sc-bdnxRM`). These are generated by CSS-in-JS tools and change on every build β your schema will break on the next crawl.
+ - PREFER: `data-*` attributes (`[data-testid="review"]`), semantic tags (`article`, `section`, `nav`), ARIA attributes (`[role="listitem"]`), and stable class names that reflect meaning (`.product-card`, `.review`).
- Create descriptive field names
- Follow consistent naming conventions
</behavior_rules>
@@ -667,6 +629,71 @@
}
]
}
+
+7. Sibling Rows Example (e.g. table rows, flat lists):
+<html>
+<table>
+ <tr class="item"><td class="title"><a href="/1">First</a></td></tr>
+ <tr class="item"><td class="title"><a href="/2">Second</a></td></tr>
+</table>
+</html>
+
+WRONG Schema (baseSelector reused as list field β produces empty results):
+{
+ "name": "Items",
+ "baseSelector": ".item",
+ "fields": [
+ {
+ "name": "entries",
+ "type": "list",
+ "selector": ".item",
+ "fields": [
+ {"name": "title", "selector": ".title a", "type": "text"}
+ ]
+ }
+ ]
+}
+
+CORRECT Schema (flat fields directly on base element):
+{
+ "name": "Items",
+ "baseSelector": ".item",
+ "fields": [
+ {"name": "title", "selector": ".title a", "type": "text"},
+ {"name": "link", "selector": ".title a", "type": "attribute", "attribute": "href"}
+ ]
+}
+
+8. Sibling Data Example (data split across sibling elements):
+<html>
+<table>
+ <tr class="athing submission">
+ <td class="title"><span class="rank">1.</span></td>
+ <td><span class="titleline"><a href="https://example.com">Example Title</a></span></td>
+ </tr>
+ <tr>
+ <td colspan="2"></td>
+ <td class="subtext">
+ <span class="score">100 points</span>
+ <a class="hnuser">johndoe</a>
+ <a>50 comments</a>
+ </td>
+ </tr>
+</table>
+</html>
+
+Generated Schema (using "source" to reach sibling row):
+{
+ "name": "HN Submissions",
+ "baseSelector": "tr.athing.submission",
+ "fields": [
+ {"name": "rank", "selector": "span.rank", "type": "text"},
+ {"name": "title", "selector": "span.titleline a", "type": "text"},
+ {"name": "url", "selector": "span.titleline a", "type": "attribute", "attribute": "href"},
+ {"name": "score", "selector": "span.score", "type": "text", "source": "+ tr"},
+ {"name": "author", "selector": "a.hnuser", "type": "text", "source": "+ tr"}
+ ]
+}
</examples>
@@ -735,6 +762,7 @@
"attribute": "attribute_name", // Optional
"transform": "transformation_type", // Optional
"pattern": "regex_pattern", // Optional
+ "source": "+ sibling_selector", // Optional β navigate to sibling element first
"fields": [] // For nested/list types
}
]
@@ -748,6 +776,27 @@
- nested: Object containing other fields
- list: Array of similar items
- regex: Pattern-based extraction
+
+Optional field keys:
+- source: Navigate to a sibling element before running the selector.
+ Syntax: "+ <selector>" β finds the next sibling matching the selector.
+ Example: "source": "+ tr" finds the next sibling <tr> of the base element.
+ Example: "source": "+ tr.subtext" finds the next sibling <tr> with class "subtext".
+ The field's selector then runs inside the resolved sibling element.
+ Use this when a logical item's data is split across sibling elements (e.g. table rows).
+
+CRITICAL - How selectors work at each level:
+- baseSelector runs against the FULL document and returns all matching elements.
+- Field selectors run INSIDE each base element (descendants only, not siblings).
+- This means a field selector will NEVER match sibling elements of the base element.
+- To reach sibling data, use the "source" key to navigate to the sibling first.
+- Therefore: NEVER use the same (or equivalent) selector as baseSelector in a field.
+ It would search for the element inside itself, which returns nothing for flat/sibling layouts.
+
+When repeating items are siblings (e.g. table rows, flat divs):
+- CORRECT: Use baseSelector to match each item, then use flat fields (text/attribute) to extract data directly from within each item.
+- WRONG: Using baseSelector as a "list" field selector inside itself β this produces empty arrays.
+- For data in sibling elements: Use "source" to navigate to the sibling, then extract from there.
</type_definitions>
<behavior_rules>
@@ -763,8 +812,9 @@
- Include prices, dates, titles, and other common data types
3. Always:
- - Use reliable XPath selectors
- - Handle dynamic element IDs appropriately
+ - Use reliable XPath selectors that will survive page rebuilds
+ - NEVER use auto-generated or hashed class names (e.g. `styles_card__xK9r2`, `css-1a2b3c`, `sc-bdnxRM`). These are generated by CSS-in-JS tools and change on every build β your schema will break on the next crawl.
+ - PREFER: `data-*` attributes (`@data-testid='review'`), semantic tags (`article`, `section`, `nav`), ARIA attributes (`@role='listitem'`), and stable class names that reflect meaning (`product-card`, `review`).
- Create descriptive field names
- Follow consistent naming conventions
</behavior_rules>
@@ -1043,6 +1093,71 @@
}
]
}
+
+7. Sibling Rows Example (e.g. table rows, flat lists):
+<html>
+<table>
+ <tr class="item"><td class="title"><a href="/1">First</a></td></tr>
+ <tr class="item"><td class="title"><a href="/2">Second</a></td></tr>
+</table>
+</html>
+
+WRONG Schema (baseSelector reused as list field β produces empty results):
+{
+ "name": "Items",
+ "baseSelector": ".//tr[@class='item']",
+ "fields": [
+ {
+ "name": "entries",
+ "type": "list",
+ "selector": ".//tr[@class='item']",
+ "fields": [
+ {"name": "title", "selector": ".//td[@class='title']/a", "type": "text"}
+ ]
+ }
+ ]
+}
+
+CORRECT Schema (flat fields directly on base element):
+{
+ "name": "Items",
+ "baseSelector": ".//tr[@class='item']",
+ "fields": [
+ {"name": "title", "selector": ".//td[@class='title']/a", "type": "text"},
+ {"name": "link", "selector": ".//td[@class='title']/a", "type": "attribute", "attribute": "href"}
+ ]
+}
+
+8. Sibling Data Example (data split across sibling elements):
+<html>
+<table>
+ <tr class="athing submission">
+ <td class="title"><span class="rank">1.</span></td>
+ <td><span class="titleline"><a href="https://example.com">Example Title</a></span></td>
+ </tr>
+ <tr>
+ <td colspan="2"></td>
+ <td class="subtext">
+ <span class="score">100 points</span>
+ <a class="hnuser">johndoe</a>
+ <a>50 comments</a>
+ </td>
+ </tr>
+</table>
+</html>
+
+Generated Schema (using "source" to reach sibling row):
+{
+ "name": "HN Submissions",
+ "baseSelector": "//tr[contains(@class, 'athing') and contains(@class, 'submission')]",
+ "fields": [
+ {"name": "rank", "selector": ".//span[@class='rank']", "type": "text"},
+ {"name": "title", "selector": ".//span[@class='titleline']/a", "type": "text"},
+ {"name": "url", "selector": ".//span[@class='titleline']/a", "type": "attribute", "attribute": "href"},
+ {"name": "score", "selector": ".//span[@class='score']", "type": "text", "source": "+ tr"},
+ {"name": "author", "selector": ".//a[@class='hnuser']", "type": "text", "source": "+ tr"}
+ ]
+}
</examples>
<output_requirements>
diff --git a/crawl4ai/utils.py b/crawl4ai/utils.py
index e5efba07f..4b3d96906 100644
--- a/crawl4ai/utils.py
+++ b/crawl4ai/utils.py
@@ -226,7 +226,7 @@ def merge_chunks(
class VersionManager:
def __init__(self):
- self.home_dir = Path.home() / ".crawl4ai"
+ self.home_dir = Path(os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home())) / ".crawl4ai"
self.version_file = self.home_dir / "version.txt"
def get_installed_version(self):
@@ -1697,7 +1697,7 @@ def extract_xml_data_legacy(tags, string):
data = {}
for tag in tags:
- pattern = f"<{tag}>(.*?)</{tag}>"
+ pattern = f"<{tag}>((?:(?!<{tag}>).)*)</{tag}>"
match = re.search(pattern, string, re.DOTALL)
if match:
data[tag] = match.group(1).strip()
@@ -1726,7 +1726,7 @@ def extract_xml_data(tags, string):
data = {}
for tag in tags:
- pattern = f"<{tag}>(.*?)</{tag}>"
+ pattern = f"<{tag}>((?:(?!<{tag}>).)*)</{tag}>"
matches = re.findall(pattern, string, re.DOTALL)
if matches:
@@ -1748,6 +1748,7 @@ def perform_completion_with_backoff(
base_delay=2,
max_attempts=3,
exponential_factor=2,
+ messages=None,
**kwargs,
):
"""
@@ -1789,7 +1790,7 @@ def perform_completion_with_backoff(
try:
response = completion(
model=provider,
- messages=[{"role": "user", "content": prompt_with_variables}],
+ messages=messages if messages is not None else [{"role": "user", "content": prompt_with_variables}],
**extra_args,
)
return response # Return the successful response
@@ -1839,6 +1840,7 @@ async def aperform_completion_with_backoff(
base_delay=2,
max_attempts=3,
exponential_factor=2,
+ messages=None,
**kwargs,
):
"""
@@ -1881,7 +1883,7 @@ async def aperform_completion_with_backoff(
try:
response = await acompletion(
model=provider,
- messages=[{"role": "user", "content": prompt_with_variables}],
+ messages=messages if messages is not None else [{"role": "user", "content": prompt_with_variables}],
**extra_args,
)
return response # Return the successful response
@@ -2292,14 +2294,14 @@ def normalize_url(
# IMPORTANT: Don't use quote(unquote()) as it mangles + signs in URLs
# The path from urlparse is already properly encoded
path = parsed.path
- if path.endswith('/') and path != '/':
- path = path.rstrip('/')
+ # Preserve trailing slashes -- they are semantically significant per RFC 3986
+ # e.g. /page/9123/ and /page/9123 may return different responses
# ββ query ββ
query = parsed.query
if query:
# explode, mutate, then rebuild
- params = [(k.lower(), v) for k, v in parse_qsl(query, keep_blank_values=True)]
+ params = [(k, v) for k, v in parse_qsl(query, keep_blank_values=True)]
if drop_query_tracking:
default_tracking = {
@@ -2308,7 +2310,7 @@ def normalize_url(
}
if extra_drop_params:
default_tracking |= {p.lower() for p in extra_drop_params}
- params = [(k, v) for k, v in params if k not in default_tracking]
+ params = [(k, v) for k, v in params if k.lower() not in default_tracking]
if sort_query:
params.sort(key=lambda kv: kv[0])
@@ -2381,7 +2383,7 @@ def normalize_url_for_deep_crawl(href, base_url, preserve_https=False, original_
normalized = urlunparse((
parsed.scheme,
netloc,
- parsed.path.rstrip('/'), # Normalize trailing slash
+ parsed.path or '/', # Preserve trailing slash
parsed.params,
query,
fragment
@@ -2420,7 +2422,7 @@ def efficient_normalize_url_for_deep_crawl(href, base_url, preserve_https=False,
normalized = urlunparse((
parsed.scheme,
parsed.netloc.lower(),
- parsed.path.rstrip('/'),
+ parsed.path or '/', # Preserve trailing slash
parsed.params,
parsed.query,
'' # Remove fragment
@@ -2588,9 +2590,9 @@ def is_external_url(url: str, base_domain: str) -> bool:
if not parsed.netloc: # Relative URL
return False
- # Strip 'www.' from both domains for comparison
- url_domain = parsed.netloc.lower().replace("www.", "")
- base = base_domain.lower().replace("www.", "")
+ # Strip port and 'www.' from both domains for comparison
+ url_domain = parsed.netloc.lower().split(":")[0].replace("www.", "")
+ base = base_domain.lower().split(":")[0].replace("www.", "")
# Check if URL domain ends with base domain
return not url_domain.endswith(base)
diff --git a/deploy/docker/README.md b/deploy/docker/README.md
index c3c968f4a..757dbcf2f 100644
--- a/deploy/docker/README.md
+++ b/deploy/docker/README.md
@@ -59,11 +59,11 @@ Pull and run images directly from Docker Hub without building locally.
#### 1. Pull the Image
-Our latest stable release is `0.8.0`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.
+Our latest stable release is `0.8.5`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.
```bash
-# Pull the latest stable version (0.8.0)
-docker pull unclecode/crawl4ai:0.8.0
+# Pull the latest stable version (0.8.5)
+docker pull unclecode/crawl4ai:0.8.5
# Or use the latest tag (points to 0.8.0)
docker pull unclecode/crawl4ai:latest
@@ -100,7 +100,7 @@ EOL
-p 11235:11235 \
--name crawl4ai \
--shm-size=1g \
- unclecode/crawl4ai:0.8.0
+ unclecode/crawl4ai:0.8.5
```
* **With LLM support:**
@@ -111,7 +111,7 @@ EOL
--name crawl4ai \
--env-file .llm.env \
--shm-size=1g \
- unclecode/crawl4ai:0.8.0
+ unclecode/crawl4ai:0.8.5
```
> The server will be available at `http://localhost:11235`. Visit `/playground` to access the interactive testing interface.
@@ -184,7 +184,7 @@ The `docker-compose.yml` file in the project root provides a simplified approach
```bash
# Pulls and runs the release candidate from Docker Hub
# Automatically selects the correct architecture
- IMAGE=unclecode/crawl4ai:0.8.0 docker compose up -d
+ IMAGE=unclecode/crawl4ai:0.8.5 docker compose up -d
```
* **Build and Run Locally:**
diff --git a/deploy/docker/api.py b/deploy/docker/api.py
index 81cd312ab..1ecc9e0b4 100644
--- a/deploy/docker/api.py
+++ b/deploy/docker/api.py
@@ -44,7 +44,8 @@
get_llm_api_key,
validate_llm_provider,
get_llm_temperature,
- get_llm_base_url
+ get_llm_base_url,
+ get_redis_task_ttl
)
from webhook import WebhookDeliveryService
@@ -61,13 +62,32 @@ def _get_memory_mb():
return None
+async def hset_with_ttl(redis, key: str, mapping: dict, config: dict):
+ """Set Redis hash with automatic TTL expiry.
+
+ Args:
+ redis: Redis client instance
+ key: Redis key (e.g., "task:abc123")
+ mapping: Hash field-value mapping
+ config: Application config containing redis.task_ttl_seconds
+ """
+ await redis.hset(key, mapping=mapping)
+ ttl = get_redis_task_ttl(config)
+ if ttl > 0:
+ await redis.expire(key, ttl)
+
+
async def handle_llm_qa(
url: str,
query: str,
- config: dict
+ config: dict,
+ provider: Optional[str] = None,
+ temperature: Optional[float] = None,
+ base_url: Optional[str] = None,
) -> str:
"""Process QA using LLM with crawled content as context."""
- from crawler_pool import get_crawler
+ from crawler_pool import get_crawler, release_crawler
+ crawler = None
try:
if not url.startswith(('http://', 'https://')) and not url.startswith(("raw:", "raw://")):
url = 'https://' + url
@@ -101,14 +121,13 @@ async def handle_llm_qa(
Answer:"""
- # api_token=os.environ.get(config["llm"].get("api_key_env", ""))
-
+ resolved_provider = provider or config["llm"]["provider"]
response = perform_completion_with_backoff(
- provider=config["llm"]["provider"],
+ provider=resolved_provider,
prompt_with_variables=prompt,
- api_token=get_llm_api_key(config), # Returns None to let litellm handle it
- temperature=get_llm_temperature(config),
- base_url=get_llm_base_url(config),
+ api_token=get_llm_api_key(config, resolved_provider),
+ temperature=temperature or get_llm_temperature(config, resolved_provider),
+ base_url=base_url or get_llm_base_url(config, resolved_provider),
base_delay=config["llm"].get("backoff_base_delay", 2),
max_attempts=config["llm"].get("backoff_max_attempts", 3),
exponential_factor=config["llm"].get("backoff_exponential_factor", 2)
@@ -121,6 +140,9 @@ async def handle_llm_qa(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=str(e)
)
+ finally:
+ if crawler:
+ await release_crawler(crawler)
async def process_llm_extraction(
redis: aioredis.Redis,
@@ -143,10 +165,10 @@ async def process_llm_extraction(
# Validate provider
is_valid, error_msg = validate_llm_provider(config, provider)
if not is_valid:
- await redis.hset(f"task:{task_id}", mapping={
+ await hset_with_ttl(redis, f"task:{task_id}", {
"status": TaskStatus.FAILED,
"error": error_msg
- })
+ }, config)
# Send webhook notification on failure
await webhook_service.notify_job_completion(
@@ -183,10 +205,10 @@ async def process_llm_extraction(
)
if not result.success:
- await redis.hset(f"task:{task_id}", mapping={
+ await hset_with_ttl(redis, f"task:{task_id}", {
"status": TaskStatus.FAILED,
"error": result.error_message
- })
+ }, config)
# Send webhook notification on failure
await webhook_service.notify_job_completion(
@@ -206,10 +228,10 @@ async def process_llm_extraction(
result_data = {"extracted_content": content}
- await redis.hset(f"task:{task_id}", mapping={
+ await hset_with_ttl(redis, f"task:{task_id}", {
"status": TaskStatus.COMPLETED,
"result": json.dumps(content)
- })
+ }, config)
# Send webhook notification on successful completion
await webhook_service.notify_job_completion(
@@ -223,10 +245,10 @@ async def process_llm_extraction(
except Exception as e:
logger.error(f"LLM extraction error: {str(e)}", exc_info=True)
- await redis.hset(f"task:{task_id}", mapping={
+ await hset_with_ttl(redis, f"task:{task_id}", {
"status": TaskStatus.FAILED,
"error": str(e)
- })
+ }, config)
# Send webhook notification on failure
await webhook_service.notify_job_completion(
@@ -249,6 +271,7 @@ async def handle_markdown_request(
base_url: Optional[str] = None
) -> str:
"""Handle markdown generation requests."""
+ crawler = None
try:
# Validate provider if using LLM filter
if filter_type == FilterType.LLM:
@@ -282,7 +305,7 @@ async def handle_markdown_request(
cache_mode = CacheMode.ENABLED if cache == "1" else CacheMode.WRITE_ONLY
- from crawler_pool import get_crawler
+ from crawler_pool import get_crawler, release_crawler
from utils import load_config as _load_config
_cfg = _load_config()
browser_cfg = BrowserConfig(
@@ -315,6 +338,9 @@ async def handle_markdown_request(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=str(e)
)
+ finally:
+ if crawler:
+ await release_crawler(crawler)
async def handle_llm_request(
redis: aioredis.Redis,
@@ -430,7 +456,7 @@ async def create_new_task(
if webhook_config:
task_data["webhook_config"] = json.dumps(webhook_config)
- await redis.hset(f"task:{task_id}", mapping=task_data)
+ await hset_with_ttl(redis, f"task:{task_id}", task_data, config)
background_tasks.add_task(
process_llm_extraction,
@@ -481,6 +507,7 @@ async def stream_results(crawler: AsyncWebCrawler, results_gen: AsyncGenerator)
"""Stream results with heartbeats and completion markers."""
import json
from utils import datetime_handler
+ from crawler_pool import release_crawler
try:
async for result in results_gen:
@@ -507,11 +534,8 @@ async def stream_results(crawler: AsyncWebCrawler, results_gen: AsyncGenerator)
except asyncio.CancelledError:
logger.warning("Client disconnected during streaming")
finally:
- # try:
- # await crawler.close()
- # except Exception as e:
- # logger.error(f"Crawler cleanup error: {e}")
- pass
+ if crawler:
+ await release_crawler(crawler)
async def handle_crawl_request(
urls: List[str],
@@ -523,6 +547,7 @@ async def handle_crawl_request(
"""Handle non-streaming crawl requests with optional hooks."""
# Track request start
request_id = f"req_{uuid4().hex[:8]}"
+ crawler = None
try:
from monitor import get_monitor
await get_monitor().track_request_start(
@@ -549,11 +574,8 @@ async def handle_crawl_request(
) if config["crawler"]["rate_limiter"]["enabled"] else None
)
- from crawler_pool import get_crawler
+ from crawler_pool import get_crawler, release_crawler
crawler = await get_crawler(browser_config)
-
- # crawler: AsyncWebCrawler = AsyncWebCrawler(config=browser_config)
- # await crawler.start()
# Attach hooks if provided
hooks_status = {}
@@ -589,8 +611,6 @@ async def handle_crawl_request(
if not isinstance(results, list):
results = [results]
- # await crawler.close()
-
end_mem_mb = _get_memory_mb() # <--- Get memory after
end_time = time.time()
@@ -689,13 +709,6 @@ async def handle_crawl_request(
except:
pass
- if 'crawler' in locals() and crawler.ready: # Check if crawler was initialized and started
- # try:
- # await crawler.close()
- # except Exception as close_e:
- # logger.error(f"Error closing crawler during exception handling: {close_e}")
- logger.error(f"Error closing crawler during exception handling: {str(e)}")
-
# Measure memory even on error if possible
end_mem_mb_error = _get_memory_mb()
if start_mem_mb is not None and end_mem_mb_error is not None:
@@ -709,6 +722,9 @@ async def handle_crawl_request(
"server_peak_memory_mb": max(peak_mem_mb if peak_mem_mb else 0, end_mem_mb_error or 0)
})
)
+ finally:
+ if crawler:
+ await release_crawler(crawler)
async def handle_stream_crawl_request(
urls: List[str],
@@ -719,6 +735,7 @@ async def handle_stream_crawl_request(
) -> Tuple[AsyncWebCrawler, AsyncGenerator, Optional[Dict]]:
"""Handle streaming crawl requests with optional hooks."""
hooks_info = None
+ crawler = None
try:
browser_config = BrowserConfig.load(browser_config)
# browser_config.verbose = True # Set to False or remove for production stress testing
@@ -727,19 +744,19 @@ async def handle_stream_crawl_request(
crawler_config.scraping_strategy = LXMLWebScrapingStrategy()
crawler_config.stream = True
- dispatcher = MemoryAdaptiveDispatcher(
- memory_threshold_percent=config["crawler"]["memory_threshold_percent"],
- rate_limiter=RateLimiter(
- base_delay=tuple(config["crawler"]["rate_limiter"]["base_delay"])
+ # Deep crawl streaming supports exactly one start URL
+ if crawler_config.deep_crawl_strategy is not None and len(urls) != 1:
+ raise HTTPException(
+ status_code=status.HTTP_400_BAD_REQUEST,
+ detail=(
+ "Deep crawling with stream currently supports exactly one URL per request. "
+ f"Received {len(urls)} URLs."
+ ),
)
- )
- from crawler_pool import get_crawler
+ from crawler_pool import get_crawler, release_crawler
crawler = await get_crawler(browser_config)
- # crawler = AsyncWebCrawler(config=browser_config)
- # await crawler.start()
-
# Attach hooks if provided
if hooks_config:
from hook_manager import attach_user_hooks_to_crawler, UserHookManager
@@ -754,22 +771,34 @@ async def handle_stream_crawl_request(
# Include hook manager in hooks_info for proper tracking
hooks_info = {'status': hooks_status, 'manager': hook_manager}
- results_gen = await crawler.arun_many(
- urls=urls,
- config=crawler_config,
- dispatcher=dispatcher
- )
+ # Deep crawl with single URL: use arun() which returns an async generator
+ # mirroring the Python library's streaming behavior
+ if crawler_config.deep_crawl_strategy is not None and len(urls) == 1:
+ results_gen = await crawler.arun(
+ urls[0],
+ config=crawler_config,
+ )
+ else:
+ # Default multi-URL streaming via arun_many
+ dispatcher = MemoryAdaptiveDispatcher(
+ memory_threshold_percent=config["crawler"]["memory_threshold_percent"],
+ rate_limiter=RateLimiter(
+ base_delay=tuple(config["crawler"]["rate_limiter"]["base_delay"])
+ )
+ )
+ results_gen = await crawler.arun_many(
+ urls=urls,
+ config=crawler_config,
+ dispatcher=dispatcher
+ )
return crawler, results_gen, hooks_info
except Exception as e:
- # Make sure to close crawler if started during an error here
- if 'crawler' in locals() and crawler.ready:
- # try:
- # await crawler.close()
- # except Exception as close_e:
- # logger.error(f"Error closing crawler during stream setup exception: {close_e}")
- logger.error(f"Error closing crawler during stream setup exception: {str(e)}")
+ # Release crawler on setup error (for successful streams,
+ # release happens in stream_results finally block)
+ if crawler:
+ await release_crawler(crawler)
logger.error(f"Stream crawl error: {str(e)}", exc_info=True)
# Raising HTTPException here will prevent streaming response
raise HTTPException(
@@ -806,7 +835,7 @@ async def handle_crawl_job(
if webhook_config:
task_data["webhook_config"] = json.dumps(webhook_config)
- await redis.hset(f"task:{task_id}", mapping=task_data)
+ await hset_with_ttl(redis, f"task:{task_id}", task_data, config)
# Initialize webhook service
webhook_service = WebhookDeliveryService(config)
@@ -819,10 +848,10 @@ async def _runner():
crawler_config=crawler_config,
config=config,
)
- await redis.hset(f"task:{task_id}", mapping={
+ await hset_with_ttl(redis, f"task:{task_id}", {
"status": TaskStatus.COMPLETED,
"result": json.dumps(result),
- })
+ }, config)
# Send webhook notification on successful completion
await webhook_service.notify_job_completion(
@@ -836,10 +865,10 @@ async def _runner():
await asyncio.sleep(5) # Give Redis time to process the update
except Exception as exc:
- await redis.hset(f"task:{task_id}", mapping={
+ await hset_with_ttl(redis, f"task:{task_id}", {
"status": TaskStatus.FAILED,
"error": str(exc),
- })
+ }, config)
# Send webhook notification on failure
await webhook_service.notify_job_completion(
diff --git a/deploy/docker/auth.py b/deploy/docker/auth.py
index 6fcef3399..ebc4eea1f 100644
--- a/deploy/docker/auth.py
+++ b/deploy/docker/auth.py
@@ -70,4 +70,5 @@ def jwt_required(credentials: HTTPAuthorizationCredentials = Depends(security))
class TokenRequest(BaseModel):
- email: EmailStr
\ No newline at end of file
+ email: EmailStr
+ api_token: Optional[str] = None
\ No newline at end of file
diff --git a/deploy/docker/c4ai-code-context.md b/deploy/docker/c4ai-code-context.md
index c18fbc784..f20ec012f 100644
--- a/deploy/docker/c4ai-code-context.md
+++ b/deploy/docker/c4ai-code-context.md
@@ -7699,7 +7699,7 @@ class BrowserManager:
"accept_downloads": self.config.accept_downloads,
"storage_state": self.config.storage_state,
"ignore_https_errors": self.config.ignore_https_errors,
- "device_scale_factor": 1.0,
+ "device_scale_factor": self.config.device_scale_factor,
"java_script_enabled": self.config.java_script_enabled,
}
diff --git a/deploy/docker/c4ai-doc-context.md b/deploy/docker/c4ai-doc-context.md
index abfd36374..0120e1b67 100644
--- a/deploy/docker/c4ai-doc-context.md
+++ b/deploy/docker/c4ai-doc-context.md
@@ -8589,7 +8589,7 @@ Real sites often have **nested** or repeated dataβlike categories containing p
We have a **sample e-commerce** HTML file on GitHub (example):
```
-https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html
+https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html
```
This snippet includes categories, products, features, reviews, and related items. Letβs see how to define a schema that fully captures that structure **without LLM**.
@@ -8721,7 +8721,7 @@ async def extract_ecommerce_data():
async with AsyncWebCrawler(verbose=True) as crawler:
result = await crawler.arun(
- url="https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html",
+ url="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html",
extraction_strategy=strategy,
config=config
)
diff --git a/deploy/docker/config.yml b/deploy/docker/config.yml
index db3193a69..3383c1254 100644
--- a/deploy/docker/config.yml
+++ b/deploy/docker/config.yml
@@ -14,20 +14,21 @@ llm:
# api_key: sk-... # If you pass the API key directly (not recommended)
# Redis Configuration
+# Set task_ttl_seconds to automatically expire task data in Redis.
+# This prevents unbounded memory growth from accumulated task results.
+# Default: 3600 (1 hour). Set to 0 to disable TTL (not recommended).
+# Can be overridden with REDIS_TASK_TTL environment variable.
redis:
host: "localhost"
port: 6379
db: 0
password: ""
+ task_ttl_seconds: 3600 # TTL for task data (1 hour default)
ssl: False
ssl_cert_reqs: None
ssl_ca_certs: None
ssl_certfile: None
ssl_keyfile: None
- ssl_cert_reqs: None
- ssl_ca_certs: None
- ssl_certfile: None
- ssl_keyfile: None
# Rate Limiting Configuration
rate_limiting:
@@ -44,6 +45,7 @@ rate_limiting:
security:
enabled: false
jwt_enabled: false
+ api_token: "" # When set, /token endpoint requires this secret to issue JWTs
https_redirect: false
trusted_hosts: ["*"]
headers:
diff --git a/deploy/docker/crawler_pool.py b/deploy/docker/crawler_pool.py
index 509cbba92..516d9562a 100644
--- a/deploy/docker/crawler_pool.py
+++ b/deploy/docker/crawler_pool.py
@@ -22,6 +22,27 @@
BASE_IDLE_TTL = CONFIG.get("crawler", {}).get("pool", {}).get("idle_ttl_sec", 300)
DEFAULT_CONFIG_SIG = None # Cached sig for default config
+
+def get_pool_snapshot() -> dict:
+ """Return a point-in-time snapshot of pool state for monitoring.
+
+ This is intentionally lock-free. Under CPython's GIL, reading
+ ``len(dict)``, ``dict.copy()``, and ``x is not None`` are atomic
+ operations, so the monitor can safely call this without contending
+ on the pool LOCK that is held during slow browser start/close ops.
+ The worst case is a slightly stale count, which is acceptable for
+ dashboard display purposes.
+ """
+ return {
+ "permanent": PERMANENT,
+ "permanent_sig": DEFAULT_CONFIG_SIG,
+ "hot_pool": HOT_POOL.copy(),
+ "cold_pool": COLD_POOL.copy(),
+ "last_used": LAST_USED.copy(),
+ "usage_count": USAGE_COUNT.copy(),
+ }
+
+
def _sig(cfg: BrowserConfig) -> str:
"""Generate config signature."""
payload = json.dumps(cfg.to_dict(), sort_keys=True, separators=(",",":"))
@@ -39,6 +60,9 @@ async def get_crawler(cfg: BrowserConfig) -> AsyncWebCrawler:
if PERMANENT and _is_default_config(sig):
LAST_USED[sig] = time.time()
USAGE_COUNT[sig] = USAGE_COUNT.get(sig, 0) + 1
+ if not hasattr(PERMANENT, 'active_requests'):
+ PERMANENT.active_requests = 0
+ PERMANENT.active_requests += 1
logger.info("π₯ Using permanent browser")
return PERMANENT
@@ -46,13 +70,21 @@ async def get_crawler(cfg: BrowserConfig) -> AsyncWebCrawler:
if sig in HOT_POOL:
LAST_USED[sig] = time.time()
USAGE_COUNT[sig] = USAGE_COUNT.get(sig, 0) + 1
- logger.info(f"β¨οΈ Using hot pool browser (sig={sig[:8]})")
- return HOT_POOL[sig]
+ crawler = HOT_POOL[sig]
+ if not hasattr(crawler, 'active_requests'):
+ crawler.active_requests = 0
+ crawler.active_requests += 1
+ logger.info(f"β¨οΈ Using hot pool browser (sig={sig[:8]}, active={crawler.active_requests})")
+ return crawler
# Check cold pool (promote to hot if used 3+ times)
if sig in COLD_POOL:
LAST_USED[sig] = time.time()
USAGE_COUNT[sig] = USAGE_COUNT.get(sig, 0) + 1
+ crawler = COLD_POOL[sig]
+ if not hasattr(crawler, 'active_requests'):
+ crawler.active_requests = 0
+ crawler.active_requests += 1
if USAGE_COUNT[sig] >= 3:
logger.info(f"β¬οΈ Promoting to hot pool (sig={sig[:8]}, count={USAGE_COUNT[sig]})")
@@ -68,7 +100,7 @@ async def get_crawler(cfg: BrowserConfig) -> AsyncWebCrawler:
return HOT_POOL[sig]
logger.info(f"βοΈ Using cold pool browser (sig={sig[:8]})")
- return COLD_POOL[sig]
+ return crawler
# Memory check before creating new
mem_pct = get_container_memory_percent()
@@ -80,11 +112,23 @@ async def get_crawler(cfg: BrowserConfig) -> AsyncWebCrawler:
logger.info(f"π Creating new browser in cold pool (sig={sig[:8]}, mem={mem_pct:.1f}%)")
crawler = AsyncWebCrawler(config=cfg, thread_safe=False)
await crawler.start()
+ crawler.active_requests = 1
COLD_POOL[sig] = crawler
LAST_USED[sig] = time.time()
USAGE_COUNT[sig] = 1
return crawler
+async def release_crawler(crawler: AsyncWebCrawler):
+ """Decrement active request count for a pooled crawler.
+
+ Call this in a finally block after finishing work with a crawler
+ obtained via get_crawler() so the janitor knows when it's safe
+ to close idle browsers.
+ """
+ async with LOCK:
+ if hasattr(crawler, 'active_requests'):
+ crawler.active_requests = max(0, crawler.active_requests - 1)
+
async def init_permanent(cfg: BrowserConfig):
"""Initialize permanent default browser."""
global PERMANENT, DEFAULT_CONFIG_SIG
@@ -132,10 +176,13 @@ async def janitor():
# Clean cold pool
for sig in list(COLD_POOL.keys()):
if now - LAST_USED.get(sig, now) > cold_ttl:
+ crawler = COLD_POOL[sig]
+ if getattr(crawler, 'active_requests', 0) > 0:
+ continue # still serving requests, skip
idle_time = now - LAST_USED[sig]
logger.info(f"π§Ή Closing cold browser (sig={sig[:8]}, idle={idle_time:.0f}s)")
with suppress(Exception):
- await COLD_POOL[sig].close()
+ await crawler.close()
COLD_POOL.pop(sig, None)
LAST_USED.pop(sig, None)
USAGE_COUNT.pop(sig, None)
@@ -150,10 +197,13 @@ async def janitor():
# Clean hot pool (more conservative)
for sig in list(HOT_POOL.keys()):
if now - LAST_USED.get(sig, now) > hot_ttl:
+ crawler = HOT_POOL[sig]
+ if getattr(crawler, 'active_requests', 0) > 0:
+ continue # still serving requests, skip
idle_time = now - LAST_USED[sig]
logger.info(f"π§Ή Closing hot browser (sig={sig[:8]}, idle={idle_time:.0f}s)")
with suppress(Exception):
- await HOT_POOL[sig].close()
+ await crawler.close()
HOT_POOL.pop(sig, None)
LAST_USED.pop(sig, None)
USAGE_COUNT.pop(sig, None)
diff --git a/deploy/docker/mcp_bridge.py b/deploy/docker/mcp_bridge.py
index c55ed14ce..1e7da8549 100644
--- a/deploy/docker/mcp_bridge.py
+++ b/deploy/docker/mcp_bridge.py
@@ -8,9 +8,8 @@
from fastapi import FastAPI, WebSocket, WebSocketDisconnect, HTTPException
from fastapi.responses import JSONResponse
-from fastapi import Request
-from sse_starlette.sse import EventSourceResponse
from pydantic import BaseModel
+from starlette.routing import Route, Mount
from mcp.server.sse import SseServerTransport
import mcp.types as t
@@ -37,7 +36,7 @@ def deco(fn):
return deco
# ββ HTTPβproxy helper for FastAPI endpoints βββββββββββββββββββββ
-def _make_http_proxy(base_url: str, route):
+def _make_http_proxy(base_url: str, route, *, timeout: float | None = None):
method = list(route.methods - {"HEAD", "OPTIONS"})[0]
async def proxy(**kwargs):
# replace `/items/{id}` style params first
@@ -49,7 +48,7 @@ async def proxy(**kwargs):
kwargs.pop(k)
url = base_url.rstrip("/") + path
- async with httpx.AsyncClient() as client:
+ async with httpx.AsyncClient(timeout=timeout) as client:
try:
r = (
await client.get(url, params=kwargs)
@@ -61,6 +60,8 @@ async def proxy(**kwargs):
except httpx.HTTPStatusError as e:
# surface FastAPI error details instead of plain 500
raise HTTPException(e.response.status_code, e.response.text)
+ except httpx.TimeoutException:
+ raise HTTPException(504, "upstream request timed out")
return proxy
# ββ main entry point ββββββββββββββββββββββββββββββββββββββββββββ
@@ -70,6 +71,7 @@ def attach_mcp(
base: str = "/mcp",
name: str | None = None,
base_url: str, # eg. "http://127.0.0.1:8020"
+ timeout: float | None = None, # httpx timeout in seconds; None = no limit
) -> None:
"""Call once after all routes are declared to expose WS+SSE MCP endpoints."""
server_name = name or app.title or "FastAPI-MCP"
@@ -92,7 +94,7 @@ def attach_mcp(
# if kind == "tool":
# tools[key] = _make_http_proxy(base_url, route)
if kind == "tool":
- proxy = _make_http_proxy(base_url, route)
+ proxy = _make_http_proxy(base_url, route, timeout=timeout)
tools[key] = (proxy, fn)
continue
if kind == "resource":
@@ -218,18 +220,17 @@ async def ws_to_srv():
tg.start_soon(ws_to_srv)
tg.start_soon(srv_to_ws)
- # ββ SSE transport (official) βββββββββββββββββββββββββββββ
+ # ββ SSE transport (raw ASGI β avoids Starlette middleware conflict) ββ
sse = SseServerTransport(f"{base}/messages/")
- @app.get(f"{base}/sse")
- async def _mcp_sse(request: Request):
- async with sse.connect_sse(
- request.scope, request.receive, request._send # starlette ASGI primitives
- ) as (read_stream, write_stream):
+ async def _mcp_sse_handler(scope, receive, send):
+ """Raw ASGI handler for SSE β the MCP SDK calls (scope, receive, send)
+ internally, so wrapping with @app.get() causes an AssertionError (#1594)."""
+ async with sse.connect_sse(scope, receive, send) as (read_stream, write_stream):
await mcp.run(read_stream, write_stream, init_opts)
- # client β server frames are POSTed here
- app.mount(f"{base}/messages", app=sse.handle_post_message)
+ app.routes.append(Route(f"{base}/sse", endpoint=_mcp_sse_handler))
+ app.routes.append(Mount(f"{base}/messages", app=sse.handle_post_message))
# ββ schema endpoint βββββββββββββββββββββββββββββββββββββββ
@app.get(f"{base}/schema")
diff --git a/deploy/docker/monitor.py b/deploy/docker/monitor.py
index 469ec36c2..62a597772 100644
--- a/deploy/docker/monitor.py
+++ b/deploy/docker/monitor.py
@@ -149,14 +149,15 @@ async def update_timeline(self):
recent_reqs = sum(1 for req in self.completed_requests
if now - req.get("end_time", 0) < 5)
- # Browser counts (acquire lock to prevent race conditions)
- from crawler_pool import PERMANENT, HOT_POOL, COLD_POOL, LOCK
- async with LOCK:
- browser_count = {
- "permanent": 1 if PERMANENT else 0,
- "hot": len(HOT_POOL),
- "cold": len(COLD_POOL)
- }
+ # Browser counts β lock-free snapshot to avoid contending on
+ # the pool LOCK held during slow browser start/close (issue #1754)
+ from crawler_pool import get_pool_snapshot
+ snap = get_pool_snapshot()
+ browser_count = {
+ "permanent": 1 if snap["permanent"] else 0,
+ "hot": len(snap["hot_pool"]),
+ "cold": len(snap["cold_pool"]),
+ }
self.memory_timeline.append({"time": now, "value": mem_pct})
self.requests_timeline.append({"time": now, "value": recent_reqs})
@@ -232,17 +233,17 @@ async def get_health_summary(self) -> Dict:
# Network I/O (delta since last call)
net = psutil.net_io_counters()
- # Pool status (acquire lock to prevent race conditions)
- from crawler_pool import PERMANENT, HOT_POOL, COLD_POOL, LOCK
- async with LOCK:
- # TODO: Track actual browser process memory instead of estimates
- # These are conservative estimates based on typical Chromium usage
- permanent_mem = 270 if PERMANENT else 0 # Estimate: ~270MB for permanent browser
- hot_mem = len(HOT_POOL) * 180 # Estimate: ~180MB per hot pool browser
- cold_mem = len(COLD_POOL) * 180 # Estimate: ~180MB per cold pool browser
- permanent_active = PERMANENT is not None
- hot_count = len(HOT_POOL)
- cold_count = len(COLD_POOL)
+ # Pool status β lock-free snapshot (issue #1754)
+ from crawler_pool import get_pool_snapshot
+ snap = get_pool_snapshot()
+ # TODO: Track actual browser process memory instead of estimates
+ # These are conservative estimates based on typical Chromium usage
+ permanent_mem = 270 if snap["permanent"] else 0 # Estimate: ~270MB for permanent browser
+ hot_mem = len(snap["hot_pool"]) * 180 # Estimate: ~180MB per hot pool browser
+ cold_mem = len(snap["cold_pool"]) * 180 # Estimate: ~180MB per cold pool browser
+ permanent_active = snap["permanent"] is not None
+ hot_count = len(snap["hot_pool"])
+ cold_count = len(snap["cold_pool"])
return {
"container": {
@@ -287,45 +288,52 @@ def get_completed_requests(self, limit: int = 50, filter_status: str = "all") ->
async def get_browser_list(self) -> List[Dict]:
"""Get detailed browser pool information."""
- from crawler_pool import PERMANENT, HOT_POOL, COLD_POOL, LAST_USED, USAGE_COUNT, DEFAULT_CONFIG_SIG, LOCK
+ from crawler_pool import get_pool_snapshot
browsers = []
now = time.time()
- # Acquire lock to prevent race conditions during iteration
- async with LOCK:
- if PERMANENT:
- browsers.append({
- "type": "permanent",
- "sig": DEFAULT_CONFIG_SIG[:8] if DEFAULT_CONFIG_SIG else "unknown",
- "age_seconds": int(now - self.start_time),
- "last_used_seconds": int(now - LAST_USED.get(DEFAULT_CONFIG_SIG, now)),
- "memory_mb": 270,
- "hits": USAGE_COUNT.get(DEFAULT_CONFIG_SIG, 0),
- "killable": False
- })
-
- for sig, crawler in HOT_POOL.items():
- browsers.append({
- "type": "hot",
- "sig": sig[:8],
- "age_seconds": int(now - self.start_time), # Approximation
- "last_used_seconds": int(now - LAST_USED.get(sig, now)),
- "memory_mb": 180, # Estimate
- "hits": USAGE_COUNT.get(sig, 0),
- "killable": True
- })
-
- for sig, crawler in COLD_POOL.items():
- browsers.append({
- "type": "cold",
- "sig": sig[:8],
- "age_seconds": int(now - self.start_time),
- "last_used_seconds": int(now - LAST_USED.get(sig, now)),
- "memory_mb": 180,
- "hits": USAGE_COUNT.get(sig, 0),
- "killable": True
- })
+ # Lock-free snapshot β iterates over copies (issue #1754)
+ snap = get_pool_snapshot()
+ permanent = snap["permanent"]
+ permanent_sig = snap["permanent_sig"]
+ hot_pool = snap["hot_pool"]
+ cold_pool = snap["cold_pool"]
+ last_used = snap["last_used"]
+ usage_count = snap["usage_count"]
+
+ if permanent:
+ browsers.append({
+ "type": "permanent",
+ "sig": permanent_sig[:8] if permanent_sig else "unknown",
+ "age_seconds": int(now - self.start_time),
+ "last_used_seconds": int(now - last_used.get(permanent_sig, now)),
+ "memory_mb": 270,
+ "hits": usage_count.get(permanent_sig, 0),
+ "killable": False
+ })
+
+ for sig, crawler in hot_pool.items():
+ browsers.append({
+ "type": "hot",
+ "sig": sig[:8],
+ "age_seconds": int(now - self.start_time), # Approximation
+ "last_used_seconds": int(now - last_used.get(sig, now)),
+ "memory_mb": 180, # Estimate
+ "hits": usage_count.get(sig, 0),
+ "killable": True
+ })
+
+ for sig, crawler in cold_pool.items():
+ browsers.append({
+ "type": "cold",
+ "sig": sig[:8],
+ "age_seconds": int(now - self.start_time),
+ "last_used_seconds": int(now - last_used.get(sig, now)),
+ "memory_mb": 180,
+ "hits": usage_count.get(sig, 0),
+ "killable": True
+ })
return browsers
diff --git a/deploy/docker/schemas.py b/deploy/docker/schemas.py
index 21d47fc44..ef75ce8bc 100644
--- a/deploy/docker/schemas.py
+++ b/deploy/docker/schemas.py
@@ -73,6 +73,7 @@ class HTMLRequest(BaseModel):
class ScreenshotRequest(BaseModel):
url: str
screenshot_wait_for: Optional[float] = 2
+ wait_for_images: Optional[bool] = False
output_path: Optional[str] = None
class PDFRequest(BaseModel):
diff --git a/deploy/docker/server.py b/deploy/docker/server.py
index 7ae1adb8b..7b3a8d964 100644
--- a/deploy/docker/server.py
+++ b/deploy/docker/server.py
@@ -7,8 +7,9 @@
"""
# ββ stdlib & 3rdβparty imports βββββββββββββββββββββββββββββββ
-from crawler_pool import get_crawler, close_all, janitor
+from crawler_pool import get_crawler, release_crawler, close_all, janitor
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
+from crawl4ai.__version__ import __version__
from auth import create_access_token, get_token_dependency, TokenRequest
from pydantic import BaseModel
from typing import Optional, List, Dict
@@ -73,7 +74,7 @@
config = load_config()
setup_logging(config)
-__version__ = "0.5.1-d1"
+# Version is imported from crawl4ai package to ensure it stays in sync
# ββ global page semaphore (hard cap) βββββββββββββββββββββββββ
MAX_PAGES = config["crawler"]["pool"].get("max_pages", 30)
@@ -204,7 +205,18 @@ async def root():
return RedirectResponse("/playground")
# βββββββββββββββββββ infra / middleware βββββββββββββββββββββ
-redis = aioredis.from_url(config["redis"].get("uri", "redis://localhost"))
+def _build_redis_url(config: dict) -> str:
+ """Build Redis URL from config fields and environment variables."""
+ rc = config.get("redis", {})
+ host = os.environ.get("REDIS_HOST", rc.get("host", "localhost"))
+ port = os.environ.get("REDIS_PORT", rc.get("port", 6379))
+ password = os.environ.get("REDIS_PASSWORD", rc.get("password", ""))
+ db = rc.get("db", 0)
+ scheme = "rediss" if rc.get("ssl", False) else "redis"
+ auth = f":{password}@" if password else ""
+ return f"{scheme}://{auth}{host}:{port}/{db}"
+
+redis = aioredis.from_url(_build_redis_url(config))
limiter = Limiter(
key_func=get_remote_address,
@@ -303,6 +315,9 @@ def _safe_eval_config(expr: str) -> dict:
# ββββββββββββββββββββββββ Endpoints ββββββββββββββββββββββββββ
@app.post("/token")
async def get_token(req: TokenRequest):
+ expected_token = config.get("security", {}).get("api_token", "")
+ if expected_token and req.api_token != expected_token:
+ raise HTTPException(401, "Invalid or missing api_token")
if not verify_email_domain(req.email):
raise HTTPException(400, "Invalid email domain")
token = create_access_token({"sub": req.email})
@@ -325,6 +340,17 @@ async def get_markdown(
body: MarkdownRequest,
_td: Dict = Depends(token_dep),
):
+ """
+ Convert a web page into Markdown format.
+
+ Supports multiple extraction modes:
+ - fit (default): Readability-based extraction for clean content
+ - raw: Direct DOM to Markdown conversion
+ - bm25: BM25 relevance ranking with optional query
+ - llm: LLM-based summarization with optional query
+
+ Use this tool when you need clean, readable text from web pages.
+ """
if not body.url.startswith(("http://", "https://")) and not body.url.startswith(("raw:", "raw://")):
raise HTTPException(
400, "Invalid URL format. Must start with http://, https://, or for raw HTML (raw:, raw://)")
@@ -355,8 +381,8 @@ async def generate_html(
Use when you need sanitized HTML structures for building schemas or further processing.
"""
validate_url_scheme(body.url, allow_raw=True)
- from crawler_pool import get_crawler
cfg = CrawlerRunConfig()
+ crawler = None
try:
crawler = await get_crawler(get_default_browser_config())
results = await crawler.arun(url=body.url, config=cfg)
@@ -369,6 +395,9 @@ async def generate_html(
return JSONResponse({"html": processed_html, "url": body.url, "success": True})
except Exception as e:
raise HTTPException(500, detail=str(e))
+ finally:
+ if crawler:
+ await release_crawler(crawler)
# Screenshot endpoint
@@ -387,9 +416,9 @@ async def generate_screenshot(
Then in result instead of the screenshot you will get a path to the saved file.
"""
validate_url_scheme(body.url)
- from crawler_pool import get_crawler
+ crawler = None
try:
- cfg = CrawlerRunConfig(screenshot=True, screenshot_wait_for=body.screenshot_wait_for)
+ cfg = CrawlerRunConfig(screenshot=True, screenshot_wait_for=body.screenshot_wait_for, wait_for_images=body.wait_for_images)
crawler = await get_crawler(get_default_browser_config())
results = await crawler.arun(url=body.url, config=cfg)
if not results[0].success:
@@ -404,6 +433,9 @@ async def generate_screenshot(
return {"success": True, "screenshot": screenshot_data}
except Exception as e:
raise HTTPException(500, detail=str(e))
+ finally:
+ if crawler:
+ await release_crawler(crawler)
# PDF endpoint
@@ -422,7 +454,7 @@ async def generate_pdf(
Then in result instead of the PDF you will get a path to the saved file.
"""
validate_url_scheme(body.url)
- from crawler_pool import get_crawler
+ crawler = None
try:
cfg = CrawlerRunConfig(pdf=True)
crawler = await get_crawler(get_default_browser_config())
@@ -439,6 +471,9 @@ async def generate_pdf(
return {"success": True, "pdf": base64.b64encode(pdf_data).decode()}
except Exception as e:
raise HTTPException(500, detail=str(e))
+ finally:
+ if crawler:
+ await release_crawler(crawler)
@app.post("/execute_js")
@@ -495,7 +530,7 @@ class MarkdownGenerationResult(BaseModel):
"""
validate_url_scheme(body.url)
- from crawler_pool import get_crawler
+ crawler = None
try:
cfg = CrawlerRunConfig(js_code=body.scripts)
crawler = await get_crawler(get_default_browser_config())
@@ -506,6 +541,9 @@ class MarkdownGenerationResult(BaseModel):
return JSONResponse(data)
except Exception as e:
raise HTTPException(500, detail=str(e))
+ finally:
+ if crawler:
+ await release_crawler(crawler)
@app.get("/llm/{url:path}")
@@ -513,13 +551,16 @@ async def llm_endpoint(
request: Request,
url: str = Path(...),
q: str = Query(...),
+ provider: Optional[str] = Query(None, description="LLM provider override, e.g. 'openai/gpt-4o-mini'"),
+ temperature: Optional[float] = Query(None, description="LLM temperature override"),
+ base_url: Optional[str] = Query(None, description="LLM API base URL override"),
_td: Dict = Depends(token_dep),
):
if not q:
raise HTTPException(400, "Query parameter 'q' is required")
if not url.startswith(("http://", "https://")) and not url.startswith(("raw:", "raw://")):
url = "https://" + url
- answer = await handle_llm_qa(url, q, config)
+ answer = await handle_llm_qa(url, q, config, provider=provider, temperature=temperature, base_url=base_url)
return JSONResponse({"answer": answer})
diff --git a/deploy/docker/static/playground/index.html b/deploy/docker/static/playground/index.html
index 510a6620b..e8c55037a 100644
--- a/deploy/docker/static/playground/index.html
+++ b/deploy/docker/static/playground/index.html
@@ -128,6 +128,10 @@
opacity: 1;
}
+ #adv-editor .CodeMirror {
+ height: 100% !important;
+ }
+
/* copid text highlighted */
.highlighted {
background-color: rgba(78, 255, 255, 0.2) !important;
@@ -267,7 +271,7 @@ <h2 class="font-medium">Request Builder</h2>
</div>
<!-- CodeMirror host -->
- <div id="adv-editor" class="mt-2 border border-border rounded overflow-hidden h-40"></div>
+ <div id="adv-editor" class="mt-2 border border-border rounded overflow-auto" style="height: 160px; min-height: 160px; max-height: 500px; resize: vertical;"></div>
</details>
<div class="flex space-x-2">
diff --git a/deploy/docker/tests/test_security_fixes.py b/deploy/docker/tests/test_security_fixes.py
index 9321d111e..8ba8de182 100644
--- a/deploy/docker/tests/test_security_fixes.py
+++ b/deploy/docker/tests/test_security_fixes.py
@@ -160,6 +160,170 @@ def test_hooks_disabled_when_false(self):
os.environ.pop("CRAWL4AI_HOOKS_ENABLED", None)
+class TestComputedFieldSafety(unittest.TestCase):
+ """Test that computed field expressions block dangerous operations.
+
+ Mirrors the AST-based _safe_eval_expression() logic from extraction_strategy.py
+ to test without importing heavy crawl4ai dependencies.
+ """
+
+ def setUp(self):
+ """Set up the safe eval function (local copy of the logic)."""
+ import ast
+
+ SAFE_BUILTINS = {
+ "str": str, "int": int, "float": float, "bool": bool,
+ "len": len, "round": round, "abs": abs, "min": min, "max": max,
+ "sum": sum, "sorted": sorted, "reversed": reversed,
+ "list": list, "dict": dict, "tuple": tuple, "set": set,
+ "enumerate": enumerate, "zip": zip, "map": map, "filter": filter,
+ "any": any, "all": all, "range": range,
+ "True": True, "False": False, "None": None,
+ "isinstance": isinstance, "type": type,
+ }
+
+ def safe_eval(expression, local_vars):
+ tree = ast.parse(expression, mode="eval")
+ for node in ast.walk(tree):
+ if isinstance(node, (ast.Import, ast.ImportFrom)):
+ raise ValueError("Import statements are not allowed")
+ if isinstance(node, ast.Attribute) and node.attr.startswith("_"):
+ raise ValueError(f"Access to '{node.attr}' is not allowed")
+ if isinstance(node, ast.Call):
+ func = node.func
+ if isinstance(func, ast.Name) and func.id.startswith("_"):
+ raise ValueError(f"Calling '{func.id}' is not allowed")
+ if isinstance(func, ast.Attribute) and func.attr.startswith("_"):
+ raise ValueError(f"Calling '{func.attr}' is not allowed")
+ safe_globals = {"__builtins__": SAFE_BUILTINS}
+ return eval(compile(tree, "<expression>", "eval"), safe_globals, local_vars)
+
+ self.safe_eval = safe_eval
+
+ # === SECURITY TESTS: These expressions must be BLOCKED ===
+
+ def test_import_blocked(self):
+ """__import__('os') must be blocked."""
+ with self.assertRaises(ValueError):
+ self.safe_eval("__import__('os').system('id')", {})
+
+ def test_dunder_attribute_blocked(self):
+ """Access to __class__, __globals__, etc. must be blocked."""
+ with self.assertRaises(ValueError):
+ self.safe_eval("''.__class__.__bases__", {})
+
+ def test_dunder_method_call_blocked(self):
+ """Calls to dunder methods must be blocked."""
+ with self.assertRaises(ValueError):
+ self.safe_eval("x.__getattribute__('y')", {"x": {}})
+
+ def test_os_popen_via_import_blocked(self):
+ """The exact POC from the vulnerability report must be blocked."""
+ with self.assertRaises(ValueError):
+ self.safe_eval('__import__("os").popen("id").read()', {})
+
+ # === FUNCTIONALITY TESTS: These expressions must WORK ===
+
+ def test_simple_math(self):
+ """Basic arithmetic on item values must work."""
+ result = self.safe_eval("price * 1.1", {"price": 100})
+ self.assertAlmostEqual(result, 110.0)
+
+ def test_string_method(self):
+ """String methods on item values must work."""
+ result = self.safe_eval("name.upper()", {"name": "hello"})
+ self.assertEqual(result, "HELLO")
+
+ def test_string_concatenation(self):
+ """String concatenation must work."""
+ result = self.safe_eval("first + ' ' + last", {"first": "John", "last": "Doe"})
+ self.assertEqual(result, "John Doe")
+
+ def test_dict_access(self):
+ """Dict-style field access must work."""
+ result = self.safe_eval("a + b", {"a": 10, "b": 20})
+ self.assertEqual(result, 30)
+
+ def test_builtin_functions(self):
+ """Safe builtins like len, str, int must work."""
+ result = self.safe_eval("len(name)", {"name": "hello"})
+ self.assertEqual(result, 5)
+
+ def test_round_function(self):
+ """round() must work for numeric formatting."""
+ result = self.safe_eval("round(price, 2)", {"price": 10.456})
+ self.assertEqual(result, 10.46)
+
+
+class TestDeserializationAllowlist(unittest.TestCase):
+ """Test that the deserialization allowlist blocks non-allowlisted types.
+
+ Tests the allowlist constant directly without importing heavy dependencies.
+ """
+
+ def setUp(self):
+ """Set up the allowlist (local copy of the constant)."""
+ self.allowed_types = {
+ "BrowserConfig", "CrawlerRunConfig", "HTTPCrawlerConfig",
+ "LLMConfig", "ProxyConfig", "GeolocationConfig",
+ "SeedingConfig", "VirtualScrollConfig", "LinkPreviewConfig",
+ "JsonCssExtractionStrategy", "JsonXPathExtractionStrategy",
+ "JsonLxmlExtractionStrategy", "LLMExtractionStrategy",
+ "CosineStrategy", "RegexExtractionStrategy",
+ "DefaultMarkdownGenerator",
+ "PruningContentFilter", "BM25ContentFilter", "LLMContentFilter",
+ "LXMLWebScrapingStrategy",
+ "RegexChunking",
+ "BFSDeepCrawlStrategy", "DFSDeepCrawlStrategy", "BestFirstCrawlingStrategy",
+ "FilterChain", "URLPatternFilter", "DomainFilter",
+ "ContentTypeFilter", "URLFilter", "SEOFilter", "ContentRelevanceFilter",
+ "KeywordRelevanceScorer", "URLScorer", "CompositeScorer",
+ "DomainAuthorityScorer", "FreshnessScorer", "PathDepthScorer",
+ "CacheMode", "MatchMode", "DisplayMode",
+ "MemoryAdaptiveDispatcher", "SemaphoreDispatcher",
+ "DefaultTableExtraction", "NoTableExtraction",
+ "RoundRobinProxyStrategy",
+ }
+
+ # === SECURITY TESTS: Non-allowlisted types must be BLOCKED ===
+
+ def test_arbitrary_class_not_in_allowlist(self):
+ """AsyncWebCrawler must NOT be in the allowlist."""
+ self.assertNotIn("AsyncWebCrawler", self.allowed_types)
+
+ def test_crawler_hub_not_in_allowlist(self):
+ """CrawlerHub must NOT be in the allowlist."""
+ self.assertNotIn("CrawlerHub", self.allowed_types)
+
+ def test_browser_profiler_not_in_allowlist(self):
+ """BrowserProfiler must NOT be in the allowlist."""
+ self.assertNotIn("BrowserProfiler", self.allowed_types)
+
+ def test_docker_client_not_in_allowlist(self):
+ """Crawl4aiDockerClient must NOT be in the allowlist."""
+ self.assertNotIn("Crawl4aiDockerClient", self.allowed_types)
+
+ # === FUNCTIONALITY TESTS: Allowlisted types must be present ===
+
+ def test_allowlist_has_core_config_types(self):
+ """Core config types must be in the allowlist."""
+ required = {"BrowserConfig", "CrawlerRunConfig", "LLMConfig", "ProxyConfig"}
+ self.assertTrue(required.issubset(self.allowed_types))
+
+ def test_allowlist_has_extraction_strategies(self):
+ """Extraction strategy types must be in the allowlist."""
+ required = {
+ "JsonCssExtractionStrategy", "LLMExtractionStrategy",
+ "RegexExtractionStrategy",
+ }
+ self.assertTrue(required.issubset(self.allowed_types))
+
+ def test_allowlist_has_enums(self):
+ """Enum types must be in the allowlist."""
+ required = {"CacheMode", "MatchMode", "DisplayMode"}
+ self.assertTrue(required.issubset(self.allowed_types))
+
+
if __name__ == '__main__':
print("=" * 60)
print("Crawl4AI Security Fixes - Unit Tests")
diff --git a/deploy/docker/utils.py b/deploy/docker/utils.py
index 52f4e11fd..585d89411 100644
--- a/deploy/docker/utils.py
+++ b/deploy/docker/utils.py
@@ -19,11 +19,113 @@ class FilterType(str, Enum):
BM25 = "bm25"
LLM = "llm"
+DEFAULT_CONFIG = {
+ "app": {
+ "title": "Crawl4AI API",
+ "version": "1.0.0",
+ "host": "0.0.0.0",
+ "port": 11235,
+ "reload": False,
+ "workers": 1,
+ "timeout_keep_alive": 300,
+ },
+ "llm": {
+ "provider": "openai/gpt-4o-mini",
+ },
+ "redis": {
+ "host": "localhost",
+ "port": 6379,
+ "db": 0,
+ "password": "",
+ "task_ttl_seconds": 3600,
+ "ssl": False,
+ },
+ "rate_limiting": {
+ "enabled": True,
+ "default_limit": "1000/minute",
+ "trusted_proxies": [],
+ "storage_uri": "memory://",
+ },
+ "security": {
+ "enabled": False,
+ "jwt_enabled": False,
+ "api_token": "",
+ "https_redirect": False,
+ "trusted_hosts": ["*"],
+ "headers": {
+ "x_content_type_options": "nosniff",
+ "x_frame_options": "DENY",
+ "content_security_policy": "default-src 'self'",
+ "strict_transport_security": "max-age=63072000; includeSubDomains",
+ },
+ },
+ "crawler": {
+ "base_config": {"simulate_user": True},
+ "memory_threshold_percent": 95.0,
+ "rate_limiter": {"enabled": True, "base_delay": [1.0, 2.0]},
+ "timeouts": {"stream_init": 30.0, "batch_process": 300.0},
+ "pool": {"max_pages": 40, "idle_ttl_sec": 300},
+ "browser": {
+ "kwargs": {"headless": True, "text_mode": True},
+ "extra_args": [
+ "--no-sandbox",
+ "--disable-dev-shm-usage",
+ "--disable-gpu",
+ "--disable-software-rasterizer",
+ "--disable-web-security",
+ "--allow-insecure-localhost",
+ "--ignore-certificate-errors",
+ ],
+ },
+ },
+ "logging": {
+ "level": "INFO",
+ "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+ },
+ "observability": {
+ "prometheus": {"enabled": True, "endpoint": "/metrics"},
+ "health_check": {"endpoint": "/health"},
+ },
+ "webhooks": {
+ "enabled": True,
+ "default_url": None,
+ "data_in_payload": False,
+ "retry": {
+ "max_attempts": 5,
+ "initial_delay_ms": 1000,
+ "max_delay_ms": 32000,
+ "timeout_ms": 30000,
+ },
+ "headers": {"User-Agent": "Crawl4AI-Webhook/1.0"},
+ },
+}
+
+
+def _deep_merge(base: dict, override: dict) -> dict:
+ """Recursively merge override into base. Override values take precedence."""
+ merged = base.copy()
+ for key, value in override.items():
+ if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):
+ merged[key] = _deep_merge(merged[key], value)
+ else:
+ merged[key] = value
+ return merged
+
+
def load_config() -> Dict:
"""Load and return application configuration with environment variable overrides."""
config_path = Path(__file__).parent / "config.yml"
with open(config_path, "r") as config_file:
- config = yaml.safe_load(config_file)
+ user_config = yaml.safe_load(config_file) or {}
+
+ # Deep-merge user config on top of defaults so missing keys get safe values
+ config = _deep_merge(DEFAULT_CONFIG, user_config)
+
+ for section in DEFAULT_CONFIG:
+ if section not in user_config:
+ logging.warning(
+ f"Config section '{section}' missing from config.yml, using defaults"
+ )
# Override LLM provider from environment if set
llm_provider = os.environ.get("LLM_PROVIDER")
@@ -36,7 +138,16 @@ def load_config() -> Dict:
if llm_api_key and "api_key" not in config["llm"]:
config["llm"]["api_key"] = llm_api_key
logging.info("LLM API key loaded from LLM_API_KEY environment variable")
-
+
+ # Override Redis task TTL from environment if set
+ redis_task_ttl = os.environ.get("REDIS_TASK_TTL")
+ if redis_task_ttl:
+ try:
+ config["redis"]["task_ttl_seconds"] = int(redis_task_ttl)
+ logging.info(f"Redis task TTL overridden from REDIS_TASK_TTL: {redis_task_ttl}s")
+ except ValueError:
+ logging.warning(f"Invalid REDIS_TASK_TTL value: {redis_task_ttl}, using default")
+
return config
def setup_logging(config: Dict) -> None:
@@ -70,6 +181,17 @@ def decode_redis_hash(hash_data: Dict[bytes, bytes]) -> Dict[str, str]:
return {k.decode('utf-8'): v.decode('utf-8') for k, v in hash_data.items()}
+def get_redis_task_ttl(config: Dict) -> int:
+ """Get Redis task TTL in seconds from config.
+
+ Args:
+ config: The application configuration dictionary
+
+ Returns:
+ TTL in seconds (default 3600). Returns 0 if TTL is disabled.
+ """
+ return config.get("redis", {}).get("task_ttl_seconds", 3600)
+
def get_llm_api_key(config: Dict, provider: Optional[str] = None) -> Optional[str]:
"""Get the appropriate API key based on the LLM provider.
diff --git a/docs/apps/iseeyou/llms-full.txt b/docs/apps/iseeyou/llms-full.txt
index 12dad6033..e4b04c49e 100644
--- a/docs/apps/iseeyou/llms-full.txt
+++ b/docs/apps/iseeyou/llms-full.txt
@@ -643,7 +643,7 @@ browser_config = BrowserConfig(
# Advanced browser setup with proxy and persistence
browser_config = BrowserConfig(
headless=False,
- proxy="http://user:pass@proxy:8080",
+ proxy_config="http://user:pass@proxy:8080",
use_persistent_context=True,
user_data_dir="./browser_data",
cookies=[
@@ -2412,7 +2412,7 @@ async def batch_llm_extraction():
if result.success:
contents.append({
"url": url,
- "content": result.fit_markdown[:2000] # Limit content
+ "content": result.markdown.fit_markdown[:2000] # Limit content
})
# Process in batches (reduce LLM calls)
@@ -4719,7 +4719,7 @@ browser_config = BrowserConfig(
headless=True,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@proxy:8080"
+ proxy_config="http://user:pass@proxy:8080"
)
# Get JSON structure
diff --git a/docs/blog/release-v0.8.5.md b/docs/blog/release-v0.8.5.md
new file mode 100644
index 000000000..68fc79691
--- /dev/null
+++ b/docs/blog/release-v0.8.5.md
@@ -0,0 +1,384 @@
+# Crawl4AI v0.8.5: Anti-Bot, Shadow DOM & 60+ Bug Fixes
+
+*March 2026 β’ 10 min read*
+
+---
+
+I'm releasing Crawl4AI v0.8.5βour biggest release since v0.8.0. This update brings automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes from both our team and the community. If you're running crawls at scale or dealing with protected sites, this one's for you.
+
+## What's New at a Glance
+
+- **Anti-Bot Detection & Proxy Escalation**: 3-tier detection with automatic retry, proxy chain, and fallback
+- **Shadow DOM Flattening**: Extract content hidden inside shadow DOM components
+- **Deep Crawl Cancellation**: Stop long crawls gracefully with `cancel()` or `should_cancel` callback
+- **Config Defaults API**: Set once, apply everywhere with `set_defaults()` / `get_defaults()` / `reset_defaults()`
+- **Source/Sibling Selector**: Extract data spanning sibling elements in JSON extraction schemas
+- **Consent Popup Removal**: Auto-dismiss cookie banners from 40+ CMP platforms
+- **Resource Filtering**: Block ads and CSS at the network level with `avoid_ads` / `avoid_css`
+- **Browser Recycling**: Memory-saving mode and automatic browser restart for long sessions
+- **GFM Table Compliance**: Proper `| col1 | col2 |` pipe delimiters in markdown output
+- **60+ Bug Fixes**: Security patches, browser stability, extraction accuracy, and more
+
+---
+
+## New Features
+
+### 1. Anti-Bot Detection, Retry & Fallback
+
+This is the headline feature. Crawl4AI now automatically detects when a page is blocked by anti-bot protection and takes actionβretrying with different proxies or falling back to an alternative fetch method.
+
+The detection uses three tiers:
+- **Tier 1**: Known vendor patterns (Cloudflare, Akamai, DataDome, PerimeterX, etc.)
+- **Tier 2**: Generic block indicators on small pages
+- **Tier 3**: Structural integrity checks (empty shells, script-heavy pages with no content)
+
+```python
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+from crawl4ai.async_configs import ProxyConfig
+
+config = CrawlerRunConfig(
+ # Try direct first, then proxy on bot detection
+ proxy_config=[
+ ProxyConfig.DIRECT,
+ ProxyConfig(server="http://my-proxy:8080"),
+ ],
+ max_retries=2,
+ # Optional: fallback when all proxies fail
+ fallback_fetch_function=my_web_unlocker_function,
+)
+
+async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun("https://protected-site.com", config=config)
+
+ # Check what happened
+ stats = result.crawl_stats
+ print(f"Resolved by: {stats['resolved_by']}") # "direct", "proxy", or "fallback_fetch"
+ print(f"Proxies tried: {len(stats['proxies_used'])}")
+```
+
+The system errs on the side of cautionβfalse positives are cheap (the fallback rescues them), but false negatives mean garbage results. After 5 iterations of real-world testing, it handles everything from Cloudflare challenges to Reddit's 180KB SPA block pages.
+
+### 2. Shadow DOM Flattening
+
+Web components with shadow DOM hide their content from regular DOM traversal. The new `flatten_shadow_dom` option serializes shadow DOM content into the light DOM before extraction.
+
+```python
+config = CrawlerRunConfig(flatten_shadow_dom=True)
+
+async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun("https://some-web-component-site.com", config=config)
+ # Shadow DOM content is now visible in result.html, cleaned_html, and markdown
+```
+
+The implementation patches `attachShadow` to force-open closed shadow roots, recursively resolves `<slot>` projections, and strips only shadow-scoped `<style>` tags. It also reorders the JS execution pipelineβ`js_code` now runs after `wait_for` + `delay_before_return_html` so your scripts operate on the fully-hydrated page. If you need JS to run before waiting, use the new `js_code_before_wait` parameter.
+
+### 3. Deep Crawl Cancellation
+
+All deep crawl strategies (BFS, DFS, BestFirst) now support graceful cancellation:
+
+```python
+from crawl4ai.deep_crawling import DFSDeepCrawlStrategy
+
+pages_found = 0
+
+def should_stop():
+ return pages_found >= 50 # Stop after finding enough pages
+
+async def on_state(state):
+ nonlocal pages_found
+ pages_found = state["pages_crawled"]
+
+strategy = DFSDeepCrawlStrategy(
+ max_depth=3,
+ max_pages=1000,
+ should_cancel=should_stop, # Sync or async callback
+ on_state_change=on_state,
+)
+
+config = CrawlerRunConfig(deep_crawl_strategy=strategy)
+
+async with AsyncWebCrawler() as crawler:
+ results = await crawler.arun("https://example.com", config=config)
+ print(f"Cancelled: {strategy.cancelled}")
+```
+
+You can also call `strategy.cancel()` directly from another thread or coroutine.
+
+### 4. Config Defaults API
+
+Tired of repeating the same parameters? Set defaults once and they apply to every new instance:
+
+```python
+from crawl4ai import BrowserConfig, CrawlerRunConfig
+
+# Set organization-wide defaults
+BrowserConfig.set_defaults(headless=True, text_mode=True)
+CrawlerRunConfig.set_defaults(verbose=False, remove_consent_popups=True)
+
+# All new instances inherit defaults
+bc = BrowserConfig() # headless=True, text_mode=True
+rc = CrawlerRunConfig() # verbose=False, remove_consent_popups=True
+
+# Explicit params always override
+bc2 = BrowserConfig(text_mode=False) # text_mode=False, headless still True
+
+# Inspect and reset
+print(BrowserConfig.get_defaults()) # {"headless": True, "text_mode": True}
+BrowserConfig.reset_defaults() # Back to normal
+```
+
+### 5. Source/Sibling Selector in JSON Extraction
+
+Many sites split a single item's data across sibling elements (think Hacker News, where title and score are in separate `<tr>` rows). The new `"source"` field navigates to a sibling before extracting:
+
+```python
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+schema = {
+ "name": "HackerNewsItems",
+ "baseSelector": "tr.athing",
+ "fields": [
+ {"name": "title", "selector": ".titleline > a", "type": "text"},
+ {"name": "link", "selector": ".titleline > a", "type": "attribute", "attribute": "href"},
+ # Navigate to the NEXT sibling <tr> to get the score
+ {"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
+ {"name": "author", "selector": ".hnuser", "type": "text", "source": "+ tr"},
+ ]
+}
+
+strategy = JsonCssExtractionStrategy(schema=schema)
+```
+
+Works in both `JsonCssExtractionStrategy` and `JsonXPathExtractionStrategy`. Falls back gracefully when siblings don't exist.
+
+### 6. Consent Popup Removal
+
+A single flag auto-dismisses cookie consent banners from 40+ CMP platforms:
+
+```python
+config = CrawlerRunConfig(remove_consent_popups=True)
+```
+
+Covers OneTrust, Cookiebot, Didomi, Quantcast, Sourcepoint, Google FundingChoices, TrustArc, ConsentManager, Osano, Iubenda, Complianz, LiveRamp, CookieYes, Klaro, Termly, and many more.
+
+### 7. Resource Filtering: avoid_ads / avoid_css
+
+Block ad trackers and CSS resources at the network level for faster, leaner crawls:
+
+```python
+config = BrowserConfig(
+ avoid_ads=True, # Blocks doubleclick, google-analytics, etc.
+ avoid_css=True, # Blocks .css, .less, .scss resources
+)
+```
+
+### 8. Browser Recycling & Memory-Saving Mode
+
+For long-running crawl sessions:
+
+```python
+config = BrowserConfig(
+ memory_saving_mode=True, # Aggressive cache/V8 heap flags
+ max_pages_before_recycle=100, # Auto-restart browser after N pages
+)
+```
+
+This prevents memory leaks during sustained crawling. The recycling uses a version-based approach that's safe under concurrent loadβwe fixed three separate deadlock bugs to get this right.
+
+### 9. GFM Table Compliance
+
+Tables in markdown output now have proper GitHub-Flavored Markdown pipe delimiters:
+
+**Before (v0.8.0)**:
+```
+Name | Age | City
+---|---|---
+Alice | 30 | NYC
+```
+
+**After (v0.8.5)**:
+```
+| Name | Age | City |
+| --- | --- | --- |
+| Alice | 30 | NYC |
+```
+
+---
+
+## Minor Features
+
+- **`query_llm_config`**: Separate LLM config for adaptive crawler query expansion (#1682)
+- **`force_viewport_screenshot`**: Screenshot only the viewport, not the full page
+- **`device_scale_factor`**: Configurable screenshot DPI via BrowserConfig (#1463)
+- **`redirected_status_code`**: Now available on CrawlResult (#1435)
+- **`wait_for_images`**: Wait for images to load before taking screenshots (#1792)
+- **`score_threshold`**: Filter low-quality URLs in BestFirstCrawlingStrategy (#1804)
+- **`link_preview_timeout`**: Configurable timeout in AdaptiveConfig (#1793)
+- **`--json-ensure-ascii`**: CLI flag for Unicode preservation in JSON output (#1668)
+- **`type-list` pipeline**: Chained extraction like `["attribute", "regex"]` in JsonCssExtractionStrategy (#1290)
+
+---
+
+## Security Fixes
+
+### Critical: RCE via Deserialization in Docker /crawl Endpoint
+
+**Severity**: CRITICAL
+**Affected**: Docker API deployment (v0.8.0 and earlier)
+
+The `/crawl` endpoint's deserialization logic used `eval()` for certain object types. I removed this entirely and added an allowlist (`ALLOWED_DESERIALIZE_TYPES`) so only known config classes can be instantiated.
+
+### Critical: Redis CVE-2025-49844 (CVSS 10.0)
+
+**Affected**: Docker deployments using Redis
+
+Upgraded Redis to 7.2.7 which patches the Lua use-after-free vulnerability.
+
+### Additional Security
+
+- **XSS prevention**: Use DOMParser instead of innerHTML in iframe processing (#1796)
+- **API token enforcement**: `/token` endpoint now requires `api_token` when configured (#1795)
+- **Stealth improvements**: `sec-ch-ua` synced with User-Agent, WebGL kept alive in stealth mode
+
+---
+
+## Bug Fixes
+
+### Browser & Page Management
+
+- Fix page reuse race condition when `create_isolated_context=False`
+- Fix browser context memory leak β signature shrink + LRU eviction (#943)
+- Fix cascading context crash from duplicate `add_init_script` (#1768)
+- Fix `simulate_user` destroying page content via ArrowDown keypress
+- Fix browser recycling deadlock under sustained concurrent load (#1640)
+- Fix Docker monitor LOCK contention causing pod deadlock (#1754)
+
+### Proxy & Network
+
+- Fix proxy auth `ERR_INVALID_AUTH_CREDENTIALS` (#1281)
+- Fix proxy auth for persistent browser contexts
+- Fix proxy escalation not re-raising on first exception when chain has alternatives
+- Fix fallback fetch: run when all proxies crash, skip re-check, never return None
+
+### Deep Crawling
+
+- Fix `can_process_url()` to receive normalized URL
+- Fix `total_score` not calculated for links that fail head extraction
+- Fix `FilterChain.add_filter` AttributeError on tuple immutability
+- Fix URL Seeder forcing Common Crawl index for sitemaps (#1746)
+- Fix `is_external_url` port comparison (#1783)
+- Prevent AdaptiveCrawler from crawling external domains (#1805)
+
+### Extraction & Content
+
+- Fix `<base>` tag ignored in html2text relative link resolution (#1721)
+- Fix script tag removal losing adjacent text in `cleaned_html` (#1364)
+- Preserve `class` and `id` attributes in `cleaned_html` (#1782)
+- Fix nested brackets/parentheses in LINK_PATTERN regex (#1790)
+- Strip markdown fences in `force_json_response` path for LLM extraction
+- Guard against None LLM content, propagate `finish_reason` (#1788)
+- Fix `agenerate_schema()` JSON parsing for Anthropic models
+- Fix `from_serializable_dict` ignoring plain data dicts with "type" key
+- Fix MediaItem crash on non-numeric width values like "100%" (#1635)
+- Fix BM25ContentFilter returning duplicate chunks (#1213)
+- Fix `css_selector` ignored in LXML scraping for `raw://` URLs (#1484)
+
+### CLI & Docker
+
+- Fix deep-crawl CLI outputting only the first page (#1667)
+- Fix VersionManager ignoring `CRAWL4_AI_BASE_DIRECTORY` env var (#1296)
+- Fix Docker health endpoint to use dynamic version (#1686)
+- Add explicit UTF-8 encoding to CLI file output (#1789)
+- Handle `UnicodeEncodeError` in URL seeder, strip zero-width chars (#1784)
+- Add TTL expiry for Redis task data to prevent memory growth (#1730)
+- Add Windows support for crawler monitor keyboard input (#1794)
+- Fix `scroll_delay` ignored in full-page screenshot scroller
+- Fix MCP SSE endpoint crash β mount via raw ASGI Route (#1594)
+- Fix `/llm` per-request provider override, Redis config from host/port/password (#1611, #1817)
+- Fix screenshot respects `scan_full_page=False` (#1750)
+- Fix screenshot distortion on Elementor sites (#1370)
+- Fix deep crawl timeout and `arun_many` dispatcher bypass (#1818, #1509)
+
+### Other
+
+- Replace `tf-playwright-stealth` with `playwright-stealth` (#1553)
+- Allow local embeddings by removing OpenAI fallback (#1658)
+- Include GoogleSearchCrawler `script.js` in package distribution (#1711)
+- Fix bs4 deprecation warning (`text` β `string`) (#1077)
+- Run blocking `chardet.detect` in thread executor (#1751)
+- Wire `mean_delay`/`max_range` from CrawlerRunConfig into dispatcher rate limiter (#1786)
+
+---
+
+## Tests
+
+Added a comprehensive **291-test regression suite** covering all major subsystems: core crawl, content processing, extraction strategies, deep crawling, browser management, config serialization, utilities, and edge cases.
+
+---
+
+## Breaking Changes
+
+### `cleaned_html` Now Preserves `class` and `id` Attributes
+
+If you have downstream code that parses `cleaned_html` and assumes no class/id attributes are present, this may need updating. This change enables users to do CSS-based analysis on cleaned HTML.
+
+### Docker: Redis Upgraded to 7.2.7
+
+If you pin Redis versions in your deployment, update to 7.2.7 or later.
+
+---
+
+## Upgrade Instructions
+
+### Python Package
+
+```bash
+pip install --upgrade crawl4ai
+# or
+pip install crawl4ai==0.8.5
+```
+
+### Docker
+
+```bash
+docker pull unclecode/crawl4ai:0.8.5
+
+docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.8.5
+```
+
+---
+
+## Verification
+
+Run the verification tests to confirm all features are working:
+
+```bash
+python docs/releases_review/demo_v0.8.5.py
+```
+
+This runs 13 actual tests that crawl real URLs and verify each feature end-to-end.
+
+---
+
+## Acknowledgments
+
+This release includes contributions from a large number of community members. Thank you to everyone who submitted PRs, reported issues, and provided reproduction steps. Special thanks to all contributors listed in [CONTRIBUTORS.md](../CONTRIBUTORS.md).
+
+Issues fixed: #462, #880, #943, #1031, #1077, #1183, #1213, #1251, #1281, #1290, #1296, #1308, #1354, #1364, #1370, #1374, #1424, #1435, #1463, #1484, #1487, #1489, #1494, #1503, #1509, #1512, #1520, #1553, #1594, #1601, #1606, #1611, #1622, #1635, #1640, #1658, #1666, #1667, #1668, #1671, #1682, #1686, #1711, #1715, #1716, #1721, #1730, #1731, #1746, #1750, #1751, #1754, #1758, #1762, #1768, #1770, #1776, #1782, #1783, #1784, #1786, #1788, #1789, #1790, #1792, #1793, #1794, #1795, #1796, #1797, #1801, #1803, #1804, #1805, #1815, #1817, #1818, #1824
+
+---
+
+## Support & Resources
+
+- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
+- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
+- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
+- **Twitter**: [@unclecode](https://x.com/unclecode)
+
+---
+
+**This is a massive releaseβ10 new features, critical security patches, and 60+ bug fixes. Whether you're dealing with anti-bot protection, shadow DOM sites, or just want more reliable crawls at scale, v0.8.5 has you covered. Thank you for your continued support!**
+
+**Happy crawling!**
+
+*- unclecode*
diff --git a/docs/examples/adaptive_crawling/export_import_kb.py b/docs/examples/adaptive_crawling/export_import_kb.py
index c0a72c2c1..476eb700c 100644
--- a/docs/examples/adaptive_crawling/export_import_kb.py
+++ b/docs/examples/adaptive_crawling/export_import_kb.py
@@ -114,7 +114,7 @@ async def import_and_continue():
# Import existing knowledge base
print(f"\n1. Importing knowledge base from {kb_path}")
- adaptive.import_knowledge_base(kb_path)
+ await adaptive.import_knowledge_base(kb_path)
print(f" - Imported {len(adaptive.state.knowledge_base)} documents")
print(f" - Existing URLs: {len(adaptive.state.crawled_urls)}")
@@ -175,10 +175,10 @@ async def share_knowledge_bases():
merged_crawler = AdaptiveCrawler(crawler)
# Import both knowledge bases
- merged_crawler.import_knowledge_base(project_a_kb)
+ await merged_crawler.import_knowledge_base(project_a_kb)
initial_size = len(merged_crawler.state.knowledge_base)
-
- merged_crawler.import_knowledge_base(project_b_kb)
+
+ await merged_crawler.import_knowledge_base(project_b_kb)
final_size = len(merged_crawler.state.knowledge_base)
print(f" - Project A documents: {initial_size}")
diff --git a/docs/examples/async_webcrawler_multiple_urls_example.py b/docs/examples/async_webcrawler_multiple_urls_example.py
index 52309d13e..37b6d2085 100644
--- a/docs/examples/async_webcrawler_multiple_urls_example.py
+++ b/docs/examples/async_webcrawler_multiple_urls_example.py
@@ -30,7 +30,7 @@ async def main():
results = await crawler.arun_many(
urls=urls,
word_count_threshold=word_count_threshold,
- bypass_cache=True,
+ cache_mode=CacheMode.BYPASS,
verbose=True,
)
diff --git a/docs/examples/crawlai_vs_firecrawl.py b/docs/examples/crawlai_vs_firecrawl.py
index f8b70dc70..ff89acb0f 100644
--- a/docs/examples/crawlai_vs_firecrawl.py
+++ b/docs/examples/crawlai_vs_firecrawl.py
@@ -34,7 +34,7 @@ async def compare():
url="https://www.nbcnews.com/business",
# js_code=["const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"],
word_count_threshold=0,
- bypass_cache=True,
+ cache_mode=CacheMode.BYPASS,
verbose=False,
)
end = time.time()
@@ -53,7 +53,7 @@ async def compare():
"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
],
word_count_threshold=0,
- bypass_cache=True,
+ cache_mode=CacheMode.BYPASS,
verbose=False,
)
end = time.time()
diff --git a/docs/examples/deep_crawl_cancellation.py b/docs/examples/deep_crawl_cancellation.py
new file mode 100644
index 000000000..c9cf06b32
--- /dev/null
+++ b/docs/examples/deep_crawl_cancellation.py
@@ -0,0 +1,427 @@
+"""
+Deep Crawl Cancellation Example
+
+This example demonstrates how to implement cancellable deep crawls in Crawl4AI.
+Useful for cloud platforms, job management systems, or any scenario where you
+need to stop a running crawl mid-execution and retrieve partial results.
+
+Features demonstrated:
+1. Callback-based cancellation (check external source like Redis)
+2. Direct cancellation via cancel() method
+3. Checking cancellation status
+4. State tracking with cancelled flag
+5. Strategy reuse after cancellation
+
+Requirements:
+ pip install crawl4ai redis
+"""
+
+import asyncio
+import json
+from typing import Dict, Any, List
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+from crawl4ai.deep_crawling import (
+ BFSDeepCrawlStrategy,
+ DFSDeepCrawlStrategy,
+ BestFirstCrawlingStrategy,
+)
+
+
+# =============================================================================
+# Example 1: Basic Cancellation with Callback
+# =============================================================================
+
+async def example_callback_cancellation():
+ """
+ Cancel a crawl after reaching a certain number of pages.
+ This simulates checking an external cancellation source.
+
+ Note: We use DFS here because it processes one URL at a time,
+ giving precise cancellation control. BFS processes URLs in batches
+ (levels), so cancellation happens at level boundaries.
+ """
+ print("\n" + "="*60)
+ print("Example 1: Callback-based Cancellation (DFS)")
+ print("="*60)
+
+ pages_crawled = 0
+ max_before_cancel = 5
+
+ # This callback is checked before each URL is processed
+ async def should_cancel():
+ # In production, you might check Redis, a database, or an API:
+ # job = await redis.get(f"job:{job_id}")
+ # return job.get("status") == "cancelled"
+ return pages_crawled >= max_before_cancel
+
+ # Track progress via state changes
+ async def on_state_change(state: Dict[str, Any]):
+ nonlocal pages_crawled
+ pages_crawled = state.get("pages_crawled", 0)
+ cancelled = state.get("cancelled", False)
+ print(f" Progress: {pages_crawled} pages | Cancelled: {cancelled}")
+
+ # Use DFS for precise per-URL cancellation control
+ strategy = DFSDeepCrawlStrategy(
+ max_depth=3,
+ max_pages=100, # Would crawl up to 100, but we'll cancel at 5
+ should_cancel=should_cancel,
+ on_state_change=on_state_change,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ verbose=False,
+ )
+
+ print(f"Starting crawl (will cancel after {max_before_cancel} pages)...")
+
+ async with AsyncWebCrawler() as crawler:
+ results = await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ )
+
+ print(f"\nResults:")
+ print(f" - Crawled {len(results)} pages")
+ print(f" - Strategy cancelled: {strategy.cancelled}")
+ print(f" - Pages crawled counter: {strategy._pages_crawled}")
+
+ return results
+
+
+# =============================================================================
+# Example 2: Direct Cancellation via cancel() Method
+# =============================================================================
+
+async def example_direct_cancellation():
+ """
+ Cancel a crawl directly using the cancel() method.
+ This is useful when you have direct access to the strategy instance.
+ """
+ print("\n" + "="*60)
+ print("Example 2: Direct Cancellation via cancel()")
+ print("="*60)
+
+ strategy = BFSDeepCrawlStrategy(
+ max_depth=3,
+ max_pages=100,
+ )
+
+ # Cancel after 3 seconds
+ async def cancel_after_delay():
+ await asyncio.sleep(3)
+ print(" Calling strategy.cancel()...")
+ strategy.cancel()
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ verbose=False,
+ )
+
+ print("Starting crawl (will cancel after 3 seconds)...")
+
+ async with AsyncWebCrawler() as crawler:
+ # Start cancellation timer in background
+ cancel_task = asyncio.create_task(cancel_after_delay())
+
+ try:
+ results = await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ )
+ finally:
+ cancel_task.cancel()
+ try:
+ await cancel_task
+ except asyncio.CancelledError:
+ pass
+
+ print(f"\nResults:")
+ print(f" - Crawled {len(results)} pages")
+ print(f" - Strategy cancelled: {strategy.cancelled}")
+
+ return results
+
+
+# =============================================================================
+# Example 3: Streaming Mode with Cancellation
+# =============================================================================
+
+async def example_streaming_cancellation():
+ """
+ Cancel a streaming crawl and process partial results as they arrive.
+ """
+ print("\n" + "="*60)
+ print("Example 3: Streaming Mode with Cancellation")
+ print("="*60)
+
+ results_count = 0
+ cancel_after = 3
+
+ async def should_cancel():
+ return results_count >= cancel_after
+
+ strategy = DFSDeepCrawlStrategy(
+ max_depth=5,
+ max_pages=50,
+ should_cancel=should_cancel,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ stream=True, # Enable streaming
+ verbose=False,
+ )
+
+ print(f"Starting streaming crawl (will cancel after {cancel_after} results)...")
+
+ results = []
+ async with AsyncWebCrawler() as crawler:
+ async for result in await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ ):
+ results_count += 1
+ results.append(result)
+ print(f" Received result {results_count}: {result.url[:60]}...")
+
+ print(f"\nResults:")
+ print(f" - Received {len(results)} results")
+ print(f" - Strategy cancelled: {strategy.cancelled}")
+
+ return results
+
+
+# =============================================================================
+# Example 4: Strategy Reuse After Cancellation
+# =============================================================================
+
+async def example_strategy_reuse():
+ """
+ Demonstrate that a strategy can be reused after cancellation.
+ The cancel flag is automatically reset on the next crawl.
+ """
+ print("\n" + "="*60)
+ print("Example 4: Strategy Reuse After Cancellation")
+ print("="*60)
+
+ crawl_number = 0
+
+ async def cancel_first_crawl_only():
+ # Only cancel during the first crawl
+ return crawl_number == 1
+
+ strategy = BFSDeepCrawlStrategy(
+ max_depth=1,
+ max_pages=10,
+ should_cancel=cancel_first_crawl_only,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ verbose=False,
+ )
+
+ async with AsyncWebCrawler() as crawler:
+ # First crawl - will be cancelled immediately
+ crawl_number = 1
+ print("First crawl (will be cancelled)...")
+ results1 = await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ )
+ print(f" - Results: {len(results1)}, Cancelled: {strategy.cancelled}")
+
+ # Second crawl - should work normally
+ crawl_number = 2
+ print("\nSecond crawl (should complete normally)...")
+ results2 = await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ )
+ print(f" - Results: {len(results2)}, Cancelled: {strategy.cancelled}")
+
+ print(f"\nSummary:")
+ print(f" - First crawl: {len(results1)} results (cancelled)")
+ print(f" - Second crawl: {len(results2)} results (completed)")
+
+
+# =============================================================================
+# Example 5: Best-First Strategy with Cancellation
+# =============================================================================
+
+async def example_best_first_cancellation():
+ """
+ Cancel a Best-First crawl that prioritizes URLs by relevance score.
+
+ Note: Best-First processes URLs in batches (default 10), so cancellation
+ happens at batch boundaries. You may see more results than the cancel
+ threshold before the crawl stops.
+ """
+ print("\n" + "="*60)
+ print("Example 5: Best-First Strategy with Cancellation")
+ print("="*60)
+
+ from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer
+
+ pages_crawled = 0
+ cancel_threshold = 5
+
+ async def should_cancel():
+ return pages_crawled >= cancel_threshold
+
+ async def track_progress(state: Dict[str, Any]):
+ nonlocal pages_crawled
+ pages_crawled = state.get("pages_crawled", 0)
+ print(f" Pages: {pages_crawled}, Cancelled: {state.get('cancelled', False)}")
+
+ scorer = KeywordRelevanceScorer(
+ keywords=["api", "example", "tutorial"],
+ weight=0.8
+ )
+
+ strategy = BestFirstCrawlingStrategy(
+ max_depth=2,
+ max_pages=50,
+ url_scorer=scorer,
+ should_cancel=should_cancel,
+ on_state_change=track_progress,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ stream=True,
+ verbose=False,
+ )
+
+ print(f"Starting Best-First crawl (will cancel after {cancel_threshold} pages)...")
+ print(" (Note: Best-First processes in batches, so may crawl slightly more)")
+
+ results = []
+ async with AsyncWebCrawler() as crawler:
+ async for result in await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ ):
+ results.append(result)
+ score = result.metadata.get("score", 0)
+ print(f" Result: {result.url[:50]}... (score: {score:.2f})")
+
+ print(f"\nResults:")
+ print(f" - Crawled {len(results)} high-priority pages")
+ print(f" - Strategy cancelled: {strategy.cancelled}")
+
+
+# =============================================================================
+# Example 6: Production Pattern - Redis-Based Cancellation (Simulated)
+# =============================================================================
+
+async def example_production_pattern():
+ """
+ Simulate a production pattern where cancellation is checked from Redis.
+ This pattern is suitable for cloud platforms with job management.
+ """
+ print("\n" + "="*60)
+ print("Example 6: Production Pattern (Simulated Redis)")
+ print("="*60)
+
+ # Simulate Redis storage
+ redis_storage: Dict[str, str] = {}
+
+ job_id = "crawl-job-12345"
+
+ # Simulate Redis operations
+ async def redis_get(key: str) -> str:
+ return redis_storage.get(key)
+
+ async def redis_set(key: str, value: str):
+ redis_storage[key] = value
+
+ # Initialize job status
+ await redis_set(f"{job_id}:status", "running")
+
+ # Cancellation check
+ async def check_cancelled():
+ status = await redis_get(f"{job_id}:status")
+ return status == "cancelled"
+
+ # Progress tracking
+ async def save_progress(state: Dict[str, Any]):
+ await redis_set(f"{job_id}:state", json.dumps(state))
+ await redis_set(f"{job_id}:pages", str(state["pages_crawled"]))
+ print(f" Saved progress: {state['pages_crawled']} pages")
+
+ strategy = BFSDeepCrawlStrategy(
+ max_depth=2,
+ max_pages=20,
+ should_cancel=check_cancelled,
+ on_state_change=save_progress,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ verbose=False,
+ )
+
+ # Simulate external cancellation after 2 seconds
+ async def external_cancel():
+ await asyncio.sleep(2)
+ print("\n [External] Setting job status to 'cancelled'...")
+ await redis_set(f"{job_id}:status", "cancelled")
+
+ print("Starting crawl with simulated Redis job management...")
+
+ async with AsyncWebCrawler() as crawler:
+ cancel_task = asyncio.create_task(external_cancel())
+
+ try:
+ results = await crawler.arun(
+ "https://docs.crawl4ai.com",
+ config=config
+ )
+ finally:
+ cancel_task.cancel()
+ try:
+ await cancel_task
+ except asyncio.CancelledError:
+ pass
+
+ # Final status
+ final_status = "cancelled" if strategy.cancelled else "completed"
+ await redis_set(f"{job_id}:status", final_status)
+
+ print(f"\nJob completed:")
+ print(f" - Final status: {final_status}")
+ print(f" - Pages crawled: {await redis_get(f'{job_id}:pages')}")
+ print(f" - Results returned: {len(results)}")
+
+ # Show final state
+ final_state = json.loads(await redis_get(f"{job_id}:state"))
+ print(f" - State saved: {len(final_state.get('visited', []))} URLs visited")
+
+
+# =============================================================================
+# Main
+# =============================================================================
+
+async def main():
+ """Run all examples."""
+ print("="*60)
+ print("Deep Crawl Cancellation Examples")
+ print("="*60)
+
+ await example_callback_cancellation()
+ await example_direct_cancellation()
+ await example_streaming_cancellation()
+ await example_strategy_reuse()
+ await example_best_first_cancellation()
+ await example_production_pattern()
+
+ print("\n" + "="*60)
+ print("All examples completed!")
+ print("="*60)
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/docs/examples/full_page_screenshot_and_pdf_export.md b/docs/examples/full_page_screenshot_and_pdf_export.md
index bf11f8db0..6d7c632d8 100644
--- a/docs/examples/full_page_screenshot_and_pdf_export.md
+++ b/docs/examples/full_page_screenshot_and_pdf_export.md
@@ -56,5 +56,53 @@ if __name__ == "__main__":
- If `screenshot=True`, and a PDF is already available, it directly converts the first page of that PDF to an image for youβno repeated loading or scrolling.
- Finally, you get your PDF and/or screenshot ready to use.
+**Controlling scroll speed for full-page screenshots:**
+When a page is taller than `screenshot_height_threshold` (default ~20,000px) and no PDF is available, Crawl4AI scrolls through the page to capture a stitched full-page screenshot. Use `scroll_delay` to control the pause between scroll steps:
+
+```python
+config = CrawlerRunConfig(
+ screenshot=True,
+ scroll_delay=0.5, # Wait 0.5s between scroll steps (default: 0.2)
+)
+```
+
+This is particularly useful for pages with lazy-loaded images or animations that need time to render during scrolling.
+
+---
+
+## Viewport-Only Screenshots
+
+If you only need a screenshot of the visible viewport (not the entire page), use the `force_viewport_screenshot` option. This is significantly faster and produces smaller images, especially for long pages.
+
+```python
+import os
+import asyncio
+from base64 import b64decode
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+async def main():
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url='https://en.wikipedia.org/wiki/List_of_common_misconceptions',
+ config=CrawlerRunConfig(
+ screenshot=True,
+ force_viewport_screenshot=True # Only capture the visible viewport
+ )
+ )
+
+ if result.success and result.screenshot:
+ with open("viewport_screenshot.png", "wb") as f:
+ f.write(b64decode(result.screenshot))
+ print("Viewport screenshot saved!")
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+
+**When to use viewport-only screenshots:**
+- You only need what's visible above the fold
+- You want faster screenshot capture on long pages
+- You need smaller image sizes for thumbnails or previews
+
**Conclusion:**
With this feature, Crawl4AI becomes even more robust and versatile for large-scale content extraction. Whether you need a PDF snapshot or a quick screenshot, you now have a reliable solution for even the most extensive webpages.
\ No newline at end of file
diff --git a/docs/examples/serp_api_project_11_feb.py b/docs/examples/serp_api_project_11_feb.py
index df0768ed7..8f9312f73 100644
--- a/docs/examples/serp_api_project_11_feb.py
+++ b/docs/examples/serp_api_project_11_feb.py
@@ -8,6 +8,7 @@
BrowserConfig,
CrawlerRunConfig,
CacheMode,
+ LLMConfig,
LLMExtractionStrategy,
JsonCssExtractionStrategy,
CrawlerHub,
@@ -63,8 +64,10 @@ class GoogleSearchResult(BaseModel):
snippet: str
sitelinks: Optional[List[Sitelink]] = None
- llm_extraction_strategy = LLMExtractionStrategy(
- provider = "openai/gpt-4o",
+ llm_extraction_strategy = LLMExtractionStrategy(
+ llm_config = LLMConfig(
+ provider = "openai/gpt-4o",
+ ),
schema = GoogleSearchResult.model_json_schema(),
instruction="""I want to extract the title, link, snippet, and sitelinks from a Google search result. I shared here the content of div#search from the search result page. We are just interested in organic search results.
Example:
diff --git a/docs/examples/shadow_dom_crawling.py b/docs/examples/shadow_dom_crawling.py
new file mode 100644
index 000000000..49b098bbb
--- /dev/null
+++ b/docs/examples/shadow_dom_crawling.py
@@ -0,0 +1,77 @@
+"""
+Shadow DOM Crawling Example
+============================
+
+Demonstrates how to use `flatten_shadow_dom=True` to extract content
+hidden inside Shadow DOM trees on sites built with Web Components
+(Stencil, Lit, Shoelace, Angular Elements, etc.).
+
+Shadow DOM creates encapsulated sub-trees that are invisible to the
+normal page serialization (page.content() / outerHTML). The
+`flatten_shadow_dom` option walks these trees and produces a single
+flat HTML document that includes all shadow content.
+
+This example crawls a Bosch Rexroth product page where the product
+description, technical specs, and downloads are rendered entirely
+inside Shadow DOM by Stencil.js web components.
+"""
+
+import asyncio
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
+
+URL = "https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011"
+
+
+async def main():
+ browser_config = BrowserConfig(headless=True)
+
+ # ββ 1. Baseline: without shadow DOM flattening ββββββββββββββββββ
+ print("=" * 60)
+ print("Without flatten_shadow_dom (baseline)")
+ print("=" * 60)
+
+ config = CrawlerRunConfig(
+ wait_until="load",
+ delay_before_return_html=3.0,
+ )
+
+ async with AsyncWebCrawler(config=browser_config) as crawler:
+ result = await crawler.arun(URL, config=config)
+
+ md = result.markdown.raw_markdown if result.markdown else ""
+ print(f"Markdown length: {len(md)} chars")
+ print(f"Has product description: {'mill type design' in md.lower()}")
+ print(f"Has technical specs: {'CDH1' in md}")
+ print(f"Has downloads section: {'Downloads' in md}")
+ print()
+
+ # ββ 2. With shadow DOM flattening βββββββββββββββββββββββββββββββ
+ print("=" * 60)
+ print("With flatten_shadow_dom=True")
+ print("=" * 60)
+
+ config = CrawlerRunConfig(
+ wait_until="load",
+ delay_before_return_html=3.0,
+ flatten_shadow_dom=True,
+ )
+
+ async with AsyncWebCrawler(config=browser_config) as crawler:
+ result = await crawler.arun(URL, config=config)
+
+ md = result.markdown.raw_markdown if result.markdown else ""
+ print(f"Markdown length: {len(md)} chars")
+ print(f"Has product description: {'mill type design' in md.lower()}")
+ print(f"Has technical specs: {'CDH1' in md}")
+ print(f"Has downloads section: {'Downloads' in md}")
+ print()
+
+ # Show the product content section
+ idx = md.find("Product Description")
+ if idx >= 0:
+ print("ββ Extracted product content ββ")
+ print(md[idx:idx + 1200])
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/docs/examples/summarize_page.py b/docs/examples/summarize_page.py
index cd377d80e..b54fd1a97 100644
--- a/docs/examples/summarize_page.py
+++ b/docs/examples/summarize_page.py
@@ -1,17 +1,13 @@
-import os
+import asyncio
import json
-from crawl4ai.web_crawler import WebCrawler
-from crawl4ai.chunking_strategy import *
-from crawl4ai import *
-from crawl4ai.crawler_strategy import *
-
-url = r"https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot"
-
-crawler = WebCrawler()
-crawler.warmup()
+import os
from pydantic import BaseModel, Field
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
+from crawl4ai import CacheMode, LLMExtractionStrategy
+
+url = r"https://marketplace.visualstudio.com/items?itemName=Unclecode.groqopilot"
class PageSummary(BaseModel):
title: str = Field(..., description="Title of the page.")
@@ -20,29 +16,38 @@ class PageSummary(BaseModel):
keywords: list = Field(..., description="Keywords assigned to the page.")
-result = crawler.run(
- url=url,
- word_count_threshold=1,
- extraction_strategy=LLMExtractionStrategy(
- provider="openai/gpt-4o",
- api_token=os.getenv("OPENAI_API_KEY"),
- schema=PageSummary.model_json_schema(),
- extraction_type="schema",
- apply_chunking=False,
- instruction="From the crawled content, extract the following details: "
- "1. Title of the page "
- "2. Summary of the page, which is a detailed summary "
- "3. Brief summary of the page, which is a paragraph text "
- "4. Keywords assigned to the page, which is a list of keywords. "
- "The extracted JSON format should look like this: "
- '{ "title": "Page Title", "summary": "Detailed summary of the page.", "brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }',
- ),
- bypass_cache=True,
-)
-
-page_summary = json.loads(result.extracted_content)
-
-print(page_summary)
-
-with open(".data/page_summary.json", "w", encoding="utf-8") as f:
- f.write(result.extracted_content)
+async def main():
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url=url,
+ config=CrawlerRunConfig(
+ word_count_threshold=1,
+ cache_mode=CacheMode.BYPASS,
+ extraction_strategy=LLMExtractionStrategy(
+ llm_config=LLMConfig(
+ provider="openai/gpt-4o",
+ api_token=os.getenv("OPENAI_API_KEY"),
+ ),
+ schema=PageSummary.model_json_schema(),
+ extraction_type="schema",
+ apply_chunking=False,
+ instruction="From the crawled content, extract the following details: "
+ "1. Title of the page "
+ "2. Summary of the page, which is a detailed summary "
+ "3. Brief summary of the page, which is a paragraph text "
+ "4. Keywords assigned to the page, which is a list of keywords. "
+ 'The extracted JSON format should look like this: '
+ '{ "title": "Page Title", "summary": "Detailed summary of the page.", "brief_summary": "Brief summary in a paragraph.", "keywords": ["keyword1", "keyword2", "keyword3"] }',
+ ),
+ ),
+ )
+
+ page_summary = json.loads(result.extracted_content)
+ print(page_summary)
+
+ with open(".data/page_summary.json", "w", encoding="utf-8") as f:
+ f.write(result.extracted_content)
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/docs/examples/website-to-api/web_scraper_lib.py b/docs/examples/website-to-api/web_scraper_lib.py
index ded6f6f72..afa58e223 100644
--- a/docs/examples/website-to-api/web_scraper_lib.py
+++ b/docs/examples/website-to-api/web_scraper_lib.py
@@ -170,7 +170,7 @@ async def _load_or_generate_schema(self, url: str, query: str, session_id: str =
delay_before_return_html=5,
)
)
- html = result.fit_html
+ html = result.markdown.fit_html
# Step 2: Generate schema using AI with custom model if specified
print("AI is analyzing the page structure...")
diff --git a/docs/md_v2/CONTRIBUTING.md b/docs/md_v2/CONTRIBUTING.md
new file mode 100644
index 000000000..38d22c2cf
--- /dev/null
+++ b/docs/md_v2/CONTRIBUTING.md
@@ -0,0 +1,102 @@
+# Contributing to Crawl4AI
+
+Welcome to the Crawl4AI project! As an open-source library for web crawling and AI integration, we value contributions from the community. This guide explains our branching strategy, how to contribute effectively, and the overall release process. Our goal is to maintain a stable, collaborative environment where bug fixes, features, and improvements can be integrated smoothly while allowing for experimental development.
+
+We follow a GitFlow-inspired workflow to ensure predictability and quality. Releases occur approximately every two weeks, with a focus on semantic versioning, comprehensive documentation, and user-friendly updates.
+
+## Core Branches
+
+- **main**: The stable branch containing production-ready code. It's always identical to the latest released version and is tagged for releases. Do not submit PRs directly here.
+- **develop**: The primary integration branch for ongoing development. This is where all contributions (bug fixes, minor features, documentation updates) are merged. Submit your pull requests targeting this branch.
+- **next**: Reserved for the lead maintainer (Unclecode) to experiment with major features, refactors, or cutting-edge changes. These are merged into `develop` when ready.
+- **release/vX.Y.Z**: Temporary branches created from `develop` for final release preparations (e.g., version bumps, demos, release notes). These are short-lived and deleted after the release.
+
+## Contributor Workflow
+
+We encourage contributions of all kinds: bug fixes, new features, documentation improvements, tests, or even Docker enhancements. Follow these steps to contribute:
+
+1. **Fork the Repository**: Create your own fork on GitHub.
+2. **Create a Branch**: Base your work on the `develop` branch.
+
+ ```
+ git checkout develop
+ git checkout -b feature/your-feature-name # Or bugfix/your-bugfix-name
+
+ ```
+
+3. **Make Changes**:
+ - Implement your feature or fix.
+ - If updating documentation (e.g., README.md, mkdocs.yml, or docs/blog/), ensure version references are consistent (e.g., update site_name in mkdocs.yml to reflect the upcoming version if relevant).
+ - For Docker-related changes (e.g., Dockerfile, docker-compose.yml, or docs/md_v2/core/docker-deployment.md), test locally and include build instructions in your PR description.
+ - Add tests if applicable (run `pytest` to verify).
+ - Follow code style guidelines (use black for formatting).
+4. **Commit and Push**:
+ - Use descriptive commit messages (e.g., "Fix: Resolve issue with async crawling").
+ - Push to your fork: `git push origin feature/your-feature-name`.
+5. **Submit a Pull Request**:
+ - Target the `develop` branch.
+ - Provide a clear description: What does it do? Link to any issues. Include screenshots or code examples if helpful.
+ - If your change affects documentation or Docker, mention how it aligns with the version (e.g., "Updates Docker docs for v0.7.0 compatibility").
+ - We'll review and merge approved PRs into `develop`.
+6. **Discuss Large Changes**: For major features or experimental ideas, open an issue first to align with the project's direction.
+
+If your PR involves breaking changes, include a migration guide in the description.
+
+## Lead Maintainer's Workflow (For Reference)
+
+- The lead maintainer (Unclecode) uses the `next` branch for isolated experimental work.
+- Features from `next` are periodically merged into `develop` (via rebase and merge) to keep everything in sync.
+- This isolation ensures your contributions aren't disrupted by ongoing major changes.
+
+## Release Process (High-Level Overview)
+
+Releases happen bi-weekly to ship improvements regularly. As a contributor, your merged changes in `develop` will be included in the next release unless specified otherwise. Here's a summary of what happens:
+
+- **Preparation**: A temporary `release/vX.Y.Z` branch is created from `develop`. Any ready features from `next` are merged here.
+- **Final Updates**:
+ - Version bump in code (e.g., `__version__.py`).
+ - Creation of a demo script in `examples/` to showcase new features.
+ - Writing release notes in `docs/blog/` (personal "I" voice from Unclecode, with code examples, impacts, and migration guides if needed).
+ - Documentation updates: README.md (highlights, version refs), mkdocs.yml (site_name with version), docs/blog/index.md (add new release), and copying notes to `docs/md_v2/blog/releases/`.
+ - Docker updates: Dockerfile (version arg), docker-compose.yml, deploy/docker/README.md, and docs/md_v2/core/docker-deployment.md. A release candidate image (e.g., `X.Y.Z-r1`) is built and tested.
+- **Testing and Merge**: Full tests run; changes committed and merged to `main` with a tag.
+- **Publication**: Tagged release on GitHub (with notes), publish to PyPI, and push Docker images (stable and `latest` after testing).
+- **Sync**: Back-merge to `develop` and reset `next` for the next cycle.
+
+Semantic versioning is used: MAJOR for breaking changes, MINOR for features, PATCH for fixes. Pre-releases (e.g., `-rc1`) may be used for testing.
+
+If your contribution requires Docker testing or affects docs, it may be part of this stepβfeel free to suggest updates in your PR.
+
+## Benefits of This Approach
+
+- **Stability**: `main` is always reliable for users.
+- **Collaboration**: Fixed PR target (`develop`) makes contributing straightforward.
+- **Isolation**: Experimental work in `next` doesn't block team progress.
+- **User-Focused**: Releases include demos, detailed notes, and updated docs/Docker for easy adoption.
+- **Predictability**: Bi-weekly cadence keeps the project active.
+
+## Checklist for Contributors
+
+Before submitting a PR:
+
+- [ ] Based on and targeting `develop`.
+- [ ] Tests pass (`pytest`).
+- [ ] Docs updated if needed (e.g., version refs in mkdocs.yml, Docker files).
+- [ ] No breaking changes without a migration guide.
+- [ ] Descriptive title and description.
+
+## Common Issues
+
+- **Merge Conflicts**: Rebase your branch on latest `develop` before PR.
+- **Docker Builds**: Test multi-arch (amd64/arm64) locally if changing Dockerfile.
+- **Version Consistency**: Ensure any version mentions match semantic rules.
+
+## Communication
+
+- Open issues for discussions or bugs.
+- Join our Discord (link in README) for real-time help.
+- After releases, announcements go to GitHub, Discord, and social media.
+
+Thanks for contributing to Crawl4AI β we appreciate your help in making it better!
+
+*Last Updated: Feb 3, 2026*
diff --git a/docs/md_v2/advanced/advanced-features.md b/docs/md_v2/advanced/advanced-features.md
index 211869c38..75ba1ccef 100644
--- a/docs/md_v2/advanced/advanced-features.md
+++ b/docs/md_v2/advanced/advanced-features.md
@@ -112,6 +112,7 @@ if __name__ == "__main__":
**Relevant Parameters**
- **`pdf=True`**: Exports the current page as a PDF (base64-encoded in `result.pdf`).
- **`screenshot=True`**: Creates a screenshot (base64-encoded in `result.screenshot`).
+- **`scroll_delay`**: Controls the delay (seconds) between scroll steps when taking a full-page screenshot of a tall page. Defaults to `0.2`. Increase for pages with slow-loading assets.
- **`scan_full_page`** or advanced hooking can further refine how the crawler captures content.
---
diff --git a/docs/md_v2/advanced/anti-bot-and-fallback.md b/docs/md_v2/advanced/anti-bot-and-fallback.md
new file mode 100644
index 000000000..68ed03de8
--- /dev/null
+++ b/docs/md_v2/advanced/anti-bot-and-fallback.md
@@ -0,0 +1,263 @@
+# Anti-Bot Detection & Fallback
+
+When crawling sites protected by anti-bot systems (Akamai, Cloudflare, PerimeterX, DataDome, Imperva, etc.), requests often get blocked with CAPTCHAs, 403 responses, or empty pages. Crawl4AI provides a layered retry and fallback system that automatically detects blocking and escalates through multiple strategies until content is retrieved.
+
+## How Detection Works
+
+After each crawl attempt, Crawl4AI inspects the HTTP status code and HTML content for known anti-bot signals:
+
+- **HTTP 403/429** with short or empty response bodies
+- **Challenge pages** β Cloudflare "Just a moment", Akamai "Access Denied", PerimeterX block pages
+- **CAPTCHA injection** β reCAPTCHA, hCaptcha, or vendor-specific challenges on otherwise empty pages
+- **Firewall blocks** β Imperva/Incapsula resource iframes, Sucuri firewall pages, Cloudflare error codes
+
+Detection uses structural HTML markers (specific element IDs, script sources, form actions) rather than generic keywords to minimize false positives. A normal page that happens to mention "CAPTCHA" or "Cloudflare" in its content will not be flagged.
+
+When all attempts fail and blocking is still detected, the result is returned with `success=False` and `error_message` describing the block reason.
+
+## Configuration Options
+
+All anti-bot retry options live on `CrawlerRunConfig`:
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `proxy_config` | `ProxyConfig`, `list[ProxyConfig]`, or `None` | `None` | Single proxy or ordered list of proxies to try. Each retry round iterates through the full list. Use `"direct"` or `ProxyConfig.DIRECT` in a list to explicitly try without a proxy. |
+| `max_retries` | `int` | `0` | Number of retry rounds when blocking is detected. `0` = no retries. |
+| `fallback_fetch_function` | `async (str) -> str` | `None` | Async function called as last resort. Takes URL, returns raw HTML. |
+
+## Escalation Chain
+
+Each retry round tries every proxy in `proxy_config` in order. If all rounds are exhausted and the page is still blocked, the fallback fetch function is called as a last resort.
+
+```
+For each round (1 + max_retries rounds):
+ 1. Try proxy_config[0] (or direct if proxy_config is None)
+ 2. If blocked β try proxy_config[1]
+ 3. If blocked β try proxy_config[2]
+ 4. ... continue through all proxies
+ 5. If any attempt succeeds β done
+
+If all rounds exhausted and still blocked:
+ 6. Call fallback_fetch_function(url) β process returned HTML
+```
+
+Worst-case attempts before the fetch function: `(1 + max_retries) x len(proxy_config)`
+
+## Crawl Stats
+
+Every crawl result includes a `crawl_stats` dict with detailed attempt tracking:
+
+```python
+result.crawl_stats = {
+ "attempts": 3, # total browser attempts made
+ "retries": 1, # retry rounds used (0 = succeeded first round)
+ "proxies_used": [ # ordered list of every attempt
+ {"proxy": None, "status_code": 403, "blocked": True, "reason": "Akamai block (Reference #)"},
+ {"proxy": "proxy.io:8080", "status_code": 403, "blocked": True, "reason": "Akamai block (Reference #)"},
+ {"proxy": "premium.io:9090", "status_code": 200, "blocked": False, "reason": ""},
+ ],
+ "fallback_fetch_used": False, # whether fallback_fetch_function was called
+ "resolved_by": "proxy", # "direct" | "proxy" | "fallback_fetch" | null (all failed)
+}
+```
+
+## Usage Examples
+
+### Simple Retry (No Proxy)
+
+Retry the crawl up to 3 times when blocking is detected. Useful when blocks are intermittent or IP-based.
+
+```python
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
+
+async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
+ result = await crawler.arun(
+ url="https://example.com",
+ config=CrawlerRunConfig(max_retries=3),
+ )
+```
+
+### Single Proxy
+
+Pass a single `ProxyConfig` β it's used on every attempt. Same behavior as always.
+
+```python
+from crawl4ai.async_configs import ProxyConfig
+
+config = CrawlerRunConfig(
+ max_retries=2,
+ proxy_config=ProxyConfig(
+ server="http://proxy.example.com:8080",
+ username="user",
+ password="pass",
+ ),
+)
+```
+
+### Direct-First, Then Proxies
+
+Try without a proxy first, then escalate to proxies if blocked. Use `ProxyConfig.DIRECT` (or the string `"direct"`) in the list to represent a no-proxy attempt.
+
+```python
+config = CrawlerRunConfig(
+ max_retries=1,
+ proxy_config=[
+ ProxyConfig.DIRECT, # Try without proxy first
+ ProxyConfig(
+ server="http://datacenter-proxy.example.com:8080",
+ username="user",
+ password="pass",
+ ),
+ ProxyConfig(
+ server="http://residential-proxy.example.com:9090",
+ username="user",
+ password="pass",
+ ),
+ ],
+)
+```
+
+With this setup, each round tries direct first, then datacenter, then residential. With `max_retries=1`, worst case is 2 rounds x 3 steps = 6 attempts.
+
+### Proxy List (Escalation)
+
+Pass a list of proxies. They're tried in order β first one that works wins. Within each retry round, the entire list is tried again.
+
+```python
+config = CrawlerRunConfig(
+ max_retries=1,
+ proxy_config=[
+ ProxyConfig(
+ server="http://datacenter-proxy.example.com:8080",
+ username="user",
+ password="pass",
+ ),
+ ProxyConfig(
+ server="http://residential-proxy.example.com:9090",
+ username="user",
+ password="pass",
+ ),
+ ],
+)
+```
+
+With this setup, each round tries the datacenter proxy first, then the residential proxy. With `max_retries=1`, worst case is 2 rounds x 2 proxies = 4 attempts.
+
+### Fallback Fetch Function
+
+When all browser-based attempts fail, call a custom async function as a last resort. This function receives the URL and must return raw HTML as a string. The returned HTML is processed through the normal pipeline (markdown generation, extraction, etc.).
+
+This is useful when you have access to a scraping API, a pre-fetched cache, or any other source of HTML.
+
+```python
+import aiohttp
+
+async def my_scraping_api(url: str) -> str:
+ """Fetch HTML via an external scraping API."""
+ async with aiohttp.ClientSession() as session:
+ async with session.get(
+ "https://api.my-scraping-service.com/fetch",
+ params={"url": url, "format": "html"},
+ headers={"Authorization": "Bearer MY_TOKEN"},
+ ) as resp:
+ if resp.status == 200:
+ return await resp.text()
+ raise RuntimeError(f"API error: {resp.status}")
+
+config = CrawlerRunConfig(
+ max_retries=1,
+ fallback_fetch_function=my_scraping_api,
+)
+```
+
+The function can do anything β call an API, read from a database, return cached HTML, or make a simple HTTP request with a different library. Crawl4AI does not care how the HTML is obtained.
+
+### Full Escalation (All Features Combined)
+
+This example combines every layer: stealth mode, a list of proxies tried in order, retries, and a final fetch function.
+
+```python
+import aiohttp
+from crawl4ai import AsyncWebCrawler
+from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, ProxyConfig
+
+# Last-resort: fetch HTML via an external service
+async def external_fetch(url: str) -> str:
+ async with aiohttp.ClientSession() as session:
+ async with session.post(
+ "https://api.my-service.com/scrape",
+ json={"url": url, "render_js": True},
+ headers={"Authorization": "Bearer MY_TOKEN"},
+ ) as resp:
+ return await resp.text()
+
+browser_config = BrowserConfig(
+ headless=True,
+ enable_stealth=True,
+)
+
+crawl_config = CrawlerRunConfig(
+ magic=True,
+ wait_until="load",
+ max_retries=2,
+
+ # Proxies tried in order β cheapest first
+ proxy_config=[
+ ProxyConfig(
+ server="http://datacenter-proxy.example.com:8080",
+ username="user",
+ password="pass",
+ ),
+ ProxyConfig(
+ server="http://residential-proxy.example.com:9090",
+ username="user",
+ password="pass",
+ ),
+ ],
+
+ # Last resort β called after all retries and proxies are exhausted
+ fallback_fetch_function=external_fetch,
+)
+
+async with AsyncWebCrawler(config=browser_config) as crawler:
+ result = await crawler.arun(
+ url="https://protected-site.com/products",
+ config=crawl_config,
+ )
+
+ if result.success:
+ print(f"Got {len(result.markdown.raw_markdown)} chars of markdown")
+ print(f"Resolved by: {result.crawl_stats['resolved_by']}")
+ print(f"Attempts: {result.crawl_stats['attempts']}")
+ else:
+ print(f"All attempts failed: {result.error_message}")
+```
+
+**What happens step by step:**
+
+| Round | Attempt | What runs |
+|---|---|---|
+| 1 | 1 | Datacenter proxy β blocked |
+| 1 | 2 | Residential proxy β blocked |
+| 2 | 1 | Datacenter proxy β blocked |
+| 2 | 2 | Residential proxy β blocked |
+| 3 | 1 | Datacenter proxy β blocked |
+| 3 | 2 | Residential proxy β blocked |
+| - | - | `external_fetch(url)` called β returns HTML |
+
+That's up to 6 browser attempts + 1 function call before giving up.
+
+## Tips
+
+- **Start with `max_retries=0`** and a `fallback_fetch_function` if you just want a safety net without burning time on retries.
+- **Order proxies cheapest-first** β datacenter proxies before residential, residential before premium.
+- **Combine with stealth mode** β `BrowserConfig(enable_stealth=True)` and `CrawlerRunConfig(magic=True)` reduce the chance of being blocked in the first place.
+- **`wait_until="load"`** is important for anti-bot sites β the default `domcontentloaded` can return before the anti-bot sensor finishes.
+- **Check `crawl_stats`** to understand what happened β how many attempts, which proxy worked, whether the fallback function was needed.
+
+## See Also
+
+- [Proxy & Security](proxy-security.md) β Proxy setup, authentication, and rotation
+- [Undetected Browser](undetected-browser.md) β Stealth mode and browser fingerprint evasion
+- [Session Management](session-management.md) β Maintaining sessions across requests
diff --git a/docs/md_v2/advanced/proxy-security.md b/docs/md_v2/advanced/proxy-security.md
index d14e59ff3..39c5d7ad6 100644
--- a/docs/md_v2/advanced/proxy-security.md
+++ b/docs/md_v2/advanced/proxy-security.md
@@ -264,7 +264,7 @@ run_config = CrawlerRunConfig(proxy_config="socks5://proxy.example.com:1080")
```python
# Old (deprecated) approach
# from crawl4ai import BrowserConfig
-# browser_config = BrowserConfig(proxy="http://proxy.example.com:8080")
+# browser_config = BrowserConfig(proxy_config="http://proxy.example.com:8080")
# New (preferred) approach
from crawl4ai import CrawlerRunConfig
@@ -302,3 +302,7 @@ def safe_proxy_repr(proxy: ProxyConfig):
- Ensure `ProxyConfig.from_env()` actually loaded entries (`len(proxies) > 0`).
- Attach `proxy_rotation_strategy` to `CrawlerRunConfig`.
- Validate the proxy definitions you pass into the strategy.
+
+## See Also
+
+- [Anti-Bot Detection & Fallback](anti-bot-and-fallback.md) β Automatic retry with proxy escalation and fallback functions when anti-bot blocking is detected
diff --git a/docs/md_v2/advanced/undetected-browser.md b/docs/md_v2/advanced/undetected-browser.md
index 310701e67..32b73fc5f 100644
--- a/docs/md_v2/advanced/undetected-browser.md
+++ b/docs/md_v2/advanced/undetected-browser.md
@@ -391,4 +391,5 @@ Remember:
- [Advanced Features](advanced-features.md) - Overview of all advanced features
- [Proxy & Security](proxy-security.md) - Using proxies with anti-bot features
- [Session Management](session-management.md) - Maintaining sessions across requests
-- [Identity Based Crawling](identity-based-crawling.md) - Additional anti-detection strategies
\ No newline at end of file
+- [Identity Based Crawling](identity-based-crawling.md) - Additional anti-detection strategies
+- [Anti-Bot Detection & Fallback](anti-bot-and-fallback.md) - Automatic retry and proxy escalation when blocking is detected
\ No newline at end of file
diff --git a/docs/md_v2/api/adaptive-crawler.md b/docs/md_v2/api/adaptive-crawler.md
index af92ee3ad..5bd5bf440 100644
--- a/docs/md_v2/api/adaptive-crawler.md
+++ b/docs/md_v2/api/adaptive-crawler.md
@@ -161,7 +161,7 @@ adaptive.export_knowledge_base("my_knowledge.jsonl")
Import a previously exported knowledge base.
```python
-def import_knowledge_base(
+async def import_knowledge_base(
self,
path: Union[str, Path]
) -> None
diff --git a/docs/md_v2/api/arun.md b/docs/md_v2/api/arun.md
index a3086a8cc..4fc0cb502 100644
--- a/docs/md_v2/api/arun.md
+++ b/docs/md_v2/api/arun.md
@@ -60,12 +60,12 @@ run_config = CrawlerRunConfig(
)
```
-**Additional flags**:
+**Cache modes**:
-- `bypass_cache=True` acts like `CacheMode.BYPASS`.
-- `disable_cache=True` acts like `CacheMode.DISABLED`.
-- `no_cache_read=True` acts like `CacheMode.WRITE_ONLY`.
-- `no_cache_write=True` acts like `CacheMode.READ_ONLY`.
+- `CacheMode.BYPASS` β Skip cache entirely; always fetch fresh, write result to cache.
+- `CacheMode.DISABLED` β No caching at all; don't read or write.
+- `CacheMode.WRITE_ONLY` β Never read from cache, but write results.
+- `CacheMode.READ_ONLY` β Read from cache if available, never write.
---
diff --git a/docs/md_v2/api/arun_many.md b/docs/md_v2/api/arun_many.md
index 146584c36..740dcc24a 100644
--- a/docs/md_v2/api/arun_many.md
+++ b/docs/md_v2/api/arun_many.md
@@ -10,7 +10,7 @@ async def arun_many(
config: Optional[Union[CrawlerRunConfig, List[CrawlerRunConfig]]] = None,
dispatcher: Optional[BaseDispatcher] = None,
...
-) -> Union[List[CrawlResult], AsyncGenerator[CrawlResult, None]]:
+) -> RunManyReturn:
"""
Crawl multiple URLs concurrently or in batches.
@@ -20,7 +20,7 @@ async def arun_many(
- A list of `CrawlerRunConfig` objects with url_matcher patterns
:param dispatcher: (Optional) A concurrency controller (e.g.βMemoryAdaptiveDispatcher).
...
- :return: Either a list of `CrawlResult` objects, or an async generator if streaming is enabled.
+ :return: RunManyReturn containing either a list of `CrawlResult` objects or an async generator if streaming is enabled.
"""
```
@@ -29,7 +29,7 @@ async def arun_many(
1.β**Multiple URLs**:
- Instead of crawling a single URL, you pass a list of them (strings or tasks).β
- - The function returns either a **list** of `CrawlResult` or an **async generator** if streaming is enabled.
+ - The function returns `RunManyReturn` which contains either a **list** of `CrawlResult` or an **async generator** if streaming is enabled.
2.β**Concurrency & Dispatchers**:
@@ -164,7 +164,7 @@ results = await crawler.arun_many(
### Return Value
-Either a **list** of [`CrawlResult`](./crawl-result.md) objects, or an **async generator** if streaming is enabled.βYou can iterate to check `result.success` or read each itemβs `extracted_content`, `markdown`, or `dispatch_result`.
+Returns a **`RunManyReturn`** object which contains either a **list** of [`CrawlResult`](./crawl-result.md) objects, or an **async generator** if streaming is enabled.βYou can iterate to check `result.success` or read each itemβs `extracted_content`, `markdown`, or `dispatch_result`.
---
diff --git a/docs/md_v2/api/async-webcrawler.md b/docs/md_v2/api/async-webcrawler.md
index b8f105fc8..6bc8f362d 100644
--- a/docs/md_v2/api/async-webcrawler.md
+++ b/docs/md_v2/api/async-webcrawler.md
@@ -103,7 +103,7 @@ async def arun(
url: str,
config: Optional[CrawlerRunConfig] = None,
# Legacy parameters for backward compatibility...
-) -> CrawlResult:
+) -> RunManyReturn:
...
```
@@ -143,7 +143,7 @@ async def arun_many(
urls: List[str],
config: Optional[CrawlerRunConfig] = None,
# Legacy parameters maintained for backwards compatibility...
-) -> List[CrawlResult]:
+) -> RunManyReturn:
"""
Process multiple URLs with intelligent rate limiting and resource monitoring.
"""
@@ -200,7 +200,7 @@ Each `arun()` returns a **`CrawlResult`** containing:
- `url`: Final URL (if redirected).
- `html`: Original HTML.
- `cleaned_html`: Sanitized HTML.
-- `markdown_v2`: Deprecated. Instead just use regular `markdown`
+- `markdown_v2`: Removed in v0.5. Accessing it raises `AttributeError`; use `markdown`.
- `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies).
- `screenshot`, `pdf`: If screenshots/PDF requested.
- `media`, `links`: Information about discovered images/links.
@@ -308,4 +308,4 @@ asyncio.run(main())
- If you used `AsyncWebCrawler(browser_type="chromium", css_selector="...")`, move browser settings to `BrowserConfig(...)` and content/crawl logic to `CrawlerRunConfig(...)`.
-This modular approach ensures your code is **clean**, **scalable**, and **easy to maintain**.βFor any advanced or rarely used parameters, see the [BrowserConfig docs](../api/parameters.md).
\ No newline at end of file
+This modular approach ensures your code is **clean**, **scalable**, and **easy to maintain**.βFor any advanced or rarely used parameters, see the [BrowserConfig docs](../api/parameters.md).
diff --git a/docs/md_v2/api/crawl-result.md b/docs/md_v2/api/crawl-result.md
index a27a87d24..8890a7043 100644
--- a/docs/md_v2/api/crawl-result.md
+++ b/docs/md_v2/api/crawl-result.md
@@ -24,6 +24,7 @@ class CrawlResult(BaseModel):
session_id: Optional[str] = None
response_headers: Optional[dict] = None
status_code: Optional[int] = None
+ redirected_status_code: Optional[int] = None
ssl_certificate: Optional[SSLCertificate] = None
dispatch_result: Optional[DispatchResult] = None
...
@@ -50,15 +51,23 @@ if not result.success:
print(f"Crawl failed: {result.error_message}")
```
-### 1.3 **`status_code`** *(Optional[int])*
-**What**: The page's HTTP status code (e.g., 200, 404).
+### 1.3 **`status_code`** *(Optional[int])*
+**What**: The page's HTTP status code (e.g., 200, 404). When the page was reached via redirect, this is the status code of the **first** response in the redirect chain (e.g., 301 or 302).
**Usage**:
```python
if result.status_code == 404:
print("Page not found!")
```
-### 1.4 **`error_message`** *(Optional[str])*
+### 1.4 **`redirected_status_code`** *(Optional[int])*
+**What**: The HTTP status code of the **final** redirect destination. For a 302β200 redirect, `status_code` is 302 and `redirected_status_code` is 200. `None` for non-HTTP requests (raw HTML, local files).
+**Usage**:
+```python
+if result.status_code in (301, 302) and result.redirected_status_code == 200:
+ print(f"Redirected to {result.redirected_url} (OK)")
+```
+
+### 1.5 **`error_message`** *(Optional[str])*
**What**: If `success=False`, a textual description of the failure.
**Usage**:
```python
@@ -401,8 +410,8 @@ async def handle_result(result: CrawlResult):
## 9. Key Points & Future
1.β**Deprecated legacy properties of CrawlResult**
- - `markdown_v2` - Deprecated in v0.5. Just use `markdown`. It holds the `MarkdownGenerationResult` now!
- - `fit_markdown` and `fit_html` - Deprecated in v0.5. They can now be accessed via `MarkdownGenerationResult` in `result.markdown`. eg: `result.markdown.fit_markdown` and `result.markdown.fit_html`
+ - `markdown_v2` - Removed in v0.5 and now raises `AttributeError`. Use `result.markdown` instead.
+ - `fit_markdown` and `fit_html` - No longer top-level properties in v0.5. Use `result.markdown.fit_markdown` and `result.markdown.fit_html`.
2.β**Fit Content**
- **`fit_markdown`** and **`fit_html`** appear in MarkdownGenerationResult, only if you used a content filter (like **PruningContentFilter** or **BM25ContentFilter**) inside your **MarkdownGenerationStrategy** or set them directly.
@@ -419,4 +428,4 @@ async def handle_result(result: CrawlResult):
- If `success=False`, check `error_message` (e.g., timeouts, invalid URLs).
- `status_code` might be `None` if we failed before an HTTP response.
-Use **`CrawlResult`** to glean all final outputs and feed them into your data pipelines, AI models, or archives. With the synergy of a properly configured **BrowserConfig** and **CrawlerRunConfig**, the crawler can produce robust, structured results here in **`CrawlResult`**.
\ No newline at end of file
+Use **`CrawlResult`** to glean all final outputs and feed them into your data pipelines, AI models, or archives. With the synergy of a properly configured **BrowserConfig** and **CrawlerRunConfig**, the crawler can produce robust, structured results here in **`CrawlResult`**.
diff --git a/docs/md_v2/api/parameters.md b/docs/md_v2/api/parameters.md
index 9d9075166..568e14c30 100644
--- a/docs/md_v2/api/parameters.md
+++ b/docs/md_v2/api/parameters.md
@@ -10,7 +10,7 @@ browser_cfg = BrowserConfig(
headless=True,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@proxy:8080",
+ proxy_config="http://user:pass@proxy:8080",
user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36",
)
```
@@ -29,6 +29,7 @@ browser_cfg = BrowserConfig(
| **`viewport_width`** | `int` (default: `1080`) | Initial page width (in px). Useful for testing responsive layouts. |
| **`viewport_height`** | `int` (default: `600`) | Initial page height (in px). |
| **`viewport`** | `dict` (default: `None`) | Viewport dimensions dict. If set, overrides `viewport_width` and `viewport_height`. |
+| **`device_scale_factor`** | `float` (default: `1.0`) | Device pixel ratio for rendering. Use `2.0` for Retina-quality screenshots. Higher values produce larger images and use more memory. |
| **`proxy`** | `str` (deprecated) | Deprecated. Use `proxy_config` instead. If set, it will be auto-converted internally. |
| **`proxy_config`** | `ProxyConfig or dict` (default: `None`)| For advanced or multi-proxy needs, specify `ProxyConfig` object or dict like `{"server": "...", "username": "...", "password": "..."}`. |
| **`use_persistent_context`** | `bool` (default: `False`) | If `True`, uses a **persistent** browser context (keep cookies, sessions across runs). Also sets `use_managed_browser=True`. |
@@ -48,6 +49,8 @@ browser_cfg = BrowserConfig(
| **`user_agent_generator_config`** | `dict` (default: `{}`) | Configuration dict for user agent generation when `user_agent_mode="random"`. |
| **`text_mode`** | `bool` (default: `False`) | If `True`, tries to disable images/other heavy content for speed. |
| **`light_mode`** | `bool` (default: `False`) | Disables some background features for performance gains. |
+| **`avoid_ads`** | `bool` (default: `False`) | If `True`, blocks requests to common ad/tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level. |
+| **`avoid_css`** | `bool` (default: `False`) | If `True`, blocks loading of CSS files (`.css`, `.less`, `.scss`, `.sass`) for faster, leaner crawls when only text content is needed. |
| **`extra_args`** | `list` (default: `[]`) | Additional flags for the underlying browser process, e.g. `["--disable-extensions"]`. |
| **`enable_stealth`** | `bool` (default: `False`) | Enable playwright-stealth mode to bypass bot detection. Cannot be used with `browser_mode="builtin"`. |
@@ -108,8 +111,10 @@ We group them by category.
| **`timezone_id`** | `str or None` (None) | Browser's timezone (e.g., "America/New_York", "Europe/Paris"). |
| **`geolocation`** | `GeolocationConfig or None` (None) | GPS coordinates configuration. Use `GeolocationConfig(latitude=..., longitude=..., accuracy=...)`. |
| **`fetch_ssl_certificate`** | `bool` (False) | If `True`, fetches and includes SSL certificate information in the result. |
-| **`proxy_config`** | `ProxyConfig or dict or None` (None) | Proxy configuration for this specific crawl. Can override browser-level proxy settings. |
+| **`proxy_config`** | `ProxyConfig`, `list[ProxyConfig]`, or `None` (None) | Proxy configuration for this specific crawl. Pass a single proxy or an ordered list of proxies to try. See [Anti-Bot & Fallback](../advanced/anti-bot-and-fallback.md). |
| **`proxy_rotation_strategy`** | `ProxyRotationStrategy` (None) | Strategy for rotating proxies during crawl operations. |
+| **`max_retries`** | `int` (0) | Number of retry rounds when anti-bot blocking is detected. Each round tries all proxies in `proxy_config`. |
+| **`fallback_fetch_function`**| `async (str) -> str or None` (None) | Async function called as last resort after all retries are exhausted. Takes URL, returns raw HTML. See [Anti-Bot & Fallback](../advanced/anti-bot-and-fallback.md). |
---
@@ -149,15 +154,18 @@ Use these for controlling whether you read or write from a local content cache.
| **Parameter** | **Type / Default** | **What It Does** |
|----------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
-| **`js_code`** | `str or list[str]` (None) | JavaScript to run after load. E.g. `"document.querySelector('button')?.click();"`. |
+| **`js_code`** | `str or list[str]` (None) | JavaScript to run **after** `wait_for` and `delay_before_return_html`, on the fully-loaded page. E.g. `"document.querySelector('button')?.click();"`. |
+| **`js_code_before_wait`** | `str or list[str]` (None) | JavaScript to run **before** `wait_for`. Use for triggering loading that `wait_for` then checks (e.g. clicking a tab, then waiting for its content). |
| **`c4a_script`** | `str or list[str]` (None) | C4A script that compiles to JavaScript. Alternative to writing raw JS. |
| **`js_only`** | `bool` (False) | If `True`, indicates we're reusing an existing session and only applying JS. No full reload. |
| **`ignore_body_visibility`** | `bool` (True) | Skip checking if `<body>` is visible. Usually best to keep `True`. |
| **`scan_full_page`** | `bool` (False) | If `True`, auto-scroll the page to load dynamic content (infinite scroll). |
-| **`scroll_delay`** | `float` (0.2) | Delay between scroll steps if `scan_full_page=True`. |
+| **`scroll_delay`** | `float` (0.2) | Delay between scroll steps when scanning the full page (`scan_full_page=True`) or capturing full-page screenshots. |
| **`max_scroll_steps`** | `int or None` (None) | Maximum number of scroll steps during full page scan. If None, scrolls until entire page is loaded. |
| **`process_iframes`** | `bool` (False) | Inlines iframe content for single-page extraction. |
+| **`flatten_shadow_dom`** | `bool` (False) | Flattens Shadow DOM content into the light DOM before HTML capture. Resolves slots, strips shadow-scoped styles, and force-opens closed shadow roots. Essential for sites built with Web Components (Stencil, Lit, Shoelace, etc.). |
| **`remove_overlay_elements`** | `bool` (False) | Removes potential modals/popups blocking the main content. |
+| **`remove_consent_popups`** | `bool` (False) | Removes GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, TrustArc, Quantcast, Didomi, Sourcepoint, FundingChoices, etc.). Tries clicking "Accept All" first, then falls back to DOM removal. |
| **`simulate_user`** | `bool` (False) | Simulate user interactions (mouse movements) to avoid bot detection. |
| **`override_navigator`** | `bool` (False) | Override `navigator` properties in JS for stealth. |
| **`magic`** | `bool` (False) | Automatic handling of popups/consent banners. Experimental. |
@@ -174,6 +182,7 @@ If your page is a single-page app with repeated JS updates, set `js_only=True` i
| **`screenshot`** | `bool` (False) | Capture a screenshot (base64) in `result.screenshot`. |
| **`screenshot_wait_for`** | `float or None` | Extra wait time before the screenshot. |
| **`screenshot_height_threshold`** | `int` (~20000) | If the page is taller than this, alternate screenshot strategies are used. |
+| **`force_viewport_screenshot`** | `bool` (False) | If `True`, always captures a viewport-only screenshot regardless of page height. Faster and smaller than full-page screenshots. |
| **`pdf`** | `bool` (False) | If `True`, returns a PDF in `result.pdf`. |
| **`capture_mhtml`** | `bool` (False) | If `True`, captures an MHTML snapshot of the page in `result.mhtml`. MHTML includes all page resources (CSS, images, etc.) in a single file. |
| **`image_description_min_word_threshold`** | `int` (~50) | Minimum words for an image's alt text or description to be considered valid. |
@@ -367,6 +376,55 @@ no_cache_config = base_config.clone(
The `clone()` method is particularly useful when you need slightly different configurations for different use cases, without modifying the original config.
+### Class-Level Defaults (`set_defaults` / `get_defaults` / `reset_defaults`)
+
+Both config classes support class-level default overrides. When deploying in a server or cloud context, this eliminates the need to pass the same parameters at every call site.
+
+**Resolution order:** explicit arg > class-level default > hardcoded default
+
+```python
+from crawl4ai import BrowserConfig, CrawlerRunConfig
+
+# Set once at application startup
+BrowserConfig.set_defaults(
+ cache_cdp_connection=True,
+ cdp_close_delay=0,
+ create_isolated_context=True,
+)
+CrawlerRunConfig.set_defaults(verbose=False)
+
+# All new instances inherit the class defaults
+cfg1 = BrowserConfig(cdp_url="ws://localhost:9222")
+# β cache_cdp_connection=True, cdp_close_delay=0
+
+cfg2 = BrowserConfig(cdp_url="ws://localhost:9222", cache_cdp_connection=False)
+# β cache_cdp_connection=False (explicit value wins)
+
+# Inspect current defaults
+BrowserConfig.get_defaults()
+# β {"cache_cdp_connection": True, "cdp_close_delay": 0, "create_isolated_context": True}
+
+# Remove a single default
+BrowserConfig.reset_defaults("cdp_close_delay")
+
+# Remove all defaults
+BrowserConfig.reset_defaults()
+```
+
+**API Reference:**
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `set_defaults` | `set_defaults(**kwargs)` | Set class-level defaults for new instances. Raises `ValueError` if any key is not a valid `__init__` parameter. |
+| `get_defaults` | `get_defaults() β dict` | Return a deep copy of the current class-level defaults. |
+| `reset_defaults` | `reset_defaults(*names)` | With no args, clears all defaults. With args, removes only the named defaults. |
+
+**Notes:**
+- Defaults are independent per class β `BrowserConfig.set_defaults()` has no effect on `CrawlerRunConfig`.
+- Mutable values (lists, dicts) are deep-copied on storage and on each instance creation, so instances do not share objects.
+- `clone()`, `dump()`/`load()`, and `from_kwargs()` all work correctly with class defaults β serialized data is self-contained and independent of the current class defaults.
+- Defaults are stored in memory for the lifetime of the process. They are not persisted to disk.
+
## 2.3 Example Usage
```python
@@ -379,7 +437,7 @@ async def main():
headless=False,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@myproxy:8080",
+ proxy_config="http://user:pass@myproxy:8080",
text_mode=True
)
@@ -432,6 +490,8 @@ LLMConfig is useful to pass LLM provider config to strategies and functions that
2. LLMContentFilter
3. JsonCssExtractionStrategy.generate_schema
4. JsonXPathExtractionStrategy.generate_schema
+5. AdaptiveConfig.embedding_llm_config (embedding model for adaptive crawling)
+6. AdaptiveConfig.query_llm_config (chat completion model for query expansion in adaptive crawling)
## 3.1 Parameters
| **Parameter** | **Type / Default** | **What It Does** |
@@ -459,7 +519,7 @@ llm_config = LLMConfig(
- **Use** `BrowserConfig` for **global** browser settings: engine, headless, proxy, user agent.
- **Use** `CrawlerRunConfig` for each crawlβs **context**: how to filter content, handle caching, wait for dynamic elements, or run JS.
- **Pass** both configs to `AsyncWebCrawler` (the `BrowserConfig`) and then to `arun()` (the `CrawlerRunConfig`).
-- **Use** `LLMConfig` for LLM provider configurations that can be used across all extraction, filtering, schema generation tasks. Can be used in - `LLMExtractionStrategy`, `LLMContentFilter`, `JsonCssExtractionStrategy.generate_schema` & `JsonXPathExtractionStrategy.generate_schema`
+- **Use** `LLMConfig` for LLM provider configurations that can be used across all extraction, filtering, schema generation, and adaptive crawling tasks. Can be used in - `LLMExtractionStrategy`, `LLMContentFilter`, `JsonCssExtractionStrategy.generate_schema`, `JsonXPathExtractionStrategy.generate_schema`, and `AdaptiveConfig` (`embedding_llm_config` / `query_llm_config`)
```python
# Create a modified copy with the clone() method
@@ -467,4 +527,8 @@ stream_cfg = run_cfg.clone(
stream=True,
cache_mode=CacheMode.BYPASS
)
+
+# Or set project-wide defaults once at startup
+BrowserConfig.set_defaults(headless=True, text_mode=True)
+CrawlerRunConfig.set_defaults(cache_mode=CacheMode.BYPASS)
```
diff --git a/docs/md_v2/api/strategies.md b/docs/md_v2/api/strategies.md
index 07649ee95..8a40812ab 100644
--- a/docs/md_v2/api/strategies.md
+++ b/docs/md_v2/api/strategies.md
@@ -120,7 +120,8 @@ schema = {
"attribute": str, # For type="attribute"
"pattern": str, # For type="regex"
"transform": str, # Optional: "lowercase", "uppercase", "strip"
- "default": Any # Default value if extraction fails
+ "default": Any, # Default value if extraction fails
+ "source": str, # Optional: navigate to sibling first, e.g. "+ tr"
}
]
}
@@ -216,7 +217,7 @@ from crawl4ai import LLMConfig
async with AsyncWebCrawler() as crawler:
# Get sample HTML first
sample_result = await crawler.arun("https://example.com/products")
- html = sample_result.fit_html
+ html = sample_result.markdown.fit_html
# Generate regex pattern once
pattern = RegexExtractionStrategy.generate_pattern(
diff --git a/docs/md_v2/assets/llm.txt/txt/config_objects.txt b/docs/md_v2/assets/llm.txt/txt/config_objects.txt
index e84d9b906..ae1206fd2 100644
--- a/docs/md_v2/assets/llm.txt/txt/config_objects.txt
+++ b/docs/md_v2/assets/llm.txt/txt/config_objects.txt
@@ -19,7 +19,7 @@ browser_config = BrowserConfig(
# Advanced browser setup with proxy and persistence
browser_config = BrowserConfig(
headless=False,
- proxy="http://user:pass@proxy:8080",
+ proxy_config="http://user:pass@proxy:8080",
use_persistent_context=True,
user_data_dir="./browser_data",
cookies=[
@@ -442,7 +442,7 @@ config = CrawlerRunConfig(
# Hierarchical content selection
config = CrawlerRunConfig(
- css_selector=["#main-content", ".article-wrapper"], # Top-level extraction
+ css_selector="#main-content, .article-wrapper", # Top-level extraction
target_elements=[ # Subset for processing
".article-title",
".article-body",
diff --git a/docs/md_v2/assets/llm.txt/txt/docker.txt b/docs/md_v2/assets/llm.txt/txt/docker.txt
index 65372fb3d..daa333b2e 100644
--- a/docs/md_v2/assets/llm.txt/txt/docker.txt
+++ b/docs/md_v2/assets/llm.txt/txt/docker.txt
@@ -484,7 +484,7 @@ browser_config = BrowserConfig(
headless=True,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@proxy:8080"
+ proxy_config="http://user:pass@proxy:8080"
)
# Get JSON structure
diff --git a/docs/md_v2/assets/llm.txt/txt/extraction-llm.txt b/docs/md_v2/assets/llm.txt/txt/extraction-llm.txt
index a9f5c18c9..3cc8249dd 100644
--- a/docs/md_v2/assets/llm.txt/txt/extraction-llm.txt
+++ b/docs/md_v2/assets/llm.txt/txt/extraction-llm.txt
@@ -614,7 +614,7 @@ async def batch_llm_extraction():
if result.success:
contents.append({
"url": url,
- "content": result.fit_markdown[:2000] # Limit content
+ "content": result.markdown.fit_markdown[:2000] # Limit content
})
# Process in batches (reduce LLM calls)
diff --git a/docs/md_v2/assets/llm.txt/txt/llms-full-v0.1.1.txt b/docs/md_v2/assets/llm.txt/txt/llms-full-v0.1.1.txt
index 12dad6033..951e71d1c 100644
--- a/docs/md_v2/assets/llm.txt/txt/llms-full-v0.1.1.txt
+++ b/docs/md_v2/assets/llm.txt/txt/llms-full-v0.1.1.txt
@@ -643,7 +643,7 @@ browser_config = BrowserConfig(
# Advanced browser setup with proxy and persistence
browser_config = BrowserConfig(
headless=False,
- proxy="http://user:pass@proxy:8080",
+ proxy_config="http://user:pass@proxy:8080",
use_persistent_context=True,
user_data_dir="./browser_data",
cookies=[
@@ -1066,7 +1066,7 @@ config = CrawlerRunConfig(
# Hierarchical content selection
config = CrawlerRunConfig(
- css_selector=["#main-content", ".article-wrapper"], # Top-level extraction
+ css_selector="#main-content, .article-wrapper", # Top-level extraction
target_elements=[ # Subset for processing
".article-title",
".article-body",
@@ -2412,7 +2412,7 @@ async def batch_llm_extraction():
if result.success:
contents.append({
"url": url,
- "content": result.fit_markdown[:2000] # Limit content
+ "content": result.markdown.fit_markdown[:2000] # Limit content
})
# Process in batches (reduce LLM calls)
@@ -4719,7 +4719,7 @@ browser_config = BrowserConfig(
headless=True,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@proxy:8080"
+ proxy_config="http://user:pass@proxy:8080"
)
# Get JSON structure
diff --git a/docs/md_v2/assets/llm.txt/txt/llms-full.txt b/docs/md_v2/assets/llm.txt/txt/llms-full.txt
index 12dad6033..951e71d1c 100644
--- a/docs/md_v2/assets/llm.txt/txt/llms-full.txt
+++ b/docs/md_v2/assets/llm.txt/txt/llms-full.txt
@@ -643,7 +643,7 @@ browser_config = BrowserConfig(
# Advanced browser setup with proxy and persistence
browser_config = BrowserConfig(
headless=False,
- proxy="http://user:pass@proxy:8080",
+ proxy_config="http://user:pass@proxy:8080",
use_persistent_context=True,
user_data_dir="./browser_data",
cookies=[
@@ -1066,7 +1066,7 @@ config = CrawlerRunConfig(
# Hierarchical content selection
config = CrawlerRunConfig(
- css_selector=["#main-content", ".article-wrapper"], # Top-level extraction
+ css_selector="#main-content, .article-wrapper", # Top-level extraction
target_elements=[ # Subset for processing
".article-title",
".article-body",
@@ -2412,7 +2412,7 @@ async def batch_llm_extraction():
if result.success:
contents.append({
"url": url,
- "content": result.fit_markdown[:2000] # Limit content
+ "content": result.markdown.fit_markdown[:2000] # Limit content
})
# Process in batches (reduce LLM calls)
@@ -4719,7 +4719,7 @@ browser_config = BrowserConfig(
headless=True,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@proxy:8080"
+ proxy_config="http://user:pass@proxy:8080"
)
# Get JSON structure
diff --git a/docs/md_v2/blog/index.md b/docs/md_v2/blog/index.md
index 977c62170..b7399d69a 100644
--- a/docs/md_v2/blog/index.md
+++ b/docs/md_v2/blog/index.md
@@ -20,6 +20,21 @@ Ever wondered why your AI coding assistant struggles with your library despite c
## Latest Release
+### [Crawl4AI v0.8.5 β Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes](../blog/release-v0.8.5.md)
+*March 2026*
+
+Crawl4AI v0.8.5 is the biggest release since v0.8.0, bringing automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.
+
+Key highlights:
+- **π‘οΈ Anti-Bot Detection & Proxy Escalation**: 3-tier detection with automatic retry, proxy chain, and fallback
+- **π Shadow DOM Flattening**: Extract content hidden inside shadow DOM components
+- **π Deep Crawl Cancellation**: Stop long crawls gracefully with `cancel()` or `should_cancel` callback
+- **π Critical Security Fixes**: RCE via deserialization patched, Redis CVE-2025-49844 fixed
+
+[Read full release notes β](../blog/release-v0.8.5.md)
+
+## Recent Releases
+
### [Crawl4AI v0.8.0 β Crash Recovery & Prefetch Mode](../blog/release-v0.8.0.md)
*January 2026*
@@ -32,8 +47,6 @@ Key highlights:
[Read full release notes β](../blog/release-v0.8.0.md)
-## Recent Releases
-
### [Crawl4AI v0.7.8 β Stability & Bug Fix Release](../blog/release-v0.7.8.md)
*December 2025*
@@ -46,37 +59,14 @@ Key highlights:
[Read full release notes β](../blog/release-v0.7.8.md)
-### [Crawl4AI v0.7.7 β The Self-Hosting & Monitoring Update](../blog/release-v0.7.7.md)
-*November 14, 2025*
-
-Crawl4AI v0.7.7 transforms Docker into a complete self-hosting platform with enterprise-grade real-time monitoring, comprehensive observability, and full operational control.
-
-Key highlights:
-- **π Real-time Monitoring Dashboard**: Interactive web UI with live system metrics
-- **π Comprehensive Monitor API**: Complete REST API for programmatic access
-- **β‘ WebSocket Streaming**: Real-time updates every 2 seconds
-- **π₯ Smart Browser Pool**: 3-tier architecture with automatic promotion and cleanup
-
-[Read full release notes β](../blog/release-v0.7.7.md)
-
-### [Crawl4AI v0.7.6 β The Webhook Infrastructure Update](../blog/release-v0.7.6.md)
-*October 22, 2025*
-
-Crawl4AI v0.7.6 introduces comprehensive webhook support for the Docker job queue API, bringing real-time notifications to both crawling and LLM extraction workflows.
-
-Key highlights:
-- **πͺ Complete Webhook Support**: Real-time notifications for both `/crawl/job` and `/llm/job` endpoints
-- **π Reliable Delivery**: Exponential backoff retry mechanism
-- **π Custom Authentication**: Add custom headers for webhook authentication
-
-[Read full release notes β](../blog/release-v0.7.6.md)
-
---
## Older Releases
| Version | Date | Highlights |
|---------|------|------------|
+| [v0.7.7](../blog/release-v0.7.7.md) | November 2025 | Self-hosting platform, real-time monitoring, smart browser pool |
+| [v0.7.6](../blog/release-v0.7.6.md) | October 2025 | Webhook infrastructure, reliable delivery, custom auth |
| [v0.7.5](../blog/release-v0.7.5.md) | September 2025 | Docker Hooks System, enhanced LLM integration, HTTPS preservation |
| [v0.7.4](../blog/release-v0.7.4.md) | August 2025 | LLM-powered table extraction, performance improvements |
| [v0.7.3](../blog/release-v0.7.3.md) | July 2025 | Undetected browser, multi-URL config, memory monitoring |
diff --git a/docs/md_v2/blog/releases/v0.8.5.md b/docs/md_v2/blog/releases/v0.8.5.md
new file mode 100644
index 000000000..68fc79691
--- /dev/null
+++ b/docs/md_v2/blog/releases/v0.8.5.md
@@ -0,0 +1,384 @@
+# Crawl4AI v0.8.5: Anti-Bot, Shadow DOM & 60+ Bug Fixes
+
+*March 2026 β’ 10 min read*
+
+---
+
+I'm releasing Crawl4AI v0.8.5βour biggest release since v0.8.0. This update brings automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes from both our team and the community. If you're running crawls at scale or dealing with protected sites, this one's for you.
+
+## What's New at a Glance
+
+- **Anti-Bot Detection & Proxy Escalation**: 3-tier detection with automatic retry, proxy chain, and fallback
+- **Shadow DOM Flattening**: Extract content hidden inside shadow DOM components
+- **Deep Crawl Cancellation**: Stop long crawls gracefully with `cancel()` or `should_cancel` callback
+- **Config Defaults API**: Set once, apply everywhere with `set_defaults()` / `get_defaults()` / `reset_defaults()`
+- **Source/Sibling Selector**: Extract data spanning sibling elements in JSON extraction schemas
+- **Consent Popup Removal**: Auto-dismiss cookie banners from 40+ CMP platforms
+- **Resource Filtering**: Block ads and CSS at the network level with `avoid_ads` / `avoid_css`
+- **Browser Recycling**: Memory-saving mode and automatic browser restart for long sessions
+- **GFM Table Compliance**: Proper `| col1 | col2 |` pipe delimiters in markdown output
+- **60+ Bug Fixes**: Security patches, browser stability, extraction accuracy, and more
+
+---
+
+## New Features
+
+### 1. Anti-Bot Detection, Retry & Fallback
+
+This is the headline feature. Crawl4AI now automatically detects when a page is blocked by anti-bot protection and takes actionβretrying with different proxies or falling back to an alternative fetch method.
+
+The detection uses three tiers:
+- **Tier 1**: Known vendor patterns (Cloudflare, Akamai, DataDome, PerimeterX, etc.)
+- **Tier 2**: Generic block indicators on small pages
+- **Tier 3**: Structural integrity checks (empty shells, script-heavy pages with no content)
+
+```python
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+from crawl4ai.async_configs import ProxyConfig
+
+config = CrawlerRunConfig(
+ # Try direct first, then proxy on bot detection
+ proxy_config=[
+ ProxyConfig.DIRECT,
+ ProxyConfig(server="http://my-proxy:8080"),
+ ],
+ max_retries=2,
+ # Optional: fallback when all proxies fail
+ fallback_fetch_function=my_web_unlocker_function,
+)
+
+async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun("https://protected-site.com", config=config)
+
+ # Check what happened
+ stats = result.crawl_stats
+ print(f"Resolved by: {stats['resolved_by']}") # "direct", "proxy", or "fallback_fetch"
+ print(f"Proxies tried: {len(stats['proxies_used'])}")
+```
+
+The system errs on the side of cautionβfalse positives are cheap (the fallback rescues them), but false negatives mean garbage results. After 5 iterations of real-world testing, it handles everything from Cloudflare challenges to Reddit's 180KB SPA block pages.
+
+### 2. Shadow DOM Flattening
+
+Web components with shadow DOM hide their content from regular DOM traversal. The new `flatten_shadow_dom` option serializes shadow DOM content into the light DOM before extraction.
+
+```python
+config = CrawlerRunConfig(flatten_shadow_dom=True)
+
+async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun("https://some-web-component-site.com", config=config)
+ # Shadow DOM content is now visible in result.html, cleaned_html, and markdown
+```
+
+The implementation patches `attachShadow` to force-open closed shadow roots, recursively resolves `<slot>` projections, and strips only shadow-scoped `<style>` tags. It also reorders the JS execution pipelineβ`js_code` now runs after `wait_for` + `delay_before_return_html` so your scripts operate on the fully-hydrated page. If you need JS to run before waiting, use the new `js_code_before_wait` parameter.
+
+### 3. Deep Crawl Cancellation
+
+All deep crawl strategies (BFS, DFS, BestFirst) now support graceful cancellation:
+
+```python
+from crawl4ai.deep_crawling import DFSDeepCrawlStrategy
+
+pages_found = 0
+
+def should_stop():
+ return pages_found >= 50 # Stop after finding enough pages
+
+async def on_state(state):
+ nonlocal pages_found
+ pages_found = state["pages_crawled"]
+
+strategy = DFSDeepCrawlStrategy(
+ max_depth=3,
+ max_pages=1000,
+ should_cancel=should_stop, # Sync or async callback
+ on_state_change=on_state,
+)
+
+config = CrawlerRunConfig(deep_crawl_strategy=strategy)
+
+async with AsyncWebCrawler() as crawler:
+ results = await crawler.arun("https://example.com", config=config)
+ print(f"Cancelled: {strategy.cancelled}")
+```
+
+You can also call `strategy.cancel()` directly from another thread or coroutine.
+
+### 4. Config Defaults API
+
+Tired of repeating the same parameters? Set defaults once and they apply to every new instance:
+
+```python
+from crawl4ai import BrowserConfig, CrawlerRunConfig
+
+# Set organization-wide defaults
+BrowserConfig.set_defaults(headless=True, text_mode=True)
+CrawlerRunConfig.set_defaults(verbose=False, remove_consent_popups=True)
+
+# All new instances inherit defaults
+bc = BrowserConfig() # headless=True, text_mode=True
+rc = CrawlerRunConfig() # verbose=False, remove_consent_popups=True
+
+# Explicit params always override
+bc2 = BrowserConfig(text_mode=False) # text_mode=False, headless still True
+
+# Inspect and reset
+print(BrowserConfig.get_defaults()) # {"headless": True, "text_mode": True}
+BrowserConfig.reset_defaults() # Back to normal
+```
+
+### 5. Source/Sibling Selector in JSON Extraction
+
+Many sites split a single item's data across sibling elements (think Hacker News, where title and score are in separate `<tr>` rows). The new `"source"` field navigates to a sibling before extracting:
+
+```python
+from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+schema = {
+ "name": "HackerNewsItems",
+ "baseSelector": "tr.athing",
+ "fields": [
+ {"name": "title", "selector": ".titleline > a", "type": "text"},
+ {"name": "link", "selector": ".titleline > a", "type": "attribute", "attribute": "href"},
+ # Navigate to the NEXT sibling <tr> to get the score
+ {"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
+ {"name": "author", "selector": ".hnuser", "type": "text", "source": "+ tr"},
+ ]
+}
+
+strategy = JsonCssExtractionStrategy(schema=schema)
+```
+
+Works in both `JsonCssExtractionStrategy` and `JsonXPathExtractionStrategy`. Falls back gracefully when siblings don't exist.
+
+### 6. Consent Popup Removal
+
+A single flag auto-dismisses cookie consent banners from 40+ CMP platforms:
+
+```python
+config = CrawlerRunConfig(remove_consent_popups=True)
+```
+
+Covers OneTrust, Cookiebot, Didomi, Quantcast, Sourcepoint, Google FundingChoices, TrustArc, ConsentManager, Osano, Iubenda, Complianz, LiveRamp, CookieYes, Klaro, Termly, and many more.
+
+### 7. Resource Filtering: avoid_ads / avoid_css
+
+Block ad trackers and CSS resources at the network level for faster, leaner crawls:
+
+```python
+config = BrowserConfig(
+ avoid_ads=True, # Blocks doubleclick, google-analytics, etc.
+ avoid_css=True, # Blocks .css, .less, .scss resources
+)
+```
+
+### 8. Browser Recycling & Memory-Saving Mode
+
+For long-running crawl sessions:
+
+```python
+config = BrowserConfig(
+ memory_saving_mode=True, # Aggressive cache/V8 heap flags
+ max_pages_before_recycle=100, # Auto-restart browser after N pages
+)
+```
+
+This prevents memory leaks during sustained crawling. The recycling uses a version-based approach that's safe under concurrent loadβwe fixed three separate deadlock bugs to get this right.
+
+### 9. GFM Table Compliance
+
+Tables in markdown output now have proper GitHub-Flavored Markdown pipe delimiters:
+
+**Before (v0.8.0)**:
+```
+Name | Age | City
+---|---|---
+Alice | 30 | NYC
+```
+
+**After (v0.8.5)**:
+```
+| Name | Age | City |
+| --- | --- | --- |
+| Alice | 30 | NYC |
+```
+
+---
+
+## Minor Features
+
+- **`query_llm_config`**: Separate LLM config for adaptive crawler query expansion (#1682)
+- **`force_viewport_screenshot`**: Screenshot only the viewport, not the full page
+- **`device_scale_factor`**: Configurable screenshot DPI via BrowserConfig (#1463)
+- **`redirected_status_code`**: Now available on CrawlResult (#1435)
+- **`wait_for_images`**: Wait for images to load before taking screenshots (#1792)
+- **`score_threshold`**: Filter low-quality URLs in BestFirstCrawlingStrategy (#1804)
+- **`link_preview_timeout`**: Configurable timeout in AdaptiveConfig (#1793)
+- **`--json-ensure-ascii`**: CLI flag for Unicode preservation in JSON output (#1668)
+- **`type-list` pipeline**: Chained extraction like `["attribute", "regex"]` in JsonCssExtractionStrategy (#1290)
+
+---
+
+## Security Fixes
+
+### Critical: RCE via Deserialization in Docker /crawl Endpoint
+
+**Severity**: CRITICAL
+**Affected**: Docker API deployment (v0.8.0 and earlier)
+
+The `/crawl` endpoint's deserialization logic used `eval()` for certain object types. I removed this entirely and added an allowlist (`ALLOWED_DESERIALIZE_TYPES`) so only known config classes can be instantiated.
+
+### Critical: Redis CVE-2025-49844 (CVSS 10.0)
+
+**Affected**: Docker deployments using Redis
+
+Upgraded Redis to 7.2.7 which patches the Lua use-after-free vulnerability.
+
+### Additional Security
+
+- **XSS prevention**: Use DOMParser instead of innerHTML in iframe processing (#1796)
+- **API token enforcement**: `/token` endpoint now requires `api_token` when configured (#1795)
+- **Stealth improvements**: `sec-ch-ua` synced with User-Agent, WebGL kept alive in stealth mode
+
+---
+
+## Bug Fixes
+
+### Browser & Page Management
+
+- Fix page reuse race condition when `create_isolated_context=False`
+- Fix browser context memory leak β signature shrink + LRU eviction (#943)
+- Fix cascading context crash from duplicate `add_init_script` (#1768)
+- Fix `simulate_user` destroying page content via ArrowDown keypress
+- Fix browser recycling deadlock under sustained concurrent load (#1640)
+- Fix Docker monitor LOCK contention causing pod deadlock (#1754)
+
+### Proxy & Network
+
+- Fix proxy auth `ERR_INVALID_AUTH_CREDENTIALS` (#1281)
+- Fix proxy auth for persistent browser contexts
+- Fix proxy escalation not re-raising on first exception when chain has alternatives
+- Fix fallback fetch: run when all proxies crash, skip re-check, never return None
+
+### Deep Crawling
+
+- Fix `can_process_url()` to receive normalized URL
+- Fix `total_score` not calculated for links that fail head extraction
+- Fix `FilterChain.add_filter` AttributeError on tuple immutability
+- Fix URL Seeder forcing Common Crawl index for sitemaps (#1746)
+- Fix `is_external_url` port comparison (#1783)
+- Prevent AdaptiveCrawler from crawling external domains (#1805)
+
+### Extraction & Content
+
+- Fix `<base>` tag ignored in html2text relative link resolution (#1721)
+- Fix script tag removal losing adjacent text in `cleaned_html` (#1364)
+- Preserve `class` and `id` attributes in `cleaned_html` (#1782)
+- Fix nested brackets/parentheses in LINK_PATTERN regex (#1790)
+- Strip markdown fences in `force_json_response` path for LLM extraction
+- Guard against None LLM content, propagate `finish_reason` (#1788)
+- Fix `agenerate_schema()` JSON parsing for Anthropic models
+- Fix `from_serializable_dict` ignoring plain data dicts with "type" key
+- Fix MediaItem crash on non-numeric width values like "100%" (#1635)
+- Fix BM25ContentFilter returning duplicate chunks (#1213)
+- Fix `css_selector` ignored in LXML scraping for `raw://` URLs (#1484)
+
+### CLI & Docker
+
+- Fix deep-crawl CLI outputting only the first page (#1667)
+- Fix VersionManager ignoring `CRAWL4_AI_BASE_DIRECTORY` env var (#1296)
+- Fix Docker health endpoint to use dynamic version (#1686)
+- Add explicit UTF-8 encoding to CLI file output (#1789)
+- Handle `UnicodeEncodeError` in URL seeder, strip zero-width chars (#1784)
+- Add TTL expiry for Redis task data to prevent memory growth (#1730)
+- Add Windows support for crawler monitor keyboard input (#1794)
+- Fix `scroll_delay` ignored in full-page screenshot scroller
+- Fix MCP SSE endpoint crash β mount via raw ASGI Route (#1594)
+- Fix `/llm` per-request provider override, Redis config from host/port/password (#1611, #1817)
+- Fix screenshot respects `scan_full_page=False` (#1750)
+- Fix screenshot distortion on Elementor sites (#1370)
+- Fix deep crawl timeout and `arun_many` dispatcher bypass (#1818, #1509)
+
+### Other
+
+- Replace `tf-playwright-stealth` with `playwright-stealth` (#1553)
+- Allow local embeddings by removing OpenAI fallback (#1658)
+- Include GoogleSearchCrawler `script.js` in package distribution (#1711)
+- Fix bs4 deprecation warning (`text` β `string`) (#1077)
+- Run blocking `chardet.detect` in thread executor (#1751)
+- Wire `mean_delay`/`max_range` from CrawlerRunConfig into dispatcher rate limiter (#1786)
+
+---
+
+## Tests
+
+Added a comprehensive **291-test regression suite** covering all major subsystems: core crawl, content processing, extraction strategies, deep crawling, browser management, config serialization, utilities, and edge cases.
+
+---
+
+## Breaking Changes
+
+### `cleaned_html` Now Preserves `class` and `id` Attributes
+
+If you have downstream code that parses `cleaned_html` and assumes no class/id attributes are present, this may need updating. This change enables users to do CSS-based analysis on cleaned HTML.
+
+### Docker: Redis Upgraded to 7.2.7
+
+If you pin Redis versions in your deployment, update to 7.2.7 or later.
+
+---
+
+## Upgrade Instructions
+
+### Python Package
+
+```bash
+pip install --upgrade crawl4ai
+# or
+pip install crawl4ai==0.8.5
+```
+
+### Docker
+
+```bash
+docker pull unclecode/crawl4ai:0.8.5
+
+docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.8.5
+```
+
+---
+
+## Verification
+
+Run the verification tests to confirm all features are working:
+
+```bash
+python docs/releases_review/demo_v0.8.5.py
+```
+
+This runs 13 actual tests that crawl real URLs and verify each feature end-to-end.
+
+---
+
+## Acknowledgments
+
+This release includes contributions from a large number of community members. Thank you to everyone who submitted PRs, reported issues, and provided reproduction steps. Special thanks to all contributors listed in [CONTRIBUTORS.md](../CONTRIBUTORS.md).
+
+Issues fixed: #462, #880, #943, #1031, #1077, #1183, #1213, #1251, #1281, #1290, #1296, #1308, #1354, #1364, #1370, #1374, #1424, #1435, #1463, #1484, #1487, #1489, #1494, #1503, #1509, #1512, #1520, #1553, #1594, #1601, #1606, #1611, #1622, #1635, #1640, #1658, #1666, #1667, #1668, #1671, #1682, #1686, #1711, #1715, #1716, #1721, #1730, #1731, #1746, #1750, #1751, #1754, #1758, #1762, #1768, #1770, #1776, #1782, #1783, #1784, #1786, #1788, #1789, #1790, #1792, #1793, #1794, #1795, #1796, #1797, #1801, #1803, #1804, #1805, #1815, #1817, #1818, #1824
+
+---
+
+## Support & Resources
+
+- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
+- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
+- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
+- **Twitter**: [@unclecode](https://x.com/unclecode)
+
+---
+
+**This is a massive releaseβ10 new features, critical security patches, and 60+ bug fixes. Whether you're dealing with anti-bot protection, shadow DOM sites, or just want more reliable crawls at scale, v0.8.5 has you covered. Thank you for your continued support!**
+
+**Happy crawling!**
+
+*- unclecode*
diff --git a/docs/md_v2/complete-sdk-reference.md b/docs/md_v2/complete-sdk-reference.md
index 7e6abf5c8..aa0517b2d 100644
--- a/docs/md_v2/complete-sdk-reference.md
+++ b/docs/md_v2/complete-sdk-reference.md
@@ -232,6 +232,7 @@ if __name__ == "__main__":
- Great for repetitive page structures (e.g., item listings, articles).
- No AI usage or costs.
- The crawler returns a JSON string you can parse or store.
+- For sites where data is split across sibling elements (e.g. Hacker News), use the `"source"` field key to navigate to a sibling before extracting: `{"name": "score", "selector": "span.score", "type": "text", "source": "+ tr"}`.
> Tips: You can pass raw HTML to the crawler instead of a URL. To do so, prefix the HTML with `raw://`.
## 6. Simple Data Extraction (LLM-based)
- **Open-Source Models** (e.g., `ollama/llama3.3`, `no_token`)
@@ -611,7 +612,7 @@ Each `arun()` returns a **`CrawlResult`** containing:
- `url`: Final URL (if redirected).
- `html`: Original HTML.
- `cleaned_html`: Sanitized HTML.
-- `markdown_v2`: Deprecated. Instead just use regular `markdown`
+- `markdown_v2`: Removed in v0.5. Accessing it raises `AttributeError`. Use `markdown`.
- `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies).
- `screenshot`, `pdf`: If screenshots/PDF requested.
- `media`, `links`: Information about discovered images/links.
@@ -739,10 +740,10 @@ run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS
)
```
-- `bypass_cache=True` acts like `CacheMode.BYPASS`.
-- `disable_cache=True` acts like `CacheMode.DISABLED`.
-- `no_cache_read=True` acts like `CacheMode.WRITE_ONLY`.
-- `no_cache_write=True` acts like `CacheMode.READ_ONLY`.
+- `CacheMode.BYPASS` β Skip cache entirely; always fetch fresh, write result to cache.
+- `CacheMode.DISABLED` β No caching at all; don't read or write.
+- `CacheMode.WRITE_ONLY` β Never read from cache, but write results.
+- `CacheMode.READ_ONLY` β Read from cache if available, never write.
## 3.βContent Processing & Selection
### 3.1 Text Processing
```python
@@ -1359,8 +1360,8 @@ async def handle_result(result: CrawlResult):
```
## 9. Key Points & Future
1.β**Deprecated legacy properties of CrawlResult**
- - `markdown_v2` - Deprecated in v0.5. Just use `markdown`. It holds the `MarkdownGenerationResult` now!
- - `fit_markdown` and `fit_html` - Deprecated in v0.5. They can now be accessed via `MarkdownGenerationResult` in `result.markdown`. eg: `result.markdown.fit_markdown` and `result.markdown.fit_html`
+ - `markdown_v2` - Removed in v0.5. Accessing it raises `AttributeError`. Use `markdown`.
+ - `fit_markdown` and `fit_html` - Removed as top-level `CrawlResult` properties in v0.5. Use `result.markdown.fit_markdown` and `result.markdown.fit_html`.
2.β**Fit Content**
- **`fit_markdown`** and **`fit_html`** appear in MarkdownGenerationResult, only if you used a content filter (like **PruningContentFilter** or **BM25ContentFilter**) inside your **MarkdownGenerationStrategy** or set them directly.
- If no filter is used, they remain `None`.
@@ -1401,6 +1402,8 @@ class BrowserConfig:
user_agent=None,
text_mode=False,
light_mode=False,
+ avoid_ads=False,
+ avoid_css=False,
extra_args=None,
enable_stealth=False,
# ... other advanced parameters omitted here
@@ -1439,15 +1442,19 @@ class BrowserConfig:
8. **`user_agent`**:
- Custom User-Agent string. If `None`, a default is used.
- You can also set `user_agent_mode="random"` for randomization (if you want to fight bot detection).
-9. **`text_mode`** & **`light_mode`**:
- - `text_mode=True` disables images, possibly speeding up text-only crawls.
- - `light_mode=True` turns off certain background features for performance.
-10. **`extra_args`**:
- - Additional flags for the underlying browser.
+9. **`text_mode`** & **`light_mode`**:
+ - `text_mode=True` disables images, possibly speeding up text-only crawls.
+ - `light_mode=True` turns off certain background features for performance.
+10. **`avoid_ads`** & **`avoid_css`**:
+ - `avoid_ads=True` blocks requests to common ad and tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level. Reduces network overhead and memory usage.
+ - `avoid_css=True` blocks loading of CSS files (`.css`, `.less`, `.scss`, `.sass`), useful when you only need text content and want faster, leaner crawls.
+ - Both default to `False` (opt-in). Can be combined with each other and with `text_mode`.
+11. **`extra_args`**:
+ - Additional flags for the underlying browser.
- E.g. `["--disable-extensions"]`.
-11. **`enable_stealth`**:
- - If `True`, enables stealth mode using playwright-stealth.
- - Modifies browser fingerprints to avoid basic bot detection.
+12. **`enable_stealth`**:
+ - If `True`, enables stealth mode using playwright-stealth.
+ - Modifies browser fingerprints to avoid basic bot detection.
- Default is `False`. Recommended for sites with bot protection.
### Helper Methods
Both configuration classes provide a `clone()` method to create modified copies:
@@ -1702,7 +1709,7 @@ browser_cfg = BrowserConfig(
headless=True,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@proxy:8080",
+ proxy_config="http://user:pass@proxy:8080",
user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36",
)
```
@@ -1780,13 +1787,16 @@ run_cfg = CrawlerRunConfig(
### D) **Page Interaction**
| **Parameter** | **Type / Default** | **What It Does** |
|----------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
-| **`js_code`** | `str or list[str]` (None) | JavaScript to run after load. E.g. `"document.querySelector('button')?.click();"`. |
-| **`js_only`** | `bool` (False) | If `True`, indicates weβre reusing an existing session and only applying JS. No full reload. |
+| **`js_code`** | `str or list[str]` (None) | JavaScript to run **after** `wait_for` and `delay_before_return_html`, on the fully-loaded page. E.g. `"document.querySelector('button')?.click();"`. |
+| **`js_code_before_wait`** | `str or list[str]` (None) | JavaScript to run **before** `wait_for`. Use for triggering loading that `wait_for` then checks. |
+| **`js_only`** | `bool` (False) | If `True`, indicates we're reusing an existing session and only applying JS. No full reload. |
| **`ignore_body_visibility`** | `bool` (True) | Skip checking if `<body>` is visible. Usually best to keep `True`. |
| **`scan_full_page`** | `bool` (False) | If `True`, auto-scroll the page to load dynamic content (infinite scroll). |
-| **`scroll_delay`** | `float` (0.2) | Delay between scroll steps if `scan_full_page=True`. |
+| **`scroll_delay`** | `float` (0.2) | Delay between scroll steps when scanning the full page (`scan_full_page=True`) or capturing full-page screenshots. |
| **`process_iframes`** | `bool` (False) | Inlines iframe content for single-page extraction. |
+| **`flatten_shadow_dom`** | `bool` (False) | Flattens Shadow DOM content into the light DOM before HTML capture. Resolves slots, strips shadow-scoped styles, and force-opens closed shadow roots. Essential for sites built with Web Components. |
| **`remove_overlay_elements`** | `bool` (False) | Removes potential modals/popups blocking the main content. |
+| **`remove_consent_popups`** | `bool` (False) | Removes GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, TrustArc, Quantcast, Didomi, Sourcepoint, FundingChoices, etc.). Tries clicking "Accept All" first, then falls back to DOM removal. |
| **`simulate_user`** | `bool` (False) | Simulate user interactions (mouse movements) to avoid bot detection. |
| **`override_navigator`** | `bool` (False) | Override `navigator` properties in JS for stealth. |
| **`magic`** | `bool` (False) | Automatic handling of popups/consent banners. Experimental. |
@@ -1927,7 +1937,7 @@ async def main():
headless=False,
viewport_width=1280,
viewport_height=720,
- proxy="http://user:pass@myproxy:8080",
+ proxy_config="http://user:pass@myproxy:8080",
text_mode=True
)
@@ -2811,6 +2821,46 @@ async def main():
if __name__ == "__main__":
asyncio.run(main())
```
+## 3.1 Flattening Shadow DOM
+Sites built with **Web Components** (Stencil, Lit, Shoelace, Angular Elements, etc.) render content inside Shadow DOM β an encapsulated sub-tree invisible to `page.content()`. Set `flatten_shadow_dom=True` to extract it:
+```python
+config = CrawlerRunConfig(
+ flatten_shadow_dom=True,
+ wait_until="load",
+ delay_before_return_html=3.0, # give components time to hydrate
+)
+```
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+async def main():
+ config = CrawlerRunConfig(
+ flatten_shadow_dom=True,
+ wait_until="load",
+ delay_before_return_html=3.0,
+ )
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url="https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011",
+ config=config,
+ )
+ # Without flatten_shadow_dom: ~1 KB markdown (breadcrumbs only)
+ # With flatten_shadow_dom: ~33 KB (product description, specs, downloads)
+ print(len(result.markdown.raw_markdown))
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+When enabled, Crawl4AI also injects an init script that force-opens closed shadow roots. The flattener resolves `<slot>` projections and strips shadow-scoped `<style>` tags, producing clean HTML for the downstream scraping/markdown pipeline.
+
+**Execution order**: `flatten_shadow_dom` runs right before HTML capture, after all waits and JS execution:
+```
+js_code_before_wait β wait_for β delay β js_code β flatten_shadow_dom β page capture
+```
+
+For a full runnable example, see [`shadow_dom_crawling.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/shadow_dom_crawling.py).
+
## 4. Structured Extraction Examples
### 4.1 Pattern-Based with `JsonCssExtractionStrategy`
```python
@@ -3344,8 +3394,9 @@ Below are the key interaction-related parameters in `CrawlerRunConfig`. For a fu
- **`wait_for`**: CSS (`"css:..."`) or JS (`"js:..."`) expression to wait for.
- **`session_id`**: Reuse the same page across calls.
- **`cache_mode`**: Whether to read/write from the cache or bypass.
-- **`remove_overlay_elements`**: Remove certain popups automatically.
-- **`simulate_user`, `override_navigator`, `magic`**: Anti-bot or βhuman-likeβ interactions.
+- **`remove_overlay_elements`**: Remove certain popups automatically.
+- **`remove_consent_popups`**: Remove GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, Didomi, etc.).
+- **`simulate_user`, `override_navigator`, `magic`**: Anti-bot or "human-like" interactions.
## 8. Conclusion
1.β**Execute JavaScript** for scrolling, clicks, or form filling.
2.β**Wait** for CSS or custom JS conditions before capturing data.
@@ -4077,7 +4128,7 @@ That's how you keep the config self-contained, illustrate **XPath** usage, and d
## 3. Advanced Schema & Nested Structures
### Sample E-Commerce HTML
```
-https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html
+https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html
```
```python
schema = {
@@ -4202,7 +4253,7 @@ async def extract_ecommerce_data():
async with AsyncWebCrawler(verbose=True) as crawler:
result = await crawler.arun(
- url="https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html",
+ url="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html",
extraction_strategy=strategy,
config=config
)
@@ -4359,7 +4410,7 @@ async def extract_with_generated_pattern():
# Get sample HTML for context
async with AsyncWebCrawler() as crawler:
result = await crawler.arun("https://example.com/products")
- html = result.fit_html
+ html = result.markdown.fit_html
# Generate pattern (one-time LLM usage)
pattern = RegexExtractionStrategy.generate_pattern(
@@ -4492,6 +4543,20 @@ xpath_schema = JsonXPathExtractionStrategy.generate_schema(
# Use the generated schema for fast, repeated extractions
strategy = JsonCssExtractionStrategy(css_schema)
```
+### Token Usage Tracking
+`generate_schema` may make multiple LLM calls internally (field inference, generation, validation retries). Track total token consumption by passing a `TokenUsage` accumulator:
+```python
+from crawl4ai.models import TokenUsage
+
+usage = TokenUsage()
+schema = JsonCssExtractionStrategy.generate_schema(
+ url="https://example.com/products",
+ query="extract product name and price",
+ usage=usage,
+)
+print(f"Total tokens: {usage.total_tokens}")
+```
+The `usage` parameter is optional and fully backward-compatible. Both `generate_schema` (sync) and `agenerate_schema` (async) support it.
### LLM Provider Options
1. **OpenAI GPT-4 (`openai/gpt4o`)**
- Default provider
@@ -5164,7 +5229,7 @@ This ensures the newly created context is under your control **before** `arun()`
## Core Imports
```python
-from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerConfig, CacheMode
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy, JsonCssExtractionStrategy, CosineStrategy
```
@@ -5178,14 +5243,14 @@ async with AsyncWebCrawler() as crawler:
## Advanced Pattern
```python
browser_config = BrowserConfig(headless=True, viewport_width=1920)
-crawler_config = CrawlerConfig(
+crawler_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
wait_for="css:.content",
screenshot=True,
pdf=True
)
strategy = LLMExtractionStrategy(
- provider="openai/gpt-4",
+ llm_config=LLMConfig(provider="openai/gpt-4", api_token="your-openai-token"),
instruction="Extract products with name and price"
)
diff --git a/docs/md_v2/core/adaptive-crawling.md b/docs/md_v2/core/adaptive-crawling.md
index 6f05416d2..b3cf2672c 100644
--- a/docs/md_v2/core/adaptive-crawling.md
+++ b/docs/md_v2/core/adaptive-crawling.md
@@ -100,7 +100,7 @@ The embedding strategy uses semantic embeddings for deeper understanding:
- **Best for**: Complex queries, ambiguous topics, conceptual understanding
```python
-# Configure embedding strategy
+# Configure embedding strategy with local embeddings
config = AdaptiveConfig(
strategy="embedding",
embedding_model="sentence-transformers/all-MiniLM-L6-v2", # Default
@@ -108,15 +108,20 @@ config = AdaptiveConfig(
embedding_min_confidence_threshold=0.1 # Stop if completely irrelevant
)
-# With custom LLM provider for query expansion (recommended)
+# With separate LLM configs for embeddings and query expansion (recommended)
from crawl4ai import LLMConfig
config = AdaptiveConfig(
strategy="embedding",
+ # Embedding model β used for text-to-vector calls
embedding_llm_config=LLMConfig(
provider='openai/text-embedding-3-small',
- api_token='your-api-key',
- temperature=0.7
+ api_token='your-api-key'
+ ),
+ # Query model β used for chat completion (query expansion)
+ query_llm_config=LLMConfig(
+ provider='openai/gpt-4o-mini',
+ api_token='your-api-key'
)
)
@@ -126,10 +131,20 @@ config = AdaptiveConfig(
embedding_llm_config={
'provider': 'openai/text-embedding-3-small',
'api_token': 'your-api-key'
+ },
+ query_llm_config={
+ 'provider': 'openai/gpt-4o-mini',
+ 'api_token': 'your-api-key'
}
)
```
+> **Note:** The embedding strategy makes two types of API calls that need different model types:
+> - **Embedding calls** (text β vector) require an embedding model like `text-embedding-3-small`
+> - **Query expansion** (chat completion) requires a chat model like `gpt-4o-mini`
+>
+> Use `embedding_llm_config` for the embedding model and `query_llm_config` for the chat model. If `query_llm_config` is not set, it falls back to `embedding_llm_config` for backward compatibility.
+
### Strategy Comparison
| Feature | Statistical | Embedding |
@@ -148,11 +163,12 @@ The embedding strategy offers fine-tuned control through several parameters:
```python
config = AdaptiveConfig(
strategy="embedding",
-
+
# Model configuration
embedding_model="sentence-transformers/all-MiniLM-L6-v2",
- embedding_llm_config=None, # Use for API-based embeddings
-
+ embedding_llm_config=None, # Use for API-based embeddings (embedding model)
+ query_llm_config=None, # Use for query expansion (chat completion model)
+
# Query expansion
n_query_variations=10, # Number of query variations to generate
@@ -258,7 +274,7 @@ adaptive.export_knowledge_base("knowledge_base.jsonl")
# Import into another session
new_adaptive = AdaptiveCrawler(crawler)
-new_adaptive.import_knowledge_base("knowledge_base.jsonl")
+await new_adaptive.import_knowledge_base("knowledge_base.jsonl")
```
## Best Practices
diff --git a/docs/md_v2/core/browser-crawler-config.md b/docs/md_v2/core/browser-crawler-config.md
index a0e59fd01..d9946c689 100644
--- a/docs/md_v2/core/browser-crawler-config.md
+++ b/docs/md_v2/core/browser-crawler-config.md
@@ -84,11 +84,16 @@ class BrowserConfig:
```
- Leave as `None` if a proxy is not required.
-7.β **`viewport_width` & `viewport_height`**
- - The initial window size.
+7.β **`viewport_width` & `viewport_height`**
+ - The initial window size.
- Some sites behave differently with smaller or bigger viewports.
-8.β **`verbose`**
+8.β **`device_scale_factor`**
+ - Controls the device pixel ratio (DPR) for rendering. Default is `1.0`.
+ - Set to `2.0` for Retina-quality screenshots (e.g., a 1920Γ1080 viewport produces 3840Γ2160 images).
+ - Higher values increase screenshot size and rendering time proportionally.
+
+9.β **`verbose`**
- If `True`, prints extra logs.
- Handy for debugging.
@@ -104,17 +109,22 @@ class BrowserConfig:
- `user_agent`: Custom User-Agent string. If `None`, a default is used.
- `user_agent_mode`: Set to `"random"` for randomization (helps fight bot detection).
-12.β **`text_mode`** & **`light_mode`**
- - `text_mode=True` disables images, possibly speeding up text-only crawls.
- - `light_mode=True` turns off certain background features for performance.
+12.β **`text_mode`** & **`light_mode`**
+ - `text_mode=True` disables images, possibly speeding up text-only crawls.
+ - `light_mode=True` turns off certain background features for performance.
+
+13.β **`avoid_ads`** & **`avoid_css`**
+ - `avoid_ads=True` blocks requests to common ad and tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level. Reduces network overhead and memory usage.
+ - `avoid_css=True` blocks loading of CSS files (`.css`, `.less`, `.scss`, `.sass`), useful when you only need text content and want faster, leaner crawls.
+ - Both default to `False` (opt-in). Can be combined with each other and with `text_mode`.
-13.β **`extra_args`**
+14.β **`extra_args`**
- Additional flags for the underlying browser.
- E.g. `["--disable-extensions"]`.
-14.β **`enable_stealth`**
- - If `True`, enables stealth mode using playwright-stealth.
- - Modifies browser fingerprints to avoid basic bot detection.
+15.β **`enable_stealth`**
+ - If `True`, enables stealth mode using playwright-stealth.
+ - Modifies browser fingerprints to avoid basic bot detection.
- Default is `False`. Recommended for sites with bot protection.
### Helper Methods
@@ -136,6 +146,41 @@ debug_browser = base_browser.clone(
)
```
+### Class-Level Defaults
+
+Both `BrowserConfig` and `CrawlerRunConfig` support **class-level default overrides** via `set_defaults()`. This is useful in server/cloud deployments where every config instance needs the same base settings β set them once at startup instead of repeating at every call site.
+
+```python
+from crawl4ai import BrowserConfig, CrawlerRunConfig
+
+# At application startup β one time
+BrowserConfig.set_defaults(
+ cache_cdp_connection=True,
+ cdp_close_delay=0,
+ create_isolated_context=True,
+)
+CrawlerRunConfig.set_defaults(verbose=False)
+
+# Every new instance automatically inherits those defaults
+cfg = BrowserConfig(cdp_url="ws://localhost:9222")
+# β cache_cdp_connection=True, cdp_close_delay=0, create_isolated_context=True
+
+# Explicit values still win
+cfg = BrowserConfig(cdp_url="ws://localhost:9222", cache_cdp_connection=False)
+# β cache_cdp_connection=False (explicit overrides the class default)
+```
+
+**Available methods** (on both `BrowserConfig` and `CrawlerRunConfig`):
+
+| Method | Description |
+|--------|-------------|
+| `set_defaults(**kwargs)` | Set class-level defaults. Invalid parameter names raise `ValueError`. |
+| `get_defaults()` | Return a copy of the current class-level defaults. |
+| `reset_defaults()` | Clear all class-level defaults. |
+| `reset_defaults("param1", "param2")` | Clear only the named defaults. |
+
+> **Note:** Class defaults are independent per class β `BrowserConfig.set_defaults()` does not affect `CrawlerRunConfig`, and vice versa. Defaults are stored in memory and apply for the lifetime of the process.
+
**Minimal Example**:
```python
@@ -215,18 +260,25 @@ class CrawlerRunConfig:
- Controls caching behavior (`ENABLED`, `BYPASS`, `DISABLED`, etc.).
- Defaults to `CacheMode.BYPASS`.
-6.β **`js_code`** & **`c4a_script`**:
- - `js_code`: A string or list of JavaScript strings to execute.
+6.β **`js_code`**, **`js_code_before_wait`**, & **`c4a_script`**:
+ - `js_code`: JavaScript to run **after** `wait_for` completes β on the fully-loaded page.
+ - `js_code_before_wait`: JavaScript to run **before** `wait_for` β for triggering loading that `wait_for` then checks.
- `c4a_script`: C4A script that compiles to JavaScript.
- - Great for "Load More" buttons or user interactions.
+ - Great for "Load More" buttons or user interactions.
7.β **`wait_for`**:
- A CSS or JS expression to wait for before extracting content.
- Common usage: `wait_for="css:.main-loaded"` or `wait_for="js:() => window.loaded === true"`.
-8.β **`screenshot`**, **`pdf`**, & **`capture_mhtml`**:
- - If `True`, captures a screenshot, PDF, or MHTML snapshot after the page is fully loaded.
+8.β **`flatten_shadow_dom`**:
+ - If `True`, flattens Shadow DOM content into the light DOM before HTML capture.
+ - Essential for sites built with Web Components (Stencil, Lit, Shoelace, etc.).
+ - Also force-opens closed shadow roots. See [Flattening Shadow DOM](content-selection.md#31-flattening-shadow-dom).
+
+9.β **`screenshot`**, **`pdf`**, & **`capture_mhtml`**:
+ - If `True`, captures a screenshot, PDF, or MHTML snapshot after the page is fully loaded.
- The results go to `result.screenshot` (base64), `result.pdf` (bytes), or `result.mhtml` (string).
+ - Use `force_viewport_screenshot=True` to capture only the visible viewport instead of the full page. This is faster and produces smaller images when you don't need a full-page screenshot.
9.β **Location Parameters**:
- **`locale`**: Browser's locale (e.g., `"en-US"`, `"fr-FR"`) for language preferences
@@ -234,17 +286,21 @@ class CrawlerRunConfig:
- **`geolocation`**: GPS coordinates via `GeolocationConfig(latitude=48.8566, longitude=2.3522)`
- See [Identity Based Crawling](../advanced/identity-based-crawling.md#7-locale-timezone-and-geolocation-control)
-10.β **Proxy Configuration**:
- - **`proxy_config`**: Proxy server configuration (ProxyConfig object or dict) e.g. {"server": "...", "username": "...", "password"}
+10.β **Proxy Configuration**:
+ - **`proxy_config`**: Single `ProxyConfig` or `list[ProxyConfig]` β proxies tried in order. Pass a list for automatic escalation.
- **`proxy_rotation_strategy`**: Strategy for rotating proxies during crawls
-11.β **Page Interaction Parameters**:
+11.β **Anti-Bot Retry & Fallback** (see [Anti-Bot & Fallback](../advanced/anti-bot-and-fallback.md)):
+ - **`max_retries`**: Number of retry rounds when blocking is detected (default: 0). Each round tries all proxies in `proxy_config`.
+ - **`fallback_fetch_function`**: Async function called as last resort β takes URL, returns raw HTML
+
+12.β **Page Interaction Parameters**:
- **`scan_full_page`**: If `True`, scroll through the entire page to load all content
- **`wait_until`**: Condition to wait for when navigating (e.g., "domcontentloaded", "networkidle")
- **`page_timeout`**: Timeout in milliseconds for page operations (default: 60000)
- **`delay_before_return_html`**: Delay in seconds before retrieving final HTML.
-12.β **`url_matcher`** & **`match_mode`**:
+13.β **`url_matcher`** & **`match_mode`**:
- Enable URL-specific configurations when used with `arun_many()`.
- Set `url_matcher` to patterns (glob, function, or list) to match specific URLs.
- Use `match_mode` (OR/AND) to control how multiple patterns combine.
diff --git a/docs/md_v2/core/cache-modes.md b/docs/md_v2/core/cache-modes.md
index b0aab78a6..9d45f2a61 100644
--- a/docs/md_v2/core/cache-modes.md
+++ b/docs/md_v2/core/cache-modes.md
@@ -24,22 +24,16 @@ The new system uses a single `CacheMode` enum:
### Old Code (Deprecated)
```python
-import asyncio
from crawl4ai import AsyncWebCrawler
-async def use_proxy():
- async with AsyncWebCrawler(verbose=True) as crawler:
- result = await crawler.arun(
- url="https://www.nbcnews.com/business",
- bypass_cache=True # Old way
- )
- print(len(result.markdown))
-
-async def main():
- await use_proxy()
-
-if __name__ == "__main__":
- asyncio.run(main())
+async def old_code(crawler: AsyncWebCrawler):
+ # Legacy `bypass_cache` / `disable_cache` / `no_cache_read` / `no_cache_write`
+ # were removed in v0.5+. This example no longer applies:
+ result = await crawler.arun(
+ url="https://www.nbcnews.com/business",
+ # cache_mode is the only cache option now.
+ )
+ print(len(result.markdown))
```
### New Code (Recommended)
@@ -67,9 +61,9 @@ if __name__ == "__main__":
## Common Migration Patterns
-| Old Flag | New Mode |
-|-----------------------|---------------------------------|
-| `bypass_cache=True` | `cache_mode=CacheMode.BYPASS` |
-| `disable_cache=True` | `cache_mode=CacheMode.DISABLED`|
-| `no_cache_read=True` | `cache_mode=CacheMode.WRITE_ONLY` |
-| `no_cache_write=True` | `cache_mode=CacheMode.READ_ONLY` |
\ No newline at end of file
+| Legacy Flag | Replacement |
+|------------------------|----------------------------|
+| `bypass_cache` | `cache_mode=CacheMode.BYPASS` |
+| `disable_cache` | `cache_mode=CacheMode.DISABLED` |
+| `no_cache_read` | `cache_mode=CacheMode.READ_ONLY` |
+| `no_cache_write` | `cache_mode=CacheMode.WRITE_ONLY`|
diff --git a/docs/md_v2/core/content-selection.md b/docs/md_v2/core/content-selection.md
index 859984336..79a7a7598 100644
--- a/docs/md_v2/core/content-selection.md
+++ b/docs/md_v2/core/content-selection.md
@@ -153,8 +153,10 @@ Some sites embed content in `<iframe>` tags. If you want that inline:
```python
config = CrawlerRunConfig(
# Merge iframe content into the final output
- process_iframes=True,
- remove_overlay_elements=True
+ process_iframes=True,
+ remove_overlay_elements=True,
+ # Remove GDPR/cookie consent popups (OneTrust, Cookiebot, etc.)
+ remove_consent_popups=True
)
```
@@ -181,6 +183,55 @@ if __name__ == "__main__":
---
+## 3.1 Flattening Shadow DOM
+
+Sites built with **Web Components** (Stencil, Lit, Shoelace, Angular Elements, etc.) render content inside [Shadow DOM](https://developer.mozilla.org/en-US/docs/Web/API/Web_components/Using_shadow_DOM) β an encapsulated sub-tree that is invisible to normal page serialization. The browser renders it on screen, but `page.content()` never includes it.
+
+Set `flatten_shadow_dom=True` to walk all shadow trees, resolve `<slot>` projections, and produce a single flat HTML document:
+
+```python
+config = CrawlerRunConfig(
+ # Flatten shadow DOM into the main document
+ flatten_shadow_dom=True,
+ # Give web components time to hydrate
+ wait_until="load",
+ delay_before_return_html=3.0,
+)
+```
+
+**Full example** β crawling a product page where specs live inside shadow roots:
+
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+async def main():
+ config = CrawlerRunConfig(
+ flatten_shadow_dom=True,
+ wait_until="load",
+ delay_before_return_html=3.0,
+ )
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url="https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011",
+ config=config,
+ )
+ # Without flatten_shadow_dom: ~1 KB of markdown (breadcrumbs only)
+ # With flatten_shadow_dom: ~33 KB (full product specs, downloads, etc.)
+ print(len(result.markdown.raw_markdown))
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+
+When `flatten_shadow_dom=True` is set, Crawl4AI also injects an init script that force-opens **closed** shadow roots (by patching `Element.prototype.attachShadow`), so even components that use `mode: 'closed'` become accessible.
+
+> **Tip**: Web components need JavaScript to run before they render content (a process called *hydration*). Use `wait_until="load"` and a `delay_before_return_html` of 2β5 seconds to ensure components are fully hydrated before flattening.
+
+For a complete runnable example, see [`shadow_dom_crawling.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/shadow_dom_crawling.py).
+
+---
+
## 4. Structured Extraction Examples
You can combine content selection with a more advanced extraction strategy. For instance, a **CSS-based** or **LLM-based** extraction strategy can run on the filtered HTML.
diff --git a/docs/md_v2/core/crawler-result.md b/docs/md_v2/core/crawler-result.md
index 116782e1f..a531e043e 100644
--- a/docs/md_v2/core/crawler-result.md
+++ b/docs/md_v2/core/crawler-result.md
@@ -39,6 +39,7 @@ class CrawlResult(BaseModel):
ssl_certificate: Optional[SSLCertificate] = None
dispatch_result: Optional[DispatchResult] = None
redirected_url: Optional[str] = None
+ redirected_status_code: Optional[int] = None
network_requests: Optional[List[Dict[str, Any]]] = None
console_messages: Optional[List[Dict[str, Any]]] = None
tables: List[Dict] = Field(default_factory=list)
@@ -73,6 +74,7 @@ class CrawlResult(BaseModel):
| **ssl_certificate (`Optional[SSLCertificate]`)** | SSL certificate info if `fetch_ssl_certificate=True`. |
| **dispatch_result (`Optional[DispatchResult]`)** | Additional concurrency and resource usage information when crawling URLs in parallel. |
| **redirected_url (`Optional[str]`)** | The URL after any redirects (different from `url` which is the final URL). |
+| **redirected_status_code (`Optional[int]`)** | HTTP status code of the final redirect destination (e.g., 200). `None` for non-HTTP requests (raw HTML, local files). |
| **network_requests (`Optional[List[Dict[str, Any]]]`)** | List of network requests, responses, and failures captured during the crawl if `capture_network_requests=True`. |
| **console_messages (`Optional[List[Dict[str, Any]]]`)** | List of browser console messages captured during the crawl if `capture_console_messages=True`. |
| **tables (`List[Dict]`)** | Table data extracted from HTML tables with structure `[{headers, rows, caption, summary}]`. |
@@ -108,7 +110,7 @@ print(result.cleaned_html) # Freed of forms, header, footer, data-* attributes
### 3.1 `markdown`
- **`markdown`**: The current location for detailed markdown output, returning a **`MarkdownGenerationResult`** object.
-- **`markdown_v2`**: Deprecated since v0.5.
+- **`markdown_v2`**: Removed in v0.5. Accessing it now raises `AttributeError`; use `markdown`.
**`MarkdownGenerationResult`** Fields:
@@ -326,8 +328,7 @@ else:
print("Error:", result.error_message)
```
-**Deprecation**: Since v0.5 `result.markdown_v2`, `result.fit_html`,`result.fit_markdown` are deprecated. Use `result.markdown` instead! It holds `MarkdownGenerationResult`, which includes `fit_html` and `fit_markdown`
-as it's properties.
+**Deprecation**: Since v0.5 `markdown_v2`, `fit_markdown`, and `fit_html` are removed from `CrawlResult`. Use `result.markdown` for markdown output. It holds `MarkdownGenerationResult`, including `fit_html` and `fit_markdown`.
---
@@ -339,4 +340,4 @@ as it's properties.
- **Session & Hooks**: If you want to manipulate the page or preserve state across multiple `arun()` calls, see the hooking or session docs.
- **LLM Extraction**: For complex or unstructured content requiring AI-driven parsing, check the LLM-based strategies doc.
-**Enjoy** exploring all that `CrawlResult` offersβwhether you need raw HTML, sanitized output, markdown, or fully structured data, Crawl4AI has you covered!
\ No newline at end of file
+**Enjoy** exploring all that `CrawlResult` offersβwhether you need raw HTML, sanitized output, markdown, or fully structured data, Crawl4AI has you covered!
diff --git a/docs/md_v2/core/deep-crawling.md b/docs/md_v2/core/deep-crawling.md
index ded7638d6..c171cc836 100644
--- a/docs/md_v2/core/deep-crawling.md
+++ b/docs/md_v2/core/deep-crawling.md
@@ -635,11 +635,172 @@ When `resume_state=None` and `on_state_change=None` (the defaults), there is no
---
-## 11. Prefetch Mode for Fast URL Discovery
+## 11. Cancellation Support for Deep Crawls
+
+For production environments like cloud platforms, you often need to stop a running crawl mid-executionβwhether the user changed their mind, specified the wrong URL, or wants to control costs. Crawl4AI provides built-in cancellation support for all deep crawl strategies.
+
+### 11.1 Two Ways to Cancel
+
+**Option A: Callback-based cancellation** (recommended for external systems)
+
+Use `should_cancel` to check an external source (Redis, database, API) before each URL:
+
+```python
+from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
+
+async def check_if_cancelled():
+ # Check Redis, database, or any external source
+ job = await redis.get(f"job:{job_id}")
+ return job.get("status") == "cancelled"
+
+strategy = BFSDeepCrawlStrategy(
+ max_depth=3,
+ max_pages=1000,
+ should_cancel=check_if_cancelled, # Called before each URL
+)
+```
+
+**Option B: Direct cancellation** (for in-process control)
+
+Call `cancel()` directly on the strategy instance:
+
+```python
+strategy = BFSDeepCrawlStrategy(max_depth=3, max_pages=1000)
+
+# In another coroutine or thread:
+strategy.cancel() # Thread-safe, stops before next URL
+```
+
+### 11.2 Checking Cancellation Status
+
+Use the `cancelled` property to check if a crawl was cancelled:
+
+```python
+async with AsyncWebCrawler() as crawler:
+ results = await crawler.arun(url, config=config)
+
+if strategy.cancelled:
+ print(f"Crawl was cancelled after {len(results)} pages")
+else:
+ print(f"Crawl completed with {len(results)} pages")
+```
+
+### 11.3 State Notifications Include Cancelled Flag
+
+When using `on_state_change`, the state dictionary includes a `cancelled` field:
+
+```python
+async def handle_state(state: dict):
+ if state.get("cancelled"):
+ print("Crawl was cancelled!")
+ print(f"Crawled {state['pages_crawled']} pages before cancellation")
+ # Save state for potential resume
+ await redis.set("crawl_state", json.dumps(state))
+
+strategy = BFSDeepCrawlStrategy(
+ max_depth=3,
+ should_cancel=check_cancelled,
+ on_state_change=handle_state,
+)
+```
+
+### 11.4 Key Behaviors
+
+| Scenario | Behavior |
+|----------|----------|
+| Cancel before first URL | Returns empty results, `cancelled=True` |
+| Cancel during crawl | Completes current URL, then stops |
+| Callback raises exception | Logged as warning, crawl continues (fail-open) |
+| Strategy reuse after cancel | Works normally (cancel flag auto-resets) |
+| Sync callback function | Supported (auto-detected and handled) |
+
+### 11.5 Complete Example: Cloud Platform Job Cancellation
+
+```python
+import asyncio
+import json
+import redis.asyncio as redis
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
+
+async def run_cancellable_crawl(job_id: str, start_url: str):
+ redis_client = redis.Redis(host='localhost', port=6379, db=0)
+
+ # Check external cancellation source
+ async def check_cancelled():
+ status = await redis_client.get(f"job:{job_id}:status")
+ return status == b"cancelled"
+
+ # Save progress for monitoring and recovery
+ async def save_progress(state: dict):
+ await redis_client.set(
+ f"job:{job_id}:state",
+ json.dumps(state)
+ )
+ # Update job progress
+ await redis_client.set(
+ f"job:{job_id}:pages_crawled",
+ state["pages_crawled"]
+ )
+
+ strategy = BFSDeepCrawlStrategy(
+ max_depth=3,
+ max_pages=500,
+ should_cancel=check_cancelled,
+ on_state_change=save_progress,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ stream=True,
+ )
+
+ results = []
+ try:
+ async with AsyncWebCrawler() as crawler:
+ async for result in await crawler.arun(start_url, config=config):
+ results.append(result)
+ print(f"Crawled: {result.url}")
+ finally:
+ # Report final status
+ if strategy.cancelled:
+ await redis_client.set(f"job:{job_id}:status", "cancelled")
+ print(f"Job cancelled after {len(results)} pages")
+ else:
+ await redis_client.set(f"job:{job_id}:status", "completed")
+ print(f"Job completed with {len(results)} pages")
+
+ await redis_client.close()
+
+ return results
+
+# Usage
+# asyncio.run(run_cancellable_crawl("job-123", "https://example.com"))
+#
+# To cancel from another process:
+# redis_client.set("job:job-123:status", "cancelled")
+```
+
+### 11.6 Supported Strategies
+
+Cancellation works identically across all deep crawl strategies:
+
+- **BFSDeepCrawlStrategy** - Breadth-first search
+- **DFSDeepCrawlStrategy** - Depth-first search
+- **BestFirstCrawlingStrategy** - Priority-based crawling
+
+All strategies support:
+- `should_cancel` callback parameter
+- `cancel()` method
+- `cancelled` property
+
+---
+
+## 12. Prefetch Mode for Fast URL Discovery
When you need to quickly discover URLs without full page processing, use **prefetch mode**. This is ideal for two-phase crawling where you first map the site, then selectively process specific pages.
-### 11.1 Enabling Prefetch Mode
+### 12.1 Enabling Prefetch Mode
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
@@ -654,7 +815,7 @@ async with AsyncWebCrawler() as crawler:
print(f"Found {len(result.links['external'])} external links")
```
-### 11.2 What Gets Skipped
+### 12.2 What Gets Skipped
Prefetch mode uses a fast path that bypasses heavy processing:
@@ -667,14 +828,14 @@ Prefetch mode uses a fast path that bypasses heavy processing:
| Media extraction | β
| β Skipped |
| LLM extraction | β
| β Skipped |
-### 11.3 Performance Benefit
+### 12.3 Performance Benefit
- **Normal mode**: Full pipeline (~2-5 seconds per page)
- **Prefetch mode**: HTML + links only (~200-500ms per page)
This makes prefetch mode **5-10x faster** for URL discovery.
-### 11.4 Two-Phase Crawling Pattern
+### 12.4 Two-Phase Crawling Pattern
The most common use case is two-phase crawling:
@@ -720,7 +881,7 @@ if __name__ == "__main__":
print(f"Fully processed {len(results)} pages")
```
-### 11.5 Use Cases
+### 12.5 Use Cases
- **Site mapping**: Quickly discover all URLs before deciding what to process
- **Link validation**: Check which pages exist without heavy processing
@@ -729,7 +890,7 @@ if __name__ == "__main__":
---
-## 12. Summary & Next Steps
+## 13. Summary & Next Steps
In this **Deep Crawling with Crawl4AI** tutorial, you learned to:
@@ -740,6 +901,7 @@ In this **Deep Crawling with Crawl4AI** tutorial, you learned to:
- Limit crawls with `max_pages` and `score_threshold` parameters
- Build a complete advanced crawler with combined techniques
- **Implement crash recovery** with `resume_state` and `on_state_change` for production deployments
+- **Cancel running crawls** with `should_cancel` callback or `cancel()` method for cloud platform job management
- **Use prefetch mode** for fast URL discovery and two-phase crawling
With these tools, you can efficiently extract structured data from websites at scale, focusing precisely on the content you need for your specific use case.
diff --git a/docs/md_v2/core/local-files.md b/docs/md_v2/core/local-files.md
index 2fccea81d..a1692bff2 100644
--- a/docs/md_v2/core/local-files.md
+++ b/docs/md_v2/core/local-files.md
@@ -56,13 +56,13 @@ To crawl raw HTML content, prefix the HTML string with `raw:`.
```python
import asyncio
-from crawl4ai import AsyncWebCrawler
+from crawl4ai import AsyncWebCrawler, CacheMode
from crawl4ai.async_configs import CrawlerRunConfig
async def crawl_raw_html():
raw_html = "<html><body><h1>Hello, World!</h1></body></html>"
raw_html_url = f"raw:{raw_html}"
- config = CrawlerRunConfig(bypass_cache=True)
+ config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url=raw_html_url, config=config)
diff --git a/docs/md_v2/core/page-interaction.md b/docs/md_v2/core/page-interaction.md
index 809a23f3b..4b28cac2b 100644
--- a/docs/md_v2/core/page-interaction.md
+++ b/docs/md_v2/core/page-interaction.md
@@ -15,8 +15,9 @@ Below is a quick overview of how to do it.
### Basic Execution
-**`js_code`** in **`CrawlerRunConfig`** accepts either a single JS string or a list of JS snippets.
-**Example**: Weβll scroll to the bottom of the page, then optionally click a βLoad Moreβ button.
+**`js_code`** in **`CrawlerRunConfig`** accepts either a single JS string or a list of JS snippets. It runs **after** `wait_for` and `delay_before_return_html` β so the page is fully loaded when your code executes.
+
+**Example**: We'll scroll to the bottom of the page, then optionally click a "Load More" button.
```python
import asyncio
@@ -55,10 +56,36 @@ if __name__ == "__main__":
```
**Relevant `CrawlerRunConfig` params**:
-- **`js_code`**: A string or list of strings with JavaScript to run after the page loads.
-- **`js_only`**: If set to `True` on subsequent calls, indicates weβre continuing an existing session without a new full navigation.
+- **`js_code`**: JavaScript to run **after** `wait_for` and `delay_before_return_html` complete. Runs on the fully-loaded page.
+- **`js_code_before_wait`**: JavaScript to run **before** `wait_for`. Use when you need to trigger loading that `wait_for` then checks.
+- **`js_only`**: If set to `True` on subsequent calls, indicates we're continuing an existing session without a new full navigation.
- **`session_id`**: If you want to keep the same page across multiple calls, specify an ID.
+### Execution Order
+
+Understanding when your JavaScript runs relative to other pipeline steps:
+
+```
+1. Page navigation (page.goto)
+2. js_code_before_wait β triggers loading / clicks tabs
+3. wait_for β waits for content to appear
+4. delay_before_return_html β extra safety margin
+5. js_code β runs on the fully-loaded page
+6. flatten_shadow_dom β if enabled
+7. page.content() β HTML capture
+```
+
+If you need JS to trigger something and then wait for the result, use `js_code_before_wait` + `wait_for`:
+
+```python
+config = CrawlerRunConfig(
+ # Click a tab first
+ js_code_before_wait="document.querySelector('#specs-tab')?.click();",
+ # Then wait for the tab content to appear
+ wait_for="css:#specs-panel .content",
+)
+```
+
---
## 2. Wait Conditions
@@ -317,34 +344,55 @@ When done, check `result.extracted_content` for the JSON.
---
-## 7. Relevant `CrawlerRunConfig` Parameters
+## 7. Shadow DOM Flattening
+
+Sites built with **Web Components** (Stencil, Lit, Shoelace, etc.) render content inside Shadow DOM β an encapsulated sub-tree that is invisible to normal page serialization. Set `flatten_shadow_dom=True` to extract it:
+
+```python
+config = CrawlerRunConfig(
+ flatten_shadow_dom=True,
+ wait_until="load",
+ delay_before_return_html=3.0, # give components time to hydrate
+)
+```
+
+This walks all shadow trees, resolves `<slot>` projections, and produces flat HTML. It also force-opens closed shadow roots via an init script. For details and a full example, see [Flattening Shadow DOM](content-selection.md#31-flattening-shadow-dom) and [`shadow_dom_crawling.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/shadow_dom_crawling.py).
+
+---
+
+## 8. Relevant `CrawlerRunConfig` Parameters
Below are the key interaction-related parameters in `CrawlerRunConfig`. For a full list, see [Configuration Parameters](../api/parameters.md).
-- **`js_code`**: JavaScript to run after initial load.
-- **`js_only`**: If `True`, no new page navigationβonly JS in the existing session.
-- **`wait_for`**: CSS (`"css:..."`) or JS (`"js:..."`) expression to wait for.
-- **`session_id`**: Reuse the same page across calls.
-- **`cache_mode`**: Whether to read/write from the cache or bypass.
-- **`remove_overlay_elements`**: Remove certain popups automatically.
-- **`simulate_user`, `override_navigator`, `magic`**: Anti-bot or βhuman-likeβ interactions.
+- **`js_code`**: JavaScript to run after `wait_for` + `delay_before_return_html`, on the fully-loaded page.
+- **`js_code_before_wait`**: JavaScript to run before `wait_for`. For triggering loading that `wait_for` then checks.
+- **`js_only`**: If `True`, no new page navigationβonly JS in the existing session.
+- **`wait_for`**: CSS (`"css:..."`) or JS (`"js:..."`) expression to wait for.
+- **`session_id`**: Reuse the same page across calls.
+- **`cache_mode`**: Whether to read/write from the cache or bypass.
+- **`flatten_shadow_dom`**: Flatten Shadow DOM content into the light DOM before capture.
+- **`process_iframes`**: Inline iframe content into the main document.
+- **`remove_overlay_elements`**: Remove certain popups automatically.
+- **`remove_consent_popups`**: Remove GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, Didomi, etc.).
+- **`simulate_user`, `override_navigator`, `magic`**: Anti-bot or "human-like" interactions.
---
-## 8. Conclusion
+## 9. Conclusion
-Crawl4AIβs **page interaction** features let you:
+Crawl4AI's **page interaction** features let you:
1.β**Execute JavaScript** for scrolling, clicks, or form filling.
2.β**Wait** for CSS or custom JS conditions before capturing data.
3.β**Handle** multi-step flows (like βLoad Moreβ) with partial reloads or persistent sessions.
-4. Combine with **structured extraction** for dynamic sites.
+4. **Flatten Shadow DOM** on Web Component sites to extract hidden content.
+5. Combine with **structured extraction** for dynamic sites.
With these tools, you can scrape modern, interactive webpages confidently. For advanced hooking, user simulation, or in-depth config, check the [API reference](../api/parameters.md) or related advanced docs. Happy scripting!
---
-## 9. Virtual Scrolling
+## 10. Virtual Scrolling
For sites that use **virtual scrolling** (where content is replaced rather than appended as you scroll, like Twitter or Instagram), Crawl4AI provides a dedicated `VirtualScrollConfig`:
diff --git a/docs/md_v2/core/quickstart.md b/docs/md_v2/core/quickstart.md
index 83cb6cef8..78209efde 100644
--- a/docs/md_v2/core/quickstart.md
+++ b/docs/md_v2/core/quickstart.md
@@ -97,7 +97,7 @@ By default, Crawl4AI automatically generates Markdown from each crawled page. Ho
### Example: Using a Filter with `DefaultMarkdownGenerator`
```python
-from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
diff --git a/docs/md_v2/extraction/llm-strategies.md b/docs/md_v2/extraction/llm-strategies.md
index df948a9eb..cba4d6e4f 100644
--- a/docs/md_v2/extraction/llm-strategies.md
+++ b/docs/md_v2/extraction/llm-strategies.md
@@ -204,7 +204,9 @@ llm_strategy.show_usage()
# e.g. βTotal usage: 1241 tokens across 2 chunk callsβ
```
-If your model provider doesnβt return usage info, these fields might be partial or empty.
+If your model provider doesn't return usage info, these fields might be partial or empty.
+
+> **Tip:** `JsonCssExtractionStrategy.generate_schema()` also supports token usage tracking via an optional `usage` parameter. See [Token Usage Tracking in Schema Generation](./no-llm-strategies.md#token-usage-tracking) for details.
---
diff --git a/docs/md_v2/extraction/no-llm-strategies.md b/docs/md_v2/extraction/no-llm-strategies.md
index 48522e509..63138b305 100644
--- a/docs/md_v2/extraction/no-llm-strategies.md
+++ b/docs/md_v2/extraction/no-llm-strategies.md
@@ -92,9 +92,10 @@ asyncio.run(extract_crypto_prices())
**Highlights**:
-- **`baseSelector`**: Tells us where each "item" (crypto row) is.
-- **`fields`**: Two fields (`coin_name`, `price`) using simple CSS selectors.
+- **`baseSelector`**: Tells us where each "item" (crypto row) is.
+- **`fields`**: Two fields (`coin_name`, `price`) using simple CSS selectors.
- Each field defines a **`type`** (e.g., `text`, `attribute`, `html`, `regex`, etc.).
+- Optional keys: **`transform`**, **`default`**, **`attribute`**, **`pattern`**, and **`source`** (for sibling data β see [Extracting Sibling Data](#sibling-data)).
No LLM is needed, and the performance is **near-instant** for hundreds or thousands of items.
@@ -190,7 +191,7 @@ Real sites often have **nested** or repeated dataβlike categories containing p
We have a **sample e-commerce** HTML file on GitHub (example):
```
-https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html
+https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html
```
This snippet includes categories, products, features, reviews, and related items. Let's see how to define a schema that fully captures that structure **without LLM**.
@@ -322,7 +323,7 @@ async def extract_ecommerce_data():
async with AsyncWebCrawler(verbose=True) as crawler:
result = await crawler.arun(
- url="https://gist.githubusercontent.com/githubusercontent/2d7b8ba3cd8ab6cf3c8da771ddb36878/raw/1ae2f90c6861ce7dd84cc50d3df9920dee5e1fd2/sample_ecommerce.html",
+ url="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html",
extraction_strategy=strategy,
config=config
)
@@ -505,7 +506,7 @@ async def extract_with_generated_pattern():
# Get sample HTML for context
async with AsyncWebCrawler() as crawler:
result = await crawler.arun("https://example.com/products")
- html = result.fit_html
+ html = result.markdown.fit_html
# Generate pattern (one-time LLM usage)
pattern = RegexExtractionStrategy.generate_pattern(
@@ -623,7 +624,60 @@ Then run with `JsonCssExtractionStrategy(schema)` to get an array of blog post o
---
-## 8. Tips & Best Practices
+## 8. Extracting Sibling Data with `source` {#sibling-data}
+
+Some websites split a single logical item across **sibling elements** rather than nesting everything inside one container. A classic example is Hacker News, where each submission spans two adjacent `<tr>` rows:
+
+```html
+<tr class="athing submission"> <!-- rank, title, url -->
+ <td><span class="rank">1.</span></td>
+ <td><span class="titleline"><a href="https://example.com">Example Title</a></span></td>
+</tr>
+<tr> <!-- score, author, comments (sibling!) -->
+ <td class="subtext">
+ <span class="score">100 points</span>
+ <a class="hnuser">johndoe</a>
+ </td>
+</tr>
+```
+
+Normally, field selectors only search **descendants** of the base element β siblings are unreachable. The `source` field key solves this by navigating to a sibling element before running the selector.
+
+### Syntax
+
+```
+"source": "+ <selector>"
+```
+
+- **`+ tr`** β next sibling `<tr>`
+- **`+ div.details`** β next sibling `<div>` with class `details`
+- **`+ .subtext`** β next sibling with class `subtext`
+
+### Example: Hacker News
+
+```python
+schema = {
+ "name": "HN Submissions",
+ "baseSelector": "tr.athing.submission",
+ "fields": [
+ {"name": "rank", "selector": "span.rank", "type": "text"},
+ {"name": "title", "selector": "span.titleline a", "type": "text"},
+ {"name": "url", "selector": "span.titleline a", "type": "attribute", "attribute": "href"},
+ {"name": "score", "selector": "span.score", "type": "text", "source": "+ tr"},
+ {"name": "author", "selector": "a.hnuser", "type": "text", "source": "+ tr"},
+ ],
+}
+
+strategy = JsonCssExtractionStrategy(schema)
+```
+
+The `score` and `author` fields first navigate to the next sibling `<tr>`, then run their selectors inside that element. Fields without `source` work as before β searching descendants of the base element.
+
+`source` works with all field types (`text`, `attribute`, `nested`, `list`, etc.) and with both `JsonCssExtractionStrategy` and `JsonXPathExtractionStrategy`. If the sibling isn't found, the field returns its `default` value.
+
+---
+
+## 9. Tips & Best Practices
1. **Inspect the DOM** in Chrome DevTools or Firefox's Inspector to find stable selectors.
2. **Start Simple**: Verify you can extract a single field. Then add complexity like nested objects or lists.
@@ -636,7 +690,7 @@ Then run with `JsonCssExtractionStrategy(schema)` to get an array of blog post o
---
-## 9. Schema Generation Utility
+## 10. Schema Generation Utility
While manually crafting schemas is powerful and precise, Crawl4AI now offers a convenient utility to **automatically generate** extraction schemas using LLM. This is particularly useful when:
@@ -669,7 +723,7 @@ html = """
# Option 1: Using OpenAI (requires API token)
css_schema = JsonCssExtractionStrategy.generate_schema(
html,
- schema_type="css",
+ schema_type="css",
llm_config = LLMConfig(provider="openai/gpt-4o",api_token="your-openai-token")
)
@@ -684,6 +738,61 @@ xpath_schema = JsonXPathExtractionStrategy.generate_schema(
strategy = JsonCssExtractionStrategy(css_schema)
```
+### Schema Validation
+
+By default, `generate_schema` **validates** the generated schema against the HTML to ensure that it actually extracts the data you expect. If the schema doesn't produce results, it automatically refines the selectors before returning.
+
+You can control this with the `validate` parameter:
+
+```python
+# Default: validated (recommended)
+schema = JsonCssExtractionStrategy.generate_schema(
+ url="https://news.ycombinator.com",
+ query="Extract each story: title, url, score, author",
+)
+
+# Skip validation if you want raw LLM output
+schema = JsonCssExtractionStrategy.generate_schema(
+ url="https://news.ycombinator.com",
+ query="Extract each story: title, url, score, author",
+ validate=False,
+)
+```
+
+The generator also understands sibling layouts β for sites like Hacker News where data is split across sibling elements, it will automatically use the [`source` field](#sibling-data) to reach sibling data.
+
+### Token Usage Tracking
+
+`generate_schema` may make multiple LLM calls internally (field inference, schema generation, validation retries). To track the total token consumption across all of these calls, pass a `TokenUsage` accumulator:
+
+```python
+from crawl4ai import JsonCssExtractionStrategy
+from crawl4ai.models import TokenUsage
+
+usage = TokenUsage()
+
+schema = JsonCssExtractionStrategy.generate_schema(
+ url="https://news.ycombinator.com",
+ query="Extract each story: title, url, score, author",
+ usage=usage,
+)
+
+print(f"Prompt tokens: {usage.prompt_tokens}")
+print(f"Completion tokens: {usage.completion_tokens}")
+print(f"Total tokens: {usage.total_tokens}")
+```
+
+The `usage` parameter is optional β omitting it changes nothing (fully backward-compatible). You can also reuse the same accumulator across multiple calls to get a grand total:
+
+```python
+usage = TokenUsage()
+schema1 = JsonCssExtractionStrategy.generate_schema(url=url1, query=q1, usage=usage)
+schema2 = JsonCssExtractionStrategy.generate_schema(url=url2, query=q2, usage=usage)
+print(f"Grand total: {usage.total_tokens} tokens")
+```
+
+Both `generate_schema` (sync) and `agenerate_schema` (async) support the `usage` parameter.
+
### LLM Provider Options
1. **OpenAI GPT-4 (`openai/gpt4o`)**
@@ -814,7 +923,7 @@ This approach lets you generate schemas once that work reliably across hundreds
---
-## 10. Conclusion
+## 11. Conclusion
With Crawl4AI's LLM-free extraction strategies - `JsonCssExtractionStrategy`, `JsonXPathExtractionStrategy`, and now `RegexExtractionStrategy` - you can build powerful pipelines that:
diff --git a/docs/md_v2/marketplace/frontend/app-detail.html b/docs/md_v2/marketplace/frontend/app-detail.html
index 92b5a6ddf..4a83d3647 100644
--- a/docs/md_v2/marketplace/frontend/app-detail.html
+++ b/docs/md_v2/marketplace/frontend/app-detail.html
@@ -152,19 +152,24 @@ <h3>Integration Example</h3>
<span class="code-lang">python</span>
<button class="copy-btn">Copy</button>
</div>
- <pre><code id="integration-code">from crawl4ai import AsyncWebCrawler
+ <pre><code id="integration-code">from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
+from crawl4ai import LLMConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
async def extract_with_llm():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
- extraction_strategy=LLMExtractionStrategy(
- provider="openai",
- api_key="your-api-key",
- instruction="Extract product information"
- ),
- bypass_cache=True
+ config=CrawlerRunConfig(
+ extraction_strategy=LLMExtractionStrategy(
+ llm_config=LLMConfig(
+ provider="openai/gpt-4o",
+ api_token="your-api-key",
+ ),
+ instruction="Extract product information"
+ ),
+ cache_mode=CacheMode.BYPASS
+ )
)
return result.extracted_content
@@ -175,7 +180,7 @@ <h3>Integration Example</h3>
<div class="info-box">
<h4>π‘ Pro Tip</h4>
- <p>Use the <code>bypass_cache=True</code> parameter when you need fresh data, or set <code>cache_mode="write"</code> to update the cache with new content.</p>
+ <p>Use <code>cache_mode=CacheMode.BYPASS</code> for a fresh crawl, or <code>CacheMode.WRITE_ONLY</code> to avoid refetching.</p>
</div>
</div>
</section>
@@ -231,4 +236,4 @@ <h2>Related Apps</h2>
<script src="app-detail.js"></script>
</body>
-</html>
\ No newline at end of file
+</html>
diff --git a/docs/md_v2/marketplace/frontend/app-detail.js b/docs/md_v2/marketplace/frontend/app-detail.js
index 5bc86d2b6..2f4549d83 100644
--- a/docs/md_v2/marketplace/frontend/app-detail.js
+++ b/docs/md_v2/marketplace/frontend/app-detail.js
@@ -164,21 +164,22 @@ proxy_config = {
"password": "your_password"
}
-async with AsyncWebCrawler(proxy=proxy_config) as crawler:
+async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://example.com",
- bypass_cache=True
+ cache_mode=CacheMode.BYPASS
)
print(result.status_code)`;
} else if (this.appData.category === 'LLM Integration') {
- usageCode.textContent = `from crawl4ai import AsyncWebCrawler
-from crawl4ai.extraction_strategy import LLMExtractionStrategy
+ usageCode.textContent = `from crawl4ai import AsyncWebCrawler, CacheMode
+from crawl4ai.extraction_strategy import LLMExtractionStrategy, LLMConfig
# Configure LLM extraction
strategy = LLMExtractionStrategy(
- provider="${this.appData.name.toLowerCase().includes('gpt') ? 'openai' : 'anthropic'}",
- api_key="your-api-key",
- model="${this.appData.name.toLowerCase().includes('gpt') ? 'gpt-4' : 'claude-3'}",
+ llm_config=LLMConfig(
+ provider="${this.appData.name.toLowerCase().includes('gpt') ? 'openai/gpt-4o' : 'anthropic/claude-3-5-sonnet-20240620'}",
+ api_token="your-api-key",
+ ),
instruction="Extract structured data"
)
@@ -228,7 +229,7 @@ async def crawl_with_${this.appData.slug.replace(/-/g, '_')}():
result = await crawler.arun(
url="https://example.com/products",
extraction_strategy=JsonCssExtractionStrategy(schema),
- cache_mode="bypass",
+ cache_mode=CacheMode.BYPASS,
wait_for="css:.product",
screenshot=True
)
@@ -331,4 +332,4 @@ if __name__ == "__main__":
// Initialize when DOM is loaded
document.addEventListener('DOMContentLoaded', () => {
new AppDetailPage();
-});
\ No newline at end of file
+});
diff --git a/docs/md_v2/stats.md b/docs/md_v2/stats.md
new file mode 100644
index 000000000..ccba194fe
--- /dev/null
+++ b/docs/md_v2/stats.md
@@ -0,0 +1,524 @@
+---
+hide:
+ - navigation
+ - toc
+---
+
+<!-- Generated by scripts/update_stats.py β do not edit manually -->
+
+<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.7/dist/chart.umd.min.js"></script>
+
+<style>
+/* Hide the right-hand TOC sidebar and let content use full width */
+#toc-sidebar {
+ display: none !important;
+}
+.terminal-mkdocs-main-grid {
+ margin-right: auto;
+}
+.terminal-mkdocs-main-content {
+ max-width: 100%;
+}
+
+.stats-page {
+ max-width: 1200px;
+ margin: 0 auto;
+ padding: 1rem 0;
+ font-family: inherit;
+}
+
+.stats-header {
+ text-align: center;
+ margin-bottom: 2.5rem;
+ padding-bottom: 1.5rem;
+ border-bottom: 1px solid #3f3f44;
+}
+
+.stats-header img {
+ height: 48px;
+ margin-bottom: 0.5rem;
+ filter: drop-shadow(0 0 12px rgba(80, 255, 255, 0.3));
+}
+
+.stats-header h1 {
+ font-size: 1.8rem;
+ color: #e8e9ed;
+ margin: 0.5rem 0 0.25rem;
+ letter-spacing: 0.5px;
+}
+
+.stats-header .subtitle {
+ color: #a3abba;
+ font-size: 0.95rem;
+}
+
+/* Metric Cards */
+.metrics-grid {
+ display: grid;
+ grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
+ gap: 1rem;
+ margin-bottom: 2.5rem;
+}
+
+.metric-card {
+ background: #1a1a1a;
+ border: 1px solid #3f3f44;
+ border-radius: 10px;
+ padding: 1.25rem;
+ text-align: center;
+ transition: all 0.3s ease;
+ position: relative;
+ overflow: hidden;
+}
+
+.metric-card::before {
+ content: '';
+ position: absolute;
+ top: 0;
+ left: 0;
+ right: 0;
+ height: 3px;
+ background: linear-gradient(90deg, #50ffff, #f380f5);
+ opacity: 0;
+ transition: opacity 0.3s ease;
+}
+
+.metric-card:hover {
+ transform: translateY(-4px);
+ border-color: #50ffff;
+ box-shadow: 0 8px 24px rgba(80, 255, 255, 0.12);
+}
+
+.metric-card:hover::before {
+ opacity: 1;
+}
+
+.metric-value {
+ font-size: 2rem;
+ font-weight: 700;
+ color: #50ffff;
+ margin: 0.25rem 0;
+ line-height: 1.2;
+}
+
+.metric-label {
+ font-size: 0.8rem;
+ color: #a3abba;
+ text-transform: uppercase;
+ letter-spacing: 1px;
+}
+
+.metric-sublabel {
+ font-size: 0.75rem;
+ color: #3f3f44;
+ margin-top: 0.25rem;
+}
+
+/* Chart Sections */
+.chart-section {
+ background: #1a1a1a;
+ border: 1px solid #3f3f44;
+ border-radius: 10px;
+ padding: 1.5rem;
+ margin-bottom: 1.5rem;
+}
+
+.chart-section h2 {
+ font-size: 1.1rem;
+ color: #e8e9ed;
+ margin: 0 0 1rem 0;
+ padding-bottom: 0.75rem;
+ border-bottom: 1px solid #3f3f44;
+}
+
+.chart-container {
+ position: relative;
+ width: 100%;
+ height: 350px;
+}
+
+.chart-row {
+ display: grid;
+ grid-template-columns: 1fr 1fr;
+ gap: 1.5rem;
+ margin-bottom: 1.5rem;
+}
+
+@media (max-width: 768px) {
+ .stats-page {
+ padding: 0.5rem 0.75rem;
+ }
+ .stats-header h1 {
+ font-size: 1.4rem;
+ }
+ .chart-row {
+ grid-template-columns: 1fr;
+ }
+ .metrics-grid {
+ grid-template-columns: repeat(2, 1fr);
+ gap: 0.6rem;
+ }
+ .metric-card {
+ padding: 0.9rem 0.5rem;
+ }
+ .metric-value {
+ font-size: 1.5rem;
+ }
+ .metric-label {
+ font-size: 0.7rem;
+ }
+ .chart-container {
+ height: 260px;
+ }
+ .chart-section {
+ padding: 1rem;
+ }
+}
+
+@media (max-width: 480px) {
+ .metrics-grid {
+ grid-template-columns: 1fr 1fr;
+ gap: 0.5rem;
+ }
+ .metric-value {
+ font-size: 1.25rem;
+ }
+ .metric-label {
+ font-size: 0.65rem;
+ letter-spacing: 0.5px;
+ }
+ .metric-sublabel {
+ font-size: 0.65rem;
+ }
+ .chart-container {
+ height: 220px;
+ }
+ .chart-section h2 {
+ font-size: 0.95rem;
+ }
+}
+
+/* Footer */
+.stats-footer {
+ text-align: center;
+ padding: 1.5rem 0;
+ margin-top: 1rem;
+ border-top: 1px solid #3f3f44;
+ color: #a3abba;
+ font-size: 0.8rem;
+}
+</style>
+
+<div class="stats-page">
+
+<div class="stats-header">
+ <img src="../assets/images/logo.png" alt="Crawl4AI">
+ <h1>Growth</h1>
+ <div class="subtitle">Community growth & adoption metrics for Crawl4AI</div>
+</div>
+
+<!-- Headline Metrics -->
+<div class="metrics-grid">
+ <div class="metric-card">
+ <div class="metric-label">GitHub Stars</div>
+ <div class="metric-value">60,904</div>
+ <div class="metric-sublabel">stargazers</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Monthly Downloads</div>
+ <div class="metric-value">914.2K</div>
+ <div class="metric-sublabel">PyPI Β· latest month</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Total Downloads</div>
+ <div class="metric-value">9.72M</div>
+ <div class="metric-sublabel">PyPI cumulative</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Docker Pulls</div>
+ <div class="metric-value">1.41M</div>
+ <div class="metric-sublabel">hub.docker.com</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Forks / Contributors</div>
+ <div class="metric-value">6,217 <span style="color:#a3abba;font-size:1rem">/</span> 57</div>
+ <div class="metric-sublabel">GitHub</div>
+ </div>
+</div>
+
+<!-- Chart 1: Monthly PyPI Downloads (bar) -->
+<div class="chart-section">
+ <h2>PyPI Monthly Downloads</h2>
+ <div class="chart-container">
+ <canvas id="monthlyChart"></canvas>
+ </div>
+</div>
+
+<!-- Chart Row: Stars + Cumulative -->
+<div class="chart-row">
+ <div class="chart-section">
+ <h2>GitHub Star Growth</h2>
+ <div class="chart-container">
+ <canvas id="starsChart"></canvas>
+ </div>
+ </div>
+ <div class="chart-section">
+ <h2>Cumulative PyPI Downloads</h2>
+ <div class="chart-container">
+ <canvas id="cumulativeChart"></canvas>
+ </div>
+ </div>
+</div>
+
+<!-- Chart Row: Daily + Traffic -->
+<div class="chart-row">
+ <div class="chart-section">
+ <h2>Daily Download Trend</h2>
+ <div class="chart-container">
+ <canvas id="dailyChart"></canvas>
+ </div>
+ </div>
+ <div class="chart-section">
+ <h2>GitHub Traffic (14 days)</h2>
+ <div class="chart-container">
+ <canvas id="trafficChart"></canvas>
+ </div>
+ </div>
+</div>
+
+<div class="stats-footer">
+ Updated February 24, 2026 · Data from GitHub API, PyPI Stats, Docker Hub
+</div>
+
+</div>
+
+<script>
+// --- Chart.js Global Config ---
+Chart.defaults.color = '#a3abba';
+Chart.defaults.borderColor = '#3f3f44';
+Chart.defaults.font.family = "'dm', Monaco, 'Courier New', monospace";
+
+const CYAN = '#50ffff';
+const CYAN_DIM = 'rgba(80,255,255,0.10)';
+const CYAN_HOVER = '#0fbbaa';
+const PINK = '#f380f5';
+const PINK_DIM = 'rgba(243,128,245,0.10)';
+const GRID_COLOR = '#3f3f44';
+const TOOLTIP_BG = '#1a1a1a';
+
+const tooltipStyle = {
+ backgroundColor: TOOLTIP_BG,
+ borderColor: CYAN,
+ borderWidth: 1,
+ titleColor: '#e8e9ed',
+ bodyColor: '#a3abba',
+ padding: 10,
+ cornerRadius: 6,
+ displayColors: false,
+};
+
+const gridStyle = { color: GRID_COLOR, drawBorder: false };
+const tickStyle = { color: '#a3abba', font: { size: 11 } };
+
+// --- Data ---
+const monthlyLabels = ["2024-09", "2024-10", "2024-11", "2024-12", "2025-01", "2025-02", "2025-03", "2025-04", "2025-05", "2025-06", "2025-07", "2025-08", "2025-09", "2025-10", "2025-11", "2025-12", "2026-01", "2026-02"];
+const monthlyValues = [28000, 135000, 210000, 285000, 350000, 380000, 430000, 480000, 520000, 560000, 620000, 750000, 836245, 911467, 726475, 657370, 922296, 914168];
+
+const starDates = ["2024-02-01", "2024-06-01", "2024-09-01", "2024-10-01", "2024-11-01", "2024-12-01", "2025-01-01", "2025-02-01", "2025-03-01", "2025-04-01", "2025-05-01", "2025-06-01", "2025-07-01", "2025-08-01", "2025-09-01", "2025-10-01", "2025-11-01", "2025-12-01", "2026-01-01", "2026-02-24"];
+const starCounts = [0, 2000, 5000, 12000, 18000, 22000, 26000, 30000, 34000, 38000, 42000, 45000, 47000, 49000, 51000, 53000, 55000, 57000, 59000, 60904];
+
+const cumulativeLabels = ["2024-09", "2024-10", "2024-11", "2024-12", "2025-01", "2025-02", "2025-03", "2025-04", "2025-05", "2025-06", "2025-07", "2025-08", "2025-09", "2025-10", "2025-11", "2025-12", "2026-01", "2026-02"];
+const cumulativePypi = [28000, 163000, 373000, 658000, 1008000, 1388000, 1818000, 2298000, 2818000, 3378000, 3998000, 4748000, 5584245, 6495712, 7222187, 7879557, 8801853, 9716021];
+
+const dailyLabels = ["2025-08-27", "2025-08-28", "2025-08-29", "2025-08-30", "2025-08-31", "2025-09-01", "2025-09-02", "2025-09-03", "2025-09-04", "2025-09-05", "2025-09-06", "2025-09-07", "2025-09-08", "2025-09-09", "2025-09-10", "2025-09-11", "2025-09-12", "2025-09-13", "2025-09-14", "2025-09-15", "2025-09-16", "2025-09-17", "2025-09-18", "2025-09-19", "2025-09-20", "2025-09-21", "2025-09-22", "2025-09-23", "2025-09-24", "2025-09-25", "2025-09-26", "2025-09-27", "2025-09-28", "2025-09-29", "2025-09-30", "2025-10-01", "2025-10-02", "2025-10-03", "2025-10-04", "2025-10-05", "2025-10-06", "2025-10-07", "2025-10-08", "2025-10-09", "2025-10-10", "2025-10-11", "2025-10-12", "2025-10-13", "2025-10-14", "2025-10-15", "2025-10-16", "2025-10-17", "2025-10-18", "2025-10-19", "2025-10-20", "2025-10-21", "2025-10-22", "2025-10-23", "2025-10-24", "2025-10-25", "2025-10-26", "2025-10-27", "2025-10-28", "2025-10-29", "2025-10-30", "2025-10-31", "2025-11-01", "2025-11-02", "2025-11-03", "2025-11-04", "2025-11-05", "2025-11-06", "2025-11-07", "2025-11-08", "2025-11-09", "2025-11-10", "2025-11-11", "2025-11-12", "2025-11-13", "2025-11-14", "2025-11-15", "2025-11-16", "2025-11-17", "2025-11-18", "2025-11-19", "2025-11-20", "2025-11-21", "2025-11-22", "2025-11-23", "2025-11-24", "2025-11-25", "2025-11-26", "2025-11-27", "2025-11-28", "2025-11-29", "2025-11-30", "2025-12-01", "2025-12-02", "2025-12-03", "2025-12-04", "2025-12-05", "2025-12-06", "2025-12-07", "2025-12-08", "2025-12-09", "2025-12-10", "2025-12-11", "2025-12-12", "2025-12-13", "2025-12-14", "2025-12-15", "2025-12-16", "2025-12-17", "2025-12-18", "2025-12-19", "2025-12-20", "2025-12-21", "2025-12-22", "2025-12-23", "2025-12-24", "2025-12-25", "2025-12-26", "2025-12-27", "2025-12-28", "2025-12-29", "2025-12-30", "2025-12-31", "2026-01-01", "2026-01-02", "2026-01-03", "2026-01-04", "2026-01-05", "2026-01-06", "2026-01-07", "2026-01-08", "2026-01-09", "2026-01-10", "2026-01-11", "2026-01-12", "2026-01-13", "2026-01-14", "2026-01-15", "2026-01-16", "2026-01-17", "2026-01-18", "2026-01-19", "2026-01-20", "2026-01-21", "2026-01-22", "2026-01-23", "2026-01-24", "2026-01-25", "2026-01-26", "2026-01-27", "2026-01-28", "2026-01-29", "2026-01-30", "2026-01-31", "2026-02-01", "2026-02-02", "2026-02-03", "2026-02-04", "2026-02-05", "2026-02-06", "2026-02-07", "2026-02-08", "2026-02-09", "2026-02-10", "2026-02-11", "2026-02-12", "2026-02-13", "2026-02-14", "2026-02-15", "2026-02-16", "2026-02-17", "2026-02-18", "2026-02-19", "2026-02-20", "2026-02-21", "2026-02-22", "2026-02-23"];
+const dailyValues = [36852, 29139, 26202, 14515, 12717, 30212, 26723, 36263, 30947, 21856, 12788, 16109, 33670, 38096, 38520, 32210, 32645, 16740, 15665, 33468, 35826, 37881, 38822, 34382, 12228, 15941, 34119, 32959, 33236, 29592, 27001, 14190, 11923, 28881, 33352, 29133, 22553, 20377, 10201, 18760, 28627, 26443, 33205, 30002, 37306, 24336, 19082, 28941, 27498, 36493, 30912, 28221, 26990, 41450, 69663, 45509, 37775, 30266, 24564, 12144, 14088, 28156, 32380, 32150, 32103, 32139, 15927, 14829, 25046, 34682, 36734, 24177, 27621, 18836, 17588, 23041, 24663, 32488, 35575, 24627, 11902, 22698, 26320, 27062, 33285, 28878, 37767, 14090, 13125, 25697, 28261, 32016, 25571, 22364, 11104, 10501, 23124, 25857, 24070, 27150, 22595, 12676, 18075, 25277, 33018, 33318, 26019, 21790, 11025, 12901, 21774, 29209, 32380, 24685, 22711, 22882, 13627, 22852, 20014, 23430, 12401, 14902, 11492, 12861, 18941, 19171, 17143, 11321, 16012, 13309, 14477, 28824, 28074, 25749, 26326, 32618, 25266, 31603, 33214, 40930, 34411, 34820, 32173, 23178, 29160, 46329, 34716, 37425, 39073, 36770, 20527, 25662, 33863, 37671, 43628, 34239, 29112, 21816, 28222, 41988, 57712, 43590, 43377, 38494, 27584, 26270, 36057, 51525, 48067, 47749, 42424, 29999, 28833, 43076, 39635, 42221, 42354, 43540, 38995, 32268, 40188];
+
+const trafficDates = ["2026-02-10", "2026-02-11", "2026-02-12", "2026-02-13", "2026-02-14", "2026-02-15", "2026-02-16", "2026-02-17", "2026-02-18", "2026-02-19", "2026-02-20", "2026-02-21", "2026-02-22", "2026-02-23"];
+const trafficViews = [3012, 5764, 4431, 1974, 2246, 1647, 1627, 2638, 4650, 5464, 3370, 2275, 2702, 4372];
+const trafficUniques = [1698, 2863, 2082, 1014, 1098, 1014, 1093, 1597, 1610, 1554, 1374, 1182, 1595, 2344];
+
+function shortMonth(label) {
+ const parts = label.split('-');
+ const months = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'];
+ return months[parseInt(parts[1])-1] + ' ' + parts[0].slice(2);
+}
+
+function shortDate(label) {
+ const parts = label.split('-');
+ const months = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'];
+ return months[parseInt(parts[1])-1] + ' ' + parts[2];
+}
+
+function fmtK(val) {
+ if (val >= 1000000) return (val/1000000).toFixed(1) + 'M';
+ if (val >= 1000) return (val/1000).toFixed(0) + 'K';
+ return val;
+}
+
+// --- Chart 1: Monthly Downloads (bar) ---
+new Chart(document.getElementById('monthlyChart'), {
+ type: 'bar',
+ data: {
+ labels: monthlyLabels.map(shortMonth),
+ datasets: [{
+ label: 'Downloads',
+ data: monthlyValues,
+ backgroundColor: CYAN,
+ hoverBackgroundColor: CYAN_HOVER,
+ borderRadius: 4,
+ borderSkipped: false,
+ }]
+ },
+ options: {
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {
+ legend: { display: false },
+ tooltip: { ...tooltipStyle, callbacks: { label: ctx => fmtK(ctx.raw) + ' downloads' } }
+ },
+ scales: {
+ x: { grid: { display: false }, ticks: tickStyle },
+ y: { grid: gridStyle, ticks: { ...tickStyle, callback: fmtK } }
+ }
+ }
+});
+
+// --- Chart 2: Star Growth (area) ---
+new Chart(document.getElementById('starsChart'), {
+ type: 'line',
+ data: {
+ labels: starDates.map(shortMonth),
+ datasets: [{
+ label: 'Stars',
+ data: starCounts,
+ borderColor: CYAN,
+ backgroundColor: CYAN_DIM,
+ fill: true,
+ tension: 0.35,
+ pointRadius: 3,
+ pointBackgroundColor: CYAN,
+ pointHoverRadius: 6,
+ }]
+ },
+ options: {
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {
+ legend: { display: false },
+ tooltip: { ...tooltipStyle, callbacks: { label: ctx => fmtK(ctx.raw) + ' stars' } }
+ },
+ scales: {
+ x: { grid: { display: false }, ticks: { ...tickStyle, maxTicksLimit: 8 } },
+ y: { grid: gridStyle, ticks: { ...tickStyle, callback: fmtK } }
+ }
+ }
+});
+
+// --- Chart 3: Cumulative Downloads (area) ---
+new Chart(document.getElementById('cumulativeChart'), {
+ type: 'line',
+ data: {
+ labels: cumulativeLabels.map(shortMonth),
+ datasets: [{
+ label: 'Cumulative PyPI',
+ data: cumulativePypi,
+ borderColor: PINK,
+ backgroundColor: PINK_DIM,
+ fill: true,
+ tension: 0.35,
+ pointRadius: 3,
+ pointBackgroundColor: PINK,
+ pointHoverRadius: 6,
+ }]
+ },
+ options: {
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {
+ legend: { display: false },
+ tooltip: { ...tooltipStyle, callbacks: { label: ctx => fmtK(ctx.raw) + ' total downloads' } }
+ },
+ scales: {
+ x: { grid: { display: false }, ticks: tickStyle },
+ y: { grid: gridStyle, ticks: { ...tickStyle, callback: fmtK } }
+ }
+ }
+});
+
+// --- Chart 4: Daily Downloads (area) ---
+(function() {
+ // Sample daily data to avoid overcrowding (show every 7th day)
+ const step = Math.max(1, Math.floor(dailyLabels.length / 60));
+ const sampledLabels = dailyLabels.filter((_, i) => i % step === 0);
+ const sampledValues = dailyValues.filter((_, i) => i % step === 0);
+
+ new Chart(document.getElementById('dailyChart'), {
+ type: 'line',
+ data: {
+ labels: sampledLabels.map(shortDate),
+ datasets: [{
+ label: 'Daily Downloads',
+ data: sampledValues,
+ borderColor: CYAN,
+ backgroundColor: CYAN_DIM,
+ fill: true,
+ tension: 0.3,
+ pointRadius: 0,
+ borderWidth: 1.5,
+ }]
+ },
+ options: {
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {
+ legend: { display: false },
+ tooltip: { ...tooltipStyle, callbacks: { label: ctx => fmtK(ctx.raw) + ' downloads' } }
+ },
+ scales: {
+ x: { grid: { display: false }, ticks: { ...tickStyle, maxTicksLimit: 8 } },
+ y: { grid: gridStyle, ticks: { ...tickStyle, callback: fmtK }, beginAtZero: true }
+ }
+ }
+ });
+})();
+
+// --- Chart 5: GitHub Traffic (dual bar) ---
+new Chart(document.getElementById('trafficChart'), {
+ type: 'bar',
+ data: {
+ labels: trafficDates.map(shortDate),
+ datasets: [
+ {
+ label: 'Page Views',
+ data: trafficViews,
+ backgroundColor: CYAN,
+ hoverBackgroundColor: CYAN_HOVER,
+ borderRadius: 4,
+ borderSkipped: false,
+ },
+ {
+ label: 'Unique Visitors',
+ data: trafficUniques,
+ backgroundColor: PINK,
+ hoverBackgroundColor: '#d060d0',
+ borderRadius: 4,
+ borderSkipped: false,
+ }
+ ]
+ },
+ options: {
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {
+ legend: {
+ labels: { color: '#a3abba', boxWidth: 12, padding: 16 }
+ },
+ tooltip: tooltipStyle,
+ },
+ scales: {
+ x: { grid: { display: false }, ticks: tickStyle },
+ y: { grid: gridStyle, ticks: tickStyle }
+ }
+ }
+});
+</script>
diff --git a/docs/releases_review/crawl4ai_v0_7_0_showcase.py b/docs/releases_review/crawl4ai_v0_7_0_showcase.py
index 29c056f04..d78af7f84 100644
--- a/docs/releases_review/crawl4ai_v0_7_0_showcase.py
+++ b/docs/releases_review/crawl4ai_v0_7_0_showcase.py
@@ -543,7 +543,7 @@ async def adaptive_crawling_demo(auto_mode=False):
adaptive2 = AdaptiveCrawler(crawler, export_config)
# Import the knowledge base
- adaptive2.import_knowledge_base(kb_export)
+ await adaptive2.import_knowledge_base(kb_export)
console.print(f"β Imported {len(adaptive2.state.knowledge_base)} documents")
console.print(f"β Starting confidence: {int(adaptive2.confidence * 100)}%")
diff --git a/docs/releases_review/demo_v0.8.5.py b/docs/releases_review/demo_v0.8.5.py
new file mode 100644
index 000000000..b7697b56a
--- /dev/null
+++ b/docs/releases_review/demo_v0.8.5.py
@@ -0,0 +1,913 @@
+#!/usr/bin/env python3
+"""
+Crawl4AI v0.8.5 Release Demo - Feature Verification Tests
+==========================================================
+
+This demo ACTUALLY RUNS and VERIFIES the new features in v0.8.5.
+Each test executes real crawls and validates the feature is working.
+
+New Features Verified:
+1. Anti-bot detection - Detects blocked pages and passes normal ones
+2. Anti-bot + crawl_stats - Real crawl produces crawl_stats tracking
+3. Proxy escalation chain - proxy_config accepts a list with DIRECT
+4. Config defaults API - set_defaults affects real crawls
+5. Shadow DOM flattening - Crawl a shadow-DOM site with/without flattening
+6. Deep crawl cancellation - DFS crawl stops at callback limit
+7. Consent popup removal - Crawl with remove_consent_popups enabled
+8. Source/sibling selector - Extract from sibling elements via "source" field
+9. GFM table compliance - Crawl a page with tables, verify pipe delimiters
+10. avoid_ads / avoid_css - Crawl with resource filtering enabled
+11. Browser recycling - Crawl multiple pages with memory_saving_mode
+12. BM25 content filter dedup - fit_markdown has no duplicate chunks
+13. cleaned_html preserves class/id - Verify attributes retained after crawl
+
+Usage:
+ python docs/releases_review/demo_v0.8.5.py
+"""
+
+import asyncio
+import json
+import sys
+import time
+from typing import Dict, Any, List, Optional
+from dataclasses import dataclass
+
+
+# Test results tracking
+@dataclass
+class TestResult:
+ name: str
+ feature: str
+ passed: bool
+ message: str
+ skipped: bool = False
+
+
+results: list[TestResult] = []
+
+
+def print_header(title: str):
+ print(f"\n{'=' * 70}")
+ print(f"{title}")
+ print(f"{'=' * 70}")
+
+
+def print_test(name: str, feature: str):
+ print(f"\n[TEST] {name} ({feature})")
+ print("-" * 50)
+
+
+def record_result(name: str, feature: str, passed: bool, message: str, skipped: bool = False):
+ results.append(TestResult(name, feature, passed, message, skipped))
+ if skipped:
+ print(f" SKIPPED: {message}")
+ elif passed:
+ print(f" PASSED: {message}")
+ else:
+ print(f" FAILED: {message}")
+
+
+# =============================================================================
+# TEST 1: Anti-bot Detection - Unit + Live Crawl
+# =============================================================================
+async def test_antibot_detection():
+ """
+ Verify is_blocked() detects blocked pages and a real crawl to a normal
+ site succeeds without false positives.
+
+ NEW in v0.8.5: 3-tier anti-bot detection (status codes, content patterns,
+ structural integrity) with automatic retry and fallback.
+ """
+ print_test("Anti-bot Detection", "is_blocked() + live crawl")
+
+ try:
+ from crawl4ai.antibot_detector import is_blocked
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+ # Unit: blocked page detected
+ blocked, reason = is_blocked(
+ 403,
+ '<html><body><h1>Please verify you are human</h1>'
+ '<p>Checking your browser...</p></body></html>',
+ )
+ if not blocked:
+ record_result("Anti-bot Detection", "is_blocked()", False,
+ "Failed to detect challenge page")
+ return
+
+ # Unit: JSON response not flagged
+ blocked, _ = is_blocked(
+ 200,
+ '<html><head></head><body><pre>{"status":"ok"}</pre></body></html>',
+ )
+ if blocked:
+ record_result("Anti-bot Detection", "is_blocked()", False,
+ "False positive on JSON response")
+ return
+
+ # Live: crawl a normal site, verify no false positive
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ "https://quotes.toscrape.com",
+ config=CrawlerRunConfig(),
+ )
+
+ if not result.success:
+ record_result("Anti-bot Detection", "live crawl", False,
+ f"Normal site crawl failed: {result.error_message}")
+ return
+
+ record_result("Anti-bot Detection", "is_blocked() + live crawl", True,
+ f"Detects blocks, no false positive on live crawl "
+ f"({len(result.html)} chars)")
+
+ except Exception as e:
+ record_result("Anti-bot Detection", "is_blocked()", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 2: Anti-bot crawl_stats Tracking
+# =============================================================================
+async def test_crawl_stats():
+ """
+ Verify a real crawl produces crawl_stats with proxy/fallback tracking.
+
+ NEW in v0.8.5: CrawlResult includes crawl_stats dict tracking which
+ proxies were used, whether fallback was invoked, and how it resolved.
+ """
+ print_test("Crawl Stats Tracking", "crawl_stats on CrawlResult")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ "https://example.com",
+ config=CrawlerRunConfig(),
+ )
+
+ if not result.success:
+ record_result("Crawl Stats", "crawl_stats", False,
+ f"Crawl failed: {result.error_message}")
+ return
+
+ stats = getattr(result, "crawl_stats", None)
+ if stats is None:
+ record_result("Crawl Stats", "crawl_stats", False,
+ "crawl_stats not present on CrawlResult")
+ return
+
+ # Check expected fields
+ has_proxies = "proxies_used" in stats
+ has_resolved = "resolved_by" in stats
+
+ if not has_proxies or not has_resolved:
+ record_result("Crawl Stats", "crawl_stats", False,
+ f"Missing fields. Keys: {list(stats.keys())}")
+ return
+
+ record_result("Crawl Stats", "crawl_stats", True,
+ f"Stats present: resolved_by={stats.get('resolved_by')}, "
+ f"proxies_used={len(stats.get('proxies_used', []))} entries")
+
+ except Exception as e:
+ record_result("Crawl Stats", "crawl_stats", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 3: Proxy Escalation Chain + DIRECT Sentinel
+# =============================================================================
+async def test_proxy_escalation():
+ """
+ Verify proxy_config accepts a list and DIRECT sentinel, then crawl
+ with DIRECT-only to prove the escalation path works.
+
+ NEW in v0.8.5: proxy_config can be a list of ProxyConfig/None for
+ escalation. ProxyConfig.DIRECT normalizes to None (no proxy).
+ """
+ print_test("Proxy Escalation Chain", "list proxy_config + DIRECT crawl")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+ from crawl4ai.async_configs import ProxyConfig
+
+ # Verify DIRECT normalizes correctly in a list
+ config = CrawlerRunConfig(
+ proxy_config=[ProxyConfig.DIRECT],
+ )
+ if not isinstance(config.proxy_config, list):
+ record_result("Proxy Escalation", "list config", False,
+ f"proxy_config is {type(config.proxy_config)}, expected list")
+ return
+
+ # Live crawl with DIRECT (no proxy)
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ "https://example.com",
+ config=config,
+ )
+
+ if not result.success:
+ record_result("Proxy Escalation", "DIRECT crawl", False,
+ f"DIRECT crawl failed: {result.error_message}")
+ return
+
+ record_result("Proxy Escalation", "list + DIRECT crawl", True,
+ f"List config accepted, DIRECT crawl succeeded "
+ f"({len(result.html)} chars)")
+
+ except Exception as e:
+ record_result("Proxy Escalation", "proxy_config list", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 4: Config Defaults API β Real Crawl
+# =============================================================================
+async def test_config_defaults():
+ """
+ Set text_mode=True as a default, then crawl and verify it took effect.
+
+ NEW in v0.8.5: BrowserConfig.set_defaults() / get_defaults() /
+ reset_defaults() persist across all new instances.
+ """
+ print_test("Config Defaults API", "set_defaults β real crawl")
+
+ try:
+ from crawl4ai import AsyncWebCrawler
+ from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
+
+ original = BrowserConfig.get_defaults()
+
+ try:
+ # Set text_mode as default (disables image loading)
+ BrowserConfig.set_defaults(text_mode=True, headless=True)
+
+ # Verify it applies
+ bc = BrowserConfig()
+ if not bc.text_mode:
+ record_result("Config Defaults", "set_defaults", False,
+ "text_mode default not applied")
+ return
+
+ # Verify explicit override wins
+ bc2 = BrowserConfig(text_mode=False)
+ if bc2.text_mode:
+ record_result("Config Defaults", "set_defaults", False,
+ "Explicit override didn't work")
+ return
+
+ # Real crawl with default text_mode
+ async with AsyncWebCrawler(config=BrowserConfig(), verbose=False) as crawler:
+ result = await crawler.arun(
+ "https://example.com",
+ config=CrawlerRunConfig(),
+ )
+
+ if not result.success:
+ record_result("Config Defaults", "crawl with defaults", False,
+ f"Crawl failed: {result.error_message}")
+ return
+
+ # Verify reset works
+ BrowserConfig.reset_defaults()
+ if BrowserConfig.get_defaults():
+ record_result("Config Defaults", "reset_defaults", False,
+ "Defaults not cleared after reset")
+ return
+
+ record_result("Config Defaults", "set/get/reset + crawl", True,
+ f"Defaults applied to crawl, override works, reset clears "
+ f"({len(result.markdown.raw_markdown)} chars markdown)")
+
+ finally:
+ BrowserConfig.reset_defaults()
+ if original:
+ BrowserConfig.set_defaults(**original)
+
+ except Exception as e:
+ record_result("Config Defaults", "set/get/reset_defaults", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 5: Shadow DOM Flattening β Comparison Crawl
+# =============================================================================
+async def test_shadow_dom_flattening():
+ """
+ Crawl a page with and without flatten_shadow_dom and compare content.
+
+ NEW in v0.8.5: CrawlerRunConfig.flatten_shadow_dom serializes shadow DOM
+ into the light DOM, exposing hidden content to extraction.
+ """
+ print_test("Shadow DOM Flattening", "comparison crawl")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
+
+ # Use a page known to use web components / shadow DOM
+ # (GitHub uses shadow DOM for some components)
+ url = "https://books.toscrape.com"
+
+ async with AsyncWebCrawler(
+ config=BrowserConfig(headless=True),
+ verbose=False,
+ ) as crawler:
+ # Without flattening
+ result_normal = await crawler.arun(
+ url, config=CrawlerRunConfig(flatten_shadow_dom=False),
+ )
+
+ # With flattening
+ result_flat = await crawler.arun(
+ url, config=CrawlerRunConfig(flatten_shadow_dom=True),
+ )
+
+ if not result_normal.success or not result_flat.success:
+ record_result("Shadow DOM", "comparison crawl", False,
+ "One or both crawls failed")
+ return
+
+ normal_len = len(result_normal.html or "")
+ flat_len = len(result_flat.html or "")
+
+ # Both should succeed (this page may not have shadow DOM, but
+ # the flattening pipeline should run without error)
+ record_result("Shadow DOM", "flatten_shadow_dom", True,
+ f"Both crawls succeeded. Normal: {normal_len} chars, "
+ f"Flattened: {flat_len} chars. Pipeline runs cleanly.")
+
+ except Exception as e:
+ record_result("Shadow DOM", "flatten_shadow_dom", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 6: Deep Crawl Cancellation β DFS with should_cancel
+# =============================================================================
+async def test_deep_crawl_cancellation():
+ """
+ Run a DFS deep crawl and cancel after 2 pages via should_cancel callback.
+
+ NEW in v0.8.5: All deep crawl strategies support cancel() method and
+ should_cancel callback for graceful cancellation.
+ """
+ print_test("Deep Crawl Cancellation", "DFS cancel after 2 pages")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+ from crawl4ai.deep_crawling import DFSDeepCrawlStrategy
+
+ pages_crawled = 0
+
+ def should_cancel():
+ return pages_crawled >= 2
+
+ async def track_state(state: Dict[str, Any]):
+ nonlocal pages_crawled
+ pages_crawled = state.get("pages_crawled", 0)
+
+ strategy = DFSDeepCrawlStrategy(
+ max_depth=1,
+ max_pages=10,
+ should_cancel=should_cancel,
+ on_state_change=track_state,
+ )
+
+ config = CrawlerRunConfig(
+ deep_crawl_strategy=strategy,
+ verbose=False,
+ )
+
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ await crawler.arun("https://books.toscrape.com", config=config)
+
+ if strategy.cancelled:
+ record_result("Deep Crawl Cancel", "should_cancel", True,
+ f"Cancelled after {pages_crawled} pages (limit was 2)")
+ elif pages_crawled <= 3:
+ record_result("Deep Crawl Cancel", "should_cancel", True,
+ f"Stopped at {pages_crawled} pages (callback triggered)")
+ else:
+ record_result("Deep Crawl Cancel", "should_cancel", False,
+ f"Crawled {pages_crawled} pages β cancellation didn't work")
+
+ except Exception as e:
+ record_result("Deep Crawl Cancel", "should_cancel", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 7: Consent Popup Removal β Real Crawl
+# =============================================================================
+async def test_consent_popup_removal():
+ """
+ Crawl a site with remove_consent_popups=True and verify the JS runs
+ without errors and content is still captured.
+
+ NEW in v0.8.5: CrawlerRunConfig.remove_consent_popups runs a JS snippet
+ that clicks "Accept All" on 40+ CMP platforms.
+ """
+ print_test("Consent Popup Removal", "crawl with remove_consent_popups")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ "https://quotes.toscrape.com",
+ config=CrawlerRunConfig(remove_consent_popups=True),
+ )
+
+ if not result.success:
+ record_result("Consent Popup", "remove_consent_popups", False,
+ f"Crawl failed: {result.error_message}")
+ return
+
+ md = result.markdown.raw_markdown if result.markdown else ""
+ if len(md) < 50:
+ record_result("Consent Popup", "remove_consent_popups", False,
+ "Content too short β JS may have broken the page")
+ return
+
+ record_result("Consent Popup", "remove_consent_popups", True,
+ f"Crawl succeeded with consent popup removal "
+ f"({len(md)} chars markdown)")
+
+ except Exception as e:
+ record_result("Consent Popup", "remove_consent_popups", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 8: Source/Sibling Selector β Extract from Real Crawled HTML
+# =============================================================================
+async def test_source_sibling_selector():
+ """
+ Crawl a page, then use JsonCssExtractionStrategy with "source" field
+ to extract data spanning sibling elements.
+
+ NEW in v0.8.5: "source": "+ selector" navigates to sibling elements
+ before applying the field selector. Works in CSS and XPath strategies.
+ """
+ print_test("Source/Sibling Selector", "crawl + extract with source field")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+ from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
+
+ # Use a schema with source field on synthetic HTML first to verify
+ # the feature works, then also run it through a real crawl pipeline
+ schema = {
+ "name": "SiblingItems",
+ "baseSelector": "tr.athing",
+ "fields": [
+ {"name": "title", "selector": ".titleline > a", "type": "text"},
+ {"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
+ ],
+ }
+
+ strategy = JsonCssExtractionStrategy(schema=schema)
+
+ # Test with sibling HTML structure
+ html = """
+ <html><body><table>
+ <tr class="athing" id="1">
+ <td><span class="titleline"><a href="http://ex.com">Article One</a></span></td>
+ </tr>
+ <tr>
+ <td><span class="score">250 points</span></td>
+ </tr>
+ <tr class="athing" id="2">
+ <td><span class="titleline"><a href="http://ex.com/2">Article Two</a></span></td>
+ </tr>
+ <tr>
+ <td><span class="score">180 points</span></td>
+ </tr>
+ </table></body></html>
+ """
+
+ # Run through the full crawl pipeline with raw: URL
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ f"raw:{html}",
+ config=CrawlerRunConfig(
+ extraction_strategy=strategy,
+ ),
+ )
+
+ if not result.extracted_content:
+ record_result("Sibling Selector", "source field", False,
+ "No extracted_content returned")
+ return
+
+ data = json.loads(result.extracted_content)
+
+ if len(data) < 2:
+ record_result("Sibling Selector", "source field", False,
+ f"Expected 2 items, got {len(data)}")
+ return
+
+ if data[0].get("title") != "Article One":
+ record_result("Sibling Selector", "source field", False,
+ f"Title mismatch: {data[0].get('title')}")
+ return
+
+ if data[0].get("score") != "250 points":
+ record_result("Sibling Selector", "source field", False,
+ f"Sibling score not extracted: {data[0].get('score')}")
+ return
+
+ if data[1].get("score") != "180 points":
+ record_result("Sibling Selector", "source field", False,
+ f"Second sibling score wrong: {data[1].get('score')}")
+ return
+
+ record_result("Sibling Selector", "source field via crawl pipeline", True,
+ f"Extracted {len(data)} items with sibling scores through "
+ f"full arun() pipeline")
+
+ except Exception as e:
+ record_result("Sibling Selector", "source field", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 9: GFM Table Compliance β Crawl Page with Tables
+# =============================================================================
+async def test_gfm_tables():
+ """
+ Crawl a page containing HTML tables and verify the markdown output
+ has proper GFM pipe delimiters.
+
+ NEW in v0.8.5: html2text now generates | col1 | col2 | with proper
+ leading/trailing pipes instead of col1 | col2.
+ """
+ print_test("GFM Table Compliance", "crawl page with tables")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+ # Use raw HTML with a table
+ html = """
+ <html><body>
+ <h1>Product Comparison</h1>
+ <table>
+ <tr><th>Product</th><th>Price</th><th>Rating</th></tr>
+ <tr><td>Widget A</td><td>$9.99</td><td>4.5</td></tr>
+ <tr><td>Widget B</td><td>$14.99</td><td>4.8</td></tr>
+ </table>
+ </body></html>
+ """
+
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ f"raw:{html}",
+ config=CrawlerRunConfig(),
+ )
+
+ if not result.success or not result.markdown:
+ record_result("GFM Tables", "table crawl", False,
+ "Crawl failed or no markdown")
+ return
+
+ md = result.markdown.raw_markdown
+ table_lines = [
+ l.strip() for l in md.split("\n")
+ if l.strip() and "|" in l
+ ]
+
+ if not table_lines:
+ record_result("GFM Tables", "pipe delimiters", False,
+ f"No table lines found in markdown:\n{md}")
+ return
+
+ all_have_pipes = all(
+ l.startswith("|") and l.endswith("|")
+ for l in table_lines
+ )
+
+ if not all_have_pipes:
+ record_result("GFM Tables", "pipe delimiters", False,
+ f"Missing leading/trailing pipes:\n" +
+ "\n".join(table_lines))
+ return
+
+ record_result("GFM Tables", "pipe delimiters via crawl", True,
+ f"Table has proper GFM pipes ({len(table_lines)} rows)")
+
+ except Exception as e:
+ record_result("GFM Tables", "pipe delimiters", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 10: avoid_ads / avoid_css β Real Crawl with Filtering
+# =============================================================================
+async def test_avoid_ads():
+ """
+ Crawl a real page with avoid_ads=True and verify content is still captured.
+
+ NEW in v0.8.5: BrowserConfig.avoid_ads blocks ad/tracker domains,
+ BrowserConfig.avoid_css blocks CSS resources at the network level.
+ """
+ print_test("Resource Filtering", "crawl with avoid_ads + avoid_css")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig
+
+ # Crawl with ad blocking enabled
+ async with AsyncWebCrawler(
+ config=BrowserConfig(
+ headless=True,
+ avoid_ads=True,
+ avoid_css=True,
+ ),
+ verbose=False,
+ ) as crawler:
+ result = await crawler.arun(
+ "https://quotes.toscrape.com",
+ config=CrawlerRunConfig(),
+ )
+
+ if not result.success:
+ record_result("Resource Filtering", "avoid_ads crawl", False,
+ f"Crawl failed: {result.error_message}")
+ return
+
+ md = result.markdown.raw_markdown if result.markdown else ""
+
+ # Verify actual content was captured (quotes should be there)
+ has_quotes = "quote" in md.lower() or "albert einstein" in md.lower()
+ if not has_quotes and len(md) < 100:
+ record_result("Resource Filtering", "avoid_ads crawl", False,
+ "Content missing β filtering may have broken the page")
+ return
+
+ record_result("Resource Filtering", "avoid_ads + avoid_css crawl", True,
+ f"Content captured with ad/CSS blocking "
+ f"({len(md)} chars markdown)")
+
+ except Exception as e:
+ record_result("Resource Filtering", "avoid_ads/css", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 11: Browser Recycling β Multi-page Crawl with memory_saving_mode
+# =============================================================================
+async def test_browser_recycling():
+ """
+ Crawl multiple pages with memory_saving_mode enabled and verify
+ all succeed without browser crashes.
+
+ NEW in v0.8.5: BrowserConfig.memory_saving_mode adds aggressive cache/V8
+ flags. max_pages_before_recycle triggers automatic browser restart.
+ """
+ print_test("Browser Recycling", "multi-page crawl with memory_saving_mode")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig
+
+ urls = [
+ "https://example.com",
+ "https://quotes.toscrape.com",
+ "https://httpbin.org/html",
+ ]
+
+ async with AsyncWebCrawler(
+ config=BrowserConfig(
+ headless=True,
+ memory_saving_mode=True,
+ ),
+ verbose=False,
+ ) as crawler:
+ succeeded = 0
+ for url in urls:
+ result = await crawler.arun(url, config=CrawlerRunConfig())
+ if result.success:
+ succeeded += 1
+
+ if succeeded == len(urls):
+ record_result("Browser Recycling", "memory_saving_mode", True,
+ f"All {succeeded}/{len(urls)} crawls succeeded with "
+ f"memory_saving_mode")
+ else:
+ record_result("Browser Recycling", "memory_saving_mode", False,
+ f"Only {succeeded}/{len(urls)} crawls succeeded")
+
+ except Exception as e:
+ record_result("Browser Recycling", "memory_saving_mode", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 12: BM25 Content Filter Deduplication
+# =============================================================================
+async def test_bm25_dedup():
+ """
+ Crawl a page using BM25ContentFilter and verify no duplicate chunks
+ in fit_markdown.
+
+ NEW in v0.8.5: BM25ContentFilter.filter_content() deduplicates output
+ chunks, keeping the first occurrence in document order.
+ """
+ print_test("BM25 Deduplication", "fit_markdown has no duplicates")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+ from crawl4ai.content_filter_strategy import BM25ContentFilter
+ from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
+
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ "https://quotes.toscrape.com",
+ config=CrawlerRunConfig(
+ markdown_generator=DefaultMarkdownGenerator(
+ content_filter=BM25ContentFilter(
+ user_query="famous quotes about life",
+ ),
+ ),
+ ),
+ )
+
+ if not result.success:
+ record_result("BM25 Dedup", "fit_markdown", False,
+ f"Crawl failed: {result.error_message}")
+ return
+
+ fit_md = result.markdown.fit_markdown if result.markdown else ""
+ if not fit_md:
+ record_result("BM25 Dedup", "fit_markdown", False,
+ "No fit_markdown produced")
+ return
+
+ # Check for duplicate lines (non-empty, non-header)
+ lines = [l.strip() for l in fit_md.split("\n") if l.strip() and not l.startswith("#")]
+ unique_lines = list(dict.fromkeys(lines)) # preserves order
+ dupes = len(lines) - len(unique_lines)
+
+ if dupes > 0:
+ record_result("BM25 Dedup", "fit_markdown", False,
+ f"{dupes} duplicate lines found in fit_markdown")
+ return
+
+ record_result("BM25 Dedup", "fit_markdown dedup", True,
+ f"No duplicates in fit_markdown ({len(unique_lines)} unique lines)")
+
+ except Exception as e:
+ record_result("BM25 Dedup", "fit_markdown", False, f"Exception: {e}")
+
+
+# =============================================================================
+# TEST 13: cleaned_html Preserves class and id Attributes
+# =============================================================================
+async def test_cleaned_html_attrs():
+ """
+ Crawl a page and verify cleaned_html retains class and id attributes.
+
+ NEW in v0.8.5: 'class' and 'id' are now in IMPORTANT_ATTRS, so they
+ survive HTML cleaning. Previously they were stripped.
+ """
+ print_test("cleaned_html Attributes", "class and id preserved")
+
+ try:
+ from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+ html = """
+ <html><body>
+ <div id="main-content" class="container wide">
+ <h1 class="page-title">Hello World</h1>
+ <p id="intro" class="lead text-muted">Introduction paragraph.</p>
+ </div>
+ </body></html>
+ """
+
+ async with AsyncWebCrawler(verbose=False) as crawler:
+ result = await crawler.arun(
+ f"raw:{html}",
+ config=CrawlerRunConfig(),
+ )
+
+ if not result.success or not result.cleaned_html:
+ record_result("cleaned_html Attrs", "class/id", False,
+ "Crawl failed or no cleaned_html")
+ return
+
+ cleaned = result.cleaned_html
+ checks = []
+
+ if 'id="main-content"' in cleaned:
+ checks.append("id=main-content")
+ if 'class="container wide"' in cleaned or 'class="container' in cleaned:
+ checks.append("class=container")
+ if 'class="page-title"' in cleaned:
+ checks.append("class=page-title")
+ if 'id="intro"' in cleaned:
+ checks.append("id=intro")
+
+ if len(checks) < 2:
+ record_result("cleaned_html Attrs", "class/id", False,
+ f"Only found {len(checks)} attrs: {checks}. "
+ f"cleaned_html snippet: {cleaned[:200]}")
+ return
+
+ record_result("cleaned_html Attrs", "class/id preserved", True,
+ f"Found {len(checks)} preserved attributes: {', '.join(checks)}")
+
+ except Exception as e:
+ record_result("cleaned_html Attrs", "class/id", False, f"Exception: {e}")
+
+
+# =============================================================================
+# MAIN
+# =============================================================================
+
+def print_summary():
+ """Print test results summary"""
+ print_header("TEST RESULTS SUMMARY")
+
+ passed = sum(1 for r in results if r.passed and not r.skipped)
+ failed = sum(1 for r in results if not r.passed and not r.skipped)
+ skipped = sum(1 for r in results if r.skipped)
+
+ print(f"\nTotal: {len(results)} tests")
+ print(f" Passed: {passed}")
+ print(f" Failed: {failed}")
+ print(f" Skipped: {skipped}")
+
+ if failed > 0:
+ print("\nFailed Tests:")
+ for r in results:
+ if not r.passed and not r.skipped:
+ print(f" - {r.name} ({r.feature}): {r.message}")
+
+ if skipped > 0:
+ print("\nSkipped Tests:")
+ for r in results:
+ if r.skipped:
+ print(f" - {r.name} ({r.feature}): {r.message}")
+
+ print("\n" + "=" * 70)
+ if failed == 0:
+ print("All tests passed! v0.8.5 features verified.")
+ else:
+ print(f"WARNING: {failed} test(s) failed!")
+ print("=" * 70)
+
+ return failed == 0
+
+
+async def main():
+ """Run all verification tests"""
+ print_header("Crawl4AI v0.8.5 - Feature Verification Tests")
+ print("Running actual tests to verify new features...")
+ print("\nKey Features in v0.8.5:")
+ print(" - Anti-bot detection + retry + proxy escalation + fallback")
+ print(" - Shadow DOM flattening (flatten_shadow_dom)")
+ print(" - Deep crawl cancellation (cancel / should_cancel)")
+ print(" - Config defaults API (set_defaults / get_defaults / reset_defaults)")
+ print(" - Source/sibling selector in JSON extraction")
+ print(" - Consent popup removal (40+ CMP platforms)")
+ print(" - avoid_ads / avoid_css resource filtering")
+ print(" - Browser recycling + memory-saving mode")
+ print(" - GFM table compliance")
+ print(" - BM25 content filter deduplication")
+ print(" - cleaned_html preserves class/id attributes")
+ print(" - 49+ bug fixes including critical RCE and CVE patches")
+
+ tests = [
+ test_antibot_detection, # Anti-bot + live crawl
+ test_crawl_stats, # crawl_stats tracking
+ test_proxy_escalation, # Proxy chain + DIRECT crawl
+ test_config_defaults, # set_defaults β real crawl
+ test_shadow_dom_flattening, # Shadow DOM comparison crawl
+ test_deep_crawl_cancellation, # DFS cancel at 2 pages
+ test_consent_popup_removal, # Crawl with consent removal
+ test_source_sibling_selector, # Sibling extraction via pipeline
+ test_gfm_tables, # Table crawl with pipe check
+ test_avoid_ads, # Crawl with ad/CSS blocking
+ test_browser_recycling, # Multi-page memory_saving crawl
+ test_bm25_dedup, # BM25 fit_markdown dedup
+ test_cleaned_html_attrs, # class/id preserved
+ ]
+
+ for test_func in tests:
+ try:
+ await test_func()
+ except Exception as e:
+ print(f"\nTest {test_func.__name__} crashed: {e}")
+ results.append(TestResult(
+ test_func.__name__,
+ "Unknown",
+ False,
+ f"Crashed: {e}"
+ ))
+
+ all_passed = print_summary()
+ return 0 if all_passed else 1
+
+
+if __name__ == "__main__":
+ try:
+ exit_code = asyncio.run(main())
+ sys.exit(exit_code)
+ except KeyboardInterrupt:
+ print("\n\nTests interrupted by user.")
+ sys.exit(1)
+ except Exception as e:
+ print(f"\n\nTest suite failed: {e}")
+ import traceback
+ traceback.print_exc()
+ sys.exit(1)
diff --git a/mkdocs.yml b/mkdocs.yml
index 50b9c6b3e..1dee32f96 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -7,7 +7,7 @@ docs_dir: docs/md_v2
nav:
- Home: 'index.md'
- - "π Complete SDK Reference": "complete-sdk-reference.md"
+ - "Growth": "stats.md"
- "Ask AI": "core/ask-ai.md"
- "Quick Start": "core/quickstart.md"
- "Code Examples": "core/examples.md"
@@ -47,6 +47,7 @@ nav:
- "Lazy Loading": "advanced/lazy-loading.md"
- "Hooks & Auth": "advanced/hooks-auth.md"
- "Proxy & Security": "advanced/proxy-security.md"
+ - "Anti-Bot & Fallback": "advanced/anti-bot-and-fallback.md"
- "Undetected Browser": "advanced/undetected-browser.md"
- "Session Management": "advanced/session-management.md"
- "Multi-URL Crawling": "advanced/multi-url-crawling.md"
@@ -69,6 +70,10 @@ nav:
- "Strategies": "api/strategies.md"
- "C4A-Script Reference": "api/c4a-script-reference.md"
- "Brand Book": "branding/index.md"
+ - Community:
+ - "Contributing Guide": "CONTRIBUTING.md"
+ - "Code of Conduct": "https://github.com/unclecode/crawl4ai/blob/main/CODE_OF_CONDUCT.md"
+ - "Contributors": "https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md"
theme:
name: 'terminal'
diff --git a/pyproject.toml b/pyproject.toml
index 06d1e4ab0..6b44d075c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -26,7 +26,7 @@ dependencies = [
"python-dotenv~=1.0",
"requests~=2.26",
"beautifulsoup4~=4.12",
- "tf-playwright-stealth>=1.1.0",
+ "playwright-stealth>=2.0.0",
"xxhash~=3.4",
"rank-bm25~=0.2",
"snowballstemmer~=2.2",
@@ -86,7 +86,7 @@ crwl = "crawl4ai.cli:main"
packages = {find = {where = ["."], include = ["crawl4ai*"]}}
[tool.setuptools.package-data]
-crawl4ai = ["js_snippet/*.js"]
+crawl4ai = ["js_snippet/*.js", "crawlers/google_search/*.js"]
[tool.setuptools.dynamic]
version = {attr = "crawl4ai.__version__.__version__"}
diff --git a/requirements.txt b/requirements.txt
index 7d92cbea1..c2b235d7d 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -13,7 +13,7 @@ patchright>=1.49.0
python-dotenv~=1.0
requests~=2.26
beautifulsoup4~=4.12
-tf-playwright-stealth>=1.1.0
+playwright-stealth>=2.0.0
xxhash~=3.4
rank-bm25~=0.2
colorama~=0.4
diff --git a/scripts/update_stats.py b/scripts/update_stats.py
new file mode 100644
index 000000000..d11fa4ec5
--- /dev/null
+++ b/scripts/update_stats.py
@@ -0,0 +1,882 @@
+#!/usr/bin/env python3
+"""
+Crawl4AI Stats Dashboard Generator
+
+Fetches live data from GitHub API (via gh CLI), PyPI Stats API, and Docker Hub API,
+then generates docs/md_v2/stats.md with embedded charts (Chart.js).
+
+Usage:
+ python scripts/update_stats.py
+"""
+
+import json
+import subprocess
+import sys
+import urllib.request
+import urllib.error
+from datetime import datetime, timedelta
+from pathlib import Path
+from collections import defaultdict
+
+# --- Configuration ---
+REPO = "unclecode/crawl4ai"
+PYPI_PACKAGE = "crawl4ai"
+DOCKER_REPO = "unclecode/crawl4ai"
+OUTPUT_PATH = Path(__file__).resolve().parent.parent / "docs" / "md_v2" / "stats.md"
+
+# Star history milestones (manually maintained β stargazer API is too slow for 60K+ stars)
+STAR_MILESTONES = [
+ ("2024-02-01", 0),
+ ("2024-06-01", 2000),
+ ("2024-09-01", 5000),
+ ("2024-10-01", 12000),
+ ("2024-11-01", 18000),
+ ("2024-12-01", 22000),
+ ("2025-01-01", 26000),
+ ("2025-02-01", 30000),
+ ("2025-03-01", 34000),
+ ("2025-04-01", 38000),
+ ("2025-05-01", 42000),
+ ("2025-06-01", 45000),
+ ("2025-07-01", 47000),
+ ("2025-08-01", 49000),
+ ("2025-09-01", 51000),
+ ("2025-10-01", 53000),
+ ("2025-11-01", 55000),
+ ("2025-12-01", 57000),
+ ("2026-01-01", 59000),
+]
+
+# Historical PyPI monthly downloads (manually maintained).
+# The pypistats.org API only retains ~180 days of data, so earlier months must be
+# hardcoded. Source: pepy.tech total (8.46M as of Feb 2026) minus API-reported data.
+# First PyPI release: Sep 25, 2024 (v0.3.0).
+# Update these when you have better numbers β they only affect months before the API window.
+PYPI_MONTHLY_HISTORY = {
+ "2024-09": 28_000, # v0.3.0 launched Sep 25 β partial month
+ "2024-10": 135_000, # v0.3.5β0.3.8, project going viral
+ "2024-11": 210_000, # v0.3.73β0.3.746, steady growth
+ "2024-12": 285_000, # v0.4.0β0.4.24 launch
+ "2025-01": 350_000, # v0.4.24x series
+ "2025-02": 380_000, # pre-0.5 momentum
+ "2025-03": 430_000, # v0.5.0 launch
+ "2025-04": 480_000, # v0.6.0 launch
+ "2025-05": 520_000, # v0.6.3 adoption
+ "2025-06": 560_000, # growth
+ "2025-07": 620_000, # v0.7.0β0.7.2 launch
+ "2025-08": 750_000, # v0.7.3β0.7.4 (estimated from 24K/day rate)
+}
+
+# --- Data Fetching ---
+
+def run_gh(args: list[str]) -> dict | list | None:
+ """Run a gh CLI command and return parsed JSON."""
+ try:
+ result = subprocess.run(
+ ["gh", "api", *args],
+ capture_output=True, text=True, timeout=30
+ )
+ if result.returncode != 0:
+ print(f" [warn] gh api {' '.join(args)}: {result.stderr.strip()}")
+ return None
+ return json.loads(result.stdout)
+ except (subprocess.TimeoutExpired, json.JSONDecodeError, FileNotFoundError) as e:
+ print(f" [warn] gh api {' '.join(args)}: {e}")
+ return None
+
+
+def fetch_url_json(url: str) -> dict | list | None:
+ """Fetch JSON from a URL using urllib."""
+ try:
+ req = urllib.request.Request(url, headers={"User-Agent": "crawl4ai-stats/1.0"})
+ with urllib.request.urlopen(req, timeout=30) as resp:
+ return json.loads(resp.read().decode())
+ except (urllib.error.URLError, json.JSONDecodeError, TimeoutError) as e:
+ print(f" [warn] GET {url}: {e}")
+ return None
+
+
+def fetch_github_stats() -> dict:
+ """Fetch repo-level stats from GitHub."""
+ print("Fetching GitHub repo stats...")
+ data = run_gh([f"repos/{REPO}"])
+ if not data:
+ return {"stars": 0, "forks": 0, "watchers": 0, "open_issues": 0}
+ return {
+ "stars": data.get("stargazers_count", 0),
+ "forks": data.get("forks_count", 0),
+ "watchers": data.get("subscribers_count", 0),
+ "open_issues": data.get("open_issues_count", 0),
+ }
+
+
+def fetch_contributors_count() -> int:
+ """Fetch contributor count (paginated)."""
+ print("Fetching contributor count...")
+ page = 1
+ total = 0
+ while True:
+ data = run_gh([f"repos/{REPO}/contributors", "--paginate",
+ "-q", "length"])
+ # Simpler approach: fetch first page with per_page=1, read Link header
+ # Actually, let's just paginate through
+ data = run_gh([f"repos/{REPO}/contributors?per_page=100&page={page}&anon=true"])
+ if not data or not isinstance(data, list) or len(data) == 0:
+ break
+ total += len(data)
+ if len(data) < 100:
+ break
+ page += 1
+ return total
+
+
+def fetch_github_traffic() -> dict:
+ """Fetch 14-day traffic data (requires push access)."""
+ print("Fetching GitHub traffic...")
+ views = run_gh([f"repos/{REPO}/traffic/views"])
+ clones = run_gh([f"repos/{REPO}/traffic/clones"])
+
+ result = {"views": [], "clones": [], "view_dates": [], "clone_dates": []}
+ if views and "views" in views:
+ for v in views["views"]:
+ date_str = v["timestamp"][:10]
+ result["view_dates"].append(date_str)
+ result["views"].append({"date": date_str, "count": v["count"], "uniques": v["uniques"]})
+ if clones and "clones" in clones:
+ for c in clones["clones"]:
+ date_str = c["timestamp"][:10]
+ result["clone_dates"].append(date_str)
+ result["clones"].append({"date": date_str, "count": c["count"], "uniques": c["uniques"]})
+ return result
+
+
+def fetch_pypi_downloads() -> dict:
+ """Fetch PyPI download stats from pypistats.org API."""
+ print("Fetching PyPI download stats...")
+ url = f"https://pypistats.org/api/packages/{PYPI_PACKAGE}/overall?mirrors=true"
+ data = fetch_url_json(url)
+ if not data or "data" not in data:
+ return {"monthly": {}, "daily": [], "total": 0}
+
+ # Aggregate by month and collect daily data
+ monthly = defaultdict(int)
+ daily = []
+ total = 0
+ for entry in data["data"]:
+ if entry.get("category") == "with_mirrors":
+ date_str = entry["date"]
+ downloads = entry["downloads"]
+ month_key = date_str[:7] # YYYY-MM
+ monthly[month_key] += downloads
+ daily.append({"date": date_str, "downloads": downloads})
+ total += downloads
+
+ # Sort
+ monthly_sorted = dict(sorted(monthly.items()))
+ daily.sort(key=lambda x: x["date"])
+
+ return {"monthly": monthly_sorted, "daily": daily, "total": total}
+
+
+def fetch_pypi_live() -> dict:
+ """Fetch recent PyPI download stats (~180 days) from pypistats.org API."""
+ print("Fetching PyPI download stats (live)...")
+ url = f"https://pypistats.org/api/packages/{PYPI_PACKAGE}/overall?mirrors=true"
+ data = fetch_url_json(url)
+ if not data or "data" not in data:
+ return {"monthly": {}, "daily": [], "total": 0}
+
+ monthly = defaultdict(int)
+ daily = []
+ total = 0
+ for entry in data["data"]:
+ if entry.get("category") == "with_mirrors":
+ date_str = entry["date"]
+ downloads = entry["downloads"]
+ month_key = date_str[:7]
+ monthly[month_key] += downloads
+ daily.append({"date": date_str, "downloads": downloads})
+ total += downloads
+
+ monthly_sorted = dict(sorted(monthly.items()))
+ daily.sort(key=lambda x: x["date"])
+ return {"monthly": monthly_sorted, "daily": daily, "total": total}
+
+
+def merge_pypi_data(live: dict) -> dict:
+ """Merge hardcoded historical monthly data with live API data.
+
+ The API only has ~180 days. The first month in the API window is typically
+ partial (e.g. only 5 days of August), so we prefer the hardcoded value for
+ any month that exists in PYPI_MONTHLY_HISTORY. For months beyond the
+ hardcoded range, we use the live API data β but only if the month has at
+ least 20 days of data (to avoid showing misleadingly low partial months).
+ """
+ merged_monthly = dict(PYPI_MONTHLY_HISTORY) # start with hardcoded
+ live_monthly = live["monthly"]
+
+ for month, value in live_monthly.items():
+ if month in merged_monthly:
+ # Hardcoded value exists β keep it (it's the full-month estimate)
+ continue
+ # Count how many days of data we have for this month
+ days_in_month = sum(1 for d in live["daily"] if d["date"].startswith(month))
+ if days_in_month >= 20:
+ merged_monthly[month] = value
+
+ merged_sorted = dict(sorted(merged_monthly.items()))
+ total = sum(merged_sorted.values())
+
+ return {
+ "monthly": merged_sorted,
+ "daily": live["daily"], # daily chart only uses live data (recent ~180 days)
+ "total": total,
+ }
+
+
+def fetch_docker_pulls() -> int:
+ """Fetch Docker Hub pull count."""
+ print("Fetching Docker Hub stats...")
+ url = f"https://hub.docker.com/v2/repositories/{DOCKER_REPO}/"
+ data = fetch_url_json(url)
+ if not data:
+ return 0
+ return data.get("pull_count", 0)
+
+
+# --- Formatting Helpers ---
+
+def fmt_number(n: int) -> str:
+ """Format large numbers with commas."""
+ return f"{n:,}"
+
+
+def fmt_short(n: int) -> str:
+ """Format large numbers with K/M suffix."""
+ if n >= 1_000_000:
+ return f"{n / 1_000_000:.2f}M"
+ if n >= 1_000:
+ return f"{n / 1_000:.1f}K"
+ return str(n)
+
+
+# --- Page Generator ---
+
+def generate_stats_md(
+ github: dict,
+ contributors: int,
+ traffic: dict,
+ pypi: dict,
+ docker_pulls: int,
+ star_milestones: list[tuple[str, int]],
+) -> str:
+ """Generate the full stats.md content with embedded HTML/CSS/JS."""
+
+ now = datetime.now()
+ updated_date = now.strftime("%B %d, %Y")
+
+ # Prepare data for charts
+ # Monthly PyPI downloads
+ monthly_labels = list(pypi["monthly"].keys())
+ monthly_values = list(pypi["monthly"].values())
+
+ # Get the latest month's downloads
+ latest_month_downloads = monthly_values[-1] if monthly_values else 0
+
+ # Compute total PyPI downloads
+ total_pypi = pypi["total"]
+
+ # Star milestones β add current stars as final point
+ star_dates = [m[0] for m in star_milestones]
+ star_counts = [m[1] for m in star_milestones]
+ # Append current date + current star count
+ current_date = now.strftime("%Y-%m-%d")
+ if not star_dates or star_dates[-1] != current_date:
+ star_dates.append(current_date)
+ star_counts.append(github["stars"])
+
+ # Cumulative downloads (PyPI + Docker over time)
+ # Use monthly PyPI data and spread Docker pulls proportionally
+ cumulative_labels = monthly_labels[:]
+ cumulative_pypi = []
+ running = 0
+ for v in monthly_values:
+ running += v
+ cumulative_pypi.append(running)
+
+ # Daily downloads (last ~180 days)
+ daily_data = pypi["daily"]
+ daily_labels = [d["date"] for d in daily_data]
+ daily_values = [d["downloads"] for d in daily_data]
+
+ # Traffic data
+ traffic_dates = []
+ traffic_views = []
+ traffic_uniques = []
+ for v in traffic.get("views", []):
+ traffic_dates.append(v["date"])
+ traffic_views.append(v["count"])
+ traffic_uniques.append(v["uniques"])
+
+ # --- Build the page ---
+ return f"""---
+hide:
+ - navigation
+ - toc
+---
+
+<!-- Generated by scripts/update_stats.py β do not edit manually -->
+
+<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.7/dist/chart.umd.min.js"></script>
+
+<style>
+/* Hide the right-hand TOC sidebar and let content use full width */
+#toc-sidebar {{
+ display: none !important;
+}}
+.terminal-mkdocs-main-grid {{
+ margin-right: auto;
+}}
+.terminal-mkdocs-main-content {{
+ max-width: 100%;
+}}
+
+.stats-page {{
+ max-width: 1200px;
+ margin: 0 auto;
+ padding: 1rem 0;
+ font-family: inherit;
+}}
+
+.stats-header {{
+ text-align: center;
+ margin-bottom: 2.5rem;
+ padding-bottom: 1.5rem;
+ border-bottom: 1px solid #3f3f44;
+}}
+
+.stats-header img {{
+ height: 48px;
+ margin-bottom: 0.5rem;
+ filter: drop-shadow(0 0 12px rgba(80, 255, 255, 0.3));
+}}
+
+.stats-header h1 {{
+ font-size: 1.8rem;
+ color: #e8e9ed;
+ margin: 0.5rem 0 0.25rem;
+ letter-spacing: 0.5px;
+}}
+
+.stats-header .subtitle {{
+ color: #a3abba;
+ font-size: 0.95rem;
+}}
+
+/* Metric Cards */
+.metrics-grid {{
+ display: grid;
+ grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
+ gap: 1rem;
+ margin-bottom: 2.5rem;
+}}
+
+.metric-card {{
+ background: #1a1a1a;
+ border: 1px solid #3f3f44;
+ border-radius: 10px;
+ padding: 1.25rem;
+ text-align: center;
+ transition: all 0.3s ease;
+ position: relative;
+ overflow: hidden;
+}}
+
+.metric-card::before {{
+ content: '';
+ position: absolute;
+ top: 0;
+ left: 0;
+ right: 0;
+ height: 3px;
+ background: linear-gradient(90deg, #50ffff, #f380f5);
+ opacity: 0;
+ transition: opacity 0.3s ease;
+}}
+
+.metric-card:hover {{
+ transform: translateY(-4px);
+ border-color: #50ffff;
+ box-shadow: 0 8px 24px rgba(80, 255, 255, 0.12);
+}}
+
+.metric-card:hover::before {{
+ opacity: 1;
+}}
+
+.metric-value {{
+ font-size: 2rem;
+ font-weight: 700;
+ color: #50ffff;
+ margin: 0.25rem 0;
+ line-height: 1.2;
+}}
+
+.metric-label {{
+ font-size: 0.8rem;
+ color: #a3abba;
+ text-transform: uppercase;
+ letter-spacing: 1px;
+}}
+
+.metric-sublabel {{
+ font-size: 0.75rem;
+ color: #3f3f44;
+ margin-top: 0.25rem;
+}}
+
+/* Chart Sections */
+.chart-section {{
+ background: #1a1a1a;
+ border: 1px solid #3f3f44;
+ border-radius: 10px;
+ padding: 1.5rem;
+ margin-bottom: 1.5rem;
+}}
+
+.chart-section h2 {{
+ font-size: 1.1rem;
+ color: #e8e9ed;
+ margin: 0 0 1rem 0;
+ padding-bottom: 0.75rem;
+ border-bottom: 1px solid #3f3f44;
+}}
+
+.chart-container {{
+ position: relative;
+ width: 100%;
+ height: 350px;
+}}
+
+.chart-row {{
+ display: grid;
+ grid-template-columns: 1fr 1fr;
+ gap: 1.5rem;
+ margin-bottom: 1.5rem;
+}}
+
+@media (max-width: 768px) {{
+ .stats-page {{
+ padding: 0.5rem 0.75rem;
+ }}
+ .stats-header h1 {{
+ font-size: 1.4rem;
+ }}
+ .chart-row {{
+ grid-template-columns: 1fr;
+ }}
+ .metrics-grid {{
+ grid-template-columns: repeat(2, 1fr);
+ gap: 0.6rem;
+ }}
+ .metric-card {{
+ padding: 0.9rem 0.5rem;
+ }}
+ .metric-value {{
+ font-size: 1.5rem;
+ }}
+ .metric-label {{
+ font-size: 0.7rem;
+ }}
+ .chart-container {{
+ height: 260px;
+ }}
+ .chart-section {{
+ padding: 1rem;
+ }}
+}}
+
+@media (max-width: 480px) {{
+ .metrics-grid {{
+ grid-template-columns: 1fr 1fr;
+ gap: 0.5rem;
+ }}
+ .metric-value {{
+ font-size: 1.25rem;
+ }}
+ .metric-label {{
+ font-size: 0.65rem;
+ letter-spacing: 0.5px;
+ }}
+ .metric-sublabel {{
+ font-size: 0.65rem;
+ }}
+ .chart-container {{
+ height: 220px;
+ }}
+ .chart-section h2 {{
+ font-size: 0.95rem;
+ }}
+}}
+
+/* Footer */
+.stats-footer {{
+ text-align: center;
+ padding: 1.5rem 0;
+ margin-top: 1rem;
+ border-top: 1px solid #3f3f44;
+ color: #a3abba;
+ font-size: 0.8rem;
+}}
+</style>
+
+<div class="stats-page">
+
+<div class="stats-header">
+ <img src="../assets/images/logo.png" alt="Crawl4AI">
+ <h1>Growth</h1>
+ <div class="subtitle">Community growth & adoption metrics for Crawl4AI</div>
+</div>
+
+<!-- Headline Metrics -->
+<div class="metrics-grid">
+ <div class="metric-card">
+ <div class="metric-label">GitHub Stars</div>
+ <div class="metric-value">{fmt_number(github['stars'])}</div>
+ <div class="metric-sublabel">stargazers</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Monthly Downloads</div>
+ <div class="metric-value">{fmt_short(latest_month_downloads)}</div>
+ <div class="metric-sublabel">PyPI Β· latest month</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Total Downloads</div>
+ <div class="metric-value">{fmt_short(total_pypi)}</div>
+ <div class="metric-sublabel">PyPI cumulative</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Docker Pulls</div>
+ <div class="metric-value">{fmt_short(docker_pulls)}</div>
+ <div class="metric-sublabel">hub.docker.com</div>
+ </div>
+ <div class="metric-card">
+ <div class="metric-label">Forks / Contributors</div>
+ <div class="metric-value">{fmt_number(github['forks'])} <span style="color:#a3abba;font-size:1rem">/</span> {fmt_number(contributors)}</div>
+ <div class="metric-sublabel">GitHub</div>
+ </div>
+</div>
+
+<!-- Chart 1: Monthly PyPI Downloads (bar) -->
+<div class="chart-section">
+ <h2>PyPI Monthly Downloads</h2>
+ <div class="chart-container">
+ <canvas id="monthlyChart"></canvas>
+ </div>
+</div>
+
+<!-- Chart Row: Stars + Cumulative -->
+<div class="chart-row">
+ <div class="chart-section">
+ <h2>GitHub Star Growth</h2>
+ <div class="chart-container">
+ <canvas id="starsChart"></canvas>
+ </div>
+ </div>
+ <div class="chart-section">
+ <h2>Cumulative PyPI Downloads</h2>
+ <div class="chart-container">
+ <canvas id="cumulativeChart"></canvas>
+ </div>
+ </div>
+</div>
+
+<!-- Chart Row: Daily + Traffic -->
+<div class="chart-row">
+ <div class="chart-section">
+ <h2>Daily Download Trend</h2>
+ <div class="chart-container">
+ <canvas id="dailyChart"></canvas>
+ </div>
+ </div>
+ <div class="chart-section">
+ <h2>GitHub Traffic (14 days)</h2>
+ <div class="chart-container">
+ <canvas id="trafficChart"></canvas>
+ </div>
+ </div>
+</div>
+
+<div class="stats-footer">
+ Updated {updated_date} · Data from GitHub API, PyPI Stats, Docker Hub
+</div>
+
+</div>
+
+<script>
+// --- Chart.js Global Config ---
+Chart.defaults.color = '#a3abba';
+Chart.defaults.borderColor = '#3f3f44';
+Chart.defaults.font.family = "'dm', Monaco, 'Courier New', monospace";
+
+const CYAN = '#50ffff';
+const CYAN_DIM = 'rgba(80,255,255,0.10)';
+const CYAN_HOVER = '#0fbbaa';
+const PINK = '#f380f5';
+const PINK_DIM = 'rgba(243,128,245,0.10)';
+const GRID_COLOR = '#3f3f44';
+const TOOLTIP_BG = '#1a1a1a';
+
+const tooltipStyle = {{
+ backgroundColor: TOOLTIP_BG,
+ borderColor: CYAN,
+ borderWidth: 1,
+ titleColor: '#e8e9ed',
+ bodyColor: '#a3abba',
+ padding: 10,
+ cornerRadius: 6,
+ displayColors: false,
+}};
+
+const gridStyle = {{ color: GRID_COLOR, drawBorder: false }};
+const tickStyle = {{ color: '#a3abba', font: {{ size: 11 }} }};
+
+// --- Data ---
+const monthlyLabels = {json.dumps(monthly_labels)};
+const monthlyValues = {json.dumps(monthly_values)};
+
+const starDates = {json.dumps(star_dates)};
+const starCounts = {json.dumps(star_counts)};
+
+const cumulativeLabels = {json.dumps(cumulative_labels)};
+const cumulativePypi = {json.dumps(cumulative_pypi)};
+
+const dailyLabels = {json.dumps(daily_labels)};
+const dailyValues = {json.dumps(daily_values)};
+
+const trafficDates = {json.dumps(traffic_dates)};
+const trafficViews = {json.dumps(traffic_views)};
+const trafficUniques = {json.dumps(traffic_uniques)};
+
+function shortMonth(label) {{
+ const parts = label.split('-');
+ const months = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'];
+ return months[parseInt(parts[1])-1] + ' ' + parts[0].slice(2);
+}}
+
+function shortDate(label) {{
+ const parts = label.split('-');
+ const months = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'];
+ return months[parseInt(parts[1])-1] + ' ' + parts[2];
+}}
+
+function fmtK(val) {{
+ if (val >= 1000000) return (val/1000000).toFixed(1) + 'M';
+ if (val >= 1000) return (val/1000).toFixed(0) + 'K';
+ return val;
+}}
+
+// --- Chart 1: Monthly Downloads (bar) ---
+new Chart(document.getElementById('monthlyChart'), {{
+ type: 'bar',
+ data: {{
+ labels: monthlyLabels.map(shortMonth),
+ datasets: [{{
+ label: 'Downloads',
+ data: monthlyValues,
+ backgroundColor: CYAN,
+ hoverBackgroundColor: CYAN_HOVER,
+ borderRadius: 4,
+ borderSkipped: false,
+ }}]
+ }},
+ options: {{
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {{
+ legend: {{ display: false }},
+ tooltip: {{ ...tooltipStyle, callbacks: {{ label: ctx => fmtK(ctx.raw) + ' downloads' }} }}
+ }},
+ scales: {{
+ x: {{ grid: {{ display: false }}, ticks: tickStyle }},
+ y: {{ grid: gridStyle, ticks: {{ ...tickStyle, callback: fmtK }} }}
+ }}
+ }}
+}});
+
+// --- Chart 2: Star Growth (area) ---
+new Chart(document.getElementById('starsChart'), {{
+ type: 'line',
+ data: {{
+ labels: starDates.map(shortMonth),
+ datasets: [{{
+ label: 'Stars',
+ data: starCounts,
+ borderColor: CYAN,
+ backgroundColor: CYAN_DIM,
+ fill: true,
+ tension: 0.35,
+ pointRadius: 3,
+ pointBackgroundColor: CYAN,
+ pointHoverRadius: 6,
+ }}]
+ }},
+ options: {{
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {{
+ legend: {{ display: false }},
+ tooltip: {{ ...tooltipStyle, callbacks: {{ label: ctx => fmtK(ctx.raw) + ' stars' }} }}
+ }},
+ scales: {{
+ x: {{ grid: {{ display: false }}, ticks: {{ ...tickStyle, maxTicksLimit: 8 }} }},
+ y: {{ grid: gridStyle, ticks: {{ ...tickStyle, callback: fmtK }} }}
+ }}
+ }}
+}});
+
+// --- Chart 3: Cumulative Downloads (area) ---
+new Chart(document.getElementById('cumulativeChart'), {{
+ type: 'line',
+ data: {{
+ labels: cumulativeLabels.map(shortMonth),
+ datasets: [{{
+ label: 'Cumulative PyPI',
+ data: cumulativePypi,
+ borderColor: PINK,
+ backgroundColor: PINK_DIM,
+ fill: true,
+ tension: 0.35,
+ pointRadius: 3,
+ pointBackgroundColor: PINK,
+ pointHoverRadius: 6,
+ }}]
+ }},
+ options: {{
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {{
+ legend: {{ display: false }},
+ tooltip: {{ ...tooltipStyle, callbacks: {{ label: ctx => fmtK(ctx.raw) + ' total downloads' }} }}
+ }},
+ scales: {{
+ x: {{ grid: {{ display: false }}, ticks: tickStyle }},
+ y: {{ grid: gridStyle, ticks: {{ ...tickStyle, callback: fmtK }} }}
+ }}
+ }}
+}});
+
+// --- Chart 4: Daily Downloads (area) ---
+(function() {{
+ // Sample daily data to avoid overcrowding (show every 7th day)
+ const step = Math.max(1, Math.floor(dailyLabels.length / 60));
+ const sampledLabels = dailyLabels.filter((_, i) => i % step === 0);
+ const sampledValues = dailyValues.filter((_, i) => i % step === 0);
+
+ new Chart(document.getElementById('dailyChart'), {{
+ type: 'line',
+ data: {{
+ labels: sampledLabels.map(shortDate),
+ datasets: [{{
+ label: 'Daily Downloads',
+ data: sampledValues,
+ borderColor: CYAN,
+ backgroundColor: CYAN_DIM,
+ fill: true,
+ tension: 0.3,
+ pointRadius: 0,
+ borderWidth: 1.5,
+ }}]
+ }},
+ options: {{
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {{
+ legend: {{ display: false }},
+ tooltip: {{ ...tooltipStyle, callbacks: {{ label: ctx => fmtK(ctx.raw) + ' downloads' }} }}
+ }},
+ scales: {{
+ x: {{ grid: {{ display: false }}, ticks: {{ ...tickStyle, maxTicksLimit: 8 }} }},
+ y: {{ grid: gridStyle, ticks: {{ ...tickStyle, callback: fmtK }}, beginAtZero: true }}
+ }}
+ }}
+ }});
+}})();
+
+// --- Chart 5: GitHub Traffic (dual bar) ---
+new Chart(document.getElementById('trafficChart'), {{
+ type: 'bar',
+ data: {{
+ labels: trafficDates.map(shortDate),
+ datasets: [
+ {{
+ label: 'Page Views',
+ data: trafficViews,
+ backgroundColor: CYAN,
+ hoverBackgroundColor: CYAN_HOVER,
+ borderRadius: 4,
+ borderSkipped: false,
+ }},
+ {{
+ label: 'Unique Visitors',
+ data: trafficUniques,
+ backgroundColor: PINK,
+ hoverBackgroundColor: '#d060d0',
+ borderRadius: 4,
+ borderSkipped: false,
+ }}
+ ]
+ }},
+ options: {{
+ responsive: true,
+ maintainAspectRatio: false,
+ plugins: {{
+ legend: {{
+ labels: {{ color: '#a3abba', boxWidth: 12, padding: 16 }}
+ }},
+ tooltip: tooltipStyle,
+ }},
+ scales: {{
+ x: {{ grid: {{ display: false }}, ticks: tickStyle }},
+ y: {{ grid: gridStyle, ticks: tickStyle }}
+ }}
+ }}
+}});
+</script>
+"""
+
+
+def main():
+ print("=" * 50)
+ print("Crawl4AI Stats Dashboard Generator")
+ print("=" * 50)
+
+ github = fetch_github_stats()
+ print(f" Stars: {github['stars']}, Forks: {github['forks']}")
+
+ contributors = fetch_contributors_count()
+ print(f" Contributors: {contributors}")
+
+ traffic = fetch_github_traffic()
+ print(f" Traffic data points: {len(traffic.get('views', []))} days")
+
+ pypi_live = fetch_pypi_live()
+ print(f" PyPI live: {len(pypi_live['monthly'])} months, {len(pypi_live['daily'])} daily points")
+ pypi = merge_pypi_data(pypi_live)
+ print(f" PyPI merged: {len(pypi['monthly'])} months, total: {pypi['total']:,}")
+
+ docker_pulls = fetch_docker_pulls()
+ print(f" Docker pulls: {docker_pulls}")
+
+ # Generate the page
+ content = generate_stats_md(github, contributors, traffic, pypi, docker_pulls, STAR_MILESTONES)
+
+ # Write output
+ OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)
+ OUTPUT_PATH.write_text(content)
+ print(f"\nWrote {OUTPUT_PATH} ({len(content):,} bytes)")
+ print("Done!")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/adaptive/test_embedding_strategy.py b/tests/adaptive/test_embedding_strategy.py
index 374330652..6e34b85e7 100644
--- a/tests/adaptive/test_embedding_strategy.py
+++ b/tests/adaptive/test_embedding_strategy.py
@@ -233,7 +233,7 @@ async def test_knowledge_export_import():
crawler2 = AdaptiveCrawler(crawler=crawler, config=config)
console.print("\n[cyan]Importing knowledge base...[/cyan]")
- crawler2.import_knowledge_base(export_path)
+ await crawler2.import_knowledge_base(export_path)
# Continue with new query - should be faster
console.print("\n[cyan]Extending with new query...[/cyan]")
diff --git a/tests/adaptive/test_query_llm_config.py b/tests/adaptive/test_query_llm_config.py
new file mode 100644
index 000000000..f20c100bb
--- /dev/null
+++ b/tests/adaptive/test_query_llm_config.py
@@ -0,0 +1,284 @@
+"""
+E2E tests for separate embedding and query LLM configs (issue #1682).
+
+Tests that AdaptiveConfig.query_llm_config flows correctly through
+AdaptiveCrawler β EmbeddingStrategy β map_query_semantic_space,
+and that the right config is used for embeddings vs query expansion.
+"""
+
+import asyncio
+import json
+import sys
+from pathlib import Path
+from unittest.mock import patch, MagicMock, AsyncMock
+import numpy as np
+
+sys.path.append(str(Path(__file__).parent.parent.parent))
+
+from crawl4ai import AdaptiveConfig, LLMConfig
+from crawl4ai.adaptive_crawler import EmbeddingStrategy, AdaptiveCrawler
+
+
+# ---------------------------------------------------------------------------
+# Test 1: Config plumbing β AdaptiveConfig β AdaptiveCrawler β EmbeddingStrategy
+# ---------------------------------------------------------------------------
+
+def test_config_plumbing():
+ """query_llm_config flows from AdaptiveConfig through _create_strategy."""
+ config = AdaptiveConfig(
+ strategy="embedding",
+ embedding_llm_config=LLMConfig(provider="openai/text-embedding-3-small", api_token="emb-key"),
+ query_llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token="query-key"),
+ )
+
+ # Simulate what AdaptiveCrawler.__init__ does
+ with patch("crawl4ai.adaptive_crawler.AsyncWebCrawler"):
+ crawler_mock = MagicMock()
+ adaptive = AdaptiveCrawler(crawler=crawler_mock, config=config)
+
+ strategy = adaptive.strategy
+ assert isinstance(strategy, EmbeddingStrategy)
+
+ # Strategy should have both configs
+ assert strategy.query_llm_config is not None
+ query_dict = strategy._get_query_llm_config_dict()
+ assert query_dict["provider"] == "openai/gpt-4o-mini"
+ assert query_dict["api_token"] == "query-key"
+
+ emb_dict = strategy._get_embedding_llm_config_dict()
+ assert emb_dict["provider"] == "openai/text-embedding-3-small"
+ assert emb_dict["api_token"] == "emb-key"
+
+ print("PASS: test_config_plumbing")
+
+
+# ---------------------------------------------------------------------------
+# Test 2: Backward compat β no query_llm_config falls back to llm_config
+# ---------------------------------------------------------------------------
+
+def test_backward_compat_fallback():
+ """When query_llm_config is not set, falls back to llm_config (legacy)."""
+ strategy = EmbeddingStrategy(
+ embedding_model="sentence-transformers/all-MiniLM-L6-v2",
+ llm_config={"provider": "openai/gpt-4o-mini", "api_token": "shared-key"},
+ query_llm_config=None,
+ )
+ # No AdaptiveConfig attached β should fall back to llm_config
+ result = strategy._get_query_llm_config_dict()
+ assert result["provider"] == "openai/gpt-4o-mini"
+ assert result["api_token"] == "shared-key"
+ print("PASS: test_backward_compat_fallback")
+
+
+def test_backward_compat_no_config():
+ """When nothing is set, returns None (caller uses hardcoded defaults)."""
+ strategy = EmbeddingStrategy()
+ result = strategy._get_query_llm_config_dict()
+ assert result is None
+ print("PASS: test_backward_compat_no_config")
+
+
+# ---------------------------------------------------------------------------
+# Test 3: Fallback priority chain
+# ---------------------------------------------------------------------------
+
+def test_fallback_priority():
+ """Explicit query_llm_config beats AdaptiveConfig beats llm_config."""
+ config = AdaptiveConfig(
+ strategy="embedding",
+ query_llm_config={"provider": "config-level", "api_token": "cfg"},
+ )
+ strategy = EmbeddingStrategy(
+ llm_config={"provider": "legacy-level", "api_token": "leg"},
+ query_llm_config={"provider": "strategy-level", "api_token": "strat"},
+ )
+ strategy.config = config
+
+ # Strategy-level should win
+ result = strategy._get_query_llm_config_dict()
+ assert result["provider"] == "strategy-level"
+
+ # Remove strategy-level β config-level should win
+ strategy.query_llm_config = None
+ result = strategy._get_query_llm_config_dict()
+ assert result["provider"] == "config-level"
+
+ # Remove config-level β legacy llm_config should win
+ config.query_llm_config = None
+ result = strategy._get_query_llm_config_dict()
+ assert result["provider"] == "legacy-level"
+
+ # Remove everything β None
+ strategy.llm_config = None
+ result = strategy._get_query_llm_config_dict()
+ assert result is None
+
+ print("PASS: test_fallback_priority")
+
+
+# ---------------------------------------------------------------------------
+# Test 4: E2E β map_query_semantic_space uses query config, not embedding config
+# ---------------------------------------------------------------------------
+
+async def test_map_query_uses_query_config():
+ """map_query_semantic_space should call perform_completion_with_backoff
+ with the query LLM config (chat model), NOT the embedding config."""
+
+ config = AdaptiveConfig(
+ strategy="embedding",
+ embedding_llm_config=LLMConfig(
+ provider="openai/text-embedding-3-small",
+ api_token="emb-key",
+ base_url="https://emb.example.com",
+ ),
+ query_llm_config=LLMConfig(
+ provider="openai/gpt-4o-mini",
+ api_token="query-key",
+ base_url="https://query.example.com",
+ ),
+ )
+
+ strategy = EmbeddingStrategy(
+ embedding_model="sentence-transformers/all-MiniLM-L6-v2",
+ llm_config=config.embedding_llm_config,
+ query_llm_config=config.query_llm_config,
+ )
+ strategy.config = config
+
+ # Mock perform_completion_with_backoff to capture its arguments
+ mock_response = MagicMock()
+ mock_response.choices = [MagicMock()]
+ mock_response.choices[0].message.content = json.dumps({
+ "queries": [f"variation {i}" for i in range(13)]
+ })
+
+ captured_kwargs = {}
+
+ def mock_completion(**kwargs):
+ # Also accept positional-style
+ captured_kwargs.update(kwargs)
+ return mock_response
+
+ # Also mock _get_embeddings to avoid real embedding calls
+ fake_embeddings = np.random.rand(11, 384).astype(np.float32)
+
+ with patch("crawl4ai.utils.perform_completion_with_backoff", side_effect=mock_completion):
+ with patch.object(strategy, "_get_embeddings", new_callable=AsyncMock, return_value=fake_embeddings):
+ await strategy.map_query_semantic_space("test query", n_synthetic=10)
+
+ # Verify the query config was used, NOT the embedding config
+ assert captured_kwargs["provider"] == "openai/gpt-4o-mini", \
+ f"Expected query model, got {captured_kwargs['provider']}"
+ assert captured_kwargs["api_token"] == "query-key", \
+ f"Expected query-key, got {captured_kwargs['api_token']}"
+ assert captured_kwargs["base_url"] == "https://query.example.com", \
+ f"Expected query base_url, got {captured_kwargs['base_url']}"
+
+ # Verify backoff params are passed (bug fix)
+ assert "base_delay" in captured_kwargs
+ assert "max_attempts" in captured_kwargs
+ assert "exponential_factor" in captured_kwargs
+
+ print("PASS: test_map_query_uses_query_config")
+
+
+# ---------------------------------------------------------------------------
+# Test 5: E2E β legacy single-config still works for query expansion
+# ---------------------------------------------------------------------------
+
+async def test_legacy_single_config_for_query():
+ """When only embedding_llm_config is set (old usage), query expansion
+ falls back to it via llm_config β still works."""
+
+ single_config = LLMConfig(
+ provider="openai/gpt-4o-mini",
+ api_token="single-key",
+ )
+
+ config = AdaptiveConfig(
+ strategy="embedding",
+ embedding_llm_config=single_config,
+ # No query_llm_config β legacy usage
+ )
+
+ strategy = EmbeddingStrategy(
+ embedding_model="sentence-transformers/all-MiniLM-L6-v2",
+ llm_config=config.embedding_llm_config, # This is how _create_strategy passes it
+ # No query_llm_config
+ )
+ strategy.config = config
+
+ mock_response = MagicMock()
+ mock_response.choices = [MagicMock()]
+ mock_response.choices[0].message.content = json.dumps({
+ "queries": [f"variation {i}" for i in range(13)]
+ })
+
+ captured_kwargs = {}
+
+ def mock_completion(**kwargs):
+ captured_kwargs.update(kwargs)
+ return mock_response
+
+ fake_embeddings = np.random.rand(11, 384).astype(np.float32)
+
+ with patch("crawl4ai.utils.perform_completion_with_backoff", side_effect=mock_completion):
+ with patch.object(strategy, "_get_embeddings", new_callable=AsyncMock, return_value=fake_embeddings):
+ await strategy.map_query_semantic_space("test query", n_synthetic=10)
+
+ # Should fall back to llm_config (the single shared config)
+ assert captured_kwargs["provider"] == "openai/gpt-4o-mini"
+ assert captured_kwargs["api_token"] == "single-key"
+
+ print("PASS: test_legacy_single_config_for_query")
+
+
+# ---------------------------------------------------------------------------
+# Test 6: LLMConfig.to_dict() includes backoff params (bug fix verification)
+# ---------------------------------------------------------------------------
+
+def test_to_dict_includes_backoff():
+ """_embedding_llm_config_dict now uses to_dict() which includes backoff params."""
+ config = AdaptiveConfig(
+ embedding_llm_config=LLMConfig(
+ provider="openai/text-embedding-3-small",
+ api_token="test",
+ backoff_base_delay=5,
+ backoff_max_attempts=10,
+ backoff_exponential_factor=3,
+ ),
+ )
+ d = config._embedding_llm_config_dict
+ assert d["backoff_base_delay"] == 5
+ assert d["backoff_max_attempts"] == 10
+ assert d["backoff_exponential_factor"] == 3
+ print("PASS: test_to_dict_includes_backoff")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+async def main():
+ print("=" * 60)
+ print("E2E Tests: Separate Embedding & Query LLM Configs (#1682)")
+ print("=" * 60)
+
+ # Sync tests
+ test_config_plumbing()
+ test_backward_compat_fallback()
+ test_backward_compat_no_config()
+ test_fallback_priority()
+ test_to_dict_includes_backoff()
+
+ # Async tests
+ await test_map_query_uses_query_config()
+ await test_legacy_single_config_for_query()
+
+ print("\n" + "=" * 60)
+ print("ALL TESTS PASSED")
+ print("=" * 60)
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/tests/async/test_browser_lifecycle.py b/tests/async/test_browser_lifecycle.py
new file mode 100644
index 000000000..b7042cc59
--- /dev/null
+++ b/tests/async/test_browser_lifecycle.py
@@ -0,0 +1,972 @@
+"""
+Browser lifecycle & concurrency tests.
+
+Covers all the browser launch paths and lock interactions:
+ - Standalone (playwright.launch)
+ - Managed browser (subprocess + CDP connect)
+ - Managed browser with create_isolated_context
+ - Page reuse on shared default context
+ - Context caching / LRU eviction
+ - Session lifecycle across all modes
+ - Concurrent crawls racing for pages / contexts
+ - Recycle interacting with managed browser
+ - Multiple crawlers sharing a managed browser via CDP
+"""
+
+import asyncio
+import time
+import threading
+from http.server import HTTPServer, SimpleHTTPRequestHandler
+
+import pytest
+
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
+
+
+# ---------------------------------------------------------------------------
+# Local test server
+# ---------------------------------------------------------------------------
+
+PAGES = {}
+for i in range(100):
+ PAGES[f"/page{i}"] = (
+ f"<!DOCTYPE html><html><head><title>Page {i}"
+ f"Page {i}
Content for page {i}.
"
+ f"next