A resilient, extensible web crawling framework inspired by mythology.
Ladon is an open-source web crawling and scraping framework designed for extensibility, reliability, and long-term maintainability. Its architecture centers around a core networking layer and a plugin-based crawler system, allowing developers to implement site-specific adapters cleanly and consistently.
The name Ladon comes from the multi-headed guardian serpent of Greek mythology — a symbolic representation of a framework capable of coordinating many "heads" (adapters) while guarding the integrity of the system.
Ladon aims to provide:
- A resilient HTTP layer with retries, backoff, rate limiting, and circuit breakers
- An adapter/plugin architecture for site-specific crawlers
- Consistent observability (structured logs, metrics, tracing)
- Polite crawling behavior (robots.txt, domain-level throttling)
- Extensibility from small scripts to large crawling systems
- Testable, modular design suitable for research, automation, and data pipelines
This project is currently in its early planning and design phase. The initial goal is to build the core networking client and the abstractions for adapters.
Ladon is under active development. Documentation, architecture notes, and roadmap details will be gradually added to the repository.
For now, the project consists of:
- This README
- Basic repository structure
- Design work happening upstream, preparing for implementation
Ladon is being built as an open-source community project. While contributions are not yet open, we will soon welcome:
- Feature proposals
- Issue reports
- Documentation contributions
- Adapter implementations
- Testing and CI improvements
A CONTRIBUTING.md guide and CODE_OF_CONDUCT.md file will be added once the initial architecture stabilizes.
Ladon will be released under a permissive open-source license (likely MIT or Apache 2.0). The final license choice will be added here soon.
Early planned milestones include:
- Core networking layer
- HttpClient abstraction
- Retry/backoff logic
- Rate limiting per domain
- Circuit breaker system
- Robots.txt handler
- Result and metadata model
- Plugin architecture for site adapters
- CLI tool for running crawlers
- Documentation site (MkDocs Material)
- Example adapters (non-proprietary websites)
- Testing framework and contract tests
Project updates and design discussions will be published directly in this repository as the implementation begins.