A powerful lead generation tool that crawls multiple platforms to collect potential leads for startups, developers, and makers.
This project combines multi-threaded scraping, Selenium automation, and crawl4ai integration to efficiently gather data while managing concurrency.
The Lead Generator is designed to scrape and aggregate information from popular product discovery and startup platforms:
- DevHunt
- ProductHunt
- TinyStartups
- Uneed
👉 (Currently, support for Peerlist is under development.)
The system is optimized for performance using Python threading, with semaphores controlling concurrency levels to prevent overload. Each scraper runs simultaneously, ensuring faster data collection.
- ⚡ Concurrent Scraping – Multiple scrapers run in parallel using threading.
- 🔒 Safe Thread Management – Semaphores prevent resource exhaustion.
- 🌐 Smart Crawling – Uses both Selenium (for dynamic content) and crawl4ai (for fast structured scraping).
- 📊 Multi-Platform Support – Collects leads from top product and startup websites.
- 🛠️ Extensible Design – Easy to plug in new sources (Peerlist integration coming soon).
- 🕵️ Planned: Tor Proxy Rotation – Future integration with Tor network to rotate IPs and avoid rate-limits.
- Python 3
- Selenium (for JavaScript-heavy pages)
- crawl4ai (for efficient crawling)
- Threading + Semaphores (for concurrency control)
- Planned: Tor (for anonymous, rotating proxies)
- Each scraper is initialized for a specific platform.
- A thread pool launches scrapers simultaneously.
- Semaphores manage the maximum number of active threads at a time.
- Data is collected, cleaned, and stored for lead generation use cases.
- (Upcoming: Tor proxy layer for rotating IPs automatically.)
# Clone the repository
git clone https://github.com/your-username/lead-generator.git
cd lead-generator
# Install dependencies
pip install -r requirements.txt
# Run the scrapers
python main.py- ✅ Add support for DevHunt, ProductHunt, TinyStartups, Uneed
- 🔄 Peerlist parser (in progress)
- 🌍 Tor proxy rotation for anonymous scraping (planned)
- 📤 Export collected leads into CSV/Excel/Database
- 🌟 Add dashboard for visualizing collected leads
Contributions are welcome! If you’d like to add support for more platforms or improve the scraping logic:
- Fork this repo
- Create a feature branch (
git checkout -b feature/new-scraper) - Commit your changes
- Open a Pull Request
This project is licensed under the MIT License.