A resilient scraper for real-time marketplace monitoring
English | Português Contributing | Changelog
GOrimpo is a continuous monitoring solution designed for the second-hand market and collectibles. It automates searches for specific terms, processes price and negative filters, and instantly notifies the user via Telegram.
Unlike simple scripts, GOrimpo was built with a focus on resilience and stealth, mimicking human behavior to avoid detection by anti-bot systems.
- Multimodal Search: Dynamic support for Chromium, Firefox, and WebKit rendering engines.
- Stealth Identity Factory: Random generation of thousands of real User-Agents at runtime.
- Adaptive Circuit Breaker: IP protection system that halts searches and scales cooldown time automatically upon detecting blocks (403 Forbidden).
- Behavioral Mimicry: Implementation of random delays (jitter) and behavioral micro-rests to break rhythmic search patterns.
- Advanced Filtering: Keyword exclusion system and detection of sponsored/featured ads that are irrelevant to the search.
- Smart Notification: Support for Telegram Topics to organize searches into different categories.
The project is built with Go v1.25.5, uses SQLite3 for data persistence, and Playwright for the scraping engine.
The project follows Hexagonal Architecture (Ports & Adapters), ensuring that the application core (Domain and Services) is completely decoupled from external technologies like databases, scraping frameworks, or notification APIs.
graph TD
subgraph External_World [External World]
OLX[OLX - Playwright Scraper]
TG[Telegram - Notifier]
DB[(SQLite - Repository)]
YAML[config.yaml - Config]
end
subgraph Adapters [Adapters]
OLXAdapter[OLX Adapter]
TelegramAdapter[Telegram Adapter]
SQLiteAdapter[SQLite Adapter]
ConfigAdapter[Config Manager]
end
subgraph Core_Ports [Core Ports]
ScraperPort((Scraper Port))
NotifierPort((Notifier Port))
RepoPort((Repository Port))
ConfigPort((Config Port))
end
subgraph Core_Domain [Core Domain & Services]
Service[Gorimpo Service]
Domain[Domain Models: Offer, Search]
end
%% Connections
OLX --> OLXAdapter
TG <--> TelegramAdapter
DB <--> SQLiteAdapter
YAML --> ConfigAdapter
OLXAdapter --> ScraperPort
TelegramAdapter --> NotifierPort
SQLiteAdapter --> RepoPort
ConfigAdapter --> ConfigPort
ScraperPort --> Service
NotifierPort --> Service
RepoPort --> Service
ConfigPort --> Service
Service --> Domain
The recommended way to run GOrimpo is through Docker, ensuring that all Playwright dependencies and browsers are isolated.
# Clone the repository
git clone https://github.com/LXSCA7/gorimpo.git
# Configure your environment variables (.env)
TELEGRAM_TOKEN=your_token
TELEGRAM_CHAT_ID=your_chat_id
GOTIFY_HOST=your_host
GOTIFY_TOKEN=your_gotify_token
# Start the container
docker-compose up -dSearch intelligence is entirely controlled via config.yaml. Example:
app:
default_notifier: "telegram" # options: gotify, telegram
use_topics: true # define if telegram will use topics
proxy:
enabled: true
provider: "proxyscrape"
strategies:
proxyscrape:
url: "proxyscrape_url"
timeout: 5
refresh_threshold: 0.5
categories:
- "🍄 nintendo"
- "🎮 playstation"
- "👾 handhelds"
# add your category here, GOrimpo will automatically create the topic on Telegram
scraper:
user_agent_count: 50 # Number of identities in the pool
min_jitter: 2
max_jitter: 10
searches:
- term: "Nintendo 64"
min_price: 200
max_price: 600
category: "nintendo"
exclude: ["box", "manuals", "broken"]Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
Please check our Contributing Guide for more details on our architecture and workflow.
- Implementation of new adapters for Enjoei, MercadoLivre, and Shopee.
- Analytical frontend using HTMX and Templ.
- Price history implementation for market trend analysis.
Developed by LXSCA ⭐️