feat(proxy): add ProxyRotator with multi-strategy rotation, health tracking, and auto-failover#38
feat(proxy): add ProxyRotator with multi-strategy rotation, health tracking, and auto-failover#38evelaa123 wants to merge 1 commit intoCloakHQ:mainfrom
Conversation
…acking, and auto-failover - Round-robin, random, least-used, least-failures strategies - Per-proxy health tracking with cooldown and auto-recovery - Sticky sessions (reuse same proxy for N requests) - Dynamic pool management (add/remove at runtime) - Context manager / withSession with auto success/failure reporting - Thread-safe implementation with Lock - Credential masking in stats and logs - SOCKS5 without auth supported; SOCKS5 with auth rejected (Chromium limitation) - Bare proxy format support (user:pass@host:port) - Full test coverage: 49 pytest, 45 vitest, 7 real-proxy, 9 real-proxy JS - Integration with launch(proxy=rotator)
e062dcc to
0aa4ea5
Compare
|
Hey, thanks for the detailed PR and the solid test coverage — this is clearly well thought out. After reviewing, we're going to pass on merging this one. The core reason: proxy rotation is orchestration logic that lives between browser sessions, not inside the launcher. Since Chrome binds the proxy at process start (it's a Compare this to We want to keep the wrapper focused on what only it can do: launching a stealthy browser. For multi-session workflows, check out CloakBrowser Manager — self-hosted profile management with persistent sessions, proxies, and noVNC. One thing we do want to take from this PR: the bare proxy format fix ( |
Summary
Adds
ProxyRotator— a thread-safe proxy pool with automatic rotation, health tracking, and failover for both Python and JavaScript/TypeScript.Background
The design was inspired by Scrapling's ProxyRotator, which provides cyclic proxy rotation with custom strategy callbacks for its spider framework. CloakBrowser's implementation is written from scratch and differs significantly:
sticky_countto keep the same proxy for a batch of requests (useful for session-based websites).threading.Lock(Python). Scrapling's rotator is not thread-safe.stats()and all log output automatically mask usernames and passwords. Scrapling does not mask credentials.socks5://user:pass@host:portat construction time with a clear error message, because Chromium does not support SOCKS5 authentication. This prevents a confusing launch-time crash.user:pass@host:portwithout a scheme prefix, which is common in proxy provider dashboards.In short: Scrapling's ProxyRotator was the starting inspiration for "a class that rotates proxies," but the actual implementation, API, and feature set are entirely different and written from scratch for CloakBrowser's use case (standalone browser automation, not a spider framework).
Features
Rotation Strategies
Health Tracking
Each proxy tracks:
use_count,fail_count,consecutive_fails,last_used,last_failed,cooldown_until.When a proxy reaches
max_failuresconsecutive failures (default: 3), it is automatically placed on cooldown forcooldownseconds (default: 300). During cooldown it is skipped by all strategies. When cooldown expires, the proxy becomes available again. Callingreport_success()immediately resets the consecutive failure counter and removes any cooldown.If all proxies are in cooldown simultaneously,
next()raisesRuntimeErrorwith a clear message.Sticky Sessions
Set
sticky_count=Nto reuse the same proxy for N consecutivenext()calls before rotating. Useful for websites that track sessions by IP. If the sticky proxy fails, rotation is forced immediately.Dynamic Pool
add(proxy)— add a proxy to the pool at runtimeremove(proxy)— remove a proxy (raises if not found or would empty the pool)Context Manager / withSession
Python:
JavaScript:
Credential Masking
stats()output and all internal logging automatically masks credentials:http://user:pass@host:8080→http://***:***@host:8080user:pass@host:8080(bare format) →***:***@host:8080socks5://host:8080(no creds) → unchangedhttp://host:8080||username→http://host:8080||***SOCKS5 Validation
Chromium does not support SOCKS5 proxy authentication. Rather than failing with a cryptic error at browser launch time,
ProxyRotatorvalidates proxies at construction andadd()time:socks5://host:port— accepted (no auth)socks5://user:pass@host:port— raisesValueErrorimmediately{"server": "socks5://host:port", "username": "u", "password": "p"}— raisesValueErrorimmediatelyIntegration with launch()
The
current()method returns the proxy that was last selected bynext(), so you can report success/failure after using it.API Reference
Python (
cloakbrowser.proxy_rotator)JavaScript/TypeScript (
cloakbrowser)Tests
Tests cover: all four strategies, health tracking with cooldown expiration, sticky sessions, context manager success/failure paths, dynamic pool add/remove edge cases, credential masking for all proxy formats, SOCKS5 validation, thread safety (10 threads × 100 calls), bare proxy format handling, round-robin index clamping after removal, and real browser integration (headless Chromium verifying external IP via ipify.org).
Files
Added
cloakbrowser/proxy_rotator.py— Python implementationjs/src/proxy-rotator.ts— TypeScript implementationtests/test_proxy_rotator.py— Python unit tests (49 tests)js/tests/proxy-rotator.test.ts— JS unit tests (45 tests)tests/test_proxy_real.py— Python real-proxy integration teststests/test_proxy_real.mjs— JS real-proxy integration testsexamples/proxy_rotation.py— Python usage exampleexamples/proxy_verify.py— Python proxy verification scriptjs/examples/proxy-rotation.ts— TypeScript usage exampleModified
cloakbrowser/__init__.py— export ProxyRotatorcloakbrowser/browser.py— integrate rotator with launch()js/src/index.ts— export ProxyRotator, maskProxy, resolveProxyRotatorjs/src/proxy.ts— integrate rotator resolutionjs/src/types.ts— add ProxyRotator-related typesjs/src/playwright.ts— resolve rotator in Playwright launcherjs/src/puppeteer.ts— resolve rotator in Puppeteer launchertests/test_proxy.py— add bare proxy format parsing tests