fix: RateLimiter burst race, Retry-After headers, deep crawl dispatcher (#1095)

[ntohidi]

Summary

• Fixes [Bug]: RateLimiter provides ineffective protection against failures #1095
• Three fixes for RateLimiter ineffectiveness under concurrent load and on rate-limited sites

1. Burst race (concurrent tasks bypass rate limiting)

wait_if_needed() had no synchronization — concurrent tasks all read last_request_time at the same instant, computed wait_time ≈ 0, and fired together. Added per-domain asyncio.Lock so tasks serialize and each waits its proper turn.

Before: 9/10 requests fire at +1.7s simultaneously (0ms gaps)
After: Requests spaced 1.2-1.8s apart across 13.7s total

2. Retry-After header support

update_delay() only accepted (url, status_code) — server rate-limit headers were completely ignored. Added optional response_headers param with parsing for Retry-After (both delay-seconds and HTTP-date formats). Both dispatcher call sites now pass result.response_headers.

Before: 429 with Retry-After: 5 → blind exponential backoff (1.9s)
After: 429 with Retry-After: 5 → delay set to 5.0s as server instructed

3. Deep crawl dispatcher configurability

BFS, DFS, and BestFirst strategies hardcoded arun_many() calls without passing a dispatcher. Added dispatcher param to all three, forwarded to every arun_many() call.

Changes

• crawl4ai/async_dispatcher.py: Per-domain lock in wait_if_needed(), response_headers param + _parse_retry_after() in update_delay(), both call sites updated
• crawl4ai/deep_crawling/bfs_strategy.py: Added dispatcher param, forwarded to arun_many()
• crawl4ai/deep_crawling/dfs_strategy.py: Forwarded self.dispatcher to arun_many()
• crawl4ai/deep_crawling/bff_strategy.py: Added dispatcher param, forwarded to arun_many()

Test plan

• Reproduction script verifies all three fixes (burst serialization, Retry-After parsing, dispatcher passthrough)
• Deep crawl with rate-limited site (e.g. gamesjobslive.niceboard.co) to verify end-to-end