-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Summary
The enrichment worker currently polls /internal/tasks/claim every 1s per worker when the queue is empty. This creates unnecessary request/log noise at idle.
Problem
- Idle traffic scales with
WORKER_CONCURRENCY(currently default 4). - Empty-queue polling is constant and noisy in API logs.
- Current behavior does not adapt to prolonged idle periods.
Proposed change
Implement exponential idle backoff in the worker claim loop:
- Start at
1safter an empty claim. - Increase on consecutive empty claims:
1s -> 2s -> 4s -> 8s ... - Cap at
30-60s(choose one explicit cap). - Reset backoff to
1simmediately after a successful claim. - Optionally apply small jitter (recommended) to avoid synchronized polling across workers.
Acceptance criteria
- Worker claim loop uses exponential idle backoff for consecutive empty claims.
- Backoff resets to minimum after a successful claim.
- Maximum idle wait is capped at a configured constant (within 30-60s).
- Existing task-processing behavior and retries remain unchanged.
- Unit tests cover backoff increase, cap behavior, and reset-on-success.
Notes
This keeps the existing pull/claim queue model (no architecture change to webhook subscriber) while substantially reducing idle load.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels