Skip to content

feat(worker): add exponential idle backoff for empty task claims #133

@mfittko

Description

@mfittko

Summary

The enrichment worker currently polls /internal/tasks/claim every 1s per worker when the queue is empty. This creates unnecessary request/log noise at idle.

Problem

  • Idle traffic scales with WORKER_CONCURRENCY (currently default 4).
  • Empty-queue polling is constant and noisy in API logs.
  • Current behavior does not adapt to prolonged idle periods.

Proposed change

Implement exponential idle backoff in the worker claim loop:

  • Start at 1s after an empty claim.
  • Increase on consecutive empty claims: 1s -> 2s -> 4s -> 8s ...
  • Cap at 30-60s (choose one explicit cap).
  • Reset backoff to 1s immediately after a successful claim.
  • Optionally apply small jitter (recommended) to avoid synchronized polling across workers.

Acceptance criteria

  • Worker claim loop uses exponential idle backoff for consecutive empty claims.
  • Backoff resets to minimum after a successful claim.
  • Maximum idle wait is capped at a configured constant (within 30-60s).
  • Existing task-processing behavior and retries remain unchanged.
  • Unit tests cover backoff increase, cap behavior, and reset-on-success.

Notes

This keeps the existing pull/claim queue model (no architecture change to webhook subscriber) while substantially reducing idle load.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions