Skip to content

feat: add tenacity retry to Grow/LevelData API requests #3590

@cbini

Description

@cbini

Problem

The Grow API resource (src/teamster/libraries/level_data/grow/resources.py) has no retry logic on HTTP requests. When grow-api.leveldata.com times out, the step fails immediately and the entire run relies on Dagster's run-level auto-retry — restarting the full run instead of retrying just the failed request.

On 2026-04-07, kipptaf__grow__multi_partition_assets_schedule failed due to a connection timeout to grow-api.leveldata.com. The Dagster auto-retry succeeded 2 minutes later, but the entire run was re-executed rather than just the timed-out request.

Proposed Change

Add tenacity retry with exponential backoff + jitter to GrowResource._request(), matching the pattern already used in:

  • ADP WFN (libraries/adp/workforce_now/api/resources.py:53-57) — @retry(stop=stop_after_attempt(5), wait=wait_exponential_jitter())
  • Google Forms (libraries/google/forms/resources.py:35) — same pattern
  • Extracts (libraries/extracts/assets.py:105-109) — same pattern

Retry on:

  • requests.exceptions.ConnectionError (includes timeouts when timeout kwarg is set)
  • requests.exceptions.Timeout
  • HTTP 429 (rate limit) — respect Retry-After header if present
  • HTTP 502, 503, 504 (transient server errors)

Scope

  • GrowResource._request() in src/teamster/libraries/level_data/grow/resources.py
  • Also consider GrowResource._get_access_token() (OAuth2 token fetch can also time out)
  • Check OvergradResource — same level_data library, also has no retry logic and a TODO noting missing exponential backoff (libraries/overgrad/resources.py:32)

Out of Scope

Broader audit of all API resources for missing retry logic (separate issue).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions