Problem
The Grow API resource (src/teamster/libraries/level_data/grow/resources.py) has no retry logic on HTTP requests. When grow-api.leveldata.com times out, the step fails immediately and the entire run relies on Dagster's run-level auto-retry — restarting the full run instead of retrying just the failed request.
On 2026-04-07, kipptaf__grow__multi_partition_assets_schedule failed due to a connection timeout to grow-api.leveldata.com. The Dagster auto-retry succeeded 2 minutes later, but the entire run was re-executed rather than just the timed-out request.
Proposed Change
Add tenacity retry with exponential backoff + jitter to GrowResource._request(), matching the pattern already used in:
- ADP WFN (
libraries/adp/workforce_now/api/resources.py:53-57) — @retry(stop=stop_after_attempt(5), wait=wait_exponential_jitter())
- Google Forms (
libraries/google/forms/resources.py:35) — same pattern
- Extracts (
libraries/extracts/assets.py:105-109) — same pattern
Retry on:
requests.exceptions.ConnectionError (includes timeouts when timeout kwarg is set)
requests.exceptions.Timeout
- HTTP 429 (rate limit) — respect
Retry-After header if present
- HTTP 502, 503, 504 (transient server errors)
Scope
GrowResource._request() in src/teamster/libraries/level_data/grow/resources.py
- Also consider
GrowResource._get_access_token() (OAuth2 token fetch can also time out)
- Check
OvergradResource — same level_data library, also has no retry logic and a TODO noting missing exponential backoff (libraries/overgrad/resources.py:32)
Out of Scope
Broader audit of all API resources for missing retry logic (separate issue).
Problem
The Grow API resource (
src/teamster/libraries/level_data/grow/resources.py) has no retry logic on HTTP requests. Whengrow-api.leveldata.comtimes out, the step fails immediately and the entire run relies on Dagster's run-level auto-retry — restarting the full run instead of retrying just the failed request.On 2026-04-07,
kipptaf__grow__multi_partition_assets_schedulefailed due to a connection timeout togrow-api.leveldata.com. The Dagster auto-retry succeeded 2 minutes later, but the entire run was re-executed rather than just the timed-out request.Proposed Change
Add
tenacityretry with exponential backoff + jitter toGrowResource._request(), matching the pattern already used in:libraries/adp/workforce_now/api/resources.py:53-57) —@retry(stop=stop_after_attempt(5), wait=wait_exponential_jitter())libraries/google/forms/resources.py:35) — same patternlibraries/extracts/assets.py:105-109) — same patternRetry on:
requests.exceptions.ConnectionError(includes timeouts whentimeoutkwarg is set)requests.exceptions.TimeoutRetry-Afterheader if presentScope
GrowResource._request()insrc/teamster/libraries/level_data/grow/resources.pyGrowResource._get_access_token()(OAuth2 token fetch can also time out)OvergradResource— samelevel_datalibrary, also has no retry logic and a TODO noting missing exponential backoff (libraries/overgrad/resources.py:32)Out of Scope
Broader audit of all API resources for missing retry logic (separate issue).