Skip to content

Forward ingest: time-aware guards to avoid after-hours/holiday churn #6

@1kuna

Description

@1kuna

Problem\nForward checks enqueue and workers process same-day tasks regardless of trading hours/holiday status. This causes churn (retries/403s) when markets are closed or data isn’t yet available. We recently added a blunt 2-day defer as a band-aid; we should replace it with time-aware logic.\n\n### Proposal\n- Gate forward task creation and execution by trading calendar + time-of-day.\n- Only schedule forward tasks on trading days; skip weekends/holidays.\n- For same-day pulls:\n - Require market open for intraday, and prefer post-close + small buffer for EOD.\n - Enforce vendor freshness windows (e.g., Massive T+1; IEX ~15-20 min delay).\n- In the worker, treat NOT_ENTITLED (e.g., Alpaca 403) as permanent for that vendor/dataset and fall back or mark failed; avoid infinite retries.\n- Keep the 15-min forward tick but make it a no-op when guards say "too early/closed".\n\n### Rationale\nAvoids infinite retry loops and queue churn after hours/holidays while still allowing near-real-time ingest when data is expected to be available.\n\n### Acceptance\n- Forward tasks are not enqueued/processed when the market is closed or vendor freshness window not met.\n- No repeated NOT_ENTITLED retries for the same vendor/dataset; fallback or fail fast.\n- Tests cover trading-day detection and time-of-day gating.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions