Accurate trading‑minutes coverage per ticker (inception & special closures aware) #3
Description
Problem
Current coverage metrics reflect what we’ve downloaded, but not how close we are to the theoretical maximum. We also don’t account for ad‑hoc market closures (natural disasters, national days of mourning/funerals, 9/11, Sandy, etc.), so % progress can be inflated or deflated.
Proposal
- Compute per‑ticker theoretical trading minutes from the ticker’s inception to today using exchange_calendars (XNYS) with adhoc closures included.
- Determine per‑ticker inception from reference (e.g., Polygon list date / Alpaca assets) or fallback to first ingested row.
- Handle symbol changes/delistings by mapping across corporate actions so coverage follows the instrument, not just a string.
- Count downloaded minutes per ticker from manifests/curated partitions and compare to theoretical to produce % coverage and gap minutes.
- Emit a daily JSON/CSV report (ops/reports/coverage) and optional alert when coverage < threshold.
Data Sources
- exchange_calendars.XNYS (includes holidays + known ad‑hoc closures)
- Reference: polygon_universe (list date), delistings, corp actions
- Manifests in S3 (manifests/*) and/or curated/raw partition counts
Implementation Sketch
- Library: compute_theoretical_minutes(ticker, start, end, calendar='XNYS') → int
- Inception resolver: get_inception_date(ticker) → date (reference→fallback to first row)
- Coverage job: coverage_progress_report(window|full) writes per‑ticker % coverage, gaps; top offenders
- Wire into monitoring/coverage and schedule alongside audit
Acceptance Criteria
- For 10 sample tickers, % coverage matches hand‑computed values within 0.5% using known closure dates
- Ad‑hoc closures (e.g., 2018‑12‑05 national day of mourning) reduce theoretical minutes appropriately
- Report persisted to S3 and visible in ops/reports; includes per‑exchange configuration hook
Config
- ENV: COVERAGE_EXCHANGE (default XNYS), COVERAGE_LOOKBACK_DAYS (default full), COVERAGE_ALERT_THRESHOLD (e.g., 95)
Notes
- exchange_calendars is already in deps; leverage its adhoc closures; extend calendars if we find gaps.