Skip to content

Handle empty environment overrides in vendor worker#4

Open
1kuna wants to merge 1 commit intomainfrom
codex/find-and-fix-a-bug
Open

Handle empty environment overrides in vendor worker#4
1kuna wants to merge 1 commit intomainfrom
codex/find-and-fix-a-bug

Conversation

@1kuna
Copy link
Owner

@1kuna 1kuna commented Oct 16, 2025

Summary

  • add a helper to safely parse integer environment overrides for the vendor worker CLI
  • default concurrency and RPM limits to zero when the corresponding environment variables are blank or invalid

Testing

  • python -m compileall trademl/workers/vendor_worker.py

https://chatgpt.com/codex/tasks/task_e_68f0a7871d2c8326bf51e7844bef8cde

1kuna added a commit that referenced this pull request Dec 9, 2025
## Critical Fixes (Data Correctness)

### Empty responses on trading days (#1)
- Added `_is_trading_day()` helper using XNYS calendar with weekday fallback
- Empty on trading day → AMBER (needs re-check), not GREEN
- Empty on weekend/holiday → GREEN with NO_SESSION

### Finnhub fundamentals phantom GREEN (#2)
- Changed to return EMPTY with qc_code="STORAGE_NOT_IMPLEMENTED"
- Worker now marks STORAGE_NOT_IMPLEMENTED as AMBER gap, not GREEN

### Nested date paths (#3)
- Fixed 8 occurrences of Parquet writes using parent.parent
- Prevents creating date=X/date=X/data.parquet nested paths

### Weekends marked RED (#4)
- Added trading day check in rows_by_date processing
- Non-trading days with 0 rows → GREEN/NO_SESSION in both workers

## Critical Fixes (Reliability)

### PARTIAL success handling (#5)
- Added FetchStatus.PARTIAL enum value
- Added failed_at_date and original_end_date to FetchResult
- Worker saves successful days, creates follow-up task for remaining
- Preserves original task.kind for follow-up tasks

### HTTP retry for 5xx errors (#6)
- Added retry with 2-4s jitter backoff for 500/502/503/504
- Counts retry attempts in budget

### Failed calls not counted in budget (#7)
- Moved budget.spend() before raise_for_status()
- Now counts ALL API calls for accurate rate limiting

## Medium Fixes

### Multi-day head-of-line blocking (#8)
- Added MAX_TASK_DAYS config per dataset (minute: 7, EOD: 30, fundamentals: 90)
- enqueue_task() auto-chunks large date ranges
- Prevents single task blocking vendor for hours

### Budget crash recovery (#9, #10, #11)
- Debounced saves (once/second) instead of every 10 calls
- Anti-burst protection: stale tokens reset to 1 on restart
- Explicit warning for missing state file, backup for corrupted files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments