Live API Connectors — Real-time municipal data ingestion

[GareBear99]

Overview

Replace static JSON source files with live API connectors to municipal open data endpoints.

Scope

• Vancouver Open Data API connector
• Calgary Open Data API connector
• Edmonton Open Data API connector
• Winnipeg Open Data API connector
• Montréal Open Data API connector
• Scheduled refresh intervals with delta detection
• Source freshness tracking per API endpoint
• Error recovery and retry logic for API failures

Technical Notes

• Current architecture reads from data/raw/source_a.json and source_b.json
• Connectors should output the same JSON schema so the resolver pipeline works unchanged
• Consider a lightweight Node.js or Python scheduler for periodic fetches
• Rate limiting and caching strategy needed per municipality

Acceptance Criteria

• At least 3 municipal APIs connected and fetching live data
• Delta detection avoids re-processing unchanged records
• Source freshness visible in pipeline health dashboard
• Graceful fallback to last-known data on API failure