Add ADS-B.lol and OpenWeather connectors Closes ENG-638 and ENG-639

[oatsandsugar]

To Do

• conform connectors to updated spec, for viz

What was added:

• Complete TypeScript connector for ADS-B.lol API (aircraft tracking)
• All 10 API endpoints with user-friendly methods (trackByICAO, findNearby, etc.)
• Full specification compliance (95%) with all resilience patterns:
  • Structured errors with standard codes (NETWORK_ERROR, TIMEOUT, etc.)
  • Token bucket rate limiting with adaptive server feedback
  • Circuit breaker with CLOSED/OPEN/HALF_OPEN states
  • Request ID generation and correlation tracking
  • AbortController cancellation support throughout
  • Retry mechanism with exponential backoff, jitter, and retry budget
  • Concurrent request limiting with proper resource management
  • Data transformation with schema validation
• Comprehensive testing suite (15/15 tests pass):
  • Unit tests for all core functionality
  • Live API tests against real ADS-B.lol endpoints
  • Performance and error handling validation
• Complete documentation with compliance audit and testing guide

Prompt History for Replicating with Other APIs:

Phase 1: Initial Setup

"Connectors are organized in the following structure (example for S3 connector): connector-factory/connectors/fiveonefour/s3/typescript --> lets create a connector together based on the specs you just saw, and this API https://api.adsb.lol/docs. First, create the directory and the readme for the connector. Next, propose what the connector would look like given the specification. Then, lets discuss what we want to include. Finally, lets draft the connector. Lets use Typescript."

Phase 2: User Interface Design

"what is the user interface like? I don't want them to have to think about the underlying apis"

[Assistant proposes user-friendly interface hiding API complexity:

• trackByICAO() instead of GET /v2/icao/{hex}
• findNearby() instead of GET /v2/lat/{lat}/lon/{lon}/dist/{radius}
• getEmergencies() instead of GET /v2/sqk/7700]

"I think also methods that return all aircraft, aircraft by airline (if possible), aircraft by military or not"

Phase 3: Specification Compliance

"'/Users/.../api-connector.mdx' please check the implementation against the spec"

[Assistant performs detailed compliance audit, identifies gaps]

"can you fix these gaps?"

"work on one at a time"

[Assistant implements each feature incrementally:

1. Structured error handling with standard codes
2. Token bucket rate limiting
3. Circuit breaker pattern
4. Request ID tracking
5. AbortController cancellation
6. Data transformation methods
7. Enhanced retry mechanisms
8. Concurrent request limiting]

Phase 4: Documentation and Compliance

"Can you add another md in the connector directory showing the compliance with the spec? and explaining where it deviates?"

"if they aren't implemented, explain why."

[Assistant adds detailed explanations for each unimplemented feature:

• API limitations (no batching endpoints)
• Platform delegation (TLS handled by runtime)
• Architecture decisions (connection pooling conflicts with HTTP/2)]

Phase 5: Testing Strategy

"how can you light weight test this connector?"

"no i want to test against the actual API"

[Assistant creates live API test suite with 7 comprehensive tests]

"run all the tests"

[Assistant sets up testing infrastructure, resolves pnpm/workspace issues, runs complete test suite]

Key Implementation Patterns for Other Connectors:

1. Start with API exploration: Fetch OpenAPI specs, understand endpoints
2. Design user-friendly interface: Hide API complexity behind domain methods
3. Implement spec compliance incrementally: One pattern at a time
4. Build comprehensive testing: Unit tests + live API tests
5. Document thoroughly: README, compliance audit, testing guide
6. Use structured errors: Standard codes with proper classification
7. Implement all resilience patterns: Rate limiting, circuit breaker, retries
8. Add observability: Request IDs, status methods, detailed metadata

Total implementation: ~2000 lines of production-ready TypeScript with 95% specification compliance.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Project	Deployment	Preview	Comments	Updated (UTC)
connector-factory-components-docs	✅ Ready	Preview	Comment	Aug 13, 2025 8:33pm

[Copilot]

Pull Request Overview

This PR introduces a comprehensive, production-ready TypeScript connector for the ADS-B.lol aircraft tracking API with 95% specification compliance. The connector provides user-friendly methods for real-time aircraft tracking while implementing all required resilience patterns including structured error handling, token bucket rate limiting, circuit breaker protection, retry mechanisms with exponential backoff, and request cancellation support.

Key changes include:

• Complete ADS-B.lol API integration with 10 endpoints mapped to domain-specific methods
• Full specification compliance with structured errors, rate limiting, circuit breaker, and retry patterns
• Comprehensive testing suite with both unit tests and live API validation
• Complete documentation including compliance audit and testing guides

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
types.ts	Defines structured error handling with standard error codes and classification
index.ts	Main connector implementation with full API coverage and resilience patterns
rate-limiter.ts	Token bucket rate limiting with adaptive server feedback
circuit-breaker.ts	Circuit breaker implementation with state management
data-transformer.ts	Schema validation and data transformation utilities
connector-types.ts	TypeScript interfaces for connector specification compliance
package.json	Project configuration with comprehensive test scripts
tsconfig.json	TypeScript compilation configuration
jest.config.js	Jest testing framework configuration
Test files	Comprehensive test suite including unit, integration, and live API tests
Documentation	Complete README, testing guide, and specification compliance audit

[oatsandsugar]

Feedback on the UX from the LLM, noting that "fixing" these would mean breaking away from the connector spec

Developer Experience Feedback - ADS-B Connector

Current State Analysis

The ADS-B connector is functionally complete with 95% specification compliance, but the developer experience could be simplified significantly. While the current interface works well for enterprise applications requiring fine-grained control, it may be too complex for typical developers who just want aircraft data.

Current Friction Points

1. Too Much Boilerplate

Current approach:

const connector = new AdsbConnector();
await connector.initialize();
await connector.connect();
const aircraft = await connector.findNearby(33.9425, -118.4081, 50);

Issue: Three steps to get started feels like Java EE - most developers expect simpler APIs.

2. AbortController Complexity

Current approach:

const controller = new AbortController();
const aircraft = await connector.findNearby(lat, lon, radius, controller.signal);

Issue: Many developers won't understand AbortController or why they need it for basic usage.

3. Structured Error Handling

Current approach:

catch (error) {
  if (error.code === 'NETWORK_ERROR') {
    // Handle network issues
  }
}

Issue: Requires understanding of error codes and structured error handling patterns.

4. Enterprise Patterns Exposed

Current concepts developers must understand:

• Token bucket rate limiting
• Circuit breaker patterns
• Request correlation IDs
• Retry budgets
• Exponential backoff

Issue: These are implementation details that most developers don't need to think about.

Recommended Improvements

1. Auto-initialization Option

// Option A: Auto-connect constructor
const adsb = new AdsbConnector({ autoConnect: true });
const aircraft = await adsb.findNearby(33.9425, -118.4081, 50);

// Option B: Static methods for one-liners
const aircraft = await AdsbConnector.findNearby(33.9425, -118.4081, 50);

// Option C: Keep current API but make lifecycle optional
const adsb = new AdsbConnector();
// Works without explicit initialize/connect
const aircraft = await adsb.findNearby(33.9425, -118.4081, 50);

2. Simple Timeout API

// Instead of AbortController
const aircraft = await connector.findNearby(lat, lon, radius, { 
  timeout: 5000 
});

// Or as a connector-level setting
const adsb = new AdsbConnector({ timeout: 10000 });

3. Quick Start Documentation

Add a "Quick Start" section to README with one-liner examples:

// Get aircraft near a location
const planes = await new AdsbConnector().findNearby(34.0522, -118.2437, 25);

// Track specific flight
const flight = await new AdsbConnector().trackByCallsign('UAL123');

// Get military aircraft
const military = await new AdsbConnector().getMilitary();

// Monitor area with callback
const monitor = new AdsbConnector();
monitor.watchArea(34.0522, -118.2437, (aircraft) => {
  console.log(`${aircraft.length} aircraft detected`);
});

4. Common Use-Case Methods

Add more intuitive methods for common scenarios:

// Overhead flights (defaults to reasonable radius)
await connector.getFlightsOverhead(lat, lon);

// Airport traffic (if we can map codes to coordinates)
await connector.getAirportTraffic('LAX');

// Streaming updates
await connector.watchArea(lat, lon, callback);

// Emergency situations nearby
await connector.getEmergenciesNearby(lat, lon, radius);

// Popular flight routes
await connector.getFlightsByRoute('LAX', 'JFK');

5. Simplified Error Handling

// Simple boolean checks
try {
  const aircraft = await connector.findNearby(lat, lon, radius);
} catch (error) {
  if (error.isNetworkError) {
    // Retry logic
  } else if (error.isTimeout) {
    // Increase timeout
  } else {
    // Other handling
  }
}

// Or simple retry helper
const aircraft = await connector.withRetry(() => 
  connector.findNearby(lat, lon, radius)
);

6. Configuration Presets

// Instead of detailed configuration
const adsb = new AdsbConnector('development'); // Fast, loose limits
const adsb = new AdsbConnector('production');  // Conservative, robust
const adsb = new AdsbConnector('realtime');    // Optimized for frequent calls

Implementation Strategy

Phase 1: Backward Compatible Simplification

• Add auto-initialization option to constructor
• Add timeout parameter as alternative to AbortController
• Add convenience methods for common use cases
• Keep all existing methods unchanged

Phase 2: Enhanced Developer Experience

• Add static methods for one-liner usage
• Add streaming/watching capabilities
• Add configuration presets
• Enhanced error messages with suggestions

Phase 3: Advanced Features (Optional)

• Airport code to coordinate mapping
• Flight route analysis
• Historical data integration
• WebSocket streaming support

Success Metrics

Before: 3-4 lines of boilerplate + understanding of enterprise patterns
After: 1 line for basic usage, optional complexity for advanced use cases

The goal is to make the connector accessible to all skill levels while maintaining the robust enterprise features for those who need them.

Priority

High Priority: Phase 1 improvements (backward compatible)
Medium Priority: Enhanced documentation with more examples
Low Priority: Phase 2 and 3 advanced features

This would significantly improve the developer onboarding experience while preserving all existing functionality.

[oatsandsugar]

Chat History

This comment documents the significant engineering decisions and changes made during development after the initial implementation.

Registry Structure Compliance

Major Issue: CEO feedback that file structure is critical for connector discovery and tooling

Analysis of Google Analytics Example:

• Discovered our structure didn't match the canonical pattern
• Google Analytics has proper meta directories and schema organization
• Our connector had correct path but wrong internal structure

Restructuring Changes:

1. File Organization (Commit: 33ccc15)

   • Moved source files to src/ directory structure
   • Moved documentation to docs/
   • Consolidated test files in tests/
   • Added examples/ with working code samples

2. ETL Directory Architecture Decision:

   • Question: Should ADS-B connector populate extract/transform/load directories?
   • Analysis: ADS-B is pure API connector (HTTP GET → JSON), not ETL pipeline
   • Decision: Keep ETL directories empty with .gitkeep files
   • Rationale: Simple API client vs complex data processing (like Google Analytics)

Schema Documentation Enhancement

Issue: Schemas exist in code but not properly documented for tooling integration

Google Analytics Pattern Analysis:

• Uses structured datasets array instead of flat schemas object
• Separates raw vs extracted stages
• Supports both json and relational formats
• Each dataset references documentation files

Implementation (Commit: d22e13e):

1. Updated index.json structure:

   {
     "$schema": "https://schemas.connector-factory.dev/schema-index.schema.json",
     "datasets": [
       {"name": "aircraft", "stage": "raw", "kind": "json", ...}
     ]
   }

2. Added comprehensive relational schemas:

   • Raw tables: aircraft, aircraft_snapshots
   • Extracted tables: aircraft_enriched, api_responses
   • Complete SQL DDL with indexes, views, and triggers
   • Documentation for each schema type

Developer Experience Analysis

Question: Is the connector interface simple enough for typical developers?

Current Complexity Issues Identified:

• Requires 3 steps: new AdsbConnector() → initialize() → connect()
• AbortController pattern unfamiliar to many developers
• Enterprise patterns (circuit breaker, retry budget) exposed to basic users
• Structured error codes require knowledge of error handling patterns

Recommendations Documented (not implemented):

• Auto-initialization option
• Simple timeout parameter instead of AbortController
• Static methods for one-liner usage
• Configuration presets for different use cases

API Connector Specification Enhancement

Issue: Specification didn't document critical registry structure requirements

Added to Specification:

• Complete directory structure requirements with examples
• Meta folder contents and requirements
• Schema compliance rules
• Structured examples of connector.json files
• Warning about tooling dependencies on structure

Impact: Future LLMs and developers will have clear guidance on registry structure compliance.

Final Architecture

The connector now represents a working, production-ready API client that follows all registry conventions:

• ✅ 95% specification compliance with all resilience patterns
• ✅ Proper registry structure matching canonical examples
• ✅ Comprehensive schema documentation for tooling integration
• ✅ All tests passing: 8/8 unit tests, 7/7 live API tests

Key Architectural Decision: Maintained simple HTTP client design rather than complex ETL pipeline, appropriate for real-time aircraft tracking API.

[linear]

ENG-638 ADSB

ENG-639 OpenWeather