Split Wonderware pipeline into reusable connector + pipeline

[514Ben]

Summary

This PR splits the Wonderware pipeline into two components:

1. Reusable Connector (connector-registry/wonderware/) - Handles data access from Wonderware Historian
2. Focused Pipeline (pipeline-registry/wonderware_to_clickhouse/) - Handles ClickHouse storage

This follows the established SAP HANA CDC pattern and enables the Wonderware connector to be reused by other pipelines.

---

🎯 Changes

✨ New: Wonderware Connector

Created complete connector in connector-registry/wonderware/ with 4-level hierarchy:

• Root → Author (514-labs) → Language (python) → Implementation (default)

Core modules:

• config.py - Connection configuration (host, port, database, credentials)
• connection_manager.py - SQLAlchemy connection pool with circuit breaker pattern
• reader.py - Data extraction (discover_tags, fetch_history_data, get_tag_count, test_connection)
• connector.py - High-level facade providing simple API
• models.py - Domain models (TagInfo, HistoryRow, ConnectorStatus)
• Complete test suite with SQLite mock fixtures

Metadata:

• Category: historian
• Capabilities: Extract ✅, Transform ❌, Load ❌
• Tags: historian, scada, aveva, wonderware, sql-server, industrial

🔄 Updated: Pipeline

Configuration changes:

• wonderware_config.py: Renamed WonderwareConfig → PipelineConfig
• Removed all connection fields (now in connector)
• Changed env prefix: WONDERWARE_ → WONDERWARE_PIPELINE_
• Kept only pipeline-specific fields: tag_chunk_size, backfill_chunk_days, sync_schedule, etc.

Workflow updates:

• wonderware_sync.py: Uses WonderwareConnector with clean imports
• wonderware_backfill.py: Uses WonderwareConnector with clean imports
• Added _get_cached_tags() helper for Redis caching in sync workflow

Infrastructure:

• Added symlink: app/wonderware → connector source (enables clean imports without path manipulation)

Tests:

• Updated test_wonderware_config.py to test PipelineConfig
• Updated conftest.py with new env var prefix
• Added test to verify connection fields are NOT in pipeline config

🗑️ Deleted

• app/workflows/lib/wonderware_client.py - Logic moved to connector

---

📋 Environment Variables

Connector (unchanged from original)

WONDERWARE_HOST           # Required
WONDERWARE_PORT           # Default: 1433
WONDERWARE_DATABASE       # Default: Runtime
WONDERWARE_USERNAME       
WONDERWARE_PASSWORD       
WONDERWARE_DRIVER         # Default: mssql+pytds

Pipeline (new prefix)

WONDERWARE_PIPELINE_TAG_CHUNK_SIZE           # Default: 10
WONDERWARE_PIPELINE_BACKFILL_CHUNK_DAYS      # Default: 1
WONDERWARE_PIPELINE_SYNC_SCHEDULE            # Default: */1 * * * *
WONDERWARE_PIPELINE_BACKFILL_OLDEST_TIME     # Default: 2025-01-01 00:00:00
WONDERWARE_PIPELINE_TAG_CACHE_TTL            # Default: 3600

---

✅ Benefits

1. Reusability - Wonderware connector can be used by other pipelines
2. Separation of Concerns - Connector handles data access, pipeline handles storage
3. Maintainability - Updates to connection logic only need to happen in the connector
4. Consistency - Follows established SAP HANA CDC pattern
5. Testing - Each component has independent test suite

---

🧪 Verification

• ✅ All connector Python files compile without syntax errors
• ✅ All pipeline Python files compile without syntax errors
• ✅ Symlink resolves correctly to connector source
• ✅ Import chain works: from wonderware import WonderwareConnector
• ✅ No path manipulation required (clean imports)

---

📊 Files Changed

• 28 files changed, 2,183 insertions(+)
• 18 new files in connector-registry
• 10 files in pipeline (new/modified)

---

🔍 Review Notes

Please verify:

• Connector metadata is correct
• Symlink works in your environment
• Environment variable naming is acceptable
• Tests are comprehensive
• Documentation is clear

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
registry	Ready	Preview, Comment	Feb 6, 2026 11:24pm