Envelope and Encoding support #39
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat: Configurable envelope formats and wire encoding for sinks
Adds support for multiple output envelope formats across all sinks, enabling Debezium compatibility and CloudEvents support without changing the internal event model.
Envelope Formats
Native - Direct Event serialization (default, lowest overhead)
Debezium - Wire-compatible with Debezium's JsonConverter schemaless mode ({"schema":null,"payload":{...}})
CloudEvents - CNCF CloudEvents 1.0 specification
Configuration
yamlsinks:
config:
id: kafka-sink
brokers: localhost:9092
topic: events
envelope:
type: debezium # native | debezium | cloudevents
encoding: json # json (default), avro planned
Changes
Core:
New envelope module with Native, Debezium, and CloudEvents implementations
New encoding module with Json encoder (extensible for Avro/Protobuf)
Event structure aligned with Debezium payload format internally
Sources:
MySQL and PostgreSQL sources refactored to adopt envelope/encode model
Fixed MySQL DDL handling and schema reload
Sinks:
All sinks (Kafka, Redis, NATS) updated to use configurable envelope and encoding
Per-sink envelope configuration via YAML
Tests:
All sink integration tests updated for envelope/encoding support
Turso integration tests temporarily disabled
Docs:
New envelopes.md documentation with wire format examples
Updated README with quick start and envelope configuration
Updated example configs
Migration
Existing pipelines continue to work unchanged (defaults to native envelope, json encoding). Users migrating from Debezium can set envelope: { type: debezium } for drop-in compatibility.
closes #36 and #37