Separate data by topic/stream_key into distinct folders

[Copilot]

Data from different Kafka topics and Redis streams was being written to the same directory, making organization and querying inefficient.

Changes

Sources

• Kafka: Group messages by topic, add topic to RecordBatch schema metadata
• Redis: Group messages by stream_key/list_key, add stream_key to schema metadata

Sinks (S3, HDFS, Local)

• Extract topic/stream_key from batch metadata during flush
• Create separate folders: prefix/{topic}/data_*.parquet
• Maintains compatibility with partitioning: prefix/{topic}/{partition}/data_*.parquet

Folder Structure

Before:

s3://bucket/raw-data/
  data_20241113_120000_000001.parquet  # mixed topics
  data_20241113_120001_000002.parquet

After:

s3://bucket/raw-data/
  events/
    date=2024-01-01/data_*.parquet
  logs/
    date=2024-01-01/data_*.parquet
  metrics/
    data_*.parquet

Backward Compatibility

Batches without topic/stream_key metadata write to base directory as before. No configuration changes required.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Sink should store data from different topics/stream_keys separately in different folders</issue_title>
<issue_description></issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Sink should store data from different topics/stream_keys separately in different folders #10

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.