Feature Req: Additional data source - Apache Kafka

[bspowell]

Overview of Problem

Helios currently supports data ingestion only through Amazon Kinesis. This limitation restricts potential users who rely on other popular streaming platforms like Apache Kafka.

Solution

Expand Helios' data ingestion capabilities to include Apache Kafka. This can be done in two ways:

1. Amazon MSK Integration:
   Integrate with Amazon Managed Service for Apache Kafka. This will allow the team to leverage existing Lambda function logic for a quicker implementation.
2. Direct Kafka Support:
   The other approach would be to re-architect the Lambda Connector function to support direct Kafka integration. This would enable broader Kafka support beyond Amazon's managed service.

Supporting Links

https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink