Kafka Auth (Source & Sink)

[turbolytics]

refs #90

• Publish test data now supports sasl/ssl config:

python3 cmd/publish-test-data.py --num-messages=1000 --topic="input-sasl-tls-1" --ssl-ca-location=$(pwd)/dev/kafka/certs/ca-cert.pem --ssl-key-location=$(pwd)/dev/kafka/certs/client-key.pem --ssl-certificate-location=$(pwd)/dev/kafka/certs/client-cert.pem --ssl-key-password=testpass --security-protocol=SASL_SSL --sasl-mechanism=PLAIN --sasl-username=user --sasl-password=bitnami

• Kafka source now supports the config as well

SQLFLOW_ROOT_DIR=$(pwd) python3 cmd/sql-flow.py run dev/config/examples/kafka.sasl-tls.yml

TODO

• Kafka Sink Supports SASL_SSL
• Test that publish still works with unauthenticated kafka
• Update schema validate for kafka sink config

Tests

Test Consume an Publish

docker-compose -f kafka-sasl-tls.yml up --build --force-recreate

python3 cmd/publish-test-data.py --num-messages=1000 --topic="input-sasl-tls-1" --ssl-ca-location=$(pwd)/dev/kafka/certs/ca-cert.pem --ssl-key-location=$(pwd)/dev/kafka/certs/client-key.pem --ssl-certificate-location=$(pwd)/dev/kafka/certs/client-cert.pem --ssl-key-password=testpass --security-protocol=SASL_SSL --sasl-mechanism=PLAIN --sasl-username=user --sasl-password=bitnami