_  __       __ _            _
| |/ /__ _ / _| | ____ _   | |    ___ _ __  ___
| ' // _` | |_| |/ / _` |  | |   / _ \ '_ \/ __|
| . \ (_| |  _|   < (_| |  | |__|  __/ | | \__ \
|_|\_\__,_|_| |_|\_\__,_|  |_____\___|_| |_|___/

Real-time particle visualization of data flowing through Kafka and Flink

Prereqs

• Docker + Docker Compose
• Python 3 + pip (for gen.py)

Start

docker compose up -d
docker ps

UIs:

• Kafka UI: http://localhost:8085
• Flink UI: http://localhost:8081

Create Kafka topics

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic input --partitions 1 --replication-factor 1

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic output --partitions 1 --replication-factor 1

Run the Flink SQL job

docker exec -it flink-jobmanager ./bin/sql-client.sh -f /sql/flink.sql

Generate test data (host -> Kafka)

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt 2>/dev/null || pip install kafka-python
python gen.py

Verify output

docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic output --from-beginning

Lens Visualization

Real-time visualization of messages flowing through the Kafka -> Flink pipeline as glowing particles on a dark canvas.

Run locally (after Kafka + gen.py are running):

python lens_bridge.py
# Open http://localhost:8090

Or via Docker — it's included in docker compose up.

• Lens UI: http://localhost:8090

Notes

• Host Kafka bootstrap: localhost:9092
• Container Kafka bootstrap: kafka:19092

Logs / troubleshooting

docker logs kafka --tail 100
docker logs flink-jobmanager --tail 100
docker logs flink-taskmanager --tail 100
docker logs kafka-ui --tail 100

Cleanup

docker compose down -v