Tutorial: How to stream to DuckLake

[turbolytics]

refs #136

• Start kafka

make start-backing-services

• Publish 1MM Events to Kafka

python3 cmd/publish-test-data.py --num-messages=1000000 --topic="input-user-clicks-ducklake"

• Start Consumer, Read 1MM events and Then Exist

SQLFLOW_BATCH_SIZE=1000 python3 cmd/sql-flow.py run dev/config/examples/kafka.ducklake.yml --max-msgs-to-process=1000000

• Verify 1MM Events are written to ducklake

D INSTALL ducklake;
D  LOAD ducklake;
D ATTACH 'ducklake:my_ducklake.ducklake' AS my_ducklake;
D       USE my_ducklake;
D SELECT COUNT(*) FROM my_ducklake.events;
┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│    1000000     │
│ (1.00 million) │
└────────────────┘