Ducklake is a new lake storage format by MotherDuck. It aims to solve some of the issues with Iceberg by centralizing metadata in Postgres, instead of directly on blob storage.

SQLFlow is a stream processing engine that ingests data from kafka, runs sql against that stream and sinks the output.

SQLFlow has a duckdb context available during stream processing. This make it trivial to stream data from Kafka to Ducklake!

https://sql-flow.com/docs/tutorials/ducklake-sink/

https://github.com/turbolytics/sql-flow