What is it? - Tansu is an Kafka compatible broker with S3 or PostgreSQL storage - Broker schema validation of Proto/Avro/JSON backed topics - Proto/Avro/JSON backed topics are also written as Parquet record batches for easy ingestion by downstream tooling Step by step tutorial: https://blog.tansu.io/articles/parquet Tansu is written in async Rust with the Tokio runtime. We build a single static linked binary that contains the broker, proxy and admin/utility commands in ~60MB. Also available as a from scratch docker container. The Kafka protocol is implemented with serde using proc macros to transform the JSON definitions into Rust data types (https://blog.tansu.io/articles/serde-kafka-protocol). Proto/Avro/JSON is converted into Apache Arrow data types, and then into Apache Parquet with a sprinkle of Apache Datafusion. We also have an example of it running on fly.io with tigris (https://blog.tansu.io/articles/fly-with-tigris) that scales brokers back to 0 between requests. Looking forward to any feedback and hope that Tansu can improve your workflow! |