Show HN: Open-Source C++ Apache Iceberg Client with Write Support

Show HN: Open-Source C++ Apache Iceberg Client with Write Support(github.com)

8 points by jovezhong 1 year ago | 3 comments

Existing OSS C++ projects like ClickHouse and DuckDB support reading from Iceberg tables. Writing requires Spark, PyIceberg, or managed services.

In this PR https://github.com/timeplus-io/proton/pull/928, we are open-sourcing a C++ implementation of Iceberg integration. It's an MVP, focusing on REST catalog and S3 read/write(S3 table support coming soon). You can use Timeplus to continuously read data from MSK and stream writes to S3 in the Iceberg format. No JVM. No Python. Just a low-overhead, high-throughput C++ engine. Docker/K8s are optional. Demo video: https://www.youtube.com/watch?v=2m6ehwmzOnc

Help us improve the code to add more integrations and features. Happy to contribute this to the Iceberg community. Or just roast the code. We’ll buy the virtual coffee.

tingfirst 1 year ago |

Pretty cool to see a C++ R/W Iceberg client without dependency, and even better open-sourced. The pipeline is all about processing and routing, ideally, to open and flexible destination with no lock-in and long-term retention. Writing into Apache Iceberg is becoming critical to give users real control, rather than into specific data warehouses or lakehouses that are hard to move out.

ZiliangXK 1 year ago |

Look like other folks are doing similar things, wondering any benchmark results so far compared with Java implementation

gangtao 1 year ago | |

can you provide more information about the other similar project, would like to take a look