Hi HN, We're Luke and Phillip, and we're building Spice.ai OSS - a lightweight, portable runtime, built in Rust and powered by Apache DataFusion to locally materialize, accelerate, and query data tables sourced from any database, data warehouse or data lake.

Phillip and I first introduced Spice on Show HN in September 2021. Since then, we’ve been schooled and humbled in every way building 100TB+ data and ML systems for the https://spice.ai cloud platform. Along with our customers, we struggled with getting fast, low-latency, high-concurrency SQL query within a budget, accessing and combining data from many sources, trade-offs between OLTP/OLAP compute engines, and managing datasets as code.

Today, we’re re-launching Spice, completely rebuilt from the ground up, to directly solve several of the problems we had in accessing data quickly and cost-effectively providing it to applications, dashboards, and machine learning. Spice provides federated SQL query across databases (MySQL, PostgreSQL, etc.), data warehouses (Snowflake, BigQuery, etc.) and data lakes (S3, MinIO, Databricks, etc.) with the ability to materialize remote datasets locally using in-memory Arrow, DuckDB, SQLite, or PostgreSQL. Accelerated engines run in your infrastructure giving you flexibility and control over price and performance.

You can read the full announcement blog post at https://blog.spiceai.org/posts/2024/03/28/adding-spice-the-n....

We’d appreciate it if you check Spice out, give us feedback, and if you'd like to contribute, we'd love to build with you.

Thanks!

GitHub: https://github.com/spiceai/spiceai

user:	dwgray
created:	September 7, 2021
karma:	0