The new local data stack: Integrating Dagster, dbt and DuckDB(georgheiler.com) |
The new local data stack: Integrating Dagster, dbt and DuckDB(georgheiler.com) |
1. Usually, you want to run some warehouse all the time. Bring their data through ETL, run transformation, and report. This goes against the local environment. Ideally, I would love a cloud warehouse, which each engineer could easily fork to their laptop.
2. Almost all companies already have some data setup. The migration path is very unclear. Most likely, this is a secondary system for the majority of companies. Ideally, I would love to describe how I can use it along big platforms (e.g. BigQuery or Snowflake).
After the initial imports and some massaging with queries that felt awesome, I found hard to step-up the game building the relationships I wanted. The last stroke before switching was the lack of managing foreign keys without recreating entire tables again. I can go over other examples.
It can be done, but it just takes you out of the flow when you're analyzing the data + cleaning it, specially because I know that I can do with psql and sqlite in a blink of an eye.
Since many etl tools don't care about the target database being these old and trusty fellas, I felt losing a lot of time just to get rid of a postgres install that is right now consuming only 200 mb of ram on a docker/podman container. Or working around some sqlite ingestion issues with simple notebooks + pandas/polars/etc.
in my pov it seems a shaky ground for an entire new stack
I appreciate duckdb taking me out of the comfort zone tho.
We run this stack in production for the last few months, and it has its downsides (I would argue due to the young ecosystem) and upsides, which we try to explain in the blog. We wanted to concentrate more on the concepts and technology change/improvement that allows us to run such a stack and explain how we see the future steps forward.
1. Running a warehouse is not a bad idea, but you must always be careful to separate the storage from the compute to scale. I experienced the limitation of such a system as described in this blog https://delta.io/blog/2022-09-14-why-migrate-lakehouse-delta... -> tough position if your solution is good but not scale. Ideally run it with external tables in order that data is visible without engine access
Another limitation is the metastore for your tables and metadata, which you usually have per workspace/environment in such a scenario. Databricks' unity catalog is an excellent way to solve it, but it is only compatible with some engines.
2. We do not think that this stack exists to exchange the snowflake or big query but to take a part of the workload away ( data transformation) and let PaaS solutions be good at what they are made for -> user interface and interaction.
Georg - one of the authors is here.
What we argue is that:
- for a great software/data engineering/creation experience we recommend such a stack that is only on when needed (when transformations occur)
- for a great data consumption experience we suggest the integration with an established PaaS platform. Not only for the sake of being available (as a serving layer of data to end users) but also for the missing fine-grained RBAC in the proposed transformation layer
We're experimenting with using DuckDB as the warehouse. Would be awesome to let people pull down parts of their warehouse locally for testing.
Georg - one of the authors is here.
In fact, this is exactly what we argue: Keep data the consumer experience high a Paas platform (Fabric, BigLake, Databricks, SF, ...) can make a lot of sense, whereas for the best data development/creator experience a high-code solution based on solid software engineering best practices should be the preferred solution - at least in my/our(authors) opinion.
I would like to quickly grab some data, queries and dashboards and run them in my local warehouses.
Also for 99% of companies and data workloads the optimal choice should usually be a janky bash script, so this provides meaningful competition for annoyingly medium data.
why remote setup is not enough for those teams?..
https://count.co/product/data-modeling/
and associated blog post:
https://count.co/blog/how-we-made-data-modeling-multiplayer/
Dagster versus dbt https://dagster.io/vs
Migrating off dbt cloud https://dagster.io/blog/migrate-off-dbt-cloud
¹ https://github.com/davidgasquez/gitcoin-grants-data-portal
What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?
The common stack is then to ingest them from Kafka into an OLAP datastore like ClickHouse or Pinot.
Then I think you are referring to what is sometimes called “user facing analytics” against that datastore.
We like to use a tool called Cube.dev as middleware between the app and OLAP database which gives APIs and a semantic layer.
> we argue that we can rethink the current state of data transformation pipelines (intro)
> In the blog we will cover:
> We show a way how you can combine the best of both worlds
> We will tackle its impact and explain more in the following development section.
edit: yeah it's ChatGPT:
> Parts of this text were adeptly generated by ChatGPT but enhanced by real humans.
I guess it's the future, turn a tweet into a 5-page essay with your AI, so your readers can summarize it back to tweet-length with their AI.
Georg - one of the authors is here.
Indeed, we used LLMs/GPT4 for proof reading and enhancing the English language (we are not native speakers).
We were thinking about breaking up the content - but decided that one long post is a better fit.
> > We show a way how you can combine the best of both worlds
In fact, that sentence I wrote by hand : )
the Dagster folks have some comparisons, of course there are popular modern alternatives other than Dagster
Please take a read here: https://georgheiler.com/2023/12/11/dagster-dbt-duckdb-as-new...
We (Georg Heiler and Aleksandar Milicevic) are keen to discuss the proposed new stack with you. Do not hesitate to reach out.
The only trick with local is that data sets of any appreciable size take ages to pull down, at least here in the US with our terrible internet (in the average Italian mountain village this would probably work great).
In such a case the network transfer (for many cases) can be almost irrelevant - at least for up to medium-ish sized datasets.
Submitted one bug report for the thing I didn't know how to fix.
Also DBT and Airflow are different tools, aimed at solving completely different problems. The Airflow competitor in this article is Dagster, which is also open source.
So thanks!