It’s Okay to Store Data in Apache Kafka(confluent.io) |
It’s Okay to Store Data in Apache Kafka(confluent.io) |
I appreciated the mid-article note that "treating Kafka as a primary store does mean raising the bar for how you run it". Databases and filesystems are silently scary.
For instance, you cannot query Kafka and this is a basic requirement of most data stores. To get queryable data, you could build up state from events in some external component and then query that external component.
From what I understand, this building of state from events is an example of an Event Sourcing architecture.
I think event sourcing as a primary datastore is a cool idea. Its fun to play with such ideas on small projects and then scale up. The downside is that Kafka is a distributed monster that is hard to run locally (you need multiple docker containers for a basic setup). I'd be very interested to see if anybody else answers your question.
https://gist.github.com/17twenty/d619f922ab209f750630824f7c6...
https://www.confluent.io/blog/publishing-apache-kafka-new-yo...
All modern databases are essentially the same: they receive changes, save them to a write-ahead log or journal, then materialize the data into indexes and tables. This is the event-sourcing + compaction + materialized views model that kafka/confluent keeps pushing but inside a single piece of software that does all the hard things for you with arbitrary queries, reliable backups, etc. There's a reason why this is the common model for decades, there's nothing new here.
Splitting up all that logic into separate event streams, processing pipelines and datastores is certainly valid but only really necessary at the 1% of companies that operate at that scale. For most people, a relational database running on a mid-range server is plenty of power and far better long-term storage.