GCP and Confluent partner to deliver a managed Apache Kafka service(cloud.google.com) |
GCP and Confluent partner to deliver a managed Apache Kafka service(cloud.google.com) |
Subtle differences in the semantics of pub/sub and message passing services can have really significant consequences for their use-cases.
Google Pubsub is at-least-once delivery, and best-effort ordering. That means any consumer pipelines need to be able tolerate duplicates and out-of-order messages - in many cases, that's not trivial to handle. If the order of your messages isn't that important, and if you can make the operations of consumers idempotent - pub/sub is awesome. But more often than not, it becomes just another message passing service to add to your plethora of message passing services, because its limitations can't extend to all of your use-cases. (I really want one ring to rule them all - Kafka gets the closest, IMHO)
Kafka is an basically pub/sub on top of an ordered, append-only log, and consumers read the log stream very much like a single process reads/seeks on a file handle - using offsets. Given infinite storage - your entire data stream can be replayed, and any data can be recreated from scratch - thats a pretty awesome thing.
Try Apache Pulsar: https://pulsar.incubator.apache.org/
If you just want an event stream with loose ordering, no throughput or partition management, and the ability to 'ack' each individual message, than GCP pubsub is a pretty good choice. It can also do push-based subscriptions. The client libraries are rather buggy though and end up with memory leaks and network failures.
Kafka meanwhile is much more compatible with many other services that are typical run these days from stream processing to databases. That being said, Kafka is also pretty easy to run and VMs are extremely reliable on any cloud so I don't see much value in managed solutions for that, unless someone just wants to completely de-risk. The bandwidth costs add up quickly for even moderate usage.
Confluent is run by the same folks from LinkedIn who built Kafka originally (it talks about this a bit more in the article), so you're in good hands with regard to the product's vision remaining consistent to the original innovation.
This enables different kinds of use cases, and can be easier to reason about.
The downside is kafka doesn't provide guaranteed delivery out of the box, the producers buffer messages and that's when the loss can happen - if you need guaranteed delivery then you need to write extra code.
I know I found it a bit non-trivial to run, so if I was still using Kafka on GCP boxes anyway I'd absolutely try this out.
If you are comfortable at operations you'll be fine. Some people are not good at ops so outsourcing the problem making the ops side someone else's issue can also be useful.
Self hosting will offer far more options when it comes to scaling and tweaking. Overall on bare hardware costs it's cheaper and faster although up front costs will be higher.
Kafka usecases are rarely elastic so don't gain that advantage in the cloud. Also Kafka's missing tierd storage makes it expensive if storing big volumes of data.
In practice, it’s better to offload things that are not core to what your company is making money on, until you hit constraint points from scaling. As one of the comments mentioned, at PB-scale processing, even bare metal may make sense. (But not always - one of my former employers went down this path early and ended up losing all productivity in R&D because of people fighting to keep that baremetal setup alive for months on end. This really hurt future revenue growth and distracted eng. leadership from key changes in their industry.)
Like with most complex questions, the immediate answer is “it depends”.
Plenty of hosted Kafka providers these days, it's not that difficult to run and the advantage of Confluent as the original authors is really just marketing. They're also working on a Kafka operator (one of many in the market) to run it even easier on Kubernetes.
This makes it possible to easily update your event processing logic or add new components to the mix while maintaining a clean architecture.
I gave a talk about this at PostgresConf US a few weeks ago, the talk's not yet available as an article, but you can get the slides from https://aiven.io/blog/aiven-talks-in-postgresconf-us-2018/ if you're interested.
The concept of "stream-table duality" tells us tables can be thought of as a materialized view of change data operations [INSERT/UPDATE/DELETE] streams. Kafka can be used as a buffer for streaming data that can materialized into a relational table at any time.
One of the more interesting use-cases is multi-target replication: feed your change-data-capture data into Kafka, and replay it on any other backend data store (SQL, Graph Database, NoSQL, etc.)[1]
Conceptually this lets you ingest data into a stream, write the data on multiple backends and keep everything in sync. Martin Klepmann has written a simple (PoC-quality) tool for doing this with Postgres databases.
[1] https://www.confluent.io/blog/bottled-water-real-time-integr...
Consumers read from the topic (the underlying partitions) and maintain the offset last read themselves, allowing for easy replay, strict ordering within a single partition, and completely disconnected pacing from producers. The optional "key" for each message allows for compaction so only the latest key/value pair is kept in a topic.
It's definitely not a database but works well for replicating them, as well as doing most of the work of a typical message queue / service bus system.
We have been hearing more and more about people using Kafka to support streaming analytics. I haven't spent a ton of time studying it, I was under the impression that it was just a giant queue, but ordering + retention is basically a database in the way I think about it.
Also consumers can be in a "consumer group" so you can have multiple clusters of consumers each reading the entire log separately, but shared within each cluster.
Also I'd recommend looking at Apache Pulsar for a next-generation architecture that combines Kafka's log semantics with the low-latency routing and individual message acknowledgements of message queues: https://pulsar.incubator.apache.org/
If Kafka supported data processing (queries or whatever) then it would be much closer to databases. Also, databases are normally aware of the structure of the data (for example, columns).
Therefore, Kafka can hardly be viewed as a DBMS because it explicitly separates two major concerns:
* data management - how to represent data (Kafka)
* data processing - how to derive/infer new data (Kafka Streams - a separate library)
Theoretically, if they could combine these two layers of functionality in one system then it would be a database.
https://engineering.linkedin.com/distributed-systems/log-wha...