Streaming Platform Comparision:Kafka/Confluent/Pulsar/AutoMQ/Redpanda/Warpstream

Streaming Platform Comparision:Kafka/Confluent/Pulsar/AutoMQ/Redpanda/Warpstream(github.com)

27 points by jackbauer24 2 years ago | 4 comments

We have been utilizing Apache Pulsar for our messaging needs, however, we've found it to be quite complex and prone to numerous bugs. These issues have unfortunately led to significant downtime within our production environment, impacting our overall operational efficiency. If there is new technology, such as AutoMQ, that can potentially enhance our operational maintenance and utilization of Kafka, why not give it a try?

jackbauer24 2 years ago |

The table summarizes the differences between various streaming platforms. It highlights how different technical architecture choices can lead to different end results. What are your thoughts on this table?

superhx 2 years ago |

Why Apache Kafka cannot run on the Spot instances?

If we utilize K8S on Spot instances, does that imply that all products can accommodate Spot instances?

jackbauer24 2 years ago | |

The following explanation from Strimzi about the difficulty of implementing autoscaling in Kubernetes can also be applied to answer this question. Using spot instances requires the computing layer to be stateless.

ref:https://github.com/orgs/strimzi/discussions/6635

For Kafka brokers, auto-scaling is complicated because of their architecture. Adding or removing brokers is simple. But directing some load to them is complicated because they are in a way form of data storage. And moving the whole partitions between brokers is expensive. The partitions often contain huge amounts of data which need to be shifted from one broker to another - that will take time, it will have a performance penalty on the other traffic and possibly cost even real money for the data transfers.