Amazon Kinesis(aws.typepad.com) |
Amazon Kinesis(aws.typepad.com) |
Like all AWS offerings, Kinesis is a platform. It looks like kafka + storm, with fully integrated ecosystem with other AWS services. From the very beginning, the reliability, real-time processing, and transparent elasticity are built in. That's all I can say.
We are choosing Kafka over other solutions like RabbitMQ because we like the persistent txn-log-style messages and how cheap consumers are.
If we add nodes to an existing Kafka cluster, those nodes own no partitions and therefore send/receive no traffic. A rebalancing event must occur for these servers to become active. Bouncing Kafka on one of the active nodes is one way to trigger such an event.
Fortunately, cluster resizing is infrequent. Unfortunately, network interruptions are not (at least on EC2).
When ZooKeeper detects a node failure (however brief), the node is removed from the active pool and the partitions are rebalanced. This is desirable. But when the node comes back online, no rebalancing takes place. The server remains inactive (as if it were a new node) until we trigger a rebalancing event.
As a result, we have to bounce Kafka on an active server every few weeks in response to network blips. 0.8 alleges to handle this better, but we'll see.
Handle-jiggling aside, I'm a fan of Kafka and the types of systems you can build around it. Happy to put you in touch with our Kafka guy, just email me (mike.babineau@rumblegames.com). Loggly's also running Kafka on AWS - would be interesting to hear their take on this.
From the press conference reported in the link: "Jeff is very excited about the AWS business and he believes - like the rest of the leadership team does – that in the fullness of time- it is very possible that AWS could be the biggest business at Amazon."
Some interesting things about these kinds of measaging platforms.
Many exhanges/algo/low-latency/hft firms have large clusters of these kinds of systems for trading. The open source stuff out there is kind of different from the typical systems that revolve around a central engine/sequencer (matching engine).
There's a large body of knowledge in the financial industry on building low-latency versions of these message processors. Here's some interesting possibilities. On an e5-2670 with 7122 solarflare cards running openonload, its possible to pump a decent 2M 100byte messages/sec with a packetization of around 200k pps.
Avergae latency through a carefully crafted system using efficient data structures and in-memory only stores can pump and process a message through in about 15 microseconds with the 99.9 percent median at around 20 micros. This is a message hitting a host, getting sent to an engine, then back to the host and back.
Using regular interrupt based processing and e1000s probably yields around 500k msgs/sec with average latency through the system at around 100 micros and 99.9% medians in the 30-40 millisecond range.
Its useful to see solarflares tuning guidelines on building uber-efficient memcache boxes that can handle something like 7-8M memcache requests/sec.
http://www.amazon.com/AmazonBasics-KU-0833-Wired-Keyboard-Bl...
If the performance and pricing are right it's going to relieve a lot of headaches in terms of infrastructure management.
how? i mean, apart from poisson stats / shot noise, obviously (and which is noise, so you can't predict it anyway).
thinking some more, i guess this (splitting and merging partitions in a non-generic way) is to handle when a consumer is slow for some reason. perhaps that partition is backing up because the consumer crashed.
but then why not say that, instead of postulating the people are going to have uneven hashes?
[edit:] maybe they allow duplicates?
Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware.
I think it's unfair to call it vaporware - Amazon doesn't tend to release vaporware. You can also be fairly confident this has been in private beta for some time, so we'll probably see a few blog posts about it from some of their privileged (big spending) clients - typically someone like Netflix or AirBnB. But I agree it would be nice to get some more information on the details.
As for the client library handling load-balancing, fault tolerance, etc - that might not be ideal, but as long as I don't have to do it myself then it might be okay.
Its less ideal from a maintenance perspective, because there will need to be feature-rich clients in Java and C (with dynamic language bindings). Applications will be running many many versions of the clients. Also, for coordination, the clients will need to communicate, so there may be configuration and/or firewall issues for the app to resolve.
It will be interesting to see Amazon make this tradeoff for what I believe is the first time.
http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record...
1MB/s writes with 1000 writes/s 2MB/s reads with 5 read/s
We at Loggly are pretty excited about Kinesis -- I was at the announcement yesterday. We're already big users of Kafka, and we see Kinesis as another potential source and sink for our customers' log data. Our intent is to support pulling streams of log data in real time from our customers' Kinesis infrastructure, if they are already stream data to there -- and then analyze and index it.
And ss a sink, where we can stream customers' log data to their Kinesis infrastructure, after we've analyzed and indexed it for them, just like we do today with AWS S3. It could work really, really well.
I'm having way more trouble with the consumer being dumb with the way it distributes topics and partitions. End up with lots of idle consumers, while others are way above max.
Your consumer problems sounds similar to one we had. Root cause was that the number of consumers exceeded the number of active partitions. The tricky part was that the topic was only distributed across part of the cluster (because of the issue described in my parent post), so we had fewer partitions than we thought.