Super cool to see this here. If you're at all interested in big systems, you should read this.

> Compounding this latency, hard drive performance is also variable depending on the other transactions in the queue. Smaller requests that are scattered randomly on the media take longer to find and access than several large requests that are all next to each other. This random performance led to wildly inconsistent behavior.

The effect of this can be huge! Given a reasonably sequential workload, modern magnetic drives can do >100MB/s of reads or writes. Given an entirely random 4kB workload, they can be limited to as little as 400kB/s of reads or writes. Queuing and scheduling can help avoid the truly bad end of this, but real-world performance still varies by over 100x depending on workload. That's really hard for a multi-tenant system to deal with (especially with reads, where you can't do the "just write it somewhere else" trick).

> To know what to fix, we had to know what was broken, and then prioritize those fixes based on effort and rewards.

This was the biggest thing I learned from Marc in my career (so far). He'd spend time working on visualizations of latency (like the histogram time series in this post) which were much richer than any of the telemetry we had, then tell a story using those visualizations, and completely change the team's perspective on the work that needed to be done. Each peak in the histogram came with it's own story, and own work to optimize. Really diving into performance data - and looking at that data in multiple ways - unlocks efficiencies and opportunities that are invisible without that work and investment.

> Armed with this knowledge, and a lot of human effort, over the course of a few months in 2013, EBS was able to put a single SSD into each and every one of those thousands of servers.

This retrofit project is one of my favorite AWS stories.

> The thing that made this possible is that we designed our system from the start with non-disruptive maintenance events in mind. We could retarget EBS volumes to new storage servers, and update software or rebuild the empty servers as needed.

This is a great reminder that building distributed systems isn't just for scale. Here, we see how building the system in a way that can seamlessly tolerate the failure of a server, and move data around without loss, makes large scale operations (everything from day-to-day software upgrades to a massive hardware retrofit project) possible that just wouldn't be possible in a "simpler" architecture. A "simpler" architecture would make these operations much harder, to the point of being impossible, making the end-to-end problem we're trying to solve for the customer harder.

Ah, this brings back memories. Reddit was one of the very first users of EBS back in 2008. I thought I was so clever when I figured out that I could get more IOPS if I build a software raid out of five EBS volumes.

At the time each volume had very inconsistent performance, so I would launch seven or eight, and then run some each write and read loads. I'd take the five best performers and then put them into a Linux software raid.

In the good case, I got the desired effect -- I did in fact get more IOPS then 5x a single node. But in the bad case, oh boy was it bad.

What I didn't realize was that if you're using a software raid, if one node is slow, the entire raid moves at the speed of the slowest volume. So this would manifest as a database going bad. It took a while to figure out it was the RAID that was the problem. And even then, removing the bad node was hard -- the software raid really didn't want to let go of the bad volume until it could finish writing out to it, which of course was super slow.

And then I would put in a new EBS volume and have to rebuild the array, which of course it was also bad at because it would be bottlenecked on the IOPS for the new volume.

We moved off of those software raids after a while. We almost never used EBS at Netflix, in part because I would tell everyone who would listen about my folly at reddit, and because they had already standardized on using only local disk before I ever got there.

And an amusing side note, when AWS had that massive EBS outage, I still worked at reddit and I was actually watching Netflix while I was waiting for the EBS to come back so I could fix all the databases. When I interviewed at Netflix one of the questions I asked them was "how were you still up during the EBS outage?", and they said, "Oh, we just don't use EBS".

It's cool to read this.

One interesting tidbit is that during the period this author writes about, AWS had a roughly 4-day outage (impacted at least EC2, EBS, and RDS, iirc), caused by EBS, that really shook folks' confidence in AWS.

It resulted in a reorg and much deeper investment in EBS as a standalone service.

It also happened around the time Apple was becoming a customer, and AWS in general was going through hockey-stick growth thanks to startup adoption (Netflix, Zynga, Dropbox, etc).

It's fun to read about these technical and operational bits, but technical innovation in production is messy, and happens against a backdrop of Real Business Needs.

I wish more of THOSE stories were told as well.

This is the bit I found curious: "adding a small amount of random latency to requests to storage servers counter-intuitively reduced the average latency and the outliers due to the smoothing effect it has on the network".

Can anyone explain why?

If you're curious, this is a talk I gave back in 2009 [0] about Amazon S3 internals. It was created from internal assets by the S3 team, and a lot in there influenced how EBS was developed.

[0]: https://vimeo.com/7330740

I liked the part about them manually retrofitting an SSD in every EBS unit in 2013. That looks a lot like a Samsung SATA SSD:

https://www.allthingsdistributed.com/images/mo-manual-ssd.pn...

I think we got SSDs installed in blades from Dell well before that, but I may be misremembering.

I/O performance was a big thing in like 2010/2011/2012. We went from spinning HDs to Flash memory.

I remember experimenting with these raw Flash-based devices, no error/wear level handling at all. Insanity, but we were all desperate for that insane I/O performance bump from spinning rust to silicon.

This gives me fond memories of building storage-as-a-service infrastructure back before we had useful opensource stuff, moving away from sun san, fibrechannel and solaris we landed on glusterfs on supermicro storage servers, running linux and nfs. We peaked almost 2Pb before I moved on in 2007.

Secondly it reminds me of the time when it simply made sense to ninja-break and rebuild mdraids with ssds in-place of the spinning drives WHILE the servers were running (sata kind of supported hotswapping the drives). Going from spinning to ssd gave us a 14x increase in IOPS in the most important system of the platform.

At the very start of my career, I got to work for a large-scale (technically/logistically, not in staff) internet company doing all the systems stuff. The number of lessons I learned in such a short time was crazy. Since leaving them, I learned that most people can go almost their whole careers without running into all those issues, and so don't learn those lessons.

That's one of the reasons why I think we should have a professional license. By requiring an apprenticeship under a master engineer, somebody can pick up incredibly valuable knowledge and skills (that you only learn by experience) in a very short time frame, and then be released out into the world to be much more effective throughout their career. And as someone who also interviews candidates, some proof of their experience and a reference from their mentor would be invaluable.

Loved this:

> While the much celebrated ideal of a “full stack engineer” is valuable, in deep and complex systems it’s often even more valuable to create cohorts of experts who can collaborate and get really creative across the entire stack and all their individual areas of depth.

The first diagram in that article is incorrect/quite outdated. Modern computers have most PCIe lanes going directly into the CPU (IO Hub or "Uncore" area of the processor), not via a separate PCH like in the old days. That's an important development for both I/O throughput and latency.

Otherwise, great article, illustrating that it's queues all the way down!

Early on, the cloud's entire point was to use "commodity hardware," but now we have hyper-specialized hardware for individual services. AWS has Graviton, Inferentia and Tranium chips. Google has TPUs and Titan security cards, Azure uses FPGA's and Sphere for security. This trend will continue.

Great article.

"EBS is capable of delivering more IOPS to a single instance today than it could deliver to an entire Availability Zone (AZ) in the early years on top of HDDs."

Dang!

What's the best way to provide a new EC2 instance with a fast ~256GB dataset directory? We're currently using EBS volumes but it's a pain to do updates to the data because we have to create a separate copy of the volume for each instance. EFS was too slow. Instance storage SSDs are ephemeral. Haven't tried FSx Lustre yet.

The most surprising thing ia that the author had no previous experience in the domain. It's almost impossible to get hired at AWS now without domain expertise, AFAIK.

So true and valid of almost all software development:

> In retrospect, if we knew at the time how much we didn’t know, we may not have even started the project!

Great read, although a shame that it didn't go any further than adding the write cache SSD solution, which must have been many years ago. I was hoping for a little more recent info on the EBS architecture.

I had no idea Werner Vogels had a systems blog. Awesome read, thanks.

I think the most fascinating thing is watching them relearn every lesson the storage industry already knew about a decade earlier. Feels like most of this could have been solved by either hiring storage industry experts or just acquiring one of the major vendors.

The cloud's point was to be a computing utility, not to use commodity hardware. They may have been using commodity hardware but it was just a means to an end. That was also the era that "commodity hardware" was a literal game changer for all forms of large-scale computing for businesses, as before then you'd be beholden to an overpriced, over-valued proprietary computing vendor with crap support and no right to fix or modify it yourself. But anyway, the businesses you're referencing are literally the largest tech companies in the world; custom hardware is pretty normal at that size.

You must be talking about very early on because it would only have taken a short time spent on practical cloud building to begin realizing that much or even most of what is in "commodity hardware" is there to serve uses cases that cloud providers don't have. Why do servers have redundant power supplies? What is the BMC good for? Who cares about these LEDs? Why would anyone use SAS? Is it very important that rack posts are 19 inches between centers, or was that a totally arbitrary decision made by AT&T 90 years ago? What's the point of 12V or 5V intermediate power rails? Is there a benefit from AC power to the host?

It was more of an opportunity initially. Bezos saw a chance to purchase extremely discounted systems. He had the foresight to move redundancy from a single host to clusters and then eventually data center level redundancy. This decision reshaped how services were implemented and scaled. Specialized devices returned because they offer a better value or in some cases enabled products that were never possible before. At the end of the day a lot of decisions are based around cost. Can you do it with a toaster or an oven cheaper? Can we cut all of our food into pieces and cook it in a Beowulf cluster of toasters? I think you get the idea.

> hyper-specialized hardware for individual services > AWS has Graviton

This is commodity hardware.

It pretty much works off ARM spec. Besides AWS owning it and offering it cheaper it's not in any way faster / better than Intel / AMD / something else.

We've had custom motherboards, server cases, etc long ago even before clouds.

If Apple silicon happens in the cloud then maybe...

The trend was to begin to use distributed software in computing clusters that allowed lower cost with commodity machines with lower individual reliability.

Now the focus is on cost-efficient machines in clusters, which is why we have the specialization.

There are substantial differences in developing hyperscale storage from the systems that were built previously. But note that many of the architects of these systems were previously storage industry experts, and acquiring an existing vendor would not have been an asset to AWS since these new systems had a wide range of issues that the vendors never had to solve.

Your comment elsewhere about NetApp solving all known problems with WAFL. Hahahaha. Have you tried deleting a 5TB file in a filesystem at 95% capacity with snapshots enabled?

What is there to learn from an "storage industry expert" or major vendors? network attached block level storage at AWS's scale hasn't been done before.

> Feels like most of this could have been solved by either hiring storage industry experts or just acquiring one of the major vendors.

VMware tried something like that with vCloud Air [0], which attempted to build a public cloud on the ESXi hypervisor with hardware and expertise sourced from storage vendors. It failed.

There were a number of reasons for the failure but perhaps the most prominent was that VMware vastly underestimated the amount of engineering necessary to build a multi-tenant public cloud. As the article pointed out you need to optimize along the entire I/O path from client request down to sectors on disk. It's a significant mindshift from running VMs over a SAN, which is where VMware came from.

(I was involved in the project in its later stages. These are personal observations.)

[0] https://en.wikipedia.org/wiki/VCloud_Air

For the right kind of people, it's tremendously satisfying for themselves to rethink and independently rediscover the best solutions. If you hire an industry expert that already knows everything, asking them to design the same things and write the same code again is not satisfying at all.

The answer to why your strategy most likely would have been a failure lies outside of technology domain