January 28th Incident Report(github.com) |
January 28th Incident Report(github.com) |
There were other learning points such as immediately going into anti DDoS mode and human communication issues that didn't realise or escalate the problem until some time after the issues started occurring.
Firmware issue meant that a large fraction of their servers could not detect the disks on reboot.
This prevented the redis cluster from starting.
They inadvertently have a hard-dependency on redis being up for the majority of their infrastructure to start.
Takeaway: If you run any complex system, ensure that each component is tested for its response to various degrees of failure in peer services, including but not limited to totally unavailable, intermittent connectivity, reduced bandwidth, lossy links, power-cycling peers.
No CI/test process was in place for hardware/firmware combos to ensure they recovered fine from power loss.
Takeaway: If you run a decent-sized cluster, ensure all new hardware ingested is tested through various power state transitions multiple times, and again after firmware updates. With software defined networking now the norm, we have little excuse not to put a machine through its paces on an automated basis before accepting it to run critical infrastructure.
No CI/test process was in place for status advisory processes to ensure they were sufficiently rapid, representative, and automated.
Takeaway: Test your status update processes as you would test any other component service. If humans are involved, drill them regularly.
Infrastructure was too dependent on a single data center.
Takeaway: Analyze worst case failure modes, which are usually entire-site and power, networking or security related. Where possible, never depend on a single site. (At a more abstract level of business, this extends to legal jurisdictions). Don't believe the promises of third party service providers (SLAs).
PS. I am available for consulting, and not expensive.
Edit this is mostly the "DR" part of tldr :P
You're welcome.
I would STFW, but searching for "HA" isn't helpful.
While this was happening at Github, I noticed several other companies facing that same issue at the same time. Atlassian was down for the most part. It could have been an issue with the service github uses, but they won't admit that. Notice they never said what the firmware issue was instead blaming it on 'hardware'.
I think they should be transparent with people about such vulnerability, but I suspect they would never say so because then they would lose revenue.
Here on my blog I talked about this issue: http://julesjaypaulynice.com/simple-server-malicious-attacks...
I think it was some ddos campaign going on over the web.
Amazon could learn a thing or two from Github in terms of understanding customer expectations.
I do the same thing, often searching Twitter for "aws" or "outage" and find people complaining about the problem which confirms my suspicions. It's a sad state of affairs when you have to do this and Amazon doesn't seem interested in fixing it.
Both Linode and Amazon suck at their status pages (though linode was quite informative about their DDoS outages that started on Christmas). Every amazon issue we've had, the status page only changed once they'd more or less fixed it. As far as I'm concerned their status page is basically useless unless it's an extended outage, at which point it's still basically useless...
Do you mean that "the cloud provider that is bigger than the next 14 combined and whose jargon has spread through the community" doesn't understand what customers are interested in and delivering on that?
The bigger a project gets, the less prioritized something like a status page often seems to get. Larger entities certainly _have_ them but I often see more things interfering as scale grows (this isn't only a MS thing, let me make clear) whether it be domain switches between engineering and social management (status is often via twitter), feeding the status page via a long telemetry/monitoring platform that has some lag, or a high threshold for what "outage" means to avoid flappy notices (at the cost of some false negatives).
I'm not even going to make a value judgement on the tradeoff of these costs at this point, (I certainly wouldn't dismiss it offhand as a net negative although equally it's not all roses) but at the very least I'd observe that something like a status page _CAN_ be serviced very well from an up and comer (for as much as Github is that any more) and it's far from a true statement that bigCOs can't take learnings from improving customer happiness from newer entities. (In fact, I wish that was a more common practice!)
But wow it is refreshing to hear a company take full responsibility and own up to a mistake/failure and apologize for it.
Like people, all companies will make mistakes and have momentary problems. It's normal. So own up to it and learn how to avoid the mistake in the future.
as an aside I feel that I'm quite fortunate to work in the EST timezone, as their outage apparently started at about 7pm my time. We have a general rule at my company to not deploy after 6pm unless an emergency fix absolutely needs to go up.
I saw the title of the story and said to myself, what outage? :P
Complex systems fail. Period. All the time. Things like the Simian Army are fantastic tools that help you identify a host of problems and remediate them in advance, but they cannot test every combinatorial possibility in a complex distributed system.
At the end of the day, the best defense is to have skilled people who are practiced at responding to problems. GitHub has those in spades, which is why they could respond to a widespread failure of their physical layer in just over 2 hours.
The biggest win with the Simian Army isn't that it improves your redundancy. It's that it gives your people opportunities to _practice_ responses.
I'm really tempted to continue with "a simbian army is actually" but this isn't Reddit so end of comment thread.
Including the principle that if your software breaks, you're the on who has to go get savaged by velociraptors to fix it.
If only things were that easy.
For all the fawning over being provided technical details, this article was pretty light on them.
I don't think Github going down for a couple hours is that big of a deal TBH. But it does seem to expose a few really basic failings in their DR planning IMO.
I also think it's ridiculous that some commenters are trying to frame this as a distributed computing problem. It's not even a clustering problem (apparently). It's just looking at the iDRAC or whatever to see why the server isn't getting past POST and putting your recovery plan into action.
This is white box vanilla stuff that happens to everybody.
That servers had to be rebuilt as part of DR says a lot.
The fact that there was a Redis dependency during bootstrap? Probably a good thing. You know as well as anyone I'm sure the last thing you want is a bunch of processes that only look like they're up. And even if they could not error without their Redis connections, if Redis is used for caching, what's that going to do to availability? Would it be a good thing to have the processes up if they can only handle 10% of the usual load?
Those are details that aren't there.
But complex distributed computing problem this is not. Not as it was presented anyways.
Usually its just cheaper to be down for an hour or two, versus architect for the end of times.
That's an awesome idea. I wish all companies published the firmware releases in simple rss feeds, so everyone could easily integrate them with their trackers.
(If someone's bored, that may be a nice service actually ;) )
I'm getting flashbacks. All of the servers in the DC reboot and NONE of them come online. No network or anything. Even remotely rebooting them again we had nothing. Finally getting a screen (which is a pain in itself) we saw they were all stuck on a grub screen. Grub detected an error and decided not to boot automatically. Needless to say we patched grubbed and removed this "feature" promptly!
Tell us which vendor shipped that firmware, so everyone else can stop buying from them.
(unless I missed some specific non capacity related outages?)
I was an active BB user a couple years ago, and the project I worked on would hg clone from BB many times a day so I would be the first one to notice a 503 or whatever error coming from their service. Typically I would see one or two outage per month, some last a few minutes, some last several hours. Most of the time the outage impacted git/hg checkout, so I think that was their technical bottleneck.
RFO: A squirrel climbed into a transformer
and a short time later they both blew up.Once it involved fire alarms, which trigger safety shutdowns within a suite. The other involved a failed static switch panel - ie, the things that aren't mean to be able to fail.
I don't mean to be blasphemous, but from a high level, is the performance issues with Ruby (and Rails) that necessitate close binding with Redis (i.e., lots of caching) part of the issue?
It sounds like the fundamental issue is not Ruby, nor Redis, but the close coupling between them. That's sort of interesting.
It has nothing to do with Ruby, or Rails or even Redis. It's just a design flaw of the application, that you often learn the hard way.
I believe the fundamental issue was just that redis availability was taken for granted by app servers so that certain code paths/requests would fail if it wasn't available, rather than merely be slower.
That would have given them immediate co text and not wasting time on DDOS protection
Flea power?
Since the firmware had a bug, bad state could be stored, completely removing power may clear that state and appears to have done so in this case. They may have also needed to pull the backup battery, and reset the firmware settings, but I wouldn't presume that just from the term "flea power."
I have never known what to call this, but have definitely been engaged in draining a few fleas.
Also, I can't believe it's been that long since google answers has been closed..
This doesn't sound very good.
I was recently involved in an outage that occurred because the sama datacenter was hit by lightning three times in a row. Everything was redundant up the wazoo and handled the first two hits just fine, but by the time the power went out for the third time within N minutes, there wasn't enough juice left in some of the batteries!
Now would it be possible to build an automated system that can withstand this? Probably. But would your time & money be better spend worrying about other failure modes? Almost certainly.
It is true that all their sentence is about recovery, however, it is disappointing that they didn't mention anything about a redundant datacenter.
A failed/failing drive present during cold boot could cause the controller to believe there were no drives present. To add insult to injury, on early BIOS versions this made the UEFI interface inaccessible. The only way to recover from this state was to re-seat the RAID controller.
There were also two bizarre cases where the operating system SSD RAID1 would be wiped and replaced with a NTFS partition after upgrading the controller firmware (and more) on an affected system (hanging/flapping drives). Attempts to enter UEFI caused a fatal crash, but reinstall (over PXE) worked fine. BIOS upgrade from within fresh install restored it.
From the changelog:
Fixes:
- Decreased latency impact for passthrough commands on SATA disks
- Improved error handling for iDRAC / CEM storage functions
- Usability improvements for CTRL-R and HII utilities
- Resolved several cases where foreign drives could not be imported
- Resolved several issues where the presence of failed drives could lead to controller hangs
- Resolved issues with managing controllers in HBA mode from iDRAC / CEM
- Resolved issues with displayed Virtual Disk and Non-RAID Drive counts in BIOS boot mode
- Corrected issue with tape media on H330 where tape was not being treated as sequential device
- resolved an issue where Inserted hard drives might not get detected properly.I seem to recall a recent post on here about how you shouldn't have such hard dependencies. It's good advice.
Incidentally, this type of dependency is unlikely to happen if you have a shared-nothing model (like PHP has, for instance), because in such a system each request is isolated and tries to connect on its own.
The thing that fixed the last problem doesn't always fix the current problem.
This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.
Redis sentinel[0] is the HA solution for redis for quite some time.
Of course Sentinel does not make Redis conceptually different from what it is from the point of view of consistency guarantees during failures. It performs best-effort attempt to select the best slave to retain writes, but under certain failure modes its possible to lose writes during a failover.
This is common with many failover solutions of *SQL systems as well btw. It depends on your use case if this is an affordable risk or not. For most Redis use cases, usually the risk of losing some writes after certain failovers is not a big issue. For other use cases it is, and a store that retains the writes during all the failure scenarios should be used.
You can replicate to read-only instances in a secondary DC and failover. It hurts but it is better than an outage imo.
Redis has been demonstrated[0][1] to lose data under network partitions. This is particularly concerning when discussing the type of partial failure that GitHub reported.
sometimes reading comments on hn makes me laugh out loud.
there's only one reason to not do this, and that's cost. what do you expect them to say about that? i mean really, you think they're going to put that in a blog post:
"Well, the reason we don't have an entire replica of our entire installation is because it costs way too much. In fact, more than double! And so far our uptime is actually 99.99% so there's no way it's worth it! You can forget about that spend! Sorry bros."
So, a very small risk of an hour or so of downtime sometime in the future which will not cause data loss, or tens of thousands of dollars a month for a failover cluster? I wouldn't replicate it either.
If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.
We should collectively be using incidents like this as an opportunity to learn, much like the GitHub team does. Our entire industry is held back by the lack of knowledge sharing when it comes to problem response and the fact that so many companies are terrified of being transparent in the face of failure.
This is very well written retrospective that gives us a glimpse into the internal review that they conducted. Imagine how much we could collectively learn if everyone was fearless about sharing.
That's the first 5 minutes after getting to a computer.
After that it doesn't really matter why they're down. You failover, get the site back up and worry about it later.
Are these systems on a SAN? That's probably the first mistake if so. Redis isn't HA. You're not going to bounce it's block devices over to another server in the event of a failure. That's just a complex, very expensive strategy that introduces a lot of novel ways to shoot yourself in the face. If you're hosting at your own data-center, you use DAS with Redis. Cheaper, simpler. I've never seen an issue where a cabinet power loss caused a JBOD failure (I'm sure it happens, but it's a far from common scenario IME), but then again, locality matters. Don't get overly clever and spread logical systems across cabinets just because you can.
Being involved with this sort of thing more frequently than I'd like to admit, I don't know the exact situation here, but 2h6m isn't necessarily anything to brag about without a lot more context.
What's pretty shameful is that a company with GitHub's resources isn't drilling failover procedures, is ignoring physical segmentation as an availability target (or maybe just got really really unlucky; stuff happens), and doesn't have a backup data-center with BGP or DNS failover. This is all stuff that (in theory if not always in practice), many of their clients wearing a "PCI Compliant" badge are already doing on their own systems.
You bet they busted their ass to get this fixed and shared their learnings with us. I'm extremely grateful for this and yeah it inconvenienced my morning but nothing more.
You make it sound so easy. If it takes the Github folks 2 hours, I can bet it would've taken us much longer.
So thank you GitHub, please keep up the good work!
So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If you want open source use Gitlab or any other service that sprinkles extra features around git.
I'll never understand this outrage of dependence on Github when you have a distributed version control system. It's not like it should be on github to setup third party repositories for you.
Ofc, Github isn't to blame for this, rather the ones that thought Github would be great to use as a CDN.
When I worry about dependency on GitHub, I'm thinking about not the inconvenient hours of downtime but the larger threat that they might disappear or turn evil.
What I would like to see even more than opensource github would be a standard for spreading over more services. For instance, syncing code, issues, pull requests, wiki, pages, etc between self-hosted gitlab and gitlab.com, or between gitlab.com and github.com. Further, I'd like to see it be easier to use common logins across services.
I don't think we can rely on Github giving us this, but if GitLab would add it between gitlab.com and gitlab ce, that would be a compelling reason to think of switching.
I don't think outages at GitHub are very frequent. This one was lengthy, so it's definitely been on a lot of peoples' minds, but this conversation always comes up when it happens.
I don't think outages at GitHub are very frequent
What would you rather have? A dependency on a bunch of projects with variable hosting of whatever means or all your dependencies hosted with the uptime of GitHub? Having an install fail because some host is down somewhere deep in your nest of dependencies is going happen a lot more if you have more hosts to worry about.
"...but we can take steps to ensure recovery occurs in a fast and reliable manner. We can also take steps to mitigate the negative impact of these events on our users."
The lessons that giants like Netflix have learned about running massive distributed applications show that you cannot avoid failure, and instead must plan for it.
Now, having a single datacenter is not a good plan if you want to give any sort of uptime guarantee, but that's a different point to make.
However, their recovery report didn't mention anything about such a plan.
<< Edited: correct a grammar error.
Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures,
I haven't kept up to date on "Sentinel 2" that was launched with 3.0 so the situation might have changed.
What you really want is a completely separate cluster running in a different data center (site). It should be isolated on its own network and ideally it should have different admin rights/credentials and a different software maintenance (patching) schedule. A completely empty site isn't much use so you'll need some kind of replication scheme. Naturally, these isolating steps make site replication difficult. You might patch one site and now the replication stream is incompatible with the other site. (You can't patch both sites at the same time because the patch might take down the cluster.) Or whatever you're using to replicate the sites, which has credentials to both sites, breaks and blows everything up. You need a way to demote and promote sites and a constraint on only one site being the "master" at a time. What happens if network connectivity is lost between sites? What happens if one site is down for an extended period of time? Maybe you need a third, tie-breaking site?
Once you work through these issues, you are still exposed to user error. Your replication scheme might be perfect... perfect enough that that an inadvertently dropped table (or whatever) is instantly replicated to the other site and is now unrecoverable without going to tape. Maybe you introduce a delay in the replication to catch these oopsies, but now your RPO is affected. Anyway, it's a bit of a shell game of compromises and margins of error.
Source: 10 years designing and building HA/DR solutions for Discover Card.
I learned they had an unfortunate power outage. Then it took about two hours to determine that the Redis servers weren't booting, and to failover.
That's honestly pretty unimpressive no matter how you slice it.
Setting up a Redis server is easy. Sharding it is easy. Setting up slave systems is easy.
What's hard is, like most things in life and tech, planning. Planning for failure. Practicing failure. Not by insisting you need a monkey-army, which sounds cool and fun, but by having a staging environment which is mundane and boring. Pull the power plug on a server. See how long it takes to identify the culprit and recover. Figure out what you could have done to identify the issue quicker. Figure out how you could recover quicker.
You're running Redis, and you haven't even setup slave systems in different cabinets? You've never tried pulling the plug on the JBOD to see what might happen? These servers apparently weren't even pinging. Why did it take more than 10 seconds to find out they weren't running? Why wasn't there a dashboard for basic system/process status across all services? Is GitHub's operations budget really that tight?
Setting up a pair of OpenBSD boxes with pf and HA-Proxy is easy. Making sure CARP picks up on the failover system when the primary fails is pretty easy. Scheduling it, testing it so you know it's actually going to work when you need it to is the hard part. Holding people accountable for it's uptime when the log says that hasn't been done this month is the hard part.
Setting up some FreeBSD boxes with DAS is easy. Giving each shard it's own slaves is easy. Monitoring the slaves to make sure they're actually useful is hard. Making sure you have a one-way failover script fire when the CARP interface fails over is easy. Snapshotting the ZFS filesystems is easy. Simulating breaks by pulling a power plug is hard. Write some garbage to the FS. Unplug the DAS. Unplug the host. Force "up but non-responsive" situations, figure out how to identify them, and how to recover from them.
Doing anything reliable with iSCSI is hard. Even Amazon has a poor reputation for it. Do your best to avoid it, and never ever ever buy into vendor promises that it'll allow you to just remount your block devices on a different host and never have to worry about data or downtime. IME. YMMV.
Using runit to keep your Ruby processes up: Easy. Proper logging, monitoring and alerts because processes are just bouncing constantly? Well, it's not rocket science. But it definitely takes discipline.
There are people a hell of a lot smarter than me that have been doing this for a lot longer. But some of this stuff is just flat out not acceptable. I've been on those angry-client phone calls. "Inadvertent" should be a trigger word to anyone in Ops. It really means somebody didn't do something they should have and it bit them in the ass (at least that's what it meant when I said it). Would the app have started fine if boot.rb didn't attempt to connect to Redis? That seems pretty far fetched. So that sounds like a red-herring. But what do I know.
Setting these things up is not the hard part. Identifying that a host isn't responding to ping isn't the hard part. Planning, procedures, discipline and execution, day in and day out when things aren't on fire to prepare for the day that they are, that's the hard part.
Maybe Github is has a much smaller Ops team and budget than I would've imagined. Maybe this should be a wake up call to the CEO. I don't know. These are just my observations from what was said and having deja-vu thinking about lessons I had to learn the hard way.
BTW, I'm not necessarily trying to bag on the guy getting paged at 3AM. Been there, done that. It sucks. You do the best you can. If anything that I'm saying has a ring of truth to it, then it's a leadership issue. And it's not about "give smart people things and get out of their way blah blah bullshit". This sort of stuff doesn't materialize out of thin air and good intentions.
OTOH it's crazy that Github is single-homed and can't afford an F5. :shrug:
BTW:
> I can bet it would've taken us much longer
One of the worst experiences of my professional life was 72 sleepless hours, mostly in a 50F data-center, trying to figure out why the SAN would sporadically drop off servers every other minute. Turns out somebody, not me for a change, set the MTU on the switches to 9000. So whenever a max-frame was used BOOM.
But yeah, I've been through Redis failures. It didn't take me two hours to get things going again. Load-balancer failures. Database failures (backup sure, but never ever plan to actually use it; it's plan Z at best). NFS failures as well. Though those might be pushing it with heads that take almost 10 minutes just to reboot.
From the outside 2 hours seems like a very long time to identify and recover from a Redis failure. (Knock on wood.) And it sounds like they didn't have Warm systems standing by for failover? That's bad...
That said we probably don't need to solve it - we just need a way to keep read remotes online that's separate from any one server. The DHT git project was a good move in the right direction.
A generous reading of "We can also take steps to mitigate the negative impact of these events on our users." would include improvements of that sort.
That said, I also didn't spot any concrete proposals for geo-redundancy in the post-mortem. Perhaps that's a detail that will be figured out in a following exercise, or perhaps they really don't have any plans for GR, in which case the generous reading would be unwarranted.
There may be specific aspects of Github's usecase that make this difficult, but please don't pretend that geo-redundancy is impossible. Look at Netflix's architecture for an example of a site that services traffic from multiple AZs.
The space of acronyms/abbreviations is quite cluttered.
That's an interesting alternate reading though...
"To ensure the integrity of our data, we need to locate another Arizona, since this one is serving us so well."
The space of acronyms/abbreviations is quite cluttered.
The opposite of this philosophy was the motivation behind creation of the internet in the first place.
http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...
https://www.jwz.org/doc/worse-is-better.html
[thanks for the hint 'thinkpad20! I don't know what I was thinking.]
The internet is designed to be highly fault tolerant, because it was based on an arpanet project to design a network that would NOT go down, even if there was damage to a significant percentage of nodes.
But then the conversation stops within days because, the fact is, hosting your own git servers and getting people to actually use them is a huge pain in the ass. More simply put: people just like using GitHub.
Furthermore, GitHub's a business. They're selling private repositories. They do open source quite a bit of code, but they're not going to open source their actual product.
Given that this is the case, I fell like GitHub is entrenched enough that they could open source their codebase and not lose any customers. People are paying them for the convenience of someone else hosting their git repository.
But yes, most of us are most comfortable with the central repository model.
I just wanted to point that now that you guys are well stablished and have huge impact in the open source community, adopting a more open approach with your end users can be very beneficial for both, GitHub and the user base. I'm sure that lot of people would contribute to your codebase and thing like this: https://github.com/dear-github/dear-github would be less frequent and notorious.
But if the enterprise edition is really the source of income, open sourcing it doesn't have any sense. I agree on that. Maybe, another way to be more open to contributions from the community ? I dunno
0: https://en.wikipedia.org/wiki/Reliability,_availability_and_...
Honestly if anyone is working on something important enough where they must be able to rebuild at a moment's notice then they should either be checking in dependencies from these package managers or setting up their own copies of what they need. But that's like backing up; most don't realize they needed to be doing that until they have an issue so I can understand the anger when Github does go down.
Rust program building, for example, seems to require that Github be up.
https://web.stanford.edu/class/cs240/old/sp2014/readings/wor...
It's funny, my original comment had the links in plaintext so copying-and-pasting was required and Referer wasn't involved. I changed that on request. b^)
People who use Redis rarely end up using it solely as a caching layer. It often also takes on the role of an RPC facilitator and pseudo-database. GitHub's post also mentions that their engineering team had to replicate Redis' dataset before they could get the alternative hardware running, which implies that they do need some data in there before the site is operational.
Personally one of my pet peeves is people throwing mission-critical data in Redis and acting like it's honky-dory. It happens all the time and seems really difficult to get people to not do. There's a reason we have a real ACID compliant database storing non-disposable data; it's ridiculous to ignore that just because it's easier to stuff it in Redis.
I think it's reasonable to have a dependency on a Redis server, but I don't think it's reasonable to depend on any data in particular being stored in that server. It should be used as a caching/acceleration layer for data that can be easily and automatically regenerated.
I have no fundamental opposition to K-V stores or NoSQL databases, but I do think most developers favor them because it's easier to stuff them with data up front. There are big tradeoffs down the road, though, which companies don't seem to understand well, and which they aren't really equipped to handle.
A lot of people have started depending on github for more than just stashing source code some place centrally accessible as they're working on it. If github takes a lax attitude toward uptime then I suspect people will start looking for alternatives.
It wasn't indicated on the status page until after it was fixed. And it was indicated as a green check in a sea of green checks. With a small "i" in the corner to represent the outage.
I love AWS. It's not without fault but overall I think it's been well architected, well documented, and well implemented.
But the status page has got to be the ultimate example of what not to do.
I think everyone complains in forums and online but doesn't actually file tickets about it. These things are worth tickets too.
1. File ticket.
2. Wait. Then wait some more. Even if you pay big money for a support contract, they take a long time to respond (often > 1 hour).
3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalling tactic.
4. Play the dumb question/obvious response dance, waiting an hour or more for a response each time.
5. If you are lucky (usually a couple hours in now) they acknowledge there's some problem (but never give you any detail) and escalate your ticket to a higher level internal team. If you are unlucky, you are calling up your account rep (do you even have one??) and getting them to harass tech support.
6. Usually around now the problem "magically" disappears if you haven't already fixed it yourself.
7. If you are lucky, a few hours, days, weeks later you get a response asking if you are still having the problem? You, of course, are NOT having the problem since you long ago solved it yourself. If you are really unlucky they try to schedule a meeting with one of their "solution architects" who is then going to waste an hour of your time telling you how to properly "design" your software for the cloud (i.e. trying to sell you on even more of their services).
8. Ticket is closed having never gotten to the bottom of the problem, maybe get a survey.
I've never seen this go down differently. Filing more tickets isn't going to change this. You want to really change things?
STOP PAYING THEM!
If a few mid-sized customers stop paying them and make a big-stink when they do it, then I guarantee you things will change! Until then, they have little incentive to improve and the big customers have a direct line to Amazon so they can circumvent all this crap. It's up to the small and mid-sized customers to push for change and the most effective way to do this would be to spend your money elsewhere.
One example: redshift. Had an expensive temporary cluster that couldn't be deleted, for days. Was stuck "pending" or "rebuilding". Assigned account rep would take forever to respond, and just didn't understand, would forward directions to using AWS console. Yeah, DOESN'T WORK. After a week decided to try getting attention on twitter, got it fixed in about 12 hours.
My experiences don't reflect this, perhaps we are familiar with different levels of support contracts. I use AWS for work only so I can only speak to one level if their support.
>3. Get a response from a first level rep who has no access to anything, has little dev experience, and asks some inane questions which I'm convinced is a purposeful stalling tactic. 4. Play the dumb question/obvious response dance, waiting an hour or more for a response each time.
I can't agree with this either. I almost always use their chat option and a rep is usually available within 15m unless there is an AWS outage.
I do however completely agree with 5 and 6, but I don't let it bother me. They can't expose too much info about their infrastructure. I'm usually just looking for a confirmation of an issue in their side or not which they have always been willing to provide.
If you're using aws for business and are unhappy with their current level if support maybe you should talk with their sales folks to find out about higher tier support plans.
Basically, if Netflix isn't the source of the complaint, they're not going to give two fucks.
/me suspects that netflix engineers get outage notifications through some other avenue than the status page.
This was the post I was googling for
http://techblog.netflix.com/2015/10/flux-new-approach-to-sys... (prepare to have your mind blown)
And what I found in the Google results
From 2015-04 http://techblog.netflix.com/2015/04/introducing-vector-netfl...
And 2014-01 http://techblog.netflix.com/2014/01/improving-netflixs-opera...
That is some crazy fast innovation there
Hasn't worked yet.
But it's not specific to us down under - the support contacts come from all over the globe. We dropped from Business to Developer support when the $A tanked in order to save a buck, and it just takes a little longer is all - no real drop in quality. I wish other large companies had their level of support quality.
Which is unacceptable for one of the largest infrastructure providers. So many times we were sitting around twiddling our thumbs waiting for our expensive amazon support to get back to us when things were broken.