Zones or regions? I don't think multi-az will help you when the whole region kicks the bucket.

The procedure is outlined in each service's SLA -- though I think they're all pretty much the same.

Annoyingly, they expect you to do the leg work to show when the outage happened and supply logs demonstrating that you were impacted.

Might want to do some napkin math first to see if the amount credit is worth your time. The couple times my org considered pursuing it, it just wasn't worth the effort. (Though, personally, I think that speaks to a larger problem with the SLA.)

Credit Request Procedure in Kinesis SLA: https://aws.amazon.com/kinesis/sla/#Credit_Request_and_Payme...

Just us-east-1.

Existing cloudfront is indeed fine. But creating or deleting distributions fails now.

(I think it's also pretty rare for an already configured cloudfront to suffer from issues on the control planes. Cloudfront configuration updates are painfully slow even under normal circumstances, and that's probably because the configuration is heavily replicated to all POPs)

In early 2019 CloudWatch had a major outage that was particularly nasty, where instead of just outright failing to report metrics it reported a small percentage of metrics. As a result a lot of autoscaling groups and DynamoDB tables that were theoretically supposed to avoid scaling in during a metric outage still scaled in, because they saw it as a 90+% traffic reduction rather than a metric outage.

This is a relevant comment. I agree. Who will hold them accountable for violating uptime guarantees? Nobody? Then what's the point or purpose of an uptime guarantee? Marketing value?

> what a scam. who can hold them accountable for cheating those who paid for uptime guarantees?

Never trust that. Deploy in multiple regions (and AZs within those regions) if you really cannot tolerate any downtime.

yeah every amplify app should be down/super slow right now.

AWS' conditions for what healthy is and everyone elses is completely different. Kinda makes me wonder what their internals are like.

I fully admit, with no reservations at all, that you're getting tired of that bullshit.

completed a job in eu-west-1 10 minutes ago

We have experienced the same on multiple accounts

You're missing the point. The point is that it isn't apple to apples. If you are honest with a dashboard, and the competitor isn't (or doesn't have one), it's not fair to compare.

Just because it happens all the time doesn't mean it isn't unscrupulous.

OK, so what's your point? The outcome of this is still a worse situation for everyone involved in the end.

Same here, I've made some huge fuck ups in my time at Amazon, one of which I was pretty new and assumed I would be fired for; but one of the principal engineers on my team told me not to worry, these things happen and it's a blameless process where we're just trying to get to the root cause of the problem and ensure it never happens again; and it was exactly as he said. The CoE we presented said "An engineer from the xyz team..." and never mentioned a name once.

Technically 100% uptime, as one of the two competing chains will pull ahead of the other. That chain has no down-time.

For there to be downtime in Bitcoin, there would need to be a rollback, where all (or most) miners go back to a previous block and mine from that point. This has only happened once, as far as I am aware (due to a bug in the protocol itself which needed correction).

Not so much older hardware is moved there as it's just the oldest region with the most baggage

It’s not that the oldest hardware is moved there, it’s just that the oldest hardware was there to begin with. There are probably still first-generation EC2 instances running in us-east-1 on their original platforms.

From what I understand AWS and Amazon are two separate dev groups. Does your statement cover all of Amazon or only AWS?

Why don't the new folk iteratively refactor their systems to remove operational burden? Isn't that part of owning any codebase you didn't write?

That design raises the question as to what happens to passwords. Do they get replicated in the global table in plaintext? Or are you still forced to do a global user password reset if you want to failover to another user pool?

Quite superficial don’t you think?

The alternative for AWS might be to be able to failover to e.g. Kinesis in another region.

fair, but if you lose your customer's trust that might not be recoverable. Depends on your business

AWS is big. Amazon is even bigger. Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general.

You don't hear a lot of people praising AWS, the same way you don't hear a lot of people saying how great it is to have an iPhone. If I am happy, I have little incentive to post about it, since that should be the default state.

But the matter of fact is simple. If you end up in a team like this, switch and raise complaints afterwards. Nothing stops you from it. There is no "toxic engineering culture" at AWS. The problem is that AWS makes you into an owner and that includes owning your career. That means if you feel something is wrong, YOU are expected to act. No one will do it for you. And there are plenty of mechanism for you to act.

This is the greatest benefit of working at Amazon but its also the downfall of people who are not able to own things.

  Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general.

As a customer I don't really care whether AWS has a toxic internal culture. I care about whether they have operational excellence and a high quality product. This information is showing cracks in operational excellence.

My company will update their status but puts the most vague responses up. Reason is because we don’t want to appear inept when we crash the website. For example, because we ran out of disk space

Our competitors would have a field day with that

The point is to manage potential external liabilities. A business doesn't want any sort of liability they have automatically costing them if they can avoid it. They're more than happy to have anything that profits them automatically generate revenue, but if something could potentially lose them thousands or millions, they want to make sure there's a human-in-the-loop from management to check off. Not meeting SLAs or service outages are a good way to cost them money.

Few companies really respect their engineering teams/divisions in any sensible form from my experience, though I'm biased (even in heavy R&D environments). You're simply a means to an ends.

I understand your point though (and identify with it), but I find any mechanism/option that provides a way of containing potentially damaging information is going to be pushed by management over the option to release damaging information that a responsible engineer may want to disclose.

You're in a culture where admitting fault or liability is like pulling teeth and ripping finger nails off. It shouldn't be IMHO (we should own up to our mistakes and be reasonably forgiven), but that's unfortunately not the culture we have.

If they host it somewhere else, it signals they lack confidence in their own product.

If they self-host it, it signals that they're overconfident in their ability to maintain an accurate status page.

Given these two options, which do you think a budget manager will have an easier time signing off on and defending upward?

That's incredibly annoying, given the mandate the replacement service had.

I'd be curious to be a fly on the wall during the next Ops meeting when it comes up that yet again the status dashboard got made in a way that makes it hard to update during an outage.

All I see in your response is, "We don't want to tell the truth because it might cost us money."

Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.

This might be an oversimplification.

With any customer that has SLAs written into their contracts, they're not just going off your status page. They most likely have a direct point of contact and exact reporting will be done in the postmortem.

The status page is for customers for which there aren't significant legal or business complications and exists to provide transparency. In my opinion you do want "random" people at your company to be able to update it in order to provide very stressed out customers with the best information you have.

As an industry we probably should recognize this more explicitly and have more standard status pages that are like "everything might be broken but we're not sure yet"

So... then what's the point of a status dashboard?

Not at AWS, retail Amazon, but what I saw was COEs were either normal business process or PIP material depending on which org you worked for. And sometimes just the excuse to get you gone.

Where I was about 99.9% of the COEs where just a lesson learned and new process to prevent it. There was one that was basically used as a tool to remove a VERY good engineer, that didn't mesh well with new leadership.

A sister org, one I worked a lot with, wouldn't COE anything. If you were the lead engineer on a product or service that had a COE you were going to get a PIP by year end review. I wasn't surprised when all the talent left that group.

Hah. I don't work for AWS (anymore) but a COE I wrote was literally on my promo doc

Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful.

In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API.

Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.

Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law

That sounds toxic.

If you're small, yes, if you're AWS, it's business as usual?

>for operators trying to resolve this issue

It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page.

If you're right, why have the status page then? It is useless by your definition yes?

> Updating the status dashboard is pretty low priority for operators trying to resolve this issue

Which is why, during incident responses, there has to be people in charge of communication. Both internal and external communication, and some of this can be further delegated.

That's a poor excuse.

> It requires escalation up the management chain and careful wording

Careful wording is more important for external stakeholders who might not have the full context. If one is stepping in eggshells with internal management too, that's bad management. Incident communication should be factual and concise.

By design. If it was a good faith attempt to report status, it would be automatically updated from a flock of canaries instead of through a slow, political process.

>Part of the problem is, engineers love shiny things. Status pages are fundamentally boring things. Who wants to work on them?

I would work on a status page. It's a interesting problem, creating tests that prove services are viable at a place like AWS would be fun. However what I don't want to deal with is some director of so and so I never heard of yelling at me at 3 in the morning because my status page reported that his service was down accurately. I suspect that plays more into the problem. The status page is a political implement not a technical one.

Given enough law suits and mistakes by dev flipping dashboards to red with a bad code change or network provider outage and your org will have a manual public facing dashboard as well.

> for many companies its a formal warning that you need to shape up or you're going to be let go.

Patently incorrect. A PIP is management telling you that you need to seek alternative employment, now.

Joking/sarcasm aside: I’ve never seen or heard someone who is placed on a PIP successfully “exit” the PIP. They exit the company or they’re exited from the company. PIPs seem to mark the start of the “we are building formal documentation to fire you” phase of losing a job.

Not Amazon but at an HR/leader meeting our HR disclosed 45% of people on PIPs end up staying with the company.

> Updated manually by whom? > I'm afraid you're shifting the complexity to a manual process.

You're right, that's 100% what I'm doing. Why? Because it shouldn't be that complicated to update an overall health status page during an outage event, and it shouldn't take other tools and services within AWS to do it.

A common pattern in cloud providers (including AWS) is that services have some kind of tiering, whereby you can't pick up a dependency on any service on a lower tier than yourselves. Tier 2 services can't rely on Tier 3 services, etc. Services like, say, IAM, would be right at the very top. It can't rely on EBS, ELB etc. Everything has to be created in-service, because everything ultimately has to rely on authentication working.

If they're going to keep an overall status page going, it needs to be seen as a top tier service, just like identity is. That's where they were headed towards when I left AWS about 5 1/2 years ago. It had been spurred by a previous major incident couldn't be reflected in the status dashboard because of a failure in a dependency.

> I agree that it doesn't have to, and perhaps should not, be fully automated. But automating some parts will help not waste time on last minute arguments.

I go in to a bit more detail in another comment within this discussion, but a status page does not even close to accurately capture the ways that cloud environments fail, which are very, very rarely affecting more than a small percentage of customers, and even then often in some very specific way under specific circumstances. That's why AWS built the personalised status page service. They want to ensure that customers have an accurate way of telling what is going on with services they're consuming, rather than the confusing situation of checking an overall status site that doesn't really reflect their experience and never could.

Situations like today's where it at least (from the outside) seemed like Kinesis was completely down, would be a good example of something that should be reflect in the main overall status page.

The status page should be manual, and should be something the incident management team can do (and have political ability to force it to happen, rather than being subject to the whims of service directors)

I think the reason they don’t work is because someone doesn’t just magically become a better employee over two months.

Only the strong survive:p

I was merely addressing the "thousands of employees" non sequitur. Org structure means that the raw number of employees is a meaningless metric. The only people who are going to potentially flip that switch are going to have some sort of direct responsibility for the product. That number is going to be very similar whether it's a large company like Amazon or a smaller one like, say, Heroku or Dreamhost.

I just snaked my sewage pipe. Can confirm this to be true

I was a warehouse clerk which meant I was responsible for picking stowing, receiving, shipping and organizing the warehouse.

It was mine. An opening appeared in the service department and I applied for it.