a) letting customers in other areas know that their data is backed up to another continent
b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"
For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.
Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.
I don’t think I’ve seen them say?
Hey EU, take note of this next time you create silly data residency requirements that don’t allow data to travel outside your region. Encryption is an easy solution to multi-region residency…as long as the European Commission doesn’t stupidly keep trying to make encryption illegal too!
Hint: Russia absolutely knows where your data centers are.
Germany and France in a debt spiral and turning inward/hyper-nationalist while massively re-militarizing means the EU is a safe place to structure your data with zero redundancy.
I wouldn’t bother worrying about key industries and functions, since, as history has shown, the EU is a bulletproof institution that no country has ever left.
And of course every data center in Europe has Israel-grade air defenses, it’s not like they are just sitting ducks for a fleet of drones to take out within 24 hours.
Uh-oh.
It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.
The wording certainly seems very careful:
"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"
DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.
And a friendly reminder that replication isn't a tested data backup.
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
The sort of global instability that is being created by all this is bad for everyone. It generally isn't good for business either.
We're so unstable and capricious now, countries are actually trying to decouple from American tech services like AWS and Amazon. That ain't great for Bezos.
Maybe inertia kicks in after Trump and things revert to the mean, but restoring confidence in the US again as a friendly, stable nation is going to be tough if we're always 4 years away from another Trump-type figure running the show.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
me-south-1 is about 250 miles away from me-central-1, but that's not far enough in this instance. Given that, I think city level location information should be good enough.
250 miles is pretty good for weather or not specifically targeted destruction (wildfire / industrial explosions / arson), but it's clearly not enough if your data is in a building targeted in a regional war. Assuming datacenters remain targets in wartime, I think it's fair to assume if one datacenter in any particular country is attacked, all the rest of the datacenters in that country are likely to be attacked, too. In that case, offline storage (tapes and things) in inconspicuous locations might be the way.
(or an AWS Outpost thingy, assuming those things work even if the mothership is down)
https://www.edpb.europa.eu/system/files/documents/2021-06/ed...
If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?)
We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.
And has lawless goverment and unaccountable tech industry making it bad place for data.
For one thing, regardless of what the US did, they did not launch the attack that damaged the datacenters. That attack was provoked by US actions, but the attack was an intentional and voluntary act by Iran/IRGC. I don't think any of the usual settings for liability for the acts of others really apply here, but I'm not a lawyer.
You'd also need to find a court of competent jurisdiction. The US is out because of soveriegn immunity. The Military Claims Act prohibits claims that "arise from action by an enemy or result directly or indirectly from an act of the armed forces of the United States in combat" [1]. The Federal Tort Claims Act bars "any claim arising in a foreign country" as well as "any claim arising out of the combatant activities of the military or naval forces, or the Coast Guard, during time of war" [2] and precedent recognize a state of war without a declaration.
The courts in the country where the data centers were attacked might not be interested in addressing conduct of the US that didn't happen in their country. The courts in Iran are likely to consider the attack on the data center a legitimate act of war and not a tort; anyway good luck collecting against the US with a judgement from an Iranian court.
Amazon might have a better time making a claim against Iran, in the court where the attacks took place, but good luck collecting a judgement. Maybe a case in the International Court of Justice, but those have to be submitted by a country and involved countries must consent.
[1] https://uscode.house.gov/view.xhtml?path=/prelim@title10/sub... chapter section 2734 (b) (3)
[2] see page 28 and 29 of https://www.congress.gov/crs_external_products/R/PDF/R45732/...
The US would obviously retaliate by seizing Bahrain's assets in the US, which probably includes 100% of Bahrain's money (small countries don't get to have independent financial systems).
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
https://www.cbsnews.com/news/cloud-computing-loudoun-county-...
So then that big thing comes to you. It’s all kind of … held together by a prayer …
Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.
For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.
Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are sharded across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't shard across data centers at all, only across racks within a data center.)
Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.
Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40–50% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).
But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 has durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!
Amazon only claims "99.999999999% durability" per year, even for the properly replicated stuff.[0]
[0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
articleAuthorName: cbsnews (hidden byline)
articleSecondaryAuthors: n/a
articleEditors: n/a
Is the author a human or machine? Google shows 1 result for "30dacdbc-6a8a-11e2-9d12-0018fe8a00b0" and Brave Search shows 5 results.
There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
Similar here, there’s a lot of detail that got hand-waved away by a sloppy “yeah we good” puff PR answer.
Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.
If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.
He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions...
AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list.
And their Well Architected Framework and other docs make this plenty clear:
"It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)."
Also...
"All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region."
And also for single-Region / Multi-AZ architectures:
"Where possible, you should also copy data backups to another AWS Region as an additional layer of protection."
"AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters."
https://aws.amazon.com/blogs/architecture/disaster-recovery-...
Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross...
This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...
They have way more skills than the technical folk in doing that
Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.
2. Since that interview, AWS has started selling versions of their storage that isn’t redundant. It is not surprising that when AWS sells non-redundant storage that some data is not recoverable.
He didn't say "but if you hit many data centres then there is a problem". The premise was if you hit data centre, user won't notice, without caveat that there is a limit.
Now, there are a few things to consider:
- AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data
- You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data
When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.
An extraordinary statement itself that shows the difference between a proper cloud where they AZs are at least 60 to 100 miles apart...and Google or Microsoft... pretend clouds...where those AZs are just firewalls across the same data center...
How big was this flood?
I may just start using AWS Bahrain 2 as an additional example.
(excluding the geopolitical incidents)
Serious question: how do we know something is or isn't an act of god?
me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat) ???
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.
See also previous discussion: https://news.ycombinator.com/item?id=49033240
The surprising thing is that multiple availability zones were bombed simultaneously.
And to my knowledge, they haven’t even yet said 2+ AZs were compromised.
It's like saying "Oh this car manufacturer claimed that their cars were the safest in the industry but couldn't prevent this driver from flying through the windshield" and omitting the fact the person wasn't wearing a seatbelt.
e: Yep
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
[1] https://news.ycombinator.com/item?id=49033240
[2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...
Regions were always the scale of disaster isolation on AWS.
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
The only way to approach readiness here is repeated investment - like one team doing the deep dive and another pulling cables and then constantly doing pre- and post-mortems.
It's more likely the entire data centre (not just backups) would need to be built underground (or cut and cover) at enormous expense. A price that perhaps for certain data sovereignty reasons the government of Bahrain (or companies in Bahrain requiring it) would be happy to pay?
Another way to do things on the cheap could be small-scale "covert hosting". Buy an apartment or house, maintain it to give an outside appearance of being an apartment or house, but inside it has a few racks of IT equipment. This has been done in the past for telephone exchanges in some countries, not for security reasons, but rather to hide an ugly bit of infrastructure that due to technology limitations of the time had to be located deep within a residential neighbourhood.
me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 me (Middle East)but one in a trillion...
Sometimes those rules can have serious consequences.
But it very much breaks down for B2 where they're now storing original data. I hope they rethink this model. You do get what you pay for. There's a reason they're cheap.
I would imagine Backblaze would offer something similar for original data storage for their customers to choose.
[1] OK, "we" here is probably just me -- it was one of those things where the chances of successfully convincing anyone was so small, and the commercial advantages of just nodding along, and then changing your product offering was so great, that really very few people raised it or had reason to. But somebody had to!
As an aside, it is quite crazy the world we live in. I am with the majority where I expected Amazon to be more redundant, but I still marvel at the assumption that a US dev can spin up multiple redundant and data sovereign servers in dozens of countries with efficient caching, failover and redundancy (enough to survive an earthquake or targeted missile attack) from their own home. Even a few hours of outage in a foreign country is considered unacceptable.
As soon as you start specify technologies, rather than "sovereignty" you end up needing to created specific legal tests to stop people getting around it.
"Data must be stored domestically" is a short hand for being held in the same legal jurisdiction. This means for somewhere like the UK, you get all that battle tested data protections law for free. (new laws require case history to be reliable. Ie, prosecuting under a new law is hard, because if its on the edge of being legal, it can create a precedent that undermines the entire law)
In civil code places, its different, but I don't know enough to offer even a half arsed opinion.
The reason why jurisdiction is important is because if you are storing data outside of your legal protection, when something goes wrong there is little you can do to discourage fuckery.
This is the problem with blinkered engineering thinking. Yes geographically distributed data storage is good. But as you also know, storing it in place with lots of other data, means that its a target. The more places its stored, the more physical security you need. This means that there is higher chance of people being bribed.
Its not a binary, its a multi-dimension graph, with no one answer. Every dimension has a tradeoff.
UAE's tradeoff was: not even trump would ignore all the wargaming that clearly shows kicking iran in the nuts would have inflation rising consequences
Yeah, that was/is just another problem. Considering how that was actually handled in the real world before data residency laws came into force, I'm glad 'we' didn't convince those countries to put their citizens data at risk.
And second, the USA is in the middle of power grab that completely ensures any data stored there will be taken hostage wherever suitable for "negotiations".
Is it a real law? I mean I would largely ignore those kinds of regulations on the basis of sheers stupidity. After all, lawmakers of the world tried to ban math for multiple times in the last few decades. Why would anyone consider conforming to laws like that? Especially since its not possible to enforce this law.
> In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss
> In response to another question, Mr Vance rejected using the word “war” to characterise US operations in Iran, saying there was “no active shooting”.
I'm not going to complain to DoorDash if my order is delayed due to a car crash
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."
> The article does not tell us what products were impacted.
… right … which conflicts with EBS being what's reported …
(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
"I was in the office, reviewing Terraform plans"
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
You talking as if this is some mom-and-pop shop that you run.
[1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
https://aws.amazon.com/s3/storage-classes/
"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."
I wonder if "can't restore some data" includes any S3 data?
I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
I don't write contracts for a living, at least yet, but my learning so far is:
* clear goals: where is the end point and what does the product look like once it gets there in all required details
* defined test criteria: this is where you get to sue when they fuck shit up
* measures: there must be predefined measures. These can be wildly unrealistic at the start and require changes as the work occurs, which is ok, but there must be defined performance criteria that all parties are held to before work completion. In other worlds this is rewarded with bonus targets and penalties
Its a cousin of the mindset that the reason people don't steal is because they think they will be caught and rationally weigh up based on the value they gain and the chance of loss that its not a worthwhile action.
No, most of the time people steal because they think its wrong, and they dont want to do it.
Public trust is a real thing and varies massively by country. America is notably extremly low on this metric
Do you want performance, or correctness.
Well, if you want performance you use write through caching and in the case of distributed storage: more nodes confirming the block before returning. Huge performance cost.
Outsourcing this just means someone else makes these tradeoffs, they will prioritise the general case- and they’re even more incentivised to move the needle towards things that are most visible to the end user.
In this case, performance.
You won’t notice that theres a third commit server off-site (unless that site is bombed), but you will notice slower writes- and the general case says that people will express comparative dissatisfaction with weaker performance and use it as a justification to use another provider.
My mistake, if you want performance you choose write-back caching, and fewer nodes need to acknowledge the write. Sorry for clumsily typing the inverse when I was in a morning haze waking up :(
So any software company/project which actually took the time and effort to fully handle the enormous complexity, they can't sell themselves based on that fact because every other company (who didn't invest the effort) is also claiming it and the customer has no mechanism to verify the claims until some major rare event occurs.
And most of the effort is required precisely to handle those 1% of rare situations.
Not disimilar to how AI naratives are pushed on executives these last two years.
I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.
It's very clearly not about you being able to set-up redundancy for yourself.
But it's not designed to survive TWO datacenters going offline, and in a permanent fashion.
Not just in terms of service costs, but in time and complexity. In many cases building out that complexity is complicated and difficult. And sometimes the functionality you need isn't supported in the regions you use.
And for that, with a simplified use case, well, they can scale up to $20k+ a month if they like, that would be ideal, and if they need a more complicated setup or enterpriseyness, migrate off with my blessing and available-not-required hands on support (again for a reasonable fee, maybe $10k if you want the white glove).
If businesses are going to cloud but without engaging / listening to competent people who know these basics -- then the blame needs to be somewhat pointed back at those very business leaders I feel.
This is not obscure magical knowledge either that is tightly controlled. Any cloud vendor will freely teach you that. Or even a google search would.
I mean, every time cloud comes up on HN we see legions of techies posting strowman arguments about why you should offload everything onto AWS, GCP, Azure, etc.
But the problem is right there: "non-technical" leaders. Never work for one if you can avoid it.
"We should build vacuum delivery tubes for passengers!"
Now I'm thinking about legal/contractual rules that might force that kind of geographic risk.
I mean, logically you could have the Allowable Location send pre-encrypted backups to anywhere in the world, except (A) laws and regulations aren't always logical and (B) you still have the problem of keeping the decryption keys somewhere safe without leaving the key jurisdiction.
They could, of course, open other regional data centers in other countries, or say "data in this geo zone may be in any of X, Y or Z" countries, but for the latter that pretty starkly limits some of the major customers they'd have, I would guess, and for the former, well, they have other geo zones already, so if people weren't replicating to them, I'm not sure why adding me-east-1 me-west-1 me-central-1 would fix that issue, they just wouldn't replicate there either.
> Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?"
Seems pretty silly to argue, but he certainly did say "one specific data center", and I don't think anyone even non-technical will conclude "it's safe if they all go down at once" from this statement.
> https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...
They are far enough apart that tornados, floods, and fires cannot simultaneously impact multiple.
But AFAIK AWS has only reported data loss in mec1-az2.
Cross-AZ services like S3 should have no data loss, unless there is more damage than AWS has currently published.
No one was forcing Amazon to make deals with oligopolies and monarchies in the middle east.
No one was forcing Amazon to engage with US imperialism.
Like these things have extremely large threads connecting to one another. Now AWS, along with other American big tech companies, are legitimate military targets. This is what happens when you engage in these practices. You don't get to serve the interests of US imperialism and claim you're innocent. They deliberately signed up for this, there is a long history of this in US imperialism. It's nothing new.
Next time you don't want missiles raining down upon your data center, don't engage yourself in imperial politics. You won't be shocked next time when someone punches you in a face.
"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office
https://docs.aws.amazon.com/fis/latest/userguide/what-is.htm...
(The obvious option here might be an encrypted backup on some hard drives in a safe in a local office.)
I don't like Oracle. GCP has no servers in UAE. All other options were by local providers with not so friendly budget options
If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.
You think most countries have one data center or less, and most offices have less than two hundred watts of electricity supply?
The insurers always know how to weasel out of it.
Did Amazon make such a promise? They didn't as far as I know. My understanding is they provide specific guarantees like given an AZ outage, your data is still safe (provided you architect correctly).
Invoking force majeure requires the entity to prove all three following to be true:
A. That the event was unexpected and therefore unavoidable.
B. That the event was outside the control of the entity.
C. That the event made it impossible for the company to resolve the issue.
War in the region is as you say rather common unfortunately. The fact that AWS is used by the IDF (https://www.972mag.com/cloud-israeli-army-gaza-amazon-google...) should be considered a factor whether or not their data centre became a more likely target or not. What remains is the ability or not for AWS to do multi-location reduncancy.
How could it be otherwise - do you expect AWS to have its own army and missile defence system?
They don't sell any of the lower and less reliable levels to the public, I suspect simply because the reputational damage from losing user data is so bad, and the news will take no notice of the fact the user got a discount for less reliable storage.
If it dies, we will just have to re-download the data.
Anyway, we had a load shedding agreement with a large bank across the street from São Paulo and we could switch over to their Unisys mainframe in a matter of minutes, and vice versa.
And if the data center is more than about 5 years old it almost certainly was not planned with adequate backup power to run racks of GPUs.
Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They did have some access to fuel on Sep 1, but pretty spotty until maybe the 3rd. Looks like power started coming back in some places Sep 8, and maybe widely restored on Sep 14th.
I would say, by 7 days in you'll probably have a good idea of if 30 days might not be enough.
[1] https://web.archive.org/web/20061206181753/http://interdicto...
https://craphound.com/overclocked/Cory_Doctorow_-_Overclocke...
A perfect example of a claim they never made.
I worked at a 2000 person shop where the CTO moved us from 1 on-prem dc (we were begging to add a secondary site for years due to outage risk) to 1 AWS region.
It's not exactly straightforward to run multi-region across an alphabet soup of AWS services without decent configuration / application awareness, paying at least double.
Durability is about: "If I successfully store an object in S3, how unlikely is S3 to permanently lose that object because of storage failures?"
It does not answer: "Will I be able to access that object, after a rain of Shahab-1 or Shahab-2 burn all data centers in the 3 availability regions across which my S3 bucket is spread out..."
It's similar to any engineered artifact, like a skyscraper. They have a structural safety factor that defines their resilience to failures that the system is designed to tolerate, it doesn't make them immune to missile strikes.
this is one of one of those things - were in the current era either a cloud provider should provide automatic backups in another geographic zone.
if you're in us-east, then your back-ups should ideally be in eu-west + africa for redundancy.
The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive, especially when you consider the daily delta, and how much of that is ephemeral or frequently mutated data that is stored. The bandwidth costs alone would be eye-watering, let alone the storage costs.
S3 cannot make any automated decision about whether data is, or isn't important, and if they did they'd only open themselves up to lawsuits if they guessed wrong. That's why it's made an option for the end user to enable replication if they want to, or choose to replicate their own data.
It did work like this! And it was! My recollection is that the first S3 was out of SEA and had no user concept of region. Then “VDC” was added in virginia. That provided API endpoints in what became us-east-1. A bucket could be accessed from either location, and the original intent was for object store to replicate between them. By the time dub/eu-west-1 came along that was obviously not tenable; itd take 10s of gbs to replicate.
So S3 became regional. But the original sea/vdc deployments still had shared APIs and data in both regions. Your object would be stored in the region of the API you geolocated to via DNS, but read from either. _eventually_ all the data migrated to IAD, but those API endpoints were transparently proxying across the continent until 2013 or so.
And of course glacier had much more interesting takes on this with cross dc/az/region erasure encoding. But i dont think any of the wacky multi dimensional cross region stuff ever materialised in practice.
PS: hi!
Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.
(Encryption handles confidentiality concerns.)
Which is why data residency is such a stupid concept.
> I can assure you that lawmakers were not thinking about efficient disaster recovery plans or back ups when they made the laws.
I generally agree, although if a small country only had half a dozen or so redundant data centers then it would be relatively easy for a powerful adversary to wipe out all of the data centers and potentially have a significant economic impact on that country.
Having a backup data center in an ally country might make sense. Kind of like how I keep an encrypted backup hard drive at my parents house. Whenever I go to visit I pull it out and backup my laptop there too.
(I can feel someone saying "but surely having redundancy in one country is good enough, so I'll just say that I know relatively sane people who try to have hemispheric redundancy in their data, and also you never know when two different-in-every-quality-but one locations will suffer from the same disaster. Floods; heat-waves; national protests and strikes. It's surprising how often rare things happen!)
On your second point, it really is crazy. And also amazing that this is a capability that is -- or should be -- available to anyone in the world, not just in the US, and not just devs. Hopefully without also having to think about their data suddenly finding itself in a warzone.
You should check your home insurance contract, for instance... It likely would not cover an ICBM strike.
The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.
Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful to understand the fault domain (single server or single building or multi-building) so you can plan.
Blending the two numbers doesn’t help you build better against the systems. And the force majeure events are rare enough that they won’t happen … until they do. I’m not sure that knowing the precise probability that Iran would attack a gulf nation would change the fact that if they do, you need to have a DR story.
Offsite backups are sadly rare these days, and aws sales is the main reason why.
_You_ are supposed to have three copies of everything - and two of those should be off AWS.
But when I put data on normal S3, isn't my money supposedly paying for triple redundancy?
Look, here's the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
"S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region."
As far as I've heard before that's not with parity and three zones means three copies. But when I search now I see things about 5+4 parity, any insight here?
AWS doesn't use parity for S3, they actually mirror the data.
And in this case, it looks like all three DCs were damaged.
Both are bad, but having a missile hit your shop is never going to mean you are causing violence.
2013-2014 would be about right, they were still in SEA when I joined in 2013 but were actively working on decommissioning it.
You want to know how reliable the service is in steady state. For example it’s useful to know that S3 is effectively lossless in steady state whereas EBS volumes have an AFR of about 0.1%. You build your apps very differently between S3 and EBS knowing this. You can build highly resilient applications on each, but you code them differently, informed by these design goals.
You separately want to understand the failure modes that will require you to fully recover from backup. For example knowing that cloud storage is resilient to everything but region failure would inform you that your backups should be out of the region, not just a bucket in the same region. You don’t get that perspective from just a 9s number.
There is nothing to say that a determined adversary may still get at your data so it needs to stay in country.
This is why the minority of politicians who actually know about how this stuff works worry about where the data resides for jurisdictional purposes. If the government where the data resides can compel the folks who have physical and/or logical access to the physical machines that contain that data to give them access to that data, then that's game over for you.
«But you just don't permit that sort of breach to happen!» you might say. To which I reply "Yeah, right.".
Substantial physical separation of datacenters is very important, but the politics and policies of the location housing the data cannot be ignored.
We'd have to explain that if the data was physically in Brazil, but hosted by a U.S. company, that would not stop that company from accessing that data remotely -- unless you specified that. We'd have to also explain that if you were intended to defend against US mass surveillance of non-US persons by the US intelligence services, intelligence services and SIGINT are univerally almost defined by their broad remit to target foreign nations on their own territory in violation of local law. And, finally, if you intended to use the prohibiting the movement of of data as a sanction against companies to punish them for violating data protection standards, as pre-GDPR law in the EU had as an ultimate last resort, and the GDPR often ends up relying on as a last resort, you would find that multinationals are more capable of putting up servers in your home territory and continuing to serve your citizens than they are of substantially changing their practices regarding data processing.
I don't want to sound nihilistic about this -- regulations can exist in these areas. But it's those politics and policies of the institutions with control over the data that are the most important part of this: not where the bits are kept. Especially when those bits are encrypted, and the keys and access controls are elsewhere.
Us: We are concerned about our citizens (US) data, how are you managing the databases. Clout Provider (CP): They are only managed by fully background check employees. Us: Yeah, but where are they? What is their citizenship? CP: Um...mostly Eastern Europe. Lots in RU. (another CP proudly said "they're pretty much all in China...for cost containment"). Us: ...
Us: We are concerned about our citizens (EU) data, how are you managing encryption? CP: Everything is perfectly encrypted with hardware HSMs and all the FIPS and stuff. Us: So...where are the folks who run the HSMs? CP: Um...mostly SV. Some in the EU. Us: But can you assemble a quorum of US citizens for the HSM? CP: Of course! Us: ...
And on and on. Not to put too fine a point on it, many of us have no faith that vendors self policing international data protection in the face of government level pressure on companies and employees would work. Not that it can't, I don't think it would.
We may not be disagreeing that much. My argument was, and is, it's not about where the data is, it's about who has control over it. The counter-argument was "well if it's in another country, then we don't have jurisdiction, so it's going to be much harder". But what you need jurisdiction over is the people. Otherwise, you end up with multi-national corporate end-runs where you have shonky companies offering to store data locally, but who knows what department has control and access.
To be fair, the context I was having these conversations was countries arguing for data residency to combat the threat of mass surveillance (corporate and governmental) in the US, and the limited protections their users had relative to US nationals. But again, the problem is that it assumes that jurisdiction remains territorial: which is not how this was ever going to play out. The next wave after data residency requirements, beyond the usual extraterritorial intelligence community actions, was laws like the US CLOUD Act, the UK's Investigatory Powers Act, and Australia's TIA law, which effectively attempts to provide regular government departments and law enforcement with the legal ability to access data that would technically be on foreign soil.
My point was not that corporations should not self-police, but the concept of "it's stored here so we can oversee it" is not as clearcut as it seemed, and it risks introducing a new level of complexity to resiliently storing data. Which may be worth the price, but was never considered at the level this was discussed.
Nah. Policies prohibit rule-followers from accessing data that you don't want accessed. Such policies are very important. But if you give your adversary effectively-unlimited physical access to the hardware where the bits are kept, that's game over. If you don't trust the governors of a region to honor the "don't tamper with this hardware" gentleman's agreement, and you very seriously care about preventing unauthorized access to the data that that hardware stores and processes, then you don't put that hardware in that region.
To point to a real-world example of this, there's not going to be an AWS Top Secret Cloud region in China, Russia, or -say- North Korea.
Best laid plans and all that.
You're saying (correct me if I'm wrong) that data sovereignty laws are unconditionally bad because they inevitably lead to totalitarian followon laws and there's nothing to stop them. I'm saying if we didn't have enforceable data sovereignty laws we would be in worse shape for data privacy and we should have prevented the followon laws from coming to be (and, true enough, we didn't).
Further, the UAE datacenter issue is a red herring. It's an engineering issue not a political one (data sovereignty without physical redundancy is...stupid?), but schadenfreude is a helluva drug.
No, I'm making no "unconditionality" claim here: there are just risks and benefits. Sometimes you're the person in the room highlighting the potential problems. The risk with doing that is that when those problems don't happen, you look like a fool. But someone should raise the problems anyway, because that's part of the risk assessment!
Of course, if the problems do happen, then you get to indulge in "I told you so". But only if you failed to convince anyone at the time.