NASA to launch 247 petabytes of data into AWS, but forgot about egress costs(theregister.co.uk) |
NASA to launch 247 petabytes of data into AWS, but forgot about egress costs(theregister.co.uk) |
https://wasabi.com/cloud-storage-pricing/
Looks like egress is free.
Maybe because it's comparably untested? Does anyone here have any experience with it?
Wasabi does not charge for egress but our pricing model is not suitable for use cases involving the hosting of videos in a manner where the ratio of egress downloads exceeds the amount of storage.
It's hard you might think, but it's not. croit.io provides all you need to deploy a scalable cluster even on multiple geographic regions.
Price for 1 PB sized cluster including everything from rack to hardware to license to labor for below 3€/TB/Month or at the Amazon Glacier price tag but with the S3-IA access.
A 120TB SSD NAS might cost over 200k€ ..imagine a 250PB one
Isn't this a rounding error for NASA?
https://web.archive.org/web/20111024223108/https://visibleea...
247 Petabyte ~ 247000 Terabyte > 50000 USD.
Network cards, bandwidth, electricity cost > I can't guess.
Couple of good engineers (hardware and software ones), which they definitely have.
May be they could have built their own cloud in < ~10-15 million USD. And that won't be recurring cost.
May be they missed article about Bank of America saving ~2 Billion USD, by building their own cloud.
> Network cards, bandwidth, electricity cost > I can't guess.
This is where a huge amount of cost is.
> And that won't be recurring cost.
Maintenance, humans, cooling, drive replacements, property, building, land tax, payroll tax are all recurring costs.
Let take another setup of same count as backup. Then another setup as back up of back up. ~150K
> This is where a huge amount of cost is.
Maintenance, humans, cooling, drive replacements cost > can't be greater than first time set up cost.
> property, building, land tax, payroll tax
Nasa runs on Government budget, I am sure they can claim some tax break there.
The point I am trying to make is, it may be cheaper to do in-house with the level of engineering talent they have.
Now I am more curious why go along with AWS instead of using Openstack. Need to find some case study of openstack vs rest of cloud provider.
I suspect that ideas like this will become more popular as the US asks itself "what happened to our resilience?"
https://en.wikipedia.org/wiki/Internet2
Use backblaze pods, demand off peak bandwidth of gilded age megacorps that own said fiber for sync/replication.
https://www.backblaze.com/b2/storage-pod.html 480TB/4U
Have 3x sites around the US the build the pods, each new pod gets preloaded with a smattering of rarely requested and low replication count objects (as a redundant backup). Then shipped to the site where it will be used. Local writes go directly to pods which are then kept in sync with the rest of the cluster.
edit, from the TFA
``` And to put a cherry on top, the report found the project's organizers didn't consult widely enough, didn't follow NIST data integrity standards, and didn't look for savings properly during internal reviews, in part because half of the review team worked on the project itself. ```
1. Using the AWS cost calculator is pointless, naturally an entity the size of NASA would get heavily discounted rates. 2. As data volume grows, the complexities of working with that data expands. NASA appears to be embracing cloud computing by embracing a paradigm where scientists push computation to where the data rests rather than downloading data [1], [2], [3], thereby paying egress on only the higher order data products. 3. The report notes that NASA has tooling to rate limit and throttle access to data. This, in itself, proves that NASA didn't "[forget] about eye-watering cloudy egress costs before lift-off".
People may scream about vendor lock in, which is a fair complaint; but acting like NASA just didn't think about egress is misleading.
NASA is ultimately a science institution, I think diverting effort away from infrastructure management and towards studying data is likely a wise decision.
[1: https://www.hec.nasa.gov/news/features/2018/cloud_computing_...] [2: https://link.springer.com/article/10.1007/s10712-019-09541-z] [3: https://ui.adsabs.harvard.edu/abs/2017AGUFMIN21F..02P/abstra...]
Having spent a lot of money with AWS, that's giving Amazon more credit than I think is warranted.
Datapoint: When $company hit very high six figures (closing in on seven) in monthly spend I found AWS was incredibly willing to cut our egress rates, often by a significant amount.
This was explained by our account team: bandwidth has some of the best margins for AWS but they’re willing to sacrifice that for their enterprise customers (read: suck us in closer to non-commodity services)
Indeed. I am glad to see them leveraging the power of an already proven infrastructure provider rather than spending X billions of dollars trying to build and maintain their own.
Every major cloud provider is using Linux network drivers written by NASA employees.
In 1994 the NASA Beowulf project pioneered the idea of clustering together cheap commodity hardware to replace mainframes (this concept was later used to bootstrap hardware for Google)
NASA helped start the OpenStack project which powers a number of cloud providers.
Heck, NASA helped invent the GRiD Compass, the first laptop computer.
True, but once you're a certain scale, outsourcing everything just because it's not your competency isn't a good excuse. You can afford to hire enough people for it to become your competency.
There are just too many solutions to egress optimization to mention (CDN edge caching, rate limit, throttling, tiered discounts, multi-year agreements).
No gov procurement deal at this scale gets sticker shock from retail prices.
I strongly doubt this.
Amazon seems to work heavily on the principle that they charge their cost + a small margin. Which means that they can't heavily discount without going below their actual costs.
In their retail business. They finance that operation through the relatively high margin AWS.
If by "small margin" you mean "several hundred percent", then sure.
Not necessarily, depending on how the users access the data. If users access the data through their own AWS accounts, NASA could leverage S3's "Requester Pays" feature [1], to let the user pay for downloading the data.
1: https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay...
TFA says:
"a March audit report [PDF] from NASA's Inspector General noticed EOSDIS hadn’t properly modeled what data egress charges would do to its cloudy plan."
'Hadn't properly modeled' is very different from 'forgot about'. And if you actually read the linked report, it says things like:
"ESDIS officials said they plan to educate end users on accessing data stored in the cloud, including providing tools to enable them to process the data in the cloud to avoid egress charges." and "To mitigate the challenges associated with potential high egress costs when end-users access data, ESDIS plans to monitor such access and “throttle” back access to the data"
Neither of those statements would be in the audit if the entire topic had been a surprise.
"In addition, ESDIS has yet to determine which data sets will transition to the cloud nor has it developed cost models with the benefit of operational experience and metrics for usage and egress."
That sounds fairly close to the headline.
YOU ARE NOT AFRAID?
'Not yet. But, er...which way to the egress, please?'
There was a pause. Then Death said, in a puzzled voice: ISN'T THAT A FEMALE EAGLE?
(See also the inspiration for the joke: https://unrealfacts.com/pt-barnum-would-trick-people-with-a-... )
I've written two articles for them and the comments are a joke. They're all anti-Cloud, anti-progressive. Try selling them Kubernetes has a solution to their problems: they'll think you've come to steal their children. I know, I've tried.
In short: this never happened. NASA didn't forget anything. It does, however, make for a great eye catching headline!
Sorry to be bitter about this, but publications like The Register serve little purpose these days. It caters to a specific kind of IT personality that can't let go of their physical tin and they think public Cloud has no place or use at all. Again I know, I've tried convincing these people of such things.
Historically, data have been stored and processed on-premise but NASA has been migrating data and processing to the cloud where it makes sense. For instance, it makes a lot of sense to burst out to the cloud for near-real-time processing during and just after natural disasters like earthquakes and forest fires.
The large missions they mention (SWOT, NISAR - big radars in Earth orbit) are drivers of the shift of more processing + data to the cloud, because they will generate an unprecedented amount of data. They are pathfinders. By percentage, very little of that data will ever egress - it's low-level and uncalibrated - so a cached strategy could be valuable.
Here are some slides giving background on the SWOT/NISAR data system. They are from 2017, so more has happened in the meantime, but they touch on some of these issues:
https://smd-prod.s3.amazonaws.com/science-red/s3fs-public/at...
Regarding the step function in data volume, see the humorous slide #4.
Numbers used:
Initial upload: 258998272 GB (1024*1024*247)
Monthly upload: 100 GB (default)
Monthly delete: 5 GB (default)
Monthly download: 1048576 GB (1 PB)
Period of Time: 12 months (default)And they've promised to pay me, something like 20 odd times. With a specific date each time.
Which is never paid.
And ever time I ask what happened, the customer service person says they'll look into it, and never get back to me.
Most recently, they've sent an email saying they're closing my account due to lack of activity. No word on what'll happen to the funds, my expectation is they'll just steal the money for themselves.
"Lack of activity"... yeah, no kidding.. People tend not to use a service when the other party is obviously full of shit and repeatedly lies. :(
As for "AWS never forgets", sure. That goes both ways.
Given that, it's maybe still cheaper to build their own serving / caching layer in front to save egress costs than to have constructed the whole storage solution themselves.
This is Cloud lock-in using data location.
Cloud data transfers are too expensive, personally I assume that it costs more to measure and bill for bandwidth than the usage itself...
This is why you build and run your own storage, similar to Backblaze (who is almost entirely bootstrapped except for one reasonable round of investment).
I used to work very closely with this department at NASA. Without saying too much, the short answer is "tenured government employees more concerned about job security than the success of the project" is how an agency could make such dumb mistakes.
In fact, I can imagine ops-teams at Nasa licking their lips at the idea of doing away with a lot of that bureaucracy once they switch to AWS... note how the report mentions that some of the controllers are actual sponsors of the move: it’s obviously a conflict of interest, but it might well arise when the org as a whole is a bit too happy to steer away from a suboptimal situation.
This said, AWS will rob them blind, simply because they can. Like all outsourcers (which is effectively what they are), they get in with the simplicity argument, then boil that frog up with extra charges. It’s good that somebody pointed out one of those charges, but I doubt anything will change substantially- Amazon will probably cut them a discount and that will be it. And once you’re invested in a cloud env to the tune of hundreds of petabytes, you’ll likely not switch away for decades.
That implies a level of dishonesty or nontransparency that AWS doesn't have. Their pricing is disclosed, up front, and they offer a calculator to model your costs out. Knowing how much data egress you're going to have is not some arcane art, NASA just plain forgot to do it.
It may be complicated, but so is any workload at this size. Figuring the cost is part of due dilligence, and they've made it as straightforward as possible.
Oh that sounds like a potential solution.
/s
Also - can't they use torrent tech? I wouldn't mind helping out a bit on space & data
(If they distributed their science data in printed form, surely they'd be allowed to charge people for the cost of printing & mailing the paper copies; that's quite different from charging for the data itself.)
Requiring you to pay S3 is little different than requiring you to have Internet access, and thus pay whichever company includes you in THAT monopoly, IMO.
I'm also a liberal, so then I also think government should give everyone a monthly quota of internet usage allowance. Universal Basic Internet Income, or something.
For many operations, you may get to a point where it makes sense to build your own cloud.
If you're a seller, you might also get to a point where you want to sell goods directly.
It partly depends on your core expertise, meaning, is this part of how your business creates value? If NASA doesn't want be a datacenter provider, they should continue to outsource it.
It also depends on whether their business model aligns with yours. AWS's egress rules specifically work when you are getting revenue from the data being downloaded. If you're selling software or other media, and you can factor the cost of downloads into the price of it, pay-for-egress is very sustainable.
Other models like pay-for-capacity don't align as well if you want to maintain a large library of media and people are attracted by the variety, but only download the popular stuff.
For NASA, pay-for-egress may be entirely justified if their budget is based on usage of the data. Or if they can simply use "requester pays" to mitigate the cost.
Snark aside, it entirely depends on what you're doing. AWS probably has better engineers, better processes, and more of them than your company.
In most scenarios, it’s not my money, and I don’t care if it’s not my money. In this case, as a taxpayer, it’s my money (our money to be specific) and I care. I intend to contact my representatives about this failure, and have already fired off a FOIA request for AWS NASA contract details.
"At least NASA seems to have bagged a good deal from AWS: The Register used Amazon’s cloudy cost calculator to tot up the cost of storing 247PB in the cloud giant’s S3 service. The promised pay-as-you-go price for us on the street was a staggering $5,439,526.92 per month, not taking into account the free tier discount of 12 cents. The audit, meanwhile, suggests an increased cloud spend of around $30m a year by 2025, on top of NASA’s $65m-per-year deal with AWS."
AWS accounts still take management and a team of people that need to maintain a whole lot of different aspects of it, so you're not really saving on headcount. You're just moving that capex to opex.
It's important to be flexible enough to be able to deploy onto a cloud provider if the situation demands (e.g., new client demands infrastructure run in $FOREIGN_REGION_X where you don't already have a DC), but everyone's insistence on going 1000% AWS is absurd and IMO totally unjustifiable.
For what AWS provides, the DIY approach would be insanely expensive and wasteful. Not to mention, it would take years to build a basic MVP. They'd have to scope out the project, hire people just to design it, and then pay for several contractors just to stand up the first iteration of a working system (which will still not compare to AWS in terms of resiliency, redundancy, and accessibility), and then maintain it....forever. Also, many people already know how to interact with AWS. NASA would also need to design and maintain user-access methods to the data, on top of just plugging in thousands of hard drives and making them all work nicely together.
Why reinvent the wheel when there is a perfectly good wheel manufacturer that has already proven extremely successful at what they do?
Have you ever been part to an enterprise-level sales cycle? Things like the official calculator are waved away, since the customer is on a special deal, so "of course is not as much as that!". The customer asks for a quote with a certain degree of detail, the vendor provides an answer with the degree of accuracy required to get them in the door. If it turns out after a year that the customer ended up paying 2x, well, too bad - clearly they must have had higher requirements than forecasted! "Did you record all your traffic? No? Well, we did, and the result is this bill, sorry. Alright, alright, I hear your complaint, I tell you what - I'll give you a big discount on your next order, what about that?" Rinse, repeat. This is not dishonesty and I'm not alleging malfeasance or anything like that, it's just how that world works in my experience.
In order to figure out the real cost of outsourcing, you need an adversarial attitude that most shops simply lack, because they've fundamentally made the choice to abandon the previous solution even before they've entered the sales cycle. This is particularly clear in a case where some controller is also part of the group promoting the switch. It's surprising it was flagged up, there must be a competing group somewhere that is desperately trying to fight on - maybe some Oracle-friendly "japanese in the jungle" or something. Or maybe bureaucratic procedures to safeguard the institution are actually working as they should, for once, but that would be pretty exceptional in itself.
All of the cloud vendors de-empathize network egress costs. It's similar to products that depend on Microsoft licensing who will always omit those types of costs. (Oh, so you needed to spend another $500k in SQL Server Enterprise?)
Many organizations lack the operational metrics to allow them to effectively measure their egress needs. And AWS/GCP/MS salesmen arent in the business of slowing down deals with awkward questions.
This is especially true where an org like NASA probably contracts out things like network services. Going from a model where you make fixed capital investments to paying for the byte is difficult to measure.
Here's the official pricing calculator[1] - note that ingress and egress costs are included in all relevant services. Also note that for something like S3 (which is probably what the article mentions the "earthdata cloud" is based on), the pricing details are right there on the description page[2].
There is no evidence of any malfeasance by AWS here, just lots of casting aspersions. What specifically do you want that was not provided?
My question whenever I hear that people didn't is who did they ask? AWS doesn't just jump in to give people free service- but if you reach out to them and tell them you need it they tend to work with companies.
Imagine for a moment that in order to access NASA data sets you had to have a Fastmail email account. Gmail won't work, Outlook won't work, it has to be Fastmail alone.
That would be very objectionable (as much as I adore Fastmail).
Ability to pay one specific cloud provider should not be a gate for public domain government data.
https://spacenews.com/report-finds-delays-and-cost-overruns-...
https://www.popsci.com/blog-network/vintage-space/nasas-vab-...
It's useful for those agencies' budgets to reflect a portion of the cost of performing that research.
The USG needs insight into what taxpayer dollars are being spent on. Lawmakers have to explain to constituents why that money is being spent.
NASA is the first tier of information, collecting the data. Its budget ought to reflect that cost.
The consuming agencies are the second tier, processing that information. Their budgets reflect the cost of gathering their information and of processing it.
NASA doesn't know which information will be useful, so it's not helpful for them to pay the cost of egress. We want them to collect as much as possible.
It's much like a music store, 90% of their sales come from the top 10, but there's a lot of value in hosting obscure stuff.
If they have to pay to store it all rather than pay for egress, they'd have to justify the cost storing data that they can only say "it might be useful some time."
Having the agencies that are working with the data pay for the egress, they can justify the cost by showing the specific work they do.
But you're arguing for inter-agency billing as the correct way to weight scientific experiments? That isn't rational.
Better run your own Internet, after all, you care more about connectivity to your friends than your ISP does!
Dogmatism is passé. There are good uses for cloud, and good times for on-premise, depending on what you need, what your skillsets are as an organization, the kinds of workloads and length of time required for that workload.
AWS and others have absolutely outstanding amounts of infrastructure and tooling. Their reliability is off the charts in the past few years, and (once it actually gets figured out by your engineers) the cloud concept of IAM is incredibly secure.
There are pitfalls - cost, up-front complexity and several other things - but I no longer rag on "the cloud".
A quick google search for “amazon outages” lists the numerous extended outages they’ve experienced.
Total cost for constructing Launch Complex 39, which includes the VAB and the crawler-transporter launchers was estimated 1t $500M in 1962 for 2 pads. A total of 153 launches have occurred from LC-39. This number is greater than 4.
I’m willing to bet NASA saves a ton of money by going to a cloud provider- US government storage setups are insanely expensive. I remember a project I was on got a quote of over $10,000/TB in 2014, and there is no way egress is actually free right now- they are paying for a government regulation compliant internet connection one way or another.
I do worry about vendor lock in to a degree, but I’m confident the agency and tax payers would save money going to any major cloud provider.
I've operated pretty significant government shared infrastructures like this in the past... we were offering fast, flash-cached disk in 2010 for about $5,000/TB. $10k/TB is not unreasonable for highly available Tier-1 storage for something like SAP, especially in that era where you couldn't use all flash in most case.
Today, cost structures can be very different. You can land high-iop storage for a fraction of the cost without the overhead of a big SAN. If you need capacity focused storage, that is also much cheaper.
An agency like NASA gets hosed on services, and cloud is no different. AWS is probably a net savings for operational workloads whose characteristics are known. Backup is a no-brainer. But for a high-volume, operationally highly variable thing like a public archive of data, AWS a square peg in a round hole because of the metered access.
Charging the user for data, even if it is on a marginal cost basis, conflicts with a mandate to give data away freely. Because “at the marginal cost of delivery” is not “free”.
(It's true that it is common for mandates to specify something like at marginal cost of delivery rather than free—sunshine laws providing copies of public records often work that way—but that's not the applicable mandate here; in fact, since without the separate mandate here the data would be available on a marginal cost basis under FOIA, the main reason for a separate mandate is to negate that cost.)
I found https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPD&c=2230&s=1, which mentions things like "Ensure public access...", but I don't see anything there mandating such public access to necessarily be at zero cost.
https://www.usatoday.com/story/tech/news/2017/02/28/amazons-...
If my app is architected for reliability, I’ll run it on bare metal and keep the costs savings. Why pay twice by building it for cloud durability and running it on expensive cloud resources? Clearly the AWS marketing is working (“you’re just building it wrong”).
We’ll see what happens when CFOs take the reins from CTOs and CIOs and start putting cost controls in place during this recession (“why exactly are we paying so much in opex when this could be capex we can depreciate?”).
Few businesses ever get to the point where they need to run their own datacenter. And when they do, the costs would be roughly even or lower to AWS due AWS' markup (for handling those DC-related things for you, plus profit).
Disclaimer: Previously a devops/infra guy, before that ops/networking/sysadmin, built out colo facilities/datacenters/hosting companies before cloud. Have done a lot of cost models for storage and compute, still do on the side.
It's generally not the same people who are going to be at their computers running awscli (or if it is, now we get to figure in how much time they're spending on tasks that are not their primary job and how many extra of them we get to hire to maintain the same velocity, not to mention the occasional bit of firefighting you get to do when you manage your own infra)
You'd be buying something like an EMC vMax that can sustain 1M+ IOPS on lots of 15K spinning drives, with caching tiers on crazy expensive flash.
To support that, you need a fibre channel network layer and a bunch of FTEs to attend to it. Usually compliance requirements require segmentation of roles, which increases cost. If you're a federal government entity, those FTEs are most likely contractors billed out at $125-300/hr. Figure $3-5M/year on labor costs alone, although that may be divided out over multiple systems.
This happens in commercial business too. I had a buddy who was making about $150k in NYC to zone luns on a SAN. Basically he kept a spreadsheet and updated a specific configuration setting 2-3x a day and spent about 60-90 minutes/day doing that. The rest was waiting or studying for his MBA.
It's pretty wacky to compare S3 to this type of storage.
200 iops per drive from 2.5" 15KRPM is good going....
Edit:iops auto spellings
If 1-2TB drives were handily $1k in 2010 (2005 $1K hot you 128GB 15KRPM)
and your array set is at least R10,
already raw storage is approaching half of ten thousand dollars.
And this ignores controllers, cabling and chassis.
And this is before we look at our storage software licenses.
Is backup, point in time SLA, replication and availability in this budget?
The overall system ended up with worse capabilities than it should have had, but it did ship.