We reduced our server costs by moving away from AWS(levelup.gitconnected.com) |
We reduced our server costs by moving away from AWS(levelup.gitconnected.com) |
For example backblaze B2 offers free egress through Cloudflare, Fastly, BunnyCDN.
https://help.backblaze.com/hc/en-us/articles/217666928-Using...
Are you really saying that AWS and other clouds are expensive? Say it ain't so :)
If it costs you $1,000,000 a year to serve 1166 requests a second, maybe you fucked up.
...at the expense of 40 eng-years (20 eng over 2 years) spent on the migration.
This company saved $800k/year. Perfect time to go in-house with this solution.
But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-house.
At the right time, you make the right transitions.
If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile.
If they saved $800k per year, and they have to hire ten additional ops engineers to run it at a cost of $1 million per year, then they've actually gone and burned $200k on something that provides no additional value to the business or their customers.
But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away.
There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people.
At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.
I guess you've never done it yourself?
I'm not sure what those 4 engineers would be doing. You purchase a few servers with 5 year on site warranty and remote management, you take a couple of days to install them in some racks, and you never visit the site for 5 years. If they break, the manufacturer sends someone to fix them on site. The rest of the time you administer them remotely - just like you would AWS. If you want VM's install proxmox.
Whether it's cheaper than renting bare metal in a data centre is an debatable - colo costs a small fortune where I live.
But they aren't comparing it to that. They are comparing it to locking themselves into a closed source system that need specialised expertise to run, versus administering an open source Linux system that even the people running EC2 should be very familiar with. Most of the stuff AWS provides has open source counterparts - hell a whole pile of it is just open source they wrap a proprietary API around and charge you for.
10 times sounds like a lot, I'm guessing it's really less. Even so I come form the camp that shakes his head in disbelief at what people will pay AWS for basic VM and storage services, dressed up in a fancy API and marketing.
I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu
I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up and what the funk is eBPF? Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.
Look at Stack Overflow's architecture which stands apart because it was never designed to work in cloud from the beginning: https://stackexchange.com/performance
I'd argue that 90% of the SaaS doesn't have SO's scale. The whole thing would work just fine on a couple of FreeBSD servers running postgres and un-dockerized monolith. Half a rack at most with redundancy and replication.
But, if you've built your whole company around proprietary lamda functions and a vast range of AWS offerings, you're setting up yourself to never get out of the mess.
And you need dev servers. And a way to keep them in sync with production because you can’t just create a branch and provision a bare metal server on the fly. And you need a deployment process to get code from dev through test, UAT and production, and this process is lengthy and fraught because you can’t just fire up a new instance and switch the elastic IP, you actually have to deploy code on the physical machine and make sure there aren’t any configuration differences between your test and prod environments that’ll cause ‘it worked before’ problems. Developer productivity tanks, people are afraid to make changes, any pretence of agile/devops goes out the window and eventually everyone gets sick of the crusty old server and decides a complete rewrite is needed.
While Stack Overflow is popular in the dev world, not so much outside of it. The show they’re only serving 300 req/s. 19 servers to power the entire stack as well. I wouldn’t call this efficient.
At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily.
How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electrician to fix some power issue?
How much to setup and run a large air-con system for the data centre. Maybe not much in the US where aircon is common but much more expensive in Europe.
The fact they could afford to do this over such a short time period speaks to having a decent amount of cash on-hand.
Co-locating has no capital investment other than hardware, and is pretty cheap.
A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year.
Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital.
All this to say is that I don’t think capital is as big a factor as you might think.
> large air-con system
You wouldn't usually jump from AWS to buying up real estate to build your own physical data center.
A sensible first step is to rent a rack at a colocation facility. They handle power, cooling, redundancy, physical access for you.
Don't underestimate the savings that can be made from switching from a big-name cloud provider to a more old school hosting provider.
At work, people keep complaining about our costs and coming up with spreadsheets showing how much money we would save with our own hardware.
They never add the engineering costs. When they do, they forget to include the ongoing maintenance. Or the new SMEs that need to be hired (and on call). Or even the opportunity cost of doing a multi-year migration to arrive at the exact spot they already are today.
All that money, and noone is looking into optimizing our systems to shrink the bill...
Yep, we should have pointed out that this has to be done in at the right time. Generally this article was about how we saved $1M expense and we was able to share this saving with our customers. Not an attack on AWS, I still use their platform for my other projects.
But to be accurate, this migration happened last year, and now we already doubled our usage and this brings us to $2M / year saving now.
I 100% agree with the comments below, use AWS while you are defining your product and growing, Todd (the founder) did this right. Spending days and nights on managing your own infrastructure or paying a devops employee to do it, in the early phase is just pure waste, and takes away focus. But we moved away from AWS at the right time.
Also, yes we had to do this to be competitive, as prerendering is a resource heavy service, and if your site is like ProductBoard (awesome tool) which will not consume terabits of traffic or use petabytes of ram, then you can stay on AWS forever and enjoy the benefit of not caring.
And what we lost? Minor inconvenience at this point, I cannot just pull up a new database in 10 minutes, but we don't really do that anymore, most of our current projects requires months of research, planning, and delivery. With those time frames we can notify our devops and get everything in place way before we need it.
So, keep on using AWS (it rocks!), just be sure to pay attention to the bill, and don't underestimate the cost of the traffic.
Have a nice one
$800k/yr is like the cost of 2-3 engineers. Even if they're capable of doing all of the work that used to have been done by aws, you don't have any room to expand without having to put up high capital costs, and certainly not on a dime.
It sounds amazing now, but wait till the future comes and you can no longer run your own data center as cheaply as amazon.
AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce your bill and often that means using AWS-specific services. If you want to stay completely "cloud agnostic" you are going to paying more than buying into a "cloud", in that scenario then you absolutely should be looking at dedicated servers. AWS is great because you can bring existing software and just run it in EC2 (or their container stuff if your software is containerized) but the AWS magic comes from using their managed services (also spinning up/down EC2 instances as needed, but if you are running them 24/7 then consider alternatives or at least pay for reserved).
I'm so amazed that somehow people completely forget that for literally decades, web host provided dedicated hosting options at fantastic prices.
Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited.
But large numbers of web host today allow you to programmatically spin up a dedicated web host instantaneously and at a fraction of the cost.
My new startup is focused on helping application owners repatriate their workloads into their own infrastructure.
Our goal is to solve the network complexity challenges with a fully open network stack (open source software with all of the hardware options you would expect, and some you wouldn’t). The solution is designed to be turnkey and require very little network experience. It will use standard DevOps tools that you’re already using.
We’re announcing it in two weeks and will be posting info here on HN!
These cloud providers are, by definition, charging you more than it would cost you to run it yourself. What you get in return is a guarantee of expertise and an ecosystem.
- may spend $250K on servers, replaced after 3 years becomes $83k/yr
- may spend $120-250K on extra staff to maintain the infrastructure
- may spend $15K for a cage in a DC
The real fun comes later when you outgrow your cage and there's not enough space left in that DC, or they just have shitty service constantly knocking out your racks, and you have to consider splitting your infra between DCs (a huge rewrite) or moving DCs (a huge literal lift and shift). Have been part of both, it's... definitely a learning experience.
AWS is not cheap because of your server costs.
AWS is cheap because of elasticity, velocity (opportunity cost of next feature), and reduced maintenance hours.
"The cloud" was never (afaik) was about getting a cheaper VPS. It was about being able to get them on demand, give them back on demand, and generally not have to maintain anything besides your code (and maybe apply updates to datastores / AMIs)
Now, if those premises are not true for your startup/business, then AWS is not the tool for you. I didnt see any analysis of ongoing maintenance costs in the 800k saved, but will it take 1-2 FTE engineers to now be more oncall, more server upgrades, more security patches etc? That's easily 1/2 that savings gone already.
Edit: for the most part these attributes apply to GCP, Azure, Heroku etc as well, its not just about AWS
So the team is now responsible for backups, hardware ordering,.forecast etc?
How big is the team now compared to before?
Does it scale?
If you price it correctly and keep the free tier small, I would either talked to AWS for better pricing or moved to another cloud Provider.
S3 on AWS is a total no-brainer, minio on bare metal might mean much more work and a bigger infra team than business actually wants.
I would also love to know what optimizations are already in place. Does cloudflare caching work? Are the results compressed on rest? Is geolocation latency relevant?
Why even Cassandra? Are websites not unique? Wouldn't a nginx and a few big servers not work?
But who knows? The article doesn't tell me that :-(
"However, all this data and processes need to happen on a server and, of course, we used AWS for it. A few years of growth later, we’re handling over 70,000 pages per minute, storing around 560 million pages, and paying well over $1,000,000 per year.
Or at least we would be paying that much if we stayed with AWS. Instead, we were able to cut costs by 80% in a little over three months with some out-of-the-box thinking and a clear plan."
You've got the FUD covered, but you also need to add at least some substance to your claim. How do you know this figure would not be accurate? Why is your (hypothetical, not offered) estimate better than the author's?
> After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio.
if they served directly from s3 that would be, stupid?
> In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster.
is also strange. it misses a lot of detail but it does not look like they just migrated away from s3...
(looks like their new hoster is hetzner, from service.prerender.io )
So an alternative way of interpreting this is more along the lines of: We may have saved up to 80% of server costs by moving from AWS, but you almost certainly won't save that much even if a bunch gets spent on developing operations and tools.
Also, if you are bigger and can start really negotiating with hardware providers.
For example, if you primarily serve large media or many tiny files to clients that don't support http Multipart Types, than AWS can cost a lot more than the alternatives. However, AWS is generally an economical cloud provider, and a good option for those who outsourced most of their IT infrastructure.
The article would be better if it cited where the variable costs arose.
Regardless, starting a new Cassandra cluster in late 2022?! I bet they can save even more by just going with scylladb
[0] https://dzone.com/articles/s3-compatible-storage-with-cassan...
I heard they are very stubborn to increase the per user limit of cloud instances.
Also how did you deal with S3? Did you switch to another provider ? Like B2?
Even if you have 2 such boxes for master/replica, it's close to 5x savings.
Couple that with the very well known fact that AWS has outrageous data egress charges and there are patterns that can emerge where you're still in cloud but not racking up massive outbound data charges.
What does it cost to run their data center? What are the salaries they are paying for internal IT efforts to administer it? Is it an apples-to-apples comparison, e.g. are they load balancing across multiple datacenters in case of an outage?
It sounds like this was a good move for Prerender but it's hard to generalize the cost claims to other situations without details.
This seems like a key detail when telling people about your migration off AWS.
Before, it was easy to justify almost any expense with the "we just need to get 1% of this $100 billion market" and now it is "hunker down and do everything you can to be ramen-profitable, in order to survive and thrive".
[0]: or 90%, or 80% or who cares, but a majority of software services seem to NOT be compute- or traffic- heavy.
It's a fascinating article, for sure. I would have been interested to hear what their backup strategy looked like though.
One of the big benefits of cloud services, that I am aware of, is the assurance that if natural disaster strikes, you don't lose all of your data. I kind of got the impression that, more than anything else, that is what you are paying for. Data protection and uptime.
I suppose big enough bills could lead a company to make the kinds of changes that Prerender did, but when that disaster does strike, and it is time to try and recover from a fire, flood, earthquake, etc. the responsibility and speed of getting your customers back online is reliant completely upon your staff - a staff who might be extremely shaken up, hurt, or pre-occupied in taking care of their own affairs. I'm not saying it's not possible, but there is a kind of cost that comes in the form of responsibility. It's a trade off that I would not fault many people from avoiding.
$1M per year bill is a lot, but the Prerender back end is extremely write-heavy. It’s constantly loading URLs in Chrome in order to update the cached HTML so that the HTML is ready for sub-second serving to Google and other crawlers.
Being a solo founder with a profitable product that was growing organically every month, I really didn’t have the time to personally embark on a big server migration with a bunch of unknown risks (since I had never run any bare metal servers before). So the architecture was set early on and AWS allowed me the flexibility to continue to scale while I focused on the rest of business.
Just for a little more context on what was part of that $1M bill, I was running 1,000+ ec2 spot instances running Chrome browsers (phantomjs in the early days). I forget which instance type but I generally tried to scale horizontally with more smaller instance sizes for a few different reasons. Those servers, the rest of the infrastructure around rendering and saving all the HTML, and some data costs ended up being a little more than 50% the bill. Running websites through Chrome at scale is not cheap!
I had something like 20 Postgres databases on RDS used for different shards containing URL metadata, like last recache date. It was so write heavy that I had to really shard the databases. For a while I had one single shard and I eventually ran into the postgres transaction ID wraparound failure. That was not fun so I definitely over provisioned RDS shards in the future to prevent that from happening again. I think RDS costs were like 10%.
All of the HTML was stored in s3 and the number of GET requests wasn’t too crazy but being so write heavy on PUT requests for recaching HTML, with a decent sized chunk of data, the servers to serve customer requests, and data-our from our public endpoint, that was probably 30%.
There were a few other things like SQS for populating recache queues, elasticache, etc.
I never bought reserved instances and I figured the new team would go down that route but they blew me away with what they were able to do with bare metal servers. So kudos to the current Prerender team for doing such great work! Maybe that helps provide a little more context for the great comments I’m seeing here.
AWS seems ideal to me because it would let us easily scale up and down as our usage varies. But some of the anti-AWS sentiment in this article has given me pause. Any reason not to do this? Any alternatives I've missed?
Our storage and transfer usage will be negligible; it's all compute.
>Do you have any advice for software engineers who are just starting out?
>Don’t be afraid to talk with the customers. Throughout my career, the best software engineers were the ones who worked with the customer to solve their problems. Sometimes you can sack a half year of development time just by learning that you can solve the customer’s issue with a single line of code. I think the best engineers are creating solutions for real world problems.
to be very good generic advice.
The point of AWS is to be flexible. You’re paying for that. It’s easy to start. It’s easy to stop. It’s easy to change capacity.
Running your own servers is none of these things. But it is cheaper at sufficient scale. You can’t ignore the labor cost (particularly engineering) however.
Where AWS shines is with highly volatile workloads. With your own servers you have to provision for peak capacity. That’s less the case with AWS.
No shade on the author of course. It’s great to read things like this.
If you already have a mature team that is doing well optimising the environment, you have stable demand and no need to develop things rapidly then it is very likely that going cloud is poor choice.
As to "cloud agnostic", don't believe this bullshit. In my opinion most projects fare much better by just letting go of this "cloud agnostic" and just getting tied and locked in to the vendor. It is not like there is a good chance Amazon (or Microsoft or Google for that matter) will suddenly pull a significant price hike or another move out of step with other vendors.
Spending effort on trying to make your app "cloud agnostic" usually ends up with far higher development costs and a chance of failure for no benefit. Embracing one vendor in this case is usually best way to achieve smaller, leaner app that is using the platform well.
It is the same story as with SQL. Trying to use frameworks to keep your app DBMS-agnostic but then nobody ever migrates their apps to another DBMS. And for various good reasons: you hired your staff for their expertise with X, they are used to it, so now going for Y will usually be far higher cost than the benefits.
> It is the same story as with SQL. Trying to use frameworks to keep your app DBMS-agnostic but then nobody ever migrates their apps to another DBMS.
I agree, I can't tell you how many hours I've wasted trying to keeps something (theoretically) cloud or DBMS agnostic, how many problems it's caused, and at the end of the day we could never "easily" move to a different cloud or DBMS without large rewrites.
I built on top of the Serverless Framework and I am constantly kicking myself for doing so (eventually I'll move off it for my main project). It's the worst of all worlds and in the Serverless Framework docs they even have sections for "GCF" vs "Lambda", if I have to write my functions/declarations differently then why am I using your layer on top? I know you get a few things for "free" with the Serverless Framework but once your project grows you run into all sorts of issues that are a PITA to work around. The truth is I'm not leaving AWS Lamda for GCF or whatever Azure has, I quite enjoy my "walled garden" and using an abstraction layer only means I get the lowest common denominator or a headache when trying to do something that is normally easy if you've fully bought in.
A small nit, I think this is partly because we've been burned before. Same idea as backups, if you don't restore from backups, you don't know if you have good backups. Similarly, if you don't use certain code paths, you can't tell if those code paths are sufficiently bug free.
I am thinking of Drupal. Pretty much everyone who uses Drupal uses MySQL/MariaDB as far as I know. I think Drupal supports Postgresql but nobody I know uses it because nobody they know uses Drupal with Postgresql. I don't know anyone who uses drupal with sqlite on production either.
If you have 100s of projects cloud agnostic infra is good. If for example Amazon buys your competitor and now you get "new contract coming up".
That said I agree "most projects fare much better without" - a lot of people don't understand they are not in position where they would benefit from that.
A notable exception to this is of you have clients whose IT departments require you to work on specific cloud providers. Although, there's tools that help abstract deploying infrastructure to different cloud providers (eg terraform, pulumi), but they still require some familiarity with the providers. With all that said, overall I agree with your sentiment
Yeah, I never got this. If actual, for really real SQL is necessary for your application, then you need an actual, for really real DBA. If it's just a really nifty flat-file interface that "just werks," go with whatever DB engine your framework is most tested/used on.
I get that sometimes you're interfacing two different things, one of which is on one DB engine, and you'd like to use the same software. That's fine. But DB agnosticism has a cost like anything else.
If you can be truly cloud-agnostic, then there is a high probability you can be cloud-free.
I feel like a variety of other circumstances can come to pass, which would negatively affect business continuity, here's what a lazy search turned up:
"My Google Cloud was suspended too" https://news.ycombinator.com/item?id=32571055
"Google suspended our domain out of the blue" https://news.ycombinator.com/item?id=32798368
"Tell HN: Google Cloud suspended our production projects at 1am on Saturday" https://news.ycombinator.com/item?id=32547912
"AWS account was permanently closed because it was suspended for 90 days" https://news.ycombinator.com/item?id=31571538
(probably happens with Azure and other smaller platforms as well, e.g. Hetzner, DigitalOcean, Vultr, Scaleway and so on)
I'd argue that it's good to build on common standards, like OCI containers and the wire protocol of some common DBMS like PostgreSQL or MySQL/MariaDB: so that you can replicate certain parts of what a managed service does in a container locally, for development and testing. In most cases it won't matter that the managed cloud offering has some clever engineering underneath it and scales better, as long as you can check whether your SQL executes as it should and none of your seeded/anonymized test data breaks.
I actually had this project with Oracle DB that ran horribly when the instance was remote and you really couldn't do migrations well without breaking things for anyone using it, because the apps using it were written in a way where hundreds if not thousands of SQL queries were done just to load some data to display a page. Which is passable (if you don't care) when the DB is running in the same data centre as the apps, but absolutely horrible when these many smaller SQL queries have the full network round trip between them, especially when the app initializes data sequentially (N+1 problem). Ergo, the only way to work with something like that is to setup a local database (e.g. Oracle XE) and work with it, after importing a baseline migration/doing automated migrations.
The same largely applies to any other DB with sub par application architectures, as well as many other services (e.g. MinIO/Zenko instead of S3, so you don't need to actually upload 100 MB files to test whether attachment logic in your app works, if you can run them locally).
As for software that should be compatible with multiple DBMSes: sometimes it makes sense (e.g. something that could be used by your customers in a self-hosted setup across numerous different setups, like Zabbix or Nextcloud), but most of the time it would negatively impact how easy it is to develop code for your app (e.g. having to rely on ORM and their abstractions like JPQL) and testing everything would usually take more effort.
This is basically the natural progression of SaaS...
There's no such thing as a managed service so proprietary that it can't be essentially cloned by another provider. I remember people making the same claims about S3 long ago, and yet there are multiple providers of S3-compatible file storage services now.
Microsoft's Azure provides migration tools for people wanting to leave Amazon's AWS, and they are not alone.
Amazon tends to be (in my opinion) a few years ahead of everybody else, and different competitors focus on different areas of competition, but migrating away from AWS isn't really as hard as the "proprietary" label might make it seem.
First, AWS is already highly profitable, they aren't dumping where this step becomes a necessity.
Also, any such move would dry up future growth from adding new customers, which is suicidal.
For the other services, it's still going to depend heavily on your usage, staff time, and operational efficiencies. For example, if you replace SQS with RabbitMQ you need to manage multiple EC2 servers for reliability but those servers might be significantly underutilized based on your traffic levels. Whether or not you save money depends on how much you pay your operators and how many messages you use: if you use less than the free tier's million messages per month, there's no way to beat it. Each million requests costs $0.40 or less, so if you pay your ops person $50k annually (haha) you'd need to be somewhat over 120 million messages per month to pay for them to spend a single hour on O&M even before factoring in your EC2 usage.
If you are using lambda and suddenly need to run a million lambda you just can. If you want to have the ec2 support to run a million lambda at some time you are going to pay for a lot of sleeping computers.
Ok so you save money on the monthly bill but what happens when Amazon decides they want to enter your market? What happens if they decide your service is too controversial and they kick you off?
If you were deploying your own services to EC2 instead of using AWS's own services you could at least setup shop elsewhere with just a bit of work.
To me it is antithetical to building a sustainable product. People just hoping their startup gets bought and then it is someone else's problem.
They are awful to develop against, awful to test against, and often times "just work" until they don't.
It's possible that they are not interested in a response. They are saying what worked for them, whether or not others want to agree.
At one point it took often days for a dedicated server to be set up. We also didn't have such nice provisioning tools.
Now it just seems like cargo cult to use cloud providers as the only option. People just completely discount dedicated servers.
developer costs >> infrastructure costs
An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense.
PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops. The major downside wasn't cost, so much as flexibility.
Dedicated servers start to make sense for:
- The very low end (e.g. personal use, or hosting something long-running for a small business)
- The very high end (e.g. once cloud costs start to hit hundred of thousands of dollars per year)
On the very high end, cloud providers will often cut a deal, though.
My problem with AWS, recently, has been reliability. Servers crash or have performance degradation a bit too often. That leads to developer costs, and might be what pushes me back to dedicated.
And if you try to use AWS just for S3 then you will pay a lot extra for the bandwidth charges of bringing the data from S3 to your server (something that is free if you were to use EC2 or other AWS services).
Their notes here are a bit vague:
> When the migration reached mid-June, we had 300 servers running very smoothly with a total 200 million cached pages. We used Apache Cassandra nodes on each of the servers that were compatible with AWS S3.
> We broke the online migration into four steps, each a week or two apart. After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio. When the writes to S3 had been stopped completely, Prerender saved $200 a day on S3 API costs and signaled we were ready to start deleting data already cached in our Cassandra cluster.
> However, the big reveal came at the end of this phase around June 24th. In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster. The daily cost of AWS was reduced to $1.1K per day, projecting to 35K per month, and the new servers’ monthly recurring cost was estimated to be around 14K.
It says (briefly, in passing) that they used Cassandra to implement the S3 API for their nodes, but maybe just to replicate the S3 API that they were previously using? That's an interesting choice I'd not heard of before. Perhaps all of their individual files are quite small?
Then they moved to MinIO, which would be the S3 equivalent that you are looking for.
EDIT: to the naysayers: I worked in various companies (from tiny to huge) maintaining their own DCs and had access to financial data. And I was involved in the setting up new DCs.
I've been getting by on like 80$/month hetzner servers (Like ancient 10 year old pizza box machines) for my businesses for a long while. Never did the cloud thing, I can run like three or four websites on one of those bad boys. I guess stuff like AWS makes sense if your computational workload happens in bursts or whatever? But 80$/month overhead costs are like nothing, I spend more than that on a good night out level.
That's not a paradox. You're not crazy in either case.
Starting in the cloud reduces costs for some strategies by removing necessary engineering. Starting in the cloud increases costs for other strategies by charging too much for commodity offerings.
It's relatively straightforward to make the choice as soon as you put a specific business in the crosshairs.
Edit: why is this downvoted?
(And before anyone falsely claims no commercial offers for this exist yet:
https://www.plusserver.com/en/products/pluscloud-open [no affiliation])
The physical network is most likely the absolute root of trust and attestation. Physical devices gain their personalities from what network they are connected to.
Physical networking is still the biggest impediment to operating on-prem infrastructure. You need to hire very expensive engineers, deal with a bizarre purchasing paradigm dictated by a few large companies (Cisco, Arista, Juniper, etc) and the management of these devices is pretty close to being from the Stone Age.
So we’re looking to fix all of that with a pure open source play, using components that everyone is familiar with.
The end goal is that a Devops person with basic network skills should be able to build their own physical network for modern cloud native applications, with very low levels of friction but similar cost efficiencies as cloud operators like AWS, Azure, GCP.
That is _not_ a given. They're charging you more than it costs _them_ to run it.
They get:
- much lower hardware prices than you
- lower bandwidth prices than you
- likely lower electricity costs than you
Realistically, a company will charge slightly less than whatever your alternative is, and they will provide products that restrict the number of alternatives you have.
AWS, GCP, and Azure provide several differentiated products that lock you in to higher prices on everything else. Effectively, these three have an oligopoly on "highly differentiated cloud services." They are only competing with each other on those services, and they are competing with "servers plus ingress/egress costs to our differentiated services" on their commodity products. That is the real reason why AWS egress costs are so high. It prevents you from picking and choosing where to buy each part of your cloud footprint, and locks you into buying AWS. Bandwidth costs are what keep you inside the AWS/GCP/Azure walled garden.
Lower-end providers like Linode, DigitalOcean, Vultr, and Cloudflare have parts of the differentiated offering, but not all of it. These people will have lower prices than AWS/GCP/Azure since their offering is less differentiated, but they will still charge you more than you would pay by renting a server, since they offer more products.
Finally, hardware rental providers like Hetzner and other operators are competing directly with you buying the hardware and paying for datacenter space and bandwidth. Datacenters typically charge a premium for power, even when they are in areas with low-cost power.
Notice that none of these companies are competing with large-scale server buyers who have the same hardware/bandwidth/power costs that they do. As such, they do not need to pass the savings they get onto you. That is where they get their profit.
It's my impression that AWS competes with the EC2 instance cost (as it that's what new customers look at), and the bandwidth/storage only becomes apparent when you are locked in.
Really what AWS should be is one or two phases of service maturity: dev infrastructure and experimentation is phase one, phase two would be the "its a couple servers" production, BUT: with a scale plan for phase 3 being not-AWS.
Having a mature/battle tested phase 2 --> phase 3 should be a market advantage in the modern business landscape, but that's also a post-acquisition/exit phase, so none of the presumed target HN crowd care about it.
There are extensive articles and public knowledgebases on various technologies and architectures, but there is effectively nothing on AWS independence out there, even though almost every mature organization will need to face it.
That's totally non-obvious and certainly false in most cases. Cloud providers have huge economies of scale, which certainly makes it less expensive to use cloud if you factor in all your costs related to running it yourself.
This is like saying buying furniture is more expensive than making your own furniture because they have a profit margin. Even if you don't account for your own time, it will certainly be false as you'll pay your raw materials much more.
AWS would pass that difference to Amazon shareholders than passing them to you.
But, now that the emphasis seems to have shifted from growth-at-all-costs to where-can-we-cut-costs, I think there may be more organizations with a large enough server load to realize that they can do it cheaper (or even just get a different cloud provider to do it cheaper).
You see a lot of posters commenting that it isn't obvious that it's more expensive, but that's a large part of why I said I'm flabbergasted at how many people don't find it obvious.
It is obvious, someone may still choose to pay it because there's a natural tradeoff, but it's absolutely obvious.
I suspect most developers don't find it obvious because they've never dealt with actual servers themselves.
It's akin to arguing changing the oil on your vehicle is more expensive than paying someone to do it. No, it's more convenient, and that may make it worth the price.
Focusing just on what you're working on is great.
I was commenting on the expense rather than why you would (or wouldn't) want to use cloud.
That's pretty obvious but years ago you would have been downvoted into oblivion for writing that.
Even if he hired 2 FTEs in Hungary to maintain (which I doubt), it would eat 200k at most (probably much less), so they still saved 600k.
For 800k he could probably hire 10 more people, to improve development, sales, marketing, support, and that would be better investment instead of burning money on AWS.
As a senior software engineer, you can make maybe $35k a year, before taxes, if you are good. You can make it $50k if you are very good.
2 years ago I was making $23k (yearly, before taxes), before I moved to Canada and started working at Amazon for $170k usd.
Europe, especially the eastern parts of europe has an extremely cheap workforce.
For a mature business, that tradeoff probably isn't worth it, but for a startup, the opportunity cost is too high. Spending a lot on cost optimization doesn't make sense; you are better off spending on growing income.
Renting a dedicated server is very similar to renting a VM. It has a much higher provisioning time, usually hours to 1-2 days. And it takes some upfront cost or a time commitment. But it also has a lot more raw power. So if you don't require the flexibility VMs and the cloud provide they can be a very effective alternative.
They run datacenters for other datacenter companies. Most people have never heard of them because they're a $6B/year revenue beast that runs behind the scenes; but they also offer direct metal servers.
They have a US DC now. Though it only gives cloud servers. However even those are still way cheaper than any alternative.
The article spends a lot of time talking about how much lower their monthly bill is, but it says nothing about how much they spent to buy those 300 servers in the first place.
The monthly opex to house and power and cool those servers isn't _negligible_, but if you're doing back of napkin math comparing MRCs to Cloud you can just deduct the costs from their bandwidth charges that have been marked up 10,000%
They lease you the equipment. Yay.
I put a box of mine in a local datacenter, to their... endless confusion (not only do they not deal with random individuals that often, I actually read contracts so asked questions about why I had to prove my worker's comp payments and some other weird stuff that they removed). Monthly rental payments are about what I was paying in a range of cloud spend, but the system is far more capable, and I can do a lot more on it.
Admittedly this was hopefully only going to apply for a few years until they finished rearchitecting everything, but I doubt it would ever reduce back down to anywhere near the original costs.
The equation is also mirrored for something that started and grew on AWS though - going to bare metal means building up tooling and processes you didn't have before. The transition will be expensive in either direction.
I thought this was pretty weird since the original value proposition (AFAIR) was to reduce costs/head count. But everywhere I’ve worked that used AWS, had specialists employed to manage AWS. And I think the value proposition is more that it’s easier to find people who know AWS - because there are training providers and certifications - than people who know how to do everything themselves.
Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs.
[1] https://www2.deloitte.com/au/en/pages/economics/articles/eco...
I don't think this true for the vast majority of applications. It's probably true for the Youtube, Netflix, OneDrive, etc critical data path, but when your ownership is more like two dozen applications ranging from tiny (one consumer on some infrequent interval) to mid sized (under 100k requests a minute, for text, json, images, generated UI components), it's damn near impossible to beat from a pricing standpoint.
Going back to on prem servers, even doing kubernetes/docker orchestration would likely increase my current team's monthly capital expenditure from ~$14000 a month to somewhere in the realm of 50-60k a month. The sad thing about those prices are that our on prem offerings are very cheap compared to the last price I worked, a $BIGBANK that has multiple data centers in multiple countries. As an internal client with the same footprint we have now, $BIGBANK global infrastructure folks provisioning would take $90-110k and 6 to God even knows how many months depending on how fast you could get through the churn of governance, compliance, security, and controls.
Edit: just realized I forgot to mention I'm talking about serverless projects almost exclusively. That's important to the cost savings, because I agree that I'd rather host something in the 9 year old server in my basement and run my own reverse proxy than use EC2/EKS from a cost perspective.
The problem with your comment is that you insisted that they gave you enough detail to definitively determine that they were either lying or mistaken.
> There is no way this figure is accurate.
I divide this into two completely separate areas:
* advisory -- you advise your client on what is prudent in their circumstances,
* software development -- you produce software to client specifications, regardless of what they asked.
You can advise them all you want but then you do what they asked and that's it -- they are always right.
The real target customer IMO for these big public clouds are non-technical corporations whose primary revenue and personnel competencies aren't derived from mastery of software but primarily around other verticals like agriculture, manufacturing, etc. Insert Werner Vogels' criticism of the HBR article emphasizing focusing upon core competencies and outsourcing non-core competencies like IT (read: because technology is so important to every business at scale now).
which is less than 80k, yes.
> You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.
Please stop assuming stupid shit, ask for clarification if you're not sure what I included in it.
Racking 2 racks took us 2 days, one on site, on on configuration.
Patching is also included in that 10%. We track that time. If you're patching manually you already fucked up, cloud or no cloud. Only time spent actually patching is "difficult" customers that need paperwork for taking down even one server of HA pair.
And it's utterly ignorant to assume cloud somehow saves you on this point, you just have more time used dealing on security stuff on dev side instead of ops side. Hell, I dare to say average VM with auto-updates will be more secure than average container deployment. Or maybe that's our devs that didn't get kicked enough to set their container pipeline correctly...
You can make that amount normally in Eastern Europe when working on local market, you do not need to be good, average is enough. If you are good then you can make 60k+ USD. And if you are really good you can make easily 100k+ USD working for US based company remotely.
You need to know the tradeoff you are making but what I am saying is that, in most cases, it costs more to spend so much effort up front to be "cloud agnostic" than any benefits of maybe being able to switch cloud environment in a hurry.
Also sometimes you don't want to minimise the cost but use the best services or reduce the risks. Let's say I go full on Azure, but after a year or two GCP offers a much better product for my use case. I will have to ignore it because it's too expensive to switch. Or if I go full on GCP/Firebase but suddenly the pricing model changes and it becomes overpriced. I will have to eat the cost for some time.
However, if I'm careful to avoid vendor lock-ins from the start, it will probably cost a bit more on average, but the maximum cost is much lower and I don't risk being stuck with a bad service.
In practice I will for example avoid to write code specific to the proprietary Azure Blob Storage API, but use an S3 compatible object storage instead. I will also rather have an abstraction layer than using AWS SQS or GCP pub/sub directly.
I used to think this sort of thing was a valuable use of my time but I now have over $200k in annual revenue and we pay less that $5 a month in raw compute (and that's nearly all S3). My Co-founder is always worried about the cost of our AWS but so far we just haven't witnessed it.
My mandate has always been the price should be able to scale to 0. So no EC2, has made building some longer running tasks a bit more complicated, but that's just because we didn't know how to do it before we started.
Anyway, you have a tiny on-demand business, as I caveated above. You could run it just fine on a nano ec2 instance if you needed, which is $2-3/mo.
In the end you replaced sysadmin with DevOps and got up charged multiples.
The lowest common denominator across clouds is a VM. If you can run your applications and your databases on VMs, you can run them on any cloud AND on-premises. If you can't run CosmosDB or AWS Whatever on a VM, your choice of DBMS/ORM doesn't affect that you're tied to that cloud provider.
But... I'm curious: has anyone here made an attempt to go "cloud-agnostic" by targeting managed kubernetes offerings? (GKE, EKS, AKS).
Just curious how much pain I'd be causing myself if I ever attempted it ;-)
Run some of your micro-services on AWS and another on Azure etc. A given micro-service then might be highly dependent on AWS but the part of your application that uses those micro-services would not know anything about internals of AWS etc.
But hey if your goal is to bill a client as much as possible by claiming that you're using the best products from each cloud, this sounds like a GREAT business idea.
MinIO is AGPL-3 though or commercial license. Pretty sure using it as cache would be considered combined work?
I understand that there are many cases when all of that you mention is needed but for the vast majority of normal businesses having main and standby servers on Hetzner / OVH with on premises backup would cover all their needs like for ever with stellar performance. All at the fraction of the cost. Many business owners just do not know it as they're not technical and their tech stuff for some inexplicable reasons are all for doing "cool" cloud architectures at owner's expense. And of course that magical "cloud" word sells itself. So many times when I come to do something a new client I hear the owner say proudly - we are on a cloud. It takes some careful efforts to explain to them that they're just wasting money while not upsetting them in a first place.
And also of course, it allows you to go faster, which can have a huge value
Understanding your licensing and warranties is another huge cost that people don't take into account. We used to spend hours and hours figuring out if we could replace systems and what it would cost us.
Finally, you have to dispose of all that hardware when it gets too old or out of warranty. If you've never had to do that you probably have no idea how hard it is to do it correctly and so that it satisfies your SOC2 auditor.
All of these things (plus more) really add up. It's not just purchasing the components and installing them in racks. The management is probably even more expensive than the hardware.
And all of these problems go away with a cloud provider.
HDD replacement is trivial and some COLOs can even do it for you - and any good vendor will have a thing setup where your appliance automatically notifies them and a new drive is overnighted to your COLO. Sure - maybe a few times a year you drive out to swap a drive.
If you do not have good monitoring setup then that is entirely fixable - there are many stellar solutions these days out there. Hardware is easier than ever.
I mean, it is fungible enough that they can learn, but they'll probably make a lot of mistakes along the way and be less efficient.
Yes I exaggerate a bit, but I do so to make a point stand out.
- Disable password-based auth for ssh (require key-based auth) - Enable fail2ban or similar to slow down brute-force login attempts - Configure firewall - Install monitoring tools, antivirus, possibly backup daemons, etc - Setup a sane swapfile for your use case, and configure monitoring tools to alert when memory pressure gets too high - Setup disk mounts, configure monitoring tools to alert when disk space is low, and consider a cron job to automatically clean up tempfiles - Either set up automated updates (typically excluding kernel upgrades), or have a standard schedule for manually applying updates
...and probably other things that I'm forgetting because I'm a developer, and it has been years since I've been a sysadmin.
SEE also: https://levelup.gitconnected.com/how-we-reduced-our-annual-s...
One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture?
I thought it was strange they would put that picture on their site and not some GCI or something. I guess at least you know they are not lying about being a real place?
they've decent building, data center tier, so power/network redundancy, etc, etc
they're just cheap as fuck when it comes to people
I really should start a blog for some of the weird research I do sometimes.
That’s how just about everyone does backend API services with AWS. You’ll be shocked at how cheap, scalable and bullet-proof it can be.
Lambda, etc, just doesn’t scale well financially if you’re doing millions of requests per day.
The other drawback of lambda is the flip side of having the server opaquely managed by AWS. It is so opaque you can't debug anything. At one startup we had a weird connectivity issue from lambda to RDS but it was impossible to diagnose given the lack of access so it went on for months.
Had it been running in a VM, I could've diagnosed that within an hour with tcpdump and bpf et.al.
I assumed phase 3 was where AWS started giving you massive discounts because you represent so much business for them, but I've never been at a place big enough to swing that stick at them.
Could it be magical thinking based on marketing hype and autoscaling fantasies?
I think you should have multicloud strategy just to improve your bargaining position. They know when you're locked in.
A couple of others spring to mind:
- if you're not regularly testing your backups, you don't have backups
- monitor SSL certs for expiry. It's amazing how many outages I've seen because this was missed.
- are you allowing direct connection to your server from the internet, or do you want to go through a bastion host. Do you need 2FA on this, say because you need to meet ISO27001.
- do all the above through CM (like Ansible or Puppet) for obvious reasons.
You don't hire a dedicated sys admin because they know some voodoo magic developers don't, you hire a sys admin because they have the _TIME_ to watch for security updates. And you _PAY_ a company like RedHat to ease that burden.
A dedicated sysadmin will typically do a better job than a non-dedicated developer, but if that developer cannot do a passable job, that's a gap in their skillset.
I may not be a network engineer, but I damned well understand DNS, MPLS, networking, gateways, and the like. Any developer who doesn't has gaps.
Some examples which come to mind:
* Comparing S3 to the on-premise tape system but ignoring the fact that it involved an expensive tape robot, DR had access times measured in days, etc.
* Comparing S3 to on-premise storage, ignoring the difference in redundancy, forcing users to handle bitrot at the application level, and the procurement process meaning that when they ran out of storage it took months of telling people they couldn't allocate more.
* Saying their devops engineer cost twice as much as their sysadmins (true) but then when you look the devops engineer is using automation and managing literally a hundred times more systems than the “cheaper” ops team.
* Saying their cost to run a VM was cheaper than EC2, which true if you looked only at the instance but not when you calculated how much they were spending on underutilized VM hosts, power / HVAC, facilities people, etc.
It's totally possible to beat a major cloud provider on costs[1] but you usually need to be operating at fairly large scale to even approach the break-even point. This is especially true when you have regulatory or policy requirements for things like security and you include the cost of monitoring all of the things which fall under the cloud provider's responsibility — management networks, firmware version management, robust logging and IAM, etc. are all easy to accidentally exclude when making comparisons.
1. Network egress as the most obvious area to attack
There are hosting providers like Hetzner that will rent you bare metal machines so you don't have to deal with actual hardware, their networking is basically unmetered (last I checked), they will also sell you spot VMs, blob storage service, etc. The pricing difference is in integer multiples and much less "gotchas" or lock-in (they are usually dumbed down).
They won't sell you fancy SAAS - but the ammount of fiddling required to get most of AWS services working in practice and debugging it - you're not very far off from using the OSS solution they are reselling - chances are your devops/sysadmin team can do that as well.
And there's plenty cloud agnostic SAAS as well.
I mean if you're a big corp that can accept the inefficiency for the sake of having one bill to pay at the end of the month, SLA you can use to cover your ass to higher ups and have the "nobody got fired for picking AWS" mentality I get it. But for startups and small companies ? All this "magical simplicity" that AWS/Azure/GCP is supposedly brining to the table - I'm not seeing it. Real redundancy (across AZ) is still really hard and requires extensive planning - no magic there. Dynamic scaling "savings" is often offset by being multiples more expensive to begin with and I have to deal with warmup issues, and the pricing is sneaky/unpredictable, easy to fuck up. Lambda architecture is pushing towards distributed microservices by default - the hardest thing to debug in practice.
So end of the day, I'm forced to work around arbitrary pricing models rather then technical limitations, debugging opaque black box services, not being "big enough" to get actual support, overly generic/complex systems built as "one size fits all" solutions, or get sold in to practically abandoned "service" because it's available etc.
And it's not like being first party service is any indication of quality - as soon as you move off of the most used stuff you get to see some really shit services in my experience.
If all you need is bare metal, you don't care about network quality, etc. then, yes, you have a lot of options. The reason why you use something like AWS/Azure/GCP/etc. is that you're looking for more than basic infrastructure services and don't want to have to manage relationships and support infrastructure for multiple providers.
This even covers basic things: for example, look at all of the reports of Hetzner customers losing data due to something like a drive failure. It's not like there isn't a way to deal with that yourself but now you're in the business of monitoring the drives, setting up software RAID and testing/benchmarking it to make sure your configuration is robust, scheduling downtime to reflash servers with storage updates following some process which depends on the hardware & software combo you're using, etc. Your competitor using AWS isn't paying their sysadmins to do that because they do it for you, and it's completely transparent.
Similarly, the major cloud providers can live migrate VMs off of failing hosts without your awareness — another thing you have to build and operate yourself on bare metal which isn't trivial to get right. When Spectre et al. came out, the major cloud providers' customers were all patched on the day of the annoucement. Bare metal hosting users had to schedule downtime, roll patches, and make sure nothing broke.
Again, I'm not saying you can't do that yourself — having done so myself at moderate scale since the turn of the century — but just not to underestimate the direct costs to provide equivalent service, as well as the opportunity costs of having your expensive staff time locked into infrastructure O&M. Business needs change and it's really nice to be able to handle curveballs – say you went all-in on Hetzner, but now you need a server outside of Central Europe/Virginia and they want Windows + SQL Server instead of your usual Linux and they need it next week but only for 6 months … how much do you need to invest getting that set up versus someone in AWS/Azure/GCP having it done in 15 minutes with infrastructure provably meeting the same standards and configuration as you use in your primary locations?
> They won't sell you fancy SAAS - but the ammount of fiddling required to get most of AWS services working in practice and debugging it - you're not very far off from using the OSS solution they are reselling - chances are your devops/sysadmin team can do that as well.
This is the opposite of my experience: with a few exceptions, you're looking at significantly more work to build an equivalent service yourself, especially if you need to worry about reliability, security, etc. That last part is important if you need to be able to make strong statements about who has access to data, whether logging can be tampered with, etc. — those are all things you _can_ setup yourself but the cost of doing so is greater than years of your usage until you're at a pretty large scale.
As a simple example, most security benchmarks require you to enable storage encryption. That protects against data leaking when drives are replaced or rotated out of service and if you do it right can also prevent data from being leaked when someone compromises part of your infrastructure or even a sysadmin. This is a turn-key service on AWS – at most check a box for the basic form, or configure KMS policies for whatever you need in the more complex cases, and it's the same set of tools for EC2 servers, databases, S3, EFS, etc. as well as your applications using KMS directly.
If you're using Hetzner, here's what they say: “We recommend that you not use server-side encryption.”
You can configure luks, etc. yourself but now you have a ton of work to configure and validate, and the problem is inherently harder because you now need to deal with things like access control & auditing yourself. If you need to demonstrate this to an auditor, you have to convince them that you have sufficient controls to limit an attacker or rogue sysadmin whereas if you're using AWS/Azure/GCP/etc. you can just point them at the platform docs and show that you configured the intended policy.
Repeat for things like immutable storage: if you need to care about that, the cost of being able to prove that your storage is protected is going to pay for exabytes of Glacier storage.
It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself.
If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.
It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”.
It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it.
Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data.
But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.
Try uploading 5PB of content and downloading it that day at 1M RPS.
Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the service on vs deploying, and maintaining a vault on bare machines.
It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services.
People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest.
You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.
In your case, you are not exchanging 2 hours of a developer for a higher bill. In my experience you will get a higher bill and 5 hours of a cloud expert, and probably a solution that for every change or problem, you will have to call this expert as no one know what to do.
Trying to use a thing that accomplishes the a similaroutcome but in a totally different way requiring a totally different skillset, it's not surprising that you found yourself needing someone who does have that skillset to assist.
A Linux server guy trying to manage Windows servers will probably need help from a Windows server guy. A car driver trying to fly a Cessna somewhere will probably need help from a pilot.
The massive number of proprietary newer services are junk, with the exception of some of the ML services. The ML services are sometimes helpful, but I expect they'll quickly be obsoleted.
That expert may not know what to do either, or... will contradict earlier folks and say "you need to redo all of this pile..." and you'll be stuck.
I've no doubt there are some workloads that really do require a level of complexity that various cloud systems offer. Much of what I've seen doesn't require it, but once someone starts down that road, they're 'justified' in learning more and tying more stuff to the cloud provider's way of doing things.
I cut my teeth setting up servers back in the 90s, and that you can spin up 'full' servers (thinking various shades of VPS) in a few seconds is crazy great. But we're somehow now 'past' that... and we have mountains more complexity to consider.
> spend way more time with Lambda/Cloud formation/ECS
As you said, it's not your expertise. No one said AWS could be used well by non-experts. The point isn't the learning curve, but the terminal velocity once you get there.
Anyone looking to hire an ML guy for 250-500$/hr flat fee, hourly rate, no taxes, no healthcare, cancel any time... get in touch!
For that money I'll gladly bark like a dog, walk on all fours and fetch your slippers.
In the US salaries are a lot higher so in that context I can see 300-500$/day be realistic.
Really surprised me… Always made me wonder if all that money was being spent well
You should see how much lawyers cost versus how much lawyers earn. The sticker price for SWEs isn't as well-advertised, but it's how businesses work.
> developer costs >> infrastructure costs
What I've expected alot is the businesses are fundamentally struggling to find sysadmins therefore paying for infrastructure costs are there only option. And now more and more sysadmins are "AWS devops" who literally only know how to manage a AWS stack with many struggling even basic stuff such as managing to figure out how many resources they will need since more and more stuff are autoscaled.
They can hire devs but can't hire sys admins. Hell some of the time they can't even hire devs.
For comparison's sake, the net salary for a software developer in Latvia is roughly between 950 and 2800 euros per month: https://www.algas.lv/en/salaryinfo/information-technology/pr...
Given the above number, some calculations of the gross salary land at between 1350 and 4000 euros per month: https://mansjumis.lv/darba-algas-kalkulators
So essentially one of your developer days would represent weeks of work for a Latvian developer (depending on additional expenses/overhead), which I find curious.
Developer costs are still likely to be higher than infrastructure costs, but not nearly by as much, which might explain why I've seen a lot of on-prem deployments locally, and people shying away from going all in with cloud technologies.
I wonder whether these differences are more pronounced for other countries where developers can be comparatively less expensive (at least in relation to using the cloud and/or hardware), like India, Russia and elsewhere, how much the development culture differs. Also how many would opt for alternative providers like Hetzner, DigitalOcean, Vultr, Scaleway, OVH and so on...
That's what we said at my first Kleiner company. All the time.
Pretty soon we had a $100k/mo server bill.
We were just webhosting. There was no need for any of it.
You should be thinking of dev time as in _investment_ not a cost. Invest developer time into high value activities like creating robust services you own. Not into low value activities like coding against a dozen different proprietary, Amazon owned APIs that can change, go away, and become more expensive any time they feel like it.
For the use-cases I was involved in (image gallery hosting, 2-3Gbps traffic), costs looked like:
AWS: $400/mo for app servers, $40,000/mo for CDN bandwidth
Bare metal: $400/mo for app servers, $400/mo for a couple of caching reverse-proxies, each with an unmetered gigabit connection
When you’re dealing with anything more bandwidth-heavy than HTML+JS+CSS, having a handful of bare-metal Varnish instances is _orders of magnitude_ cheaper than the cloud…
(We have also looked at dedicated CDN providers like cloudflare, but we’re in that sweet spot where we’re too large for the publicly advertised plans, and the “call us for a custom contract” plans want a minimum spend around $3,000/mo :/ )
MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive.
On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent.
On the other, that's expensive but running your own is complex if you're not setup to do it already.
MinIO is but one part of a data storage puzzle. Though a very important one.
Right, you understand, but maybe you don't really understand?
If you work at a company large enough, and with enough DevOps, NetOps, Ops people to go it alone, and can get favorable lease terms on hardware, then by all means, host it yourself! Paying all of those people and also paying for AWS seems odd, for sure.
But if you work at a company without a deep bench in various Ops groups, then good news! You don't have to hire them! Instead you can rely on Amazon's deep bench, and focus your salary budget on developers rather than Ops people.
There are scales at which, even not counting enormous budget expenditures on staff rather than managed services, AWS doesn't make sense. And if you're just using EC2, you'll hit those numbers more quickly than if you make good use of things like Lambdas and SNS/SQS and Dynamo and so on. The cost savings can be enormously staggering, and the more you lean into AWS managed services, the more that's true.
We see the edge cases on HN, the rare instances of someone being kicked off of somewhere, but for most people, the uptime and long-term reliability of AWS managed services is higher than trying to go it alone.
For most businesses the savings in money and time are worth the risks you pointed out here.
If those risks are too high for you (and fair enough if they are), you’ll pay one way or another to avoid them.
Continuing what you said, Google uses macOS and Windows for client machines. That is ok because if Apple or Microsoft were to decide to cut off Google altogether, Google would have bigger problems.
However, I think from a business point of view, this is similar to how so many businesses use cloudinary. Cloudinary is a software as a service company that provides cloud based image and video management services. What if cloudinary kicks you out? There is a lot of custom code out there to interface with cloudinary API. Where do we draw the line between using Cloudinary is ok but Amazon aws lambda is not?
To be fair to the Stack Overflow guys, AWS had fewer offerings when SO was built, so the comparison wasn't the same then as it is now.
To run an equivalent stack myself would require significant effort (upfront and ongoing) and I'm not sure I could get the costs that low. Another great aspect of my stack is it's all scales on it's own without any intervention on my part.
What you are comfortable using is a sliding scale with some people thinking if you aren't hosting it in your own datacenter then you are crazy all the way up to "let's just use Firebase" (or similar). For me I've found that buying in fully can be very enjoyable and the risk is rather low (for me).
> Ok so you save money on the monthly bill...
Sounds like you do understand it.
Who cares if it's proprietary? You're either locked in to a proprietary stack or you have to go out of your way to cobble together a FOSS stack on bare metal, in which case you're essentially locked into that.
> Ok so you save money on the monthly bill but what happens when Amazon decides they want to enter your market?
If AWS helps me get to market faster and cheaper than the alternatives, I don't particularly care what the rest of Amazon does.
> What happens if they decide your service is too controversial and they kick you off?
Normal businesses don't have to worry about this. I've honestly never heard of anyone getting kicked off of AWS unless they were doing something extremely sketchy.
> If you were deploying your own services to EC2 instead of using AWS's own services you could at least setup shop elsewhere with just a bit of work.
True, if this is truly all you're doing, you can probably find better options elsewhere at this point. But pretty soon, you're going to be interesting in managed solutions for other things that are important but not core to your business's value proposition, things like managed databases and logs and auditing and object storage and containers and data lakes.
If you don't want to use the cloud services that actually make the cloud convenient to use, then yeah, I don't know why you'd use the cloud.
But you could also run, say, Kubernetes on EKS and Fargate and still have a relatively easy time porting your software elsewhere in the future. And the other cloud providers have their own versions of things like S3 and Lambda and RDS. It's never easy to port production software to another provider, but it's not like it's impossible. And it's also an extremely unlikely scenario. I'd love to see some statistics on how many companies actually move off of their cloud providers; it's got to be one of the stickiest businesses out there.
> To me it is antithetical to building a sustainable product. People just hoping their startup gets bought and then it is someone else's problem.
At this point, just about every Fortune 500 company is using AWS, Azure, or Google Cloud in some capacity. For most companies, building out your own infrastructure is not part of your value proposition. So why not just pay somebody else to do the undifferentiated heavy lifting of figuring out how to run code in a generic way?
That idea was not expressed in the comment that you replied to. The comment you replied to was in response to a strong claim that ignored the existence of economies of scale, and therefore concluded that it's so obvious that cloud providers must always be more expensive than self-hosting.
Dell can bulk order parts and has a strong negotiating stance, which makes the hardware __CHEAPER FOR THEM__.
It says nothing about how much the market can bear for Dell. And in fact, companies use Dell because of the _guarantees_, not the cost. At any point in time, Dell will replace hardware with no questions asked. They can do this because of the margins between what it costs them and what it costs you.
There are benefits to cloud, cost aint one of them.
edit:
Also, this shit?
> That's just not how this works.
It never makes you sound right.
AWS's economies of scale have as much to do with its pricing as the phases of the moon, and citing them as a reason why AWS could be cheaper than self-hosting is pretty ignorant in itself.
It is not obvious that growing your own wheat will always be cheaper than buying bread.
Whether economies of scale, or the efficiency of managed services at scale, or use-based loss leaders, there are many ways in which AWS services could be, and often are, cheaper than self-hosting. Of course, not always.
Your parent comment wasn't saying that AWS is necessarily cheaper than running hardware yourself. It was saying that AWS is not necessarily not cheaper.
You're responding to someone who said "can," not "must."
What they said was "there's no requirement that cloud costs more than traditional hosting. Their scale allows them lower fundamental costs. They could outcompete traditional hosting on cost if they wanted to."
There's no misunderstanding of how comapnies work in what parent poster said; in reality they are 100% correct, and this is the path that CloudFlare is beginning to follow with services like workers and R2.
It's a CDN, not a filestore. CDNs mean hundreds of locations.
AWS may be able to charge a surcharge for being the biggest and best-known cloud offering that most people have expertise with, raising the de facto cost of switching.
If you're using dumb boxes to deploy something like a k8s cluster why do you care about drive failure ? You rent out an extra machine to account for the downtime (and peaks) and you're still well below AWS infra cost.
>Similarly, the major cloud providers can live migrate VMs off of failing hosts without your awareness — another thing you have to build and operate yourself on bare metal which isn't trivial to get right. When Spectre et al. came out, the major cloud providers' customers were all patched on the day of the annoucement. Bare metal hosting users had to schedule downtime, roll patches, and make sure nothing broke.
Again cattle not pets.
>say you went all-in on Hetzner
But that's what I am saying - going "all in on Hetzner" doesn't really mean much if you're using dumb blocks available elsewhere (plain container/VM hosting, network object storage).
>This is the opposite of my experience: with a few exceptions, you're looking at significantly more work to build an equivalent service yourself, especially if you need to worry about reliability, security, etc. That last part is important if you need to be able to make strong statements about who has access to data, whether logging can be tampered with, etc. — those are all things you _can_ setup yourself but the cost of doing so is greater than years of your usage until you're at a pretty large scale.
That's the problem I guess - I haven't built a on-prem system in 10 years now, maybe I forgot the pain points, maybe the automation tools really don't work outside of big cloud providers.
But right now all I'm seeing is insane margins on cloud services and we still have a bunch of devops that are busting their ass off to monkeypatch everything together.
k8s doesn't mean you don't care about data integrity or performance, and if you don't build and debug more automation yourself dealing with failures creates more manual labor recovering from them. Again, you can definitely do that yourself but it adds time commitments which blow the cost savings away unless you are running at least hundreds servers to have the savings exceed the staff cost.
> Again cattle not pets.
That's an orthogonal concern — neither environment will prevent you from treating servers as pets, and not having things like auto-scaling or higher-level service abstractions definitely encourages pet servers because the overhead of managing a full server and improving utilization adds up.
> But that's what I am saying - going "all in on Hetzner" doesn't really mean much if you're using dumb blocks available elsewhere (plain container/VM hosting, network object storage).
Actually, it does for the reasons I mentioned before. If you care about reliability, performance, security, etc. you're not just picking any random hoster around the world — you have to spend time selecting vendors, signing contracts, getting set up with their environment and tools, etc. Again, that's totally possible and there's a break-even point where those costs are canceled out by lower service costs but it's higher than people typically assume.
> (plain container/VM hosting, network object storage)
This is true but also means you need to factor in the cost of greater maintenance commitments and lower productivity.
I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.
Is the issue big data or small machine?
Memory size of the largest AWS/GCP/... instances is a good indicator for small machine.
It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X.
And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.
3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors.
Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess
This is a reference to the country Brazil [0], which centers on...
This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk
[0] https://www.workhuman.com/blog/a-brief-history-of-bureaucrac...
It's obvious that baking your own bread is a bad idea if you're okay with what you can buy from the competitive market. It's equally obvious that baking specialty, artisanal bread (and buying the commodity ingredients, such as flour, salt, and yeast) is cheaper than going around to several grocery stores to try to find the exact kind of loaf you want.
AWS is in an oligopoly, so it should be obvious that its efficiencies (in this case, its economies of scale) are not passed on to the customer. The strong, clearly expressed claim was fine.
Our AWS DevOps team isn't any smaller than they'd be if they were running a handful of servers in a colocation facility.
For all new dev, which eventually including migrated some functionality from that initial workload, we used AWS-specific managed services like Lambda, Dynamo, and SNS/SQS, which required MUCH less effort from DevOps. Two of the existing DevOps people were able to handle everything the so-called serverless development required in addition to their previous normal work.
I mean, the point of managed services is that the vendor takes care of managing them, so you don't need to deal with upgrades, security patches, etc, so if you're still doing just as much DevOps work, it's likely you're not taking good advantage of managed services!
So we had some initial effort to automate and standardize CloudFormation configuration, and then developers were able to launch new code bringing in millions in revenue with very minimal oversight from DevOps.
The advantage there, though, was mostly from "serverless" and managed services, not just EC2 vs datacenter. We still had a DBA for the Postgres stuff, we just didn't need one for the new Dynamo stuff.
You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
Most automation that you know of as “devops tools” are borne from sysadmins.
Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too.
It is revisionist to think sysadmins were not automating their jobs.
Old school sysadmins used to know C, bash and Perl.
New school devops just traded that for a handful of DSLs and Python.
I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)
The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.
I’m with you on most of your post, but this is straight up wrong.
That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there.
The growth of perl, for example, was in great part from the sysadmin community automating everything.
But nowadays if you want to manage your own metal then you can use those tools as well. You can still run hypervisors VMs on your own servers. You're just not paying Amazon 6x the price of what it would cost you, and you're not relegated to low-performing compute, memory, and IO elements that Amazon standardized on.
At my last company, providing SaaS for the education market, moving from a datacenter to AWS saved almost 70% year-over-year. In the datacenter, we ran machines to cover our peak load, which only happened a few times a year. In AWS, we scaled way down and auto-scaled up during those peak weeks.
Is the entire education market "exceptionally bursty?" I suppose it could be considered so. Bar exams and midterms and finals certainly don't happen every week.
I'll wait for people to tell me how AWS was the wrong solution, and how we did everything wrong before that, but the bottom line is we saved a lot of money, accelerated our development schedule by building all new functionality using the so-called "serverless" stack, and succeeded so well during the pandemic that another company acquired us... which is why it's my former company.
I don't know how big your peak was compared to your average load, but if it was anything like that, serverless was a great call. Buying enough hardware for the bar exam and idling it 99.5% of the time sounds incredibly wasteful. However, this is an exceptionally bursty workload.
Most services do not have this level of traffic variability in such a way that a CDN can't handle it for you. 10x peak to trough variability (after your CDN) is fairly common, but still considered bursty, and in that case, AWS serverless still doesn't look great compared to DO droplets. Many services have daily or weekly cycles (without events like the bar exam), and run analytics workloads in their off hours.
Cloudflare is a CDN, which, like all of its competitors, is a lot cheaper than doing it yourself because CDNs are a competitive market. There are at least 10 (if not more) CDN providers out there that have footprints in several thousand datacenters, and the product is completely undifferentiated. In a competitive market, you have to charge related to your costs, because your competitors will cut prices to their costs.
In other words, in a competitive market, economies of scale get passed on to consumers. In monopoly and oligopoly markets, that is not the case.
Object stores are becoming a competitive market, with R2, B3, and Wasabi entering the game, but buying your own machines and using Ceph is still cheaper. You still pay for "value adds" like not having to manage the machines yourself.
A CDN is a thing where you can put things, and request them by web, and have them geo-distributed.
R2 is actually a CDN. So is S3. Yes, I'm aware, the vending companies also sell different products called CDNs. Nonetheless, R2 and S3 (unlike B2) are CDNs. You can make distributed hits to it across the planet to addresses you don't control. You can hit it directly from the web, and it will serve to you from a local node you never knew anything about. Its goals are speed and locality. You can look up programmers debating whether to use CloudFront or S3 as their CDN based on time to invalidate vs actual speed and cost control.
Attempting to describe it as "an object store instead of a CDN" is kind of missing the point. It's both. It's also a webserver. It's also a backup system. It's many things.
You cannot replicate R2 on a single machine. R2 delivers locality and redundancy. Arguing over its title is irrelevant; the reason to point out that it is a CDN is to establish requirements for replacing it with a homebrew solution.
The lowest practical cost for reliable machines is arguably VPSes at around two dollars a month. Those VPSes tend to give you 100-250 megabits of traffic monthly with around 1gig of storage. Assuming two for redundancy, you'll get maybe 500 megabits for about $4 a month. (I personally would not feel safe with two, but we're talking about cost cutting.)
That same 1g of storage on R2 falls in the free forever tier.
To eat up the $4 a month in class B actions, you will first need to consume the 10 million free events, then spend 36 cents per further million. This is a further 11.1 million, or 21.1 million requests per month.
Most low-tier VPSes will struggle around 10 requests per second using NGINX, because their disks and memory are terribly over-burdened. There are 86,400 seconds in a day, suggesting you'll get around 860,000 requests per day if you don't consider time shaping. Considering time shaping - you don't have full requests coming in at every time zone - you'll most likely have closer to 350,000 realistic. This means that your two VPSes will, just barely, be able to close the same amount of traffic.
Very, very slowly. You'll be lucky to get 200ms responses locally, and to get VPSes that cheap, you'll need one in the Netherlands in the burden corridor, and one in the United States, most likely in Texas or Georgia.
Most of your customers will now be getting 300ms round trips.
Why? Because you wanted to save against four dollars a month. This isn't even enough money to biggie-size a value meal.
All so that you could make a two-computer CDN on VPSes.
In the meantime, I've worked at two unicorns and neither of them had traffic anywhere near this large.
You can also do the math with an unmetered 1u. Ten megabit unmetered with 95% availability typically goes for about $19 a month, or $30 a month with a cheap 1u attached. That's about 3.2t of traffic a month.
As soon as you try to put actual specific numbers to it, and calculate actual specific costs, this just falls apart.
.
> There are at least 10 (if not more) CDN providers out there that have footprints in several thousand datacenters
There's more than two thousand of them from the United States alone. Akamai's CDN has over 400,000 nodes at over 50,000 POPs.
I can think of more than 40 in-house just from the US of that size (eg Netflix, Steam.)
What's your point?
.
> R2, B3, and Wasabi entering the game
Wasabi has been around since 2015. B2 isn't really the same kind of thing as R2 or S3.
.
> because CDNs are a competitive market ... Object stores are becoming a competitive market
Object stores have been around decades longer than CDNs.
.
> but buying your own machines and using Ceph is still cheaper.
Not really, no. I just did the numbers, and they did not pan out the way you claim.
Buying your own machines? You're lucky to find a 1u for less than $1500, and colo for a 1u is typically at least $40/mo. That covers 30 million writes and 300 million reads a month for six months at R2. (For scale, the Washington Post gets 86 million uniques a month, so if WaPo is doing four hits per load, which seems high if you sprite your images, then you're talking about larger than WaPo traffic. Spriting your images is a lot less work than building a CDN from scratch.)
The R2 cost for Washington Post is about $230 a month. You ... you really want to fight a cost like that, at a scale like that?
At that price, you're saving about $190 a month on bandwidth, so disregarding the cost of upkeep, it takes about eight months of WaPo tier traffic per slab to break even on each $1,500 slab.
In order to compare against R2 you need multiple locations (redundancy is critical to safety,) so for this to make sense, you need to be able to deliver multiple Washington Posts of traffic. If you can do that, well, I can make $230 a month not a big deal to you. Let's talk.
If you're trying to save money against four dollars a month, which I think was the actual realistic price here, I'd warrant it would be a good idea to look at your salary, convert it to hourly, and figure out how many years you'd have to run that savings to recoup your first hour of thinking about it. (Heck, I would even say that about $230 a month - after all, if you make $125k that's about an hour of your time, and I doubt even a very talented engineer could set up a distributed R2 replacement, including buying and siting machines, that fast.)
The vertical step cost is larger than the total cost in almost any practical system.
In order for the self-hosting costs to pan out against a distributed system like R2, you need to be looking at weekly terabytes of traffic and hundreds of nodes.
Respectfully, no, doing this yourself isn't actually cheaper. Not even close. You'll need a cost outlay to convince me, and it'll have to explain why the one I just did is wrong.
A realistic "buy your own hardware" CDN^H^H^Hobject store doesn't have barebones 1us with tiny drives. A realistic object store has a couple attached JBODs. (But you know, this is some individual's personal file server, which has been lionized into being a "competitor to R2" by ignoring almost everything that R2 actually does.)
And of course, the second you actually pull that cost outlay off, I'm just going to start talking about how much cheaper it would be to build your own datacenter, and then your own internet. Because as long as we're not focusing on there being no business need to justify the outlay, the larger you build, the better.
But it's "object hosting," not an R2 competitor, and you want to self-build. So maybe we just knock this down to two nodes. And then it's only $4,200 up front and $1,700 of traffic a month to break even, plus engineer costs for building, maintaining, and debugging it.
(And honestly, if you're just using it for storage, why not just set up FTP or RCP?)
So I googled "how much traffic should your website get." I got hubspot. https://blog.hubspot.com/blog/tabid/6307/bid/5092/how-many-v...
That article - and I'm not entirely certain that authoritative even exists here if someone defines things, which I haven't - but that article is probably better to listen to than me, at least.
They claim they surveyed about 400 "traffic analysts." There's a common sense line here of "they work for people who can afford entire staff for a job like that," so I guess I assume by default that these folks mostly work for very large sites, and the remainder merely for large sites.
Like I don't think any private pages with 200 hits a month have a "traffic analyst," you know?
Sure enough, even in that survey, 46% of them were in the 1k-15k a month bucket, and less than half a percent were in the 10m+ bucket. And remember, we needed 300 million per server-month to break even.
My strongly held, evidence based opinion is that the scale at which CDNs are cost-competable is fundamentally an irrelevant scale to all but the very largest of sites, in the way that a local single-location restaurant should not be looking into making its own flatware or furniture to control overhead.
And none of this counts the engineer costs.
My opinion is that in order to cost compete against CDNs, you need to have hundreds of large websites as customers.
My opinion is that viewing R2 and S3 as "just a place to store files" is akin to viewing a car as a place to listen to music - not even the primary use case. S3 has had http access from day one.
And yes, you can make a far cheaper radio than a car. But in general, if you try to sell the result to car purchasers, what I believe you will find is that you've misunderstood what the market is attempting to purchase.
My experience is that 90% of the people I know using S3 are using it for web fronting directly, or to back CloudFront.
.
Null hypotheses are important. So is actually having the ability to use what you build.
Imagine how cheap I could make the CDN if you just keep adding zeroes to the userbase all day, right? But it would be a hollow demonstration, because I can't actually deliver the customers to justify it.
The reason I've never heard of Ceph is that nobody I know uses it. Nobody I know uses it because nobody I know would invest this much effort into getting rid of a couple dollar a month datastore.
I know thousands of people who are making SAASes and other sites.
I am of the opinion that recreating object stores and CDNs is one of the smallest levers that you can think about turning, unless you're some starkly uncommon kind of site like a video host.
R2 is not a CDN. S3 is not a CDN. Both are object stores. They store objects with some redundancy in a local area. CDNs are globally distributed. Object stores usually aren't. You can use an object store for backups and for holding webpages, but that doesn't make them backup services or webservers either. Using only S3 (without deduplication or compression) for backups is ridiculously expensive, and using it as a webserver limits you to a static site.
In terms of the costs, if you are small, the free tiers (and cloud platforms in general) are great. If you are at the point where you are paying $5-10k/month, you are almost certainly overpaying by using cloud services.
Also, you don't even need an object store for a small app. You can use a server with a hard drive if you have a lot of data to store (plus backups and a redundant server in another place for HA). Put a CDN in front of it - a real CDN - and don't worry about it. Pay the monthly cost for the CDN. Also pay for your backups.
It's fine if you want to pay AWS so that you can focus on things other than efficiency. Most SaaS companies make that trade. But you are paying for it.
> R2 is actually a CDN. So is S3. Yes, I'm aware, the vending companies also sell different products called CDNs. Nonetheless, R2 and S3 (unlike B2) are CDNs. You can make distributed hits to it across the planet to addresses you don't control. You can hit it directly from the web, and it will serve to you from a local node you never knew anything about. Its goals are speed and locality. You can look up programmers debating whether to use CloudFront or S3 as their CDN based on time to invalidate vs actual speed and cost control.
S3 does not serve you from a local node (by my understanding of 'local'), it always goes to the source AWS region. It can take advantage of availability zones but it is still regional. It is missing the "D" part of "CDN". Perhaps this argument is just semantics, but the conventionally held understanding of "CDN" is that it's much much more widely geo-distributed. Otherwise you're right that they both provide a storage function.
> Wasabi has been around since 2015. B2 isn't really the same kind of thing as R2 or S3.
B2 added an S3-compatible API, so they're now directly comparable.
Other recent entrants in this market are IDrive e2 (4 USD/mo/TB) and Storj (4 USD/mo/TB). It's definitely a competitive market aside from the anti-competitive pressures that keep people using S3 anyway, such as zero-rated transfer within an AWS region to your other AWS resources.
Storj is the closest competitive object-store to a CDN in that it is possible to get data from the nearest geolocated node, although you'll (A) have to use the non-S3-compatible version of their API to do it, and (B) tolerate a certain degree of crypto-adjacency,
We frequently see links at HN that fall over and die after they're linked. On the one hand, that's usually because they've misconfigured a personal blog and lack caching. On the other hand, some services just weren't ready for HN traffic and failed to scale up. Which.... they wouldn't, were they running on a cloud provider like AWS.
It's possible for any service to be surprisingly bursty, but I suspect more than you think are bursty on a regular basis.
P.S. DO Droplets are Digital Ocean's response to AWS Lambda, and are still "cloud."
The core point, that you appear to have missed in the actual math, is that you will never personally be at that point.
The pay increased, the titles and tools changed but the mentality didn’t.
People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are.
Sounds like they were automating though.
“Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.
Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc.
That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.
The idea that I would need to automate the installation of an OS and applications on a fleet of machines was never contemplated because it didn’t make sense. I had one machine and OS upgrades arrived in the post every 6 months or so - on cartridge tape.
It was about need, not competence.
https://www.linkedin.com/in/mitchellh/
Or are you saying that he's not the author? Commits seem to suggest otherwise.
commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b
Author: Mitchell Hashimoto <mitchell.hashimoto@gmail.com>
Date: Fri May 23 11:03:38 2014 -0700
config: better error message
commit 089822a36f0a5e4c583ab401b7496b48bfa31d65
Author: Mitchell Hashimoto <mitchell.hashimoto@gmail.com>
Date: Fri May 23 10:52:19 2014 -0700
config: some comments
commit ec3f72703c82f50787ef86498d856a5798900ad8
Author: Mitchell Hashimoto <mitchell.hashimoto@gmail.com>
Date: Thu May 22 16:56:28 2014 -0700
Initial work on config
commit 649cf336e86da10910f6e50ea3cfaad507a2e336
Author: Mitchell Hashimoto <mitchell.hashimoto@gmail.com>
Date: Wed May 21 16:28:53 2014 -0700
Initial commitAn operations engineer and a sysadmin are two entirely different roles.
A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.
Sysadmins automate their work. Operations Engineer is just another title for sysadmin.
Systems Engineer is also another title for the same thing.
Platform Engineering (depending on where you are) is yet another variation.
If you're responsible for the operating system, monitoring, clustering, hardware etc; then you're firmly in "sysadmin" land.
I worked in the mid-00s and it wasn’t a lot better in the dev space. People passing USB keys to each other was pretty common, SVN and CVS were around but there was a lot of developer code outside of it and peer review was going to someone’s desk and walking through the new code, nothing at all like what we have today for processing change requests.
You’re talking about systems administration when it was going from pets to cattle, the primordial period where people were automating but not applying software development practices on themselves yet because even the software development practices weren’t well defined.
Sysadmin was always lagging 5y behind development w.r.t. programming practises. The same is true today of devops.
tell me how many terraform repositories have unit tests or infra scripts for that matter which would be much simpler.
I did know some fantastic sysadmins at a local university around that time period. They had huge labs of Alphas and Suns and automated everything! Unfortunately, those folks didn't go into industry.
The closest I’ve seen to devops nirvana has been NixOS, but its frustratingly divergent terminology and file formats make it very hard to cross the chasm.
Pedantry is requisite.
So do janitors. What's your point?
> Operations Engineer is just another title for sysadmin.
It really isn't. That makes as much sense as claiming that DevOps is just a sysadmin that's also a developer.
Many "operations engineers" are actually sysadmins. Many "devops" are really sysadmins also doing development.