2M users but no money in the bank(exercism.org) |
2M users but no money in the bank(exercism.org) |
Huh?
we need a lot more Byung-Chul-Han somehow
I hear the pain-points about the donation-driven model not rising to the task, but I do wonder how prominent that particular call to action is on the site. On one hand you don't want to bombard your users with annoying jimmy-wales-style boo-hoo donation banners. On the other hand you don't want your users to be completely oblivious to the fact that their favourite service is about to be shut down if everybody thinks that "it's ok, they probably have enough money / I wouldn't make a difference anyway".
Have you considered having a discreet yet prominent progress bar on the website, showing your monthly donation needs/goals? This could act as a good prompt for users to consider donating. One suggestion would be to accompany this by an information button, which when clicked takes you to a page explaining the different costs that need to be covered, where your money goes when you donate, and a detailed breakdown of what is currently still required / missing. Optics-wise, I would also try to find a way to differentiate between "large" donors and donation "trickles". E.g. you could color code on that progress bar different "tiers" of donation (or perhaps display this separately as a pie-chart in the breakdown information page). The point of this is to show to people who might be tempted to donate "small" amounts but are sitting on the fence because they don't think it'll make a difference, that it "can" actually make a cumulative difference, and push that kind of user past that threshold and donate "something".
Alternatively / additionally (you may do this already), if and when you see that a user has already extracted value from your service, such as, e.g., when they have submitted their 100th exercise, you could show a "congratulations" banner that says that you hope they're finding the service useful, and politely asks them if they could consider donating (oftentimes worded as "buy us a beer or two"). If they decide to donate, then after their donation, you could also have a tickbox at the bottom of this banner to ask in a respectful manner (i.e. not a dark-pattern-worded / confirmshaming one) if they would like to be reminded again after some time / number of exercises, or if they simply prefer to make a one-off donation and that's fine.
Finally, depending on your resources, if people "do" buy you a "beer", it would also be a nice touch to thank them for it, in an as personalised a manner as possible. Better yet, you could give them some sort of 'reward' / 'recognition' on the site; e.g. I think something silly such as revealing a hidden feature that allows a user to then tick a box to add a silly 'holding a beer' overlay to their avatar icon (in a similar way to how stackoverflow gives 'hats' close to christmas time), would actually go a long way to motivate people, after seeing these beer mugs on other people's avatars. And it's "silly" enough that people might donate to get this just for the fun and silliness of it.
The donation model is an odd one. Obviously, not everyone donates, but I think given some prompting and lack of friction in doing so, there's a significant number of people who would be inclined to donate out of a sense of gratitude, but who don't do so if they haven't been given a reasonable opportunity / visibility to do so, or some sort of confirmation that their donation doesn't just go to greedy shareholders on top of some sort of pre-existing dodgy revenue stream, but makes an actual difference to real people running the service.
Submitted title was "Exercise has 2M users but no money in the bank". Solution is not to fix the typo but to follow the site guidelines: "Please use the original title, unless it is misleading or linkbait". It's not necessary to add the company name since the domain is displayed next to the title.
There’s your problem right there. Investors want profit. If you want investors then you have to convince them you’ll deliver ROI.
Altruism is not a business model. There is nothing wrong with altruism but it is not compatible (except in short term or insignificant ways) with most of the business world, which is profit driven.
Edit: I think the comment was taken quite literally. The point I was trying to make was that the user base exists, monetization for profit is definitely hard, but monetization for keeping the lights on can work. You could try a flat fee for all users, you could do seasonal exclusive contents to support the platform, you could try selling merch, you could literally put a banner annually like Wikipedia does etc. The point being, if you have 2M users and you provide enough value, there are ways to make it work. Donations unfortunately don’t work, because it’s not top of mind for people and people need a little pushing.
“I think it's fair to say that at this stage I've lost faith in the nonprofit business model working in a way that allows Exercism to reach any of its potential.”
Removing the context that makes it clear that a quote is only applying to a very limited situation is a bit dishonest.
Of the 2M users, guessing the MAU is much lower, which would render the ad model non viable.
Still always sucks to see solid products rendered unsupportable
God please no. This is already dreadfully normalized and the web is utterly drowning in low-effort link spam, dick pills, cryptocurrency and hot Russian singles. I can't think of anything sadder than taking this lovely product and letting Google piss all over it.
I can
People tend to spend an intense few weeks learning a language, then disappear for a year. Then come back and learn another language. A large part of the 2M is still "alive" (I don't know exactly what number that would be, but I'd think around 600k) but Exercism isn't the sort of product you use day-in-day-out.
For only 2M registered (not even monthly) users? Tell me you are using AWS without telling me you are using AWS.
Let's think about their heaviest workload, which is likely running user-submitted code server-side to test those exercise submissions. Say it's really bad performance-wise and costs 1s of single-core CPU time per submission. 45m submission gets us 520 days of CPU time.
I'm going to go out on a limb and claim their entire site could run off a $50/month dedicated server, and it would be unimpressed.
Life takes energy. You need to make some of it back to survive. Asking for money isn't a dirty thing.
- Profits are reinvested to support the organization's goals and activities. - There are no owners or shareholders who receive dividend payments. - Any surplus funds are typically used to expand services, improve operations, or build financial reserves for future needs.
Non-profits still need to be financially secure.
But $12/yr. is probably doable.
your service is loved.
If you have that amount of users and you cant even pay 3(!) people, then your business model is massivly fci*g wrongly configured, Jesus!
You have that much traffic and you are unable to monetiz - i guess, this is because the "founder" is also the lead developer and the product genius.
You need some business guys to tune this!
Jesus, if this product fails, it will be a great example in whatever Stanford Business 101 course :-))
The goal isn’t to get investors! It is to get money from customers. A lot of startups seem to get this wrong.
Get some inexpensive bare metal servers and you maybe can save 80% cost.
For example, I serve roughly 3 million users on a $40/month VPS. I only run Postgres + Django, if I use Supabase and all these other fancy cloud SaaS I wouldn't be able to sustain it as the costs would increase 10~20x fold.
They can very well keep the REPL/"run code from users" bit on AWS, but serve their actual website, application and database from fixed-price bare-metal servers.
Non-profit =/= the product is free. The vast majority of hospitals and universities are non-profit, and their product/service sure as hell isn't free.
Nonprofits operate under a non-distribution constraint, meaning any surplus revenues must be reinvested into the organization's mission rather than distributed to private ownership.
What they've lost faith in is running a business with a free product.
One of the things I learned is that you should never offer even one tiny ounce of added value which the average user is not capable of perceiving.
Literally the entire idea of profit rests on arbitraging between perception and reality. You don't want to pay for surplus value that is beyond your customer's perceptive capabilities.
I would love for this site to continue and thrive.
I also agree with some others here that the hosting at AWS is probably best left to organizations that have a lot of spare cash. It is way overpriced. I self host a fairly busy site. It is mostly hands-off. I have never really fully understood the AWS premium.
Thanks for all the comments, especially those kind supportive ones. I've replied to a few, but I thought it would making a couple of general points for context.
1. I feel something that's missed is that if Exercism just continues to help tens of thousands of people per month and has no-one working full-time on it in the future, that's not the end of the world. Our donations cover server costs (thank you, donors!) so there's not an existential threat. And our wonderful volunteers keep our content growing and up to date (we've added a dozen new languages already this year). For most of Exercism's existence, it has had no paid staff. Me and Erik having to become volunteers again is ok - it just means things will improve slower.
The blog post was written as an update to our community, and our community has been on this journey with me over the last few year. They've seen us go from zero money, to raising over $1M in donations and hiring people, to not having follow-up donations and having to let those people go. They have a lot of context that's maybe not seen by people new to the story. The post is sad, because Erik, who's well loved is leaving, and because the last couple of years haven't financially gone in the direction we'd have liked it to. But the post is not me complaining.
Exercism hasn't been financially successful during the last 18 months, but it's fulfilled the core part of its mission by helping hundreds of thousands of people. And it will continue to help people moving forward.
2. Our aim is to get people to learn a programming language and then to get on with their lives. It's actively not to monetise people. That's important, because it changes the model a little. We don't expect people to stick around month-after-month doing stuff (one person said that having 70k MAU is terrible - but you can also see it as totally expected - you sign up, learn, leave - that's the point). Which means having some recurring Premium offering doesn't massively work. Nearly all the people who donate are also contributors, power-learners (learning multiple languages), or people who feel aligned to a world where something like Exercism exists, and want to ensure it continues to do so.
So the only real end-user model is to charge an upfront fee for using Exercism, which actively goes against the mission and doesn't honour the commitment under which people have contributed.
3. To all the other monetisation ideas, thank you for them. I appreciate you all wanting to help. We've considered and experimented with lots of them over the years. We've not found one that fits well yet but maybe/hopefully that'll change in the future. I have areas I will continue to peruse and am hopeful about.
4. There are numerous threads on server costs, and while I appreciate the opinions there, saving some cost there wouldn't massively shift the needle. Our donations cover our server costs. Hiring good developers and other people costs a lot more money than the ~$3k/month we'd save. I'd also say that there's some general under-appreciation of the complexity of running student's code in 73 programming language sandboxes in real-time. Exercism is a lot more complex than a webserver/database model - we have over 400 GitHub repos. I'm certain some ops gurus would be able to do a better job at building a server infrastructure than I've done for less money, but the one I've built works with basically zero maintenance and not having to be on call 24/7/365 is something I need/value.
As I said in my blog post, I'm going to take a couple of weeks out now, and come back hopefully re-energised to keep building Exercism, and the new project alongside it, which hopefully will help keep Exercism funded. Thank you again for all the support, ideas and well-wishes.
---
Changelog: - Added "For most of Exercism's existence..." to the end of that paragraph.
Shouldn't you be doing this the other way?
If you have 2 million users, then just make something that you can sell to them.
Whenever you have a bunch of customers paying you $X there is a subset of those customers that will pay you $(X+Y).
You don't have to "make them the product" in the sense that you would be selling their eyeballs, but you're teaching people to code, surely some of them will want to get a job as a coder afterwards.
What about creating a paid placement program where you give recruiters at companies hungry for tech talent access to the coders that perform the best on some tests? The coders can voluntarily take the tests, so they know their info is going to recruiters.
You could also offer certifications as a service to companies looking to test proficiency of their existing staff or create a subscription model for use by companies who want to upskill their existing staff.
With 2m users surely you have people using it from all over the world. Companies might pay a premium to hire the best up and coming talent on Upwork because generally their hourly rates are not as high as experienced devs on the platform, but you can tell based on the code they're writing that, despite their relative inexperience, they're an outlier in terms of coding skill.
You could get affiliate revenue or commissions by selling other, paid certifications such as for SAP/ABAP, Cisco, Microsoft and the like.
It just seems like you can take this as the platform product and upsell, rather than trying to create another product that you sell to people who aren't already using Exercism.
I used it [throughout 2023](https://xavd.id/blog/post/12-languages-in-12-months/) to tour new languages and have been on a deep dive of Rust this year in their [48in24 program](https://exercism.org/challenges/48in24). The UI and tooling is great, there are real humans available to mentor and everything is open source and volunteer driven. They make learning fun and that's no small feat.
Of course, things like this cost money to run, especially when there's so much going on behind the scenes. Sad that the current iteration hasn't worked out financially, but I have every faith in the team that there are good things ahead.
One thing that was obvious to me is that if you want to focus on a specific topic the $15/month subscription is too cheap, you can pay, take the course, cancel because you are done with the course, all within the month.
The other thing that is challenging is that the majority of programmer is either too cheap or "too broke", $15 USD is a meaningful amount of money in other countries with weaker currency, and in countries with strong currency I've met people paying thousands for a university diploma, doing a well paid internship that "could not afford exercism". That's though, I hope I never have to build a business for software developers.
All of that shared experience comes from 3-4 years ago, so prices, models might have changed since that, but if they go under it's sad, their UI and their UX was so clean, fun, interactive, pleasant, etc, it was a joy to study using their material.
Without revenue any company will fail regardless of popularity. A firm may acquire the company for the customer/lead list, but I wouldn't count on that kind of valuation matching the work your team put into the projects. =3
It is time investors businesses and started looking at the bottom line again, even non-profits. I mean can you imagine if we based the success of a store with how many people walked in even if they did not buy anything?
Even non-profits need to make money, so their business model was either wrong or they do not have a product worth paying for. On top of that we are probably in economic stagflation so we will be seeing more stories like this as the months go on.
So you need to upcharge. It is 100% free… to start
and then there is a subscription that is paid to finish the language course, shown in monthly cost but charged annually by default
more languages? charge more
You can sell the account data to recruitment firms. Everyone hates this but you know that your users are the product, your users know that they are supposed to be the product. Keep a cut with the recruiters so they have to pay you a little commission when candidates land a job
85% off on the annual plan, just that it’s $200, but at least you’re not paying $99/month anymore
the interesting thing about the not “not-for-profit” model is that all you have to do is just undercut educative.io by a tiny bit
Business model: "I've lost faith in the nonprofit business model"
Also noting changing in the IT market, tech layoffs, AI revolution, less demand for tech workers.
Hope it will work out for you!
A lot of people will probably tell you to add a paywall, not sure it's breaking the "non-profit business model" if you keep the price really low. But at the same time, even a low price would deter some users (myself included) and it would also devalue your content :/
Have you tried selling access to that data to the usual suspects ?
You created a site that has 2 million users and don't know how to monetize? Just charge people $5 a month to use. It will solve all your problems.
You can set the minimum to zero and a default above your target, and perhaps ask only after a short free trial period. I heard that it often yields better results than we would expect.
We're not trying to be some business that sucks people into using our site every day and monetise them. We're aiming to teach people effectively and efficiently and then get them moving on with their lives.
We could make the site less efficient, or change it's purpose, but that's not the mission of the organisation.
If is low all you can do is lower costs, which should also be straightforward.
our postgres offering costs roughly equivalent of RDS so if you use it with Django I don’t think you will come close to 20-40x a VPS, ymmv, depending on your workload
We have an Auth service, but since you use Django you wouldn’t need it - sounds like you’d only need postgres (like a lot of our customers)
And thats fine, you have features on top that can be worth paying for certain customers and expenses of your own to pay. Those of us that have been around in this business for 30 years know how expensive PAAS and IAAS can be and how easy it is to roll your own solutions and when not to do that.
I love exercism. It's beautiful, works in the browser and from CLI. The lessons are really well designed. It's the perfect coding school environment.
So, I'm sad it isn't viable.
But, I think I am really saddened by the comments here, second guessing all the decisions they made. If I were to second guess the people behind most of the comments, I would assume no one has ever run a business with payroll, or built something with this much reach. The comments ring really hollow for the most part and seem really callous given this person has sunk his life into making the world, and my life, a better place. I'm scared for the moment he is in life and sad there are not more positive comments, and even better, comments from people where they would actually help by making a donation.
I'm ok. I'm hopeful for the future. I believe there's lots of amazing stuff we can do and that Exercism has a great future. But right now it's in a hard, low place. The last few months have been pretty tough emotionally, but I think getting to this place is one of acceptance. The only direction now is up :)
Even if it gives a few useful features to subscribers, that's better than shutting down.
I’m donating today and I wish you the best.
Would you email me (chris@extrastatic.com). I've got a few people I think would help depending on what direction you want to go in.
Fwiw moving away from AWS would probably get that bill from 7500 to 750pcm or lower. Cloud is very expensive if you've not made special arrangements or can burn VC money/free credits for your org.
As someone already mentioned please consider adding ways for monetizing, ads + remove ads, paywall, paid exercises, etc. to stay afloat.
Also please get in touch, email in bio, I don‘t have anything concrete in mind but would like to be in touch. In another life I would gladly be teaching coding, who knows maybe it works out.
Somewhere along the line between 2010 - 2012 and definitely post 2013, most of the HN comments are no longer founder focused or business minded. It is unfortunate.
Or maybe they have and therefore know you can’t sustain a “business” without collecting money to pay for your expenses?
Yes, I have deep respect for the people building such a site for the better good, but asking the "how will we sustain this" from Day 1 is important (not even how to get rich, just how to get break-even).
Maybe you can reopen a free tier later, but for now you are in panic mode.
By any chance do you also donate?
I know you mentioned you don't wanna do ads, which makes sense. But if you're getting 1200 sign ups a day thats worth a lot, and id be happy to chat and see if there's a sponsor opportunity thats uninvasive and that could at least keep you going.
anyhow, you can reach me on Twitter @wagslane or via email: lane @ boot dev.
either way, hope things turn around
You've probably thought of this, but it might be easier to charge a very small amount -- say $1/month, which most people even students, could afford, rather than rely on donations. If you get 1% of users paying that, it's still $20K month which is more than you're getting now. (Of course managing the payments themselves costs money).
It can still be non-profit (which is a great thing), so no one is getting rich off of it, and it doesn't poison the mission. But it could pay the bills.
Update: Processing fees are a problem as posters have pointed out, so you'd probably want to make it $10/year, or $5/six months. It's still an easily decision for a very large number of people.
Patreon has $1/month options, not sure how they do it though.
Given the state of every service becoming subscription based, I can definitely see a lot of subscription fatigue. Maybe an alternative route you could try is the annual wikipedia "We want to keep this free. Please help us buy donating". I think a lot of people appreciate the service but a lot of the times it's easy to forget that running a service like exercism can get expensive, and would be willing to donate, given a compelling enough reminder of this.
You already have the sandbox and the support for the most popular languages.
It may take some development to build out the feature but you probably wouldn't need too many to sign on to cover your costs and you could still leave your core offering free.
Have you thought of adding a hiring feature? Where people upload their CVs, and companies search (anonymously) for talent?
There is a hiring model called PPA (Pay Per Application), or monthly fees to search CVs.
Your unique selling point would be companies are not getting hundreds of custom AI CVs flooding their application; they search for what they want, and you're helping both sides.
Good luck!
edit: grammar
If you have an email list of 2 million… a lot of companies will pay $50-100k for a mention.
E.g., there are orgs to promote computer literacy among women; and among poor people in India; and among middle-schoolers in Africa, and so on. And he has users who are women, and who ware poor people in India, and who are middle-schoolers in Africa, and so on. But none of the orgs will give him a grant, because they can't guarantee the money will only be spent for their own target groups.
I can see why that would be kind of discouraging.
“I think it's fair to say that at this stage I've lost faith in the nonprofit business model working in a way that allows Exercism to reach any of its potential.”
Which provides additional context. Why’d you remove it? It makes it clear that he’s talking about the non-profit business model as applied to his company.
Then he goes on to say that he is upset that 96% of people stop the program. Dude, coding is BORING, but coding is hyped so much that people have high expectations so you get delusional people coming and signing up to your website.
$7,500 monthly is nothing to a large tech company. It sort of does makes sense for big companies to subsidize this so that the quality and amount of software engineers that join their company increases.
This value prop is intangible, but I feel like at least one big company would go for it.
What is all this computing power going towards?
- Nonprofit business model does not equal "everything is free for the user forever", I'm guessing you already know that, but the wording on why you don't believe in nonprofit business models explicitly mentioned keeping everything free as the reason. You can earn revenue from users in a nonprofit business.
- You have a big audience with good engagement for the segment, there are multiple ways to make money without abandoning the core mission (job boards, screencast upsells for advanced courses, premium content, whatever else, look at how Remoteok.com makes money, copy-paste as the founder is super open on his process)
- Being a for-profit business and fundraising, will temporarily solve your issue of having funds to run a business. It will not solve the issue of not knowing how to/being afraid to charge users or other parties for the value they get out of your product. You could already be solving this problem today, and you have a 2million audience pipeline built in to solve that issue.
I'm not dismissing the challenge of some business segments being extremely difficult to make money in despite the value being meaningful, I work in healthcare, so I know, but since your new business will effectively be in the same segment, do focus on the revenue aspect much sooner and much than you think you'll need to, because you already know what happens if you don't.
And big respect for what you've built in a super crowded space, you obviously have the product and user empathy chops needed, wishing you the best of luck on nailing the business chops!
Some thoughts before I donate:
> I've lost faith in the nonprofit business model
Great! You will find a way to generate revenue streams. This is a good pivot! I would recommend going fully open source and public domain as you add a profit model. IMO public domain for-profit companies are superior to (C) non-profit companies. Profits don't make you evil, closed source and copyright does.
> every PR in a live Exercism repository should get a review before it can be merged
Why? I recommend just merging in. Quickly fixing/undoing problematic commits.
> $7,500 in monthly donations. That covers our server cost
OMG!!! Stop! This money should go to you 3, not cloud providers.
Server costs should be $0. This should be a local first, entirely client side site. Maybe 1 $50 a month droplet if you need some read/write user stuff. With less than 100K active users in a day, 1 server should be more than enough.
Finally, ditch the (c) symbol at bottom. Go public domain!
Here's my user test and authoring of this comment: https://www.loom.com/share/763e480438c4481ba0aa056c6ef0cbbb?...
Y-Combinator says “build something people want”. Two million sounds like a lot of people.
I’ve only glanced over this so I could be missing some critical fact, but superficially it sounds like pmarca or pg should write a check before the rest of us stop believing in the startup lottery.
To me that begs the obvious question... what about trying some automated advertising like Google Ads? Probably not the most theoretically optimum use of their ad space, but minimal effort is involved.
What I am trying to say, there is no free lunch.
I don't have a solution for you, and you probably tried a lot of prudent things. I assume you have already contacted people that buy into educational software (e.g. Kevin O'Leary of 'Shark Tank' fame)?
I wish you some kind of miracle that turns things around; sometimes once conversation can change everything. Surely this user base deserves that the site finds a way to sustain.
Feels like you've been kickstarting a competitor/pivot on the side.
At my org it’s hard to get justification for donations.
However if this was something like ~100$ a year or whatever per dev, I’d be paying it today for my team.
They created a subscription plan that had no real value except maybe adding some trivial "feature" like a gold star next to their name (I don't remember the details). This made it possible for the companies to receive a standard invoice that had a greater chance of approval.
Appreciate the comment. If you have thoughts on that, I'd love to hear them here: https://forum.exercism.org/t/exercism-teams-coming-soon/1266...
maybe everything that could have been tried was tried but this comes off like some of the effort spent working on the next company could have been spent keeping this one alive
Amusing. I walk away from all business partners that think this way
They hand waive away “who will pay what”
You don't know more about the company than them.
Most likely they saw this coming months ahead of time, and needed to figure out how to pay their own rents after actual doomsday and needed to pre-emptively start figuring out what's next.
Founders who don't start with a pre-existing financial cushion have very, very little job security if they underpay themselves and their businesses shut down without an exit. Also, founders tend to be generalists, and most big companies don't need more generalists, so it's not easy to get hired, doubly so if you have wasted a lot of your brain pitching to investors and having coffee chats with clients. The harsh reality is, the more you are forced to chat with investors and customers instead of working heads-down, the more you lose your hard technical abilities that other people would hire you for.
As a founder of a shutting down company, if you want to pay your rent on time, you need to do one of 3 things: (a) pay higher salaries prior to shutdown to give you some time to figure yourself out (controversial), (b) start studying to be a specialist well ahead of shutdown and get on the interview treadmill, (c) start working on your next thing well ahead of shutdown.
If you do none of those things, things can get really dangerous to your personal finances.
I used to contribute to Exercism a lot (I'm still technically a maintainer but haven't meaningfully helped in a while), I think it was the first OSS project I contributed a large amount of effort to. I was more or less the sole maintainer for C++ for a while and also helped with the C track.
It's definitely a good service and OSS project, I recommend it a lot. Fundamentally though people don't want to pay for things that they can get for free, or they're used to getting for free.
If people actually want OSS projects to survive, you should be giving a little bit of recurring money. I set aside $XX/month to distribute to projects and I don't even work anymore.
Its bc we are skeptical and frugal.
It seems to me if you can’t get 1 in 500 of your users to give you $10 per year then maybe your product isn’t actually valuable or useful and should just shut down.
I say this without knowing anything whatsoever about this product or its utility; it just seems like a general thing. If you have this many signups and you can’t charge even a tiny tiny fraction of them, it means the signup number is irrelevant.
0.2% of people who use wikipedia donate to it. Unless you are doing tens of millions of unique views a month, it's probably not gonna be sustainable.
People really hate paying for things. Even things they use and love.
Endless telethons for public television in America annoy 100 people for every one that donates. Despite that you see ads on public TV for the Archer Daniels Midland corporation all the time even though, allegedly, there is no advertising on public TV.
even getting people to click on affiliate links that won't cost them anything I find is tough as most people remove the referral link or purchase on another device probably out of privacy concerns.
Kinda like a dating app, where maximizing usage means not getting you hitched (too quickly).
One of my theses as an edtech founder is that the education can't be the core business model long term.
PayPal's traditional fee (for USD)[1] = 3.49% + $.3, for $1 = $0.3349
PayPal's micropayment fee (for USD)[2] = 4.99% + $.09, for $1 = $0.1349
Using that model would make sense in OP's case.
[1]: https://www.paypal.com/us/webapps/mpp/merchant-fees#statemen...
[2]: https://www.paypal.com/us/webapps/mpp/merchant-fees#micropay...
Most organizations aim to survive as long as possible, so financial sustainability ($) is crucial.
I would then give a free account to anyone who asks. ("We know that even $1/month is a big ask for some folks in certain countries or they may even have a method by which to pay it. If that's you, let us know and apply for a free account here.") Then basically auto-approve (with some bot-screening mechanism) accounts that apply. Sure you'll get some abuse and people lying, but not that many people are going to lie just to save $1/month.
measuring success by how much people are willing to shift scarce resources in your direction is a perfectly good way to measure success. It's "weighted usage+reach". the remarkable effectiveness of markets in allocating resources this way is a huge achievement for civilization. I really had to scratch my head at their surprise that a non-profit oxymoron model, i mean, business model wasn't viable.
What? No.
If a business isn't able to sustain itself with its revenue, it has failed and will need to shut down. That's the definition of failure.
Lots of things can be "successful" by your metric by using unsustainable pricing.
I'm guessing fees make it not worthwhile, maybe they should negotiate lowered fees from their payment service, or figure out someone else who would give them drastically lower fee rates.
I offer a ton of free information online. I received raving feedback from people, including some that said I saved them thousands of euros. Just this week readers jokingly suggested giving me a Nobel prize or a statue. Still, donations are one percent of my revenue on a good month. They would not cover my groceries.
This is fine for me, but it's always on the back of my mind when people suggest donations as a funding method.
In my (limited) experience they mostly don't work because the people/projects that need donations are way too coy about asking for money. Donation buttons get tucked out of sight and/or cloaked in twee language like "buy me a coffee!"
I like to point to https://archiveofourown.org as a counterexample. Hardly the most critical project in the world - it's just fanfiction - and yet they raise a few hundred thousand dollars a year in public donations (solidly surpassing their budget goals) because they run an actual, clear, focused fundraising campaign twice a year.
I run a number of apps and services that don't make a dime. In fact, they cost me thousands. I don't mind.
Monetization is a real pain in the ass, and I stay away from it (for these services).
I have the ability to make a lot of money, if I choose, but that's not much fun. I don't really need the money, and I like having full control of everything I do.
I am regularly approached by folks with monetization ideas. Some, are quite good. Others ... not so much ...
Maybe I 'll pursue something, eventually, but I'm in no hurry.
Thanks for your service. Looks like a great site.
I'm sure when all is said and done, there just are not enough people like me to make a difference, but I sure hope things swing the other way. Subscriptions for everything and tips for everything are two cultural things that need to die in a fire. They are ways of squeezing more out of people, especially by taking advantage of human tendencies. It's gross and at this point I think it is unethical. Maybe not all implementations, but there are plenty that are.
So my $6 per month VPS would be about $2,000 upfront.
And a $500 game console would be a $1.50 monthly subscription. (Obviously glossing over tons of differences in say, a PS2 and a Stadia subscription)
It feels easier to ask customers for $1 per month than $360 up front... Especially since I don't know if my service will be up forever
The only way I can see this working is a recurring donation with a low enough amount that people will just set it once and forget it.
So instead of a $1/month subscription, you could try a $1/month recurring donation.
The general argument is - we can specifically identify talent and motivated learners/skill-possessing people, who will look for jobs. That's exactly what companies need and companies pay $5k-$10k a head during good markets for this rare STEM skill, like programming!
Problem - The people on these platforms are 70% likely to give up within 6 months or go into another field
Problem - Experience means 10x more than education to employers, especially fundamental experience
Problem - Your information on potential hires as the provider becomes outdated fast and supply-demand changes in hiring market can reduce your commissions by 50-75% in one year (as happened from 2022-2023)
I would go as far as saying these users - because they are typically earlier career - are actually worse for connecting with jobs boards/hiring than literally any random iPhone user for advertising purposes. And companies don't pay much to advertise to random iPhone users. I wish it wasn't so, but it is so according to how companies act.
If OP wants to monetise this way they should consider an affiliate programme to a CV upload or Jobs board site, it will test any appetite for hiring these people at the moment much quicker.
You could follow the Sublime Text model and basically have an honor system—if you're actively employed making more than $X, please pay us. If you're not, you're welcome to use it for free!
The Tech Resume Inside Out [0] uses this model (though not on the honor system) for similar reasons.
I would look to reduce server costs instead. $7k/month for 2 million users seems like a lot. Do they execute your code on their servers? I couldn't really tell from their website.
We're paying ~$1k/month for all our tooling servers (so the 150 different test runners, representers, analyzers that are used to check peoples code. There's >=1 of those running every second). Bare in mind, we're running student's code in over 70 languages here. Each is a docker container (often many gb large) - so we pay for HDD too.
The biggest actual cost is the database at $2k/month. We have about 600 queries per second, and around 10MB per second (spiking to 47MB per second) of read throughput. It's an autoscaling database, but AWS determines that it's at the level it needs to be, and if I turn that down, performance suffers (I've tried).
Beyond that, all the other individual services are ~$300/m, so quite small amounts, but for things we rely on (e.g. caching servers, a shared filesystem amongst all those servers, and other things).
$1.2k on tax is also fun.
From the numbers you've provided, it seems like your total cost is around $5.5k, so I assume the remaining $2k is attributed to traffic.
Everything looks quite reasonable, except for the database and traffic costs. I've run MySQL servers handling 150k reads/sec and 50k updates/sec with no issues, even on very cheap machines (around €30 per month). Years ago, we were serving over 100 million pages (of heavy content) per month, and we didn’t even bother looking at traffic statistics because, here in Germany, it’s hard to hit the traffic limits that most hosting providers impose.
That being said, AWS is less expensive than I initially thought. At the same time, I’m confident you could reduce your hosting bill by up to $2k without even leaving AWS by setting up your own database server. Moving away from AWS entirely might be challenging, as managing a fleet of about 30 servers would likely take one or two days of work per week (I'm managing a dozen mostly idling servers and I work one day per month on them). When your hosting bill reaches $30k, I'm very sure it would be cheaper to hire someone (hint, hint ;-)), that moves everything to dedicated servers and manages them.
RDS is extremely expensive. All managed databases are.
That said, it's a trade off of convenience and being in the AWS bubble, and weighing up the pros/cons of separating out services. Data Transfer is another thing to consider too of course. Sticking your database elsewhere might cost more in egress traffic communicating with it from your other AWS infrastructure. If you're all in on other AWS services, sometimes the RDS price is just worth it when it comes to the total price. Sound like this might be the case for your setup.
I hope you do manage to work things out. The service you have is great.
PS - Side note on RDS sizing. You might already know but sometimes it's worth increasing the storage size on gp3 type storage above 400GB (if you haven't already) as you get 12,000 IOPS baseline against 500MiB/s throughput[0] when you have that much storage. That's 4 times the below 400GB baseline performance but you only pay for the additional storage cost. It can make a difference if you're IOPS constrained or trying to deal with bursty traffic but want to use the smallest instance size possible otherwise to save costs.
[0]https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...
Anyhow that database server was probably ~$35,000 all in. That's 5 months of your current AWS spend. One of the things i did during that time was take a 2 generation newer server, a $35,000 1u Dell with 512GB of ram, and mirrored the postgres database into tmpfs and enabled replication, then we set that machine as primary. The new machine didn't break a sweat. So much so that one of the things me and the (really very awesome and nice; Hi, Chuck, if you're out there!) DBA did was set postgres to use no more than 640KB of memory, then ran the entire site, with 4k QPS, on that postgres instance with 640KB of memory (not counting the 280GB of tmpfs storage, of course!), just to prove it would work. It did - although some of the bookeeping queries (not sure what they're called) were taking a very long time, and would have had to be refactored to use less temporary memory, and such.
anyhow my point is, there are people out there that can do things cheaper, or faster, or more efficiently than whatever you got goin on right now. Your statistics on "per second" usage and the like don't sound too demanding. If you could squirrel away $500/month for a few months, and you ask around for someone that can rack metal and has peering, there are people (including me) who could get you co-lo in <16U[0] with redundancy, where your only monthly infra charges would be the co-lo fees.
[0] Old, extremely beefy, but large servers are generally 4U, but dirt cheap for what you get. Ex: 80 thread, 512GB RAM, 8 SAS bay HP server, $800 shipped. And i bought those 6 years ago. However: 5950x, 128GB RAM, 24 SATA port can be had for <$2000 (i'm guessing based on what i paid a few years ago), and that's roughly equivalent in power (kernel compile takes 3 seconds longer on the 5950x but it uses 1/4th the power at the wall). The reason i tagged on <16U is because at most you're gunna need 4x4U, two "front end" and two "back end" machines, with duties split and everything redundant in the rack. I haven't looked in a while to see what's available on ebay as far as more density, but for sure 16U or less!
The issue becomes: how do you find someone who knows how to do all that, that is willing to work for next to nothing because they believe in the ngo/nfp? Maybe there's a tech forum that people like that read, who knows.
good luck, and thank you for doing things to help other people. I hope it all works out in the end.
email in profile.
That is not much, considering they need to cover all aspects of running a nonprofit; they need enough technical staff to maintain a 24/7 on call rota while letting people occasionally go on holiday; and they need to pay healthcare and pensions and suchlike on top of salaries.
And it's not like they can offer a meagre salary now but promise vast wealth in the future, like a startup might.
It buys you 150 machines for a month 12 core 24GB VPS with unlimited traffic on a 1Gbps link
See my comment below.
Yes!
https://www.ionos.com/servers/vps
$7500/month would get them 150 such servers.
Maybe they should cut the AWS costs and hire their developer back.
Id be really interested to hear the breakdown of their AWS bill. It would be a crime if they were blowing what money they have giving Amazon 9 cents per gigabyte egress.
Edit: It appears that their "not-for-profit" message in their footer is a bit misleading, as explained by those who replied to my comment.
A true non-profit company in the UK would be a registered Community Interest Company, or a registered Charity.
Typically you'd form a Charitable Trust or Charitable Incorporated Organisation, and Exercism isn't listed as either on the Charity Comission website.
Markets are what people do when they’re not having sex. No sense in fighting markets.
Cartels and market failures are what happens when the holders of capital aren’t terrified of Lina Khan or Maximilian Robespierre for longer than 12 seconds.
The Battery Club is for kingmakers who decide things together for their mutual benefit behind closed doors. They learned their lesson after Don’t Poach Gate: it’s not in writing anymore.
Having 1600 people sign in daily and never logging back again - creating 2M zombie accounts is I suppose critical fact that you are missing.
To get from 800 real people to two million user id rows one would be running some kind of spam scam, and if that’s the case I apologize.
Busy investing in AI?
I disagree. Wikipedia is similar to Exercism on that count. A tiny fraction of expenses are hosting the encyclopedia. It's also necessary to pay people to program the software underlying MediaWiki.
People aren't bothering to edit anymore either. Search engine summaries, voice assistants, and now large-language models repackage information from Wikipedia in a way that discourages people from going to the website itself. Most low-hanging fruit is gone, and the standard of quality has increased.
There's a whole project dedicated to helping undergraduate students edit Wikipedia as a part of their coursework now. It's a great learning opportunity because a Wikipedia editor will check your sources more than the average TA. The flipside is that an undergraduate degree in the liberal arts isn't necessarily enough to write at a Wikipedia level, which is a problem.
These external factors are decimating the volunteer community and the visitor population. Spending on outreach, staffing, and grants to local affiliates is much costlier than having people wander onto Wikipedia and contribute, but that's no longer the era we live in.
I do know a lot about noncommercial broadcasting in the US though, and your comments appear superficial.
A sponsorship message is different from a call-to-action advertisement. The former is permitted, the latter is not. They are not different enough, in my opinion, but they are discernible.
Public TV and radio station finances are public information. Audience fundraisers are a meaningful portion of their operating budget (~25-50%), though it varies by market. As are program underwriting (up to 50% in strong markets), and often CPB funds contribute a few points (this is highly politicized and if you overindex on it, your news sources are likely polluted).
Personally I don't want to guilt trip my readers into donating. There are affiliate links on the website. It's a business and it works well enough to support me. It feels like double dipping.
Above all, There's a point where it operates exactly like an ad and annoys people just like one. At the moment, American tipping culture is taking over in Germany and it feels like another instance of that.
The nice middle ground would be to ask after doing something especially nice. "That tool sure saved you a lot of time. Donate?" I think that the link in my email signature works especially well because of that.
If that doesn't work, they could introduce premium features. Stuff that obviously has to cost money would make the pill easier to swallow, for example, features that depends on third party services (only LLMs come to mind right now), or human assistance (real-life instructors). Admittedly, this would be a compromise. One could argue it fails on the 100% part, but another could argue that 100% of existing features remain free.
Another thing. There are several podcasts who have a free feed and paid one. Sam Harris’ Making Sense comes to mind with their free feed and a paid one for full access to all content. But they understand not everyone can pay. If one is unable to pay, they can simply email to ask for a free annual membership.
Pay what you want albums on Bandcamp are another example of a sliding scale. Suggested price $10, but artists can define their minimum.
Someone has to pay (paid) for this business a.k.a. is actively loosing money.
On the other hand, it allows me to work on what I love full time. This would not be possible if I relied on user donations or even public funding.
Thought of handing off some of those money-making opportunities to others, after you vet them beforehand?
I could use an alternative income stream and I am very tired of the employment grind. But I never networked and that has been biting me on the rear for the last 10-ish years.
Apologies if that's too direct. I am keenly feeling the lack of freedom and the lack of "frak you" money in the bank.
Community Interest Companies do have sharedholders and can pay dividends. CIC's are definitely not charities in the scope of the word that you're using it.
I don't want to argue that AWS is cheap, but I call hyperbole here. Where can I find an m5.large (2 core 3.1GHz CPU, 8GiB RAM, 10Gbps network, $70/month) for $7 USD a month?
Then there are all the services that ain't plain servers, those are generally priced at at least a 10x markup, some of them much more than that.
maybe i misspoke - at one point there was battery backed DDR ram they used in PCIe, but by the time i came around they were using Fusion IO PCIe devices, which i guess were NAND flash, not DDR. or, alternatively, that is how it was explained during onboarding - "it's like DDR on a PCIe card, so the iops are 1000x that of SAS 10k drives"
unless you're talking about our experiment of tmpfs - then yeah, the use case was "genewitch heard bill gates say 640k should be enough for anyone; here's a super beefy machine to test that theory; theory tested." We didn't run the site live on that machine for more than 10 minutes or so, we switched it back to the fusion-io backed server immediately. It was a proof of concept about one of the things we could do with these new servers - read replicas with the DB in tmpfs for extreme speed and no IO blocking.
I do think they really can have legit sign ups, but people "wanting to learn code" usually fizzle out quick. I have my experience as a dev when friends/family/colleagues ask for guidance, I provide them with links resources and some mentoring and after 2-3 days they don't follow up ever again.
But not using metric like daily active users, returning users, might indicate that they want to inflate their worth or they convinced themselves to believe in the wrong metric.
If the millions of users thing is even a little real, that’s a galactically less risky play than friggin LangChain for Pets or whatever.
I hope I’m being paranoid but I get the sense that someone pissed off someone important.
A better term for OpenAI would be “rogue state”.
https://www.bloomberg.com/opinion/articles/2023-11-20/who-co...
It’s clearly not if the founder isn’t taking a salary and they just had to lay off their only employee..
Most have adopted EKS-like services i.e. kubernetes.
There is fly.io that's closer. Hope they improve on the reliability aspect.
almos zero sysadmin troubles. might even reduce the troubles of working with eks/ecs.
now for storage and db, that's a different story.
I hope you would consider a "hard sell" banner ad for donations in the vein of Wikipedia before giving up on the model. I wonder if there are a lot of potential donors out there who would respond if they understood the stakes.
Redesign the donate/payment page for clarity. Enable Apple Pay to reduce friction.
https://www.dictionary.com/browse/business
Someone's profession (their "business") can certainly be working at a nonprofit, but that's because they are getting paid. The nonprofit organization itself is not a business. You might be looking for the word "corporation", which can include nonprofits.
Nonprofits in the US literally are corporations, just with a different tax status. They need revenue that exceeds their costs. Many of them sell things. They can even turn a profit. Ikea operates as a nonprofit in the US.
I don't know how much effort they've put into experimenting with pricing, but it'd be worth trying to increase donations, especially for such a well-designed site.
But yes, rather than shutting down, I'd probably just put the whole thing behind a paywall to try and keep it alive.
In the best case, you get to change the world and make a bunch of money. Sometimes, you just have a viable business selling sugar water. And other times, you're doing something really good that's just not going to get you paid.
There's nothing wrong with the last thing, other than its potential lack of sustainability.
I'm now a teacher; I could easily make 10x my comp elsewhere; possibly 100x. Measured on your scale, the only value is what I'm getting paid. But it turns out that those good feelings are worth something to me.
This. I feel that this has been missed in the thread a little.
Non-profits denote nothing about their financial good-standing or moral good-standing. You can be an amoral non-profit, a moral for-profit or vice-versa. Your profit structure has nothing to do with the morals of the people inside of it.
Additionally, there a lot of non-profit executives that are profiting very healthily off of their work.
The average for-profit company can't be trusted/expected to do what's best for anyone outside the profit structure (e.g. users or the public at large). Of course you can have exceptions in either direction, but on average for-profit orgs will likely be willing to fuck over anyone else to increase profits - intentionally or not - because profit growth is literally their purpose.
'Non-profit' advertises that you don't have to worry about that specific, powerful motivation of the org contending with your best interests.
It's a less powerful motivation, in that there are not shareholders who will benefit from growth.
But in practice, executives and managers end up making more, which provides an incentive for growth.
A sustainable business demands some kind of ongoing income.
That is how stupid costly AWS is and 99% of people using AWS are wasting money like there is now tomorrow.
You don't even need to use Hetzner, I'm actually quite surprised they are spending 2k on DB. For the env isolation i understand they need ec2 but...
it shouldn't cost anywhere that amount, for instance I have 3 million users on a $40/month digital ocean VPS with just Django + Postgres and I have far more reads and writes.
for env isolation you could spin up a $5/month VPS and shut it down when idle
Why do you think that is? Are that many people that dumb or are there other reasons they use managed cloud?
Requiring more revenue than costs does not a business make. You starting a nonprofit does not give you special insight into the meaning of words. I am the treasurer on the board of a nonprofit. I have had to go through the books quarterly, confirm we’ve got enough cash for a year of runway, etc. The understanding of what nonprofits do is not the issue. This is a boring dispute over words (semantics).
Remember, you initiated this dispute over words when you made an argument “by definition” and you criticized KolmogorovCamp’s correct use of the distinction between nonprofits and businesses.
I get it, AWS looks expensive, but a bunch of their foundational services are real force-multipliers if you don't have the cash to build out entire operational teams.
Its just not true that AWS doesn't need expensive experts to get stuff done - it really does.
Anyone who is half decent on the command line in Linux can get all those servers installed and running without the spaghetti complexity of AWS.
The cloud as a magical place of simplicity and ease of use and infinite scalability in every direction - I think its the opposite of that - AWS is a nightmarish tangle of complexity and hard to configure, understand, relate and maintain systems.
Its MUCH easier just to load up a single powerful machine with everything you need. I'm not saying that works for all workloads but a single machine or a few machines can take you an awful long way.
For the core service I tend to favour monoliths too, but I would say you are vastly underestimating the halo of other crap needed to operationalise a real website/SaaS.
Where is your load balancer? Your database redundancy? Where are backups stored? Where are you streaming your logs for long-term retention? Where are you handling metrics/alarming?
Bare metal is great, but you have to build a ton of shit to actually ship product.
You get all if not most of this on DigitalOcean, Linode, Upcloud, Scaleway, etc all of a LOT cheaper.
> but a bunch of their foundational services are real force-multipliers if you don't have the cash to build out entire operational teams.
No, it's not. As above and for a lot of things AWS' complexity and silly factor can make it even worse. In GCP I can setup a dual region bucket. As simple as that. In AWS I need to setup 2 buckets, a replication role, bucket policies, lifecycle policies and a lot more just to get the same. Force multiplier? As in make it slower? EKS takes longer than the default Terraform timeout to provision. The list goes on...
Why go to the trouble of running Kubernetes on top of AWS, when ECS does roughly the same job at a fraction of the complexity?
Why use Terraform when CloudFormation maps better to the underlying primitives?
The problem here is that we all have opinions about hosting, but not so many useful opinions about business models, so hosting feedback is what this person is going to get.
Even if things have changed now, I can't imagine you'd find a great dev with 10 YoE for less than $80K/yr, and that's hiring globally (with the time zone issues that come with it). You can probably get 1 OK dev and 1 bad dev with 10 YoE for that price though, but you'd usually be better off just hiring one great dev than any other combination.
Also note that a lot of the things you are paying for (CPU cores, traffic, network throughput) in those nodes are shared resources (that Gbit link especially) and/or have “fair use” policies attached to them, and while the same might be true of cloud providers those policies are often either more generous or (perhaps more important from the business stability PoV) at least better defined.
“Cloud” is still expensive compared to buying and managing individual nodes, even if you add in all the above and the things I no doubt forgot to mention, but it does give a lot more than the same cost in individual nodes than this sort of comparison suggests. And sometimes just not having to deal with all that, keeping the business more focused on its core competencies, is worth the extra expense.
In DayJob we use Azure a lot, and sometimes I see the costs of certain things¹² and balk, and we do still have infrastructure people to manage the platform, but overall it works better for us than managing our own resources more directly. We have an extra complication due to our client base (regulated companies like banks and insurers, who are storing PII of both their own people and their customers with us) in that we have to give a lot of assurances on security and such which would be more work (it is already a _lot_ of work as anyone else in that sort of B2B arena can attest) if we self-managed everything.
----
[1] $2,400/yr for SFTP access to a storage account if you need it available 24/7?! Especially given we have at least one such account per client as their requirements understandably require that level of separation. I think we'll keep using the relay & management dashboard I setup in a few cheap VMs, thanks…
[2] and the performance given the costs: AzureSQL³ I'm looking at you!
[3] though again, some of that cost is in things like the scaling flexibility and other infrastructure convenience, which the business finds worth paying for
History has proven that most of the time these reduce availability than increase them. Any sort of failover and the complicated setups to get it going introduces bugs and issues more than the redundancy it provides.
Have we forgotten the number of large single server applications running on single linux machines that never needed an unplanned restart or had a crash for years? And you can't beat AWS us-east-1 or Azure or GCP in outages lately.
And I doubt any service like this needs auto-scaling. Most services barely will use up a proper single server i.e. something with >96 cores >1TB of RAM.
> “Cloud” is still expensive compared to buying and managing individual nodes > And sometimes just not having to deal with all that, keeping the business more focused on its core competencies, is worth the extra expense.
There are ways to not manage all that and still be in the cloud. It's called don't use AWS or Azure.
No.
I run multi-site Ceph+Nomad clusters with NixOS on Hetzner for our startup and maintaining those takes less than 5% of my time.
By using great tools and understanding them well you can do it with little manpower. I learned all those tools in around 3 months total -- so around as much as getting a basic understanding of AWS IAM ;-)
The only thing you don't get with that from your list is auto-scaling. But the with Hetzner the price difference vs AWS is 10x for storage, 20x for compute, and 10000x for traffic, so we just over-provision a little. And my 5% time /includes/ manual upscaling.
Yes, I am oncall 24/7 to manage that infra, but I'd be as well when using hosted cloud services. Yes, fixing a Ceph issue, or handling Hashicorp Consul not handling an out-of-disk situation correctly is more complicated than waiting for S3 go come back from its outage, but the savings are massive. Testing whether your backup restore works is something you need to do equally with hosted services.
So it is definitely possible to self-manage everything, for 5% of one engineer.
“and understanding them well” is doing a lot of legwork there. From a standing start how does a startup that has the skills & experience to make the product but not necessarily manage the infrastructure get to the point of understanding the tools well, or even knowing which tools are best to learn to the point of understanding well?
> So it is definitely possible to self-manage everything, for 5% of one engineer.
I can accept that as true, if you have the right person/people, and they are willing (particularly the on-call part).
What would you prefer, this website eventually shutting down because the donations barely cover hosting costs and there's nobody to maintain it, or the website occasionally going down but otherwise actually being profitable enough that the founder can continue maintaining it on a part-time basis and keeping the site alive?
What we lose sight of, is that those things aren't as important as we, as SREs would like to think. When you're a corporation of one person, trying to stay afloat, you can just rely on a single big box and spend your time dealing with all the other problems first. Make sure you have an escape hatch so you can scale up if need be, but don't overengineer for a problem you won't run in to.
Who cares? At this point it's a hobby project that makes zero profit and is bleeding money. No more money is going to be lost if they lose the DB tomorrow. No more money is going to be lost if they go down for an hour or a day or a week (in fact, they might _save_ money if they don't get more AWS charges during the outage).
They have nothing to lose, and about 6k/month to gain by moving to cost-effective hosting, which could actually make this a decent side-project.
Because it doesn't. ECS has its own complexities - perhaps as a result of EC2 fleets / autoscaling groups and more. Suddenly you need launch templates and it goes on. Have you tried updating ECS via CLI? It's largely confusing.
> Why use Terraform when CloudFormation maps better to the underlying primitives?
If only. It has gotten better, but historically CloudFormation has a lot of missing features and still do. Cloudformation can get stuck for hours and you'd just have to wait. The cross region support is terrible. Not to say Terraform doesn't have its quirks, but it's definitely not "worse".
And I disagree that AWS is less complex. Managing services across AWS is complex, K8s is as well but I would rather manage K8S on bare instances.
My devex team has developed helm charts we can use that automatically detect EKS, AKS or gcp K8s and configure the parts of an app to work with each environment, but the end users of the helm chart don’t really have to care.
I thought "surely somebody must have invented a better system for this" and NixOps was exactly that. Deploying config changes always took a few seconds with that, instead of 10 minutes.
> what resources did you find helpful for learning NixOS?
This was already in 2017 so documentation was worse than it is today.
On a flight I read the Nix, NixOS, Nixpkgs manuals top to bottom. I also read some of the nix-pills, but didn't like that they went so deep into the weeds of packaging when my primary interest at the time was OS configuration management. In retrospect, I should have read those also front to end to save some time later when packaging our own software and some specific dependencies became more important for us. I also read various blog posts, examples, and asked some questions in the IRC channel (now Matrix), where there were some people that simply knew every detail and were willing to spend hours sharing their knowledge (thanks cleverca22!).
I also read key NixOS logic source code, such as the `switch-to-configuration` script that switches between 2 declarative configs (like many, I do not like that this is written in Perl, and I'm sure it will eventually be switched).
A thing I did wrong was to learn too late how to write my own NixOS modules; I wrote our own systems as "plain nix functions" but they would have been better as NixOS modules, because those allow overriding parts of the config from outside, and make code more composable (see also https://news.ycombinator.com/item?id=41355203).
I spent 2 months prototyping all our infra in NixOps and learned by doing.
I also learned specifically where the gaps are: NixOS generally handles what's running on a single machine (with systemd units), and with e.g. NixOps you can access the global config of other machines (to render e.g. a Wireguard config file where you need to put in all machines to connect to, so {all machines IPs} \ {own IP}). It does not handle active cross-machine coordination, e.g. if some GlusterFS or Ceph tutorial says "first run this command on this machine, then afterwards that command on that other machine", or "run this command on any machine, but only run it once". So I learned Consul as a distributed lock service to coordinate (mutex) commands across machines. Luckily, the amount of software that needs "installation by human operator running commands" is continuously going down, declarative config becomes more of a norm.
With NixOS, a good thing is that while it is reasonably complex, it is simple enough that you can understand it fully, that is, for any given behaviour you _know_ where in the nixpkgs code it is. I recommend to use that approach (spend a few months to understand it fully), because it makes you massively more productive.
I also believe that this is a big benefit of NixOS vs e.g. containers on Kubernetes: Kubernetes is big and complicated, with likely more lines of code than anybody could read, and the mechanisms are more involved (for example, you need to know a lot of iptables to know how a request is routed eventually to your application code). NixOS is simpler (packaging software and rendering systemd units); it uses a more radically different fundament but in turn advanced features on top of it are straightforward (multiple versions of libraries on the same machine, knowing for every binary exactly which source code built it, running _only_ what's declared, automatic transparent build caching, spawning VMs that mimic your physical servers). NixOS provides less than cluster orchestrators like Nomad and Kubernetes (e.g. no multi-machine rolling deploys with automatic rollbacks), but one person can keep it all in their head, and it is very good at building things that run in cluster orchestrators. (Disclosure: I know much more about NixOS than Kubernetes; maybe Kubernetes disagree with me and think that a single person can understand Kubernetes source entirely to get the fast directed debugging I claim is possible with NixOS.)
Often, you also don't need a cluster orchestrator. Our Ceph runs straight on NixOS on Hetzner dedicated machines, it does not run in our Nomad. We use Nomad to schedule our application-specific jobs onto our machines -- that is, we use the cluster orchestrator for their original design goal (ball-packing CPU + memory jobs across machines), and do not use the cluster orchestrator as a "code packaging and deployment tool", which is what much of current Docker+Kubernetes is used for. We find that Nix is simpler and better for the latter.
Starting from NixOps, we Nixified all our our tooling (e.g. build our Haskell / C++ / Python / TypeScript with Nix), fixed things in nixpkgs in our submodule and made lots of upstream PRs for it (I'm currently at ~300 nixpkgs commits). NixOS works extra well if you upstream stuff your company needs, because it will reduce your maintenance burden and make other industrial users' life easier too. Especially recommended is to upstream NixOS VM tests for services you rely on; for example, I contributed the Consul multi-machine VM test [2], which automatically runs for any version upgrade to Consul in nixpkgs so nobody will break our infra that way.
Hope this helps!
[1]: https://flyingcircus.io/en/about-us/blog-news/details-view/t...
[2]: https://github.com/NixOS/nixpkgs/blob/72936c3bf6272f05922812...
But It will probably hire less then half of a US developer that then pisses all the other money away on overpriced AWS Servers, so that is right.
I actually agree with you that 90K gets you top-10% of the developers in many countries, but you were saying that 45K would get you a great developer with 10 YoE.
Devs with 10 YoE are already probably less than 20% of developers, and there are devs with 10 YoE who aren't "great" developers. Perhaps this is my own western near-sightedness talking, but I'm under the impression that great developers may constitute fewer than 5% of developers in some of these countries in question (I assume due to a combination lower English proficiency which is important for working with Western employers, different cultural norms which cause many developers from some countries to adopt anti-patterns for working effectively with teams, and more limited access to reliable internet and better equipment, which could result in lengthened feedback loops during their learning process).
But my primary argument wasn't that you'd be unable to get a great dev with 10YoE, it was that you'd be unable to get two great devs with 10 YoE for 90K/yr, or 45K each, or as tpacek pointed out, more like 35K/yr take-home.
So the person claiming you can get 2 "great" devs with 10 YoE for 90K/yr is effectively saying there are 2 "great" devs with 10 YoE somewhere in the world who are willing to work for like $35K/yr (or something in that realm less than 45K/yr).
I don't think the market has gotten that bad, but again, I'm not really paying attention to the job market right now.
That's not to say it can't be figured out! But people telling them that they should reduce hosting costs are doing a disservice.