How We Built Fly Postgres(fly.io) |
How We Built Fly Postgres(fly.io) |
As a proof, check out fly.io forums. They are full with posts about suddenly broken Postgres instances.
So should many of the people in our forum. This is part of why we wrote this! That said, the vast majority of PG users on Fly have no issues. Most issues are the result of people trying to spend as little money as possible on their underlying infrastructure. When you have a 1GB disk, you'll run it out of space pretty quickly. You may not pay us for it, though, which is what they value most.
We tell people to use CrunchyBridge _all_ the time. It's better than our Postgres for many people.
1. https://webstack.dancroak.com/
2. https://github.com/croaky/webstack
*Disclaimer I work at one of those fully managed database providers.
I could see a Crunchy Data managed version of Fly Postgres's doing super well. Combining the best aspects of both your experience managing HA Postgres and backup, with the distributed read replicas from Fly.
I assume you use SSL, and these are not made on demand (connection pooling ?) On 'cold start' do you get massive latency ?
What kind of latency is there between client and server ?
What made you choose Heroku and not a specific managed postgresql service ?
Did you try flys postgres service?
What is the HackerNews opinion on: who is their actual customer?
Obviously lots of companies pay for managed databases. It's not an uncrowded market for a reason.
But like... pricing wise... it seems so expensive? What is the HackerNews take on the valuable proposition specifically for hosted databases? Is the answer basically 1:1 with anything cloud hosting related?
Thank god someone is. I’ve lost more of my life to consul partition failures that’s any other part of the nutech stack.
This stuff, for all the hype about raft and things, is brittle enough and requires enough special attention.
Which is why I’d gladly rather have it be fly.io’s problem.
The reality is: we've got a fair bit of experience with Consul at this point, we respect the hell out of it for the problems it was designed to solve, and we're unlikely to stretch it any further than we've already stretched it. Distributed lock service for Postgres clusters? Sure. Source of truth for all our app state? We've built our own thingy ("corrosion", a Rust distribute state system) to phase Consul out with. I'll get Jerome to say things about it.
But to my actual question: I remember seeing somewhere that Fly Postgres is not a managed database and isn't as fire-and-forget (somewhere in the docs), and that honestly scares me a bit. It shouldn't, because at the end of the day, RDS is probably downright hairy and ugly under the hood, it's just hidden from us.
I've also seen a handful of reliability issues on the forums around postgres. So what is Fly's position on actually running these clusters? Are you guys feeling pretty good about its stability for critical workloads?
On a related note, these folks that are continuing to use RDS via fly... what are the latencies like with a us-east-1 RDS? I know you're in Ashburn so it must be sub 5ms?
That itself would be worth keeping our RDS and moving everything else to Fly to be honest.
I'm getting 7ms RTT on us-east-1. Of course I messed up my setup the first time and had RDS in Ohio... That was 30ms.
You would probably get sub 5ms if you did public RDS connection. Don't know if that would ever pass an audit though.
Solid and boring is often a good choice. I'm glad to see startups in this space.
What's the latest on adoption of Spanner-like databases?
While not Spanner, it is essentially an open source db like AlloyDB or Aurora, pushing replication and scale out to the storage layer (in this case via FoundationDB). The most interesting bit of mvsqlite is it's multi-writer capabilities, using FoundationDB to perform page-level locks.
I'm neither the creator nor using it in production, but I'd love to see more DBs using FoundationDB as storage. It's a pretty cool solution.
The lead developer on mvsqlite has since joined Deno...
Given that the deploy model of a jettified jar is so clean in comparison to the various hipster stacks ;) that they do have primary support for, I'm not sure why they don't have much in the way of documentation for it. I did find a support thread that refers to https://archive.is/o9YE1, which seems like a fairly gross and opaque sequence of steps.
@tptacek, what are the chances that somebody at Fly could produce a more streamlined recipe and/or documentation for JVM apps?
Am I supposed to be looking for serverless deployment? Deploy a master/parent version of the app on one main VPS then several sub versions on other VPSes distributed around?
I understand why Fly doesn't want customers to expose the database to the rest of the internet (ecosystem and such), but am very happy that Railway allows it.
Very easy to setup stuff. Would recommend using prepaid credits because there's no capped billing.
Getting closer and closer to heroku deploy and auto config handles everything.
Still love the company tho but can’t support it more :/
If you come from Heroku, we seem too cheap: https://community.fly.io/t/comparison-of-prices-to-heroku/89...
If you are most familiar with AWS, we seem pretty close to what you're used to. But very cheap for egress.
If you are a DigitalOcean user, VMs seem pretty ok, but bandwidth seems more expensive.
If you are a CloudFlare user, you think everything else is too expensive until they try to sell you an Enterprise plan.
If you're a happy user of Hetzner or OVH, you pay something close to the same price as we do for servers. And might be surprised at our prices. Because we also want to have reasonable margins.
People really like us when they have an app that seems valuable to them, they don't want to think hard about running servers, but they do know they can "eject" and run the rest of their infra too: https://community.fly.io/t/startup-credit-program/6709/3?u=k...
I am curious about the freemium model for PaaS systems. I've always wondered what percent of compute ends up being free and if the paid prices have to be higher to subsidize the free tier. Would it be better for the paying customers if the service was 30% cheaper and there was no free tier? Of course, I might be incredibly far off on how much the free tier customers cost.
I think for people that think Fly.io is expensive, it just feels like what Fly.io does should be table stakes rather than a premium service in 2022 - and yet it's so hard to find! Heroku is 15 years old and Fly.io feels like the first platform I've used since that just gets it.
I would say that a collaboration between you and Neon (https://neon.tech/) would be pretty cool. While your site does link to Neon as a recommendation, Neon's datacenters often aren't that proximal to Fly.io's - Ohio isn't that close to Chicago, Virginia, or New Jersey. Maybe that'll get better in the future.
I'd always love it if Fly.io were cheaper, but more than that I'm glad that Fly.io seems to really get what customers need.
tbf, Cloudflare's Enterprise plan includes usage, too. Just like Fly's own plans: https://fly.io/plans
Besides, the Cloudflare platform is way more reliable and their products actually make it to GA :)
But then why not just use AWS App Runner and keep everything in AWS? No egress costs and you get the best support in the industry on legitimately enterprise-grade infra. Don't like AWS? Ok, GCP has Google Cloud Run.
Why would I ever choose Fly when the competition already has a Fly-alike with global infra spend 1000x yours?
There are trade-offs of course (https://onlineornot.com/self-hosting-vs-managed-services-dec...) between the two options.
I've been hoping fly.io builds a strong automated middle ground for a while now!
Still more than 2x the cost, but not nearly $760
Anyone who wants a managed database + to run Docker contains on VMs without having to do much ops work or deal with the complexities of the big clouds.
> But like... pricing wise... it seems so expensive?
It's quite a bit cheaper than Heroku who run a similar service and have bootloads of customers.
At what cost though? Why not run a VM that runs Postgres? It might take a week to set up in terms of automated backups, cluster, failover, etc. (ok, maybe a few more weeks) but hosted DB costing 5x the underlying VPS is insane?
Some people certainly do work for companies like that. I'm sure big places like Facebook or Apple or Netflix applaud and promote people for this - and have the scale at which having people working on these problems makes sense. At your startup with a few dozen people? Probably not. If you're not building the product, you're not helping the company succeed. Ok, you're saving the company a bit of money, but at the cost of your time and the cost of the company actually getting product out the door, finding product-market-fit, etc.
Do you want to use your employee time setting up a HA database cluster and saving the company $1,000 per month or developing your app?
That's a key question: pay Linode $1,560/mo for a 3-node 32GB RAM cluster or launch 3 32GB boxes for $720, figure out PostgreSQL replication, make sure you setup the replication users, make sure you don't open any security holes, make sure you have Patroni or Stolon so that you can switch over when the primary fails, make sure you have etcd or Consul or something to handle that coordination, make sure that you have Barman or pgBackRest setup to take your WAL and persist it to S3, setup your S3 buckets, setup a full backup schedule so that you can restore easily, make sure that you're regularly testing your restores, make sure that you're testing your failover (do you even know if Patroni is actually working?), figure out how your app is going to cut over to the new primary when that happens (is there a shared IP that you need to move, are you using a proxy like HAProxy that's checking health-checks to see which it should proxy to, etc). Or would you rather just pay someone $1,000/mo so that you don't have to deal with that?
I hate paying up for something I feel like I should be able to do myself, but it does make some sense. If I decide I don't want to pay for Google Cloud Run and my servers all die, I can boot up some new boxes and get my app running again with some downtime. That's not great, but recoverable. If I don't want to pay for Google Cloud SQL and my servers die, now I'm hoping that my backups were working, that I can bring a much more complicated deployment back online than just some random process or container, etc. One of those two just carries more risk. Yes, backups should work and should be tested and you should even test backups in a managed service, but if you're a startup trying to move fast and find product market fit, I'm guessing that the premium is worth saving your engineers that time. I hate saying that because cloud providers are pushing such high margins, but it's probably true.
As a curiosity, do you run your own databases? If so, which? Do you find that everything Just Works or that it's a pain to get everything running, debugging things, testing backups, etc.? Is this in a high-traffic, commercial situation or just as a hobby? I think hosting your own database is relatively easy if you're just going to have one server and pg_dump -> rsync a backup nightly. In the rare event that you lose a server, maybe have an hour of downtime. If you're able to recover within 50 minutes and you lose a server every week, you're still at 99.5% uptime. If you can recover in 40 minutes and you lose a server every month, you're at 99.9% uptime. Do we need more? How often does a VM go down (note, don't say "I have 1,000 servers and I lose one every other day" - that would mean the average instance is lasting several years)? Won't Google's live-migration of VMs handle a lot of that?
So, it's trade-offs, but I think we're not living in a world where companies say "we're not going to architect for HA and we'll suffer an hour or two of downtime every year or two when we lose a box." Sometimes we certainly get ourselves into situations where we've made complicated systems that end up having complicated failure scenarios too.
Still, I think managed databases are an easy sell, even at their premium price point.
Little bit off topic: It's funny that all the time you see comments on HN saying something along the line: why using Kubernetes, it's just overly complex. But after that are, most likely, returning to their self managed DB clusters that need all that maintenance and setup just listed. While, if they were using Kubernetes, they would have Operators or Helm Charts that do 90% of the stuff.
(Now back to topic) Don't get me wrong, I'll chose a managed DB all day every day over something self managed but sometimes (e.g. startup without a indefinitely amount of VC money) self management of infrastructure is the fastest and cheapest way to go.
Except Fly Postgres isn't a managed offering unlike, say, CrunchyBridge or PlanetScale or Alloy or Aurora. It is pretty much a "Fly app" you'd have to tend to yourself.
Different companies or even teams within different companies will have different risk acceptance. The thing with managed services, is part of the premium is your getting all the bells and whistles... but that may not be aligned with what customers need, that paying the managed premium is buying them.
So I suspect the important thing here is that customers realize what they're getting and have proper expectations set. In this case, unless I'm missing something, that you're not getting a managed database service from fly.io, you're getting an OSS tool that makes running postgres on fly.io a bit easier. Kind of like a database controller for kubernetes... helps you automate some things, but it's still just software running on your cluster.
And then it's up to those customers to decide, whether that's acceptable risk to them or not. Maybe a bunch of customers get to vote with their wallets on whether this works for them or not. Maybe there will be enough demand where some partner specializing in database tech like neon, crunchy, cockroach, etc will have a service targeting fly.io specifically, or maybe fly.io will get stuck building it themselves if customers demand it.
Lots of maybes, so at least I'll be interested to follow this and see how it develops.
There's also security to think about (do you run a CA? how do you handle Postgres certificates, etcd certificates, rotation, revocation, etc).
Some hosted providers also have nice value-adds like query-level performance monitoring and DBA services
Can anyone tell me I should or shouldn’t trust I’ll be ok?
Feels like a better test would be for the healthcheck to time the query RTT and report that back. Completely remove the web request from the equation.
Looking forward to the blog post!
Note that this is still the on-demand price.
Although it's still not Apple to Apple, no one (sane) would useRDS 24x7 without RI. That probably brings you down another 50 bucks or so.
Yes. I wrote: "I was recently just one of those affected by a random issue!"
I don't notice any measurable latency. Heroku exposes their postgres instances directly on the public web (secured with a strong user/pass combo of course), so there's no difference, networking-wise. There's no additional reverse proxy or some tunnel to route through.
Heroku was the cheapest for the very small instance I needed.
The connections are made on-demand and are insanely fast.
And in general, nomad worked pretty well for us but our consul cluster kept mysteriously failing. I think that caused our nomad cluster to fail because it was backed by Consul.
The one complaint I did have about nomad (same as consul) was that the recovery process was manual where you had to manually generate a peers.json.
I was shocked when I saw that. Truly one of the "finding how the sausage is made" moments even though I've managed linux servers for two decades – I always assumed it would use zeroconf/bonjour/multicast DNS (remember cloud auto-join?) or something similarly elegant to auto discover other nodes in the network and just reconnect and rebuild a cluster. I mean what's the point of all this stuff if it can't be used to recover a cluster and just Do The Right Thing™? The shiny new experience is stellar (like sales, or setting up a new cluster), but the flip side (when things go wrong) is a mess. That's why we eventually said "nope!" to all the custom stuff and went with boring, plain vanilla ECS, which is itself too much now that we've started using fly.
Don't ever want to even think about having to hand-write a peers.json file to recover a cluster, boot things up, and pray to the ancient gods that it works.
We don't have time for that nonsense. Please, take my money, Fly/Render/everyone else. Your costs are a margin of error compared to what I had to pay a devops person to build our own stack. (I'm not even exaggerating. It was six figures. DevOps people are worth every penny but they cost many, many pennies.) Ultimately, we never used the infra.
I want to focus on building solutions for my customers and not fiddling with weird server stuff.
The consul clusters would keep failing in QA (they were running on t2.nanos – but that should be plenty of bandwidth for raft not to blow up every couple weeks, same happened with t2.micros too).
Before we pulled the plug we had started seeing something about ec2 health checks failing and the autoscaling groups yanking servers out and replacing them with new servers. but this is exactly the kind of case where consul should've just added the new machine right in. Instead, the 3 node cluster (now 2 nodes) would just sit there saying "hey I can't find a leader... aaaah. I can't find a leader" – well, to paraphrase Mike Myers on SNL, TALK AMONGST YUHSELVES and figure it out, there are two of you remaining.
[edit]
Also other managed DB hosts have similar anti-funnels for pricing, so I'm not trying to pick on Crunchy Data.
Cheers
2. The 3rd item on there goes to the "pricing" page that doesn't actually tell you the pricing (it says "Starting from $10/mo" but you need to scroll down and click through to get to the actual pricing table) :/
3. The 4th item "pricing calculator" is great; I just didn't find it.
My last gig had an MSSQL database component that pushed $700,000 a year.
It wasn't _that_ big but it was pushing the upper published limits of what you could do with an MSSQL RDS. One day replication stopped working, and amazon business support of whatever couldn't resolve it.
I've been around the block and have continuously come to the conclusion that it's usually better to have the skills to run them yourself, and then it's just a balance of whether or not you're time or cash poor.
I think the people who are doing this calculation wrong are mostly confusing and conflating those two outcomes. Just because your database servers are very busy, does not mean your product achieved some commercial success. You can't just pay extra for the full-service treatment and expect to receive a commensurate output in value,
(and the people who won't pay for a real person to own their databases full-time are likely in a circular Venn diagram with people who won't pay attention to optimizing their database queries, so your point can probably stand unmodified... except there is an important case where the PaaS provides value, and it's nearest to the starting point of practically everyone.)
If you look at every dollar spent on IT/databases as a sunk cost which cannot be recovered from future recurring expenses, it's very bleak indeed to put pennies on top of pennies and pile them higher every day... or you could pay attention to the signal that SaaS is providing every month (the bill for usage), and ramp up the attention paid to optimizing queries before it gets too late in terms of dollars and cents? The dollar impact of slow queries cannot be understated.
But some business ideas will also not ever make it that far. You could spend the money on DBA salary or DIY and never know how much it was really being wasted, comparatively, if you didn't ever get a handle on your own usage metrics.
Opportunity cost and all that.