Do you need separate systems when you already have Postgres?(postgresisenough.dev) |
Do you need separate systems when you already have Postgres?(postgresisenough.dev) |
All the eng. managers I've worked with in the last decade have sworn by Postgres, in the decade before that they were swearing at mongo and getting betrayed by Arangodb switching to a restrictive license and seeing other innovative databases going down the same path means for new side projects I go postgres.
Postgres doesn't do replication, HA or columnar/OLAP workloads, for example.
I'm thinking of Redis in particular. If you're using it as incredibly fast but not critical storage, it's trivial to set up and it ~never crashes or requires maintenance. It creates no headaches, and in exchange gives me a k/v store that I can thrash without worrying about performance (I know it's fast), downstream impact (am I slowing down critical-path SQL queries), etc. Especially in the age of LLMs, which I've found to be great at devops-type tasks, I feel slightly less compelled to simplify my stack.
I'm far more hesitant to throw a more formal MQ in the mix though... mostly from experience in that a lot of mid level ("senior") developers don't really understand queues very well at all. Even if conceptually, using tables for queues is more complex.
That said, I will use PG for workflows similar to K/V, Document (JSONB) and other structures over reaching for say MongoDB, etc.
It is not about ops cost in infrastructure, but ops cost in debugging consistency errors.
This has two implications:
1) make sure if you use Postgres for anything beyond core rdbms functionality that there is no dependency between the two, so you can rip out the additional functionality and move to a different platform when you end up needing to reduce the load on your db server
2) if using Postgres for non-essentials complicates your db backup workflow, risks the data integrity, makes it difficult to maintain or upgrade your Postgres instance (eg you have to wait months or years for compatibility with newer Postgres versions), or loads relatively shoddy or unstable code into the beating heart of your application, then you should either use a different Postgres server/install/container for these ancillary services or bite the bullet and introduce an alternative dependency, depending on which makes more sense.
An example of a table that would benefit from this would be rate-limits / concurrency-limits, which are commonly implemented using Redis instead of Postgres.
If you use ordinary READ COMMITTED transactions, they will advance their xmin horizons on each query (and allow old version cleanups) up until their own transaction id but unfortunately not beyond that. For a given table, this is important since that transaction is uncommitted and it might modify the table. If you could make long-running transactions readonly on those specific tables, then you could use a different xmin horizon specifically for those tables. It would require a lot of duplicative bookkeeping in shared memory though. You could probably fake this today by using 2 databases on the same machine and using two-phase-commit+dblink/fdw for cross-database transactions/queries (fdw uses repeatable read in transactions, so it won't allow the xmin to advance).
What I don't want is long running unrelated transactions keeping old tuples alive just it case they're needed. My third suggestion where transactions attempting to read old versions fail is probably better than the first two where isolation gets silently downgraded, since it avoid that problem.
> only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative
I love Postgres as a DB but, really, this is ridiculous. No doubt these extensions can do the job well-enough but you might as well invest in learning the right tool for the problem from the start, when the stakes are still pretty low. Why wait until, ahem, Postgres is pushed to the limit before you spin up a Redis cluster?
You don't get free opcost by using Postgres for everything. Arguably if you end up with a monolith of a database, you are paying a higher opcost (imagine if too much caching can affect all CRUD ops in your platform). Or you can manage a cluster of PG instances but that's no less complex---each plugin still comes with its own opcost!
No Silver Bullet, No Free Lunch, and all that. If your problem domain really warrants something outside of relational storage, you're gonna pay that complexity cost one way or another. You can't escape it by shoehorning everything in Postgres, fantastic as a DB as it is.
I have started just using Postgres to back queues. It is simpler (although I have spun up new apps with Redis so many times it is only a small improvement) but more importantly it is cheaper. I really do try a lot of stuff out, so completely removing a infrastructure piece is a nice money saver. Again, not massive but it's cheaper.
The downsides of doing this in the prototype/MVP context are minimal. At the scale of prototype and MVPs, I certainly don't see any difference in performance.
I did note in the graphic it listed Kafka but in the lower table graphic I did not see any Postgres replacement for Kafka. If I am at the scale where I really want Kafka, it is probably for performance and I just can't believe there's anyway Postgres could provide that.
So, I love the flexibility and all-in-one abilities of Postgres for prototyping and making MVPs. But at my job, nobody is proposing exclusively using Postgres for persistence.
Unless of course, you already know how to operate rabbitmq.
It was ok. Never really failed and didn't affect the user-initiated CRUD ops though ofc the bg jobs ran when the users slept. Still wouldn't bet my life on it.
SELECT FOR UPDATE SKIP LOCKED works great for turning Postgres into a job queue. It does not work the same way on CRDB and you will likely have to rewrite those queries or use an external job queue.
I've seen a few "Use Postgres for Everything!" posts lately. It seems to be fashionable. It reminds me of the Choose Boring Technology[1] thing from 2018 or so, but more specific to a database.
I think the ideas of "don't add unnecessary dependencies" and "ruthlessly evaluate tradeoffs" and "prefer simplicity" and so on are general and have very little to with postgres, so when I see things like "All you need is X" I roll my eyes a little, because these decisions are highly dependent on your use case, and taken as blanket advice it is generally _bad_ advice even if the underlying rationale is sound.
[1]: https://mcfunley.com/choose-boring-technology
ETA: I am going through their list and so much of this means that you are going to manage your own PG cluster and not take advantage of Aurora or RDS, which means you're already committing to a major tradeoff if you want to use a lot of these custom extensions.
Love FerretDB, but it doesn't really replace MongoDB's GridFS which is main reason why most people who are really using Mongo now day. Anyone knows a good replacement for GridFS?
I'm not in the "use Postgres for everything" camp, but only because I think it's too complex to be used like that. It should be replaced with a bunch of simple primitives in this role. No SQL and query planning magic please.
The problem of all the datastores is that are applications (like Wordpress) so you are too late to fix anything deep.
We need "frameworks" (so each sub-component can be used as-is or even swapped) and even wondering what a "system level" data engine could be.
I don't believe it's ever easier to manage updating two technologies than one.
Postgres is a good middle ground, though.
https://arango.ai/wp-content/uploads/2025/11/ADB-Community-L...
and it is dead to me. I want my head! I can accept GPL, Apache, MIT or some legit open source license. For my projects I see two possible paths which I want to have open: (1) building a commercial service on top of a database (like my RSS reader) where you can't necessarily draw a clear line between what is allowed and what is not allowed, for instance I have an adaptation layer that makes postgres look like the part of arangodb that I actually use (I do manually rewrite AQL queries into a DSL that extends AlchemyAPI) and if I did something similar over arango is this reselling? (2) an open source project where I want to tell people "go forth and use this code" and not have to hire a lawyer to know what they can and can't do.
Once a vendor has shown they have this attitude, I expect them to change their license for the worse in the future -- I just don't want to invest my time and energy in their platform.
It also has the effect of making software adopting such licenses getting removed from open source distributions.
First is this Oxide and Friends episode [1] where Bryan and gang explains war stories related to operating Postgres during their Joyent days and why they went with Cockroach DB for Oxide.
Second is this amazing blog from brandur which explains several issues with using Postgres as high throughput queue and some mitigations.
Online forums like Hacker news can be a bit echo chamber-y. It is always good to ensure that the people you are taking advice from are solving the same problem as you.
[1] https://oxide-and-friends.transistor.fm/episodes/whither-coc... [2] https://brandur.org/postgres-queues
The two critical questions IMO are:
1. At what scale do these issues arise? 2. Does Postgres make it harder to solve them when you reach that scale than another solution would have?
Some of that probably depends on the libraries or tools you use on top of Postgres but being able to easily swap to Redis or a proper queue at scale should mitigate the risk of starting with Postgres if you don't already have the scale.
I guess its more the rapid start-up mindset to get it up and running fast to sell the company, and leave the problem for someone else which is why a lot of our world is falling apart...
For example: I worked with Rails from 2009-2024 and I haven't come across a single Rails project in the wild that didn't have the queue du jour installed: sideqik+redis, delayed-job before, etc. And then since it's there, people just end up using.
> No Silver Bullet, No Free Lunch, and all that. If your problem domain really warrants something outside of relational storage, you're gonna pay that complexity cost one way or another. You can't escape it by shoehorning everything in Postgres, fantastic as a DB as it is.
So as not to bloat the system, you will end up bloating your Postgres installation instead.
Also, Unix principle/KISS: do one thing and only one thing well. Of course this comes back down to your definition of "one thing". "_Any_ Data Storage" seems to be your one thing but not for me. "Relational Data Storage" is a well-scoped one thing for me.
I'm not saying don't take advantage of the plugin system. But we must exercise good sense so as not to abuse it when we avail of its advantages. TFA's suggestion of "only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative" does not sound like good sense. It sounds like a personal and an organizational burnout waiting to happen. Wait until PG is at its limit does not sound like technical debt to you?
> If Postgres can truly handle all of these situations, then mastery of that one tool should be focused on.
Thus you have conceived the conundrum of The Database of Theseus: after how many plugins is your Postgres cluster no longer a mere Postgres cluster to the point where it is unreasonable for a seasoned/certified Postgres DBA to have a good grasp of the system in a reasonable time span?
This is almost textbook https://en.wikipedia.org/wiki/Inner-platform_effect in that this is the second paragraph of the "Examples" section. I acknowledge that Postgres' plugin system mitigates this somewhat but I just can't be convinced that all those plugins will play well with each other for all eternity and especially under stress (unless you make a PG instance for each sub-system, in which case, refer to my original comment).
For the record
- I've used PG for caching with UNLOGGED. Still had Redis for a message queue and other caching. - I've used PostGIS because it made sense to be able to make geo query when latlon is already a (small) part of your schema. - I've used JSONB but only because the alternative is Mongo. I still had an Elasticsearch cluster indexing all that data for search.
Like right now I am thinking about a system that has a password reset process and you need to keep track of a user id and a reset token, one or two timestamps, maybe a state variable and a flag or two. That's well under 100 bytes and the cluster size for a typical fs is 4kb or more plus there is the cost of the directory entry. If the OS is Windows it has to ask the Security Manager when ever you open it or delete it which is even more heavyweight.
It's common now for applications that handle lots of little "files" to store them as blobs in SQLlite! See https://sqlite.org/fasterthanfs.html
I am being sarcastic to show what a ridiculous idea it is to shoehorn everything into Postgres. If you find eschewing a relational DB for the filesystem incredulous, you should maybe reconsider if eschewing specialized software in favor of Postgres is advisable.
If you have bunch of files and don't have any structure, yes filesystem is great but the moment when you need consistency & performance (which you need sooner rather than later) use databases.
Investing early doesn't hurt when you build a product that you know will have many writers.
And if they aren’t calling themselves Open Source, then why do you care?
A better response to my comment could have been "just let it go" or something. ₍₍(˶>ᗜ<˶)⁾⁾
It's not that it's necessarily complex. But if you don't need it, don't use it. The business could use your time elsewhere
For a persistent store.. sure... but that's true for PostgreSQL as well, which has some pretty painful major version migrations by comparison to other options.
But I don’t have a problem with people inserting clauses to prevent Amazon from taking over. I don’t expect free work from people forever. If I’m going to use an open source project to build a commercial product, I would only do so if I’m ok forking and maintaining the project myself if necessary.
I don't have any fear that Postgres will get relicensed with a worse license than it has. But I see any relicensing or license that is more restrictive than a standard license as a slippery slope that makes me think "I don't want to invest in this platform" thanks to that experience.
I'm a software developer, not a lawyer. I understand standard software licenses and don't feel I have to re-read them or think too much about them. By using one and being dependent on systems that use them I feel like I'm reducing the burden on people who might adopt my open source.
The first is that there are many what most people would call standard licenses that contain tons of restrictions. Many of them like AGPL (and GPL to a lesser extent) aren't even fully understood by lawyers.
Second, when people are talking about standard open source licenses, they are generally talking about licensees approved by the OSI. The problem is that the OSI is primarily financed by tech companies including Amazon, Microsoft, and Google. They're not ever going to add a new category of licenses or promote anything that harms those companies business interests.
As far as relicensing being a slippery slope. There's no such thing as relicensing. Once software is released under a certain license, it is always available under that license.
Anyone can choose to license future work using a different license. That is functionally equivalent to the original authors of a project ceasing work, and someone else starting work on a new fork with a different license. Nothing about Postgres using a standard license prevents this (for what it's worth Postgres is released under its own custom "PostgreSQL License" even if it is very similar to other open source licenses).
What you are actually worried about is that someone stops doing free work, which is the practical consequence of "relicensing". If that's the case you need to look into much more than the license of a project, you need to look into size of the organization, funding, size of the community, time in operation etc... I don't know if operating under a non-standard license is even in any way correlated with an organization or person relicensing or any other form of ceasing development.
Otherwise it's the same trap, just one level deeper.