When building evaluation infrastructure for generative AI systems at Gentrace [1], we needed a task queue that could handle thousands of tasks per day while remaining simple to self-host.

We considered traditional options like RabbitMQ and Kafka but wanted to avoid the operational overhead for our self-hosted customers. That's when we turned to PostgreSQL.

Using PostgreSQL's FOR UPDATE SKIP LOCKED, we built a task queue that supports retries, parallelism, and large payloads, all without adding new infrastructure. Sometimes, PostgreSQL really is all you need!

It's been running in production at Gentrace, handling 8-10k tasks daily with peaks of 45k during large-scale evaluations. The architecture is simple, reliable, and easy to maintain.

We decided to open-source a version of this implementation as Simple Task Queue. It includes:

- A task queue built on PostgreSQL - Type-safe task definitions and scheduling in TypeScript - Support for retries, parallel processing, and scheduled tasks

We also wrote a deep dive about how we built it on our blog: https://gentrace.ai/blog/task-queue-simplification

We'd love feedback! Try it out, and let us know how it works for you.

- Vivek

[1]: https://gentrace.ai/