When building our open-source reverse ETL platform, built on Ruby on Rails & Postgres, as part of the ETL process running SQL queries in background and extracting data from data warehouses was critical and we needed a way to handle long running workloads.

We wanted something that was durable & that could handle retries, our initial thoghts were to use Rails Sidekiq, but the lack of durability and the need to handle retries made us look for other options, also we didn't want to create a dependency on Redis.

When we first did a POC with Temporal IO, we were amazed by the performance of running long tasks using Postgres as data store, we were able to benchmark by running a workload that tool 3-4 minutes to complete and processed 100K records from a data warehouse. Of course, we had to make some changes to our code to make it work with Temporal IO, but it was worth it.

it also provided a nice looking UI to monitor the Syncs and the ability to retry failed tasks, which was a big plus for us.

If anyone looking to build a long running workloads, I would highly recommend Temporal IO, it just works.

Link to our Github Repo: https://github.com/Multiwoven/multiwoven