Temporal raises $550M at a $12.55B valuation(temporal.io) |
Temporal raises $550M at a $12.55B valuation(temporal.io) |
Durable Execution offers three key benefits:
It improves application reliability by providing fault tolerance.
It simplifies code by allowing it to focus on the goal instead of potential problems.
It accelerates development by eliminating the need to write complex error-handling logic.
Why is it such a big deal? Genuine question, not trying to be snarky
You need to reserve some inventory, charge a card, eventually email a customer and steps can go wrong. So you have queues, and retries and ways to back things out, undo changes.
To my understanding, Temporal's idea is to factor that part out, the queues and retries, and offsetting actions, so you write the logic and not the workflow orchestration.
Basically, a multi-step event handling system, where the data could be spread across multiple databases/systems, so there's no way to rollback a transaction atomically across all the databases. Instead, you explicitly codify "compensations" to undo your previous commits so that you eventually end up in a consistent state.
Temporal is the orchestrator/framework/library to implement the above in an easier way.
In a past job, they tried to implement a similar thing on much lower scale with bidirectional database migrations.
Fortunately, they were mostly ran in one direction.
In Temporal, you use their SDK to mark which code is either:
- Deterministic without external dependencies on network, disk, clock, etc.
- Non-deterministic (e.g. accesses a filesystem, dependent on clock time, talks over the network)
You can write the code without handling flakiness or retries and the Temporal control plane handles all the progress tracking and retries for you. Progress is tracked at the individual line of code for deterministic code, or for non-deterministic code, tracked at function boundaries defined by the programmer. You buy into more complexity upfront, but it makes the application code way simpler and easier to manage overall.
They do a lot of other cool stuff on top of all this, and their Temporal Worker Controller architecture is particularly well suited to running massive scale processing/AI workloads on Kubernetes (handles a lot of stuff like autoscaling without interrupting work, rainbow version rollout, etc.)
temporal centralizes all the error handling, retry handling, ... some web pages to manage failed, pending tasks.
it's been good for us, but a real step function in complexity of the app.
Functionally, it's a "workflow" runner (ex - you can mostly treat it like a queue, where you've got workers that are picking up work to do).
But it wraps a couple of pretty handy features on top like:
- It preserves most arguments to actions, and it makes system details deterministic for retries (ex you can re-run a workflow at a later time, and temporal will make sure the code sees details like date and time as though it were the original run, and will yell at you if you try to write code that won't be deterministic on retries)
- It supports very long waits easily. (ex - very easy to have a workflow do a couple things, wait a week, then do some more things).
- It has decent profiling and UI tools
- It can "restart" a failed workflow deterministically from the step at which it failed (using details from the original point)
---
Basically - it's a background worker service that's put a lot of time and thought into ways to handle failures better.
It absolutely still has some considerable pain points though, and I find it difficult to use for larger tasks (ex - their message gRPC size limit of 4mb is a b*&^% to work around, since it often breaks a lot of the utility they provide, and the history cap at 50mb is also really painful in certain situations.)
Really - I think it was just the right tool at the right time to make calling LLMs with long waits relatively easy and somewhat foolproof.
Also having a unified control plane is handy even at a pretty small scale if your alternative is going full “cloud native” on hyperscaler microservices. The ability to see what’s happening when and where across a single workflow run is a dream compared to all the traditional logging approaches I’ve seen.
It's used by OpenAI.
Yes...with eventual consistency.
If your process OOMs, if the machine restarts, etc.
Plus some other distributed execution advantages.
We're using temporal as the foundation for agents running on our platform doing accounting work for clients, it's proven extremely effective and I love how durable workflows are within temporal.
+100 on the technology, but realizing they are also VC funded makes me a little more cautious when it comes to vendor lock in.
Congrats to Temporal! They'd just previously raised in February, https://temporal.io/blog/temporal-raises-usd300m-series-d-at...
but the money being raised - what is it for ? go to market ? sponsorship & I like that temporal sponsored Crystal Palace.
we've so many good durable execution engines.
this is the type of VC pollution that will cause companies to end up being acquired by an Italian silverware company (bending spoons).
Buying DRAM?
A few key differences. Restate has a more flexible programming model. You don't write workflows with activitities, but just durable processes/handlers. Durable steps execute inline and get persisted over an open streaming connection in Restate (low latency, lower overhead per durable step, sharing resources like sandboxes) instead of working with a pull-model where each activity executes remotely on a worker. Restate has a lean deployment model with a single binary that can be deployed multiple times to have a highly-available cluster (potentially spread across multiple regions). It is used for large-scale production clusters, and so lightweight here does not mean less reliable than Temporal.
You can do the same things with Temporal like sleep for months etc. You can learn more here: https://restate.dev/vs/temporal
Temporal has 3 types of activities:
* local: executed in the same process as the orchestrator code. Many local activities can be executed locally before their results are sent to a backend server in a single RPC call.
* task queue based: executed by a pool of worker processes that poll from the queue. This is the most flexible model as it supports flow control, priorities, fair queueing out of the box.
* eager dispatch: task queue based but executed locally if possible as a performance optimization.
Temporal also supports stand alone activities that are invoked without a workflow and dispatched through a task queue.
Restate only supports local activities (using Temporal terminology).
I wouldn't call it a "more flexible programming model".
Restate made several decisions I consider questionable for the system's availability and stability, like pushing work to handlers instead of dispatching it through a queue. Any design decision has tradeoffs. It would be nice if you mentioned these trade-offs in your posts instead of making claims that sound like pure marketing.
Naively without digging into the code I would look at a “streaming over an open connection” as likely to strictly more brittle.
Sure, Temporal is more mature and battle-tested, but Restate is quite nice and I honestly prefer it in most cases.
EDIT:
I started drafting this reply before seeing someone from the Restate team chimed in.
I'm sure there is a place for restate.dev, but it isn't in the same place as Temporal.
Like Temporal, Restate supports workflows that sleep for months at a time and let's them reliably finish.
It actually does many more useful things. For example, you can have services that remember state across invocations and there is no need for continue-as-new.
Since Restate allows inlining durable steps into your workflow, it is very easy to co-locate your workflows with expensive resources such as sandboxes or other per-node resources. And the nice thing is that each durable step is really cheap and adds only minimal overhead.
Restate's spiritual father is Stateful Functions (https://nightlies.apache.org/flink/flink-statefun-docs-maste...) a library for event driven applications built on top of Apache Flink. If you want to learn more about why we started building Restate, I recommend this excellent blog post https://restate.dev/blog/why-we-built-restate.
I understand that Restate and Temporal look similar from a superficial perspective but Restate is not only a durable execution engine but a durable runtime that also provides consistent state and reliable communication. These are the building blocks to build reliable agents and applications w/o having to fit them into a workflow-activity like model. If you want to learn more about how Temporal differs from Restate, check out https://restate.dev/vs/temporal.
That is, they require good taste and opinionated API design first, porting skills second. The SDK for each language ends up being substantially different.
Frontier models are not very good at this yet. I've tried. Ultimately I couldn't justify the effort when there are other workflow engines out there.
Honestly I never did come to a satisfying conclusion on whether I thought it was worth it. The teams I worked on that used it found it neither simple nor easy to use, and I was never sure whether we were really reaping the benefits of correctness in the face of failure at the scale we were running. We eventually migrated everything off of it, and everyone was happier, but perhaps (probably) we had more lurking bugs in distributed failure cases. But to my knowledge we never tracked down an incident to the kind of problem that temporal solves.
I guess I'd say that I like durable execution (or at least temporal specifically, it's the only system like this that I've used) in theory, but not really in practice.
You can scale the workflows up and down to as complex or simple as you want for whatever your business is. The example above is one I’ve seen, but it works on a smaller scale as well. But you have to write the implementation code in very specific ways to get the benefits, and the overhead ends up being very difficult to plan and reason about, and a lot of business logic actually gets hidden amongst the weeds of the tool’s overhead.
The overhead of doing something in this way is quite large, but, as this handled literal money, it was definitely worth it.
Unfortunately it appears someone vibe-coded a port of the Java SDK for .NET and namesquatted the `Dbos.Transact` package on NuGet [1], making it look like an official SDK. It's also not a very good port, mirroring the Java implementation instead of using idiomatic C#.
If you ever plan on releasing a .NET library, I suggest opening up a dispute resolution [2] with the owner (it really is just an email to the guy, CCing NuGet support) and reserving the `Dbos` prefix [3].
[1]: https://www.nuget.org/packages/Dbos.Transact/0.0.0-alpha.0.4... [2]: https://learn.microsoft.com/en-us/nuget/nuget-org/policies/d... [3]: https://learn.microsoft.com/en-us/nuget/nuget-org/id-prefix-...
While the workflow is actively doing work (like calling other services, accessing state or interacting with external services, for example) the server is connected to the workflow deployment via a low-latency bidirectional streaming connection to receive and acknowledge progress that the workflow makes. That way the workflow can finish as fast as possible.
A nice side effect of this model is that you can co-locate your workflow with expensive resources, such as a sandbox, which should be used by all durable actions that the workflow executes. The reason this works is because Restate can inline durable steps (what would be modeled in Temporal as activities, I believe).
So you have fast persistence of events while a handler can make progress, and suspensions while waiting.
While the durable function does fast work and adds steps, it pushes it through a stream. When a wait point comes, it closes and replays on resumption (typical durable execution style).
That gives you the best of both worlds: same long-running workflows with long sleets and suspensions, but also ability to add steps with few ms overhead only.
Temporal is fully open source, you can run the entire thing on your laptop or your own server. People pay for the SaaS anyway, because dealing with the distributed computing problems is hard.
(There are some other projects inspired by Temporal as well, but if you want to run something in-house, you're probably funding a minimum of 2-3 engineers to handle on-call, and for that price the SaaS is competitive).
(1) You can model the equivalent of local activities and activities that run on other workers in Restate.
A local activity is a step in the workflow function. An activity supposed to run on a different worker is a function called by the workflow function. Since these calls are just Restate events, exactly-one, suspendable, this gives you a full-fledged workflow/remote-activity pattern. Including concurrency, separate retry policies, etc.
(2) Restate steps commit individually, unlike local activities.
Imagine a two-step workflow, where you want one step durable before starting the second. Account withdrawal before deposit. Restate steps allow you to do that, each step is durable committed before the next step runs.
Per Temporal's own docs, Temporal Local Activity results become durable only when the enclosing Workflow Task completes. That's different than Restate, which can durably commit every individual ctx.run before proceeding to the next step.
Making actual durable commits fast, so you can have sequences of fast durable steps building on each other is super valuable. If an agent can commit the guardrail evaluation in low milliseconds before it starts the tool call, that's great, do it. If committing this involves dispatching another workflow or activity task, the consideration is harder.
When we see someone migrate a Temporal workflow, they often end up using many more durable steps in Restate than they used activities before.
(3) Why do we consider it flexible?
(a) Virtual Objects: Keep state around across the workflow, without doing tricks like "keep the workflow running, signal only, continue_as_new" after a while. Virtual Objects are a natural way to model concurrent stateful entities.
(b) non-workflow communication patterns: We have seen users build lot's of different patterns. It can get as crazy as graphs of functions/objects sending each other durable messages. All end-to-end idempotent (or exactly once, for Virtual Object state). That is outside the hierarchical workflow/subworkflow/activity abstraction.
(4) availability and stability
I don't know where the perception with stability comes from, Restate pushes some pretty high volumes for customers, like 100k+ actions/sec.
The push model in Restate is internally dispatched through queues as well. The application just don't see it as task queues. Limits are implemented through virtual-queues in Restate 1.7.
-----
Of course the systems make different trade-offs. The Restate design (vqueues, push model) is took us longer to build than a task queue model would have, because it puts more work onto the dispatcher that is otherwise handled just implicitly by the worker pools.
But once it is there, it is sooo nice, in how easy it integrates into infra, and how it can handle flow-control with high hierarchical limits in ways I genuinely haven't seen in any other system achieve (see https://restate.dev/blog/announcing-restate-1-7 )
Stay humble / nice ego dude.
Necessary reference: https://share.google/7uReiRkd6mY3RcCQQ
I certainly don't think I'm superior to them in any way and have great respect for the very capable team they are. We know each other personally. I even presented at the Flink conference while at Uber.
My message is that I'd prefer a more technical discussion of the merits of our products rather than simplified marketing attacks.