Skip to main content

What is Durable Execution?

Durable Execution is an abstraction of common backend engineering design patterns.

It lets you build production-grade backend system easily. Your system will have a simpler architecture that is highly scalable, cost efficient, performant, observable with comprehensive production operation tooling.

Why existing databases fall short?​

Today we have all kinds of durable database/storage — like PostgreSQL, MySQL, S3.

They expose APIs to persist data, and APIs to retrieve the data. The APIs are passive.

Usually, there are two main requirements when building a reliable backend:

  1. It should run on multiple machines for availability, throughput, and scale (not single point of failure)
  2. An API could be implemented as calling multiple other APIs, services. A single database transaction is usually not enough.

More specifically, a backend API usually:

  • Calls several APIs in sequence or in parallel
  • Schedules future work, like reminders or any business timeouts
  • Coordinate with other systems, like human input or general events
  • Handles all kinds of failures (may differently per step) with retry, rollback, DLQ (with manual intervention and recovery)
  • Needs good observability — inspect inputs, errors, and in-flight work at each step
  • Must stay reliable as system evolve and getting higher traffic, including high availability, linear scaling, acceptable & predictable cost, good latency & performance

None of them are provided out of the box by those passive storage or database.

Therefore today, engineers still have to write complex code to solve these problems, over and over and over again.

Example: process order by two steps​

Let's take the simplest example of order processing. There are only two steps, each step is a single API call.

  • charge the buyer
  • ship the item

For a prototype you might write in the code directly:

function processOrder(order){
    callChargeAPI(order)
    callShipAPI(order)
}

That works until any failure happens. Process or machine crashes after charge succeeds, or network times out at the API call. The order ends up with charging the buyer but never ship the item.

Fix 1 — database row + polling workers​

Then a smart engineer thinks of inserting a row (status = pending), return to the client, and let background workers pick up work.

We get minimal durability, but here are more problems:

  • How do multiple workers claim work without conflicting each other?
  • What if a worker also crashes after picking up the work, before the work is done?

Fix 2 — message queues between steps​

More experienced backend engineers usually evolve to use message queues: "charge-request" and "ship-requests" queue. So the order start with publishing to charge-requests, consume, then publish to ship-requests.

That solves the hard problems in Fix 1, but still, harder problems come:

  • Most queues cannot atomically commit “charge done” and “enqueue ship”. Queue 2 may succeed while queue 1 times out, so both steps run concurrently for an order.
    • Usually it ends up with using a database row as source of truth for dedup
  • Backoff retry and DLQ may not be provided out of the box
    • E.g. Kafka requires separate topics, SQS requires visibility timeout to work around
    • Need to build tooling for DLQ
  • If the frontend should return after charge succeeds (“payment received, order processing”), we need to be polling against a database, ugly and inefficient.

More over, as business evolve, it's very common that the order processing need to

  • If shipping API fails after retry, refund the payment to buyer (refund after shipping API fails for an hour)
  • Seller approval before ship, or providing the tracking number manually
  • A reminder to seller to approve or provide the tracking number

Those all require non-trivial architecture changes to support in this message queue design.

How Durable Execution helps​

Durable Execution provides the solution out of the box, without you writing a single line of code for those problems. You only write the business logic code for the real order process.

For this charge → approval → ship flow, you write something like:

  1. Step 1 — Charge with its own retry policy. When it succeeds, move to Step 2 atomically.
  2. Step 2 — Ship
    1. Wait for seller approval or a durable timer.
    2. When the timer fires or seller approval arrives, invoke the code to send a reminder and keep waiting, or ship the item.
    3. Ship execute with retry; on exhaustion, go to a refund Step.
  3. Step 3 — Refund process the refund to buyer.
  4. Frontend code calls an API out of the box, to wait until Step 1 completes as “payment received”.

The platform handles the rest all for free -- records each Step's inputs, outputs, and errors; operators can inspect and time travel, etc.

Isn't that amazing?

See Why Dex? for how Dex implements this.