Compared to Database

View as MarkdownView as Markdown

Most backends keep their state in a database and build each process around it. Below, one order process is built that way, one design at a time, and then as a Dex Flow.

The full order process charges the buyer and ships the item. It waits for the seller's approval before shipping and reminds the seller while it waits. It refunds the buyer when shipping still fails after an hour. These are the parts each design needs for it:

Database and queues

14parts you build and run

Tables

• Orders table with status
• Approvals table

Queues

• charge-requests
• ship-requests
• refund-requests
• Retry topics or visibility timeouts
• Dead-letter queue (DLQ)

Services and jobs

• Charge consumers
• Ship consumers
• Refund consumers
• Reminder cron job
• Approval endpoint
• Status polling endpoint
• DLQ inspect and repair tools

Runs on a database and a queue broker(not counted)

Dex

5parts you write and run

One Flow

• Charge Step
• Ship Step
• Refund Step
• approve RPC

Services

• Workers that host the Flow

Runs on Dex Server, which provides(not counted)

• Retries per Step
• Durable Timers
• Waits on Channels
• Client wait API
• Step history and rerun

What a database does not do

Databases and storage services such as PostgreSQL, MySQL, and S3 store data and return it when you ask. They are passive: they act only when your code calls them.

A reliable backend needs more than storage. It runs on several machines, so that one failure does not stop it and it can handle more traffic. One API often calls several other APIs and services, so one database transaction cannot cover it.

A typical backend API also:

• Calls several APIs, in sequence or in parallel.
• Schedules future work, such as reminders and business timeouts.
• Waits for other systems, such as a human approval or an external event.
• Handles failures with retries, rollback, and a dead-letter queue (DLQ) for manual repair. Each step may need a different policy.
• Shows the inputs, errors, and in-progress work of each step.
• Stays available, fast, and affordable as traffic grows.

A database provides none of this. Teams write that code again for every service.

The example: process an order in two steps

The order process has two steps, and each step is one API call:

• Charge the buyer.
• Ship the item.

A prototype calls the two APIs in a row:

process_order.pyPython

123

def process_order(order):
    call_charge_api(order)
    call_ship_api(order)

This works until something fails. If the process crashes after the charge succeeds, or the ship call times out, the buyer is charged but the item never ships.

CLIENTFrontend API

Returns after both calls finish — or fails halfway.

SERVICEOrder handler

charge(); ship(); in one request thread.

• Crash or timeout after charge leaves corrupted state
• Fine for a POC. Breaks in production.

Design 1: a database row and polling workers

The next idea is to insert an order row with status = pending, return to the client, and let background workers pick up the row.

CLIENTFrontend API

Insert row, return request id.

STORAGEOrders table

status = pending | charging | shipping | done

poll / claim rows

WORKER POOLWorker A

SELECT … FOR UPDATE

Worker B

same table scan

The order is now saved, but new problems appear:

• How do several workers claim rows without taking the same one?
• What happens when a worker crashes after it claims a row but before it finishes?

Design 2: a message queue between steps

Experienced teams usually move to message queues, one for each step: charge-requests and ship-requests. An order starts as a message on charge-requests. The charge consumer handles it and publishes a message to ship-requests.

CLIENTFrontend API

Publish to charge queue, return early.

QUEUE 1charge-requests

Charge consumer

Call charge API, publish to ship queue.

no cross-queue transaction

QUEUE 2ship-requests

Ship consumer

Call ship API, maybe write status row.

• The ship message is sent, but the charge ack times out: charge and ship both run twice.
• Step 3 means queue 3 and another consumer fleet.
• Backoff, DLQ, and approval waits become more queues and cron.

Queues solve the problems of Design 1, but they bring harder ones:

• Most queues cannot commit "charge done" and "enqueue ship" together. If the ship message is published but the acknowledgement of the charge message times out, the queue delivers the charge message again. The charge step then runs again while the ship step runs for the same order.
  • Teams usually add a database row as the source of truth to drop duplicates.
• Retries with backoff and a DLQ are often missing.
  • For example, Kafka needs separate retry topics, and SQS needs a visibility timeout as a workaround.
  • You also have to build tools to inspect and recover messages in the DLQ.
• The frontend may need to return as soon as the charge succeeds ("payment received, order processing"). It then has to poll the database, which is clumsy and wasteful.

The business also changes over time. The order process often needs to:

• Refund the buyer when the ship API still fails after retries, for example after an hour of failures.
• Wait for seller approval before shipping, or let the seller enter the tracking number by hand.
• Remind the seller to approve or to enter the tracking number.

Each of these adds queues, consumers, tables, or cron jobs to the queue design.

The same process as a Dex Flow

In Dex you write only the business logic. Dex provides the retries, timers, waits, crash recovery, and the record of each Step.

CLIENTFrontend API

Starts the Flow, then waits for the Charge Step with the Client API.

DEXDex Server

Records each Step result and the move to the next Step together.

Retries per StepDurable TimersWaits on ChannelsStep history and rerun

sends each Step task to one Worker

YOUR WORKERSWorker A

Charge, Ship, Refund Steps

Worker B

Same Flow code

Steps call the APIs

Charge API

Ship API

Refund API

The charge, approve, and ship process becomes:

1. Charge Step: charge the buyer, with its own retry policy. When the charge succeeds, Dex moves the order to the Ship Step. Dex records the result and the move together, so a crash cannot leave the order between the two Steps.
2. Ship Step:
   1. Wait for the seller's approval or a durable timer.
   2. When the timer fires, send a reminder and keep waiting. When the approval arrives, ship the item.
   3. Retry the ship call. When the retries run out, move to the Refund Step.
3. Refund Step: refund the buyer.
4. The frontend calls a built-in Client API that waits until the Charge Step completes, then shows "payment received".

Dex records the inputs, outputs, and errors of every Step. Operators can inspect each order and, after a fix, rerun it from an earlier Step (time travel).

Problem	Database and queues	Dex Flow
Crash between two steps	The buyer can be charged while the item never ships	Dex records each Step result and the move to the next Step together
Several workers	You build row claiming or rely on queue delivery	Dex sends each task to one of your Workers

When requirements change

Each new requirement adds parts to the queue design. In Dex it adds at most one Step or RPC and changes the Ship Step.

Database and queues addDex adds

Refund the buyer after an hour of ship failures

Database and queues add

• refund-requests
• Refund consumers

Ship consumers record the first failure and give up after an hour.

Dex adds

• Refund Step

Ship Step: retry for one hour, then move to the Refund Step.

Wait for the seller's approval before shipping

Database and queues add

• Approvals table
• Approval endpoint

The approval endpoint, not the charge consumer, publishes to ship-requests.

Dex adds

• approve RPC

Ship Step waits for a message on the seller Channel.

Remind the seller

Database and queues add

• Reminder cron job

The job scans the approvals table for orders that still wait.

Dex adds

Ship Step also waits for a Timer. When it fires, the Step sends a reminder and waits again.

5new partsDatabase and queues2new partsDex

Why Dex? shows this Flow as Dex code, and the Quick Start runs it.