Failure and redelivery

View as MarkdownView as Markdown

Retries and failure routes handle attempts that fail. Some failures leave Dex without knowing whether the work ran. A Step with an external effect that must not repeat, such as a payment or an email, has to plan for these cases. This page states what an application can rely on.

Heartbeats are not acknowledged

A heartbeat call returns once the Worker has queued or sent it. It does not wait for Dex to store anything. Python's heartbeat, Go's RecordHeartbeat, Java's and TypeScript's recordHeartbeat, and Rust's record_heartbeat all behave this way. A returned heartbeat call does not prove that Dex stored the checkpoint. This holds for SYNC Steps too, even though a SYNC Step runs user code only after Dex made the attempt start durable.

Heartbeat checkpoints are lossy. Dex batches them and keeps only the latest value of each batch. A checkpoint can also be lost when the attempt ends, such as on a timeout or a cancellation, or when a Dex Server node fails. After an interruption, the next attempt receives the last checkpoint that Dex stored. That checkpoint can be older than the last one the Step sent. Do not use a checkpoint as the only record that an external effect happened.

An attempt starts when Dex hands it to a Worker

A Step attempt starts at the moment Dex hands it to a Worker. Dex first reserves an idle Worker slot. It then records the start. This is the record that pins the attempt's input snapshot and takes its Attribute locks. Dex sends the task last. For a SYNC Step, the start is durable in object storage before the Worker receives the task. An ASYNC Step does not wait for that write.

Once Dex handed an attempt to a Worker, Dex never hands the same attempt number to another Worker. If the Worker crashes or its connection breaks, the attempt fails whether or not it sent output. The retry policy then runs: it consumes a retry and waits for the backoff. If a Worker stops responding without disconnecting, the attempt fails when its heartbeat timeout expires. A server restart or failover does the same for every attempt that was handed out. Those attempts fail with the failure kind STARTED_ATTEMPT_INTERRUPTED. SYNC and ASYNC behave the same in these cases.

A few cases return the attempt to pending instead. The attempt keeps its attempt number, its input snapshot, and its locks, and it consumes no retry. This happens only when the Worker certainly never ran the attempt:

• The Worker declines the Step type as unknown. Dex then avoids that Worker for that Step type.
• The task never reached the Worker, because the stream closed before the send or the send failed.
• A Worker that uses reserved-slot dispatch closes its stream cleanly while the task has no output yet. This is a normal Worker shutdown. A shutting-down Worker finishes its running tasks, sends their results, and then closes. Tasks that arrive while it closes are not run and return to Dex.
• An older SDK Worker declines the task for lack of capacity.

An ASYNC Step has one more case. If a Dex Server node crashes before Dex wrote the ASYNC start to object storage, Dex rolls that start back. The same attempt number then runs again, with no retry consumed and possibly on different input. So for ASYNC, one attempt number can run twice, and the two executions can overlap. Step Stream writes can then repeat in the same source.

For both durabilities, the attempts of one Step are not strictly sequential. Cancellation is cooperative, so an interrupted execution may still run while the next attempt or the failure route starts.

	SYNC (default)	ASYNC
Workers one attempt number reaches	At most one, except when the attempt returns to pending as described above.	At most one, except a rollback after a node crash can run the same number again.
Start and execution order	The start is durable before the Worker receives the task.	The Worker can run before the start is durable.
Worker disconnect or node failover	The attempt fails and the retry policy runs.	The same. A node crash before the start was written is the rollback case.
attempt_progress	NOT_STARTED or NOT_COMPLETED.	Always NOT_COMPLETED.
Overlapping attempts	Possible after cancellation, because cancellation is cooperative.	Possible after cancellation, and between two executions of the same attempt number.
Step Stream writes	Can repeat across attempts.	Can also repeat within one attempt number, in the same source.
When to use	Effects that need the tightest guarantees.	Short Steps where latency matters and a rare rerun is acceptable.

A failure route cannot tell whether the work ran

A Step attempt that no Worker takes does not fail. It waits in the durable task queue, uses no retry, and runs when a Worker takes it. Only an RPC fails when no Worker takes it within the dispatch wait. A Step attempt that does fail can fail before or after a Worker received it, and its error type and detail alone do not always tell which.

The recovery error (RecoveryErrorInfo) also carries attempt_progress. It is the furthest progress across all attempts of the failed method call:

• NOT_STARTED: no attempt ever reached a Worker that could have run it. The handler certainly did not run.
• NOT_COMPLETED: at least one attempt was handed to a Worker and may have run, but none completed.
• UNSPECIFIED: an older server did not report progress, or the call started before the upgrade. Treat it like NOT_COMPLETED.

ASYNC failures always report NOT_COMPLETED, because a rolled-back execution may have run.

NOT_COMPLETED is conservative. Dex can report it although the Worker never ran the handler:

• The server crashed after the start was durable but before the task reached the Worker.
• The connection broke right after Dex wrote the task but before the Worker read it.
• The Worker declined the task or closed cleanly, but the server crashed before it recorded that.
• A failure inside the SDK before it called the handler, such as reading input blobs or decoding. Dex reports this as a normal Worker error. A misconfigured FlowServiceAddress makes every attempt with blob inputs fail this way. values.hydrateBeforeDispatch avoids it.

A failure route can rely on NOT_STARTED to mean the work did not run. It cannot rely on NOT_COMPLETED to mean the work ran. The Flow's history records what Dex knows, and Dex Web shows it.

Design for these limits

• Make each external effect idempotent. Key it by an ID that stays the same across attempts, such as the Flow ID and the Step execution ID, or your own business key. Send the key to the service that applies the effect, so that it applies the effect once.
• Treat a Step with a single attempt as at most once only within these limits. An interrupted attempt may have made its call, and a cancelled execution may still run. Its failure route can run although the work happened, or although it never ran. For ASYNC, the same attempt number can also run twice.
• Reconcile an unknown outcome. When a failure route sees NOT_COMPLETED, check the target system by the idempotency key or your business key before you repeat or report the effect. Record the outcome you find in the Flow.

For the agent packages' single-attempt tool Steps, see Tool call limits.