---
title: "Failure and redelivery"
description: "Retries and failure routes handle attempts that fail. Some failures leave Dex"
canonical_url: "https://docs.superdurable.io/reference/python/1.37.x/flows-and-steps/failure-and-redelivery/"
source: "docs/reference-python_versioned_docs/version-1.37.x/flows-and-steps/failure-and-redelivery.mdx"
locale: "en"
---
# Failure and redelivery
Retries and failure routes handle attempts that fail. Some failures leave Dex
without knowing whether the work ran. A Step with an external effect that must
not repeat, such as a payment or an email, has to plan for these cases. This
page states what an application can rely on.
## Heartbeats are not acknowledged
A heartbeat call returns once the Worker has queued or sent it. It does not
wait for Dex to store anything. Python's **heartbeat**, Go's
**RecordHeartbeat**, Java's and TypeScript's **recordHeartbeat**, and Rust's
**record_heartbeat** all behave this way. A returned heartbeat call does not
prove that Dex stored the checkpoint. This holds for SYNC Steps too, even
though a SYNC Step runs user code only after Dex made the attempt start
durable.
Heartbeat checkpoints are lossy. Dex batches them and keeps only the latest
value of each batch. A checkpoint can also be lost when the attempt ends, such
as on a timeout or a cancellation, or when a Dex Server node fails. After an
interruption, the next attempt receives the last checkpoint that Dex stored.
That checkpoint can be older than the last one the Step sent. Do not use a
checkpoint as the only record that an external effect happened.
## An attempt starts when Dex hands it to a Worker
A Step attempt starts at the moment Dex hands it to a Worker. Dex first
reserves an idle Worker slot. It then records the start. This is the record
that pins the attempt's input snapshot and takes its Attribute locks. Dex sends
the task last. For a SYNC Step, the start is durable in object storage before
the Worker receives the task. An ASYNC Step does not wait for that write.
Once Dex handed an attempt to a Worker, Dex never hands the same attempt number
to another Worker. If the Worker crashes or its connection breaks, the attempt
fails whether or not it sent output. The retry policy then runs: it consumes a
retry and waits for the backoff. If a Worker stops responding without
disconnecting, the attempt fails when its heartbeat timeout expires. A server
restart or failover does the same for every attempt that was handed out. Those
attempts fail with the failure kind STARTED_ATTEMPT_INTERRUPTED. SYNC and ASYNC
behave the same in these cases.
A few cases return the attempt to pending instead. The attempt keeps its
attempt number, its input snapshot, and its locks, and it consumes no retry.
This happens only when the Worker certainly never ran the attempt:
- The Worker declines the Step type as unknown. Dex then avoids that Worker for
that Step type.
- The task never reached the Worker, because the stream closed before the send
or the send failed.
- A Worker that uses reserved-slot dispatch closes its stream cleanly while the
task has no output yet. This is a normal Worker shutdown. A shutting-down
Worker finishes its running tasks, sends their results, and then closes.
Tasks that arrive while it closes are not run and return to Dex.
- An older SDK Worker declines the task for lack of capacity.
An ASYNC Step has one more case. If a Dex Server node crashes before Dex wrote
the ASYNC start to object storage, Dex rolls that start back. The same attempt
number then runs again, with no retry consumed and possibly on different
input. So for ASYNC, one attempt number can run twice, and the two executions
can overlap. Step Stream writes can then repeat in the same source.
For both durabilities, the attempts of one Step are not strictly sequential.
[Cancellation](https://docs.superdurable.io/reference/python/1.37.x/flows-and-steps/step-options.md#cancellation) is cooperative, so an interrupted execution may
still run while the next attempt or the failure route starts.
| | SYNC (default) | ASYNC |
| ---------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Workers one attempt number reaches | At most one, except when the attempt returns to pending as described above. | At most one, except a rollback after a node crash can run the same number again. |
| Start and execution order | The start is durable before the Worker receives the task. | The Worker can run before the start is durable. |
| Worker disconnect or node failover | The attempt fails and the retry policy runs. | The same. A node crash before the start was written is the rollback case. |
| **attempt_progress** | **NOT_STARTED** or **NOT_COMPLETED**. | Always **NOT_COMPLETED**. |
| Overlapping attempts | Possible after cancellation, because cancellation is cooperative. | Possible after cancellation, and between two executions of the same attempt number. |
| Step Stream writes | Can repeat across attempts. | Can also repeat within one attempt number, in the same source. |
| When to use | Effects that need the tightest guarantees. | Short Steps where latency matters and a rare rerun is acceptable. |
## A failure route cannot tell whether the work ran
A Step attempt that no Worker takes does not fail. It waits in the durable task
queue, uses no retry, and runs when a Worker takes it. Only an RPC fails when
no Worker takes it within the dispatch wait. A Step attempt that does fail can
fail before or after a Worker received it, and its error type and detail alone
do not always tell which.
The recovery error (**RecoveryErrorInfo**) also carries **attempt_progress**.
It is the furthest progress across all attempts of the failed method call:
- **NOT_STARTED**: no attempt ever reached a Worker that could have run it. The
handler certainly did not run.
- **NOT_COMPLETED**: at least one attempt was handed to a Worker and may have
run, but none completed.
- **UNSPECIFIED**: an older server did not report progress, or the call started
before the upgrade. Treat it like **NOT_COMPLETED**.
ASYNC failures always report **NOT_COMPLETED**, because a rolled-back execution
may have run.
**NOT_COMPLETED** is conservative. Dex can report it although the Worker never
ran the handler:
- The server crashed after the start was durable but before the task reached
the Worker.
- The connection broke right after Dex wrote the task but before the Worker
read it.
- The Worker declined the task or closed cleanly, but the server crashed before
it recorded that.
- A failure inside the SDK before it called the handler, such as reading input
blobs or decoding. Dex reports this as a normal Worker error. A misconfigured
**FlowServiceAddress** makes every attempt with blob inputs fail this way.
**values.hydrateBeforeDispatch** avoids it.
A failure route can rely on **NOT_STARTED** to mean the work did not run. It
cannot rely on **NOT_COMPLETED** to mean the work ran. The Flow's history
records what Dex knows, and Dex Web shows it.
## Design for these limits
- Make each external effect idempotent. Key it by an ID that stays the same
across attempts, such as the Flow ID and the Step execution ID, or your own
business key. Send the key to the service that applies the effect, so that it
applies the effect once.
- Treat a Step with a single attempt as at most once only within these limits.
An interrupted attempt may have made its call, and a cancelled execution may
still run. Its failure route can run although the work happened, or although
it never ran. For ASYNC, the same attempt number can also run twice.
- Reconcile an unknown outcome. When a failure route sees **NOT_COMPLETED**,
check the target system by the idempotency key or your business key before
you repeat or report the effect. Record the outcome you find in the Flow.
For the agent packages' single-attempt tool Steps, see
[Tool call limits](https://docs.superdurable.io/agent-frameworks.md#tool-call-outcomes).