---
title: "Failure and retries"
description: "When a Step method attempt fails, its retry policy and failure policy decide"
canonical_url: "https://docs.superdurable.io/zh-Hans/reference/rust/failure-and-retries/"
source: "docs/reference/rust/failure-and-retries.mdx"
locale: "zh-Hans"
---
# Failure and retries
When a Step method attempt fails, its retry policy and failure policy decide
what happens next. When Dex cannot tell whether the work ran, see
[Failure and redelivery](https://docs.superdurable.io/zh-Hans/reference/rust/failure-and-retries.md#failure-and-redelivery).
## Retry and failure policies
- The method **Timeout** limits one attempt. See
[Timeouts and total duration](https://docs.superdurable.io/zh-Hans/reference/rust/step-options.md#timeouts-and-total-duration).
- The **retry policy** calls the same method of the same Step execution again,
until its attempts are used up.
- **Failure policy** decides what happens **after** retries end. The default
fails the Flow. **WaitFor** can run **Execute** anyway, and **Execute** can
go to a recovery Step.
```rust
fn options(&self) -> StepOptions {
StepOptions::new().execute_retry(
RetryPolicy::new()
.initial_interval(Duration::from_secs(1))
.backoff_coefficient(2.0)
.maximum_attempts(5),
)
}
```
Example: [examples/rust/src/primitives/step/retry_flow.rs](https://github.com/superdurable/dex/tree/main/examples/rust/src/primitives/step/retry_flow.rs)
### WaitFor failure policy
When **WaitFor** retries are exhausted:
- **WaitForFailurePolicy::FailFlow** fails the Flow. This is the default.
- **WaitForFailurePolicy::Proceed** runs **Execute** anyway.
With **Proceed**, the Step's **Execute** receives the Step input
unchanged. **wait_for_method_failed()** returns true, and **recovery_error()** holds
the error type and detail of the last failed **WaitFor** attempt.
```rust
fn options(&self) -> StepOptions {
StepOptions::new()
.wait_for_failure(WaitForFailurePolicy::Proceed)
.wait_for_retry(RetryPolicy::new().maximum_attempts(2))
}
fn wait_for(&self, _context: &mut Context, _input: Self::Input) -> HandlerResult {
Err(HandlerError::new("ProceedOnWaitFailure", "planned WaitFor failure"))
}
fn execute(&self, context: &mut Context, input: Self::Input) -> HandlerResult {
if !context.wait_for_method_failed() {
return Err(HandlerError::new(
"ProceedOnWaitFailure",
"waitFor failure was not reported",
));
}
Ok(StepDecision::go_to(&Finish, format!("{input}_recovered")))
}
```
Example: [examples/rust/src/primitives/proceed_on_wait_failure/flow.rs](https://github.com/superdurable/dex/tree/main/examples/rust/src/primitives/proceed_on_wait_failure/flow.rs)
### Execute failure policy
To go to a **recovery Step** when **Execute** retries are used up,
call **on_execute_failure_proceed_to** on **StepOptions**.
```rust
fn options(&self) -> StepOptions {
StepOptions::new()
.execute_retry(RetryPolicy::new().maximum_attempts(3))
.on_execute_failure_proceed_to(&Compensate)
}
```
Example: [examples/rust/src/patterns/recovery/flow.rs](https://github.com/superdurable/dex/tree/main/examples/rust/src/patterns/recovery/flow.rs)
The recovery Step receives the failed Step's input unchanged.
**recovery_error()** holds the error type and detail of the last failed
**Execute** attempt, and its [attempt progress](https://docs.superdurable.io/zh-Hans/reference/rust/failure-and-retries.md#unknown-outcome). See the
[Failure Handling](https://docs.superdurable.io/zh-Hans/design-patterns/failure-handling.md) pattern and the
[money-transfer](https://docs.superdurable.io/zh-Hans/use-cases/money-transfer.md) use case.
### Last failure info
- When a Step finally fails, its **StepExecuteFailed** or
**StepWaitForFailed** event carries the failure.
- While retries are still running, before the failure policy runs, Step events
and the live Flow state show **lastFailureInfo**: the error type, detail, and
Worker metadata of the latest attempt.
- In Dex Web, event details show it as **Last failure** while retries run.
Active Steps in the Flow overview show **lastFailureInfo** too.
## RetryAfter
Use **RetryAfter** to choose the delay before the next attempt. It overrides
the retry schedule of the **RetryPolicy**.
Retries are for recovering from failures. To wait for an external system, use
the [Polling](https://docs.superdurable.io/zh-Hans/design-patterns/polling.md) pattern, not **RetryAfter**.
A retry of an **Execute** call also cannot see new state. It reuses the
[state snapshot](https://docs.superdurable.io/zh-Hans/reference/rust/step-options.md#selective-state-loading) of the first attempt, so it does
not see Attribute writes made since. To wait for a write from outside the Step,
have the writer also publish a [Channel](https://docs.superdurable.io/zh-Hans/concepts/primitives/channel.md) message,
directly or from an RPC handler. Wait for that message in the **WaitFor** of the
next Step. Or have the next Step wait on a [Timer](https://docs.superdurable.io/zh-Hans/concepts/primitives/timer.md); it
reads fresh state when it runs.
```rust
if i32::try_from(context.attempt()).unwrap_or(i32::MAX) < ready_after_attempt {
return Err(HandlerError::retry_after(
7,
"CustomRetry",
format!("not ready on attempt {}", context.attempt()),
));
}
Ok(StepDecision::graceful_complete(String::from("ready")))
```
Example: [examples/rust/src/primitives/custom_retry/flow.rs](https://github.com/superdurable/dex/tree/main/examples/rust/src/primitives/custom_retry/flow.rs)
## Failure and redelivery
Some failures leave Dex unable to tell whether the work ran. A Step with an
external effect that must not repeat, such as a payment or an email, must plan
for these cases. This section states what you can rely on.
### Heartbeats are not acknowledged
**record_heartbeat** returns once the Worker has queued or sent the heartbeat. It does
not wait for Dex to store anything, so its return does not prove that Dex
stored the checkpoint. This holds for SYNC Steps too, although a SYNC Step's
code runs only after Dex stored the attempt start.
Dex can lose checkpoints. It batches them and keeps only the latest value of
each batch. A checkpoint can also be lost when the attempt ends, for example on
a timeout or a cancellation, or when a Dex Server node fails. After an
interruption, the next attempt receives the last checkpoint that Dex stored.
That checkpoint can be older than the last one the Step sent. Do not use a
checkpoint as the only record that an external effect happened.
### An attempt starts when Dex hands it to a Worker
To hand an attempt to a Worker, Dex first reserves an idle Worker slot. It
then records the attempt start, which fixes the attempt's input snapshot and
takes its Attribute locks. Dex sends the task last. For a SYNC Step, the start
is stored in object storage before the Worker receives the task. An ASYNC Step
does not wait for that write.
Once Dex has handed an attempt to a Worker, it never hands the same attempt
number to another Worker. If the Worker crashes or its connection breaks, the
attempt fails, whether or not it sent output. The retry policy then uses a
retry and waits for the backoff. If a Worker stops responding without
disconnecting, the attempt fails when its heartbeat timeout expires. A server
restart or failover does the same for every attempt that was handed out. Those
attempts fail with the failure kind STARTED_ATTEMPT_INTERRUPTED. SYNC and ASYNC
behave the same in these cases.
In a few cases, Dex returns the attempt to pending instead. The attempt keeps
its attempt number, its input snapshot, and its locks, and it uses no retry.
This happens only when the Worker certainly never ran the attempt:
- The Worker declines the Step type as unknown. Dex then avoids that Worker for
that Step type.
- The task never reached the Worker, because the stream closed before the send
or the send failed.
- A Worker that uses reserved-slot dispatch closes its stream cleanly while the
task has no output yet. This is a normal Worker shutdown. A shutting-down
Worker finishes its running tasks, sends their results, and then closes.
Tasks that arrive while it closes are not run and return to Dex.
- An older SDK Worker declines the task for lack of capacity.
An ASYNC Step has one more case. If a Dex Server node crashes before the ASYNC
start reaches object storage, Dex rolls the start back. The same attempt number
then runs again, using no retry and possibly with different input. So one ASYNC
attempt number can run twice, and the two executions can overlap. Their Step
Stream writes can repeat in the same source.
With either durability, the attempts of one Step can overlap. The SDK cannot
stop running code on [cancellation](https://docs.superdurable.io/zh-Hans/reference/rust/step-options.md#cancellation), so an
interrupted attempt may still be running when the next attempt or the failure
route starts.
| | SYNC (default) | ASYNC |
| ---------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Workers one attempt number reaches | At most one, except when the attempt returns to pending as described above. | At most one, except a rollback after a node crash can run the same number again. |
| Start and execution order | The start is durable before the Worker receives the task. | The Worker can run before the start is durable. |
| Worker disconnect or node failover | The attempt fails and the retry policy runs. | The same. A node crash before the start was written is the rollback case. |
| **attempt_progress()** | **NOT_STARTED** or **NOT_COMPLETED**. | Always **NOT_COMPLETED**. |
| Overlapping attempts | Possible after cancellation, because the SDK cannot stop running code. | Possible after cancellation, and between two executions of the same attempt number. |
| Step Stream writes | Can repeat across attempts. | Can also repeat within one attempt number, in the same source. |
| When to use | Effects that need the tightest guarantees. | Short Steps where latency matters and a rare rerun is acceptable. |
### A failure route cannot tell whether the work ran
A Step attempt that no Worker takes does not fail. It waits in the durable task
queue, uses no retry, and runs when a Worker takes it. An RPC is different: it
fails when no Worker takes it within the dispatch wait. A Step attempt that
does fail may have failed before or after a Worker received it. Its error type
and detail do not always tell which.
The recovery error, a **RecoveryErrorInfo**, also has **attempt_progress()**:
the furthest progress that any attempt of the failed method call reached.
- **NOT_STARTED**: no attempt ever reached a Worker that could have run it. The
handler certainly did not run.
- **NOT_COMPLETED**: at least one attempt was handed to a Worker and may have
run, but none completed.
- **UNSPECIFIED**: an older server did not report progress, or the call started
before the upgrade. Treat it like **NOT_COMPLETED**.
ASYNC failures always report **NOT_COMPLETED**, because a rolled-back execution
may have run.
Dex can also report **NOT_COMPLETED** when the Worker never ran the method:
- The server crashed after the start was durable but before the task reached
the Worker.
- The connection broke right after Dex wrote the task but before the Worker
read it.
- The Worker declined the task or closed cleanly, but the server crashed before
it recorded that.
- The SDK failed before it called the method, for example while reading input
blobs or decoding. Dex reports this as a normal Worker error. A wrong
Flow service address makes every attempt with blob inputs fail this way.
The server setting **values.hydrateBeforeDispatch** avoids it.
A failure route can rely on **NOT_STARTED** to mean the work did not run. It
cannot rely on **NOT_COMPLETED** to mean the work ran. The Flow's history
records what Dex knows, and Dex Web shows it.
### Design for these limits
- Make each external effect idempotent. Key it by an ID that stays the same
across attempts, such as the Flow ID and the Step execution ID, or your own
business key. Send the key to the service that applies the effect, so that it
applies the effect once.
- A Step with a single attempt runs at most once only within these limits.
An interrupted attempt may have made its call, and a cancelled execution may
still run. Its failure route can run although the work happened, or although
it never ran. For ASYNC, the same attempt number can also run twice.
- Reconcile an unknown outcome. When a failure route sees **NOT_COMPLETED**,
check the target system by the idempotency key or your business key before
you repeat or report the effect. Record the outcome you find in the Flow.
For the agent packages' single-attempt tool Steps, see
[At-most-once has one exception](https://docs.superdurable.io/zh-Hans/agent-frameworks.md#tool-runs-twice).
## Version notes
Changed in 0.1.17 (Server protocol 4, required by SDK 1.35):
- A Step attempt deadline is the hand-out time plus the method timeout. It is
no longer shortened to the remaining retry budget. Queue and lock waits are
excluded.
- An interrupted attempt that was handed out consumes a retry.
- **TotalDuration** never ends a running attempt. The server no longer
produces the RETRY_DEADLINE failure kind.