Failure and retries

查看 Markdown查看 Markdown

When a Step method attempt fails, its retry policy and failure policy decide what happens next. When Dex cannot tell whether the work ran, see Failure and redelivery.

Retry and failure policies

• The method Timeout limits one attempt. See Timeouts and total duration.
• The retry policy calls the same method of the same Step execution again, until its attempts are used up.
• Failure policy decides what happens after retries end. The default fails the Flow. WaitFor can run Execute anyway, and Execute can go to a recovery Step.

Java

12345678910

@Override
public StepOptions getStepOptions() {
    return StepOptions.newBuilder()
            .executeRetry(RetryPolicy.newBuilder()
                    .initialInterval(Duration.ofSeconds(1))
                    .backoffCoefficient(2)
                    .maximumAttempts(5)
                    .build())
            .build();
}

Example: examples/java/src/main/java/io/superdurable/dex/primitives/step/RetryFlow.java

WaitFor failure policy

When WaitFor retries are exhausted:

• WaitForFailurePolicy.FAIL_FLOW fails the Flow. This is the default.
• WaitForFailurePolicy.PROCEED runs Execute anyway.

With PROCEED, the Step's Execute receives the Step input unchanged. waitForMethodFailed() returns true, and getRecoveryError() holds the error type and detail of the last failed WaitFor attempt.

Java

1234567891011121314151617181920

@Override
public StepOptions getStepOptions() {
    return StepOptions.newBuilder()
            .waitForRetry(RetryPolicy.newBuilder().maximumAttempts(2).build())
            .waitForFailure(WaitForFailurePolicy.PROCEED)
            .build();
}

@Override
public Wait waitFor(final Context context, final String input) {
    throw new IllegalStateException("planned WaitFor failure");
}

@Override
public StepDecision execute(final Context context, final String input) {
    if (!context.waitForMethodFailed()) {
        throw new IllegalStateException("waitFor failure was not reported");
    }
    return StepDecision.goTo(Finish.class, input + "_recovered");
}

Example: examples/java/src/main/java/io/superdurable/dex/primitives/proceedonwaitfailure/ProceedOnWaitFailureFlow.java

Execute failure policy

To go to a recovery Step when Execute retries are used up, call onExecuteFailureProceedTo on the StepOptions builder.

Java

1234567

@Override
public StepOptions getStepOptions() {
    return StepOptions.newBuilder()
            .onExecuteFailureProceedTo(UpdateQuantityRecovery.class)
            .executeRetry(RetryPolicy.newBuilder().maximumAttempts(5).build())
            .build();
}

Example: examples/java/src/main/java/io/superdurable/dex/patterns/recovery/FailureRecoveryFlow.java

The recovery Step receives the failed Step's input unchanged. getRecoveryError() holds the error type and detail of the last failed Execute attempt, and its attempt progress. See the Failure Handling pattern and the money-transfer use case.

Last failure info

• When a Step finally fails, its StepExecuteFailed or StepWaitForFailed event carries the failure.
• While retries are still running, before the failure policy runs, Step events and the live Flow state show lastFailureInfo: the error type, detail, and Worker metadata of the latest attempt.
• In Dex Web, event details show it as Last failure while retries run. Active Steps in the Flow overview show lastFailureInfo too.

RetryAfter

Use RetryAfter to choose the delay before the next attempt. It overrides the retry schedule of the RetryPolicy.

Retries are for recovering from failures. To wait for an external system, use the Polling pattern, not RetryAfter.

A retry of an Execute call also cannot see new state. It reuses the state snapshot of the first attempt, so it does not see Attribute writes made since. To wait for a write from outside the Step, have the writer also publish a Channel message, directly or from an RPC handler. Wait for that message in the WaitFor of the next Step. Or have the next Step wait on a Timer; it reads fresh state when it runs.

Java

123

throw RetryAfterException.after(
        Duration.ofSeconds(7),
        new IllegalStateException("not ready on attempt " + context.getAttempt()));

Example: examples/java/src/main/java/io/superdurable/dex/primitives/customretry/CustomRetryFlow.java

Failure and redelivery

Some failures leave Dex unable to tell whether the work ran. A Step with an external effect that must not repeat, such as a payment or an email, must plan for these cases. This section states what you can rely on.

Heartbeats are not acknowledged

recordHeartbeat returns once the Worker has queued or sent the heartbeat. It does not wait for Dex to store anything, so its return does not prove that Dex stored the checkpoint. This holds for SYNC Steps too, although a SYNC Step's code runs only after Dex stored the attempt start.

Dex can lose checkpoints. It batches them and keeps only the latest value of each batch. A checkpoint can also be lost when the attempt ends, for example on a timeout or a cancellation, or when a Dex Server node fails. After an interruption, the next attempt receives the last checkpoint that Dex stored. That checkpoint can be older than the last one the Step sent. Do not use a checkpoint as the only record that an external effect happened.

An attempt starts when Dex hands it to a Worker

To hand an attempt to a Worker, Dex first reserves an idle Worker slot. It then records the attempt start, which fixes the attempt's input snapshot and takes its Attribute locks. Dex sends the task last. For a SYNC Step, the start is stored in object storage before the Worker receives the task. An ASYNC Step does not wait for that write.

Once Dex has handed an attempt to a Worker, it never hands the same attempt number to another Worker. If the Worker crashes or its connection breaks, the attempt fails, whether or not it sent output. The retry policy then uses a retry and waits for the backoff. If a Worker stops responding without disconnecting, the attempt fails when its heartbeat timeout expires. A server restart or failover does the same for every attempt that was handed out. Those attempts fail with the failure kind STARTED_ATTEMPT_INTERRUPTED. SYNC and ASYNC behave the same in these cases.

In a few cases, Dex returns the attempt to pending instead. The attempt keeps its attempt number, its input snapshot, and its locks, and it uses no retry. This happens only when the Worker certainly never ran the attempt:

• The Worker declines the Step type as unknown. Dex then avoids that Worker for that Step type.
• The task never reached the Worker, because the stream closed before the send or the send failed.
• A Worker that uses reserved-slot dispatch closes its stream cleanly while the task has no output yet. This is a normal Worker shutdown. A shutting-down Worker finishes its running tasks, sends their results, and then closes. Tasks that arrive while it closes are not run and return to Dex.
• An older SDK Worker declines the task for lack of capacity.

An ASYNC Step has one more case. If a Dex Server node crashes before the ASYNC start reaches object storage, Dex rolls the start back. The same attempt number then runs again, using no retry and possibly with different input. So one ASYNC attempt number can run twice, and the two executions can overlap. Their Step Stream writes can repeat in the same source.

With either durability, the attempts of one Step can overlap. The SDK cannot stop running code on cancellation, so an interrupted attempt may still be running when the next attempt or the failure route starts.

	SYNC (default)	ASYNC
Workers one attempt number reaches	At most one, except when the attempt returns to pending as described above.	At most one, except a rollback after a node crash can run the same number again.
Start and execution order	The start is durable before the Worker receives the task.	The Worker can run before the start is durable.
Worker disconnect or node failover	The attempt fails and the retry policy runs.	The same. A node crash before the start was written is the rollback case.
getAttemptProgress()	NOT_STARTED or NOT_COMPLETED.	Always NOT_COMPLETED.
Overlapping attempts	Possible after cancellation, because the SDK cannot stop running code.	Possible after cancellation, and between two executions of the same attempt number.
Step Stream writes	Can repeat across attempts.	Can also repeat within one attempt number, in the same source.
When to use	Effects that need the tightest guarantees.	Short Steps where latency matters and a rare rerun is acceptable.

A failure route cannot tell whether the work ran

A Step attempt that no Worker takes does not fail. It waits in the durable task queue, uses no retry, and runs when a Worker takes it. An RPC is different: it fails when no Worker takes it within the dispatch wait. A Step attempt that does fail may have failed before or after a Worker received it. Its error type and detail do not always tell which.

The recovery error, a RecoveryErrorInfo, also has getAttemptProgress(): the furthest progress that any attempt of the failed method call reached.

• NOT_STARTED: no attempt ever reached a Worker that could have run it. The handler certainly did not run.
• NOT_COMPLETED: at least one attempt was handed to a Worker and may have run, but none completed.
• UNSPECIFIED: an older server did not report progress, or the call started before the upgrade. Treat it like NOT_COMPLETED.

ASYNC failures always report NOT_COMPLETED, because a rolled-back execution may have run.

Dex can also report NOT_COMPLETED when the Worker never ran the method:

• The server crashed after the start was durable but before the task reached the Worker.
• The connection broke right after Dex wrote the task but before the Worker read it.
• The Worker declined the task or closed cleanly, but the server crashed before it recorded that.
• The SDK failed before it called the method, for example while reading input blobs or decoding. Dex reports this as a normal Worker error. A wrong Flow service address makes every attempt with blob inputs fail this way. The server setting values.hydrateBeforeDispatch avoids it.

A failure route can rely on NOT_STARTED to mean the work did not run. It cannot rely on NOT_COMPLETED to mean the work ran. The Flow's history records what Dex knows, and Dex Web shows it.

Design for these limits

• Make each external effect idempotent. Key it by an ID that stays the same across attempts, such as the Flow ID and the Step execution ID, or your own business key. Send the key to the service that applies the effect, so that it applies the effect once.
• A Step with a single attempt runs at most once only within these limits. An interrupted attempt may have made its call, and a cancelled execution may still run. Its failure route can run although the work happened, or although it never ran. For ASYNC, the same attempt number can also run twice.
• Reconcile an unknown outcome. When a failure route sees NOT_COMPLETED, check the target system by the idempotency key or your business key before you repeat or report the effect. Record the outcome you find in the Flow.

For the agent packages' single-attempt tool Steps, see At-most-once has one exception.

Version notes

Changed in 0.1.17 (Server protocol 4, required by SDK 1.35):

• A Step attempt deadline is the hand-out time plus the method timeout. It is no longer shortened to the remaining retry budget. Queue and lock waits are excluded.
• An interrupted attempt that was handed out consumes a retry.
• TotalDuration never ends a running attempt. The server no longer produces the RETRY_DEADLINE failure kind.