Failover and Deploys

View as Markdown

Dex saves each finished Step, so a crash or a deploy repeats only the work that was in progress. This page lists each event and what runs again.

Event	What runs again	What you do
A Worker crashes or stops	The Step attempts it was running	Key external calls with an idempotency key.
A Dex Server node fails over	Every attempt that a Worker had received	The same. Each attempt also consumes a retry.
You deploy new Step code	Nothing that already finished	Keep open Flows working. See Versioning Flow code.
You change `WaitFor` code	Nothing that already waits	Expect old conditions on Steps that already wait.
An ASYNC Step's start is lost	That attempt, on a new snapshot	Use SYNC when the Step must see one snapshot.
An ASYNC Step's result is lost	That Step, and the work that started from it up to the first SYNC Step	Make ASYNC Steps and the work after them safe to repeat.
You run Time Travel	Every Step after the chosen point	Key external calls so a repeat has no effect.

A Worker crashes or stops

A Worker that dies in the middle of a Step leaves that attempt unfinished. Dex fails the attempt and retries it on another Worker, under the Step's retry policy. The retry reuses the state snapshot of the first attempt. Finished Steps do not run again. A Step that waits holds no Worker, so a crash during a wait repeats nothing. Step guarantees has the details.

A graceful stop drains first. The Worker gives running methods up to 30 seconds by default to finish, then asks them to cancel. See Workers and the task queue.

A Dex Server node fails over

When the active node crashes, a standby takes over after the lease expires, plus 1 second. The lease is 10 seconds by default. Recovery then fails every attempt that a Worker had received, with NOT_COMPLETED, and each attempt consumes a retry. So a Step that was running during a failover runs again, even if its Worker kept running. The reason is that the new node did not hand out those attempts. The old node's task streams ended with it, so the new node cannot learn whether each attempt finished, and it retries them all.

There is one exception. An ASYNC attempt whose start the node had not yet saved runs again with the same attempt number and consumes no retry. See Lease, hot standby, and step down.

If the original Worker finishes after recovery failed its attempt, Dex ignores that late result. It discards the result's Attribute writes, Channel publications, and decision. The external calls the method made have already happened, so the rerun must use the same idempotency keys.

A Step without heartbeat checkpoints starts from the beginning after a failover. With the default method timeout of 2 hours, that can repeat up to 2 hours of work. A long Step does not have to start over. Send a heartbeat with a checkpoint value as the work progresses, such as the last batch it finished. The next attempt receives the latest stored checkpoint and resumes from it. A checkpoint can be lost, so the work after it must be safe to repeat. See Heartbeat in the SDK reference (the same section exists in every SDK).

A graceful step down releases the lease, so a standby takes over at once.

Agent tools and server upgrades

An agent's single-attempt tool that was running during a failover ends as outcome_unknown, because its attempt reached a Worker and then failed. See Tool Calls. Some server upgrades stop every server process, which fails in-flight attempts the same way. Before such an upgrade, stop the Workers gracefully first. A graceful stop starts no new tasks and lets running handlers finish and send their results, for up to the drain timeout, 30 seconds by default. Then stop and upgrade the server, and start the Workers again. Calls made while no Worker runs wait in the queue, or fail with Unavailable for an RPC. See Workers and the task queue and Upgrades.

You deploy new code

Each Step attempt runs the code of the Worker that takes it; Dex does not replay earlier Steps. A Flow that waits in a Step today runs the new Execute when its conditions are met.

During a rolling deploy, a Worker that lacks a Step type or RPC declines it. Dex hands the call to another Worker, and the Step attempt is not used up. See Routing and availability.

Keep a Step type or RPC until no open Flow can call it. A stored version Attribute decides which path a Flow takes. See Versioning Flow code.

You change WaitFor code

WaitFor returns a Wait, and Dex stores it. A Step that already waits keeps the conditions its WaitFor returned. New WaitFor code applies to Step executions that start after the deploy. To change what an old Step waits for, publish to its Channel or skip its Timer, or start a new Flow.

An ASYNC Step's start or result is lost

If a node fails before it saves an ASYNC Step's start or result, that Step runs again on a new state snapshot, and so does later work up to the first SYNC Step. SYNC and ASYNC Steps explains why the repeat stops there.

You run Time Travel

Time Travel moves a Flow run back to an earlier point and runs it again from there. Every Step after that point runs again, with its external calls. See Recover a Flow with Time Travel.