--- title: "Failover and Deploys" description: "What runs again when a Worker crashes, a Dex Server node fails over, or you deploy new code, and what you must do about it." canonical_url: "https://docs.superdurable.io/concepts/failover-and-deploys/" source: "docs/content/concepts/failover-and-deploys.mdx" locale: "en" --- > Index of every page for AI agents: https://docs.superdurable.io/llms.txt # Failover and Deploys Dex saves each finished Step, so a crash or a deploy repeats only the work that was in progress. This page lists each event and what runs again. | Event | What runs again | What you do | | ------------------------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | A Worker crashes or stops | The Step attempts it was running | Key external calls with an [idempotency key](https://docs.superdurable.io/concepts/step.md#idempotency-keys). | | A Dex Server node fails over | Every attempt that a Worker had received | The same. Each attempt also consumes a retry. | | You deploy new Step code | Nothing that already finished | Keep open Flows working. See [Versioning Flow code](https://docs.superdurable.io/concepts/change-flow-code.md#versioning-flow-code). | | You change `WaitFor` code | Nothing that already waits | Expect old conditions on Steps that already wait. | | An ASYNC Step's start is lost | That attempt, on a new snapshot | Use SYNC when the Step must see one snapshot. | | An ASYNC Step's result is lost | That Step, and the work that started from it up to the first SYNC Step | Make ASYNC Steps and the work after them safe to repeat. | | You run Time Travel | Every Step after the chosen point | Key external calls so a repeat has no effect. | ## A Worker crashes or stops A Worker that dies in the middle of a Step leaves that attempt unfinished. Dex fails the attempt and retries it on another Worker, under the Step's retry policy. The retry reuses the state snapshot of the first attempt. Finished Steps do not run again. A Step that waits holds no Worker, so a crash during a wait repeats nothing. [Step guarantees](https://docs.superdurable.io/concepts/step.md#guarantees) has the details. A graceful stop drains first. The Worker gives running methods up to 30 seconds by default to finish, then asks them to cancel. See [Workers and the task queue](https://docs.superdurable.io/reference/server/workers.md). ## A Dex Server node fails over When the active node crashes, a standby takes over after the lease expires, plus 1 second. The lease is 10 seconds by default. Recovery then fails every attempt that a Worker had received, with `NOT_COMPLETED`, and each attempt consumes a retry. So a Step that was running during a failover runs again, even if its Worker kept running. The reason is that the new node did not hand out those attempts. The old node's task streams ended with it, so the new node cannot learn whether each attempt finished, and it retries them all. There is one exception. An ASYNC attempt whose start the node had not yet saved runs again with the same attempt number and consumes no retry. See [Lease, hot standby, and step down](https://docs.superdurable.io/reference/server/high-availability.md#lease). If the original Worker finishes after recovery failed its attempt, Dex ignores that late result. It discards the result's Attribute writes, Channel publications, and decision. The external calls the method made have already happened, so the rerun must use the same [idempotency keys](https://docs.superdurable.io/concepts/step.md#idempotency-keys). A Step without heartbeat checkpoints starts from the beginning after a failover. With the default method timeout of 2 hours, that can repeat up to 2 hours of work. A long Step does not have to start over. Send a heartbeat with a checkpoint value as the work progresses, such as the last batch it finished. The next attempt receives the latest stored checkpoint and resumes from it. A checkpoint can be lost, so the work after it must be safe to repeat. See [Heartbeat](https://docs.superdurable.io/reference/python/step-options.md#heartbeat) in the SDK reference (the same section exists in every SDK). A graceful step down releases the lease, so a standby takes over at once. ### Agent tools and server upgrades An agent's single-attempt tool that was running during a failover ends as `outcome_unknown`, because its attempt reached a Worker and then failed. See [Tool Calls](https://docs.superdurable.io/agent-frameworks/tool-calls.md). Some server upgrades stop every server process, which fails in-flight attempts the same way. Before such an upgrade, stop the Workers gracefully first. A graceful stop starts no new tasks and lets running handlers finish and send their results, for up to the drain timeout, 30 seconds by default. Then stop and upgrade the server, and start the Workers again. Calls made while no Worker runs wait in the queue, or fail with `Unavailable` for an RPC. See [Workers and the task queue](https://docs.superdurable.io/reference/server/workers.md) and [Upgrades](https://docs.superdurable.io/reference/server/upgrades.md). ## You deploy new code Each Step attempt runs the code of the Worker that takes it; Dex does not [replay](https://docs.superdurable.io/concepts/change-flow-code.md#versioning-flow-code) earlier Steps. A Flow that waits in a Step today runs the new `Execute` when its conditions are met. During a rolling deploy, a Worker that lacks a Step type or RPC declines it. Dex hands the call to another Worker, and the Step attempt is not used up. See [Routing and availability](https://docs.superdurable.io/reference/server/workers.md#routing-and-availability). Keep a Step type or RPC until no open Flow can call it. A stored version Attribute decides which path a Flow takes. See [Versioning Flow code](https://docs.superdurable.io/concepts/change-flow-code.md#versioning-flow-code). ## You change WaitFor code `WaitFor` returns a Wait, and Dex stores it. A Step that already waits keeps the conditions its `WaitFor` returned. New `WaitFor` code applies to Step executions that start after the deploy. To change what an old Step waits for, publish to its Channel or skip its Timer, or start a new Flow. ## An ASYNC Step's start or result is lost If a node fails before it saves an ASYNC Step's start or result, that Step runs again on a new state snapshot, and so does later work up to the first SYNC Step. [SYNC and ASYNC Steps](https://docs.superdurable.io/concepts/step/advanced-guarantees.md#sync-and-async) explains why the repeat stops there. ## You run Time Travel Time Travel moves a Flow run back to an earlier point and runs it again from there. Every Step after that point runs again, with its external calls. See [Recover a Flow with Time Travel](https://docs.superdurable.io/concepts/change-flow-code.md#time-travel). ## Next - [Step guarantees](https://docs.superdurable.io/concepts/step.md#guarantees) - [Limits](https://docs.superdurable.io/intro/limits.md) - [Run Dex in Production](https://docs.superdurable.io/components/production.md)