Skip to main content
Super Durable
Menu

Advanced Step Guarantees

View as Markdown

Read this page once Steps run in parallel, take a long time, or use ASYNC durability. Step covers the guarantees every Step has.

What Dex saves

In short: a method's writes are saved together, but they are not isolated from other Steps and RPCs.

Dex saves each WaitFor or Execute call on its own. It saves the call's Attribute writes and Channel publications together with its result, and only when the method succeeds. Calls to external systems are not part of this save.

A Step method's changes take effect all at once. They are not protected from other Steps and RPCs, though. A method reads the Flow state as it was when Dex handed the method to a Worker. Other Steps and RPCs can change that state before the method's changes take effect. Dex does not check that the values the method read are still current. So two Steps or RPCs that read and then write the same Attribute can overwrite each other's update. To prevent this, have both declare a lock on that Attribute. Dex then runs them one at a time. See Attribute locks.

Failure and redelivery

In short: when a method's outcome is uncertain, Dex runs the method again or tells you the outcome is unknown.

These cases repeat a method or leave its outcome unknown. They hold for every SDK:

  • A Worker stops in the middle of a method. The attempt fails, and the retry policy runs the method again. Its external calls may already have happened. A server restart or failover does the same to every attempt that a Worker had.
  • Heartbeats are fire-and-forget. A heartbeat checkpoint can be lost, so a rerun can start from an older checkpoint.
  • A canceled method keeps running. Dex does not stop its code. Its external calls still take effect.
  • An ASYNC Step can repeat an attempt. See SYNC and ASYNC Steps.
  • Retries can run out without an answer. The recovery Step then learns that the work certainly did not run (NOT_STARTED) or that it may have run (NOT_COMPLETED). On NOT_COMPLETED, check the target system by the idempotency key before you repeat or report the effect.

Failure and redelivery in the SDK reference gives the exact rules for each case: Python · Go · Java · TypeScript · Rust.

SYNC and ASYNC Steps

A Step's durability chooses when Dex stores its progress.

  • SYNC is the default. A SYNC Step waits for two log commits: the attempt start, before the Worker receives the task, and the result, before the Flow moves on. Each commit is one write to object storage. Dex groups the commands that are ready at the same moment into one commit, so a commit can carry the work of many Flows.
  • ASYNC hands the task to the Worker, and starts the next work, before the start and the result are stored. Dex batches these writes over about 20 milliseconds, so a single Flow runs faster. If a Dex Server node fails before the write, Dex runs the same attempt again on a new state snapshot, so it can read different Attribute values.
  • A lost ASYNC result reruns later work only up to the first SYNC Step. Dex writes a Flow's commands to its log in order. Once a SYNC Step's start is stored, every earlier command of that Flow is stored too, so a SYNC Step that a Worker has received does not run again because of a lost ASYNC result.

Choose SYNC for effects that need the tightest guarantees. Choose ASYNC for short Steps where latency matters and a rare rerun is acceptable. Good ASYNC candidates are fast reads, internal bookkeeping, and calls that a key already makes safe to repeat. Keep SYNC for a payment, an email, or any call whose rerun on a newer snapshot could send different arguments. A StepOptions setting picks the durability of one Step, and FlowConfig sets it for a whole Flow. Performance gives measured latency for both.

Cancellation, completion, and order

Canceling a Step discards everything its current method has not saved: its Attribute writes, its Channel publications, and its decision. What its WaitFor already saved stays. A decision or an RPC never cancels the Steps that it starts itself. Channel guarantees say what happens to the messages a canceled Step consumed.

A canceled method's code keeps running until the method returns. The method should check for cancellation and return. Its calls to external systems still take effect, even after the Flow has closed.

GracefulComplete ends its own branch and asks Dex to complete the Flow. Dex begins to complete the Flow as soon as no Step is active. Until then, other Steps and RPCs can still start new Steps, and those Steps delay completion too. A later branch that ends with DeadEnd does not withdraw the request. After the last Step ends, the Flow can stay RUNNING briefly while Dex finishes pending Attribute Sync writes and transactional RPCs that already started.

A Flow with no active Step keeps running until a GracefulComplete is pending, and an RPC can still start new Steps in it. End the last branch with a completing decision. If a Flow must not wait forever, set a Flow timeout.

Dex starts ready Steps roughly in the order they became ready, but not strictly. Do not rely on the order in which parallel Steps start or finish. Use a Channel or an Attribute lock when the order matters.

Next