Skip to main content
Super Durable
Menu

Change Flow Code

View as Markdown

Flows can stay open for days or months, so new code often meets Flows that started under old code. This page shows how to change code safely and how to rerun a Flow after a fix.

Versioning Flow code

Dex decides what a Flow does next from its recorded state, without re-running your earlier Step code. So deploying changed code never causes a replay error. You still have to keep open Flows working across a deployment. They can meet changed business rules, missing Step implementations, renamed RPCs, or incompatible data.

Lock business behavior at Flow start

Persist a business version value when a Flow starts, such as an Attribute or a field in the start state. Branch on that stored value, not on the Worker build currently serving the call. Existing Flows keep their old behavior; new Flows can use the new behavior after deployment.

Use this when a change alters a decision, payment rule, retry policy, deadline, or other customer-visible meaning. Deploying new code alone is not enough, because a long-running Flow can call a newly deployed Worker later.

Evolve data and contracts additively

Add optional fields. Code that reads a payload must handle fields that older runs did not set, and ignore unknown fields that newer components send. Supply a safe default for an absent field, and keep the previous meaning of existing field values. For an incompatible payload or output contract, introduce a new Step type or RPC name instead of silently changing the old one.

If a newer Worker cannot decode a stored input or Attribute, the attempt fails with a decoding error, like any error a Step method raises. Each retry fails the same way, so the Step fails when its retry window ends (4 hours by default), and its failure policy applies. By default the Flow then fails. Dex Web shows the error type and message on the attempt. Fix the code, then rerun the Step with Time Travel.

To catch this before the retry window ends, watch Dex Server's dexzd_step_attempt_failures_total metric with failure_kind="worker_error" after each deploy. A decoding error counts there like any Step error. The metric has no Step type label, so open the failing attempts in Dex Web to see which Step and which error. See Metrics.

Keep old definitions until old Flows finish

Keep a Flow type, Step type, or RPC implementation registered under its name while an open Flow can still call it. Deploy the new definition alongside the old one. New Flows reach it in one of two ways: a decision that branches on the version stored at start goes to the new Step type, or new code starts a new Flow type. Keep the old implementation available through retries and any approved Time Travel recovery.

During a rolling deployment, a Worker that lacks a requested Step or RPC can hand the call to another Worker that has it. This does not use up a Step attempt. See Routing and availability.

Before removing the old definition, search for the legacy Flow type and confirm that no running, waiting, retrying, or recoverable terminal Flow needs it. Keep the search result with the change record. The exact query depends on your search fields. Start with a search by Flow type:

dexcli flow search --query 'FlowType = "OldFlowType"'

Only remove the old code after the result is empty or every remaining Flow has an explicitly approved retirement plan.

Check before you deploy

To test a new Worker build against open Flows, run these checks:

  1. List the open Flows of the type. Search for running Flows, and for the Step types they wait in. ActiveStepTypes is a built-in search field.

    dexcli flow search --query 'FlowType = "OrderFlow" AND ExecutionStatus = "Running"'
    dexcli flow search --query 'FlowType = "OrderFlow" AND ActiveStepTypes = "Ship"'

  2. Confirm the new build registers every Step type and RPC those Flows can reach. A Worker that lacks one hands the call to another Worker. If none has it, a Step waits in the queue and an RPC fails.

  3. Run the integration tests of the new build against Flows started by the old one. Test Your Flows shows the setup. The Quick Start's restart test is the shape to copy: it stops one Worker in the middle of a Flow and finishes the Flow with another. For a version check:

    1. Check out the old build. Start a Worker from it, start Flows, and wait until each reaches the Step you care about, with wait_for_step_completion or a search on ActiveStepTypes.
    2. Stop that Worker gracefully.
    3. Start a Worker from the new build against the same Dex Server. Skip any Timers with skip-timer, send the messages the Flows wait for, and wait for each Flow to finish.
    4. Check each Flow's result and Attributes, and that the new build raised no decoding error.

These checks are manual, since Dex has no single command for them yet.

Example: add a fee to Charge

The order Flow from Order Processing charges the order amount. A new rule adds a handling fee. Orders that are already open must keep the old price.

  1. Record the version at start. The new start handler sets an Attribute, such as pricing-version, to 2 when it starts a Flow. Flows that started earlier have no value, which means version 1.
  2. Branch on the stored value. Charge reads pricing-version. It charges the amount alone for version 1 and the amount plus the fee for version 2. A Flow that waits in Ship today never runs Charge again, so it is not affected.
  3. Add a Step type for a changed contract. Suppose the refund now needs a new input field that old Flows never stored. Add a new Step type, such as RefundWithFee, and go to it only from version 2. Keep Refund for version 1 Flows.
  4. Deploy both. One Worker build contains Charge with the branch, Refund, and RefundWithFee. During a rolling deployment, an older Worker that lacks RefundWithFee hands that call to a newer one.
  5. Retire the old path. When no open Flow has version 1, remove the version 1 branch and Refund. Search for such Flows first. If pricing-version is an indexed Attribute, the search can filter on it as well as on the Flow type.

The stored Attribute decides which path a Flow takes, whatever Worker build serves it, so Step code needs no version marker.

Recover a Flow with Time Travel

Time Travel moves a Flow run back to an earlier point in its history and runs it again from there. Use it after you deploy a fix for a Step that failed. The run keeps its Run ID and starts a new attempt.

Steps after that point run again. So do their external calls. Each external call needs an idempotency key so a repeat has no effect.

Time Travel restores the run's state as it was at that point: its Attributes, its Channel messages, and the Steps that were waiting or ready. Input that came from outside the run after that point is then applied again by default: RPC calls, Channel publications, and Attribute writes. This is write reapply. Turn it off with skip_writes_reapply (dexcli: -skip-writes-reapply) when that input should not count again. The FlowService call is named ResetFlow; these docs call it Time Travel.

The target point must be inside the Flow's recent history window. By default the window keeps about the newest 50,000 history events of each run on serve, and 5,000 on dev. The server key is history.windowEvents, and a Flow can set its own size in its Flow configuration. Time Travel to the beginning of the run is always available. Start it from Dex Web, dexcli, or a Client. Time travel in the server reference covers the request options and limits.

Next