Pull Workers

Every Dex Worker is a pull Worker. It opens outbound task streams to Dex Server and never listens for incoming calls. Workers need only an outbound connection to Dex Server, so they can run behind NAT or firewalls without an inbound port. Pull Workers also offer sticky routing when a Worker keeps useful state in memory between calls.

Start a Worker

Set the Worker's Dex Server address. The pull options are optional. Use them to tune task capacity, registration, and shutdown. A Client needs only the Dex Server address. Dex sends every Flow, SubFlow, Step, and RPC call to pull Workers, so there is no Worker address or target to configure.

This example serves a Greeting Flow through a pull Worker. Its Step stores a greeting, and the greeting RPC reads it. See the Go, Python, Java, TypeScript, and Rust READMEs for startup details.

Dex Web v2 starts a Flow through a pull Worker. Start Flow does not ask for a Worker address, and the start request can send workerTargetPull: true. Dex Web does not check whether a pull Worker is registered before the start; availability is determined when Dex dispatches work. See Dex Web.

async def start_pull_greeting(
    cache: BlobCache, server_address: str
) -> tuple[GreetingFlow, AsyncWorker, AsyncClient, asyncio.Task[None]]:
    flow = GreetingFlow()
    registry = Registry((flow,), allow_async_handlers=True)
    worker = AsyncWorker(
        registry,
        cache,
        WorkerOptions(
            server_address=server_address,
            pull=PullWorkerOptions(
                max_concurrent_tasks=8,
                # Keep one slot for RPCs so Flows stay responsive.
                reserved_rpc_tasks=1,
            ),
        ),
    )
    worker_task = asyncio.create_task(worker.start())
    client = AsyncClient(
        registry,
        cache,
        ClientOptions(server_address=server_address),
    )
    try:
        await worker.wait_pull_registered(timeout=30)
    except BaseException:
        await stop_pull_greeting(worker, client, worker_task)
        raise
    return flow, worker, client, worker_task


async def stop_pull_greeting(
    worker: AsyncWorker, client: AsyncClient, worker_task: asyncio.Task[None]
) -> None:
    await client.close()
    await worker.close()
    await worker_task

Example: examples/python/dex_examples/production/pull_workers.py

Readiness

Starting with Go SDK 1.13.0, WaitPullRegistered gates startup until every Flow type queue and the app Worker ID queue has received a registration acknowledgement. Java provides waitPullRegistered(Duration) and getPullRegistrationState(). Python provides wait_pull_registered(timeout) and pull_registration_state() on both Worker and AsyncWorker; await the AsyncWorker wait. Java's stopOnPermanentStreamRejection and Python's stop_on_permanent_stream_rejection provide the same opt-in shutdown behavior as Go. Start the Go Worker in a goroutine, then wait with a startup deadline before activating it or stopping the old Worker during a rolling deployment. Registration does not guarantee spare handler capacity.

TypeScript exposes waitPullRegistered, pullRegistrationState, and stopOnPermanentStreamRejection. Rust exposes wait_pull_registered, pull_registration_state, and stop_on_permanent_stream_rejection with the same readiness semantics. TypeScript waits accept an AbortSignal or timeout in milliseconds; Rust waits can be cancelled by dropping the future or bounded with a Tokio timeout. Opt-in fatal rejections are returned by TypeScript run and Rust start after draining.

WaitPullRegistered is safe to call concurrently, including before Start. It returns a PullRegistrationError with the queue name and underlying gRPC error on a permanent rejection before readiness when the queue has no other registered stream. A rejected surplus stream is recorded and retried while another registered stream holds that queue. Startup errors from Start also reach the waiter. Cancellation returns the context error; stopping before readiness returns an error. Once all queues have registered, this wait keeps returning successfully.

The runnable examples wait for registration with a 30-second startup deadline and stop the Worker if startup fails. Go provides readiness in SDK 1.13.0; Java, Python, TypeScript, and Rust provide it in SDK 1.14.0. Register the Go HTTP handler at your readiness probe endpoint. In the other languages, call the boolean helper from your probe endpoint and return 200 when true or 503 when false.

def is_pull_greeting_ready(worker: AsyncWorker) -> bool:
    return worker.pull_registration_state().is_ready

Example: examples/python/dex_examples/production/pull_workers.py

Queues reports each queue's HasRegisteredStream and LastPermanentError. After initial registration, reconnects, ownership moves, transient disconnects, and backoff keep IsReady true. It becomes false only when a queue has no registered stream and a permanent error. Successful registration clears that queue's error and restores readiness when no other queue is failing. Stop alone does not clear readiness; remove the Worker from service during shutdown separately.

StopOnPermanentStreamRejection defaults to false, preserving retries. Set it to true to make Start return a PullRegistrationError for the first permanent rejection that leaves its queue without a registered stream, even after readiness. Rejections while another stream holds the same queue do not stop the Worker. Other queues may still hold streams while Start drains them. Draining uses StopDrainTimeout, then requests cancellation and waits for handlers to return. Handlers must respond to cancellation. Always call Stop before exiting; it remains safe after Start returns.

Routing and availability

Each pull Worker polls its own app Worker ID queue and every Flow type it registers. Dex first tries the app Worker ID that served the Flow previously. When no stream for that ID remains connected, such as after a Worker restart or deployment, Dex immediately tries the Flow type queue. Otherwise, it tries the Flow type queue after the sticky wait so another Worker can take the call. Sticky routing is best effort; keep durable state in Dex rather than depending on one Worker's memory. Flow type names must be unique within a Dex Server. An app Worker ID must identify one Worker process and change when that process restarts. The SDK generates one unless you set it explicitly.

Dex waits for an available Worker within the matching dispatch budget. If none is available, a Step receives a retryable unavailable error that consumes a Step retry. An RPC caller receives an Unavailable error directly. The default dispatch wait is 10 seconds, including up to 5 seconds for a sticky queue with connected streams. The wait never extends past the method deadline of the attempt. See Pull Worker dispatch for the wait settings.

When a pull Worker has the Flow type but lacks the requested Step type or RPC name, it declines the task before running a handler. Dex immediately tries the Flow type queue and excludes that Worker from this dispatch, even if the Flow was sticky to it. This reassignment does not consume a Step attempt. If no capable Worker accepts within the dispatch wait, the Step follows its failure and retry policy with an error that names the Step or RPC. An RPC caller gets the named error directly after waiting up to the dispatch wait of 10 seconds. This reassignment requires a Dex Server release that supports it. Against older Servers, the Worker returns the not-registered error as before.

Delivery is at least once. A call can be assigned again when its Worker disconnects before Dex receives any output. Make external side effects idempotent, for example by using the Flow ID and Step execution ID as an idempotency key.

Capacity and shutdown

The concurrent task limit caps active calls. The idle stream count keeps streams ready on each queue. Defaults are the CPU count and one idle stream per queue. Each stream handles one call at a time. At capacity, a Worker keeps its idle streams and declines an extra assignment so Dex can try another Worker.

SDK	Concurrent tasks	Idle streams per queue	Stop drain timeout	Shutdown
Go	MaxConcurrentTasks	IdleStreamsPerQueue	StopDrainTimeout	Stop
Python	max_concurrent_tasks	idle_streams_per_queue	stop_drain_timeout	stop
Java	maxConcurrentTasks	idleStreamsPerQueue	stopDrainTimeout	stop
TypeScript	maxConcurrentTasks	idleStreamsPerQueue	stopDrainTimeoutMs	close
Rust	max_concurrent_tasks	idle_streams_per_queue	stop_drain_timeout	stop

The Java and Rust option names are builder methods. The formula below uses the TypeScript spellings for the two capacity values; the same bound applies to every SDK.

A Worker gracefully closes surplus idle streams after tasks finish. Each queue keeps at least idleStreamsPerQueue registered idle streams, and readiness stays true during this cleanup. With F registered Flow types and no recent reserve declines, the steady-state bound after load drops is maxConcurrentTasks + (F + 1) × idleStreamsPerQueue streams. The extra queue is the Worker's app Worker ID queue. A queue with a recent reserve decline can keep one more idle stream for 30 seconds after its last decline. Allow up to F + 1 additional streams when sizing matching.maxConcurrentTasksCap, which defaults to 256 streams per app Worker ID. matching.maxStreams defaults to 100,000 registered streams on the node. The transient peak during a burst remains bounded by the matching caps. matching.maxQueueLength, matching.maxFlowTypes, and matching.maxFlowTypeLength also limit registration. Their defaults are 256 each.

Stop the Worker gracefully during deployment. The shutdown method stops taking new calls and waits for active handlers for the configured drain timeout, 30 seconds by default. After the deadline, it requests cancellation and waits for handlers to return. Long-running handlers should respond to cancellation. A Worker also finishes an active call before a server-directed reconnect.

Reserved RPC capacity

Since SDK 1.15.0, Workers reserve capacity so synchronous RPC calls can stay responsive while Steps run for a long time. Set ReservedRPCTasks in Go, reservedRpcTasks in Java and TypeScript, or reserved_rpc_tasks in Python and Rust. When unset, the reserve is one if the resolved task limit is at least two, otherwise zero. With the CPU-count default, Steps get one fewer slot than before. Set the reserve to zero to opt out. With a task limit of one and no explicit reserve, Worker start logs one warning that RPCs may wait behind long-running Steps and recommends a limit of at least two or an explicit reserve. The reserved slot keeps Flows responsive while Steps occupy the remaining capacity.

The reserve must be nonnegative and less than the concurrent task limit after its CPU-count default is resolved. Invalid values fail Worker construction.

RPC tasks include both API-direct and transactional InvokeWorkerRPC tasks. RPC and Step handlers count together against the task limit. Every Step task, WaitFor and Execute alike, is declined with NO_CAPACITY when the busy count reaches the task limit minus the reserve. RPC tasks can use the full limit. For example, a limit of eight and a reserve of two lets Steps start only while fewer than six handlers are busy.

Dex tries other Workers without consuming a Step attempt. If all Workers remain at their Step limit, a Step can wait until the dispatch wait expires, then receive NO_WORKER_AVAILABLE, consume an attempt, and follow its retry policy, as when all Workers are full. Idle streams stay available on every queue. A queue that declines a Step for reserved RPC capacity keeps one additional idle stream until 30 seconds after its last decline. Other queues keep their configured idle count. The Worker does not open this extra stream when all task slots are busy. Set idleStreamsPerQueue to at least two for Workers with heavy Step backlogs. Reserve declines do not close streams or change registration readiness. Each SDK emits a debug log with the Flow type for each reserve decline. Each Worker also exposes a cumulative reserve-decline counter: PullReservedRPCDeclines() in Go, getPullReservedRpcDeclines() in Java, pullReservedRpcDeclines in TypeScript, pull_reserved_rpc_declines in Python, and pull_reserved_rpc_declines() in Rust. Full-capacity rejections and unknown names do not increment it. It counts decline events, including repeated declines of the same Step dispatch while Dex retries it, so read its rate. The count never resets and remains available after shutdown. See each SDK README for logger configuration.

The startup examples set the task limit to eight and explicitly reserve one slot for RPCs, matching the default. These diagnostics helpers expose registration readiness and the cumulative reserve-decline counter for monitoring. A rising counter indicates Step capacity pressure; it does not make the Worker unready.

@dataclass(frozen=True)
class PullGreetingDiagnostics:
    is_ready: bool
    reserved_rpc_declines: int


def get_pull_greeting_diagnostics(worker: AsyncWorker) -> PullGreetingDiagnostics:
    return PullGreetingDiagnostics(
        is_ready=worker.pull_registration_state().is_ready,
        reserved_rpc_declines=worker.pull_reserved_rpc_declines,
    )

Example: examples/python/dex_examples/production/pull_workers.py

Scale to zero

server-zd has no Worker wake hook. When Workers may scale to zero, size the dispatch wait and Step retries for Worker startup time, so a Step retry can wait for a Worker to start.

Server and network setup

Workers connect to the server's FlowService port, which is api.listen. Preserve bidirectional HTTP/2 streaming through any load balancer or proxy. The SDK sends upstream Pong frames every 20 seconds while a task runs. Upgrade the SDK to get this behavior. Dex sends Ping frames to idle streams at matching.pingInterval. Set load balancer and proxy idle timeouts above both 20 seconds and matching.pingInterval, with margin for network delays. Older SDKs need idle timeouts above the longest task duration. The Pull Worker dispatch section lists the matching limits and wait settings.

All five SDKs, dexcli, and Dex Web can send the required bearer credential. Workers and Clients have separate credential options. See the authentication settings.