Run Agents in Production

View as Markdown

Agent sessions run on the same Dex Server as every other Flow. Check each item before you put sessions in front of users. Each line links to the page with the details.

Item	What to know	Details
Dex Server	Dex Server runs in Dex Cloud or on your own infrastructure with object storage. One active node serves a cluster.	Choose How to Run Dex, Run Dex in Production
Authentication	Public API authentication is off by default. Turn it on before the server is reachable from other machines.	Authentication
Retention	A closed session stays readable for 7 days by default, then the server may delete it. Keep transcripts you need longer in your own store.	Retention
Session size	A session ends when its active state reaches the state limit, 64 MiB by default. Message retention, and compaction where the harness supports it, keep it below that.	Session limits
Model spend	Check spend per account or API key around each model call; Dex leaves spend limits to your code.	Spend limits
Credentials	Keep model API keys and tool credentials in the Worker's environment or a secret store, never in session input or messages.	Credentials
Approval deadlines	Set a deadline when a call must not wait forever. An approval with no deadline waits until someone answers, and the turn waits with it.	Approval deadlines
Upgrades	A deploy takes effect at the next model call or tool call. Some releases cannot run next to the one before: stop every older Worker before a newer one starts. Such a stop turns a running single-attempt tool into `outcome_unknown` and makes RPCs fail with Unavailable until Workers return; stop Workers gracefully first.	Upgrades
Workers	Each running model call, tool call, and RPC holds one Worker task slot. Size Workers for parallel tool calls.	Workers and timeouts
Streaming	The AgentEvents Stream carries live text to your UI. Size it for your traffic.	AgentEvents Stream
Tracing	Sessions can emit OpenTelemetry spans for each model call and tool call. Content capture is off by default.	Tracing
Deployment shape	Web replicas hold a Client. Worker processes run the session Flow.	Deployment shape
Calls during a deploy	`send_user_message` and `resolve_interrupt` are RPCs. They need a running Worker, and a retried `send_user_message` can queue a message twice.	RPCs during a deploy
Unknown outcomes	A write that a crash interrupted records `outcome_unknown` and does not run again. Decide who reconciles it.	Reconcile outcome_unknown

Deployment shape

A production setup has three kinds of process:

USERSWeb app replicas

Hold a Client.

RPCs, reads

Dex Server

Session state in object storage.

pull tasks (outbound only)

Worker processes

Run the session Flow.

Web app replicas hold a Client. They start sessions, send messages, and stream replies. They need the session Flow's definition to encode calls, but never run a Step.
Worker processes run the session Flow: every model call, tool call, and approval wait. For a small app, the web process can host the Worker too, as the Agent Quick Start program does.
Dex Server keeps every session. Workers connect out to it, so they need no inbound port.

During a rolling deploy, each Step runs the code of the Worker that takes it. A turn in progress finishes its next model call or tool call on the new code. A Worker that lacks a Step type declines it, and another Worker takes it. A few releases cannot run next to the one before, and need every older Worker stopped first; the Upgrades release notes name them.

RPCs during a deploy

Your application talks to a session through RPCs, such as send_user_message and resolve_interrupt. An RPC runs on a Worker. It never waits in a queue: if no Worker takes it within 10 seconds, the call fails with Unavailable. During a deploy that stops every Worker, these calls fail until a new Worker connects. The session itself loses nothing.

Retry on Unavailable. Retry with backoff until a Worker is back. A failed call with Unavailable changed nothing.
resolve_interrupt is safe to retry. An approval that already went through returns False the second time, and nothing runs again.
send_user_message can send twice. Dex runs a repeated RPC again. If the response is lost, a retry can queue the same message twice. Before you resend, read the session with get_agent_snapshot and look for the message in messages and queued_messages. If it is queued twice, remove the copy with delete_queued_message.

This helper does that. Each tab shows the whole module. It retries a send that may not have arrived, checks the session before each retry, and deletes an extra queued copy. It tells messages apart by their text, so call it once for each message a user sends.

safe_send.pyPython
123456789101112131415161718192021222324252627282930
"""Sends a user message to a session once, even when a send must be retried."""
from __future__ import annotations

import asyncio
from typing import Protocol

import grpc  # type: ignore[import-untyped]
from dex import DexServiceError
from dex_agents import AgentSnapshot, MessageRole

# A lost response leaves a send's outcome unknown; these codes mean "check first".
UNCERTAIN_CODES = (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.DEADLINE_EXCEEDED)

class MessageSessions(Protocol):
    async def get_agent_snapshot(self, session_id: str) -> AgentSnapshot: ...

    async def send_user_message(self, session_id: str, text: str) -> bool: ...

    async def delete_queued_message(self, session_id: str, message_id: str) -> bool: ...

async def send_user_message_once(
    sessions: MessageSessions,
    session_id: str,
    text: str,
    *,
    attempts: int = 5,
    retry_delay_seconds: float = 1.0,
) -> None:
    """Queues one user message, retrying a failed send without queuing it twice.

… 37 more lines

Example: examples/python/agent-harnesses/agent_harness_examples/quick_start/safe_send.py

quick-start-safe-send.tsTypeScript
101112131415161718192021222324252627282930
… 9 import lines
// gRPC codes UNAVAILABLE and DEADLINE_EXCEEDED: the send may or may not have arrived.
const UNCERTAIN_CODES: readonly number[] = [14, 4];

export interface MessageSessions {
  getAgentSnapshot(sessionId: string): Promise<AgentSnapshot>;
  sendUserMessage(sessionId: string, text: string): Promise<boolean>;
  deleteQueuedMessage(sessionId: string, messageId: string): Promise<boolean>;
}

export function messageSessions(client: Client, flow: AgentSessionFlow): MessageSessions {
  return {
    getAgentSnapshot: (sessionId) => readAgentSnapshot(client, flow, sessionId),
    sendUserMessage: (sessionId, text) => sendAgentUserMessage(client, flow, sessionId, { text }),
    deleteQueuedMessage: (sessionId, messageId) =>
      client.invokeRPC(flow.deleteQueuedMessage, sessionId, { messageId }),
  };
}

/**
 * Queues one user message, retrying a failed send without queuing it twice.
… 51 more lines

Example: examples/typescript/src/products/agent-harnesses/quick-start/quick-start-safe-send.ts

Limits lists the other requests that can repeat.

Limits: the defaults and known gaps of Dex as a whole.
Sessions: what each Step adds to a turn.

Deployment shape

RPCs during a deploy

Next