Crashes and tool call outcomes
View as MarkdownA crash repeats at most the one call that was running. This page lists what each failure repeats, and how a tool Step's failure decides the call's record. Agent Frameworks explains the outcomes and how to reconcile outcome_unknown.
Crash matrix
Each row of this table is a Step and the moment it is interrupted. The last three columns name the integration tests for three failures:
- Worker kill: the Worker gets SIGKILL, and a new Worker starts.
- Server restart: the Dex Server node gets SIGKILL and restarts on its data directory. The Worker keeps running.
- Takeover: the active node gets SIGKILL, and a hot standby takes the lease. The Worker keeps running.
Each language names the file that holds the test. Not tested means that no test covers the cell yet, so its text states the design, not a checked result.
| Step | Interrupted | What happens | Worker kill | Server restart | Takeover |
|---|---|---|---|---|---|
| RunModel | Before any output reaches Dex | That model call runs again, up to 3 attempts within 30 minutes. Then the turn fails. If tools already ran, a note in the history tells the next turn's model not to repeat them. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| RunModel | After output reached Dex | The same. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| CallTool, for an idempotent write | Before the first heartbeat reaches Dex | The call runs again with the same idempotency key. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| CallTool, for an idempotent write | While the tool runs | The same. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| CallTool, for a read | While the tool runs | The call runs again. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| CallTool | Every attempt | After the last attempt, a read records failed and a write records outcome_unknown. | Python, TypeScript | Python, TypeScript | Not tested |
| CallSingleAttemptTool | Before the first heartbeat reaches Dex | The call records outcome_unknown and does not run again. The model sees that the outcome is unknown. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| CallSingleAttemptTool | While the tool runs | The same. | Python, TypeScript, Go | Python, TypeScript | Python, TypeScript |
| AwaitToolApproval | While it waits | Nothing runs again. The wait lives in Dex Server. A later approval runs the tool once. | Python, TypeScript, Go | Python, TypeScript | Python, TypeScript |
| AwaitQuestionAnswer | While it waits | Nothing runs again. A later answer resumes the turn. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| AwaitTimer | While it waits | Nothing runs again. The Timer fires once. A Timer that came due during an outage fires when Dex Server is back. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| PollToolTask | Between two polls | Polling goes on with the same task. No second task starts. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| PollToolTask | During a poll | The poll runs again. No second task starts. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| CompactContext | During the summary call | Only the summary call runs again, up to 3 attempts within 30 minutes. The session stores one summary. If every attempt fails, the turn fails, and the next user message compacts again. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| A wait for a user message | While the session is idle | Nothing runs again. The next message starts a turn. | Python, TypeScript | Python, TypeScript | Python, TypeScript |
| Any Step | Queued, before Dex hands it to a Worker | The Step stays in the queue and uses no attempt. It runs as attempt 1 when a Worker takes it. | Python, TypeScript | server-zd | server-zd |
| CallTool for a read, PollToolTask, CompactContext | Before the first heartbeat reaches Dex | By design, the same as the rows above for that Step. | Not tested | Not tested | Not tested |
| AwaitToolApproval, AwaitQuestionAnswer, AwaitTimer | After the wait ended, while the handler that moves the turn on runs | By design, the handler runs again. The resolved approval, answer, or Timer stays recorded. | Not tested | Not tested | Not tested |
| Any Step | After the Worker sent the result, before Dex committed it | By design, Dex either commits the result or fails the attempt. A failed attempt follows the row for that Step. | Not tested | Not tested | Not tested |
| A tool the framework cannot defer | While it runs | The tool runs inside RunModel and repeats when RunModel runs again. It is at-least-once. | Not tested | Not tested | Not tested |
| A Claude Agent SDK turn segment | While it runs | Deferred custom tools run in their own CallTool or CallSingleAttemptTool Steps, as above. Model calls, inline read tools, and CLI tools such as Bash, Write, Edit, and Task run inside RunModel and repeat when it runs again. They are at-least-once. | Python, TypeScript | Not tested | Not tested |
Each adapter package repeats the Worker kill cases for RunModel and its tool Steps in its own test suite.
A running Worker survives both server failures. It reconnects by itself, and Dex fails the attempt that it held, so the row for that Step applies. Workers and clients must reach the new active node through an address that follows readiness. The Streams of the failed node are gone. See Stream backend.
The takeover tests use a hot standby and SIGKILL. A cold standby and a planned step down are covered at the server level: see the failover tests and the drain tests.
A single-attempt tool can still run twice in one narrow case: a cancelled attempt that keeps running. See At-most-once has one exception. For deploys and the drain timeout, see Graceful stop.
A call that never ran
When a tool Step fails, its failure route reads attempt_progress from the recovery error. See Failure and redelivery. CallTool and CallSingleAttemptTool use it the same way:
| attempt_progress | What the call records |
|---|---|
| NOT_STARTED | No attempt reached a Worker, so the tool did not run. The call records failed. Its error starts with "the tool did not run, so it is safe to call again". The model sees a failed tool call and can call the tool again. A call whose tool already asked the user a question ran before, so it follows the next row. |
| NOT_COMPLETED, or UNSPECIFIED when the server reported no progress | The tool may have run. A read records failed. Any other call records outcome_unknown, and the session does not run its tool again. |
To recognize such a call in get_agent_snapshot or your own code, compare the start of its error with TOOL_CALL_NOT_RUN_ERROR_PREFIX in Python, toolCallNotRunErrorPrefix in TypeScript, or ToolCallNotRunErrorPrefix in Go.
NOT_COMPLETED is conservative, so a call that never ran can still record outcome_unknown. For example, the server crashed after it recorded the attempt start but before the task reached the Worker. Treat that outcome as unknown. To check it, see Reconcile outcome_unknown.
A tool Step that waits in the queue does not fail and uses no attempt. See Keep tool Steps dispatchable.