---
title: "Install and configure server-zd"
description: "Install and start"
canonical_url: "https://docs.superdurable.io/reference/server/0.1.23/configuration/"
source: "docs/reference-server_versioned_docs/version-0.1.23/configuration.mdx"
locale: "en"
---
# Install and configure server-zd
## Install and start
Each **server-zd/vX.Y.Z** GitHub Release publishes **dex-server-zd** archives
for Linux amd64, Linux arm64, macOS amd64, and macOS arm64. The release also includes a
**checksums.txt** file. Verify each archive against it before you extract it.
Linux archives require glibc 2.28 or newer.
The **dex-server-zd** container image is published for linux/amd64 and
linux/arm64. It runs **serve** with the configuration file
**/etc/dex-server-zd/config.yaml**. The process runs as UID 10001 and GID 10001.
```bash
dex-server-zd serve --config /etc/dex-server-zd/config.yaml
```
**serve** reads every setting from one YAML file. The
[example configuration](https://github.com/superdurable/dex/blob/server-zd/v0.1.3/server-zd/config/dex-server-zd.example.yaml)
lists every key with its default. An unknown key fails startup. An invalid
value also fails startup, and the error names the YAML path of the rejected
field. The image includes the example at
**/etc/dex-server-zd/dex-server-zd.example.yaml** for reference only.
## Minimal production configuration
```yaml
cluster:
nodeId: dex-zd
objectStore:
region: us-east-1
bucket: BUCKET
prefix: PREFIX
localStorage:
directory: /var/lib/dex-server-zd/store
lease:
duration: 60s
retention:
closedFlowDuration: 7days
metrics:
listen: 0.0.0.0:9090
```
**cluster.nodeId**, **objectStore.bucket**, and **localStorage.directory** are
required. Replace **BUCKET** and **PREFIX** with your bucket and key prefix.
Leave **objectStore.endpoint** empty to use AWS endpoint resolution. For an
S3-compatible store, set it to an HTTP or HTTPS URL. Set
**objectStore.forcePathStyle** when that store needs path-style addressing.
A node ID has 1 to 63 characters. It uses lowercase letters, digits, and
hyphens, and it starts with a letter or a digit.
One object storage prefix holds one Dex cluster, and a cluster runs one node.
The first start records **cluster.nodeId** and **cluster.partitionCount**.
Neither can change later. A start with a different node ID fails and names the
recorded one. A start with a different partition count fails and reports both
counts. Never point two deployments at the same prefix.
## Single node on local disk
**dex-server-zd local** runs one production node without external object
storage. It starts an embedded s3local object store in the same process and
keeps all data in one directory on local disk:
```bash
dex-server-zd local --data-dir /var/lib/dex-server-zd [--listen 127.0.0.1:8801] [--log-dir ] [--config ] [--attribute-store-config ] [--durability strict|process-crash-safe|relaxed]
```
The data directory holds the embedded object store in **s3**, the local store in
**store**, and the lock file **dev.lock**. The embedded object store is the
source of truth, as the bucket is for **serve**. Put the data directory on a
persistent disk. **--log-dir** also writes the log to
**dex-server-zd.log** in that directory.
The optional **--config** file holds overrides. It uses the keys of the
[example configuration](https://github.com/superdurable/dex/blob/server-zd/v0.1.3/server-zd/config/dex-server-zd.example.yaml).
Put only the keys you want to change in it. Every key in the file replaces the
single-node default. Do not copy the whole example. A copy sets **api.listen**
to **0.0.0.0:8801**, which has no authentication unless the file also
configures authenticators. It also sets **lease.duration** to 10 seconds.
**local** sets the object storage connection, the node ID **dev**, and the
local store directory itself. **--listen** overrides **api.listen** from the
file.
The optional **--attribute-store-config** flag reads only the
**attributeStore** section of a YAML file. That section must define at least
one store. The **--config** file must not also set **attributeStore**.
The node ID **dev**, the bucket **dex-dev**, the lock file **dev.lock**, and the
**dev** section of the overrides keep their **dev** names on purpose. **local**
shares them with **dev**, so a data directory can move between the two
commands.
### Partition count
Since server-zd v0.1.20, **dev** and **local** default
**cluster.partitionCount** to 32. **serve** defaults it to 1024. One node owns
every partition. The default number of applier lanes is the larger of 8 and the
CPU count, at most the partition count, so 32 partitions lower it only on hosts
with more than 32 CPUs. With the builtin search backend, fewer partitions also
make each SearchFlows call scan less. Set **cluster.partitionCount** in the
overrides to choose another power of two.
The first start records the partition count, and it cannot change later. A
later start with a different count fails. The error names the data directory
and both counts, and asks for the recorded count.
server-zd before v0.1.20 and dexcli before 1.37.0 created data directories with
1024 partitions. To keep using such a directory, add this to the overrides.
With **dexcli dev**, pass the file with **--zd-config**.
```yaml
cluster:
partitionCount: 1024
```
If you do not need its data, delete the data directory instead. The next start
creates it with 32 partitions.
An explicit **commit.applierLanes** must not exceed the partition count. If the
overrides set it above 32 without **cluster.partitionCount**, **dev** and
**local** fail to start. The error asks you either to set
**cluster.partitionCount** or to lower **commit.applierLanes** to at most 32.
Use 1024 for a data directory created before v0.1.20.
To open a data directory created with 32 partitions using server-zd before
v0.1.20 or dexcli before 1.37.0, set **cluster.partitionCount** to 32 in the
overrides. Those releases default to 1024.
### Durability
**dev.durabilityMode** selects how the embedded object store writes
acknowledged data. The **--durability** flag overrides it with the same modes,
spelled **strict**, **process-crash-safe**, and **relaxed**.
| Mode | Survives | Cost |
| -------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| **strict** (default) | A process kill, a host crash, and power loss. | Every write is flushed with fsync before the reply. |
| **processCrashSafe** | A process kill. A host crash or power loss can lose recent writes. | Faster. Every write reaches the files before the reply, without fsync. |
| **relaxed** | Only a graceful stop. | Writes stay in memory until a graceful stop. After an abrupt exit, every start fails until you delete the data directory. |
Keep the default **strict** mode in production.
### One node
**local** runs exactly one node. It holds an exclusive lock on **dev.lock** for
its whole run, so a second **local** or **dev** process on the same data
directory fails to start. It has no standby. It rejects **standby.mode: hot**
and **objectStore.hardDeleteStore**. While it holds the lock, no other process
can write its object store. A restart after a crash therefore takes over the
held lease at once. **lease.duration** defaults to 30 seconds. On a filesystem
without file locks, the commands cannot exclude each other, and a restart waits
for the previous lease to expire. Run only one process per data directory
there.
On SIGTERM, **local** drains running Step attempts for up to
**lifecycle.attemptDrainTimeout**, 30 seconds by default, and then stops as
described in [Graceful shutdown](https://docs.superdurable.io/reference/server/0.1.23/kubernetes.md#graceful-shutdown). Give a service manager at
least 45 seconds to stop it.
[Streams](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#streams) use the **memory** backend by default, as in **serve**.
Stream messages are lost when the process stops or restarts.
**local** listens on **127.0.0.1:8801** by default, so only the same host can
reach it. To serve clients and Workers on other hosts, pass
**--listen 0.0.0.0:8801** or set **api.listen** in the overrides. **local** has
no [authentication](https://docs.superdurable.io/reference/server/0.1.23/authentication.md) unless the overrides configure
authenticators. When it listens on a non-loopback address without
authenticators, it logs a warning at startup. Configure authentication before
you expose it. Set **metrics.listen** in the overrides to enable
[metrics](https://docs.superdurable.io/reference/server/0.1.23/metrics.md).
### Disk use
**dev** and **local** default **snapshot.logDeletionDelay** to 1 hour.
**serve** defaults it to 24 hours. An explicit value in the overrides always
wins, and it must be greater than zero. A log object is deleted once a snapshot
that covers it is this old. The age counts from the snapshot's creation time,
before its upload finishes. A snapshot beyond **snapshot.retainedCount** is
deleted once the newer snapshot that displaced it is this old. A longer delay
keeps older snapshots and their log for fallback and lets a stopped node catch
up from the log, at the cost of disk space. The delay must also exceed the
longest snapshot upload. Changed in 0.1.21: before, **dev** and **local** also
defaulted to 24 hours.
The embedded object store shares the disk with the local store. Plan for the
size of **store**, plus the object store: the snapshots that
**snapshot.retainedCount** keeps, and the log and displaced snapshots written
during **snapshot.logDeletionDelay**. Together, the object store can reach two
to three times the data written during the delay.
A Flow that rewrites a large value on every Step needs more disk. Its history
records wait for a newer [checkpoint](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#history-checkpoints) before they can be
deleted. For such Flows, lower **history.checkpointCommandThreshold** from its
default of 1000. Better, store incremental values, one entry per item, instead
of rewriting one large value.
### Back up and restore
To back up, stop **local** with SIGTERM, wait for it to exit, and copy the
whole data directory. A copy of a running data directory can be inconsistent.
To restore, stop **local**, replace the data directory with the copy, and
start **local** again. Writes acknowledged after the backup are lost.
The general rules in [Backup and restore](https://docs.superdurable.io/reference/server/0.1.23/object-storage.md#backup-and-restore) apply too. With a
PostgreSQL or ParadeDB search backend, the restored data directory needs a new
**search.postgresSchema** or **search.paradeDbSchema**, as that section
explains.
### Compared with dev and serve
| | **dev** | **local** | **serve** |
| ------------------------------------- | --------------------------------------- | ------------------------------------- | ----------------------------------------------- |
| Purpose | Development, started by **dexcli dev**. | Single-node production. | Production on external object storage. |
| Object storage | Embedded, in the data directory. | Embedded, in the data directory. | An external S3 or S3-compatible bucket. |
| Nodes | One. | One. | One active process, with optional hot standbys. |
| Step attempt drain on stop | 2 seconds. | 30 seconds. | 30 seconds. |
| Lease duration | 30 seconds. | 30 seconds. | 10 seconds. |
| Default **snapshot.logDeletionDelay** | 1 hour. | 1 hour. | 24 hours. |
| Default **history.windowEvents** | 5000 | 5000 | 50000 |
| Default partition count | 32 | 32 | 1024 |
| Default listen address | 127.0.0.1:8801 | 127.0.0.1:8801 | 0.0.0.0:8801 |
| Configuration | Flags and an optional overrides file. | Flags and an optional overrides file. | One YAML file. |
**dev** and **local** use the same data directory layout. Either command can
serve a data directory that the other one wrote. Stop one before you start the
other.
## Retention
**retention.closedFlowDuration** sets how long a closed Flow run stays
available. It defaults to 7 days. After that time, the run and its retained data
become eligible for deletion. Write the value as a duration, such as **72h** or
**30days**.
Each run records the value when it starts. A new value applies only to runs that
start after the node restarts with it. Runs that already started keep their
recorded duration.
From server-zd v0.1.8, a transactional or locking RPC on a closed run returns
NOT_FOUND with FLOW_NOT_EXISTS before a Worker runs its handler. A retry with an
already recorded request ID returns its earlier reply. Other RPCs still run
their handlers on a closed run that is not yet purged, so they can read it.
server-zd rejects any write that such a handler returns.
## History window
server-zd keeps a recent window of each run's history. The window holds the
**FlowStart** event and about the newest **history.windowEvents** history
events: 50,000 on **serve** and 5,000 on **dev** and **local** by default. Older
events are pruned in chunks of up to 1,000. Pruning never rejects a command or
fails a Flow.
The window counts history events. A command that changes the Flow but records
no event, such as a failed Step attempt that will be retried, counts as one
event. Pruning stops only between commands, so a run keeps slightly more or
fewer events than the window size. One command moves the window past at most
about 2,000 history records, so a larger chunk takes several commands. Deleting
the pruned records from local disk also proceeds a bounded amount per command.
Since 0.1.22, a run that stops receiving commands, such as an idle or closed
run, still finishes that deletion. About one minute after the run's last
command, the lease holder commits internal cleanup commands for the run until
the deletion is done. They record no history event.
A run can set its own window size with the **history_window_events** field of
its Flow configuration. Set it at start with the Flow configuration override, or
change it later with **UpdateFlowConfig**. A smaller window takes effect at the
run's next pruning. A larger window keeps more of the events recorded after the
change; pruned events never come back. Once set, the run's window can be changed
but not cleared. A child Flow inherits its parent's window unless its own
override sets one. Values must be positive and at most
**history.maxWindowEvents**. StartFlow and UpdateFlowConfig reject other values
with INVALID_ARGUMENT. A Step result that starts a SubFlow with such an override
is an invalid Worker response: the Step attempt fails and is retried under the
Step's retry policy.
History reads, such as **dexcli flow history --all**, return only the retained
events. [Time travel](https://docs.superdurable.io/reference/server/0.1.23/runtime-client.md#time-travel) can target only
a point inside the window. Time travel to the beginning is always available.
Each history page also reports the range of event IDs that the window removed.
Step method events record the event ID of the event that started their Step
execution. Execute events also record the event ID of the Step's WaitFor
completion. A client can use these IDs to tell a removed related event from one
on another page.
| Key | Default | Meaning |
| ------------------------------------------------------------------------------ | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **history.windowEvents** | 50000 for **serve**; 5000 for **dev** and **local** | Approximate number of the newest history events kept in the window of a run that sets no window of its own. Pruning starts when the window exceeds this value plus one prune chunk. Must be positive. The deprecated name **history.windowSemanticEvents** is still accepted with a startup warning; setting both names is an error. Changed in 0.1.21: the window counted semantic commands, not events, and the key was named **history.windowSemanticEvents**. A run that started before 0.1.21 keeps its recorded size, which now counts events, so its window reaches a little less far back. Changed in 0.1.23: the default was 10000 in every command. Runs that started earlier keep the size they recorded. |
| **history.pruneChunkSize** | 1000 | Most old history events pruned at a time. A run prunes at most one tenth of its window at a time, and at least one event, when that is smaller. Must be positive and must not exceed **windowEvents**. Changed in 0.1.21: the unit was semantic commands. |
| **history.maxWindowEvents** | 100000 | Largest window that a run's **history_window_events** may set. StartFlow and UpdateFlowConfig with a larger value fail with INVALID_ARGUMENT. A Step result that starts a SubFlow with a larger value is an invalid Worker response, and the Step attempt is retried under the Step's retry policy. It does not limit **windowEvents**. Must be positive. Changing it affects later requests only. Since 0.1.21. |
| Each run records **windowEvents** and **pruneChunkSize** when it starts. A new | | |
| value applies only to runs that start after the node restarts with it. Changed | | |
| in 0.1.23: a child Flow that a SubFlow starts records them too; before, it | | |
| recorded 10000 and 1000. | | |
Changed in 0.1.23: **history.maxRetainedSupersededBytes** was removed. It capped
the superseded values that a run without history offload kept on local disk for
time travel. A run that starts with 0.1.23 or later moves its window only by its
event count. Without offload, it deletes a superseded value when the window
passes it. With offload, it deletes the value once its history is uploaded. The
key is still accepted, has no effect, and logs a warning at startup. A run that
started earlier keeps the cap it recorded.
### Checkpoints
server-zd writes checkpoints of each run's current attempt state. Time travel
restores the nearest checkpoint before its target and replays the commands
after it. A command is one change that server-zd commits for a run, such as a
Client write, a Step completion, or an internal acknowledgment. Each new attempt
starts with a checkpoint. After that, server-zd writes a checkpoint when either
threshold below is reached, whichever comes first. Both counts restart at each
checkpoint.
| Key | Default | Meaning |
| -------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **history.checkpointCommandThreshold** | 1000 | Commands since the last checkpoint that trigger a new one. Every command counts, including bookkeeping commands such as acknowledgments, not only commands that record history events. Time travel replays at most this many commands. |
| **history.checkpointRowWriteBytesThreshold** | 1048576 (1 MiB) | Bytes of attempt-scoped rows written since the last checkpoint that trigger a new one. Attempt-scoped rows are the current attempt's Attribute, Channel message, Channel metadata, Step execution, Step counter, lock, waiter, and timer rows. Every write of such a row adds its size, so rewriting one row counts each time. A large value stored apart from its row counts only as its reference. The effective threshold is the larger of this value and the smaller of two sizes: the run's current state rows and the rows of the last checkpoint. A larger state therefore makes checkpoints sparser. |
Lower values write checkpoints more often. Time travel then replays fewer
commands, but each checkpoint copies the attempt's rows, so the run stores more
checkpoint bytes and spends more work per command. Higher values store fewer
checkpoints, and time travel replays more commands. Both values must be
positive.
Each run records these values when it starts. A new value applies only to runs
that start after the node restarts with it. Runs that already started keep
their recorded values.
The lease holder computes each time travel. It restores the checkpoint, replays
the commands after it in memory, and then commits the result.
| Key | Default | Meaning |
| --------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **history.timeTravelReplayMemoryBytes** | 1073741824 (1 GiB) | Most bytes that one time travel holds in memory while it computes its result: the commands it replays from the checkpoint to its target, the restored state and its values, the external input it reapplies, and the staged result. When write reapply is enabled, memory grows with the number and size of writes after the target, even if the restored state is small. A ResetFlow that needs more fails with RESOURCE_EXHAUSTED and changes nothing. Choose a later target point, set **skip_writes_reapply** to **true**, or raise this value. Must be positive. A new value applies to ResetFlow requests after the node restarts with it. Since 0.1.22. |
Changed in 0.1.21: **history.checkpointCommandThreshold** was renamed from
**history.checkpointCommandInterval**, and its default changed from 10000 to
1000\. **history.checkpointRowWriteBytesThreshold** was renamed from
**history.checkpointMinWrittenBytes**. The old keys still work and log a
warning at startup. Setting one key under both names fails startup with a
configuration error.
### History offload
Each run decides when it starts whether it offloads its history to object
storage. The **offload_history** field of the run's Flow configuration holds the
choice, and it defaults to true. Set it in the Flow configuration override of
StartFlow, or in the override of a SubFlow. A child Flow inherits its parent's
choice unless its own override sets one. A run keeps its choice for its whole
life: UpdateFlowConfig rejects a request that sets it with INVALID_ARGUMENT.
Since 0.1.23.
An offloaded run seals its history into segments and uploads them to object
storage. It then deletes the uploaded history from local disk. Local disk keeps
the run's active state, the history that waits for upload, and a small metadata
row for each point in the history window. The lease holder deletes a segment's
object after the history window passes the whole segment.
A run without offload keeps all of its history on local disk and never writes a
history object. The history window still applies to it. The run prunes its
oldest events and deletes the superseded values that the window passed.
Choose for each run:
- Keep offload, the default, for most runs and for every long-running run. Local
disk then holds only the newest history and the current state of the run.
- Turn it off for many short runs that close soon after they start. Each
offloaded run writes at least one history object when it closes, and deletes
it when its retention ends. Short runs at a high rate therefore add object
storage requests, while their small history costs little local disk until
their retention ends.
- A run without offload keeps every superseded value that its window covers on
local disk. No byte cap applies. A run that rewrites large values can then use
much local disk. Keep offload for such a run, or give it a smaller
**history_window_events**.
Runs that started before 0.1.23 recorded no choice. Each follows, for its whole
life, the offload mode that its partition recorded at the upgrade:
- Where offload was off at the upgrade, these runs keep their new history on local
disk and never migrate to object storage. That is the case on **serve** unless
**history.offloadEnabled** was on, and in a **dev** or **local** data
directory that 0.1.23 starts directly after 0.1.21.
- A partition where an earlier release turned offload on and then off again is
off as well: its runs seal no new segments and stop migrating. Segments sealed
before can still finish uploading, and uploaded history stays readable.
- Where offload was on at the upgrade, as 0.1.22 sets it by default on **dev**
and **local**, the runs it offloaded stay offloaded. A run that recorded
history before offload was turned on migrates that history lazily, as
described below.
- These runs also keep the superseded byte cap they recorded and follow their
delete switch. UpdateFlowConfig can still change that switch.
Changed in 0.1.23: the server switches of history offload were removed.
- **history.offloadEnabled** was removed. **true** is still accepted, has no
effect, and logs a warning at startup. **false** fails startup with a
configuration error that asks you to remove the key. To keep a run's history
on local disk, set the run's **offload_history** to false instead.
- **history.deleteExpiredObjects** was removed. It is still accepted, has no
effect, and logs a warning at startup. A run that starts with 0.1.23 or later
and offloads always deletes the objects of its expired segments.
- A run that starts with 0.1.23 or later ignores its
**FlowConfig.delete_expired_history_objects**. StartFlow, SubFlow overrides,
and UpdateFlowConfig still accept the field, and StartFlow drops it.
- Partitions no longer record an offload mode at startup, so
**dexzd_history_offload_mode_skipped_partitions_total** was removed.
| Key | Default | Meaning |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **history.segmentTargetBytes** | 4194304 (4 MiB) | Since 0.1.21. Bytes of history records, checkpoints, and superseded values after which an offloaded run seals its open history segment for upload. From 1 byte to 268435456 (256 MiB). |
| **history.supersededValueGracePeriod** | 10m | Since 0.1.22. How long a superseded value of an uploaded segment stays on local disk after the command that superseded it, measured in command time. Must be at least 1 microsecond. Each run records the value when it starts, so a new value applies only to runs that start after the node restarts with it. Runs that started before 0.1.22 keep 10 minutes. |
| **history.uploadConcurrency** | 256 | Since 0.1.21. Most sealed history segments the lease holder reads, encodes, and uploads at once. From 1 to 1024. Each upload in flight holds about one encoded segment in memory, and **history.uploadMaxInFlightBytes** bounds their total. The limit adapts between **history.uploadMinConcurrency** and this value; see [Upload concurrency](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#history-upload-concurrency). Changed in 0.1.23: the default was 16, and the limit stayed fixed at this value. |
| **history.uploadMinConcurrency** | 16 | Since 0.1.23. Uploads the lease holder allows at once while no upload waits. From 1 to 1024. A value above **history.uploadConcurrency** counts as **history.uploadConcurrency**, so a configuration that sets only a smaller **history.uploadConcurrency** keeps a fixed limit. |
| **history.uploadMaxInFlightBytes** | 134217728 (128 MiB) | Since 0.1.23. Most bytes that the lease holder's uploads in flight hold at once for encoding history segments and keeping the encoded objects. From 1048576 (1 MiB) to 68719476736 (64 GiB). Each upload reserves the most memory its encoding can allocate before it encodes, and waits while the budget is used. A segment larger than the budget uploads alone. See [Upload concurrency](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#history-upload-concurrency). |
| **history.readCacheMaxBytes** | 1073741824 (1 GiB) | Since 0.1.22. Most bytes of uploaded history that the node keeps in its disk read cache. 0 disables the cache. |
| **history.readCacheDirectory** | **localStorage.directory** with the suffix **-history-read-cache**, such as **/var/lib/dex-server-zd/store-history-read-cache** | Since 0.1.22. Directory of the disk read cache, which the cache owns. It must be outside **localStorage.directory**, which a local store rebuild sets aside, and must not contain **..** components; startup fails otherwise. If the node cannot create it, the node logs one warning and reads from object storage without a cache. |
| **history.objectReadConcurrency** | 32 | Since 0.1.22. Most object storage reads of uploaded history segments that the node runs at once. From 1 to 1024. A history read beyond it waits for a free read. |
An offloaded run divides its history into segments. A segment is sealed when it
reaches **history.segmentTargetBytes**, when 10,000 values were superseded inside
it, or when the run closes. Each run records **history.segmentTargetBytes** when
it starts, so a new value applies only to runs that start after the node restarts
with it.
Once a segment is uploaded, the run deletes its history records and checkpoints
from local disk. The run's first record and checkpoint stay. The small metadata
row of each checkpoint, and of each history record that is a time travel point
or holds events, stays until the window passes it. A superseded value is a
stored value that the run no longer holds, such as the earlier value of a
rewritten Attribute. A superseded value of an uploaded segment is deleted from
local disk once **history.supersededValueGracePeriod**, 10 minutes by default,
has passed since the command that superseded it, measured in command time. A
value reference that a Worker or client just read therefore still loads from
local disk. A Worker result, client write, or RPC result that hands back a
reference to such a value also succeeds after that period. Before it records
the result, the lease holder reads the value from its uploaded segment and
restores it on local disk. A restored value that no state holds again can stay on
local disk until the window passes it. A reference to a value that the window
passed is rejected, as without offload. Each command deletes a bounded amount, and later commands of
the run continue. Since 0.1.22, a run that receives no more commands, such as an
idle or closed run, still finishes. The lease holder commits internal cleanup
commands for the run, which record no history event. The first comes about one
minute after the run's last command, or when the period of a waiting superseded
value ends, whichever is later. While a run waits for such periods, it gets at
most one cleanup command a minute. Values superseded in the open segment of an
idle running run stay on local disk until that segment is sealed.
A run that recorded history without offload, on a partition whose offload an
earlier release then turned on and that was still on at the upgrade, migrates
its older history lazily. Only such runs migrate. Each command that records history and leaves the run running
seals one more segment of that older history, oldest first. When a segment
reaches the command itself, the run continues like a run that started with
offload. Nothing else drives the migration. A run that receives no more
commands keeps its older history on local disk, for as long as it stays idle.
Events that the history window removed before the migration stay removed. A run
that is already closed does not migrate: its local history stays until its
retention ends and the run is deleted. A run that closes during its migration
keeps the rest of its older history on local disk until it is deleted.
A **dev** or **local** data directory where an earlier release turned offload
off and on again can hold history of commands that belong to no segment. That
history stays on local disk and follows the window rule.
The lease holder uploads each sealed segment to the primary object store, under
the **history/** prefix of the bucket that holds the commit log. It uploads a
run's segments oldest first and records each upload before the next one. A
segment's content depends only on committed state. A new lease holder therefore
writes the same object again and accepts an existing object with the same
content. An existing object with different content is replaced; the node logs an
error and increments **dexzd_history_upload_checksum_mismatches_total**.
Uploads never delay commands. While object storage fails them, they retry
with backoff, sealed segments stay on local disk, and the upload backlog metrics
grow; see [Metrics](https://docs.superdurable.io/reference/server/0.1.23/metrics.md). A permanent rejection, such as
denied access, is retried once a minute instead. Retention
postpones the deletion of a closed run until its sealed segments are uploaded.
Deleting a run also deletes every object under its **history/** prefix. That
deletion starts a minute after the run's deletion begins, so that uploads an
earlier lease holder abandoned have ended on its side. An upload that object
storage accepted just before can still leave an object of the deleted run; no
read returns it. Since 0.1.22, the lease holder lists **history/** every six
hours and deletes such objects once they are an hour old. Do not configure a
bucket lifecycle rule that expires objects under **history/**. The lease holder
deletes old history objects itself, as described next.
#### Upload concurrency
Changed in 0.1.23: the number of uploads in flight adapts to the backlog. Before,
it stayed at **history.uploadConcurrency**, 16 by default. Every closed run
seals and uploads at least one segment, so short runs need one object storage
write each. With a write round trip of about 30 ms, 16 uploads at once drain
about 530 segments a second.
The limit starts at **history.uploadMinConcurrency**. Each upload that object
storage accepts while other uploads wait raises the limit by one, up to
**history.uploadConcurrency**. The limit therefore about doubles each round trip
while a backlog grows. Each upload that ends while no upload waits lowers the
limit by one, back to **history.uploadMinConcurrency**.
A retryable object storage failure, such as an S3 503 SlowDown reply, halves
the limit, but not below **history.uploadMinConcurrency**. Uploads that started
before the last reduction do not reduce it again. After a reduction, the limit
grows again only after as many successful uploads as the reduced limit. The
failed upload retries with backoff, as before.
Memory stays bounded by **history.uploadMaxInFlightBytes**. After an upload reads
a segment's rows, it computes the most memory encoding them can allocate: the
block buffers, the compressed frames, the header, and the object. It reserves
that much before it encodes, then keeps only the encoded object's size until
the upload ends. An upload waits while the reservations use the budget.
Reservations count whole KiB. Only a segment larger than the whole budget
exceeds it, and that segment uploads alone. The budget does not count the rows
an upload reads from local disk before it reserves, or the compressor's working
state.
**dexzd_history_upload_concurrency_limit** shows the current limit and
**dexzd_history_upload_in_flight_bytes** the reserved bytes. To keep the limit
fixed, set **history.uploadMinConcurrency** and **history.uploadConcurrency** to
the same value.
Since 0.1.22, a segment expires once the history window passes every command it
holds. GetHistoryEvents, LoadBlobs, and ResetFlow never read an expired segment,
from object storage or from the disk read cache, whether or not its object still
exists. The lease holder deletes the object of each expired segment about one
hour after it expired, so reads that started before the expiry can finish.
Object storage use therefore stays bounded by the window, also for a run that
never closes. Deleting a run deletes its remaining objects with the rest.
Changed in 0.1.23: a run that starts with 0.1.23 or later and offloads always
deletes the objects of its expired segments. A run that started under 0.1.22
keeps following its delete switch. When that switch is off, the run's expired
objects stay in object storage until its retention ends and the run is deleted.
Until then, anyone who can read the bucket can read the old history they hold,
including earlier values of rewritten Attributes. To delete them sooner, turn
the run's switch on with UpdateFlowConfig. SDK 1.40.0 and later no longer send
the field, so use an earlier SDK or a direct gRPC request. For sensitive values,
use hard-delete Attributes.
GetHistoryEvents reads the events of uploaded segments from object storage and
every other event from local disk. Each event comes from one place, so paging
returns every event once, also while uploads continue. Events of a run without
offload, and of commands that belong to no segment, stay on local disk and
follow the window rule. A read never returns events that the window
pruned, even when their segment object still exists. A read that object storage
fails, or whose uploaded segment object is missing, fails with UNAVAILABLE; the
client may retry it. A page that needs many segments can hold fewer events than
requested; its next page token continues the read.
ResetFlow reads history the same way. The node computes the time travel from
uploaded segments and local disk. It fetches the segments it needs while it
holds no view of the local store. A ResetFlow whose segment read fails, or whose
uploaded segment object is missing, fails with UNAVAILABLE and changes nothing;
the client may retry it. The history it reads from object storage counts against
**history.timeTravelReplayMemoryBytes** apart from the computation itself.
LoadBlobs reads a superseded value that offload deleted from local disk from its
uploaded segment. Values of one request that share a segment take one object
storage read. A value that the window passed is not returned, as before. A read
that object storage fails, or whose uploaded segment object is missing, fails
with UNAVAILABLE. A ResetFlow whose target state holds such a value reads it
from object storage in the same way. Such values, whether read from the cache or
from object storage, count against **history.timeTravelReplayMemoryBytes**.
Changed in 0.1.22: while history offload is on, a LoadBlobs request names at
most 1000 values. A request with more entries fails with INVALID_ARGUMENT when
the partition of a Flow it names offloads history. With offload off, LoadBlobs
takes any number of entries, as before.
Changed in 0.1.23: the limit counts the entries whose run offloads history. A
request with more than 1000 such entries fails with INVALID_ARGUMENT. Entries of
runs without offload are not limited.
The node keeps uploaded segment headers in memory and keeps segment headers,
records, and values in its disk read cache, up to **history.readCacheMaxBytes**.
Uploaded segments never change, so cached entries are never stale. The cache
survives a restart and is never authoritative: deleting its directory while the
node is stopped only costs object storage reads. The node never sets the cache
aside with a local store it rebuilds.
## Limits
server-zd enforces these limits. A request that a limit rejects fails at once;
it does not wait for capacity. The request ID limit never rejects a request; it
evicts the oldest request IDs instead.
| Key | Default | Meaning |
| ------------------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **values.maxValueBytes** | 16777216 (16 MiB) | Largest serialized application value, such as an Attribute value or a Channel message. A larger value is rejected. |
| **api.grpcMaxMessageBytes** | 33554432 (32 MiB) | Largest incoming or outgoing gRPC message. Must exceed **values.maxValueBytes**. Since 0.1.21, a GetHistoryEvents page ends before an event that would make the response larger, and its next page token continues the read. A single event larger than this limit fails the read with RESOURCE_EXHAUSTED and names the event ID. |
| **limits.maxFlowStateBytes** | 67108864 (64 MiB) | Active state budget of one Flow run. Active state is the run's current rows and the values they hold: Attributes, Channel messages, Step executions with their inputs, locals, and attempt snapshots, timers, locks, request IDs kept for deduplication, and outbox rows. A Step that completed, or that a cancellation without a Flow close ended, keeps no Step execution row. A Failed Step and a Step that a Flow close canceled keep a small one. It also counts a small bookkeeping row, under 100 bytes, for each stored value, superseded ones included, in runs that started before 0.1.23. It does not count history records, checkpoints, or the superseded values that history keeps for time travel. server-zd checks it when a command increases the run's active state while that active state exceeds the budget. The checked commands are a Flow start, a Client Attribute write, a Client Channel publish, an RPC that changes Flow state, a Step heartbeat, and a Step completion. An RPC changes Flow state when it writes or deletes Attributes, publishes or deletes Channel messages, or starts or cancels Steps. The increase leaves out what a Step attempt holds only until its method ends, such as the Attributes and Channel messages it loaded and its locks. Such a Flow start, Client write, or RPC fails with RESOURCE_EXHAUSTED. Such a Step heartbeat or completion fails the attempt without retry. Client deletions, reads, and RPCs that change no Flow state still run. On a run over its budget, an RPC that changes no Flow state keeps no reply value or request ID for deduplication. A retry with the same request ID is then handled as a new RPC: it runs again while the Flow is running, and fails with NOT_FOUND once the Flow has closed, even if that RPC's own commit closed it. The history of a transactional RPC still records its output. A child Flow result that reaches its parent is not checked; **limits.maxChildFlowResultBytes** bounds it instead. Changed in 0.1.21: superseded values no longer count. A Flow that already failed on this limit stays failed; time travel can resume it. Changed in 0.1.23: the rows of completed Steps and of Steps canceled without a Flow close no longer count, so a long-lived run's active state stops growing with every Step it ran. Changed in 0.1.23: a run that starts with 0.1.23 or later no longer counts the bookkeeping rows of superseded values. Those rows wait for the history window or for an upload, like history. Before, a run that offloads history and rewrites values often could fill a small budget with them before its first history segment was sealed and uploaded. |
| **limits.maxChildFlowResultBytes** | 16777216 (16 MiB) | Largest total size of the result outputs that a child Flow delivers to its parent. A Step completion that would make a child Flow's collected outputs larger fails the attempt without retry. Unless the Step's failure policy recovers, the child Flow fails, and the parent receives that failure. A Flow without a parent has no such limit. |
| **limits.maxFlowIdBytes** | 1000 | Longest Flow ID, in UTF-8 bytes. |
| **limits.maxRequestIdsPerRun** | 1000 | Newest request IDs that one run keeps for deduplication. A retry with an older request ID runs as a new request. |
| **limits.maxLongPollWaitersPerFlow** | 1000 | Most concurrent waits on one Flow, such as WaitForStepCompletion and WaitForAttributeMatch calls. A wait beyond it fails with RESOURCE_EXHAUSTED. |
| **limits.maxLongPollWaitersPerNode** | 100000 | Most concurrent waits on the node. A wait beyond it fails with RESOURCE_EXHAUSTED. |
| **limits.maxInFlightRpcsPerNode** | 1000 | Most shared transactional or locking RPC tasks in flight on the node. A new request ID beyond it fails with RESOURCE_EXHAUSTED. |
Changed in 0.1.22: a GetHistoryEvents page holds at most 1000 events. A
request whose **estimate_page_size** is larger returns 1000 events and a next
page token. Earlier versions had no such limit.
Each run records **values.maxValueBytes**, **limits.maxFlowStateBytes**,
**limits.maxChildFlowResultBytes**, and **limits.maxRequestIdsPerRun** when it starts. A new value applies only to runs that start after the node
restarts with it.
## Performance tuning
The defaults suit most nodes. Change these keys only when a load test shows a
bottleneck. They change throughput and memory use, not Flow results. One
exception: **commit.maxUnappliedBytes** bounds the backlog of accepted commands,
so a smaller value makes more requests fail with UNAVAILABLE under load. Each
key applies when the node starts.
| Key | Default | Meaning |
| ---------------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **commit.applierLanes** | The larger of 8 and the CPUs available to the process, at most **cluster.partitionCount** and 1024 | Lanes that apply committed commands to the local store in parallel. One partition always uses the same lane, so its commands apply in order. Applying a command reads and writes the local store, so even a small host uses at least 8 lanes. An explicit value must be from 1 to the smaller of **cluster.partitionCount** and 1024. If the CPU count cannot be read, the default is 8. |
| **commit.applierQueueFrames** | 64 | Committed partition batches queued for each lane. When a lane's queue is full, the commit path waits for it. |
| **commit.batchWindow** | 20ms | How long an ASYNC command waits for more commands before its batch is sealed. A SYNC command seals its batch at once while the write pipeline has room. Both write the same data. The speculation limits apply to both, so Step proposals retry without failing the Step, while Client calls can get UNAVAILABLE. |
| **commit.maxBatchRecords** | 10000 | Most commands in one log batch. A batch is sealed when it reaches this count, **commit.sealAtBytes**, or **commit.batchWindow**. |
| **commit.maxUnappliedBytes** | 67108864 (64 MiB) | Encoded bytes of accepted commands that the node has not applied locally yet. A command that does not fit fails with UNAVAILABLE. Must be at least the smaller of **api.grpcMaxMessageBytes** and 32 MiB, so that the largest command can fit. |
| **localStorage.writeBufferBytes** | 67108864 (64 MiB) | Size of one RocksDB memtable for each column family. RocksDB keeps up to two memtables for each of the six column families, so memtables can use 12 times this value (768 MiB at the default) in addition to **localStorage.blockCacheBytes**. **localStorage.maxWalBytes** forces flushes when the write-ahead log reaches it, so raise it together with this value. Must be from 64 KiB to **localStorage.maxWalBytes**. |
| **localStorage.maxBackgroundJobs** | Half the CPUs available to the process, from 2 to 1024 | Most RocksDB flushes and compactions that run at once. RocksDB starts one thread for each job. An explicit value must be from 1 to 1024. |
| **localStorage.maxSubcompactions** | 1 | Threads that one RocksDB compaction may split into. A larger value can reduce write stalls under sustained writes. |
The node logs the resolved **applierLanes** and **maxBackgroundJobs** values at
startup.
## Streams
Set **streamStore.backend** to **memory** (default) or **disabled** to control
best-effort [Streams](https://docs.superdurable.io/concepts/primitives/stream.md). Memory Streams live only in the
active server process. They start empty after a restart, a standby takeover, or
a step down, and trimming can remove older messages. While the backend is
disabled, Client Stream writes and reads fail with **FAILED_PRECONDITION**. A
Step Stream write is dropped and logged without failing the Step, and the server
logs a warning at startup.
The memory backend bounds the memory it uses:
- **streamStore.maxTotalBytes** (default 1 GiB) caps the bytes held across every
Stream on the node. A Client write past it fails with **RESOURCE_EXHAUSTED**.
Background trimming then removes the oldest messages of the largest Streams
until usage falls to **trimTargetPercent** of the bound, or lower when the
rejected message needs more room. The client retries the write.
- A message larger than its Stream's capacity can never fit. It fails with
**RESOURCE_EXHAUSTED** without trimming, and a retry fails again.
- **streamStore.maxStreamCapacityBytes** (default 64 MiB) is the largest
capacity one Stream can use. A larger **streamCapacityBytes** is reduced to it.
- Each message counts its Value, Flow ID, source, and
**estimatedMessageOverheadBytes** (default 512).
- **streamStore.maxMessageBytes** (default 100 KiB) caps the size of one
Stream message.
- Background trimming starts when a Stream reaches **trimTriggerPercent**
(default 90) of its capacity, or all Streams reach that share of
**maxTotalBytes**. It trims down to **trimTargetPercent** (default 80).
- **streamStore.maxReadMessages** (default 1000, at most 1000) is the largest
**ListStreamMessages** page size. A larger page size fails with
**INVALID_ARGUMENT**.
- **backgroundTrimBatchSize**, **trimBatchYieldTime**, and **trimWorkers** pace
trimming.
A Step Stream write that the server drops never fails the Step.
**dexzd_step_stream_writes_dropped_total** counts these writes by **reason**:
**disabled**, **invalid_argument**, **message_too_large**,
**stream_capacity_exceeded**, **total_capacity_exceeded**, or **closed**.