Install and configure server-zd
View as MarkdownInstall and start
Each server-zd/vX.Y.Z GitHub Release publishes dex-server-zd archives for Linux amd64, Linux arm64, macOS amd64, and macOS arm64. The release also includes a checksums.txt file. Verify each archive against it before you extract it. Linux archives require glibc 2.28 or newer.
The dex-server-zd container image is published for linux/amd64 and linux/arm64. It runs serve with the configuration file /etc/dex-server-zd/config.yaml. The process runs as UID 10001 and GID 10001.
dex-server-zd serve --config /etc/dex-server-zd/config.yaml
serve reads every setting from one YAML file. The example configuration lists every key with its default. An unknown key fails startup. An invalid value also fails startup, and the error names the YAML path of the rejected field. The image includes the example at /etc/dex-server-zd/dex-server-zd.example.yaml for reference only.
Minimal production configuration
cluster:
nodeId: dex-zd
objectStore:
region: us-east-1
bucket: BUCKET
prefix: PREFIX
localStorage:
directory: /var/lib/dex-server-zd/store
lease:
duration: 60s
retention:
closedFlowDuration: 7days
metrics:
listen: 0.0.0.0:9090
cluster.nodeId, objectStore.bucket, and localStorage.directory are required. Replace BUCKET and PREFIX with your bucket and key prefix. Leave objectStore.endpoint empty to use AWS endpoint resolution. For an S3-compatible store, set it to an HTTP or HTTPS URL. Set objectStore.forcePathStyle when that store needs path-style addressing.
A node ID has 1 to 63 characters. It uses lowercase letters, digits, and hyphens, and it starts with a letter or a digit.
One object storage prefix holds one Dex cluster, and a cluster runs one node. The first start records cluster.nodeId and cluster.partitionCount. Neither can change later. A start with a different node ID fails and names the recorded one. A start with a different partition count fails and reports both counts. Never point two deployments at the same prefix.
Single node on local disk
dex-server-zd local runs one production node without external object storage. It starts an embedded s3local object store in the same process and keeps all data in one directory on local disk:
dex-server-zd local --data-dir /var/lib/dex-server-zd [--listen 127.0.0.1:8801] [--log-dir <dir>] [--config <overrides.yaml>] [--attribute-store-config <dex.yaml>] [--durability strict|process-crash-safe|relaxed]
The data directory holds the embedded object store in s3, the local store in store, and the lock file dev.lock. The embedded object store is the source of truth, as the bucket is for serve. Put the data directory on a persistent disk. --log-dir also writes the log to dex-server-zd.log in that directory.
The optional --config file holds overrides. It uses the keys of the example configuration. Put only the keys you want to change in it. Every key in the file replaces the single-node default. Do not copy the whole example. A copy sets api.listen to 0.0.0.0:8801, which has no authentication unless the file also configures authenticators. It also sets lease.duration to 10 seconds. local sets the object storage connection, the node ID dev, and the local store directory itself. --listen overrides api.listen from the file.
The optional --attribute-store-config flag reads only the attributeStore section of a YAML file. That section must define at least one store. The --config file must not also set attributeStore.
The node ID dev, the bucket dex-dev, the lock file dev.lock, and the dev section of the overrides keep their dev names on purpose. local shares them with dev, so a data directory can move between the two commands.
Partition count
Since server-zd v0.1.20, dev and local default cluster.partitionCount to 32. serve defaults it to 1024. One node owns every partition. The default number of applier lanes is the larger of 8 and the CPU count, at most the partition count, so 32 partitions lower it only on hosts with more than 32 CPUs. With the builtin search backend, fewer partitions also make each SearchFlows call scan less. Set cluster.partitionCount in the overrides to choose another power of two.
The first start records the partition count, and it cannot change later. A later start with a different count fails. The error names the data directory and both counts, and asks for the recorded count.
server-zd before v0.1.20 and dexcli before 1.37.0 created data directories with 1024 partitions. To keep using such a directory, add this to the overrides. With dexcli dev, pass the file with --zd-config.
cluster:
partitionCount: 1024
If you do not need its data, delete the data directory instead. The next start creates it with 32 partitions.
An explicit commit.applierLanes must not exceed the partition count. If the overrides set it above 32 without cluster.partitionCount, dev and local fail to start. The error asks you either to set cluster.partitionCount or to lower commit.applierLanes to at most 32. Use 1024 for a data directory created before v0.1.20.
To open a data directory created with 32 partitions using server-zd before v0.1.20 or dexcli before 1.37.0, set cluster.partitionCount to 32 in the overrides. Those releases default to 1024.
Durability
dev.durabilityMode selects how the embedded object store writes acknowledged data. The --durability flag overrides it with the same modes, spelled strict, process-crash-safe, and relaxed.
| Mode | Survives | Cost |
|---|---|---|
| strict (default) | A process kill, a host crash, and power loss. | Every write is flushed with fsync before the reply. |
| processCrashSafe | A process kill. A host crash or power loss can lose recent writes. | Faster. Every write reaches the files before the reply, without fsync. |
| relaxed | Only a graceful stop. | Writes stay in memory until a graceful stop. After an abrupt exit, every start fails until you delete the data directory. |
Keep the default strict mode in production.
One node
local runs exactly one node. It holds an exclusive lock on dev.lock for its whole run, so a second local or dev process on the same data directory fails to start. It has no standby. It rejects standby.mode: hot and objectStore.hardDeleteStore. While it holds the lock, no other process can write its object store. A restart after a crash therefore takes over the held lease at once. lease.duration defaults to 30 seconds. On a filesystem without file locks, the commands cannot exclude each other, and a restart waits for the previous lease to expire. Run only one process per data directory there.
On SIGTERM, local drains running Step attempts for up to lifecycle.attemptDrainTimeout, 30 seconds by default, and then stops as described in Graceful shutdown. Give a service manager at least 45 seconds to stop it.
Streams use the memory backend by default, as in serve. Stream messages are lost when the process stops or restarts.
local listens on 127.0.0.1:8801 by default, so only the same host can reach it. To serve clients and Workers on other hosts, pass --listen 0.0.0.0:8801 or set api.listen in the overrides. local has no authentication unless the overrides configure authenticators. When it listens on a non-loopback address without authenticators, it logs a warning at startup. Configure authentication before you expose it. Set metrics.listen in the overrides to enable metrics.
Disk use
dev and local default snapshot.logDeletionDelay to 1 hour. serve defaults it to 24 hours. An explicit value in the overrides always wins, and it must be greater than zero. A log object is deleted once a snapshot that covers it is this old. The age counts from the snapshot's creation time, before its upload finishes. A snapshot beyond snapshot.retainedCount is deleted once the newer snapshot that displaced it is this old. A longer delay keeps older snapshots and their log for fallback and lets a stopped node catch up from the log, at the cost of disk space. The delay must also exceed the longest snapshot upload. Changed in 0.1.21: before, dev and local also defaulted to 24 hours.
The embedded object store shares the disk with the local store. Plan for the size of store, plus the object store: the snapshots that snapshot.retainedCount keeps, and the log and displaced snapshots written during snapshot.logDeletionDelay. Together, the object store can reach two to three times the data written during the delay.
A Flow that rewrites a large value on every Step needs more disk. Its history records wait for a newer checkpoint before they can be deleted. For such Flows, lower history.checkpointCommandThreshold from its default of 1000. Better, store incremental values, one entry per item, instead of rewriting one large value.
Back up and restore
To back up, stop local with SIGTERM, wait for it to exit, and copy the whole data directory. A copy of a running data directory can be inconsistent. To restore, stop local, replace the data directory with the copy, and start local again. Writes acknowledged after the backup are lost.
The general rules in Backup and restore apply too. With a PostgreSQL or ParadeDB search backend, the restored data directory needs a new search.postgresSchema or search.paradeDbSchema, as that section explains.
Compared with dev and serve
| dev | local | serve | |
|---|---|---|---|
| Purpose | Development, started by dexcli dev. | Single-node production. | Production on external object storage. |
| Object storage | Embedded, in the data directory. | Embedded, in the data directory. | An external S3 or S3-compatible bucket. |
| Nodes | One. | One. | One active process, with optional hot standbys. |
| Step attempt drain on stop | 2 seconds. | 30 seconds. | 30 seconds. |
| Lease duration | 30 seconds. | 30 seconds. | 10 seconds. |
| Default snapshot.logDeletionDelay | 1 hour. | 1 hour. | 24 hours. |
| Default history.windowEvents | 5000 | 5000 | 50000 |
| Default partition count | 32 | 32 | 1024 |
| Default listen address | 127.0.0.1:8801 | 127.0.0.1:8801 | 0.0.0.0:8801 |
| Configuration | Flags and an optional overrides file. | Flags and an optional overrides file. | One YAML file. |
dev and local use the same data directory layout. Either command can serve a data directory that the other one wrote. Stop one before you start the other.
Retention
retention.closedFlowDuration sets how long a closed Flow run stays available. It defaults to 7 days. After that time, the run and its retained data become eligible for deletion. Write the value as a duration, such as 72h or 30days.
Each run records the value when it starts. A new value applies only to runs that start after the node restarts with it. Runs that already started keep their recorded duration.
From server-zd v0.1.8, a transactional or locking RPC on a closed run returns NOT_FOUND with FLOW_NOT_EXISTS before a Worker runs its handler. A retry with an already recorded request ID returns its earlier reply. Other RPCs still run their handlers on a closed run that is not yet purged, so they can read it. server-zd rejects any write that such a handler returns.
History window
server-zd keeps a recent window of each run's history. The window holds the FlowStart event and about the newest history.windowEvents history events: 50,000 on serve and 5,000 on dev and local by default. Older events are pruned in chunks of up to 1,000. Pruning never rejects a command or fails a Flow.
The window counts history events. A command that changes the Flow but records no event, such as a failed Step attempt that will be retried, counts as one event. Pruning stops only between commands, so a run keeps slightly more or fewer events than the window size. One command moves the window past at most about 2,000 history records, so a larger chunk takes several commands. Deleting the pruned records from local disk also proceeds a bounded amount per command. Since 0.1.22, a run that stops receiving commands, such as an idle or closed run, still finishes that deletion. About one minute after the run's last command, the lease holder commits internal cleanup commands for the run until the deletion is done. They record no history event.
A run can set its own window size with the history_window_events field of its Flow configuration. Set it at start with the Flow configuration override, or change it later with UpdateFlowConfig. A smaller window takes effect at the run's next pruning. A larger window keeps more of the events recorded after the change; pruned events never come back. Once set, the run's window can be changed but not cleared. A child Flow inherits its parent's window unless its own override sets one. Values must be positive and at most history.maxWindowEvents. StartFlow and UpdateFlowConfig reject other values with INVALID_ARGUMENT. A Step result that starts a SubFlow with such an override is an invalid Worker response: the Step attempt fails and is retried under the Step's retry policy.
History reads, such as dexcli flow history --all, return only the retained events. Time travel can target only a point inside the window. Time travel to the beginning is always available.
Each history page also reports the range of event IDs that the window removed. Step method events record the event ID of the event that started their Step execution. Execute events also record the event ID of the Step's WaitFor completion. A client can use these IDs to tell a removed related event from one on another page.
| Key | Default | Meaning |
|---|---|---|
| history.windowEvents | 50000 for serve; 5000 for dev and local | Approximate number of the newest history events kept in the window of a run that sets no window of its own. Pruning starts when the window exceeds this value plus one prune chunk. Must be positive. The deprecated name history.windowSemanticEvents is still accepted with a startup warning; setting both names is an error. Changed in 0.1.21: the window counted semantic commands, not events, and the key was named history.windowSemanticEvents. A run that started before 0.1.21 keeps its recorded size, which now counts events, so its window reaches a little less far back. Changed in 0.1.23: the default was 10000 in every command. Runs that started earlier keep the size they recorded. |
| history.pruneChunkSize | 1000 | Most old history events pruned at a time. A run prunes at most one tenth of its window at a time, and at least one event, when that is smaller. Must be positive and must not exceed windowEvents. Changed in 0.1.21: the unit was semantic commands. |
| history.maxWindowEvents | 100000 | Largest window that a run's history_window_events may set. StartFlow and UpdateFlowConfig with a larger value fail with INVALID_ARGUMENT. A Step result that starts a SubFlow with a larger value is an invalid Worker response, and the Step attempt is retried under the Step's retry policy. It does not limit windowEvents. Must be positive. Changing it affects later requests only. Since 0.1.21. |
| Each run records windowEvents and pruneChunkSize when it starts. A new | ||
| value applies only to runs that start after the node restarts with it. Changed | ||
| in 0.1.23: a child Flow that a SubFlow starts records them too; before, it | ||
| recorded 10000 and 1000. |
Changed in 0.1.23: history.maxRetainedSupersededBytes was removed. It capped the superseded values that a run without history offload kept on local disk for time travel. A run that starts with 0.1.23 or later moves its window only by its event count. Without offload, it deletes a superseded value when the window passes it. With offload, it deletes the value once its history is uploaded. The key is still accepted, has no effect, and logs a warning at startup. A run that started earlier keeps the cap it recorded.
Checkpoints
server-zd writes checkpoints of each run's current attempt state. Time travel restores the nearest checkpoint before its target and replays the commands after it. A command is one change that server-zd commits for a run, such as a Client write, a Step completion, or an internal acknowledgment. Each new attempt starts with a checkpoint. After that, server-zd writes a checkpoint when either threshold below is reached, whichever comes first. Both counts restart at each checkpoint.
| Key | Default | Meaning |
|---|---|---|
| history.checkpointCommandThreshold | 1000 | Commands since the last checkpoint that trigger a new one. Every command counts, including bookkeeping commands such as acknowledgments, not only commands that record history events. Time travel replays at most this many commands. |
| history.checkpointRowWriteBytesThreshold | 1048576 (1 MiB) | Bytes of attempt-scoped rows written since the last checkpoint that trigger a new one. Attempt-scoped rows are the current attempt's Attribute, Channel message, Channel metadata, Step execution, Step counter, lock, waiter, and timer rows. Every write of such a row adds its size, so rewriting one row counts each time. A large value stored apart from its row counts only as its reference. The effective threshold is the larger of this value and the smaller of two sizes: the run's current state rows and the rows of the last checkpoint. A larger state therefore makes checkpoints sparser. |
Lower values write checkpoints more often. Time travel then replays fewer commands, but each checkpoint copies the attempt's rows, so the run stores more checkpoint bytes and spends more work per command. Higher values store fewer checkpoints, and time travel replays more commands. Both values must be positive.
Each run records these values when it starts. A new value applies only to runs that start after the node restarts with it. Runs that already started keep their recorded values.
The lease holder computes each time travel. It restores the checkpoint, replays the commands after it in memory, and then commits the result.
| Key | Default | Meaning |
|---|---|---|
| history.timeTravelReplayMemoryBytes | 1073741824 (1 GiB) | Most bytes that one time travel holds in memory while it computes its result: the commands it replays from the checkpoint to its target, the restored state and its values, the external input it reapplies, and the staged result. When write reapply is enabled, memory grows with the number and size of writes after the target, even if the restored state is small. A ResetFlow that needs more fails with RESOURCE_EXHAUSTED and changes nothing. Choose a later target point, set skip_writes_reapply to true, or raise this value. Must be positive. A new value applies to ResetFlow requests after the node restarts with it. Since 0.1.22. |
Changed in 0.1.21: history.checkpointCommandThreshold was renamed from history.checkpointCommandInterval, and its default changed from 10000 to 1000. history.checkpointRowWriteBytesThreshold was renamed from history.checkpointMinWrittenBytes. The old keys still work and log a warning at startup. Setting one key under both names fails startup with a configuration error.
History offload
Each run decides when it starts whether it offloads its history to object storage. The offload_history field of the run's Flow configuration holds the choice, and it defaults to true. Set it in the Flow configuration override of StartFlow, or in the override of a SubFlow. A child Flow inherits its parent's choice unless its own override sets one. A run keeps its choice for its whole life: UpdateFlowConfig rejects a request that sets it with INVALID_ARGUMENT. Since 0.1.23.
An offloaded run seals its history into segments and uploads them to object storage. It then deletes the uploaded history from local disk. Local disk keeps the run's active state, the history that waits for upload, and a small metadata row for each point in the history window. The lease holder deletes a segment's object after the history window passes the whole segment.
A run without offload keeps all of its history on local disk and never writes a history object. The history window still applies to it. The run prunes its oldest events and deletes the superseded values that the window passed.
Choose for each run:
- Keep offload, the default, for most runs and for every long-running run. Local disk then holds only the newest history and the current state of the run.
- Turn it off for many short runs that close soon after they start. Each offloaded run writes at least one history object when it closes, and deletes it when its retention ends. Short runs at a high rate therefore add object storage requests, while their small history costs little local disk until their retention ends.
- A run without offload keeps every superseded value that its window covers on local disk. No byte cap applies. A run that rewrites large values can then use much local disk. Keep offload for such a run, or give it a smaller history_window_events.
Runs that started before 0.1.23 recorded no choice. Each follows, for its whole life, the offload mode that its partition recorded at the upgrade:
- Where offload was off at the upgrade, these runs keep their new history on local disk and never migrate to object storage. That is the case on serve unless history.offloadEnabled was on, and in a dev or local data directory that 0.1.23 starts directly after 0.1.21.
- A partition where an earlier release turned offload on and then off again is off as well: its runs seal no new segments and stop migrating. Segments sealed before can still finish uploading, and uploaded history stays readable.
- Where offload was on at the upgrade, as 0.1.22 sets it by default on dev and local, the runs it offloaded stay offloaded. A run that recorded history before offload was turned on migrates that history lazily, as described below.
- These runs also keep the superseded byte cap they recorded and follow their delete switch. UpdateFlowConfig can still change that switch.
Changed in 0.1.23: the server switches of history offload were removed.
- history.offloadEnabled was removed. true is still accepted, has no effect, and logs a warning at startup. false fails startup with a configuration error that asks you to remove the key. To keep a run's history on local disk, set the run's offload_history to false instead.
- history.deleteExpiredObjects was removed. It is still accepted, has no effect, and logs a warning at startup. A run that starts with 0.1.23 or later and offloads always deletes the objects of its expired segments.
- A run that starts with 0.1.23 or later ignores its FlowConfig.delete_expired_history_objects. StartFlow, SubFlow overrides, and UpdateFlowConfig still accept the field, and StartFlow drops it.
- Partitions no longer record an offload mode at startup, so dexzd_history_offload_mode_skipped_partitions_total was removed.
| Key | Default | Meaning |
|---|---|---|
| history.segmentTargetBytes | 4194304 (4 MiB) | Since 0.1.21. Bytes of history records, checkpoints, and superseded values after which an offloaded run seals its open history segment for upload. From 1 byte to 268435456 (256 MiB). |
| history.supersededValueGracePeriod | 10m | Since 0.1.22. How long a superseded value of an uploaded segment stays on local disk after the command that superseded it, measured in command time. Must be at least 1 microsecond. Each run records the value when it starts, so a new value applies only to runs that start after the node restarts with it. Runs that started before 0.1.22 keep 10 minutes. |
| history.uploadConcurrency | 256 | Since 0.1.21. Most sealed history segments the lease holder reads, encodes, and uploads at once. From 1 to 1024. Each upload in flight holds about one encoded segment in memory, and history.uploadMaxInFlightBytes bounds their total. The limit adapts between history.uploadMinConcurrency and this value; see Upload concurrency. Changed in 0.1.23: the default was 16, and the limit stayed fixed at this value. |
| history.uploadMinConcurrency | 16 | Since 0.1.23. Uploads the lease holder allows at once while no upload waits. From 1 to 1024. A value above history.uploadConcurrency counts as history.uploadConcurrency, so a configuration that sets only a smaller history.uploadConcurrency keeps a fixed limit. |
| history.uploadMaxInFlightBytes | 134217728 (128 MiB) | Since 0.1.23. Most bytes that the lease holder's uploads in flight hold at once for encoding history segments and keeping the encoded objects. From 1048576 (1 MiB) to 68719476736 (64 GiB). Each upload reserves the most memory its encoding can allocate before it encodes, and waits while the budget is used. A segment larger than the budget uploads alone. See Upload concurrency. |
| history.readCacheMaxBytes | 1073741824 (1 GiB) | Since 0.1.22. Most bytes of uploaded history that the node keeps in its disk read cache. 0 disables the cache. |
| history.readCacheDirectory | localStorage.directory with the suffix -history-read-cache, such as /var/lib/dex-server-zd/store-history-read-cache | Since 0.1.22. Directory of the disk read cache, which the cache owns. It must be outside localStorage.directory, which a local store rebuild sets aside, and must not contain .. components; startup fails otherwise. If the node cannot create it, the node logs one warning and reads from object storage without a cache. |
| history.objectReadConcurrency | 32 | Since 0.1.22. Most object storage reads of uploaded history segments that the node runs at once. From 1 to 1024. A history read beyond it waits for a free read. |
An offloaded run divides its history into segments. A segment is sealed when it reaches history.segmentTargetBytes, when 10,000 values were superseded inside it, or when the run closes. Each run records history.segmentTargetBytes when it starts, so a new value applies only to runs that start after the node restarts with it.
Once a segment is uploaded, the run deletes its history records and checkpoints from local disk. The run's first record and checkpoint stay. The small metadata row of each checkpoint, and of each history record that is a time travel point or holds events, stays until the window passes it. A superseded value is a stored value that the run no longer holds, such as the earlier value of a rewritten Attribute. A superseded value of an uploaded segment is deleted from local disk once history.supersededValueGracePeriod, 10 minutes by default, has passed since the command that superseded it, measured in command time. A value reference that a Worker or client just read therefore still loads from local disk. A Worker result, client write, or RPC result that hands back a reference to such a value also succeeds after that period. Before it records the result, the lease holder reads the value from its uploaded segment and restores it on local disk. A restored value that no state holds again can stay on local disk until the window passes it. A reference to a value that the window passed is rejected, as without offload. Each command deletes a bounded amount, and later commands of the run continue. Since 0.1.22, a run that receives no more commands, such as an idle or closed run, still finishes. The lease holder commits internal cleanup commands for the run, which record no history event. The first comes about one minute after the run's last command, or when the period of a waiting superseded value ends, whichever is later. While a run waits for such periods, it gets at most one cleanup command a minute. Values superseded in the open segment of an idle running run stay on local disk until that segment is sealed.
A run that recorded history without offload, on a partition whose offload an earlier release then turned on and that was still on at the upgrade, migrates its older history lazily. Only such runs migrate. Each command that records history and leaves the run running seals one more segment of that older history, oldest first. When a segment reaches the command itself, the run continues like a run that started with offload. Nothing else drives the migration. A run that receives no more commands keeps its older history on local disk, for as long as it stays idle. Events that the history window removed before the migration stay removed. A run that is already closed does not migrate: its local history stays until its retention ends and the run is deleted. A run that closes during its migration keeps the rest of its older history on local disk until it is deleted.
A dev or local data directory where an earlier release turned offload off and on again can hold history of commands that belong to no segment. That history stays on local disk and follows the window rule.
The lease holder uploads each sealed segment to the primary object store, under the history/ prefix of the bucket that holds the commit log. It uploads a run's segments oldest first and records each upload before the next one. A segment's content depends only on committed state. A new lease holder therefore writes the same object again and accepts an existing object with the same content. An existing object with different content is replaced; the node logs an error and increments dexzd_history_upload_checksum_mismatches_total.
Uploads never delay commands. While object storage fails them, they retry with backoff, sealed segments stay on local disk, and the upload backlog metrics grow; see Metrics. A permanent rejection, such as denied access, is retried once a minute instead. Retention postpones the deletion of a closed run until its sealed segments are uploaded. Deleting a run also deletes every object under its history/ prefix. That deletion starts a minute after the run's deletion begins, so that uploads an earlier lease holder abandoned have ended on its side. An upload that object storage accepted just before can still leave an object of the deleted run; no read returns it. Since 0.1.22, the lease holder lists history/ every six hours and deletes such objects once they are an hour old. Do not configure a bucket lifecycle rule that expires objects under history/. The lease holder deletes old history objects itself, as described next.
Upload concurrency
Changed in 0.1.23: the number of uploads in flight adapts to the backlog. Before, it stayed at history.uploadConcurrency, 16 by default. Every closed run seals and uploads at least one segment, so short runs need one object storage write each. With a write round trip of about 30 ms, 16 uploads at once drain about 530 segments a second.
The limit starts at history.uploadMinConcurrency. Each upload that object storage accepts while other uploads wait raises the limit by one, up to history.uploadConcurrency. The limit therefore about doubles each round trip while a backlog grows. Each upload that ends while no upload waits lowers the limit by one, back to history.uploadMinConcurrency.
A retryable object storage failure, such as an S3 503 SlowDown reply, halves the limit, but not below history.uploadMinConcurrency. Uploads that started before the last reduction do not reduce it again. After a reduction, the limit grows again only after as many successful uploads as the reduced limit. The failed upload retries with backoff, as before.
Memory stays bounded by history.uploadMaxInFlightBytes. After an upload reads a segment's rows, it computes the most memory encoding them can allocate: the block buffers, the compressed frames, the header, and the object. It reserves that much before it encodes, then keeps only the encoded object's size until the upload ends. An upload waits while the reservations use the budget. Reservations count whole KiB. Only a segment larger than the whole budget exceeds it, and that segment uploads alone. The budget does not count the rows an upload reads from local disk before it reserves, or the compressor's working state.
dexzd_history_upload_concurrency_limit shows the current limit and dexzd_history_upload_in_flight_bytes the reserved bytes. To keep the limit fixed, set history.uploadMinConcurrency and history.uploadConcurrency to the same value.
Since 0.1.22, a segment expires once the history window passes every command it holds. GetHistoryEvents, LoadBlobs, and ResetFlow never read an expired segment, from object storage or from the disk read cache, whether or not its object still exists. The lease holder deletes the object of each expired segment about one hour after it expired, so reads that started before the expiry can finish. Object storage use therefore stays bounded by the window, also for a run that never closes. Deleting a run deletes its remaining objects with the rest.
Changed in 0.1.23: a run that starts with 0.1.23 or later and offloads always deletes the objects of its expired segments. A run that started under 0.1.22 keeps following its delete switch. When that switch is off, the run's expired objects stay in object storage until its retention ends and the run is deleted. Until then, anyone who can read the bucket can read the old history they hold, including earlier values of rewritten Attributes. To delete them sooner, turn the run's switch on with UpdateFlowConfig. SDK 1.40.0 and later no longer send the field, so use an earlier SDK or a direct gRPC request. For sensitive values, use hard-delete Attributes.
GetHistoryEvents reads the events of uploaded segments from object storage and every other event from local disk. Each event comes from one place, so paging returns every event once, also while uploads continue. Events of a run without offload, and of commands that belong to no segment, stay on local disk and follow the window rule. A read never returns events that the window pruned, even when their segment object still exists. A read that object storage fails, or whose uploaded segment object is missing, fails with UNAVAILABLE; the client may retry it. A page that needs many segments can hold fewer events than requested; its next page token continues the read.
ResetFlow reads history the same way. The node computes the time travel from uploaded segments and local disk. It fetches the segments it needs while it holds no view of the local store. A ResetFlow whose segment read fails, or whose uploaded segment object is missing, fails with UNAVAILABLE and changes nothing; the client may retry it. The history it reads from object storage counts against history.timeTravelReplayMemoryBytes apart from the computation itself.
LoadBlobs reads a superseded value that offload deleted from local disk from its uploaded segment. Values of one request that share a segment take one object storage read. A value that the window passed is not returned, as before. A read that object storage fails, or whose uploaded segment object is missing, fails with UNAVAILABLE. A ResetFlow whose target state holds such a value reads it from object storage in the same way. Such values, whether read from the cache or from object storage, count against history.timeTravelReplayMemoryBytes.
Changed in 0.1.22: while history offload is on, a LoadBlobs request names at most 1000 values. A request with more entries fails with INVALID_ARGUMENT when the partition of a Flow it names offloads history. With offload off, LoadBlobs takes any number of entries, as before.
Changed in 0.1.23: the limit counts the entries whose run offloads history. A request with more than 1000 such entries fails with INVALID_ARGUMENT. Entries of runs without offload are not limited.
The node keeps uploaded segment headers in memory and keeps segment headers, records, and values in its disk read cache, up to history.readCacheMaxBytes. Uploaded segments never change, so cached entries are never stale. The cache survives a restart and is never authoritative: deleting its directory while the node is stopped only costs object storage reads. The node never sets the cache aside with a local store it rebuilds.
Limits
server-zd enforces these limits. A request that a limit rejects fails at once; it does not wait for capacity. The request ID limit never rejects a request; it evicts the oldest request IDs instead.
| Key | Default | Meaning |
|---|---|---|
| values.maxValueBytes | 16777216 (16 MiB) | Largest serialized application value, such as an Attribute value or a Channel message. A larger value is rejected. |
| api.grpcMaxMessageBytes | 33554432 (32 MiB) | Largest incoming or outgoing gRPC message. Must exceed values.maxValueBytes. Since 0.1.21, a GetHistoryEvents page ends before an event that would make the response larger, and its next page token continues the read. A single event larger than this limit fails the read with RESOURCE_EXHAUSTED and names the event ID. |
| limits.maxFlowStateBytes | 67108864 (64 MiB) | Active state budget of one Flow run. Active state is the run's current rows and the values they hold: Attributes, Channel messages, Step executions with their inputs, locals, and attempt snapshots, timers, locks, request IDs kept for deduplication, and outbox rows. A Step that completed, or that a cancellation without a Flow close ended, keeps no Step execution row. A Failed Step and a Step that a Flow close canceled keep a small one. It also counts a small bookkeeping row, under 100 bytes, for each stored value, superseded ones included, in runs that started before 0.1.23. It does not count history records, checkpoints, or the superseded values that history keeps for time travel. server-zd checks it when a command increases the run's active state while that active state exceeds the budget. The checked commands are a Flow start, a Client Attribute write, a Client Channel publish, an RPC that changes Flow state, a Step heartbeat, and a Step completion. An RPC changes Flow state when it writes or deletes Attributes, publishes or deletes Channel messages, or starts or cancels Steps. The increase leaves out what a Step attempt holds only until its method ends, such as the Attributes and Channel messages it loaded and its locks. Such a Flow start, Client write, or RPC fails with RESOURCE_EXHAUSTED. Such a Step heartbeat or completion fails the attempt without retry. Client deletions, reads, and RPCs that change no Flow state still run. On a run over its budget, an RPC that changes no Flow state keeps no reply value or request ID for deduplication. A retry with the same request ID is then handled as a new RPC: it runs again while the Flow is running, and fails with NOT_FOUND once the Flow has closed, even if that RPC's own commit closed it. The history of a transactional RPC still records its output. A child Flow result that reaches its parent is not checked; limits.maxChildFlowResultBytes bounds it instead. Changed in 0.1.21: superseded values no longer count. A Flow that already failed on this limit stays failed; time travel can resume it. Changed in 0.1.23: the rows of completed Steps and of Steps canceled without a Flow close no longer count, so a long-lived run's active state stops growing with every Step it ran. Changed in 0.1.23: a run that starts with 0.1.23 or later no longer counts the bookkeeping rows of superseded values. Those rows wait for the history window or for an upload, like history. Before, a run that offloads history and rewrites values often could fill a small budget with them before its first history segment was sealed and uploaded. |
| limits.maxChildFlowResultBytes | 16777216 (16 MiB) | Largest total size of the result outputs that a child Flow delivers to its parent. A Step completion that would make a child Flow's collected outputs larger fails the attempt without retry. Unless the Step's failure policy recovers, the child Flow fails, and the parent receives that failure. A Flow without a parent has no such limit. |
| limits.maxFlowIdBytes | 1000 | Longest Flow ID, in UTF-8 bytes. |
| limits.maxRequestIdsPerRun | 1000 | Newest request IDs that one run keeps for deduplication. A retry with an older request ID runs as a new request. |
| limits.maxLongPollWaitersPerFlow | 1000 | Most concurrent waits on one Flow, such as WaitForStepCompletion and WaitForAttributeMatch calls. A wait beyond it fails with RESOURCE_EXHAUSTED. |
| limits.maxLongPollWaitersPerNode | 100000 | Most concurrent waits on the node. A wait beyond it fails with RESOURCE_EXHAUSTED. |
| limits.maxInFlightRpcsPerNode | 1000 | Most shared transactional or locking RPC tasks in flight on the node. A new request ID beyond it fails with RESOURCE_EXHAUSTED. |
Changed in 0.1.22: a GetHistoryEvents page holds at most 1000 events. A request whose estimate_page_size is larger returns 1000 events and a next page token. Earlier versions had no such limit.
Each run records values.maxValueBytes, limits.maxFlowStateBytes, limits.maxChildFlowResultBytes, and limits.maxRequestIdsPerRun when it starts. A new value applies only to runs that start after the node restarts with it.
Performance tuning
The defaults suit most nodes. Change these keys only when a load test shows a bottleneck. They change throughput and memory use, not Flow results. One exception: commit.maxUnappliedBytes bounds the backlog of accepted commands, so a smaller value makes more requests fail with UNAVAILABLE under load. Each key applies when the node starts.
| Key | Default | Meaning |
|---|---|---|
| commit.applierLanes | The larger of 8 and the CPUs available to the process, at most cluster.partitionCount and 1024 | Lanes that apply committed commands to the local store in parallel. One partition always uses the same lane, so its commands apply in order. Applying a command reads and writes the local store, so even a small host uses at least 8 lanes. An explicit value must be from 1 to the smaller of cluster.partitionCount and 1024. If the CPU count cannot be read, the default is 8. |
| commit.applierQueueFrames | 64 | Committed partition batches queued for each lane. When a lane's queue is full, the commit path waits for it. |
| commit.batchWindow | 20ms | How long an ASYNC command waits for more commands before its batch is sealed. A SYNC command seals its batch at once while the write pipeline has room. Both write the same data. The speculation limits apply to both, so Step proposals retry without failing the Step, while Client calls can get UNAVAILABLE. |
| commit.maxBatchRecords | 10000 | Most commands in one log batch. A batch is sealed when it reaches this count, commit.sealAtBytes, or commit.batchWindow. |
| commit.maxUnappliedBytes | 67108864 (64 MiB) | Encoded bytes of accepted commands that the node has not applied locally yet. A command that does not fit fails with UNAVAILABLE. Must be at least the smaller of api.grpcMaxMessageBytes and 32 MiB, so that the largest command can fit. |
| localStorage.writeBufferBytes | 67108864 (64 MiB) | Size of one RocksDB memtable for each column family. RocksDB keeps up to two memtables for each of the six column families, so memtables can use 12 times this value (768 MiB at the default) in addition to localStorage.blockCacheBytes. localStorage.maxWalBytes forces flushes when the write-ahead log reaches it, so raise it together with this value. Must be from 64 KiB to localStorage.maxWalBytes. |
| localStorage.maxBackgroundJobs | Half the CPUs available to the process, from 2 to 1024 | Most RocksDB flushes and compactions that run at once. RocksDB starts one thread for each job. An explicit value must be from 1 to 1024. |
| localStorage.maxSubcompactions | 1 | Threads that one RocksDB compaction may split into. A larger value can reduce write stalls under sustained writes. |
The node logs the resolved applierLanes and maxBackgroundJobs values at startup.
Streams
Set streamStore.backend to memory (default) or disabled to control best-effort Streams. Memory Streams live only in the active server process. They start empty after a restart, a standby takeover, or a step down, and trimming can remove older messages. While the backend is disabled, Client Stream writes and reads fail with FAILED_PRECONDITION. A Step Stream write is dropped and logged without failing the Step, and the server logs a warning at startup.
The memory backend bounds the memory it uses:
- streamStore.maxTotalBytes (default 1 GiB) caps the bytes held across every Stream on the node. A Client write past it fails with RESOURCE_EXHAUSTED. Background trimming then removes the oldest messages of the largest Streams until usage falls to trimTargetPercent of the bound, or lower when the rejected message needs more room. The client retries the write.
- A message larger than its Stream's capacity can never fit. It fails with RESOURCE_EXHAUSTED without trimming, and a retry fails again.
- streamStore.maxStreamCapacityBytes (default 64 MiB) is the largest capacity one Stream can use. A larger streamCapacityBytes is reduced to it.
- Each message counts its Value, Flow ID, source, and estimatedMessageOverheadBytes (default 512).
- streamStore.maxMessageBytes (default 100 KiB) caps the size of one Stream message.
- Background trimming starts when a Stream reaches trimTriggerPercent (default 90) of its capacity, or all Streams reach that share of maxTotalBytes. It trims down to trimTargetPercent (default 80).
- streamStore.maxReadMessages (default 1000, at most 1000) is the largest ListStreamMessages page size. A larger page size fails with INVALID_ARGUMENT.
- backgroundTrimBatchSize, trimBatchYieldTime, and trimWorkers pace trimming.
A Step Stream write that the server drops never fails the Step. dexzd_step_stream_writes_dropped_total counts these writes by reason: disabled, invalid_argument, message_too_large, stream_capacity_exceeded, total_capacity_exceeded, or closed.