server-zd operation

server-zd is a Dex Server whose only runtime dependency is object storage, such as Amazon S3 or an S3-compatible store. It keeps Flow state in a local store and commits every write to object storage before it acknowledges the write. The local disk is only a cache. A node rebuilds it from the latest snapshot and the command log in object storage.

server-zd serves the FlowService and WorkerTaskService API. It supports only pull Workers. Run Dex Web as a separate process; server-zd does not serve it. For local development, run dexcli dev, which starts server-zd and Dex Web. This page covers server-zd 0.1.

Unsupported features

server-zd rejects these options with INVALID_ARGUMENT:

• A Flow start delay, which is flow_start_delay_seconds greater than zero. Tracked in issue 995.
• A Flow-level retry policy, which is retry_policy in the Flow start options. Tracked in issue 996. Step-level retry is supported.

The only Attribute Store is PostgreSQL. Redis is tracked in issue 616.

Install and start

Each server-zd/vX.Y.Z GitHub Release publishes dex-server-zd archives for Linux amd64, Linux arm64, macOS amd64, and macOS arm64. The release also includes a checksums.txt file. Verify each archive against it before you extract it. Linux archives require glibc 2.28 or newer.

The dex-server-zd container image is published for linux/amd64 and linux/arm64. It runs serve with the configuration file /etc/dex-server-zd/config.yaml. The process runs as UID 10001 and GID 10001.

dex-server-zd serve --config /etc/dex-server-zd/config.yaml

serve reads every setting from one YAML file. The example configuration lists every key with its default. An unknown key fails startup. An invalid value also fails startup, and the error names the YAML path of the rejected field. The image includes the example at /etc/dex-server-zd/dex-server-zd.example.yaml for reference only.

Minimal production configuration

cluster:
  nodeId: dex-zd
objectStore:
  region: us-east-1
  bucket: BUCKET
  prefix: PREFIX
localStorage:
  directory: /var/lib/dex-server-zd/store
lease:
  duration: 60s
retention:
  closedFlowDuration: 7days
metrics:
  listen: 0.0.0.0:9090

cluster.nodeId, objectStore.bucket, and localStorage.directory are required. Replace BUCKET and PREFIX with your bucket and key prefix. Leave objectStore.endpoint empty to use AWS endpoint resolution. For an S3-compatible store, set it to an HTTP or HTTPS URL. Set objectStore.forcePathStyle when that store needs path-style addressing.

A node ID has 1 to 63 characters. It uses lowercase letters, digits, and hyphens, and it starts with a letter or a digit.

One object storage prefix holds one Dex cluster, and a cluster runs one node. The first start records cluster.nodeId and cluster.partitionCount. Neither can change later. A start with a different node ID fails and names the recorded one. Never point two deployments at the same prefix.

Kubernetes deployment

One replica in a StatefulSet

Run server-zd as a StatefulSet with exactly one replica. A StatefulSet creates the replacement pod only after the old pod is gone. A Deployment can start a new pod while the old one still runs.

Never force-delete the server pod. A force delete removes the pod from the API before Kubernetes confirms that its process stopped. On an unreachable node, the old process can keep running next to its replacement.

This example uses a PersistentVolumeClaim for the local store and a ConfigMap named dex-server-zd that holds config.yaml. The image is the published multi-architecture release image for amd64 and arm64.

apiVersion: v1
kind: Service
metadata:
  name: dex-server-zd
spec:
  selector:
    app: dex-server-zd
  ports:
    - name: grpc
      port: 8801
      targetPort: grpc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dex-server-zd
spec:
  serviceName: dex-server-zd
  replicas: 1
  selector:
    matchLabels:
      app: dex-server-zd
  template:
    metadata:
      labels:
        app: dex-server-zd
    spec:
      serviceAccountName: dex-server-zd
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      containers:
        - name: dex-server-zd
          image: docker.io/superdurable/dex-server-zd:0.1.3
          ports:
            - name: grpc
              containerPort: 8801
            - name: metrics
              containerPort: 9090
          startupProbe:
            grpc:
              port: 8801
            periodSeconds: 5
            failureThreshold: 60
          readinessProbe:
            grpc:
              port: 8801
            periodSeconds: 5
          livenessProbe:
            grpc:
              port: 8801
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /etc/dex-server-zd/config.yaml
              subPath: config.yaml
              readOnly: true
            - name: data
              mountPath: /var/lib/dex-server-zd
      volumes:
        - name: config
          configMap:
            name: dex-server-zd
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

Graceful shutdown

Set terminationGracePeriodSeconds to at least 30 seconds. On SIGTERM, the node stops within about 14 seconds, in this order:

1. gRPC health reports NOT_SERVING.
2. Workers are told to reconnect, and long polls close.
3. Open calls get up to 4 seconds to finish.
4. Background work and the engine stop.
5. Lease renewal stops, and the node releases its lease with a conditional write.
6. The metrics listener and the Stream Store close.
7. The process exits.

The process exits 0 when every step completes. A replacement process with the same cluster.nodeId then acquires the lease at once. The process exits 1 when some shutdown step does not complete in time or fails, for example when object storage rejects the lease release. A replacement may then wait up to lease.duration plus about 1 second.

A SIGTERM during startup interrupts lease acquisition or recovery. If the node holds its lease, it stops startup and releases the lease within the shutdown budget. A failed release makes the process exit 1, and the next start may wait for the old lease to expire. If the outcome of a lease write is still unknown when the SIGTERM arrives, the process exits 0 without releasing that write, and the next start may wait for it to expire.

Local store volume

The local store is a cache, because every acknowledged write is already in object storage. The volume type only changes restart time.

With an emptyDir volume, a new pod restores the newest snapshot and replays the log written after it. An emptyDir survives a container restart in the same pod, but not a new pod. With a PersistentVolumeClaim, a new pod reuses the local store and replays only the log written since it stopped. Either way, the first snapshot after a restart uploads the full local checkpoint to object storage.

Mount the volume at a parent directory, such as /var/lib/dex-server-zd. Set localStorage.directory to a subdirectory, such as /var/lib/dex-server-zd/store. When the node cannot reuse an existing local store, it renames the directory to a sibling with a .discarded- suffix and rebuilds it. This happens when RocksDB reports the store corrupt, missing files, or in an incompatible format while opening it, or when the store cannot be updated from the retained log. That rename fails when the directory is the mount point itself.

Other errors that stop the node from opening the store fail startup and leave the directory in place. Examples are another process that holds the store open and a permission error. Stop the other process or fix the permissions, then restart. If the store itself cannot be read, for example after a disk error, stop the node, move or delete the store directory, and restart. The node then rebuilds the store from object storage.

A store that is newer than the metadata in object storage is also set aside, and the node logs an error. That copy may be the only local evidence that the metadata went backwards.

After each successful start, the node deletes older set-aside directories. It keeps the newest localStorage.retainedSetAsideStoreCount of them for diagnosis, 1 by default. A copy that the current start set aside always counts as the newest. Set the count to 0 to keep none. The node then deletes every copy at once, including a copy of a store that is newer than the metadata. The node logs each path it deletes. A failed deletion logs a warning and does not stop startup. The node deletes only directories that it set aside for this store. It never sets aside or deletes the active store while another process holds its lock. The node never opens a set-aside copy again. Cleanup does not check whether another tool has a set-aside copy open.

Releases up to server-zd v0.1.2 never delete set-aside directories. The first successful start after an upgrade deletes all but the newest. To keep older copies, raise the count or move them out of the parent directory before you upgrade. Those releases also set the store aside after any open error; later releases do so only in the cases above.

Size the volume for the live state, plus localStorage.maxWalBytes of write-ahead log (256 MiB by default), plus room for uncompacted files after a bulk replay. The node compacts the store after such a replay, which returns the directory to about the live state size.

Also leave room for the retained set-aside copies. Each copy can be as large as the store was when the node set it aside. After a successful start, the volume holds at most localStorage.retainedSetAsideStoreCount copies. A start that sets the store aside holds one extra copy until it succeeds. Starts that fail after setting the store aside, such as a crash loop, add a copy each time until a start succeeds. With the default of 1, plan for at least two old copies next to the rebuilt store.

To use an emptyDir instead of the claim template, replace volumeClaimTemplates with a volume like this:

      volumes:
        - name: data
          emptyDir:
            sizeLimit: 20Gi

Health checks

Port 8801 serves the standard gRPC health service on api.listen. Health reports NOT_SERVING during startup, while the node waits for its lease and replays the log. It reports SERVING after recovery completes.

Use gRPC health for all three probes:

• The startupProbe holds off the liveness probe until the first SERVING. Size it so that failureThreshold × periodSeconds is at least lease.duration + 1 s + the worst-case recovery time. The lease part covers a previous owner that did not release its lease. Recovery time grows with live state and the log tail. A cold restore of an empty store took about 4 s; plan for more with real data. The example allows 5 minutes.
• The readinessProbe keeps traffic away while the node is not serving.
• The livenessProbe restarts a node that stops answering. Do not use a TCP probe for liveness: a stalled process can still accept TCP connections in the kernel backlog while it serves nothing.

A serving node never returns to NOT_SERVING without exiting, because a lost lease ends the process. Liveness on gRPC health therefore does not cause restart loops.

This probe set is for a single replica, the supported topology. A standby replica stays NOT_SERVING until it takes over, so it would never pass this startup probe. A standby needs a TCP or exec startup probe and liveness probe instead.

The process exits on its own after a fatal error, such as a lost lease or a failed commit. During startup, transient or unknown-outcome object storage read errors are retried with jittered exponential backoff. recovery.startupReadRetryBudget sets a wall-clock window for each read, starting at its first transient or unknown-outcome failure. The default is 30 seconds and must be positive. Backoff and subsequent read calls count toward that read's window. Concurrent reads do not share time, and a later read gets a new window. The error names the operation, object key, and last error when the window expires. Missing required objects, corrupt data, access denial, and lost lease ownership fail startup without read retries. Lease validity or shutdown may end retries sooner. Lease renewal continues during recovery retries. The window bounds each read, not the whole startup. After the lease is acquired, startup has no overall time limit. A persistent outage still fails startup within one read's window, and lease renewal keeps the lease while slow reads make progress. The startup probe bounds the total startup time. If object storage is unreachable at startup, the conditional-write self-check also exits 1. If startup fails, the process exits 1. Kubernetes restarts the container with backoff.

Lease

The node holds a lease object in object storage. lease.duration defaults to 10 seconds. The node renews the lease after one third of the duration.

dex-server-zd dev defaults lease.duration to 30 seconds, unless its override file sets it. A dev node has no standby, so the longer lease costs no failover time. It lets the node ride out longer host stalls. A dev node holds an exclusive lock on dev.lock in its data directory, so a second dev process on that directory fails to start. While it holds the lock, a held lease belongs to a stopped process. The dev node then takes the lease over at once with a raised epoch, so a restart does not wait, even after a crash. On a filesystem that cannot lock files, the dev node logs a warning and waits for the lease to expire, up to 31 seconds. The default dev.durabilityMode, processCrashSafe, keeps every acknowledged write across such a crash or kill, but not across a host crash or power loss. strict also survives a host crash. After an abrupt exit in relaxed mode, the dev node refuses to start until you delete its data directory. See local development.

Data safety does not depend on the duration. Each lease acquisition raises the node epoch, and every log write is conditional. Together they fence a process that lost its lease.

The duration affects takeover time after a crash. A graceful shutdown releases the lease, so the replacement starts at once. After a crash, a SIGKILL, or the loss of a Kubernetes node, the replacement waits until the old lease expires, plus 1 second. That wait is at most lease.duration plus 1 second. Recovery starts after it.

The duration only partly affects object storage outages:

• A renewal attempt that fails with a transient error, such as a 5xx or throttling response, is retried after a backoff.
• While another attempt still fits, each attempt is cut off after one sixth of the duration, between 1 and 5 seconds. The last attempt may use all the remaining time, so with a long lease it can run longer than 5 seconds.
• Retries continue only while the next backoff ends before the renewal cutoff, 100 ms before the current lease period ends. When no retry fits, or the last attempt fails, the node stops.
• A renewal write that lands after its attempt was cut off still counts if a later attempt finds it. A later attempt finds it by reading the lease after its own write fails. The node adopts that write only before the renewal cutoff.
• A renewal lost to another holder, or a permanent storage error, stops the node at once.
• A log write that object storage does not accept within 10 seconds also stops the node. This limit is fixed and does not depend on the lease duration.

A longer lease therefore lets an idle node ride out a stalled object store or a stalled host for longer. After any of these stops, the process exits 1.

For a single-replica deployment, set lease.duration to 60 seconds. Lease renewals then make about 180 PUT requests per hour, compared with about 1,080 at 10 seconds. Keep the duration at or below about 120 seconds. A longer lease saves few requests, but the unavailable time after a crash keeps growing.

By default, a starting node waits while another process holds the lease. Set lease.failIfHeld to true to fail startup instead.

Object storage

Credentials

objectStore.credentialsSource selects the credentials:

Value	Credentials
defaultChain (default)	The AWS SDK default credential provider chain
environment	DEX_ZD_OBJECT_STORE_ACCESS_KEY_ID, DEX_ZD_OBJECT_STORE_SECRET_ACCESS_KEY, and the optional DEX_ZD_OBJECT_STORE_SESSION_TOKEN
anonymous	Unsigned requests, for a local emulator or a bucket that allows anonymous access

On Amazon EKS, keep defaultChain and give the pod's service account an IAM role, for example through IAM Roles for Service Accounts or EKS Pod Identity.

Prefix-confined IAM policy

For a shared bucket, set a non-empty objectStore.prefix. Then grant the server role only these actions. Replace BUCKET and PREFIX/ with the bucket and the configured prefix, and keep the trailing slash.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET",
      "Condition": { "StringLike": { "s3:prefix": "PREFIX/*" } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::BUCKET/PREFIX/*"
    }
  ]
}

The server lists only under its prefix, including when it checks for a missing object. With this grant, AWS S3 returns 404 for a missing object. Narrower grants can return 403. The server handles either response when the prefixed listing is allowed.

The s3:ListBucket condition must use StringLike with the trailing wildcard shown above. An exact-prefix StringEquals grant is not supported. Neither is a policy without s3:ListBucket. With a non-empty prefix, startup lists under it first. A denied listing fails startup with a message that starts with this text:

startup requires s3:ListBucket with StringLike s3:prefix "PREFIX/*"

Startup also writes and deletes a probe object under the prefix. It uses this to check that conditional writes work. No bucket-wide access or bucket administration action is needed.

Hard-delete Attribute values use the primary bucket and prefix by default. To keep them in a separate place, configure objectStore.hardDeleteStore with its own endpoint, region, bucket, prefix, and credentialsSource. Its environment credentials use the DEX_ZD_HARD_DELETE_STORE_ variables with the same suffixes. Grant the same policy for that bucket and prefix.

Encryption

server-zd sends no server-side encryption headers, so the bucket's default encryption applies. Default SSE-KMS works. Conditional writes use If-None-Match and If-Match, and the server treats ETags as opaque values. It does not rely on ETags being MD5 digests.

For a customer-managed KMS key, grant the role kms:GenerateDataKey and kms:Decrypt on that key. Allow both actions in the key policy too.

Versioning and lifecycle rules

An unversioned bucket works. server-zd does not use object versions.

If bucket versioning is on, every object the server deletes stays as a noncurrent version. This includes old log objects and snapshots, so storage only grows. Add a lifecycle rule that expires noncurrent versions. A noncurrent version of a hard-delete Attribute value also keeps the deleted value until the version expires.

Add a rule that aborts incomplete multipart uploads after 1 to 7 days, whether or not versioning is on. A crash can leave an incomplete upload behind. A prefix-confined role cannot list or clean up those uploads.

This lifecycle configuration covers both rules. Drop the second rule for an unversioned bucket.

{
  "Rules": [
    {
      "ID": "dex-server-zd-abort-incomplete-uploads",
      "Status": "Enabled",
      "Filter": { "Prefix": "PREFIX/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 2 }
    },
    {
      "ID": "dex-server-zd-expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": { "Prefix": "PREFIX/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 7 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

Retention

retention.closedFlowDuration sets how long a closed Flow run stays available. It defaults to 7 days. After that time, the run and its retained data become eligible for deletion. Write the value as a duration, such as 72h or 30days.

Each run records the value when it starts. A new value applies only to runs that start after the node restarts with it. Runs that already started keep their recorded duration.

From server-zd v0.1.8, a transactional or locking RPC on a closed run returns NOT_FOUND with FLOW_NOT_EXISTS before a Worker runs its handler. A retry with an already recorded request ID returns its earlier reply. Other RPCs still run their handlers on a closed run that is not yet purged, so they can read it. server-zd rejects any write that such a handler returns.

History window

server-zd keeps a recent window of each run's history. The window holds the FlowStart event and about the newest 10,000 semantic events. Older events are pruned in chunks of 1,000. The window also shrinks when the superseded values kept for time travel exceed 256 MiB.

History reads, such as dexcli flow history --all, return only the retained events. Time travel can target only a point inside the window. Time travel to the beginning is always available.

Each history page also reports the range of event IDs that the window removed. Step method events record the event ID of the event that started their Step execution. Execute events also record the event ID of the Step's WaitFor completion. A client can use these IDs to tell a removed related event from one on another page.

Key	Default	Meaning
history.windowSemanticEvents	10000	Approximate number of the newest semantic events kept in the window. Pruning starts when the window exceeds this value plus one prune chunk. A semantic event can appear as several CLI history rows.
history.pruneChunkSize	1000	Old semantic events pruned at a time. Must not exceed windowSemanticEvents.
history.maxRetainedSupersededBytes	268435456 (256 MiB)	Most superseded value bytes kept for time travel inside the window.

Each run records these values when it starts. A new value applies only to runs that start after the node restarts with it.

Limits

server-zd enforces these limits. A request that a limit rejects fails at once; it does not wait for capacity. The request ID limit never rejects a request; it evicts the oldest request IDs instead.

Key	Default	Meaning
values.maxValueBytes	16777216 (16 MiB)	Largest serialized application value, such as an Attribute value or a Channel message. A larger value is rejected.
api.grpcMaxMessageBytes	33554432 (32 MiB)	Largest incoming or outgoing gRPC message. Must exceed values.maxValueBytes.
limits.maxFlowStateBytes	67108864 (64 MiB)	Storage budget of one Flow run. It counts current state, Attribute and Channel values, and the superseded values kept for time travel. server-zd checks it only for a Flow start, a Client Attribute write, a Client Channel publish, a Step heartbeat, and a Step completion, and only when that command increases the run's current storage while its total storage exceeds the budget. Such a Flow start or Client write fails with RESOURCE_EXHAUSTED. Such a Step heartbeat or completion fails the attempt without retry. Other writes, such as transactional RPC writes, are not checked, so the budget does not bound all run state.
limits.maxFlowIdBytes	1000	Longest Flow ID, in UTF-8 bytes.
limits.maxRequestIdsPerRun	1000	Newest request IDs that one run keeps for deduplication. A retry with an older request ID runs as a new request.
limits.maxLongPollWaitersPerFlow	1000	Most concurrent waits on one Flow, such as WaitForStepCompletion and WaitForAttributeMatch calls. A wait beyond it fails with RESOURCE_EXHAUSTED.
limits.maxLongPollWaitersPerNode	100000	Most concurrent waits on the node. A wait beyond it fails with RESOURCE_EXHAUSTED.
limits.maxInFlightRpcsPerNode	1000	Most shared transactional or locking RPC tasks in flight on the node. A new request ID beyond it fails with RESOURCE_EXHAUSTED.

Each run records values.maxValueBytes, limits.maxFlowStateBytes, and limits.maxRequestIdsPerRun when it starts. A new value applies only to runs that start after the node restarts with it.

Performance tuning

The defaults suit most nodes. Change these keys only when a load test shows a bottleneck. They change throughput and memory use, not Flow results. One exception: commit.maxUnappliedBytes bounds the backlog of accepted commands, so a smaller value makes more requests fail with UNAVAILABLE under load. Each key applies when the node starts.

Key	Default	Meaning
commit.applierLanes	The larger of 8 and the CPUs available to the process, at most cluster.partitionCount and 1024	Lanes that apply committed commands to the local store in parallel. One partition always uses the same lane, so its commands apply in order. Applying a command reads and writes the local store, so even a small host uses at least 8 lanes. An explicit value must be from 1 to the smaller of cluster.partitionCount and 1024. If the CPU count cannot be read, the default is 8.
commit.applierQueueFrames	64	Committed partition batches queued for each lane. When a lane's queue is full, the commit path waits for it.
commit.maxBatchRecords	10000	Most commands in one log batch. A batch is sealed when it reaches this count, commit.sealAtBytes, or commit.batchWindow.
commit.maxUnappliedBytes	67108864 (64 MiB)	Encoded bytes of accepted commands that the node has not applied locally yet. A command that does not fit fails with UNAVAILABLE. Must be at least the smaller of api.grpcMaxMessageBytes and 32 MiB, so that the largest command can fit.
localStorage.writeBufferBytes	67108864 (64 MiB)	Size of one RocksDB memtable for each column family. RocksDB keeps up to two memtables for each of the six column families, so memtables can use 12 times this value (768 MiB at the default) in addition to localStorage.blockCacheBytes. localStorage.maxWalBytes forces flushes when the write-ahead log reaches it, so raise it together with this value. Must be from 64 KiB to localStorage.maxWalBytes.
localStorage.maxBackgroundJobs	Half the CPUs available to the process, from 2 to 1024	Most RocksDB flushes and compactions that run at once. RocksDB starts one thread for each job. An explicit value must be from 1 to 1024.
localStorage.maxSubcompactions	1	Threads that one RocksDB compaction may split into. A larger value can reduce write stalls under sustained writes.

The node logs the resolved applierLanes and maxBackgroundJobs values at startup.

Attribute Store sync

server-zd v0.1.3 or later can sync Attributes to PostgreSQL. See Custom Attribute Store for the table contract. A Flow marks an Attribute with syncToAttributeStore and lists a configured store name in its FlowConfig. server-zd then writes the Attribute to that store's tables.

Behavior

• Only type: postgres is supported. Any other store type fails validation, so startup fails.
• Step, RPC, and SetAttributes commits never wait for the table write.
• Rows normally appear within a second of the commit. Treat the table as eventually consistent.
• While a store is unavailable, Flows keep running. When the store recovers, server-zd writes the backlog with the latest values.
• Entries still pending syncRetryPolicy.totalDuration after they were first marked are dropped. Later writes of the same Attributes sync normally.
• After time travel, server-zd rewrites every row the run synced with the state after the time travel.
• Each database needs a fencing table.
• server-zd v0.1.3 closes a Flow without waiting for its synced Attributes. Rows can land after the Flow completes.
• server-zd v0.1.4 or later waits. A Step decision that completes or fails the Flow, exhausted Step retries, or the Flow timeout stops the Steps at once. The run then stays RUNNING until every pending write is written or dropped. WaitForFlow returns only then. Entries written before the wait ended are in the table. A dropped entry does not guarantee the row's value, because earlier writes of the same run may have committed. If dexzd_attribute_sync_dropped_entries_total increases, compare the affected rows with the Flow's Attributes and write the Attributes again.
• While a close waits, SetAttributes, PublishToChannel, and RPCs that are neither transactional nor locking are accepted. Their synced writes extend the wait, and they start no Step. Transactional or locking RPCs, WaitForStepCompletion, WaitForAttribute, and DeleteChannelMessage calls made during the wait fail with FAILED_PRECONDITION.
• In server-zd v0.1.4 or later, a natural close also waits for every transactional or locking RPC admitted before it, even without an Attribute Store. Such an RPC still commits, and its synced writes join the wait. The close is recorded once those RPCs commit, fail, or time out.
• syncRetryPolicy.totalDuration limits how long each pending write is retried, not the whole close. During a PostgreSQL outage a close waits until its pending writes are dropped. Every synced write accepted while the close waits starts its own retry window and extends the wait.
• StopFlow and Terminate never wait. StopFlow closes the run as soon as it commits, even while a close waits, and the pending writes keep syncing after the close. A later SetAttributes on that run returns NOT_FOUND. From server-zd v0.1.5, it succeeds only when every write erases a currently stored hard-delete Attribute with null. This exception applies only before the run is purged. It deletes the stored objects without reopening the run. The run's Flow history records the erase after the close. Write what a Flow needs before stopping it.
• Pending writes for a store that is no longer configured are dropped at once and counted in dexzd_attribute_sync_dropped_entries_total.

Configuration

attributeStore:
  stores:
    orders:
      type: postgres
      dsn: ""
      tableName: public.orders
      rowKey: order-{order_id}
      columnNaming: snake_case
      attributeMaps:
        Items:
          tableName: public.order_items
          instanceKeyColumn: item_key
      fencingTableName: dexzd_attribute_sync_fencing
  schemaSyncInterval: 1m
  syncBatchSize: 100
  syncAttemptTimeout: 30s
  syncRetryPolicy:
    initialInterval: 100ms
    backoffCoefficient: 2.0
    maximumInterval: 30s
    maximumAttempts: 0
    totalDuration: 1h

An explicit zero for a duration, syncBatchSize, or a retry policy field means its default. The example configuration shows the section with comments.

Key	Default	Notes
stores	Empty	Maps each store name that a FlowConfig may list to a PostgreSQL destination. Empty disables sync and rejects every store name that a FlowConfig lists.
stores.NAME.type	None	Must be postgres.
stores.NAME.dsn	Empty	PostgreSQL connection string. When it is empty, server-zd reads an environment variable instead. Logs redact it.
stores.NAME.tableName	None	Main table, as table or schema.table.
stores.NAME.rowKey, columnNaming, and attributeMaps	Empty	See Custom Attribute Store.
stores.NAME.fencingTableName	dexzd_attribute_sync_fencing	Exists only in server-zd. See Fencing table.
schemaSyncInterval	1m	Time between table schema refreshes, with 90–110% jitter. A write to an unknown column refreshes the schema at once.
syncBatchSize	100	Most Attributes and AttributeMap instances of one run that one transaction writes. A run with more pending values uses several transactions.
syncAttemptTimeout	30s	Upper bound of one write transaction. See Write timeout.
syncRetryPolicy.initialInterval	100ms	Delay after the first failed write.
syncRetryPolicy.backoffCoefficient	2	Multiplies the delay after each failed write. Must be positive and finite.
syncRetryPolicy.maximumInterval	30s	Longest delay between attempts. Must not be less than initialInterval.
syncRetryPolicy.maximumAttempts	0, unlimited	Consecutive failed writes of one run's entries, after which the entries are dropped. The count lives in memory, so it starts again after a restart.
syncRetryPolicy.totalDuration	1h	Entries still pending this long after they were first marked are dropped.

Leave dsn empty to keep the connection string out of the configuration file. server-zd then reads DEX_ZD_ATTRIBUTE_STORE_DSN_ followed by the store name in upper case. Every character other than an ASCII letter or digit becomes an underscore. For the store orders, the variable is DEX_ZD_ATTRIBUTE_STORE_DSN_ORDERS. A store without a DSN in either place fails validation.

Fencing table

Every write transaction raises a fencing token row to the node's epoch. A node that lost its lease therefore cannot overwrite newer values.

Each database needs this table, named by fencingTableName. One table serves every store and every cluster in the database. server-zd never creates or alters tables. Create the table as the table owner before startup. Then grant SELECT, INSERT, and UPDATE on it to the role that server-zd connects as. Replace dex with that role.

CREATE TABLE dexzd_attribute_sync_fencing (
  store_name text NOT NULL,
  cluster_id text NOT NULL,
  scope text NOT NULL,
  fencing_token bigint NOT NULL,
  updated_at timestamptz NOT NULL,
  PRIMARY KEY (store_name, cluster_id, scope)
);
GRANT SELECT, INSERT, UPDATE ON dexzd_attribute_sync_fencing TO dex;

You may delete the rows of a permanently stopped cluster. Never delete the rows of a running cluster.

Startup connects to every store and checks its tables. Any failure fails startup, including an unreachable store or a missing fencing table. For a missing fencing table, the error includes the CREATE TABLE statement. Startup does not check grants. Without them, the fencing table looks missing, or every write fails with a permission error.

Write timeout

A write transaction lasts at most syncAttemptTimeout or a quarter of lease.duration, whichever is smaller. A quarter of lease.duration is also the lease margin. A write starts only while more than the transaction timeout plus the margin of lease validity remains. The PostgreSQL statement_timeout equals the transaction timeout. A COMMIT sent just before the deadline therefore still lands before the lease ends.

The timeout plus the margin is at most half of lease.duration. That is less than the two thirds of validity left when a lease renewal starts, so writes rarely wait for a renewal. With the default 10-second lease, a transaction lasts at most 2.5 seconds. With a 60-second lease, it lasts at most 15 seconds.

Retries and dropped entries

Failed writes are retried with the syncRetryPolicy backoff until totalDuration. The failure decides what backs off:

• Store. A connection failure, or another failure of the whole store, backs off every run of that store.
• Run. A failure caused by the written values or by a conflict with another session backs off only that run. Other Flows keep syncing. Examples are a constraint violation, a value too long for its column, a deadlock, and a serialization failure.
• Timeout. A transaction that exceeds the transaction timeout, for example while it waits for a row lock, also backs off only that run.

After a timeout, the run's retries use half as many statements per transaction, down to one. After a timeout, one statement writes all of the run's plain Attributes in one transaction, so the main row is not partly updated. Each AttributeMap instance is one statement, so a timed-out run's instances can land in different transactions. Each transaction that commits is acknowledged, so a retry writes only the rest.

When three runs of a store time out with no commit in between, the whole store backs off. The retry after a store backoff starts with other runs than the one that failed. A newer run of the same Flow ID waits for the older run, so rows keep run order.

Entries still pending syncRetryPolicy.totalDuration after they were first marked are dropped with an error log. Later writes of the same Attributes sync normally. dexzd_attribute_sync_dropped_entries_total counts dropped entries. It also counts entries that the store rejected with a non-retryable error.

Monitoring

Metric	Meaning
dexzd_attribute_sync_backlog_entries	Entries waiting to be written, as of the last scan
dexzd_attribute_sync_oldest_pending_age_seconds	Age of the oldest pending entry's retry window, 0 when none
dexzd_attribute_sync_write_failures_total with failure_kind	Failed write transactions. failure_kind is retryable or non_retryable. Non-retryable failures drop their entries.
dexzd_attribute_sync_dropped_entries_total	Entries dropped after the retry policy was exhausted or the store rejected them
dexzd_attribute_sync_fenced_writes_total	Writes rejected because a newer owner raised the fencing token

# Dropped Attribute Sync entries
increase(dexzd_attribute_sync_dropped_entries_total[10m]) > 0

# Attribute Sync entries pending for more than 5 minutes
dexzd_attribute_sync_oldest_pending_age_seconds > 300

Local development

dex-server-zd dev runs one local node with embedded object storage. Its --attribute-store-config flag reads only the attributeStore section of a YAML file. That section must define at least one store. The --config overrides file must not also set attributeStore.

dex-server-zd dev --data-dir ./zd-data --attribute-store-config ./attribute-store.yaml

The database still needs its tables and the fencing table. dexcli dev --attribute-store-config passes the file to this flag. It needs dexcli v1.19.0 or later.

Metrics

Set metrics.listen to serve plain HTTP GET /metrics on its own listener, separate from api.listen. The default empty value disables it. The endpoint has no authentication, so never expose it publicly. In Kubernetes, bind it to 0.0.0.0:9090 and restrict access with a NetworkPolicy.

The response uses the OpenMetrics text format. Metrics are available while the node recovers or waits for its lease. Every label value comes from a closed set. No series has a Flow, Flow type, partition, tenant, or key label.

What to watch	Metric family and labels
API requests and errors	dexzd_api_requests_total with rpc_class and code
API latency	dexzd_api_request_latency_seconds with rpc_class; unary calls only
Step failures	dexzd_step_attempt_failures_total with durability and failure_kind; dexzd_step_retries_exhausted_total with durability
Flow closes	dexzd_flow_closes_total with close_status; dexzd_flow_end_to_end_latency_seconds
Dispatch delay	dexzd_step_schedule_to_start_latency_seconds
Workers	dexzd_worker_stream_events_total with event; dexzd_matching_worker_streams; dexzd_matching_queued_tasks
Commit latency	dexzd_commit_latency_seconds
Object storage	dexzd_object_store_requests_total with operation and object_kind; dexzd_object_store_request_errors_total with operation
Lease and node state	dexzd_node_role with role; dexzd_lease_epoch; dexzd_lease_renewals_total with outcome
Recovery	dexzd_recovery_last_duration_seconds; dexzd_recovery_last_replayed_bytes
Timers	dexzd_timer_lag_seconds
Snapshots	dexzd_snapshots_total with outcome; dexzd_snapshot_last_duration_seconds
Local disk	dexzd_local_storage_bytes
Attribute Store sync	See Monitoring

rpc_class is one of flow_write, flow_read, search, worker_poll, and worker_respond. code is the gRPC status code in snake case, such as ok or unavailable. role is one of starting, standby, recovering, serving, and draining.

Worker polls use one persistent stream, so there is no empty-poll count. dexzd_worker_stream_events_total counts each task delivery and each stream close. Long-poll waits count toward flow_read latency.

# Failed API calls by class and code
sum by (rpc_class, code) (rate(dexzd_api_requests_total{code!="ok"}[5m]))

# A node that is not serving
dexzd_node_role{role="serving"} == 0

# Failed lease renewals
increase(dexzd_lease_renewals_total{outcome="error"}[10m]) > 0

The full catalog, with types, histogram buckets, and Linux process metrics, is in the server-zd README.

Pull Worker dispatch

For Worker setup and capacity planning, see Pull Workers. server-zd runs as one node, so it owns every Worker queue itself. Workers and Clients connect to api.listen, which is 0.0.0.0:8801 by default.

matching:
  dispatchWait: 10s
  stickyWait: 5s
  maxConcurrentTasksCap: 256
  maxStreams: 100000
  maxQueueLength: 256
  maxFlowTypes: 256
  maxFlowTypeLength: 256
  pingInterval: 20s

Key	Default	Meaning
matching.dispatchWait	10s	Total time to find a Worker for one attempt.
matching.stickyWait	5s	Time to try a connected sticky Worker before using the Flow type queue. Must not exceed dispatchWait. When no stream for the previous app Worker ID is connected, server-zd uses the Flow type queue at once.
matching.maxConcurrentTasksCap	256	Most streams for one app Worker ID on the node.
matching.maxStreams	100000	Most registered Worker streams on the node.
matching.maxQueueLength	256	Longest queue name, in characters.
matching.maxFlowTypes	256	Most Flow types in one Worker registration.
matching.maxFlowTypeLength	256	Longest registered Flow type, in characters.
matching.pingInterval	20s	Interval between Ping frames on idle Worker streams.

If no Worker is available within dispatchWait, a Step receives a retryable unavailable error that consumes a Step retry. An RPC caller receives an Unavailable error directly.

Current SDKs send upstream Pong frames every 20 seconds while a task runs. Configure load balancers and proxies for long-lived HTTP/2 streams, and set their idle timeouts above both 20 seconds and matching.pingInterval, with margin for network delays. Workers that use older SDKs need idle timeouts above the longest task duration. Preserve bidirectional streaming through the public endpoint.

Streams

Set streamStore.backend to disabled (default) or memory to control best-effort Streams. Memory Streams live in the server process and start empty after a restart. While the backend is disabled, a Stream write is dropped without failing the Step.

Authentication

Public API authentication is off by default. Configure auth.authenticators to require a bearer credential on every FlowService and WorkerTaskService call. gRPC health needs no credential. Authenticators are tried in order.

auth:
  authenticators:
    - kind: jwt
      issuer: https://issuer.example.com
      audiences: ["dex"]
      jwksURL: ""
      allowInsecureJWKSURL: false
      caFile: ""
    - kind: staticTokens
      file: /etc/dex-server-zd/tokens.yaml

allowInsecureJWKSURL allows a plain HTTP jwksURL on a trusted network. An empty jwksURL discovers the key set through the issuer's OpenID configuration. caFile adds a PEM bundle to the system roots.

The static token file lists SHA-256 hashes, never plaintext tokens:

tokens:
  - sha256: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

The file must be valid at startup. Changes take effect within a second. A malformed update keeps the last valid content. An empty or missing file revokes every token.

The server checks a credential once, when a call starts. An open pull Worker stream stays open after its token expires or is revoked. The Worker needs a valid credential when it reconnects.

api.listen serves plaintext gRPC, so bearer credentials cross the network in clear text. Terminate TLS in front of the server, or restrict the network to trusted callers.

Server and SDK protocol compatibility

server-zd and each SDK release declare inclusive protocol intervals. A Worker can start when those intervals overlap. It negotiates the highest common protocol, then synchronizes Attribute indexes, then binds WorkerTaskService. The server and SDK artifact versions are diagnostic and do not decide compatibility.

A new SDK can use an older server when it retains the older protocol, and a new server can use an older SDK while their intervals still overlap. A breaking server release raises its minimum protocol. Stop old Workers and their traffic before that server upgrade, then restart traffic. Alternatively, use isolated blue and green environments so deployments with disjoint intervals never communicate. Running Workers do not renegotiate after a server upgrade, so stop them explicitly before a breaking change.

Protocol compatibility covers the Worker-to-server API only. It does not replace the compatibility rules for open Flows, including Flow type names, Step graphs, and persisted payloads.

Backup and restore

The object storage prefix is the only source of truth. The local store is rebuilt from it. An external search projection is rebuilt from the local store after startup. With search.backend: postgres, restoring an older prefix can leave the database's Attribute registry version ahead of the restored registry. Startup then fails with an Attribute slot registry mismatch. Before restarting, configure a new search.postgresSchema or a new PostgreSQL database so the projection and slot assignments rebuild from the restored registry. When objectStore.hardDeleteStore is configured, its prefix holds hard-delete Attribute values. Protect it the same way.

Protect the prefix with bucket versioning or with replication to another bucket. With versioning, the noncurrent-version expiry sets how far back you can restore. Replication is asynchronous, so a replica can lag the source.

To restore:

1. Stop server-zd. For example, scale the StatefulSet to 0 replicas and wait for the pod to terminate.
2. Restore every object under the prefix to the same point in time. Remove objects written after that point.
3. Empty the local store directory. With a PersistentVolumeClaim, delete the directory contents or the claim. An emptyDir is already gone once the pod terminates.
4. Start server-zd. It recovers from the newest snapshot and the log in the restored prefix.

Writes acknowledged after the restore point are lost.

PostgreSQL search

Choose search.backend: postgres to run indexed search on PostgreSQL 15 or newer without extensions. It does not require pg_search, btree_gin, or any other extension. Choose ParadeDB when you need more indexed Attributes than the PostgreSQL slot limits allow.

search:
  backend: postgres
  postgresConnectionString: "postgres://USER:PASSWORD@HOST:5432/DATABASE"
  postgresConnectionPoolSize: 10
  postgresSchema: public
  readYourWritesWait: 2s

The server connects to postgresConnectionString using the PostgreSQL protocol. It neither binds nor advertises that address. Leave the field empty to read DEX_ZD_POSTGRES_CONNECTION_STRING. The connection supports the same TLS options as paradeDbConnectionString, and serialized configuration redacts it. The default pool size is 10, and the default schema is public.

Each tenant has 10 Keyword slots and 3 slots each for Int, Datetime, Double, Bool, KeywordArray, and Text. Registrations take slots in registration order. System fields, including DexWorkQueuePermissions, do not consume slots. The registry is shared by the applications on the server. Registering another Attribute after a type reaches its limit returns FAILED_PRECONDITION. Startup also fails if the existing registry exceeds these limits. The registry has no delete operation. A full registry requires a new server.

An indexed Keyword value or KeywordArray element may contain at most 1024 UTF-8 bytes. A larger new write returns INVALID_ARGUMENT. A rejected Worker Step result causes that Step to fail. A previously committed document with an oversized value whose stored type matches its registered Keyword or KeywordArray type is omitted from search and recorded as a skipped document. Its Flow state remains committed. Unregistered keys do not trigger this skip.

A newly registered Attribute uses the existing JSONB query path until its slot backfill finishes. Results stay correct during that window, but queries can be slower. Backfill commits bounded batches and resumes after a restart. It does not pause projection of other partitions. Unregistered index keys also use JSONB. IS NULL, IS NOT NULL, and negations use stored key presence, including when the stored type differs from the registered type. Int and Double values remain distinct. A scalar string stored under KeywordArray matches as one element. One mismatch has a stricter result: positive Keyword predicates do not match a stored KeywordArray. The document remains searchable, and negations retain the builtin evaluator result. Negative predicates, such as != and NOT IN, often cannot use an index. Explicit ORDER BY StartTime and ORDER BY CloseTime sort all matching rows. Their cost grows with the number of matching rows.

Text matches all query terms. It does not rank results by BM25 relevance. Projection is asynchronous, and the same readYourWritesWait applies to both external search backends.

A schema records its search backend. Starting with a different backend fails. To switch between PostgreSQL and ParadeDB, select a new database or schema. The server rebuilds its search projection from committed state automatically. Existing ParadeDB version-three schemas gain the dialect marker without clearing data or changing their version. Older releases that accept schema version 3 can still use them. PostgreSQL uses schema version 4; older releases reject it.