Kubernetes deployment
View as MarkdownOne replica in a StatefulSet
Run server-zd as a StatefulSet with exactly one replica. A StatefulSet creates the replacement pod only after the old pod is gone. A Deployment can start a new pod while the old one still runs.
Never force-delete the server pod. A force delete removes the pod from the API before Kubernetes confirms that its process stopped. On an unreachable node, the old process can keep running next to its replacement.
This example uses a PersistentVolumeClaim for the local store and a ConfigMap named dex-server-zd that holds config.yaml. The image is the published multi-architecture release image for amd64 and arm64.
apiVersion: v1
kind: Service
metadata:
name: dex-server-zd
spec:
selector:
app: dex-server-zd
ports:
- name: grpc
port: 8801
targetPort: grpc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dex-server-zd
spec:
serviceName: dex-server-zd
replicas: 1
selector:
matchLabels:
app: dex-server-zd
template:
metadata:
labels:
app: dex-server-zd
spec:
serviceAccountName: dex-server-zd
terminationGracePeriodSeconds: 45
securityContext:
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
containers:
- name: dex-server-zd
image: docker.io/superdurable/dex-server-zd:0.1.3
ports:
- name: grpc
containerPort: 8801
- name: metrics
containerPort: 9090
startupProbe:
grpc:
port: 8801
periodSeconds: 5
failureThreshold: 60
readinessProbe:
grpc:
port: 8801
periodSeconds: 5
livenessProbe:
grpc:
port: 8801
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: config
mountPath: /etc/dex-server-zd/config.yaml
subPath: config.yaml
readOnly: true
- name: data
mountPath: /var/lib/dex-server-zd
volumes:
- name: config
configMap:
name: dex-server-zd
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
Graceful shutdown
On SIGTERM, the node stops handing out new Step tasks and reports NOT_SERVING. It keeps its lease and pull Worker streams open while running attempts finish. Heartbeats, completions, and failures continue to commit. The node releases the lease only after those attempts finish and their results commit, or after lifecycle.attemptDrainTimeout expires. A hot standby follows the commits and takes over after the release.
During the drain, new Flow RPCs answer UNAVAILABLE, and the SDKs retry them. Steps that are not yet handed to a Worker wait. The successor dispatches them after it takes the lease. A Step whose start was already being written when the drain began still goes to its Worker and is drained like a running attempt. So does a Step that a Worker took from the Step task queue before the drain began. Steps still in the task queue stay there for the successor.
lifecycle.attemptDrainTimeout defaults to 30s. Set it to 0s to disable attempt draining. Step method and heartbeat timeouts still apply during the drain; the drain does not extend them. Choose a drain timeout long enough for the remaining execution time of the Step methods you want to finish. dexcli dev uses a 2s drain unless its configuration sets this field, because it stops the server after 10 seconds. dex-server-zd local keeps the 30s default.
Set Kubernetes terminationGracePeriodSeconds longer than the drain timeout, with at least 15 extra seconds for teardown. For the default 30s drain, allow at least 45 seconds. Otherwise Kubernetes may kill the process before it can commit results and release its lease.
After the attempt drain, Workers reconnect, long polls and the Stream Store close, and in-flight RPCs get up to 4 seconds to finish. Background work stops, then the engine releases the lease. Other open calls end with UNAVAILABLE so clients retry on the new active process. The metrics listener closes last. This teardown has a separate budget of about 15 seconds.
The Step attempt drain ended log records attempts_waited, attempts_finished, and attempts_running, including attempts still waiting for their final commit. Its outcome field is completed when every attempt finished, timed_out when the drain timeout expired, or node_stopped when the node stopped for another reason first. With a 0s timeout, the node logs Step attempt drain disabled instead.
The process exits 0 when every step completes. A replacement process with the same cluster.nodeId then acquires the lease at once. The process exits 1 when some shutdown step does not complete in time or fails, for example when object storage rejects the lease release. A replacement may then wait up to lease.duration plus about 1 second.
Attempts that Dex has not handed to a Worker yet do not fail. The successor dispatches them. Attempts that are running when the process lets go of the lease are failed by the new active process with NOT_COMPLETED, and each consumes a retry.
A SIGTERM during startup interrupts lease acquisition or recovery. If the node holds its lease, it stops startup and releases the lease within the shutdown budget. A failed release makes the process exit 1, and the next start may wait for the old lease to expire. If the outcome of a lease write is still unknown when the SIGTERM arrives, the process exits 0 without releasing that write, and the next start may wait for it to expire.
Local store volume
The local store is a cache, because every acknowledged write is already in object storage. The volume type only changes restart time.
With an emptyDir volume, a new pod restores the newest snapshot and replays the log written after it. An emptyDir survives a container restart in the same pod, but not a new pod. With a PersistentVolumeClaim, a new pod reuses the local store and replays only the log written since it stopped. Either way, the first snapshot after a restart uploads the full local checkpoint to object storage.
Mount the volume at a parent directory, such as /var/lib/dex-server-zd. Set localStorage.directory to a subdirectory, such as /var/lib/dex-server-zd/store. When the node cannot reuse an existing local store, it renames the directory to a sibling with a .discarded- suffix and rebuilds it. This happens when RocksDB reports the store corrupt, missing files, or in an incompatible format while opening it, or when the store cannot be updated from the retained log. That rename fails when the directory is the mount point itself.
Other errors that stop the node from opening the store fail startup and leave the directory in place. Examples are another process that holds the store open and a permission error. Stop the other process or fix the permissions, then restart. If the store itself cannot be read, for example after a disk error, stop the node, move or delete the store directory, and restart. The node then rebuilds the store from object storage.
A store that is newer than the metadata in object storage is also set aside, and the node logs an error. That copy may be the only local evidence that the metadata went backwards.
After each successful start, the node deletes older set-aside directories. It keeps the newest localStorage.retainedSetAsideStoreCount of them for diagnosis, 1 by default. A copy that the current start set aside always counts as the newest. Set the count to 0 to keep none. The node then deletes every copy at once, including a copy of a store that is newer than the metadata. The node logs each path it deletes. A failed deletion logs a warning and does not stop startup. The node deletes only directories that it set aside for this store. It never sets aside or deletes the active store while another process holds its lock. The node never opens a set-aside copy again. Cleanup does not check whether another tool has a set-aside copy open.
Size the volume for the live state, plus localStorage.maxWalBytes of write-ahead log (256 MiB by default), plus room for uncompacted files after a bulk replay. The node compacts the store after such a replay, which returns the directory to about the live state size.
Also leave room for the retained set-aside copies. Each copy can be as large as the store was when the node set it aside. After a successful start, the volume holds at most localStorage.retainedSetAsideStoreCount copies. A start that sets the store aside holds one extra copy until it succeeds. Starts that fail after setting the store aside, such as a crash loop, add a copy each time until a start succeeds. With the default of 1, plan for at least two old copies next to the rebuilt store.
To use an emptyDir instead of the claim template, replace volumeClaimTemplates with a volume like this:
volumes:
- name: data
emptyDir:
sizeLimit: 20Gi
Health checks
Port 8801 serves the standard gRPC health service on api.listen. Health reports NOT_SERVING during startup, while the node waits for its lease and replays the log. It reports SERVING after recovery completes.
Use gRPC health for all three probes:
- The startupProbe holds off the liveness probe until the first SERVING. Size it so that failureThreshold × periodSeconds is at least lease.duration + 1 s + the worst-case recovery time. The lease part covers a previous owner that did not release its lease. Recovery time grows with live state and the log tail. A cold restore of an empty store took about 4 s; plan for more with real data. The example allows 5 minutes.
- The readinessProbe keeps traffic away while the node is not serving.
- The livenessProbe restarts a node that stops answering. On a node that never steps down, prefer gRPC health to a TCP probe: a stalled process can still accept TCP connections in the kernel backlog while it serves nothing.
A serving node that never steps down returns to NOT_SERVING only by exiting, because a lost lease ends the process. Liveness on gRPC health therefore does not cause restart loops for it. A step down makes the active process report NOT_SERVING while it keeps running as a standby, and a liveness probe on gRPC health would restart it. A process that may step down needs the TCP or exec liveness probe of a standby.
This probe set is for a single replica. A standby replica stays NOT_SERVING until it takes over, so it would never pass this startup probe. A standby needs a TCP or exec startup probe and liveness probe instead. See hot standby.
The process exits on its own after a fatal error, such as a lost lease or a failed commit. During startup, transient or unknown-outcome object storage read errors are retried with jittered exponential backoff. recovery.startupReadRetryBudget sets a wall-clock window for each read, starting at its first transient or unknown-outcome failure. The default is 30 seconds and must be positive. Backoff and subsequent read calls count toward that read's window. Concurrent reads do not share time, and a later read gets a new window. The error names the operation, object key, and last error when the window expires. Missing required objects, corrupt data, access denial, and lost lease ownership fail startup without read retries. Lease validity or shutdown may end retries sooner. Lease renewal continues during recovery retries. The window bounds each read, not the whole startup. After the lease is acquired, startup has no overall time limit. A persistent outage still fails startup within one read's window, and lease renewal keeps the lease while slow reads make progress. The startup probe bounds the total startup time. If object storage is unreachable at startup, the conditional-write self-check also exits 1. If startup fails, the process exits 1. Kubernetes restarts the container with backoff.