--- title: "Object storage and backup" description: "Credentials" canonical_url: "https://docs.superdurable.io/reference/server/0.1.23/object-storage/" source: "docs/reference-server_versioned_docs/version-0.1.23/object-storage.mdx" locale: "en" --- # Object storage and backup ## Credentials **objectStore.credentialsSource** selects the credentials: | Value | Credentials | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | **defaultChain** (default) | The AWS SDK default credential provider chain | | **environment** | **DEX_ZD_OBJECT_STORE_ACCESS_KEY_ID**, **DEX_ZD_OBJECT_STORE_SECRET_ACCESS_KEY**, and the optional **DEX_ZD_OBJECT_STORE_SESSION_TOKEN** | | **anonymous** | Unsigned requests, for a local emulator or a bucket that allows anonymous access | On Amazon EKS, keep **defaultChain** and give the pod's service account an IAM role, for example through IAM Roles for Service Accounts or EKS Pod Identity. ## Prefix-confined IAM policy For a shared bucket, set a non-empty **objectStore.prefix**. Then grant the server role only these actions. Replace **BUCKET** and **PREFIX/** with the bucket and the configured prefix, and keep the trailing slash. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET", "Condition": { "StringLike": { "s3:prefix": "PREFIX/*" } } }, { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"], "Resource": "arn:aws:s3:::BUCKET/PREFIX/*" } ] } ``` The server lists only under its prefix, including when it checks for a missing object. With this grant, AWS S3 returns 404 for a missing object. Narrower grants can return 403. The server handles either response when the prefixed listing is allowed. The **s3:ListBucket** condition must use **StringLike** with the trailing wildcard shown above. An exact-prefix **StringEquals** grant is not supported. Neither is a policy without **s3:ListBucket**. With a non-empty prefix, startup lists under it first. A denied listing fails startup with a message that starts with this text: ```text startup requires s3:ListBucket with StringLike s3:prefix "PREFIX/*" ``` Startup also writes and deletes a probe object under the prefix. It uses this to check that conditional writes work. No bucket-wide access or bucket administration action is needed. Hard-delete Attribute values use the primary bucket and prefix by default. To keep them in a separate place, configure **objectStore.hardDeleteStore** with its own **endpoint**, **region**, **bucket**, **prefix**, and **credentialsSource**. Its **environment** credentials use the **DEX_ZD_HARD_DELETE_STORE\_** variables with the same suffixes. Grant the same policy for that bucket and prefix. ## Encryption server-zd sends no server-side encryption headers, so the bucket's default encryption applies. Default SSE-KMS works. Conditional writes use **If-None-Match** and **If-Match**, and the server treats ETags as opaque values. It does not rely on ETags being MD5 digests. For a customer-managed KMS key, grant the role **kms:GenerateDataKey** and **kms:Decrypt** on that key. Allow both actions in the key policy too. ## Versioning and lifecycle rules An unversioned bucket works. server-zd does not use object versions. If bucket versioning is on, every object the server deletes stays as a noncurrent version. This includes old log objects and snapshots, so storage only grows. Add a lifecycle rule that expires noncurrent versions. A noncurrent version of a hard-delete Attribute value also keeps the deleted value until the version expires. Do not add a rule that expires current objects under **PREFIX/history/**. The server reads uploaded run history there for as long as the history window covers it, and a lifecycle rule cannot tell which objects the window still needs. A read of a missing object fails with UNAVAILABLE. The server deletes old history objects of offloaded runs itself once the window has passed them; see [History offload](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#history-offload). Add a rule that aborts incomplete multipart uploads after 1 to 7 days, whether or not versioning is on. A crash can leave an incomplete upload behind. A prefix-confined role cannot list or clean up those uploads. This lifecycle configuration covers both rules. Drop the second rule for an unversioned bucket. ```json { "Rules": [ { "ID": "dex-server-zd-abort-incomplete-uploads", "Status": "Enabled", "Filter": { "Prefix": "PREFIX/" }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 2 } }, { "ID": "dex-server-zd-expire-noncurrent-versions", "Status": "Enabled", "Filter": { "Prefix": "PREFIX/" }, "NoncurrentVersionExpiration": { "NoncurrentDays": 7 }, "Expiration": { "ExpiredObjectDeleteMarker": true } } ] } ``` ## Snapshot uploads after ownership changes The first snapshot after a takeover or restart uploads the full local checkpoint to a new lineage. It cannot reuse objects from the previous lineage, even when the local RocksDB directory is reused. Allow for the full checkpoint size, manifest data, upload throughput, and per-file request time. Later snapshots from the same owner can reuse unchanged files. ## History offload A run that offloads its history seals it into segments and uploads them to the primary object store under **PREFIX/history/**. Uploaded segments let the server reclaim local history rows and, after a grace period, superseded values. Segments expire when the history window passes them, and the server deletes their objects about an hour later. Each run chooses at start whether it offloads, and it does by default. A run without offload never writes under **PREFIX/history/**. See the [offload configuration](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#history-offload) for the per-run choice, the grace periods, and the history window. Changed in 0.1.23: runs choose history offload one by one, also on **serve**, and **history.offloadEnabled** was removed. Runs that started before the upgrade keep the offload mode that their partition recorded at the upgrade. A **serve** deployment whose offload was off at the upgrade therefore seals no new history of its existing runs. This includes one that turned **history.offloadEnabled** on and then off again: segments it sealed before can still finish uploading, and uploaded history stays readable. One whose offload was on at the upgrade keeps uploading the history of its existing runs, and those runs keep migrating. Runs that start after the upgrade offload unless they turn it off. Stop the standby and active processes and upgrade both together. Start the active process first, then the standby. Do not run an earlier release on the prefix afterwards: history that new runs removed from local disk needs object storage reads. See [Upgrades and compatibility](https://docs.superdurable.io/reference/server/0.1.23/upgrades.md). Never add lifecycle expiry rules for current objects under **PREFIX/history/**; follow the [lifecycle guidance](https://docs.superdurable.io/reference/server/0.1.23/object-storage.md#versioning-and-lifecycle-rules) above. Watch **dexzd_history_upload_backlog_segments**, **dexzd_history_upload_backlog_bytes**, and **dexzd_history_upload_oldest_sealed_age_seconds**. Alert if the oldest sealed segment's age keeps growing or uploads fail, as counted by **dexzd_history_uploads_total** with **outcome="failed"**. A growing backlog means more history stays on local disk. Since 0.1.23, the number of uploads in flight adapts to the backlog; see [Upload concurrency](https://docs.superdurable.io/reference/server/0.1.23/configuration.md#history-upload-concurrency). Use **dexzd_history_reads_total** to track reads from local disk, cache, and object storage, and **dexzd_history_value_revivals_total** to track values restored to local disk. Since 0.1.22, **dexzd_history_expired_objects_deleted_total** and **dexzd_history_orphan_objects_deleted_total** count object cleanup. See [Metrics and alerts](https://docs.superdurable.io/reference/server/0.1.23/metrics.md) for labels and alert thresholds. ## Backup and restore The object storage prefix is the only source of truth. The local store is rebuilt from it. An external search projection is not rebuilt: after a restore, configure a new **search.paradeDbSchema** or **search.postgresSchema**, or a new database, so the server loads it from the restored state. See [Initial load](https://docs.superdurable.io/reference/server/0.1.23/search.md#search-projection-initial-load). Reusing the old schema fails: the restored node epoch is older than the one that the schema records, so the projection stops and SearchFlows returns **UNAVAILABLE**. With **search.backend: postgres**, reusing the old schema can also leave the database's Attribute registry version ahead of the restored registry. Startup then fails with an Attribute slot registry mismatch. When **objectStore.hardDeleteStore** is configured, its prefix holds hard-delete Attribute values. Protect it the same way. Protect the prefix with bucket versioning or with replication to another bucket. With versioning, the noncurrent-version expiry sets how far back you can restore. Replication is asynchronous, so a replica can lag the source. To restore: 1. Stop server-zd. For example, scale the StatefulSet to 0 replicas and wait for the pod to terminate. 2. Restore every object under the prefix to the same point in time. Remove objects written after that point. 3. Empty the local store directory. With a PersistentVolumeClaim, delete the directory contents or the claim. An emptyDir is already gone once the pod terminates. 4. Start server-zd. It recovers from the newest snapshot and the log in the restored prefix. Writes acknowledged after the restore point are lost.