Object storage and backup

查看 Markdown查看 Markdown

Credentials

objectStore.credentialsSource selects the credentials:

Value	Credentials
defaultChain (default)	The AWS SDK default credential provider chain
environment	DEX_ZD_OBJECT_STORE_ACCESS_KEY_ID, DEX_ZD_OBJECT_STORE_SECRET_ACCESS_KEY, and the optional DEX_ZD_OBJECT_STORE_SESSION_TOKEN
anonymous	Unsigned requests, for a local emulator or a bucket that allows anonymous access

On Amazon EKS, keep defaultChain and give the pod's service account an IAM role, for example through IAM Roles for Service Accounts or EKS Pod Identity.

Prefix-confined IAM policy

For a shared bucket, set a non-empty objectStore.prefix. Then grant the server role only these actions. Replace BUCKET and PREFIX/ with the bucket and the configured prefix, and keep the trailing slash.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET",
      "Condition": { "StringLike": { "s3:prefix": "PREFIX/*" } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::BUCKET/PREFIX/*"
    }
  ]
}

The server lists only under its prefix, including when it checks for a missing object. With this grant, AWS S3 returns 404 for a missing object. Narrower grants can return 403. The server handles either response when the prefixed listing is allowed.

The s3:ListBucket condition must use StringLike with the trailing wildcard shown above. An exact-prefix StringEquals grant is not supported. Neither is a policy without s3:ListBucket. With a non-empty prefix, startup lists under it first. A denied listing fails startup with a message that starts with this text:

startup requires s3:ListBucket with StringLike s3:prefix "PREFIX/*"

Startup also writes and deletes a probe object under the prefix. It uses this to check that conditional writes work. No bucket-wide access or bucket administration action is needed.

Hard-delete Attribute values use the primary bucket and prefix by default. To keep them in a separate place, configure objectStore.hardDeleteStore with its own endpoint, region, bucket, prefix, and credentialsSource. Its environment credentials use the DEX_ZD_HARD_DELETE_STORE_ variables with the same suffixes. Grant the same policy for that bucket and prefix.

Encryption

server-zd sends no server-side encryption headers, so the bucket's default encryption applies. Default SSE-KMS works. Conditional writes use If-None-Match and If-Match, and the server treats ETags as opaque values. It does not rely on ETags being MD5 digests.

For a customer-managed KMS key, grant the role kms:GenerateDataKey and kms:Decrypt on that key. Allow both actions in the key policy too.

Versioning and lifecycle rules

An unversioned bucket works. server-zd does not use object versions.

If bucket versioning is on, every object the server deletes stays as a noncurrent version. This includes old log objects and snapshots, so storage only grows. Add a lifecycle rule that expires noncurrent versions. A noncurrent version of a hard-delete Attribute value also keeps the deleted value until the version expires.

Add a rule that aborts incomplete multipart uploads after 1 to 7 days, whether or not versioning is on. A crash can leave an incomplete upload behind. A prefix-confined role cannot list or clean up those uploads.

This lifecycle configuration covers both rules. Drop the second rule for an unversioned bucket.

{
  "Rules": [
    {
      "ID": "dex-server-zd-abort-incomplete-uploads",
      "Status": "Enabled",
      "Filter": { "Prefix": "PREFIX/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 2 }
    },
    {
      "ID": "dex-server-zd-expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": { "Prefix": "PREFIX/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 7 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

Backup and restore

The object storage prefix is the only source of truth. The local store is rebuilt from it. An external search projection is not rebuilt: after a restore, configure a new search.paradeDbSchema or search.postgresSchema, or a new database, so the server loads it from the restored state. See Initial load. Reusing the old schema fails: the restored node epoch is older than the one that the schema records, so the projection stops and SearchFlows returns UNAVAILABLE. With search.backend: postgres, reusing the old schema can also leave the database's Attribute registry version ahead of the restored registry. Startup then fails with an Attribute slot registry mismatch. When objectStore.hardDeleteStore is configured, its prefix holds hard-delete Attribute values. Protect it the same way.

Protect the prefix with bucket versioning or with replication to another bucket. With versioning, the noncurrent-version expiry sets how far back you can restore. Replication is asynchronous, so a replica can lag the source.

To restore:

1. Stop server-zd. For example, scale the StatefulSet to 0 replicas and wait for the pod to terminate.
2. Restore every object under the prefix to the same point in time. Remove objects written after that point.
3. Empty the local store directory. With a PersistentVolumeClaim, delete the directory contents or the claim. An emptyDir is already gone once the pod terminates.
4. Start server-zd. It recovers from the newest snapshot and the log in the restored prefix.

Writes acknowledged after the restore point are lost.