External Attribute Sync

View as MarkdownView as Markdown

External Attribute Sync projects selected Flow Attribute writes into PostgreSQL tables that you own. Each destination is called an Attribute Store. Dex uses asynchronous, latest-state writes for one Flow at a time. Delivery is attempted at least once, so every destination write is idempotent.

PostgreSQL is the only supported Attribute Store. Redis support is tracked in issue 616.

Dex never creates, migrates, or backfills a destination. Create the SQL table before starting the server.

How it fits with the other components

External Attribute Sync is a server feature. A Flow marks an Attribute with syncToAttributeStore and lists a configured store name in its FlowConfig. server-zd then writes the Attribute to that store's tables. Your application and your own reports query those tables with SQL.

Configuration

attributeStore:
  stores:
    reporting:
      type: postgres
      dsn: postgres://user:password@host/database
      tableName: public.flow_attributes
    orders:
      type: postgres
      dsn: postgres://user:password@host/database
      tableName: public.orders
      rowKey: order-{customer_id}-{order_id}
      columnNaming: snake_case
      attributeMaps:
        LineItems:
          tableName: public.order_line_items
          instanceKeyColumn: line_item_id
  schemaSyncInterval: 1m
  syncBatchSize: 100
  syncAttemptTimeout: 30s
  syncRetryPolicy:
    initialInterval: 100ms
    maximumInterval: 30s
    backoffCoefficient: 2
    maximumAttempts: 0
    totalDuration: 1h

Every store requires type: postgres, tableName, and a DSN, which is dsn or an environment variable. server-zd never logs DSNs. tableName accepts table or schema.table. Identifiers are parsed and quoted. Attribute values always use driver parameters. See Attribute Store sync configuration for every key and its default.

Delivery and retries

Attribute Sync stores latest state, not an event history. A later write may repeat after an ambiguous failure. Upserts therefore use the Flow ID as their idempotency key.

Writes never block a Step, RPC, or SetAttributes commit. Failed writes are retried until syncRetryPolicy.totalDuration, then dropped with an error log. Dex does not automatically replay old Attributes or backfill a newly configured store. Filtered writes, missing targets, and retry exhaustion emit error logs.

See Attribute Store sync in the Server reference for the fencing table, closing behavior, retries, metrics, row keys, AttributeMap tables, and the SQL type contract.

Local development

dex-server-zd dev runs one local node with embedded object storage. Its --attribute-store-config flag reads only the attributeStore section of a YAML file. That section must define at least one store. The --config overrides file must not also set attributeStore.

dex-server-zd dev --data-dir ./zd-data --attribute-store-config ./attribute-store.yaml

The database still needs its tables and the fencing table. dexcli dev --attribute-store-config passes the file to this flag.