Skip to main content

Sequentially Chunked AttributeMap

Use this pattern for append-only data that must retain insertion order, such as agent messages, subscriber history, or audit events.

A single Attribute grows into one large blob. Every append then reads and rewrites the entire value. A sequentially chunked AttributeMap keeps the active write set bounded. The sample stores at most 100 subscribers in one instance.

Data layout​

ChunkedSubscriberFlow is an RPC-only Flow. Start it once with no Flow timeout.

  • SubscriberArchiveState stores the next sequence and the newest archived chunk token.
  • SubscriberChunks[current] is the only mutable chunk.
  • A full chunk moves to an immutable instance named by its zero-padded first sequence, such as 00000000000000000101.

RegisterSubscriber locks the state and current instance. Most calls rewrite one bounded chunk. When current already contains 100 records, the same commit writes the archived chunk, creates the next current chunk, and advances the state.

Lock acquisition does not queue callers. Concurrent appends can receive an RPCLockConflict. Retry specifically after that error with application-appropriate backoff. Successful appends serialize through the same state and current locks, so concurrent requests do not lose or duplicate sequences.

GetSubscriberPage starts at current unless the caller supplies a page token. The application adapter uses RPCInvokeOptions to load only that instance. Pages return newest-first records plus the next older token.

page = await app_state.client.invoke_rpc(
    flow.get_subscriber_page,
    FLOW_ID,
    GetSubscriberPageInput(page_token),
    options=RPCInvokeOptions(
        load_attribute_map_instances=(flow.subscriber_chunks.load(page_token),)
    ),
)

Example: examples/python/dex_examples/patterns/sequentially-chunked-attribute-map/controller.py

Pagination rules​

The current token is always valid after the Flow starts. An empty current returns an empty page. Archive tokens must contain exactly 20 decimal digits and identify a chunk boundary. A malformed token or a valid-looking token without a stored chunk returns an explicit error.

Archived chunks are immutable. This keeps pagination stable and makes retry behavior easy to reason about. Updating older records requires a different pattern or a separate index.

HTTP API​

POST /patterns/sequentially-chunked-attribute-map/start
POST /patterns/sequentially-chunked-attribute-map/register
GET  /patterns/sequentially-chunked-attribute-map/subscribers?pageToken=current

The fixed chunk size is part of the persisted layout. A later version can use a new chunk size for new archives, but its token traversal must understand both layouts.

Run the HTTP example from the examples playground.