Advanced: Multiple Parents with Partitioning
Route requests across several parent Flow IDs so the total number of active SubFlows can grow beyond one parent's concurrency limit.
Use a stable partition key, such as a request ID hash modulo the number of parents. Every request with the same key reaches the same parent. Each parent independently applies its Channel buffer and SubFlow concurrency limit. Increasing the parent count therefore increases total capacity without making one Flow execution unbounded.
Changing the number or order of partitions remaps keys. Use a versioned parent set or consistent hashing when stable affinity matters during scaling.
Core implementation
The submitter first invokes the selected parent's RPC. If that parent is absent or no longer active, it starts the parent with the request as its initial input. If another submitter wins the start race, the start fails and SubmitStep retries. Its next execution invokes the RPC on the active parent. The initial request is not lost because the parent's InitStep publishes it to RequestChannel.
async def enqueue_request(client, parent_flow, parent_id: str, request: str) -> bool:
try:
return await client.invoke_rpc(parent_flow.send_request, parent_id, request)
except FlowNotActiveError:
await client.start_flow(
parent_flow,
parent_id,
ParentInput([request], DEFAULT_CONCURRENCY),
StartFlowOptions(id_reuse_policy=IdReusePolicy.ALLOW_IF_NOT_RUNNING),
)
return True
def partition(request: str, partitions: int) -> int:
hash_value = 2_166_136_261
for byte in request.encode():
hash_value ^= byte
hash_value = hash_value * 16_777_619 & 0xFFFFFFFF
return hash_value % partitions
Example: examples/python/dex_examples/patterns/parallel-subflows/submit_request_flow.py