Metrics and alerts
查看 MarkdownSet metrics.listen to serve plain HTTP GET /metrics on its own listener, separate from api.listen. The default empty value disables it. The endpoint has no authentication, so never expose it publicly. In Kubernetes, bind it to 0.0.0.0:9090 and restrict access with a NetworkPolicy.
The response uses the OpenMetrics text format. Metrics are available while the node recovers or waits for its lease. Every label value comes from a closed set. No series has a Flow, Flow type, partition, tenant, or key label.
| What to watch | Metric family and labels |
|---|---|
| API requests and errors | dexzd_api_requests_total with rpc_class and code |
| API latency | dexzd_api_request_latency_seconds with rpc_class; unary calls only |
| Step failures | dexzd_step_attempt_failures_total with durability and failure_kind, which includes started_attempt_interrupted; dexzd_step_retries_exhausted_total with durability |
| Returned attempts | dexzd_step_attempts_returned_total with reason |
| Unsupported types | dexzd_matching_unsupported_type_entries; dexzd_matching_unsupported_type_evictions_total |
| Worker stream protocol errors | dexzd_worker_stream_protocol_errors_total with error |
| Flow closes | dexzd_flow_closes_total with close_status; dexzd_flow_end_to_end_latency_seconds |
| Dispatch delay | dexzd_step_schedule_to_start_latency_seconds |
| Workers | dexzd_worker_stream_events_total with event; dexzd_matching_worker_streams; dexzd_matching_queued_tasks (in-memory waiters only) |
| Dropped Step Stream writes | dexzd_step_stream_writes_dropped_total with reason |
| Step task queue | dexzd_step_task_queue_entries; dexzd_step_task_queue_oldest_age_seconds; dexzd_matching_unserved_task_queues with reason; dexzd_matching_heads_restored_refs_total; dexzd_matching_heads_incomplete_scans_total; dexzd_step_task_queue_incomplete_seeks_total |
| Commit latency | dexzd_commit_latency_seconds |
| Object storage | dexzd_object_store_requests_total with operation and object_kind; dexzd_object_store_request_errors_total with operation |
| Lease and node state | dexzd_node_role with role; dexzd_lease_epoch; dexzd_lease_renewals_total with outcome |
| Standby | dexzd_hot_standby_state with state; dexzd_hot_standby_lag_seconds; dexzd_hot_standby_lag_batches; dexzd_hot_standby_error; dexzd_takeovers_total with standby_mode and store_source; dexzd_takeover_duration_seconds |
| Recovery | dexzd_recovery_last_duration_seconds; dexzd_recovery_last_replayed_bytes |
| Timers | dexzd_timer_lag_seconds |
| Snapshots | dexzd_snapshots_total with outcome; dexzd_snapshot_last_duration_seconds |
| Local disk | dexzd_local_storage_bytes |
| Attribute Store sync | See Monitoring |
| External search projection | See Monitoring |
rpc_class is one of flow_write, flow_read, search, worker_poll, and worker_respond. code is the gRPC status code in snake case, such as ok or unavailable. role is one of starting, standby, recovering, serving, and draining. state is one of inactive, bootstrapping, following, behind, restarting, and failed. Only following is hot. The active process and a cold standby report inactive. standby_mode is hot or cold. store_source is hot_standby when the node serves the store that its hot standby kept current, and restored when it opened its own store or restored a snapshot, such as when the standby store was missing or stale. dexzd_takeover_duration_seconds covers hot takeovers only, by store_source.
reason of dexzd_step_attempts_returned_total counts attempts that went back to pending without consuming a retry. It is one of unsupported (a Worker declined the Step type), legacy_no_capacity (an older SDK Worker declined for capacity), and not_delivered (the task never reached the Worker). legacy_no_capacity should drop to zero once every SDK is upgraded. error of dexzd_worker_stream_protocol_errors_total is one of no_capacity_on_reserved_slot, half_close_after_output, and reject_after_output. With no_capacity_on_reserved_slot, a reserved-slot Worker declined a task for capacity. The attempt fails as not run, and the stream ends. With half_close_after_output, a Worker closed its stream after output but before the final result. The attempt fails with NOT_COMPLETED. With reject_after_output, a Worker declined a task after it had already sent output for it. The attempt fails as possibly run, with NOT_COMPLETED. The server ends that stream with INVALID_ARGUMENT, so it gets no further task. dexzd_matching_unsupported_type_evictions_total is a counter. A steady rise means you should raise matching.unsupportedTypeCacheEntries. An evicted pair is not sent to its Worker again until that Worker completes a Step. While the cache overflows, a type that a Worker declined can therefore be offered to it again after each Step it completes. Raise matching.unsupportedTypeCacheEntries when this counter rises or the server logs the full-cache WARN.
Worker polls use one persistent stream, so there is no empty-poll count. dexzd_worker_stream_events_total counts each stream close and, with event="delivered", every task frame sent, including RPC task frames. A returned attempt that Dex hands out again counts again. Long-poll waits count toward flow_read latency.
# Failed API calls by class and code
sum by (rpc_class, code) (rate(dexzd_api_requests_total{code!="ok"}[5m]))
# A node that is not serving
dexzd_node_role{role="serving"} == 0
# Failed lease renewals
increase(dexzd_lease_renewals_total{outcome="error"}[10m]) > 0
The full catalog, with types, histogram buckets, and Linux process metrics, is in the server-zd README.