--- title: "Metrics and alerts" description: "Set metrics.listen to serve plain HTTP GET /metrics on its own listener," canonical_url: "https://docs.superdurable.io/zh-Hans/reference/server/metrics/" source: "docs/reference/server/metrics.mdx" locale: "zh-Hans" --- # Metrics and alerts Set **metrics.listen** to serve plain HTTP **GET /metrics** on its own listener, separate from **api.listen**. The default empty value disables it. The endpoint has no authentication, so never expose it publicly. In Kubernetes, bind it to **0.0.0.0:9090** and restrict access with a NetworkPolicy. The response uses the OpenMetrics text format. Metrics are available while the node recovers or waits for its lease. Every label value comes from a closed set. No series has a Flow, Flow type, partition, tenant, or key label. | What to watch | Metric family and labels | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | API requests and errors | **dexzd_api_requests_total** with **rpc_class** and **code** | | API latency | **dexzd_api_request_latency_seconds** with **rpc_class**; unary calls only | | Step failures | **dexzd_step_attempt_failures_total** with **durability** and **failure_kind**, which includes **started_attempt_interrupted**; **dexzd_step_retries_exhausted_total** with **durability** | | Returned attempts | **dexzd_step_attempts_returned_total** with **reason** | | Unsupported types | **dexzd_matching_unsupported_type_entries**; **dexzd_matching_unsupported_type_evictions_total** | | Worker stream protocol errors | **dexzd_worker_stream_protocol_errors_total** with **error** | | Flow closes | **dexzd_flow_closes_total** with **close_status**; **dexzd_flow_end_to_end_latency_seconds** | | Dispatch delay | **dexzd_step_schedule_to_start_latency_seconds** | | Workers | **dexzd_worker_stream_events_total** with **event**; **dexzd_matching_worker_streams**; **dexzd_matching_queued_tasks** (in-memory waiters only) | | Dropped Step Stream writes | **dexzd_step_stream_writes_dropped_total** with **reason** | | Step task queue | **dexzd_step_task_queue_entries**; **dexzd_step_task_queue_oldest_age_seconds**; **dexzd_matching_unserved_task_queues** with **reason**; **dexzd_matching_heads_restored_refs_total**; **dexzd_matching_heads_incomplete_scans_total**; **dexzd_step_task_queue_incomplete_seeks_total** | | Commit latency | **dexzd_commit_latency_seconds** | | Object storage | **dexzd_object_store_requests_total** with **operation** and **object_kind**; **dexzd_object_store_request_errors_total** with **operation** | | Lease and node state | **dexzd_node_role** with **role**; **dexzd_lease_epoch**; **dexzd_lease_renewals_total** with **outcome** | | Standby | **dexzd_hot_standby_state** with **state**; **dexzd_hot_standby_lag_seconds**; **dexzd_hot_standby_lag_batches**; **dexzd_hot_standby_error**; **dexzd_takeovers_total** with **standby_mode** and **store_source**; **dexzd_takeover_duration_seconds** | | Recovery | **dexzd_recovery_last_duration_seconds**; **dexzd_recovery_last_replayed_bytes** | | Timers | **dexzd_timer_lag_seconds** | | Snapshots | **dexzd_snapshots_total** with **outcome**; **dexzd_snapshot_last_duration_seconds** | | Local disk | **dexzd_local_storage_bytes** | | Attribute Store sync | See [Monitoring](https://docs.superdurable.io/zh-Hans/reference/server/attribute-store-sync.md#attribute-store-monitoring) | | External search projection | See [Monitoring](https://docs.superdurable.io/zh-Hans/reference/server/search.md#search-projection-monitoring) | **rpc_class** is one of **flow_write**, **flow_read**, **search**, **worker_poll**, and **worker_respond**. **code** is the gRPC status code in snake case, such as **ok** or **unavailable**. **role** is one of **starting**, **standby**, **recovering**, **serving**, and **draining**. **state** is one of **inactive**, **bootstrapping**, **following**, **behind**, **restarting**, and **failed**. Only **following** is hot. The active process and a cold standby report **inactive**. **standby_mode** is **hot** or **cold**. **store_source** is **hot_standby** when the node serves the store that its hot standby kept current, and **restored** when it opened its own store or restored a snapshot, such as when the standby store was missing or stale. **dexzd_takeover_duration_seconds** covers hot takeovers only, by **store_source**. **reason** of **dexzd_step_attempts_returned_total** counts attempts that went back to pending without consuming a retry. It is one of **unsupported** (a Worker declined the Step type), **legacy_no_capacity** (an older SDK Worker declined for capacity), and **not_delivered** (the task never reached the Worker). **legacy_no_capacity** should drop to zero once every SDK is upgraded. **error** of **dexzd_worker_stream_protocol_errors_total** is one of **no_capacity_on_reserved_slot**, **half_close_after_output**, and **reject_after_output**. With **no_capacity_on_reserved_slot**, a reserved-slot Worker declined a task for capacity. The attempt fails as not run, and the stream ends. With **half_close_after_output**, a Worker closed its stream after output but before the final result. The attempt fails with NOT_COMPLETED. With **reject_after_output**, a Worker declined a task after it had already sent output for it. The attempt fails as possibly run, with NOT_COMPLETED. The server ends that stream with INVALID_ARGUMENT, so it gets no further task. **dexzd_matching_unsupported_type_evictions_total** is a counter. A steady rise means you should raise **matching.unsupportedTypeCacheEntries**. An evicted pair is not sent to its Worker again until that Worker completes a Step. While the cache overflows, a type that a Worker declined can therefore be offered to it again after each Step it completes. Raise **matching.unsupportedTypeCacheEntries** when this counter rises or the server logs the full-cache WARN. Worker polls use one persistent stream, so there is no empty-poll count. **dexzd_worker_stream_events_total** counts each stream close and, with **event="delivered"**, every task frame sent, including RPC task frames. A returned attempt that Dex hands out again counts again. Long-poll waits count toward **flow_read** latency. ```text # Failed API calls by class and code sum by (rpc_class, code) (rate(dexzd_api_requests_total{code!="ok"}[5m])) # A node that is not serving dexzd_node_role{role="serving"} == 0 # Failed lease renewals increase(dexzd_lease_renewals_total{outcome="error"}[10m]) > 0 ``` The full catalog, with types, histogram buckets, and Linux process metrics, is in the [server-zd README](https://github.com/superdurable/dex/blob/server-zd/v0.1.3/server-zd/README.md#metrics).