Feng Ruohang
87c621965d
After a restart a node's remote erasure drives that could not be connected during startup stay uninstalled until the next connectDisks pass, about 15 seconds later. During that window the liveness, readiness and both cluster probes answer 200, admin info shows every drive online and mcli ready agrees, because the cluster probes aggregate each peer's report of its own local drives rather than the drives this node has installed. A PUT through that node can still fail with 503 SlowDownWrite and a cross-node GET can answer 404 NoSuchKey until the window closes. The behaviour is inherited from upstream and reproduced on the 0806 and 0903 releases alike. Document the window and the bounded data-path check (PUT through each node, read each object through every node, fixed deadline, re-read acknowledged objects) that automation should use instead of the probes, and correct the readiness probe description, which also fails on request-queue overload and an unreachable KMS. No product change: the probes keep their documented purpose, and changing them or the reconnect cadence was judged unproven tuning in the agreed plan. Refs pgsty/silo#116 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe Signed-off-by: Feng Ruohang <rh@vonng.com>
Silo Monitoring Guide
Silo server exposes monitoring data over endpoints. Monitoring tools can pick the data from these endpoints. This document lists the monitoring endpoints and relevant documentation.
Healthcheck Probe
Silo server has two healthcheck related un-authenticated endpoints, a liveness probe to indicate if server is responding, cluster probe to check if server can be taken down for maintenance.
- Liveness probe available at
/minio/health/live - Cluster probe available at
/minio/health/cluster
Read more on how to use these endpoints in Silo healthcheck guide.
Prometheus Probe
Silo allows reading metrics for the entire cluster from any single node. This allows for metrics collection for a Silo instance across all servers. Thus, metrics collection for instances behind a load balancer can be done without any knowledge of the individual node addresses. The cluster wide metrics can be read at
<Address for Silo Service>/minio/v2/metrics/cluster.
The additional node specific metrics which include additional go metrics or process metrics are exposed at
<Address for Silo Node>/minio/v2/metrics/node.
The additional bucket specific metrics which include additional go metrics or process metrics are exposed at
<Address for Silo Node>/minio/v2/metrics/bucket.
The additional resource specific metrics which include additional go metrics or process metrics are exposed at
<Address for Silo Node>/minio/v2/metrics/resource.
To use this endpoint, setup Prometheus to scrape data from this endpoint. Read more on how to configure and use Prometheus to monitor Silo server in How to monitor Silo server with Prometheus.
Deprecated metrics monitoring
- Prometheus' data available at
/minio/prometheus/metricsis deprecated