mirror of
https://github.com/pgsty/minio.git
synced 2026-08-09 15:53:28 +03:00
Feng Ruohang
2ff594f4bb
Add 'silo healthcheck [live|ready|cluster|cluster-read]', a thin anonymous HTTP client for the server's own /minio/health/* endpoints, so containers without a shell, curl, or mc can still run health checks. Design: silo.pgsty.com/compatibility/feature/healthcheck/ The check vocabulary maps 1:1 onto the health API paths; the probe target is derived from the server's own --address/MINIO_ADDRESS contract with HTTPS auto-detected from the certs directory, and can be overridden with --url. Exit codes are 0/1 only (Docker reserves 2); diagnostics (x-minio-server-status, quorum headers) go into a single output line for docker inspect. The request is strictly anonymous (a credentialed request would be rejected by the reserved-path guard), the transport bypasses HTTP_PROXY, and certificate verification is skipped to match kubelet HTTPS probe behavior. Cluster checks default to a 15s deadline so the server's 10s cluster_deadline can elapse. Compatibility notes: the preserved /minio/health/* path literals and the MINIO_ADDRESS env var are upstream wire/config surface, reused on purpose; the rebrand-guard baseline is regenerated for the new route literals (tests included) with zero new exported symbols. The docker entrypoint argv translation learns the new command name. Verified: unit tests, entrypoint tests, go vet, plus an end-to-end run against a live server covering all four checks, --maintenance (412), --json, usage errors, unreachable and timeout paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
123 lines
3.9 KiB
Markdown
123 lines
3.9 KiB
Markdown
# Silo Healthcheck
|
|
|
|
Silo server exposes three un-authenticated, healthcheck endpoints liveness probe and a cluster probe at `/minio/health/live` and `/minio/health/cluster` respectively.
|
|
|
|
## Native CLI probe
|
|
|
|
The `silo` binary can probe those endpoints itself, which makes health checking possible in containers that ship no shell, `curl`, or `mc`:
|
|
|
|
```
|
|
silo healthcheck [FLAGS] [live|ready|cluster|cluster-read]
|
|
```
|
|
|
|
The check name maps 1:1 onto `/minio/health/<path>`; `live` is the default. The exit code is `0` when healthy and `1` otherwise, and one diagnostic line (including the `x-minio-server-status` and quorum headers on failure) is printed for `docker inspect` to capture. The probe target is derived the same way the server derives its own listen address — `--address` / `MINIO_ADDRESS`, with HTTPS auto-detected from `public.crt` and `private.key` in `--certs-dir` — or overridden wholesale with `--url`. Certificate verification is skipped, matching the kubelet's behavior for HTTPS probes.
|
|
|
|
Use it as an image `HEALTHCHECK` (exec form, since there may be no shell):
|
|
|
|
```
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=2m --start-interval=2s --retries=3 \
|
|
CMD ["/usr/bin/silo", "healthcheck", "ready"]
|
|
```
|
|
|
|
or as a Docker Compose healthcheck:
|
|
|
|
```
|
|
healthcheck:
|
|
test: ["CMD", "/usr/bin/silo", "healthcheck", "ready"]
|
|
interval: 5s
|
|
timeout: 5s
|
|
retries: 5
|
|
```
|
|
|
|
`silo healthcheck --maintenance cluster` answers the pre-drain question documented below: exit `0` when the node can be taken down safely, exit `1` (HTTP 412) when doing so would lose HA. Keep the `cluster` checks out of per-container liveness probes — they reflect cluster-wide quorum, not this process.
|
|
|
|
## Liveness probe
|
|
|
|
This probe always responds with '200 OK'. Only fails if 'etcd' is configured and unreachable. When liveness probe fails, Kubernetes like platforms restart the container.
|
|
|
|
```
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /minio/health/live
|
|
port: 9000
|
|
scheme: HTTP
|
|
initialDelaySeconds: 120
|
|
periodSeconds: 30
|
|
timeoutSeconds: 10
|
|
successThreshold: 1
|
|
failureThreshold: 3
|
|
```
|
|
|
|
## Readiness probe
|
|
|
|
This probe always responds with '200 OK'. Only fails if 'etcd' is configured and unreachable. When readiness probe fails, Kubernetes like platforms turn-off routing to the container.
|
|
|
|
```
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /minio/health/ready
|
|
port: 9000
|
|
scheme: HTTP
|
|
initialDelaySeconds: 120
|
|
periodSeconds: 15
|
|
timeoutSeconds: 10
|
|
successThreshold: 1
|
|
failureThreshold: 3
|
|
```
|
|
|
|
## Cluster probe
|
|
|
|
### Cluster-writeable probe
|
|
|
|
The reply is '200 OK' if cluster has write quorum if not it returns '503 Service Unavailable'.
|
|
|
|
```
|
|
curl http://silo1:9001/minio/health/cluster
|
|
HTTP/1.1 503 Service Unavailable
|
|
Accept-Ranges: bytes
|
|
Content-Length: 0
|
|
Server: Silo
|
|
Vary: Origin
|
|
X-Amz-Bucket-Region: us-east-1
|
|
X-Minio-Write-Quorum: 3
|
|
X-Amz-Request-Id: 16239D6AB80EBECF
|
|
X-Xss-Protection: 1; mode=block
|
|
Date: Tue, 21 Jul 2020 00:36:14 GMT
|
|
```
|
|
|
|
### Cluster-readable probe
|
|
|
|
The reply is '200 OK' if cluster has read quorum if not it returns '503 Service Unavailable'.
|
|
|
|
```
|
|
curl http://silo1:9001/minio/health/cluster/read
|
|
HTTP/1.1 503 Service Unavailable
|
|
Accept-Ranges: bytes
|
|
Content-Length: 0
|
|
Server: Silo
|
|
Vary: Origin
|
|
X-Amz-Bucket-Region: us-east-1
|
|
X-Minio-Write-Quorum: 3
|
|
X-Amz-Request-Id: 16239D6AB80EBECF
|
|
X-Xss-Protection: 1; mode=block
|
|
Date: Tue, 21 Jul 2020 00:36:14 GMT
|
|
```
|
|
|
|
### Checking cluster health for maintenance
|
|
|
|
You may query the cluster probe endpoint to check if the node which received the request can be taken down for maintenance, if the server replies back '412 Precondition Failed' this means you will lose HA. '200 OK' means you are okay to proceed.
|
|
|
|
```
|
|
curl http://silo1:9001/minio/health/cluster?maintenance=true
|
|
HTTP/1.1 412 Precondition Failed
|
|
Accept-Ranges: bytes
|
|
Content-Length: 0
|
|
Server: Silo
|
|
Vary: Origin
|
|
X-Amz-Bucket-Region: us-east-1
|
|
X-Amz-Request-Id: 16239D63820C6E76
|
|
X-Xss-Protection: 1; mode=block
|
|
X-Minio-Write-Quorum: 3
|
|
Date: Tue, 21 Jul 2020 00:35:43 GMT
|
|
```
|