Merge pull request #335 from botdeniska/fix/decom-keys-below-a-key

fix(decom): repeat the walk of a bucket while it removes source versions

Signed-off-by: Feng Ruohang <rh@vonng.com>

CHANGELOG.md

@@ -20,13 +20,26 @@
   `svcacct info` reports with `Policy: implied`, and set the policy again on
   those that were meant to be restricted.
 
+- S3 `DeleteObjects` on a single pool stops at an entry whose name fails the
+  object name check: the entry returns `XMinioInvalidObjectName` and is not
+  locked or sent to a drive, as with a single-object DELETE and with more than
+  one pool. Until now such an entry still reached the drives of the node, where
+  a name with a `..` segment was joined to the request bucket. A caller allowed
+  to delete in one bucket could remove a version in another bucket, including
+  one under Object Lock retention, and from a versioned bucket could put a
+  delete marker on an object of another bucket or on stored system metadata
+  such as a bucket's configuration. The other entries of a request are deleted
+  as before.
+
 ## RELEASE.2026-11-09T00-00-00Z
 
-Package version: `20261109000000.0.0`. Publication status and artifacts are
+Package version: `20261109000000.0.0`; published from source `d998292387cc`.
-listed in [GitHub Releases](https://github.com/pgsty/silo/releases). This
+Artifacts are listed in the [GitHub release](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z).
-section covers the changes after the published 20261026 source `c3ad836aed0d`.
+The coordinated [English release notes](https://silo.pgsty.com/blog/release/silo-20261109/)
-Most fixes come from reports and patches by
+and [Chinese release notes](https://silo.pgsty.com/zh/blog/release/silo-20261109/)
-[@botdeniska](https://github.com/botdeniska); see [CONTRIBUTORS.md](CONTRIBUTORS.md).
+cover all components, upgrade guidance, and validation scope since the
+published 20261026 source `c3ad836aed0d`. Most fixes come from reports and
+patches by [@botdeniska](https://github.com/botdeniska); see [CONTRIBUTORS.md](CONTRIBUTORS.md).
 
 Lifecycle, multi-pool and tiering:
 
@@ -41,7 +54,8 @@ Lifecycle, multi-pool and tiering:
   `retainVersions: 0`) on the expired key instead of the keys below it; a key
   with a tiered or Object Lock protected version is refused
   ([#297](https://github.com/pgsty/silo/issues/297)).
-- Stop taking the free version of a deleted tiered version for a version
+- Exclude the internal free-version marker of a deleted tiered version from
+  visible version history and lifecycle decisions
   ([#263](https://github.com/pgsty/silo/issues/263)), apply rules whose only
   action is `DelMarkerExpiration` ([#268](https://github.com/pgsty/silo/issues/268)),
   and report and account only the removals that happened

CONTRIBUTING.md

@@ -165,8 +165,11 @@ license grant to SILO or upstream MinIO maintainers is required.
 
 ### How does Silo manage dependencies?
 
-Silo uses Go modules. Preserve the compatibility module and import paths in
+Silo uses Go modules. Shared packages in `github.com/pgsty/silo-pkg/v3` are
-`go.mod`; downstream forks are selected with explicit `replace` directives.
+required and imported directly under their own module path, upstream
+`github.com/minio/minio-go/v7` is pinned to a verified commit, and Console
+(`github.com/minio/console`) and MC (`github.com/minio/mc`) are selected with
+explicit `replace` directives in `go.mod`.
 
 - Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
 

PULL_REQUESTS_ETIQUETTE.md

@@ -42,7 +42,8 @@ PRs must flow smoothly through review to reach production. Large PRs should be s
    - Document testing steps in the PR summary if manual testing was performed.
 
 4. **Before Submitting**:
-   - Run `make verify` to check formatting, linting, and tests.
+   - Sign off every commit under the Developer Certificate of Origin (`git commit -s`).
+   - Run `make verifiers` to check formatting, linting, code generation, and `rebrand-guard`.
    - Reference related issues (e.g., “Closes #1234”).
    - Notify team members via GitHub `@mentions` if urgent or complex.
 

README.md

@@ -36,7 +36,7 @@
 
 PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
 
-It follows one rule — **the product and its delivery surfaces are renamed; the protocol and your data are not.** Everything else lives on [silo.pgsty.com](https://silo.pgsty.com/).
+It follows one rule: **the product and its delivery surfaces are renamed; the protocol and your data are not.** Full documentation is published at [silo.pgsty.com](https://silo.pgsty.com/).
 
 **Related:** [`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
 
@@ -44,10 +44,10 @@ It follows one rule — **the product and its delivery surfaces are renamed; the
   <img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo Console">
 </p>
 
-#### SILO 20261109
+### SILO 20261109
 
 > [!WARNING]
-> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) fixes bugs and security vulnerabilities. We strongly recommend scheduling an upgrade to reduce security risks.
+> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) (`20261109000000.0.0`, source `d998292387cc`) fixes storage, lifecycle, native restore, Object Lock, batch authorization, and IAM defects, embedding Console `v2.7.0` with `mcli 20261109`, `silo-pkg v3.16.0`, and Go `1.27.3`. Review the [release notes and upgrade requirements](https://silo.pgsty.com/blog/release/silo-20261109/) before upgrading.
 
 > [!NOTE]
 > On 2026-08-06, `pgsty/minio` became `pgsty/silo`, and the default branch changed from `master` to `main`. The last MinIO-branded release is [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z) on the [`minio`](https://github.com/pgsty/silo/tree/minio) branch.
@@ -86,7 +86,7 @@ docker exec silo mcli mb local/demo && docker exec silo mcli ls local
 | Kubernetes | Helm chart, see [Download & Install](https://silo.pgsty.com/download/) |
 | Source | `go build -o silo . && ./silo --version` |
 
-Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/); migrating from upstream MinIO — taking over an existing `minio.service` and its `/etc/default/minio`, and keeping data ownership stable with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in — is covered by the [migration guide](https://silo.pgsty.com/compatibility/migration/) and the [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
+Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/). To migrate from upstream MinIO, take over `minio.service` and `/etc/default/minio`, and preserve data ownership with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in. See the [migration guide](https://silo.pgsty.com/compatibility/migration/) and [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
 
 ## Compatibility
 
@@ -98,7 +98,7 @@ Every divergence from upstream is listed in the code-verified [compatibility aud
 
 ### TLS and Go upgrades
 
-Server 20260916 restored Go's default key exchange; Server 20261018 retains it with Go 1.27.2.
+Server 20260916 restored Go's default key exchange; Server 20261109 retains it with Go 1.27.3.
 TLS key exchange follows Go's defaults across the S3 listener, node links,
 replication, identity providers, etcd, and external HTTP services. If an endpoint
 cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges

README_ZH.md

@@ -36,7 +36,7 @@
 
 上游停止社区发行后，Silo 为开源 MinIO 服务端维护一条持续可用的版本线：构建、软件包、多架构镜像、安全修复与完整 Web 控制台。Pigsty 在生产环境中用它承载 PostgreSQL 备份存储。
 
-它只遵循一条原则：**改名的是产品与交付物，不是协议与你的数据。** 其余内容都在 [silo.pgsty.com](https://silo.pgsty.com/zh/)。
+它只遵循一条原则：**改名的是产品与交付物，不是协议与你的数据。** 完整文档发布在 [silo.pgsty.com](https://silo.pgsty.com/zh/)。
 
 **相关项目：**[`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
 
@@ -44,10 +44,10 @@
   <img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo 控制台">
 </p>
 
-#### SILO 20261109
+### SILO 20261109
 
 > [!WARNING]
-> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) 修复了缺陷与安全漏洞，强烈建议尽快安排升级，以降低安全风险。
+> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z)（包版本 `20261109000000.0.0`，源码 `d998292387cc`）修复了多池生命周期、原生恢复、Object Lock、批处理授权与 IAM 等缺陷，内嵌 Console `v2.7.0` 并配套 `mcli 20261109`、`silo-pkg v3.16.0` 与 Go `1.27.3`。升级前请阅读[发布说明与升级要求](https://silo.pgsty.com/zh/blog/release/silo-20261109/)。
 
 > [!NOTE]
 > 2026-08-06，`pgsty/minio` 更名为 `pgsty/silo`，默认分支由 `master` 改为 `main`。最后一个 MinIO 品牌发布是 [`minio`](https://github.com/pgsty/silo/tree/minio) 分支上的 [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z)。
@@ -98,9 +98,9 @@ Silo 保留 S3 与存储格式兼容性，包括既有 `MINIO_*` 环境变量、
 
 ### TLS 与 Go 升级
 
-Server 20260916 已恢复 Go 默认密钥交换策略，20261018 在 Go 1.27.2 下保留该行为。
+Server 20260916 恢复了 Go 默认密钥交换策略，Server 20261109 在 Go 1.27.3 下保留该行为。S3 监听端口、节点间连接、站点与桶复制、身份提供方、etcd 及外部 HTTP 服务均使用 Go 默认 TLS 密钥交换。若对端无法接受 ML-KEM，设置 `GODEBUG=tlsmlkem=0` 可在当前进程禁用默认混合密钥交换（证书校验保持开启）；该选项不会禁用 ML-DSA 签名，也不能解决所有 TLS 连接重置问题，建议在升级不兼容对端后移除该临时配置。若仅新增的 SecP 混合组引发问题，可设置 `GODEBUG=tlssecpmlkem=0` 仅禁用这些组并保留 `X25519MLKEM768`。
-`GODEBUG=tlsmlkem=0`、`tlssecpmlkem=0` 的适用范围、macOS 根证书来源变化，
+
-以及 OIDC discovery 的诊断方法见 [Go 1.27 TLS 与 OIDC 指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
+在 Go 1.27 构建中，macOS 上设置 `SSL_CERT_FILE` 或 `SSL_CERT_DIR` 会以磁盘根证书和 Go 验证器替代系统钥匙串信任；陈旧或不完整的 CA 路径可能导致原先受信任的连接失败，清除继承的环境变量即可恢复钥匙串信任。配置的 `CAs` 目录中的显式证书仍会叠加加入所选根证书池。Go 1.27 二进制要求 macOS 13 或更高版本。详见 [Go 1.27 发行说明](https://go.dev/doc/go1.27)与 [Go 1.27 TLS 与 OIDC 发现指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
 
 ## 文档归属
 

SECURITY.md

@@ -38,7 +38,7 @@ For vulnerabilities in this fork:
 
 1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
 2. Prefer this repository's [private GitHub security advisory](https://github.com/pgsty/silo/security/advisories/new) workflow.
-3. If you cannot use GitHub private reporting, email [ron@pgsty.com](mailto:ron@pgsty.com) or rh@vonng.com
+3. If you cannot use GitHub private reporting, email [ron@pgsty.com](mailto:ron@pgsty.com) or [rh@vonng.com](mailto:rh@vonng.com)
    directly. Do not publish vulnerability details or exploit code in public
    issues or pull requests.
 4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.

cmd/batch-authz-review6_test.go

@@ -330,7 +330,7 @@ func TestBatchReviewRotationKeepsMigrationCleanup(t *testing.T) {
 				t.Errorf("rotation changed a draining source after its version was copied (rotation error: %v)", rotationErr)
 			}
 			snapshot := []FileInfo{{VersionID: before.VersionID, ModTime: beforeInfo.ModTime}}
-			if err := z.removeMovedVersions(t.Context(), bucket, object, 0, snapshot, nil); err != nil {
+			if _, err := z.removeMovedVersions(t.Context(), bucket, object, 0, snapshot, nil); err != nil {
 				t.Fatalf("rotation prevented migration cleanup: %v", err)
 			}
 			if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: before.VersionID}); !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {

cmd/bulk-delete-invalid-name_test.go

@@ -0,0 +1,156 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"bytes"
+	"encoding/xml"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"testing"
+
+	"github.com/minio/minio/internal/auth"
+)
+
+// A bulk delete joins each name to the request bucket on the drives, so a name
+// with a ".." segment names a path in another bucket or in the meta bucket.
+// Such a name is the error of its own entry and reaches no drive, and the
+// other entries of the request are deleted as usual. This holds for the one
+// bulk lock of a single node, for a pool that locks per set, and for more than
+// one pool.
+func TestPoolsBulkDeleteInvalidNames(t *testing.T) {
+	for _, topology := range []struct {
+		name string
+		new  func(t *testing.T) (*erasureServerPools, string)
+	}{
+		{"single node", func(t *testing.T) (*erasureServerPools, string) { return bulkDeletePools(t, 1) }},
+		{"set locks", func(t *testing.T) (*erasureServerPools, string) { return ldPool(t, ldLocal) }},
+		{"two pools", func(t *testing.T) (*erasureServerPools, string) { return bulkDeletePools(t, 2) }},
+	} {
+		t.Run(topology.name, func(t *testing.T) {
+			z, bucket := topology.new(t)
+			ctx := t.Context()
+			other := "bulk-delete-other"
+			if err := z.MakeBucket(ctx, other, MakeBucketOptions{VersioningEnabled: true}); err != nil {
+				t.Fatal(err)
+			}
+			old := batchPutVersion(t, z, other, "kept", nil)
+			latest := batchPutVersion(t, z, other, "kept", nil)
+			batchPutVersion(t, z, other, "visible", nil)
+			metaKey := pathJoin(bucketMetaPrefix, other, bucketMetadataFile)
+			if !batchVersionExists(t, z, minioMetaBucket, metaKey, "") {
+				t.Fatal("the bucket metadata is not an object of the meta bucket")
+			}
+			for _, key := range []string{"own", "dir/"} {
+				if _, err := z.PutObject(ctx, bucket, key, mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""), ObjectOptions{}); err != nil {
+					t.Fatal(err)
+				}
+			}
+
+			names := []ObjectToDelete{
+				{ObjectV: ObjectV{ObjectName: "../" + other + "/kept", VersionID: old.VersionID}}, // purge in another bucket
+				{ObjectV: ObjectV{ObjectName: "own"}},
+				{ObjectV: ObjectV{ObjectName: "a/../../" + other + "/visible"}},         // marker in another bucket
+				{ObjectV: ObjectV{ObjectName: "../" + minioMetaBucket + "/" + metaKey}}, // marker on bucket metadata
+				{ObjectV: ObjectV{ObjectName: "dir/"}},
+				{ObjectV: ObjectV{ObjectName: ""}},
+			}
+			invalid := map[int]bool{0: true, 2: true, 3: true, 5: true}
+			// Without a version ID these options ask for a delete marker, the
+			// one write a bulk delete makes.
+			deleted, errs := z.DeleteObjects(ctx, bucket, slices.Clone(names), ObjectOptions{Versioned: true})
+			for i, name := range names {
+				_, rejected := errs[i].(ObjectNameInvalid)
+				if rejected != invalid[i] || !invalid[i] && errs[i] != nil {
+					t.Errorf("%d %q: error = %v, rejected as invalid = %v, want %v", i, name.ObjectName, errs[i], rejected, invalid[i])
+				}
+				if deleted[i].ObjectName != name.ObjectName || deleted[i].found == invalid[i] {
+					t.Errorf("%d %q: result = %+v", i, name.ObjectName, deleted[i])
+				}
+			}
+
+			for _, version := range []string{old.VersionID, latest.VersionID} {
+				if !batchVersionExists(t, z, other, "kept", version) {
+					t.Errorf("version %q of the object in the other bucket was removed", version)
+				}
+			}
+			if !batchVersionExists(t, z, other, "visible", "") {
+				t.Error("the object of the other bucket is hidden by a marker")
+			}
+			if !batchVersionExists(t, z, minioMetaBucket, metaKey, "") {
+				t.Error("the metadata of the other bucket is hidden by a marker")
+			}
+			for _, key := range []string{"own", "dir/"} {
+				if batchVersionExists(t, z, bucket, key, "") {
+					t.Errorf("%q of the request bucket was not deleted", key)
+				}
+			}
+		})
+	}
+}
+
+// The names of an S3 DeleteObjects request come from its body, which the path
+// checks of the request URL do not see. A user allowed to delete in one bucket
+// reaches nothing outside it through such a name: a version under COMPLIANCE
+// retention in another bucket stays, and no delete marker lands on that object
+// or on the stored metadata of its bucket.
+func TestBulkDeleteObjectLockNameOutsideBucket(t *testing.T) {
+	defer DetectTestLeak(t)()
+	ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
+		t: t, endpoints: []string{"DeleteMultipleObjects"},
+		makeBucketOptions: MakeBucketOptions{VersioningEnabled: true},
+		objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, _ auth.Credentials, t *testing.T) {
+			ctx := t.Context()
+			user := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:DeleteObject","s3:DeleteObjectVersion"`)
+			locked := getRandomBucketName()
+			if err := obj.MakeBucket(ctx, locked, MakeBucketOptions{VersioningEnabled: true, LockEnabled: true}); err != nil {
+				t.Fatal(err)
+			}
+			enableBucketObjectLock(t, locked)
+			worm := batchPutVersion(t, obj, locked, "secret", batchLockMeta("COMPLIANCE"))
+			batchPutVersion(t, obj, bucket, "own", nil)
+			metaKey := pathJoin(bucketMetaPrefix, locked, bucketMetadataFile)
+
+			outside := []ObjectToDelete{
+				{ObjectV: ObjectV{ObjectName: "../" + locked + "/secret", VersionID: worm.VersionID}},
+				{ObjectV: ObjectV{ObjectName: "../" + locked + "/secret"}},
+				{ObjectV: ObjectV{ObjectName: "../" + minioMetaBucket + "/" + metaKey}},
+			}
+			body := encodeResponse(DeleteObjectsRequest{Objects: append(slices.Clone(outside), ObjectToDelete{ObjectV: ObjectV{ObjectName: "own"}})})
+			req, err := newTestSignedRequestV4(http.MethodPost, getDeleteMultipleObjectsURL("", bucket), int64(len(body)), bytes.NewReader(body),
+				user.AccessKey, user.SecretKey, nil)
+			if err != nil {
+				t.Fatal(err)
+			}
+			rec := httptest.NewRecorder()
+			router.ServeHTTP(rec, req)
+			var resp DeleteObjectsResponse
+			if rec.Code != http.StatusOK || xml.Unmarshal(rec.Body.Bytes(), &resp) != nil {
+				t.Fatalf("%s: status %d: %s", instance, rec.Code, rec.Body.String())
+			}
+
+			if len(resp.DeletedObjects) != 1 || resp.DeletedObjects[0].ObjectName != "own" || len(resp.Errors) != len(outside) {
+				t.Errorf("%s: want the three names outside the bucket refused and \"own\" deleted: %s", instance, rec.Body.String())
+			}
+			for _, e := range resp.Errors {
+				if e.Code != errorCodes[ErrInvalidObjectName].Code {
+					t.Errorf("%s: %q: code %s, want %s", instance, e.Key, e.Code, errorCodes[ErrInvalidObjectName].Code)
+				}
+			}
+			if !batchVersionExists(t, obj, locked, "secret", worm.VersionID) {
+				t.Errorf("%s: the version under COMPLIANCE retention in the other bucket was removed", instance)
+			}
+			if !batchVersionExists(t, obj, locked, "secret", "") {
+				t.Errorf("%s: a delete marker hides the object of the other bucket", instance)
+			}
+			if !batchVersionExists(t, obj, minioMetaBucket, metaKey, "") {
+				t.Errorf("%s: a delete marker hides the stored metadata of the other bucket", instance)
+			}
+			if batchVersionExists(t, obj, bucket, "own", "") {
+				t.Errorf("%s: the object of the request bucket was not deleted", instance)
+			}
+		},
+	})
+}

cmd/erasure-server-pool-consistency.go

@@ -271,16 +271,17 @@ func (z *erasureServerPools) applyMigrationExpiry(ctx context.Context, bucket, o
 // in the source until the ordinary expiry path has deleted them: that path
 // owns replication state and tier cleanup, and it re-evaluates the current
 // lifecycle rule. A migration cleanup must never turn an asynchronous ILM
-// decision into an unconditional physical purge.
+// decision into an unconditional physical purge. It returns how many source
-func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) error {
+// versions it deleted.
+func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) (removed int, err error) {
 	if err := z.applyMigrationExpiry(ctx, bucket, object, source, expired); err != nil {
-		return err
+		return 0, err
 	}
 	object = encodeDirObject(object)
 	lk := z.NewNSLock(bucket, object)
 	lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
 	if err != nil {
-		return err
+		return 0, err
 	}
 	ctx = lkctx.Context()
 	defer lk.Unlock(lkctx)
@@ -294,7 +295,7 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
 			continue
 		}
 		if err != nil {
-			return err
+			return removed, err
 		}
 		var src *ObjectInfo
 		var retained []PoolObjInfo
@@ -309,7 +310,7 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
 			continue
 		}
 		if !src.ModTime.Equal(version.ModTime) || src.DeleteMarker != version.Deleted {
-			return PreConditionFailed{}
+			return removed, PreConditionFailed{}
 		}
 		if isExpiredVersion(expired, version) {
 			// The logical expiry path owns this version. Keeping the source is
@@ -319,17 +320,20 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
 		// Successful movement is not enough if the destination changed again.
 		// Keep the source unless another copy still preserves its read result.
 		if len(retained) == 0 || retained[0].ObjInfo.ModTime.Before(src.ModTime) {
-			return PreConditionFailed{}
+			return removed, PreConditionFailed{}
 		}
 		_, err = z.serverPools[source].DeleteObject(ctx, bucket, object, ObjectOptions{
 			VersionID: vid, NoLock: true, NoAuditLog: true,
 			SkipFreeVersion: true, // preserve the old prefix cleanup's tier-GC behavior
 		})
 		if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
-			return err
+			return removed, err
+		}
+		if err == nil {
+			removed++
 		}
 	}
-	return nil
+	return removed, nil
 }
 
 // ilmDeletesVersionOnly reports whether the ILM action queued for a version

cmd/erasure-server-pool-decom-keys-below_test.go

@@ -0,0 +1,66 @@
+// Copyright (c) 2026 Denis Babchuk
+//
+// This file is part of MinIO Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import "testing"
+
+// A key below another key ("k/b" under "k") is a directory inside the
+// directory of that key. The listing treats the directory of an object as a
+// leaf, so a decommission walk does not see "k/b" while "k" is still in the
+// pool. The walk has to be repeated once the keys above are gone, or the
+// check after the bucket finds them and the decommission fails.
+
+func keysBelowDecommission(t *testing.T, z *erasureServerPools) PoolDecommissionInfo {
+	t.Helper()
+	if err := z.StartDecommission(t.Context(), 0); err != nil {
+		t.Fatal(err)
+	}
+	z.doDecommissionInRoutine(t.Context(), 0)
+	z.poolMetaMutex.RLock()
+	defer z.poolMetaMutex.RUnlock()
+	return *z.poolMeta.Pools[0].Decommission
+}
+
+func keysBelowPools(t *testing.T) (*erasureServerPools, string, []string) {
+	t.Helper()
+	z, _ := migrationTestPools(t)
+	const bucket = "keys-below"
+	if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
+		t.Fatal(err)
+	}
+	keys := []string{"k", "k/b", "k/b/c"}
+	for _, key := range keys {
+		putConsistencyObject(t, z, bucket, key, 0, key, ObjectOptions{Versioned: true})
+	}
+	return z, bucket, keys
+}
+
+func TestDecommissionMovesKeysBelowAKey(t *testing.T) {
+	z, bucket, keys := keysBelowPools(t)
+	if info := keysBelowDecommission(t, z); !info.Complete || info.Failed {
+		t.Errorf("decommission: complete %v, failed %v; want complete", info.Complete, info.Failed)
+	}
+	for _, key := range keys {
+		if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, key, ObjectOptions{}); !isErrObjectNotFound(err) {
+			t.Errorf("%s still in pool 0: %v", key, err)
+		}
+		if body := migrationTestBody(t, z.GetObjectNInfo, bucket, key, ""); body != key {
+			t.Errorf("%s reads %q", key, body)
+		}
+	}
+}

cmd/erasure-server-pool-decom.go

@@ -28,6 +28,7 @@ import (
 	"slices"
 	"sort"
 	"strings"
+	"sync/atomic"
 	"time"
 
 	"github.com/dustin/go-humanize"
@@ -752,7 +753,25 @@ func (set *erasureObjects) listObjectsToDecommission(ctx context.Context, bi dec
 	return err
 }
 
+// decommissionPool moves bucket bi out of pool idx. A key below another key
+// ("k/b" under "k") is a directory inside the directory of that key, and the
+// listing does not descend into the directory of an object: a walk sees "k/b"
+// only once "k" has left the pool. The walk is repeated while it removes
+// source versions, so each pass moves the keys that the previous one
+// uncovered. Each repeat needs a version removed from the pool, which takes
+// no new writes, so the repeats end.
 func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) error {
+	for {
+		removed, err := z.decommissionPoolWalk(ctx, idx, pool, bi)
+		if err != nil || removed == 0 || contextCanceled(ctx) {
+			return err
+		}
+	}
+}
+
+// decommissionPoolWalk walks bucket bi of pool idx once and returns the number
+// of keys whose source versions it removed.
+func (z *erasureServerPools) decommissionPoolWalk(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) (int64, error) {
 	ctx = logger.SetReqInfo(ctx, &logger.ReqInfo{})
 
 	const envDecomWorkers = "_MINIO_DECOMMISSION_WORKERS"
@@ -766,9 +785,11 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 	// add that many extra workers.
 	workerSize += len(pool.sets)
 
+	var removed atomic.Int64
+
 	wk, err := workers.New(workerSize)
 	if err != nil {
-		return err
+		return 0, err
 	}
 
 	var vc *versioning.Versioning
@@ -779,24 +800,24 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 	if bi.Name != minioMetaBucket {
 		vc, err = globalBucketVersioningSys.Get(bi.Name)
 		if err != nil {
-			return err
+			return 0, err
 		}
 
 		// Check if the current bucket has a configured lifecycle policy
 		lc, lifecycleUpdatedAt, err = globalBucketMetadataSys.GetLifecycleConfig(bi.Name)
 		if err != nil && !errors.Is(err, BucketLifecycleNotFound{Bucket: bi.Name}) {
-			return err
+			return 0, err
 		}
 
 		// Check if bucket is object locked.
 		lr, err = globalBucketObjectLockSys.Get(bi.Name)
 		if err != nil {
-			return err
+			return 0, err
 		}
 
 		rcfg, err = getReplicationConfig(ctx, bi.Name)
 		if err != nil {
-			return err
+			return 0, err
 		}
 	}
 
@@ -1019,7 +1040,10 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 			// if all versions were decommissioned, then we can delete the object versions.
 			if decommissioned == len(fivs.Versions) {
 				stopFn := globalDecommissionMetrics.log(decomMetricDecommissionRemoveObject, idx, bi.Name, entry.name)
-				err := z.removeMovedVersions(ctx, bi.Name, entry.name, idx, fivs.Versions, expired)
+				n, err := z.removeMovedVersions(ctx, bi.Name, entry.name, idx, fivs.Versions, expired)
+				if n > 0 {
+					removed.Add(1)
+				}
 				if err == nil {
 					err = z.removeFreeVersions(ctx, bi.Name, entry.name, idx, fivs.FreeVersions)
 				}
@@ -1062,7 +1086,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 		}(setIdx)
 	}
 	wk.Wait()
-	return nil
+	return removed.Load(), nil
 }
 
 //msgp:ignore decomMetrics

cmd/erasure-server-pool-lifecycle-authority_test.go

@@ -20,6 +20,7 @@ package cmd
 import (
 	"context"
 	"crypto/sha256"
+	"encoding/binary"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -29,6 +30,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/cespare/xxhash/v2"
 	"github.com/google/uuid"
 	"github.com/minio/madmin-go/v3"
 	"github.com/minio/minio/internal/bucket/lifecycle"
@@ -632,6 +634,176 @@ func TestPoolsLifecycleLegacyHistory(t *testing.T) {
 	}
 }
 
+// authoritySignature2021 is the header signature of o, a version whose part
+// ETags are all empty, as the releases from RELEASE.2021-11-24 until
+// RELEASE.2022-01-25 computed it: with these ETags.
+func authoritySignature2021(o xlMetaV2Object) (sig [4]byte) {
+	o.ErasureIndex = 0
+	crc := hashDeterministicString(o.MetaUser) ^ hashDeterministicBytes(o.MetaSys)
+	o.MetaUser, o.MetaSys = nil, nil
+	bts, _ := o.MarshalMsg(nil)
+	crc ^= xxhash.Sum64(bts)
+	binary.LittleEndian.PutUint32(sig[:], uint32(crc^(crc>>32)))
+	return sig
+}
+
+// A drive holds the header that the release which wrote a version there
+// stored. Before RELEASE.2024-04-28 it has no erasure coding, and until
+// RELEASE.2022-01-25 its signature counted the empty part ETags of versions
+// written before RELEASE.2021-11-24. A heal writes the header of its own
+// release and leaves the other drives alone, so the drives of an old version
+// differ there for good once some were replaced. That is one revision, as
+// reads and heal take it, on however many of the drives.
+func TestPoolsLifecycleHeaderEras(t *testing.T) {
+	id := [16]byte{1}
+	current := func(drive int) xlMetaV2ShallowVersion { return authorityRevision(t, id, 1, drive+1, "") }
+	// stored returns the version as drive holds it after change, under the
+	// header that header derives from the one a current release writes.
+	stored := func(drive int, change func(*xlMetaV2Object), header func(xlMetaV2VersionHeader, xlMetaV2Object) xlMetaV2VersionHeader) xlMetaV2ShallowVersion {
+		ver := current(drive)
+		var v xlMetaV2Version
+		if _, err := v.UnmarshalMsg(ver.meta); err != nil {
+			t.Fatal(err)
+		}
+		change(v.ObjectV2)
+		meta, err := v.MarshalMsg(nil)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return xlMetaV2ShallowVersion{header: header(ver.header, *v.ObjectV2), meta: meta}
+	}
+	eras := map[string]func(drive int) xlMetaV2ShallowVersion{
+		"no-erasure-coding": func(drive int) xlMetaV2ShallowVersion {
+			return stored(drive, func(*xlMetaV2Object) {}, func(h xlMetaV2VersionHeader, _ xlMetaV2Object) xlMetaV2VersionHeader {
+				h.EcN, h.EcM = 0, 0
+				return h
+			})
+		},
+		"empty-part-etags-signed": func(drive int) xlMetaV2ShallowVersion {
+			return stored(drive, func(o *xlMetaV2Object) { o.PartETags = []string{""} }, func(h xlMetaV2VersionHeader, o xlMetaV2Object) xlMetaV2VersionHeader {
+				h.EcN, h.EcM, h.Signature = 0, 0, authoritySignature2021(o)
+				return h
+			})
+		},
+	}
+	metaVs := slices.Repeat([]uint8{xlMetaVersion}, 16)
+	readQuorum := func(xlMetaV2ShallowVersion, uint8) int { return 12 }
+	for name, old := range eras {
+		t.Run(name, func(t *testing.T) {
+			if old(0).header == current(0).header {
+				t.Fatal("fixture: the old header is the current one")
+			}
+			if versionRevision(old(0), xlMetaVersion) != versionRevision(current(7), xlMetaVersion) {
+				t.Error("one version under the headers of two releases has two keys")
+			}
+			// The header of the key orders the merge: it is the current one.
+			if versionRevision(old(0), xlMetaVersion).header != current(0).header {
+				t.Error("the key does not carry the header of the current release")
+			}
+			for healed := range 17 {
+				metas := make([][]xlMetaV2ShallowVersion, 16)
+				for i := range metas {
+					if i < healed {
+						metas[i] = []xlMetaV2ShallowVersion{current(i)}
+					} else {
+						metas[i] = []xlMetaV2ShallowVersion{old(i)}
+					}
+				}
+				merged, err := authoritativeHistory(t.Context(), metas, metaVs, xlMetaVersion, 9, readQuorum)
+				if err != nil || len(merged) != 1 {
+					t.Errorf("%d healed drives: %d versions, err=%v", healed, len(merged), err)
+				}
+			}
+		})
+	}
+	// Control: the erasure coding still tells revisions apart, by the
+	// metadata. Another one under the same header is another revision.
+	other := stored(0, func(o *xlMetaV2Object) { o.ErasureM, o.ErasureN = 8, 8 }, func(h xlMetaV2VersionHeader, _ xlMetaV2Object) xlMetaV2VersionHeader { return h })
+	if versionRevision(other, xlMetaVersion) == versionRevision(current(0), xlMetaVersion) {
+		t.Error("another erasure coding under one header keeps the revision")
+	}
+}
+
+// The same on drives, through a heal: two versions with headers from before
+// RELEASE.2024-04-28 on a set where half of the drives were replaced. GET
+// reads them and heal has nothing left to do. Their expiry has to run as
+// well: of a noncurrent version with several pools, and of all versions of a
+// key with one pool too, which reads the history like the pools do. A
+// restore takes the drives for one generation, not for a failed status write.
+func TestPoolsLifecycleHealedOldHeaders(t *testing.T) {
+	for _, single := range []bool{false, true} {
+		name, key := "noncurrent-version", "ncv-old-headers"
+		if single {
+			name, key = "all-versions-one-pool", "eav-old-headers"
+		}
+		t.Run(name, func(t *testing.T) {
+			s := newSplitILM(t, versioning.Enabled, splitILMRules)
+			ctx := t.Context()
+			s.write(key, splitAt("old", 5, 0))
+			s.write(key, splitAt("new", 3, 0))
+			ids := []string{s.ids[key+" old"], s.ids[key+" new"]}
+			if single {
+				prior := s.z.serverPools
+				t.Cleanup(func() { s.z.serverPools = prior })
+				s.z.serverPools = s.z.serverPools[:1]
+			}
+			set := s.z.serverPools[0].getHashedSet(key)
+			disks := set.getDisks()
+			for _, disk := range disks {
+				rewriteTierTestMeta(t, disk, s.bucket, key, func(xl *xlMetaV2) {
+					for i := range xl.versions {
+						// Control: the key of a version this release wrote
+						// carries its header as stored.
+						if versionRevision(xl.versions[i], xl.metaV).header != xl.versions[i].header {
+							t.Fatal("the key of a current version carries another header")
+						}
+						xl.versions[i].header.EcN, xl.versions[i].header.EcM = 0, 0
+					}
+				})
+			}
+			replaced := len(disks) / 2
+			for _, disk := range disks[:replaced] {
+				if err := disk.Delete(ctx, s.bucket, key, DeleteOptions{Recursive: true, Immediate: true}); err != nil {
+					t.Fatal(err)
+				}
+				for _, id := range ids {
+					if _, err := s.z.HealObject(ctx, s.bucket, key, id, madmin.HealOpts{ScanMode: madmin.HealNormalScan}); err != nil {
+						t.Fatalf("heal %s: %v", id, err)
+					}
+				}
+			}
+			for i, disk := range disks {
+				if authorityHeader(t, disk, s.bucket, key, ids[0]).hasEC() != (i < replaced) {
+					t.Fatalf("fixture: drive %d of %d, %d replaced, holds the other header", i, len(disks), replaced)
+				}
+			}
+			authorityPresent(t, s, 0, key, ids[0])
+			res, err := s.z.HealObject(ctx, s.bucket, key, ids[0], madmin.HealOpts{DryRun: true, ScanMode: madmin.HealNormalScan})
+			if err != nil || slices.ContainsFunc(res.Before.Drives, func(d madmin.HealDriveInfo) bool { return d.State != madmin.DriveStateOk }) {
+				t.Fatalf("fixture: heal has work left: %v", err)
+			}
+
+			if gen, err := set.readRestoreGeneration(ctx, s.bucket, key, ids[0]); err != nil || gen.split {
+				t.Errorf("restore generation: split=%v, err=%v", gen.split, err)
+			}
+			if single {
+				_, err = s.z.DeleteObject(ctx, s.bucket, key, authorityDeleteAll)
+			} else {
+				err = authorityExpireVersion(t, s, key, ids[0])
+			}
+			if err != nil {
+				t.Fatalf("expiry: %v", err)
+			}
+			if _, err := s.z.GetObjectInfo(ctx, s.bucket, key, ObjectOptions{VersionID: ids[0]}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
+				t.Errorf("the expired version: %v", err)
+			}
+			if !single {
+				authorityPresent(t, s, 0, key, ids[1])
+			}
+		})
+	}
+}
+
 // authorityDeleteAll is the expiry of all versions of a key as the expiry
 // worker sends it: a purge of all versions, approved by the lifecycle.
 var authorityDeleteAll = ObjectOptions{DeleteAllVersions: true, Expiration: ExpirationOptions{Expire: true}}
@@ -1494,7 +1666,7 @@ func TestPoolsMigrationLifecycleRefusedExpiryKeepsSource(t *testing.T) {
 		t.Fatal(err)
 	}
 	version := FileInfo{Name: key, VersionID: id, ModTime: source.ModTime}
-	err = s.z.removeMovedVersions(ctx, s.bucket, key, 0, []FileInfo{version}, []migrationExpiredVersion{{
+	_, err = s.z.removeMovedVersions(ctx, s.bucket, key, 0, []FileInfo{version}, []migrationExpiredVersion{{
 		version:            version,
 		event:              lifecycle.Event{Action: lifecycle.DeleteAction},
 		lifecycleUpdatedAt: updatedAt,

cmd/erasure-server-pool-lifecycle.go

@@ -187,9 +187,17 @@ type revisionKey struct {
 // input is canonical: each map, including the part checksums of a legacy
 // version, is its own section with its entry count and sorted,
 // length-prefixed entries, since msgp encodes maps in iteration order and an
-// entry must not move between maps unnoticed. The header of a
+// entry must not move between maps unnoticed.
-// legacy version carries a signature over its per-shard checksums, so the
+//
-// key replaces it with one from the digest.
+// A drive holds the header that the release which wrote the version there
+// stored, and a heal writes the one of its own release: headers from before
+// RELEASE.2024-04-28 carry no erasure coding, and the signature counted
+// empty part ETags until RELEASE.2022-01-25. Reads and heal take such drives
+// for one version (matchesNotStrict), and nothing rewrites them. The key has
+// both as the current release computes them from the metadata, so that the
+// header of a version it wrote stays as stored: the signature also orders
+// versions of one time in the merge. The signature of a legacy version is
+// over its per-shard checksums; the key has one from the digest.
 func versionRevision(ver xlMetaV2ShallowVersion, metaV uint8) revisionKey {
 	h := sha256.New()
 	writeUint := func(n int) {
@@ -267,8 +275,12 @@ func versionRevision(ver xlMetaV2ShallowVersion, metaV uint8) revisionKey {
 			section("fields", 1)
 			writeField(bts)
 			h.Sum(key.digest[:0])
-			if v.Type == LegacyType {
+			switch v.Type {
+			case LegacyType:
 				copy(key.header.Signature[:], key.digest[:])
+			case ObjectType:
+				key.header.Signature = v.ObjectV2.Signature()
+				key.header.EcN, key.header.EcM = uint8(v.ObjectV2.ErasureN), uint8(v.ObjectV2.ErasureM)
 			}
 			return key
 		}
@@ -350,7 +362,7 @@ func authoritativeHistory(ctx context.Context, metas [][]xlMetaV2ShallowVersion,
 	// its count, and one that shares the header could stand in for the
 	// accepted one, so the merge only sees the accepted revision of each
 	// version and free versions as they are. It sees the headers of the
-	// keys, which do not differ between the drives of a legacy version.
+	// keys, which do not differ between the drives of a revision.
 	filtered := make([][]xlMetaV2ShallowVersion, len(metas))
 	for i, versions := range metas {
 		for j, ver := range versions {

cmd/erasure-server-pool-migration-delete_test.go

@@ -132,7 +132,7 @@ func TestPoolsMigrationCleanupRechecksSource(t *testing.T) {
 				_, err := z.serverPools[1].DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: nullVersionID})
 				mustIAM(t, err)
 			}
-			err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot, nil)
+			_, err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot, nil)
 			if change != "delete-uuid" && err == nil {
 				t.Fatal("cleanup accepted changed or uncertain state")
 			}

cmd/erasure-server-pool-rebalance.go

@@ -779,7 +779,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
 			// if all versions were rebalanced, we can delete the object versions.
 			if rebalanced == len(fivs.Versions) {
 				stopFn := globalRebalanceMetrics.log(rebalanceMetricRebalanceRemoveObject, poolIdx, bucket, entry.name)
-				err := z.removeMovedVersions(ctx, bucket, entry.name, poolIdx, fivs.Versions, expired)
+				_, err := z.removeMovedVersions(ctx, bucket, entry.name, poolIdx, fivs.Versions, expired)
 				stopFn(0, err)
 				auditLogRebalance(ctx, "Rebalance:DeleteObject", bucket, entry.name, "", err)
 				if err != nil {

cmd/erasure-server-pool.go

@@ -26,6 +26,7 @@ import (
 	"math/rand"
 	"net/http"
 	"path"
+	"slices"
 	"sort"
 	"strconv"
 	"strings"
@@ -1543,6 +1544,27 @@ func (z *erasureServerPools) DeleteObjects(ctx context.Context, bucket string, o
 		objSets.Add(objects[i].ObjectName)
 	}
 
+	// A name that fails the check keeps its error and goes no further. Joined
+	// to the bucket it can name a path in another bucket or in the meta
+	// bucket, so it is not locked, looked up or sent to a drive. The other
+	// names are deleted as a request of their own: everything below sees only
+	// names that passed.
+	if slices.ContainsFunc(derrs, func(err error) bool { return err != nil }) {
+		valid, at := make([]ObjectToDelete, 0, len(objects)), make([]int, 0, len(objects))
+		for i, obj := range objects {
+			if derrs[i] == nil {
+				valid, at = append(valid, obj), append(at, i)
+			}
+		}
+		if len(valid) > 0 {
+			deleted, errs := z.DeleteObjects(ctx, bucket, valid, opts)
+			for j, i := range at {
+				dobjects[i], derrs[i] = deleted[j], errs[j]
+			}
+		}
+		return dobjects, derrs
+	}
+
 	// A single pool locks each name in the set its writers lock it in, one
 	// set at a time. A single node keeps one lock table for all sets, so it
 	// keeps the one bulk lock below: its local lock takes the names one by

docs/metrics/healthcheck/README.md

@@ -1,6 +1,6 @@
 # Silo Healthcheck
 
-Silo server exposes three un-authenticated, healthcheck endpoints liveness probe and a cluster probe at `/minio/health/live` and `/minio/health/cluster` respectively.
+Silo server exposes four unauthenticated healthcheck endpoints: `/minio/health/live`, `/minio/health/ready`, `/minio/health/cluster`, and `/minio/health/cluster/read`.
 
 ## Native CLI probe
 
@@ -76,7 +76,7 @@ readinessProbe:
 The reply is '200 OK' if cluster has write quorum if not it returns '503 Service Unavailable'.
 
 ```
-curl http://silo1:9001/minio/health/cluster
+curl http://silo1:9000/minio/health/cluster
 HTTP/1.1 503 Service Unavailable
 Accept-Ranges: bytes
 Content-Length: 0
@@ -94,7 +94,7 @@ Date: Tue, 21 Jul 2020 00:36:14 GMT
 The reply is '200 OK' if cluster has read quorum if not it returns '503 Service Unavailable'.
 
 ```
-curl http://silo1:9001/minio/health/cluster/read
+curl http://silo1:9000/minio/health/cluster/read
 HTTP/1.1 503 Service Unavailable
 Accept-Ranges: bytes
 Content-Length: 0
@@ -112,7 +112,7 @@ Date: Tue, 21 Jul 2020 00:36:14 GMT
 You may query the cluster probe endpoint to check if the node which received the request can be taken down for maintenance, if the server replies back '412 Precondition Failed' this means you will lose HA. '200 OK' means you are okay to proceed.
 
 ```
-curl http://silo1:9001/minio/health/cluster?maintenance=true
+curl http://silo1:9000/minio/health/cluster?maintenance=true
 HTTP/1.1 412 Precondition Failed
 Accept-Ranges: bytes
 Content-Length: 0

helm/silo/README.md

@@ -34,9 +34,10 @@ helm install silo ./helm/silo \
   --set rootPassword='replace-with-a-long-random-secret'
 ```
 
-The default server and post-install image is `docker.io/pgsty/silo`. The image
+The default server image is `docker.io/pgsty/silo` (which contains the `silo`
-contains the `silo` server, `mcli`, and an `mc` compatibility symlink. It does
+server, `mcli`, and an `mc` client compatibility symlink, without a `minio`
-not contain a `minio` server binary.
+server binary). Post-install resource jobs use `mcImage` (`docker.io/pgsty/mc`)
+by default.
 
 ## Compatibility contract
 
@@ -100,6 +101,19 @@ new chart invokes `silo server`; an old image does not provide that executable.
 The new image accepts a legacy first argv token of `minio` only to support old
 chart commands during the forward migration.
 
+When upgrading an existing Silo release to `RELEASE.2026-11-09T00-00-00Z` (chart
+`7.0.4`):
+
+- `helm upgrade --reuse-values` keeps the previously saved `image.tag` and
+  `mcImage.tag`; pass `--set-string image.tag=RELEASE.2026-11-09T00-00-00Z` and
+  `--set-string mcImage.tag=RELEASE.2026-11-09T00-00-00Z` explicitly.
+- Drain or cancel active batch jobs before upgrading; `20261026` cannot resume
+  new submitter-bound batch manifests on downgrade.
+- Single-pool clusters whose erasure sets do not all share the same pods/hosts
+  require a drained stop-all upgrade rather than a rolling pod restart, and
+  `POST ?restore` returns `503` while mixed versions are serving. See the
+  [20261109 release notes](https://silo.pgsty.com/blog/release/silo-20261109/#upgrade).
+
 ## Persistence and TLS
 
 The default PersistentVolumeClaim mounts at `/export`. Use an existing claim

helm/silo/templates/NOTES.txt

@@ -1,43 +1,44 @@
+{{- $scheme := ternary "https" "http" .Values.tls.enabled -}}
 {{- if eq .Values.service.type "ClusterIP" "NodePort" }}
 Silo can be accessed via port {{ .Values.service.port }} on the following DNS name from within your cluster:
-{{ template "silo.fullname" . }}.{{ .Release.Namespace }}.{{ .Values.clusterDomain }}
+{{ template "silo.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}
 
-To access Silo from localhost, run the below commands:
+To access Silo from localhost, run the commands below:
 
   1. export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
 
-  2. kubectl port-forward $POD_NAME 9000 --namespace {{ .Release.Namespace }}
+  2. kubectl port-forward $POD_NAME {{ .Values.service.port }}:{{ .Values.minioAPIPort | default 9000 }} --namespace {{ .Release.Namespace }}
 
-Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/
+Read more about port forwarding here: https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
 
-You can now access Silo server on http://localhost:9000. Follow the below steps to connect to Silo server with mc client:
+You can now access the Silo server on {{ $scheme }}://localhost:{{ .Values.service.port }}. Connect with the mcli client:
 
-  1. Download the Silo mc client - https://silo.pgsty.com/reference/minio-mc/#quickstart
+  1. Download mcli: https://silo.pgsty.com/download/#client
 
-  2. export MC_HOST_{{ template "silo.fullname" . }}_local=http://$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:{{ .Values.service.port }}
+  2. mcli alias set local {{ $scheme }}://localhost:{{ .Values.service.port }} "$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootUser}" | base64 --decode)" "$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)"
 
-  3. mc ls {{ template "silo.fullname" . }}_local
+  3. mcli ls local
 
 {{- end }}
 {{- if eq .Values.service.type "LoadBalancer" }}
 Silo can be accessed via port {{ .Values.service.port }} on an external IP address. Get the service external IP address by:
-kubectl get svc --namespace {{ .Release.Namespace }} -l app={{ template "silo.fullname" . }}
+kubectl get svc --namespace {{ .Release.Namespace }} -l "app={{ template "silo.name" . }},release={{ .Release.Name }}"
 
-Note that the public IP may take a couple of minutes to be available.
+Note that the public IP may take a couple of minutes to become available.
 
-You can now access Silo server on http://<External-IP>:9000. Follow the below steps to connect to Silo server with mc client:
+You can now access the Silo server on {{ $scheme }}://<External-IP>:{{ .Values.service.port }}. Connect with the mcli client:
 
-  1. Download the Silo mc client - https://silo.pgsty.com/reference/minio-mc/#quickstart
+  1. Download mcli: https://silo.pgsty.com/download/#client
 
-  2. export MC_HOST_{{ template "silo.fullname" . }}_local=http://$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)@<External-IP>:{{ .Values.service.port }}
+  2. mcli alias set external {{ $scheme }}://<External-IP>:{{ .Values.service.port }} "$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootUser}" | base64 --decode)" "$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootPassword}" | base64 --decode)"
 
-  3. mc ls {{ template "silo.fullname" . }}
+  3. mcli ls external
 
-Alternately, use a browser, an S3 SDK, or a MinIO-compatible SDK to access Silo - https://silo.pgsty.com/reference/minio-server/minio-server/
+Alternately, use a browser, an S3 SDK, or a MinIO-compatible SDK to access Silo: https://silo.pgsty.com/reference/minio-server/
 {{- end }}
 
 {{ if and (.Values.networkPolicy.enabled) (not .Values.networkPolicy.allowExternal) }}
 Note: Since NetworkPolicy is enabled, only pods with label
-{{ template "silo.fullname" . }}-client=true"
+{{ template "silo.fullname" . }}-client=true
 will be able to connect to this Silo cluster.
 {{- end }}

helm/silo/values.yaml

@@ -11,7 +11,7 @@ fullnameOverride: ""
 ##
 clusterDomain: cluster.local
 
-## Set default image, imageTag, and imagePullPolicy. mode is used to indicate the
+## Set default Silo server image repository, tag, and pull policy.
 ##
 image:
   repository: pgsty/silo
@@ -517,7 +517,7 @@ postJob:
 ## Add environment variables for Silo. The MINIO_* names remain the stable
 ## configuration interface and are passed to the Silo container(s).
 environment:
-  ## Please refer for comprehensive list https://silo.pgsty.com/reference/minio-server/minio-server/
+  ## See https://silo.pgsty.com/reference/minio-server/settings/ for available settings.
   ## MINIO_BROWSER: "off"
 
 ## The name of a secret in the same kubernetes namespace which contain secret values