Merge pull request #335 from botdeniska/fix/decom-keys-below-a-key

fix(decom): repeat the walk of a bucket while it removes source versions

Signed-off-by: Feng Ruohang <rh@vonng.com>

CHANGELOG.md

@@ -33,11 +33,13 @@
 
 ## RELEASE.2026-11-09T00-00-00Z
 
-Package version: `20261109000000.0.0`. Publication status and artifacts are
+Package version: `20261109000000.0.0`; published from source `d998292387cc`.
-listed in [GitHub Releases](https://github.com/pgsty/silo/releases). This
+Artifacts are listed in the [GitHub release](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z).
-section covers the changes after the published 20261026 source `c3ad836aed0d`.
+The coordinated [English release notes](https://silo.pgsty.com/blog/release/silo-20261109/)
-Most fixes come from reports and patches by
+and [Chinese release notes](https://silo.pgsty.com/zh/blog/release/silo-20261109/)
-[@botdeniska](https://github.com/botdeniska); see [CONTRIBUTORS.md](CONTRIBUTORS.md).
+cover all components, upgrade guidance, and validation scope since the
+published 20261026 source `c3ad836aed0d`. Most fixes come from reports and
+patches by [@botdeniska](https://github.com/botdeniska); see [CONTRIBUTORS.md](CONTRIBUTORS.md).
 
 Lifecycle, multi-pool and tiering:
 
@@ -52,7 +54,8 @@ Lifecycle, multi-pool and tiering:
   `retainVersions: 0`) on the expired key instead of the keys below it; a key
   with a tiered or Object Lock protected version is refused
   ([#297](https://github.com/pgsty/silo/issues/297)).
-- Stop taking the free version of a deleted tiered version for a version
+- Exclude the internal free-version marker of a deleted tiered version from
+  visible version history and lifecycle decisions
   ([#263](https://github.com/pgsty/silo/issues/263)), apply rules whose only
   action is `DelMarkerExpiration` ([#268](https://github.com/pgsty/silo/issues/268)),
   and report and account only the removals that happened

CONTRIBUTING.md

@@ -165,8 +165,11 @@ license grant to SILO or upstream MinIO maintainers is required.
 
 ### How does Silo manage dependencies?
 
-Silo uses Go modules. Preserve the compatibility module and import paths in
+Silo uses Go modules. Shared packages in `github.com/pgsty/silo-pkg/v3` are
-`go.mod`; downstream forks are selected with explicit `replace` directives.
+required and imported directly under their own module path, upstream
+`github.com/minio/minio-go/v7` is pinned to a verified commit, and Console
+(`github.com/minio/console`) and MC (`github.com/minio/mc`) are selected with
+explicit `replace` directives in `go.mod`.
 
 - Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
 

PULL_REQUESTS_ETIQUETTE.md

@@ -42,7 +42,8 @@ PRs must flow smoothly through review to reach production. Large PRs should be s
    - Document testing steps in the PR summary if manual testing was performed.
 
 4. **Before Submitting**:
-   - Run `make verify` to check formatting, linting, and tests.
+   - Sign off every commit under the Developer Certificate of Origin (`git commit -s`).
+   - Run `make verifiers` to check formatting, linting, code generation, and `rebrand-guard`.
    - Reference related issues (e.g., “Closes #1234”).
    - Notify team members via GitHub `@mentions` if urgent or complex.
 

README.md

@@ -36,7 +36,7 @@
 
 PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
 
-It follows one rule — **the product and its delivery surfaces are renamed; the protocol and your data are not.** Everything else lives on [silo.pgsty.com](https://silo.pgsty.com/).
+It follows one rule: **the product and its delivery surfaces are renamed; the protocol and your data are not.** Full documentation is published at [silo.pgsty.com](https://silo.pgsty.com/).
 
 **Related:** [`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
 
@@ -44,10 +44,10 @@ It follows one rule — **the product and its delivery surfaces are renamed; the
   <img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo Console">
 </p>
 
-#### SILO 20261109
+### SILO 20261109
 
 > [!WARNING]
-> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) fixes bugs and security vulnerabilities. We strongly recommend scheduling an upgrade to reduce security risks.
+> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) (`20261109000000.0.0`, source `d998292387cc`) fixes storage, lifecycle, native restore, Object Lock, batch authorization, and IAM defects, embedding Console `v2.7.0` with `mcli 20261109`, `silo-pkg v3.16.0`, and Go `1.27.3`. Review the [release notes and upgrade requirements](https://silo.pgsty.com/blog/release/silo-20261109/) before upgrading.
 
 > [!NOTE]
 > On 2026-08-06, `pgsty/minio` became `pgsty/silo`, and the default branch changed from `master` to `main`. The last MinIO-branded release is [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z) on the [`minio`](https://github.com/pgsty/silo/tree/minio) branch.
@@ -86,7 +86,7 @@ docker exec silo mcli mb local/demo && docker exec silo mcli ls local
 | Kubernetes | Helm chart, see [Download & Install](https://silo.pgsty.com/download/) |
 | Source | `go build -o silo . && ./silo --version` |
 
-Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/); migrating from upstream MinIO — taking over an existing `minio.service` and its `/etc/default/minio`, and keeping data ownership stable with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in — is covered by the [migration guide](https://silo.pgsty.com/compatibility/migration/) and the [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
+Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/). To migrate from upstream MinIO, take over `minio.service` and `/etc/default/minio`, and preserve data ownership with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in. See the [migration guide](https://silo.pgsty.com/compatibility/migration/) and [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
 
 ## Compatibility
 
@@ -98,7 +98,7 @@ Every divergence from upstream is listed in the code-verified [compatibility aud
 
 ### TLS and Go upgrades
 
-Server 20260916 restored Go's default key exchange; Server 20261018 retains it with Go 1.27.2.
+Server 20260916 restored Go's default key exchange; Server 20261109 retains it with Go 1.27.3.
 TLS key exchange follows Go's defaults across the S3 listener, node links,
 replication, identity providers, etcd, and external HTTP services. If an endpoint
 cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges

README_ZH.md

@@ -36,7 +36,7 @@
 
 上游停止社区发行后，Silo 为开源 MinIO 服务端维护一条持续可用的版本线：构建、软件包、多架构镜像、安全修复与完整 Web 控制台。Pigsty 在生产环境中用它承载 PostgreSQL 备份存储。
 
-它只遵循一条原则：**改名的是产品与交付物，不是协议与你的数据。** 其余内容都在 [silo.pgsty.com](https://silo.pgsty.com/zh/)。
+它只遵循一条原则：**改名的是产品与交付物，不是协议与你的数据。** 完整文档发布在 [silo.pgsty.com](https://silo.pgsty.com/zh/)。
 
 **相关项目：**[`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
 
@@ -44,10 +44,10 @@
   <img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo 控制台">
 </p>
 
-#### SILO 20261109
+### SILO 20261109
 
 > [!WARNING]
-> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) 修复了缺陷与安全漏洞，强烈建议尽快安排升级，以降低安全风险。
+> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z)（包版本 `20261109000000.0.0`，源码 `d998292387cc`）修复了多池生命周期、原生恢复、Object Lock、批处理授权与 IAM 等缺陷，内嵌 Console `v2.7.0` 并配套 `mcli 20261109`、`silo-pkg v3.16.0` 与 Go `1.27.3`。升级前请阅读[发布说明与升级要求](https://silo.pgsty.com/zh/blog/release/silo-20261109/)。
 
 > [!NOTE]
 > 2026-08-06，`pgsty/minio` 更名为 `pgsty/silo`，默认分支由 `master` 改为 `main`。最后一个 MinIO 品牌发布是 [`minio`](https://github.com/pgsty/silo/tree/minio) 分支上的 [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z)。
@@ -98,9 +98,9 @@ Silo 保留 S3 与存储格式兼容性，包括既有 `MINIO_*` 环境变量、
 
 ### TLS 与 Go 升级
 
-Server 20260916 已恢复 Go 默认密钥交换策略，20261018 在 Go 1.27.2 下保留该行为。
+Server 20260916 恢复了 Go 默认密钥交换策略，Server 20261109 在 Go 1.27.3 下保留该行为。S3 监听端口、节点间连接、站点与桶复制、身份提供方、etcd 及外部 HTTP 服务均使用 Go 默认 TLS 密钥交换。若对端无法接受 ML-KEM，设置 `GODEBUG=tlsmlkem=0` 可在当前进程禁用默认混合密钥交换（证书校验保持开启）；该选项不会禁用 ML-DSA 签名，也不能解决所有 TLS 连接重置问题，建议在升级不兼容对端后移除该临时配置。若仅新增的 SecP 混合组引发问题，可设置 `GODEBUG=tlssecpmlkem=0` 仅禁用这些组并保留 `X25519MLKEM768`。
-`GODEBUG=tlsmlkem=0`、`tlssecpmlkem=0` 的适用范围、macOS 根证书来源变化，
+
-以及 OIDC discovery 的诊断方法见 [Go 1.27 TLS 与 OIDC 指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
+在 Go 1.27 构建中，macOS 上设置 `SSL_CERT_FILE` 或 `SSL_CERT_DIR` 会以磁盘根证书和 Go 验证器替代系统钥匙串信任；陈旧或不完整的 CA 路径可能导致原先受信任的连接失败，清除继承的环境变量即可恢复钥匙串信任。配置的 `CAs` 目录中的显式证书仍会叠加加入所选根证书池。Go 1.27 二进制要求 macOS 13 或更高版本。详见 [Go 1.27 发行说明](https://go.dev/doc/go1.27)与 [Go 1.27 TLS 与 OIDC 发现指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
 
 ## 文档归属
 

SECURITY.md

@@ -38,7 +38,7 @@ For vulnerabilities in this fork:
 
 1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
 2. Prefer this repository's [private GitHub security advisory](https://github.com/pgsty/silo/security/advisories/new) workflow.
-3. If you cannot use GitHub private reporting, email [ron@pgsty.com](mailto:ron@pgsty.com) or rh@vonng.com
+3. If you cannot use GitHub private reporting, email [ron@pgsty.com](mailto:ron@pgsty.com) or [rh@vonng.com](mailto:rh@vonng.com)
    directly. Do not publish vulnerability details or exploit code in public
    issues or pull requests.
 4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.

cmd/batch-authz-review6_test.go

@@ -330,7 +330,7 @@ func TestBatchReviewRotationKeepsMigrationCleanup(t *testing.T) {
 				t.Errorf("rotation changed a draining source after its version was copied (rotation error: %v)", rotationErr)
 			}
 			snapshot := []FileInfo{{VersionID: before.VersionID, ModTime: beforeInfo.ModTime}}
-			if err := z.removeMovedVersions(t.Context(), bucket, object, 0, snapshot, nil); err != nil {
+			if _, err := z.removeMovedVersions(t.Context(), bucket, object, 0, snapshot, nil); err != nil {
 				t.Fatalf("rotation prevented migration cleanup: %v", err)
 			}
 			if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: before.VersionID}); !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {

cmd/erasure-server-pool-consistency.go

@@ -271,16 +271,17 @@ func (z *erasureServerPools) applyMigrationExpiry(ctx context.Context, bucket, o
 // in the source until the ordinary expiry path has deleted them: that path
 // owns replication state and tier cleanup, and it re-evaluates the current
 // lifecycle rule. A migration cleanup must never turn an asynchronous ILM
-// decision into an unconditional physical purge.
+// decision into an unconditional physical purge. It returns how many source
-func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) error {
+// versions it deleted.
+func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) (removed int, err error) {
 	if err := z.applyMigrationExpiry(ctx, bucket, object, source, expired); err != nil {
-		return err
+		return 0, err
 	}
 	object = encodeDirObject(object)
 	lk := z.NewNSLock(bucket, object)
 	lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
 	if err != nil {
-		return err
+		return 0, err
 	}
 	ctx = lkctx.Context()
 	defer lk.Unlock(lkctx)
@@ -294,7 +295,7 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
 			continue
 		}
 		if err != nil {
-			return err
+			return removed, err
 		}
 		var src *ObjectInfo
 		var retained []PoolObjInfo
@@ -309,7 +310,7 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
 			continue
 		}
 		if !src.ModTime.Equal(version.ModTime) || src.DeleteMarker != version.Deleted {
-			return PreConditionFailed{}
+			return removed, PreConditionFailed{}
 		}
 		if isExpiredVersion(expired, version) {
 			// The logical expiry path owns this version. Keeping the source is
@@ -319,17 +320,20 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
 		// Successful movement is not enough if the destination changed again.
 		// Keep the source unless another copy still preserves its read result.
 		if len(retained) == 0 || retained[0].ObjInfo.ModTime.Before(src.ModTime) {
-			return PreConditionFailed{}
+			return removed, PreConditionFailed{}
 		}
 		_, err = z.serverPools[source].DeleteObject(ctx, bucket, object, ObjectOptions{
 			VersionID: vid, NoLock: true, NoAuditLog: true,
 			SkipFreeVersion: true, // preserve the old prefix cleanup's tier-GC behavior
 		})
 		if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
-			return err
+			return removed, err
+		}
+		if err == nil {
+			removed++
 		}
 	}
-	return nil
+	return removed, nil
 }
 
 // ilmDeletesVersionOnly reports whether the ILM action queued for a version

cmd/erasure-server-pool-decom-keys-below_test.go

@@ -0,0 +1,66 @@
+// Copyright (c) 2026 Denis Babchuk
+//
+// This file is part of MinIO Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import "testing"
+
+// A key below another key ("k/b" under "k") is a directory inside the
+// directory of that key. The listing treats the directory of an object as a
+// leaf, so a decommission walk does not see "k/b" while "k" is still in the
+// pool. The walk has to be repeated once the keys above are gone, or the
+// check after the bucket finds them and the decommission fails.
+
+func keysBelowDecommission(t *testing.T, z *erasureServerPools) PoolDecommissionInfo {
+	t.Helper()
+	if err := z.StartDecommission(t.Context(), 0); err != nil {
+		t.Fatal(err)
+	}
+	z.doDecommissionInRoutine(t.Context(), 0)
+	z.poolMetaMutex.RLock()
+	defer z.poolMetaMutex.RUnlock()
+	return *z.poolMeta.Pools[0].Decommission
+}
+
+func keysBelowPools(t *testing.T) (*erasureServerPools, string, []string) {
+	t.Helper()
+	z, _ := migrationTestPools(t)
+	const bucket = "keys-below"
+	if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
+		t.Fatal(err)
+	}
+	keys := []string{"k", "k/b", "k/b/c"}
+	for _, key := range keys {
+		putConsistencyObject(t, z, bucket, key, 0, key, ObjectOptions{Versioned: true})
+	}
+	return z, bucket, keys
+}
+
+func TestDecommissionMovesKeysBelowAKey(t *testing.T) {
+	z, bucket, keys := keysBelowPools(t)
+	if info := keysBelowDecommission(t, z); !info.Complete || info.Failed {
+		t.Errorf("decommission: complete %v, failed %v; want complete", info.Complete, info.Failed)
+	}
+	for _, key := range keys {
+		if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, key, ObjectOptions{}); !isErrObjectNotFound(err) {
+			t.Errorf("%s still in pool 0: %v", key, err)
+		}
+		if body := migrationTestBody(t, z.GetObjectNInfo, bucket, key, ""); body != key {
+			t.Errorf("%s reads %q", key, body)
+		}
+	}
+}

cmd/erasure-server-pool-decom.go

@@ -28,6 +28,7 @@ import (
 	"slices"
 	"sort"
 	"strings"
+	"sync/atomic"
 	"time"
 
 	"github.com/dustin/go-humanize"
@@ -752,7 +753,25 @@ func (set *erasureObjects) listObjectsToDecommission(ctx context.Context, bi dec
 	return err
 }
 
+// decommissionPool moves bucket bi out of pool idx. A key below another key
+// ("k/b" under "k") is a directory inside the directory of that key, and the
+// listing does not descend into the directory of an object: a walk sees "k/b"
+// only once "k" has left the pool. The walk is repeated while it removes
+// source versions, so each pass moves the keys that the previous one
+// uncovered. Each repeat needs a version removed from the pool, which takes
+// no new writes, so the repeats end.
 func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) error {
+	for {
+		removed, err := z.decommissionPoolWalk(ctx, idx, pool, bi)
+		if err != nil || removed == 0 || contextCanceled(ctx) {
+			return err
+		}
+	}
+}
+
+// decommissionPoolWalk walks bucket bi of pool idx once and returns the number
+// of keys whose source versions it removed.
+func (z *erasureServerPools) decommissionPoolWalk(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) (int64, error) {
 	ctx = logger.SetReqInfo(ctx, &logger.ReqInfo{})
 
 	const envDecomWorkers = "_MINIO_DECOMMISSION_WORKERS"
@@ -766,9 +785,11 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 	// add that many extra workers.
 	workerSize += len(pool.sets)
 
+	var removed atomic.Int64
+
 	wk, err := workers.New(workerSize)
 	if err != nil {
-		return err
+		return 0, err
 	}
 
 	var vc *versioning.Versioning
@@ -779,24 +800,24 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 	if bi.Name != minioMetaBucket {
 		vc, err = globalBucketVersioningSys.Get(bi.Name)
 		if err != nil {
-			return err
+			return 0, err
 		}
 
 		// Check if the current bucket has a configured lifecycle policy
 		lc, lifecycleUpdatedAt, err = globalBucketMetadataSys.GetLifecycleConfig(bi.Name)
 		if err != nil && !errors.Is(err, BucketLifecycleNotFound{Bucket: bi.Name}) {
-			return err
+			return 0, err
 		}
 
 		// Check if bucket is object locked.
 		lr, err = globalBucketObjectLockSys.Get(bi.Name)
 		if err != nil {
-			return err
+			return 0, err
 		}
 
 		rcfg, err = getReplicationConfig(ctx, bi.Name)
 		if err != nil {
-			return err
+			return 0, err
 		}
 	}
 
@@ -1019,7 +1040,10 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 			// if all versions were decommissioned, then we can delete the object versions.
 			if decommissioned == len(fivs.Versions) {
 				stopFn := globalDecommissionMetrics.log(decomMetricDecommissionRemoveObject, idx, bi.Name, entry.name)
-				err := z.removeMovedVersions(ctx, bi.Name, entry.name, idx, fivs.Versions, expired)
+				n, err := z.removeMovedVersions(ctx, bi.Name, entry.name, idx, fivs.Versions, expired)
+				if n > 0 {
+					removed.Add(1)
+				}
 				if err == nil {
 					err = z.removeFreeVersions(ctx, bi.Name, entry.name, idx, fivs.FreeVersions)
 				}
@@ -1062,7 +1086,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 		}(setIdx)
 	}
 	wk.Wait()
-	return nil
+	return removed.Load(), nil
 }
 
 //msgp:ignore decomMetrics

cmd/erasure-server-pool-lifecycle-authority_test.go

@@ -1666,7 +1666,7 @@ func TestPoolsMigrationLifecycleRefusedExpiryKeepsSource(t *testing.T) {
 		t.Fatal(err)
 	}
 	version := FileInfo{Name: key, VersionID: id, ModTime: source.ModTime}
-	err = s.z.removeMovedVersions(ctx, s.bucket, key, 0, []FileInfo{version}, []migrationExpiredVersion{{
+	_, err = s.z.removeMovedVersions(ctx, s.bucket, key, 0, []FileInfo{version}, []migrationExpiredVersion{{
 		version:            version,
 		event:              lifecycle.Event{Action: lifecycle.DeleteAction},
 		lifecycleUpdatedAt: updatedAt,

cmd/erasure-server-pool-migration-delete_test.go

@@ -132,7 +132,7 @@ func TestPoolsMigrationCleanupRechecksSource(t *testing.T) {
 				_, err := z.serverPools[1].DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: nullVersionID})
 				mustIAM(t, err)
 			}
-			err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot, nil)
+			_, err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot, nil)
 			if change != "delete-uuid" && err == nil {
 				t.Fatal("cleanup accepted changed or uncertain state")
 			}

cmd/erasure-server-pool-rebalance.go

@@ -779,7 +779,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
 			// if all versions were rebalanced, we can delete the object versions.
 			if rebalanced == len(fivs.Versions) {
 				stopFn := globalRebalanceMetrics.log(rebalanceMetricRebalanceRemoveObject, poolIdx, bucket, entry.name)
-				err := z.removeMovedVersions(ctx, bucket, entry.name, poolIdx, fivs.Versions, expired)
+				_, err := z.removeMovedVersions(ctx, bucket, entry.name, poolIdx, fivs.Versions, expired)
 				stopFn(0, err)
 				auditLogRebalance(ctx, "Rebalance:DeleteObject", bucket, entry.name, "", err)
 				if err != nil {

docs/metrics/healthcheck/README.md

@@ -1,6 +1,6 @@
 # Silo Healthcheck
 
-Silo server exposes three un-authenticated, healthcheck endpoints liveness probe and a cluster probe at `/minio/health/live` and `/minio/health/cluster` respectively.
+Silo server exposes four unauthenticated healthcheck endpoints: `/minio/health/live`, `/minio/health/ready`, `/minio/health/cluster`, and `/minio/health/cluster/read`.
 
 ## Native CLI probe
 
@@ -76,7 +76,7 @@ readinessProbe:
 The reply is '200 OK' if cluster has write quorum if not it returns '503 Service Unavailable'.
 
 ```
-curl http://silo1:9001/minio/health/cluster
+curl http://silo1:9000/minio/health/cluster
 HTTP/1.1 503 Service Unavailable
 Accept-Ranges: bytes
 Content-Length: 0
@@ -94,7 +94,7 @@ Date: Tue, 21 Jul 2020 00:36:14 GMT
 The reply is '200 OK' if cluster has read quorum if not it returns '503 Service Unavailable'.
 
 ```
-curl http://silo1:9001/minio/health/cluster/read
+curl http://silo1:9000/minio/health/cluster/read
 HTTP/1.1 503 Service Unavailable
 Accept-Ranges: bytes
 Content-Length: 0
@@ -112,7 +112,7 @@ Date: Tue, 21 Jul 2020 00:36:14 GMT
 You may query the cluster probe endpoint to check if the node which received the request can be taken down for maintenance, if the server replies back '412 Precondition Failed' this means you will lose HA. '200 OK' means you are okay to proceed.
 
 ```
-curl http://silo1:9001/minio/health/cluster?maintenance=true
+curl http://silo1:9000/minio/health/cluster?maintenance=true
 HTTP/1.1 412 Precondition Failed
 Accept-Ranges: bytes
 Content-Length: 0

helm/silo/README.md

@@ -34,9 +34,10 @@ helm install silo ./helm/silo \
   --set rootPassword='replace-with-a-long-random-secret'
 ```
 
-The default server and post-install image is `docker.io/pgsty/silo`. The image
+The default server image is `docker.io/pgsty/silo` (which contains the `silo`
-contains the `silo` server, `mcli`, and an `mc` compatibility symlink. It does
+server, `mcli`, and an `mc` client compatibility symlink, without a `minio`
-not contain a `minio` server binary.
+server binary). Post-install resource jobs use `mcImage` (`docker.io/pgsty/mc`)
+by default.
 
 ## Compatibility contract
 
@@ -100,6 +101,19 @@ new chart invokes `silo server`; an old image does not provide that executable.
 The new image accepts a legacy first argv token of `minio` only to support old
 chart commands during the forward migration.
 
+When upgrading an existing Silo release to `RELEASE.2026-11-09T00-00-00Z` (chart
+`7.0.4`):
+
+- `helm upgrade --reuse-values` keeps the previously saved `image.tag` and
+  `mcImage.tag`; pass `--set-string image.tag=RELEASE.2026-11-09T00-00-00Z` and
+  `--set-string mcImage.tag=RELEASE.2026-11-09T00-00-00Z` explicitly.
+- Drain or cancel active batch jobs before upgrading; `20261026` cannot resume
+  new submitter-bound batch manifests on downgrade.
+- Single-pool clusters whose erasure sets do not all share the same pods/hosts
+  require a drained stop-all upgrade rather than a rolling pod restart, and
+  `POST ?restore` returns `503` while mixed versions are serving. See the
+  [20261109 release notes](https://silo.pgsty.com/blog/release/silo-20261109/#upgrade).
+
 ## Persistence and TLS
 
 The default PersistentVolumeClaim mounts at `/export`. Use an existing claim

helm/silo/templates/NOTES.txt

@@ -1,43 +1,44 @@
+{{- $scheme := ternary "https" "http" .Values.tls.enabled -}}
 {{- if eq .Values.service.type "ClusterIP" "NodePort" }}
 Silo can be accessed via port {{ .Values.service.port }} on the following DNS name from within your cluster:
-{{ template "silo.fullname" . }}.{{ .Release.Namespace }}.{{ .Values.clusterDomain }}
+{{ template "silo.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}
 
-To access Silo from localhost, run the below commands:
+To access Silo from localhost, run the commands below:
 
   1. export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
 
-  2. kubectl port-forward $POD_NAME 9000 --namespace {{ .Release.Namespace }}
+  2. kubectl port-forward $POD_NAME {{ .Values.service.port }}:{{ .Values.minioAPIPort | default 9000 }} --namespace {{ .Release.Namespace }}
 
-Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/
+Read more about port forwarding here: https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
 
-You can now access Silo server on http://localhost:9000. Follow the below steps to connect to Silo server with mc client:
+You can now access the Silo server on {{ $scheme }}://localhost:{{ .Values.service.port }}. Connect with the mcli client:
 
-  1. Download the Silo mc client - https://silo.pgsty.com/reference/minio-mc/#quickstart
+  1. Download mcli: https://silo.pgsty.com/download/#client
 
-  2. export MC_HOST_{{ template "silo.fullname" . }}_local=http://$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:{{ .Values.service.port }}
+  2. mcli alias set local {{ $scheme }}://localhost:{{ .Values.service.port }} "$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootUser}" | base64 --decode)" "$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)"
 
-  3. mc ls {{ template "silo.fullname" . }}_local
+  3. mcli ls local
 
 {{- end }}
 {{- if eq .Values.service.type "LoadBalancer" }}
 Silo can be accessed via port {{ .Values.service.port }} on an external IP address. Get the service external IP address by:
-kubectl get svc --namespace {{ .Release.Namespace }} -l app={{ template "silo.fullname" . }}
+kubectl get svc --namespace {{ .Release.Namespace }} -l "app={{ template "silo.name" . }},release={{ .Release.Name }}"
 
-Note that the public IP may take a couple of minutes to be available.
+Note that the public IP may take a couple of minutes to become available.
 
-You can now access Silo server on http://<External-IP>:9000. Follow the below steps to connect to Silo server with mc client:
+You can now access the Silo server on {{ $scheme }}://<External-IP>:{{ .Values.service.port }}. Connect with the mcli client:
 
-  1. Download the Silo mc client - https://silo.pgsty.com/reference/minio-mc/#quickstart
+  1. Download mcli: https://silo.pgsty.com/download/#client
 
-  2. export MC_HOST_{{ template "silo.fullname" . }}_local=http://$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)@<External-IP>:{{ .Values.service.port }}
+  2. mcli alias set external {{ $scheme }}://<External-IP>:{{ .Values.service.port }} "$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootUser}" | base64 --decode)" "$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootPassword}" | base64 --decode)"
 
-  3. mc ls {{ template "silo.fullname" . }}
+  3. mcli ls external
 
-Alternately, use a browser, an S3 SDK, or a MinIO-compatible SDK to access Silo - https://silo.pgsty.com/reference/minio-server/minio-server/
+Alternately, use a browser, an S3 SDK, or a MinIO-compatible SDK to access Silo: https://silo.pgsty.com/reference/minio-server/
 {{- end }}
 
 {{ if and (.Values.networkPolicy.enabled) (not .Values.networkPolicy.allowExternal) }}
 Note: Since NetworkPolicy is enabled, only pods with label
-{{ template "silo.fullname" . }}-client=true"
+{{ template "silo.fullname" . }}-client=true
 will be able to connect to this Silo cluster.
 {{- end }}

helm/silo/values.yaml

@@ -11,7 +11,7 @@ fullnameOverride: ""
 ##
 clusterDomain: cluster.local
 
-## Set default image, imageTag, and imagePullPolicy. mode is used to indicate the
+## Set default Silo server image repository, tag, and pull policy.
 ##
 image:
   repository: pgsty/silo
@@ -517,7 +517,7 @@ postJob:
 ## Add environment variables for Silo. The MINIO_* names remain the stable
 ## configuration interface and are passed to the Silo container(s).
 environment:
-  ## Please refer for comprehensive list https://silo.pgsty.com/reference/minio-server/minio-server/
+  ## See https://silo.pgsty.com/reference/minio-server/settings/ for available settings.
   ## MINIO_BROWSER: "off"
 
 ## The name of a secret in the same kubernetes namespace which contain secret values