Merge main after R4 validation into R7 candidate

Signed-off-by: Feng Ruohang <rh@vonng.com>

CHANGELOG.md

@@ -1,11 +1,11 @@
 # Changelog
 
-## Unreleased — main as of 2026-09-13
+## Unreleased
 
-The coordinated source is merged through `5d955b5b7444f8a3ab550ce92713607998f89c0d`.
+The entries below describe source changes on main since the latest published Server.
 **The latest published Server remains 20260903.** These changes are not in its
 binaries, packages or images. See the [component matrix](https://silo.pgsty.com/compatibility/versions/)
-and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-09-03T13-18-01Z...5d955b5b7444f8a3ab550ce92713607998f89c0d).
+and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-09-03T13-18-01Z...main).
 
 ### Authorization and security
 
@@ -22,10 +22,38 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
 
 ### Object storage and replication
 
-- Add GET-frequency-based movement across pools, then preserve versions and
+- Evaluate conditional multipart completion against the logical current object
-  isolate writes during movement. Serialize and reconcile multi-pool object
+  across all pools while holding the existing object lock. A stale `If-Match`
-  writes, metadata updates, healing and conditional deletion; preserve shared
+  can no longer replace newer data in another pool, and the current ETag is no
-  remote-tier references until their last local owner is removed.
+  longer rejected because the upload resides next to an older copy. Conditions
+  are evaluated once; a current delete marker counts as an absent object.
+  **Availability change:** if metadata cannot be read from any pool, conditional
+  completion fails even when another pool can still serve GET/HEAD. This also
+  applies when the unreadable pool may not hold the object: absence cannot be
+  verified. Retry after the pool recovers. Unconditional completion and the
+  single-pool path retain their existing behavior.
+
+- Reconcile ordinary single-object version DELETE across all pools, including
+  null versions, delete markers and unqualified directory-marker DELETE. This
+  applies the deletion to every resolved pool copy under existing quorum
+  rules. Pending outbound delete replication retains versions until the
+  existing replication worker completes their purge; a successful response
+  does not imply immediate physical removal from every drive. Unreadable
+  pools now consistently return 503 instead of depending on pool traversal
+  order; insufficient read quorum returns `SlowDownRead`. This extends the
+  existing failure surface. Retry after recovery.
+  Cleanup failures also return an error. Batch deletion already fans out across
+  pools; replication and scanner cleanup keep their existing contracts. See
+  [scope and limitations](docs/bucket/lifecycle/access-tiering-removal.md#version-deletion-scope).
+
+- Remove the opt-in GET-frequency pool-tiering feature from PR #60, including
+  its tracker, mover, scanner hooks, configuration, XML actions and metrics.
+  Accept and ignore retired configuration/XML and preserve ordinary statistics
+  when reading v9 caches. See [migration notes](docs/bucket/lifecycle/access-tiering-removal.md).
+  The [decision record](docs/investigations/access-tiering-revert.md) preserves
+  the feature's introduction, subsequent fixes, rollback scope and review history.
+- Preserve the independent multi-pool write, metadata, healing and conditional
+  deletion fixes from PR #178, including shared remote-tier reference protection.
 - Enforce `If-Match` on DELETE, preserve retention and independently ordered
   Object Lock/tag updates, and correctly retransmit encrypted replicas.
 - Preserve plaintext part sizes and raw SSE-C replicas; prevent SSE-C

CONTRIBUTORS.md

@@ -62,7 +62,7 @@ the Git history and [NOTICE](NOTICE); this record covers activity in the PGSTY r
 | :-- | :-- | :-- |
 | [@Vonng](https://github.com/Vonng) | Maintains SILO, Console, mcli, shared packages, releases, and documentation | 85 merged PRs across five repositories; [complete maintainer record](#maintainer-record) |
 | [@h5vx](https://github.com/h5vx) | Implemented per-bucket CORS configuration and enforcement | [pgsty/silo#71](https://github.com/pgsty/silo/pull/71) — Merged ([e4e3007da](https://github.com/pgsty/silo/commit/e4e3007da6d7d1198a6a050e34f84566d40a9654)) |
-| [@mrjavadseydi](https://github.com/mrjavadseydi) | Fixed effective bucket quota metrics; proposed access-frequency ILM | [pgsty/silo#132](https://github.com/pgsty/silo/pull/132) — Merged ([ad873c735](https://github.com/pgsty/silo/commit/ad873c73571b121293791f13f8dc46ddf5264d60))<br>[pgsty/silo#60](https://github.com/pgsty/silo/pull/60) — Open |
+| [@mrjavadseydi](https://github.com/mrjavadseydi) | Fixed effective bucket quota metrics; proposed access-frequency ILM | [pgsty/silo#132](https://github.com/pgsty/silo/pull/132) — Merged ([ad873c735](https://github.com/pgsty/silo/commit/ad873c73571b121293791f13f8dc46ddf5264d60))<br>[pgsty/silo#60](https://github.com/pgsty/silo/pull/60) — Merged; access-frequency feature subsequently removed |
 | [@Dansyuqri](https://github.com/Dansyuqri) | Added ChecksumType to multipart completion responses | [pgsty/silo#57](https://github.com/pgsty/silo/pull/57) — Merged ([a96116b12](https://github.com/pgsty/silo/commit/a96116b128bbf2aa42f85eafbf75eb6636cd36ee)) |
 | [@ycjlin](https://github.com/ycjlin) | Fixed missing-bucket ListObjects semantics | [pgsty/silo#37](https://github.com/pgsty/silo/pull/37) — Merged ([49c8aeac4](https://github.com/pgsty/silo/commit/49c8aeac403916f52f8588bbe8ee42753d86eeef)) |
 | [@pinginfo](https://github.com/pinginfo) | Repaired bucket notification streaming | [pgsty/silo#34](https://github.com/pgsty/silo/pull/34) — Merged ([b7f52ca43](https://github.com/pgsty/silo/commit/b7f52ca4336bc45a48b81b39c8983a5e6882a6fa)) |

buildscripts/rebrand-guard/compat-baseline.json

@@ -289,16 +289,6 @@
     "MINIO_IDENTITY_TLS_SKIP_VERIFY",
     "MINIO_IDENTITY_TLS_STS_EXPIRY",
     "MINIO_IDLE_TIMEOUT",
-    "MINIO_ILM_ACCESS_BINS",
-    "MINIO_ILM_ACCESS_BIN_WIDTH",
-    "MINIO_ILM_ACCESS_FLUSH",
-    "MINIO_ILM_ACCESS_MAX_SIZE",
-    "MINIO_ILM_ACCESS_MAX_TRACKED",
-    "MINIO_ILM_ACCESS_MIN_RESIDENCY",
-    "MINIO_ILM_ACCESS_POOLS",
-    "MINIO_ILM_ACCESS_PROMOTE_WATERMARK",
-    "MINIO_ILM_ACCESS_TIERING",
-    "MINIO_ILM_ACCESS_WORKERS",
     "MINIO_ILM_EXPIRATION_WORKERS",
     "MINIO_ILM_TRANSITION_WORKERS",
     "MINIO_INTERFACE",
@@ -745,7 +735,6 @@
     "/idp/ldap/policy/{operation}",
     "/idp/openid/list-access-keys-bulk",
     "/ilm",
-    "/ilm/access",
     "/import-bucket-metadata",
     "/import-iam",
     "/import-iam-v2",
@@ -840,6 +829,7 @@
     "/site-replication/peer/bucket-ops",
     "/site-replication/peer/edit",
     "/site-replication/peer/iam-item",
+    "/site-replication/peer/iam-revisions",
     "/site-replication/peer/idp-settings",
     "/site-replication/peer/join",
     "/site-replication/peer/remove",
@@ -888,6 +878,7 @@
     "/v2/metrics/cluster",
     "/v2/metrics/node",
     "/v2/metrics/resource",
+    "/v3/site-replication/peer/iam-revisions",
     "/var/vcap/bosh",
     "/verifybinary",
     "/version",
@@ -915,6 +906,7 @@
     ".minio.sys/config/config.json",
     ".minio.sys/config/hello.txt",
     ".minio.sys/config/iam/${username}/identity.json",
+    ".minio.sys/config/ilm/access",
     ".minio.sys/format.json",
     ".minio.sys/multipart",
     ".minio.sys/multipart/bucket/object/uploads.json",

cmd/admin-router.go

@@ -388,6 +388,7 @@ func registerAdminRouter(router *mux.Router, enableConfigOps bool) {
 		adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/join").HandlerFunc(adminMiddleware(adminAPI.SRPeerJoin))
 		adminRouter.Methods(http.MethodPut).Path(adminVersion+"/site-replication/peer/bucket-ops").HandlerFunc(adminMiddleware(adminAPI.SRPeerBucketOps)).Queries("bucket", "{bucket:.*}").Queries("operation", "{operation:.*}")
 		adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/iam-item").HandlerFunc(adminMiddleware(adminAPI.SRPeerReplicateIAMItem))
+		adminRouter.Methods(http.MethodGet, http.MethodPut).Path(adminVersion + "/site-replication/peer/iam-revisions").HandlerFunc(adminMiddleware(adminAPI.SRPeerIAMRevisions))
 		adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/bucket-meta").HandlerFunc(adminMiddleware(adminAPI.SRPeerReplicateBucketItem))
 		adminRouter.Methods(http.MethodGet).Path(adminVersion + "/site-replication/peer/idp-settings").HandlerFunc(adminMiddleware(adminAPI.SRPeerGetIDPSettings))
 		adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/edit").HandlerFunc(adminMiddleware(adminAPI.SiteReplicationEdit))

cmd/bucket-lifecycle-audit.go

@@ -39,7 +39,6 @@ const (
 	lcEventSrc_s3PutObject
 	lcEventSrc_s3CopyObject
 	lcEventSrc_s3CompleteMultipartUpload
-	lcEventSrc_AccessTier
 )
 
 //revive:enable:var-naming

cmd/bucket-metadata_test.go

@@ -39,3 +39,28 @@ func TestBucketMetadataCorsRoundTrip(t *testing.T) {
 		t.Fatalf("CorsConfigUpdatedAt not preserved")
 	}
 }
+
+// A persisted retired extension must not prevent the whole bucket's metadata
+// from loading, including unrelated versioning and ordinary lifecycle rules.
+func TestBucketMetadataRetiredAccessTiering(t *testing.T) {
+	meta := newBucketMetadata("retired-access")
+	meta.LifecycleConfigXML = []byte(`<LifecycleConfiguration><AccessTierQuota>500GiB</AccessTierQuota><Rule><ID>access</ID><Status>Enabled</Status><Filter><Prefix>logs/</Prefix></Filter><AccessTransition><Window>10m</Window><PromoteAfterAccesses>10</PromoteAfterAccesses></AccessTransition></Rule><Rule><ID>ordinary</ID><Status>Enabled</Status><Filter><Prefix>expired/</Prefix></Filter><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>`)
+	meta.VersioningConfigXML = []byte(`<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Enabled</Status></VersioningConfiguration>`)
+	data, err := meta.MarshalMsg(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := newBucketMetadata(meta.Name)
+	if _, err := got.UnmarshalMsg(data); err != nil {
+		t.Fatal(err)
+	}
+	if err := got.parseAllConfigs(t.Context(), nil); err != nil {
+		t.Fatalf("bucket metadata failed to load: %v", err)
+	}
+	if got.lifecycleConfig == nil || got.lifecycleConfig.HasActiveRules("logs/") || !got.lifecycleConfig.HasActiveRules("expired/") {
+		t.Fatal("unexpected lifecycle behavior")
+	}
+	if got.versioningConfig == nil || !got.versioningConfig.Enabled() {
+		t.Fatal("unrelated versioning lost")
+	}
+}

cmd/config-current.go

@@ -227,7 +227,7 @@ func initHelp() {
 		},
 		config.HelpKV{
 			Key:         config.ILMSubSys,
-			Description: "manage ILM settings for expiration, transition, and access-tier workers",
+			Description: "manage ILM settings for expiration and transition workers",
 			Optional:    true,
 		},
 	}
@@ -704,9 +704,6 @@ func applyDynamicConfigForSubSys(ctx context.Context, objAPI ObjectLayer, s conf
 			if globalExpiryState != nil {
 				globalExpiryState.ResizeWorkers(ilmCfg.ExpirationWorkers)
 			}
-			if globalAccessTierState != nil {
-				globalAccessTierState.UpdateWorkers(ilmCfg.AccessWorkers)
-			}
 			globalILMConfig.update(ilmCfg)
 		}
 	}

cmd/data-scanner.go

@@ -899,7 +899,6 @@ type scannerItem struct {
 	objectName  string // Only the object name without prefixes.
 	replication replicationConfig
 	lifeCycle   *lifecycle.Lifecycle
-	poolIdx     int
 	Typ         fs.FileMode
 	heal        struct {
 		enabled bool
@@ -910,7 +909,6 @@ type scannerItem struct {
 
 type sizeSummary struct {
 	totalSize       int64
-	hotTierSize     int64
 	versions        uint64
 	deleteMarkers   uint64
 	replicatedSize  int64
@@ -1161,14 +1159,6 @@ eventLoop:
 		globalExpiryState.enqueueNoncurrentVersions(i.bucket, toDel, noncurrentEvents)
 	}
 	i.alertExcessiveVersions(remainingVersions, cumulativeSize)
-	if globalILMConfig.accessTieringEnabled() {
-		for idx, oi := range objInfos {
-			if oi.IsLatest && events[idx].Action == lifecycle.NoneAction {
-				applyAccessTransition(ctx, i, oi)
-				break
-			}
-		}
-	}
 }
 
 func evalActionFromLifecycle(ctx context.Context, lc lifecycle.Lifecycle, lr lock.Retention, rcfg *replication.Config, obj ObjectInfo) lifecycle.Event {
@@ -1484,8 +1474,6 @@ const (
 	ILMFreeVersionDelete = "ilm:free-version-delete"
 	// ILMTransition - audit trail for ILM transitioning.
 	ILMTransition = " ilm:transition"
-	// ILMAccessTier - audit trail for moving objects between server pools.
-	ILMAccessTier = "ilm:access-tier"
 )
 
 func auditLogLifecycle(ctx context.Context, oi ObjectInfo, event string, tags map[string]string, traceFn func(event string, metadata map[string]string, err error)) {
@@ -1497,8 +1485,6 @@ func auditLogLifecycle(ctx context.Context, oi ObjectInfo, event string, tags ma
 		apiName = "ILMFreeVersionDelete"
 	case ILMTransition:
 		apiName = "ILMTransition"
-	case ILMAccessTier:
-		apiName = "ILMAccessTier"
 	}
 	auditLogInternal(ctx, AuditLogOptions{
 		Event:     event,

cmd/data-usage-cache-v9_test.go

@@ -0,0 +1,89 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"bytes"
+	"encoding/json"
+	"os"
+	"reflect"
+	"testing"
+)
+
+// Read a real pre-removal scanner cache with nonzero hot-tier bytes. A v8
+// payload with a relabeled header would not exercise the ignored hts field.
+func TestDataUsageCacheReadV9(t *testing.T) {
+	raw, err := os.ReadFile("testdata/data-usage-v9/data-usage-v9.bin")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(raw) == 0 || raw[0] != 9 {
+		t.Fatal("fixture is not v9")
+	}
+	expected, err := os.ReadFile("testdata/data-usage-v9/data-usage-v9.json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var want, got dataUsageCache
+	if err := json.Unmarshal(expected, &want); err != nil {
+		t.Fatal(err)
+	}
+	if err := got.deserialize(bytes.NewReader(raw)); err != nil {
+		t.Fatal(err)
+	}
+	if !got.Info.LastUpdate.Equal(want.Info.LastUpdate) {
+		t.Fatal("cache timestamp changed")
+	}
+	// msgp restores local time while JSON preserves the UTC representation.
+	want.Info.LastUpdate = got.Info.LastUpdate
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("v9 ordinary cache fields changed:\ngot: %#v\nwant: %#v", got, want)
+	}
+	flat := got.flatten(*got.root())
+	if flat.Size != 74962 || flat.Objects != 3 || flat.Versions != 5 || flat.DeleteMarkers != 2 {
+		t.Fatalf("statistics changed: %+v", flat)
+	}
+	if flat.AllTierStats == nil || flat.AllTierStats.Tiers["COLD"] != (tierStats{TotalSize: 65536, NumVersions: 1, NumObjects: 1}) {
+		t.Fatalf("remote tier statistics changed: %+v", flat.AllTierStats)
+	}
+	buckets := []BucketInfo{{Name: "v9-bucket"}}
+	if gotInfo, wantInfo := got.dui(dataUsageRoot, buckets), want.dui(dataUsageRoot, buckets); !reflect.DeepEqual(gotInfo, wantInfo) {
+		t.Fatalf("bucket usage aggregation changed: %+v != %+v", gotInfo, wantInfo)
+	}
+	var buf bytes.Buffer
+	if err := got.serializeTo(&buf); err != nil {
+		t.Fatal(err)
+	}
+	if buf.Bytes()[0] != 8 {
+		t.Fatalf("wrote cache version %d, want 8", buf.Bytes()[0])
+	}
+	var roundtrip dataUsageCache
+	if err := roundtrip.deserialize(&buf); err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(got, roundtrip) {
+		t.Fatal("v8 round trip lost ordinary fields")
+	}
+	if err := roundtrip.deserialize(bytes.NewReader(raw[:len(raw)/2])); err == nil {
+		t.Fatal("truncated v9 cache accepted")
+	}
+	raw[0] = 10
+	if err := roundtrip.deserialize(bytes.NewReader(raw)); err == nil {
+		t.Fatal("unknown cache version accepted")
+	}
+}

cmd/data-usage-cache.go

@@ -61,7 +61,6 @@ type dataUsageEntry struct {
 	Children dataUsageHashMap `msg:"ch"`
 	// These fields do no include any children.
 	Size          int64             `msg:"sz"`
-	HotTierSize   int64             `msg:"hts"`
 	Objects       uint64            `msg:"os"`
 	Versions      uint64            `msg:"vs"` // Versions that are not delete markers.
 	DeleteMarkers uint64            `msg:"dms"`
@@ -135,8 +134,8 @@ func (ts tierStats) add(u tierStats) tierStats {
 	}
 }
 
-//msgp:encode ignore dataUsageEntryV2 dataUsageEntryV3 dataUsageEntryV4 dataUsageEntryV5 dataUsageEntryV6 dataUsageEntryV7 dataUsageEntryV8
+//msgp:encode ignore dataUsageEntryV2 dataUsageEntryV3 dataUsageEntryV4 dataUsageEntryV5 dataUsageEntryV6 dataUsageEntryV7
-//msgp:marshal ignore dataUsageEntryV2 dataUsageEntryV3 dataUsageEntryV4 dataUsageEntryV5 dataUsageEntryV6 dataUsageEntryV7 dataUsageEntryV8
+//msgp:marshal ignore dataUsageEntryV2 dataUsageEntryV3 dataUsageEntryV4 dataUsageEntryV5 dataUsageEntryV6 dataUsageEntryV7
 
 //msgp:tuple dataUsageEntryV2
 type dataUsageEntryV2 struct {
@@ -200,30 +199,14 @@ type dataUsageEntryV7 struct {
 	Compacted     bool              `msg:"c"`
 }
 
-// dataUsageEntryV8 is the on-disk shape before access-tier accounting was
-// introduced. Keep it so caches written by the previous release decode
-// without being discarded.
-type dataUsageEntryV8 struct {
-	Children dataUsageHashMap `msg:"ch"`
-	// These fields do no include any children.
-	Size          int64             `msg:"sz"`
-	Objects       uint64            `msg:"os"`
-	Versions      uint64            `msg:"vs"`
-	DeleteMarkers uint64            `msg:"dms"`
-	ObjSizes      sizeHistogram     `msg:"szs"`
-	ObjVersions   versionsHistogram `msg:"vh"`
-	AllTierStats  *allTierStats     `msg:"ats,omitempty"`
-	Compacted     bool              `msg:"c"`
-}
-
 // dataUsageCache contains a cache of data usage entries latest version.
 type dataUsageCache struct {
 	Info  dataUsageCacheInfo
 	Cache map[string]dataUsageEntry
 }
 
-//msgp:encode ignore dataUsageCacheV2 dataUsageCacheV3 dataUsageCacheV4 dataUsageCacheV5 dataUsageCacheV6 dataUsageCacheV7 dataUsageCacheV8
+//msgp:encode ignore dataUsageCacheV2 dataUsageCacheV3 dataUsageCacheV4 dataUsageCacheV5 dataUsageCacheV6 dataUsageCacheV7
-//msgp:marshal ignore dataUsageCacheV2 dataUsageCacheV3 dataUsageCacheV4 dataUsageCacheV5 dataUsageCacheV6 dataUsageCacheV7 dataUsageCacheV8
+//msgp:marshal ignore dataUsageCacheV2 dataUsageCacheV3 dataUsageCacheV4 dataUsageCacheV5 dataUsageCacheV6 dataUsageCacheV7
 
 // dataUsageCacheV2 contains a cache of data usage entries version 2.
 type dataUsageCacheV2 struct {
@@ -261,12 +244,6 @@ type dataUsageCacheV7 struct {
 	Cache map[string]dataUsageEntryV7
 }
 
-// dataUsageCacheV8 contains a cache of data usage entries version 8.
-type dataUsageCacheV8 struct {
-	Info  dataUsageCacheInfo
-	Cache map[string]dataUsageEntryV8
-}
-
 //msgp:ignore dataUsageEntryInfo
 type dataUsageEntryInfo struct {
 	Name   string
@@ -295,7 +272,6 @@ type dataUsageCacheInfo struct {
 
 func (e *dataUsageEntry) addSizes(summary sizeSummary) {
 	e.Size += summary.totalSize
-	e.HotTierSize += summary.hotTierSize
 	e.Versions += summary.versions
 	e.DeleteMarkers += summary.deleteMarkers
 	e.ObjSizes.add(summary.totalSize)
@@ -315,7 +291,6 @@ func (e *dataUsageEntry) merge(other dataUsageEntry) {
 	e.Versions += other.Versions
 	e.DeleteMarkers += other.DeleteMarkers
 	e.Size += other.Size
-	e.HotTierSize += other.HotTierSize
 
 	for i, v := range other.ObjSizes[:] {
 		e.ObjSizes[i] += v
@@ -456,7 +431,6 @@ func (d *dataUsageCache) dui(path string, buckets []BucketInfo) DataUsageInfo {
 	flat := d.flatten(*e)
 	dui := DataUsageInfo{
 		LastUpdate:              d.Info.LastUpdate,
-		ScannerCycle:            d.Info.NextCycle,
 		ObjectsTotalCount:       flat.Objects,
 		VersionsTotalCount:      flat.Versions,
 		DeleteMarkersTotalCount: flat.DeleteMarkers,
@@ -807,7 +781,6 @@ func (d *dataUsageCache) bucketsUsageInfo(buckets []BucketInfo) map[string]Bucke
 		flat := d.flatten(*e)
 		bui := BucketUsageInfo{
 			Size:                    uint64(flat.Size),
-			HotTierSize:             uint64(max(flat.HotTierSize, 0)),
 			VersionsCount:           flat.Versions,
 			ObjectsCount:            flat.Objects,
 			DeleteMarkersCount:      flat.DeleteMarkers,
@@ -1007,8 +980,8 @@ func (d *dataUsageCache) save(ctx context.Context, store objectIO, name string)
 // Bumping the cache version will drop data from previous versions
 // and write new data with the new version.
 const (
-	dataUsageCacheVerCurrent = 9
+	dataUsageCacheVerCurrent = 8
-	dataUsageCacheVerV8      = 8
+	dataUsageCacheVerV9      = 9 // Retired access-tier cache; only adds the ignored "hts" entry key.
 	dataUsageCacheVerV7      = 7
 	dataUsageCacheVerV6      = 6
 	dataUsageCacheVerV5      = 5
@@ -1210,34 +1183,7 @@ func (d *dataUsageCache) deserialize(r io.Reader) error {
 		}
 
 		return nil
-	case dataUsageCacheVerV8:
+	case dataUsageCacheVerCurrent, dataUsageCacheVerV9:
-		// Zstd compressed.
-		dec, err := zstd.NewReader(r, zstd.WithDecoderConcurrency(2))
-		if err != nil {
-			return err
-		}
-		defer dec.Close()
-		dold := &dataUsageCacheV8{}
-		if err = dold.DecodeMsg(msgp.NewReader(dec)); err != nil {
-			return err
-		}
-		d.Info = dold.Info
-		d.Cache = make(map[string]dataUsageEntry, len(dold.Cache))
-		for k, v := range dold.Cache {
-			d.Cache[k] = dataUsageEntry{
-				Children:      v.Children,
-				Size:          v.Size,
-				Objects:       v.Objects,
-				Versions:      v.Versions,
-				DeleteMarkers: v.DeleteMarkers,
-				ObjSizes:      v.ObjSizes,
-				ObjVersions:   v.ObjVersions,
-				AllTierStats:  v.AllTierStats,
-				Compacted:     v.Compacted,
-			}
-		}
-		return nil
-	case dataUsageCacheVerCurrent:
 		// Zstd compressed.
 		dec, err := zstd.NewReader(r, zstd.WithDecoderConcurrency(2))
 		if err != nil {

cmd/data-usage-cache_gen.go

@@ -1591,145 +1591,6 @@ func (z *dataUsageCacheV7) Msgsize() (s int) {
 	return
 }
 
-// DecodeMsg implements msgp.Decodable
-func (z *dataUsageCacheV8) DecodeMsg(dc *msgp.Reader) (err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, err = dc.ReadMapHeader()
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, err = dc.ReadMapKeyPtr()
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "Info":
-			err = z.Info.DecodeMsg(dc)
-			if err != nil {
-				err = msgp.WrapError(err, "Info")
-				return
-			}
-		case "Cache":
-			var zb0002 uint32
-			zb0002, err = dc.ReadMapHeader()
-			if err != nil {
-				err = msgp.WrapError(err, "Cache")
-				return
-			}
-			if z.Cache == nil {
-				z.Cache = make(map[string]dataUsageEntryV8, zb0002)
-			} else if len(z.Cache) > 0 {
-				clear(z.Cache)
-			}
-			for zb0002 > 0 {
-				zb0002--
-				var za0001 string
-				za0001, err = dc.ReadString()
-				if err != nil {
-					err = msgp.WrapError(err, "Cache")
-					return
-				}
-				var za0002 dataUsageEntryV8
-				err = za0002.DecodeMsg(dc)
-				if err != nil {
-					err = msgp.WrapError(err, "Cache", za0001)
-					return
-				}
-				z.Cache[za0001] = za0002
-			}
-		default:
-			err = dc.Skip()
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	return
-}
-
-// UnmarshalMsg implements msgp.Unmarshaler
-func (z *dataUsageCacheV8) UnmarshalMsg(bts []byte) (o []byte, err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, bts, err = msgp.ReadMapKeyZC(bts)
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "Info":
-			bts, err = z.Info.UnmarshalMsg(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Info")
-				return
-			}
-		case "Cache":
-			var zb0002 uint32
-			zb0002, bts, err = msgp.ReadMapHeaderBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Cache")
-				return
-			}
-			if z.Cache == nil {
-				z.Cache = make(map[string]dataUsageEntryV8, zb0002)
-			} else if len(z.Cache) > 0 {
-				clear(z.Cache)
-			}
-			for zb0002 > 0 {
-				var za0002 dataUsageEntryV8
-				zb0002--
-				var za0001 string
-				za0001, bts, err = msgp.ReadStringBytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "Cache")
-					return
-				}
-				bts, err = za0002.UnmarshalMsg(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "Cache", za0001)
-					return
-				}
-				z.Cache[za0001] = za0002
-			}
-		default:
-			bts, err = msgp.Skip(bts)
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	o = bts
-	return
-}
-
-// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
-func (z *dataUsageCacheV8) Msgsize() (s int) {
-	s = 1 + 5 + z.Info.Msgsize() + 6 + msgp.MapHeaderSize
-	if z.Cache != nil {
-		for za0001, za0002 := range z.Cache {
-			_ = za0002
-			s += msgp.StringPrefixSize + len(za0001) + za0002.Msgsize()
-		}
-	}
-	return
-}
-
 // DecodeMsg implements msgp.Decodable
 func (z *dataUsageEntry) DecodeMsg(dc *msgp.Reader) (err error) {
 	var field []byte
@@ -1762,12 +1623,6 @@ func (z *dataUsageEntry) DecodeMsg(dc *msgp.Reader) (err error) {
 				err = msgp.WrapError(err, "Size")
 				return
 			}
-		case "hts":
-			z.HotTierSize, err = dc.ReadInt64()
-			if err != nil {
-				err = msgp.WrapError(err, "HotTierSize")
-				return
-			}
 		case "os":
 			z.Objects, err = dc.ReadUint64()
 			if err != nil {
@@ -1946,12 +1801,12 @@ func (z *dataUsageEntry) DecodeMsg(dc *msgp.Reader) (err error) {
 // EncodeMsg implements msgp.Encodable
 func (z *dataUsageEntry) EncodeMsg(en *msgp.Writer) (err error) {
 	// check for omitted fields
-	zb0001Len := uint32(10)
+	zb0001Len := uint32(9)
-	var zb0001Mask uint16 /* 10 bits */
+	var zb0001Mask uint16 /* 9 bits */
 	_ = zb0001Mask
 	if z.AllTierStats == nil {
 		zb0001Len--
-		zb0001Mask |= 0x100
+		zb0001Mask |= 0x80
 	}
 	// variable map header, size zb0001Len
 	err = en.Append(0x80 | uint8(zb0001Len))
@@ -1981,16 +1836,6 @@ func (z *dataUsageEntry) EncodeMsg(en *msgp.Writer) (err error) {
 			err = msgp.WrapError(err, "Size")
 			return
 		}
-		// write "hts"
-		err = en.Append(0xa3, 0x68, 0x74, 0x73)
-		if err != nil {
-			return
-		}
-		err = en.WriteInt64(z.HotTierSize)
-		if err != nil {
-			err = msgp.WrapError(err, "HotTierSize")
-			return
-		}
 		// write "os"
 		err = en.Append(0xa2, 0x6f, 0x73)
 		if err != nil {
@@ -2055,7 +1900,7 @@ func (z *dataUsageEntry) EncodeMsg(en *msgp.Writer) (err error) {
 				return
 			}
 		}
-		if (zb0001Mask & 0x100) == 0 { // if not omitted
+		if (zb0001Mask & 0x80) == 0 { // if not omitted
 			// write "ats"
 			err = en.Append(0xa3, 0x61, 0x74, 0x73)
 			if err != nil {
@@ -2136,12 +1981,12 @@ func (z *dataUsageEntry) EncodeMsg(en *msgp.Writer) (err error) {
 func (z *dataUsageEntry) MarshalMsg(b []byte) (o []byte, err error) {
 	o = msgp.Require(b, z.Msgsize())
 	// check for omitted fields
-	zb0001Len := uint32(10)
+	zb0001Len := uint32(9)
-	var zb0001Mask uint16 /* 10 bits */
+	var zb0001Mask uint16 /* 9 bits */
 	_ = zb0001Mask
 	if z.AllTierStats == nil {
 		zb0001Len--
-		zb0001Mask |= 0x100
+		zb0001Mask |= 0x80
 	}
 	// variable map header, size zb0001Len
 	o = append(o, 0x80|uint8(zb0001Len))
@@ -2158,9 +2003,6 @@ func (z *dataUsageEntry) MarshalMsg(b []byte) (o []byte, err error) {
 		// string "sz"
 		o = append(o, 0xa2, 0x73, 0x7a)
 		o = msgp.AppendInt64(o, z.Size)
-		// string "hts"
-		o = append(o, 0xa3, 0x68, 0x74, 0x73)
-		o = msgp.AppendInt64(o, z.HotTierSize)
 		// string "os"
 		o = append(o, 0xa2, 0x6f, 0x73)
 		o = msgp.AppendUint64(o, z.Objects)
@@ -2182,7 +2024,7 @@ func (z *dataUsageEntry) MarshalMsg(b []byte) (o []byte, err error) {
 		for za0002 := range z.ObjVersions {
 			o = msgp.AppendUint64(o, z.ObjVersions[za0002])
 		}
-		if (zb0001Mask & 0x100) == 0 { // if not omitted
+		if (zb0001Mask & 0x80) == 0 { // if not omitted
 			// string "ats"
 			o = append(o, 0xa3, 0x61, 0x74, 0x73)
 			if z.AllTierStats == nil {
@@ -2246,12 +2088,6 @@ func (z *dataUsageEntry) UnmarshalMsg(bts []byte) (o []byte, err error) {
 				err = msgp.WrapError(err, "Size")
 				return
 			}
-		case "hts":
-			z.HotTierSize, bts, err = msgp.ReadInt64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "HotTierSize")
-				return
-			}
 		case "os":
 			z.Objects, bts, err = msgp.ReadUint64Bytes(bts)
 			if err != nil {
@@ -2429,7 +2265,7 @@ func (z *dataUsageEntry) UnmarshalMsg(bts []byte) (o []byte, err error) {
 
 // Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
 func (z *dataUsageEntry) Msgsize() (s int) {
-	s = 1 + 3 + z.Children.Msgsize() + 3 + msgp.Int64Size + 4 + msgp.Int64Size + 3 + msgp.Uint64Size + 3 + msgp.Uint64Size + 4 + msgp.Uint64Size + 4 + msgp.ArrayHeaderSize + (dataUsageBucketLen * (msgp.Uint64Size)) + 3 + msgp.ArrayHeaderSize + (dataUsageVersionLen * (msgp.Uint64Size)) + 4
+	s = 1 + 3 + z.Children.Msgsize() + 3 + msgp.Int64Size + 3 + msgp.Uint64Size + 3 + msgp.Uint64Size + 4 + msgp.Uint64Size + 4 + msgp.ArrayHeaderSize + (dataUsageBucketLen * (msgp.Uint64Size)) + 3 + msgp.ArrayHeaderSize + (dataUsageVersionLen * (msgp.Uint64Size)) + 4
 	if z.AllTierStats == nil {
 		s += msgp.NilSize
 	} else {
@@ -3422,438 +3258,6 @@ func (z *dataUsageEntryV7) Msgsize() (s int) {
 	return
 }
 
-// DecodeMsg implements msgp.Decodable
-func (z *dataUsageEntryV8) DecodeMsg(dc *msgp.Reader) (err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, err = dc.ReadMapHeader()
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	var zb0001Mask uint8 /* 1 bits */
-	_ = zb0001Mask
-	for zb0001 > 0 {
-		zb0001--
-		field, err = dc.ReadMapKeyPtr()
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "ch":
-			err = z.Children.DecodeMsg(dc)
-			if err != nil {
-				err = msgp.WrapError(err, "Children")
-				return
-			}
-		case "sz":
-			z.Size, err = dc.ReadInt64()
-			if err != nil {
-				err = msgp.WrapError(err, "Size")
-				return
-			}
-		case "os":
-			z.Objects, err = dc.ReadUint64()
-			if err != nil {
-				err = msgp.WrapError(err, "Objects")
-				return
-			}
-		case "vs":
-			z.Versions, err = dc.ReadUint64()
-			if err != nil {
-				err = msgp.WrapError(err, "Versions")
-				return
-			}
-		case "dms":
-			z.DeleteMarkers, err = dc.ReadUint64()
-			if err != nil {
-				err = msgp.WrapError(err, "DeleteMarkers")
-				return
-			}
-		case "szs":
-			var zb0002 uint32
-			zb0002, err = dc.ReadArrayHeader()
-			if err != nil {
-				err = msgp.WrapError(err, "ObjSizes")
-				return
-			}
-			if zb0002 != uint32(dataUsageBucketLen) {
-				err = msgp.ArrayError{Wanted: uint32(dataUsageBucketLen), Got: zb0002}
-				return
-			}
-			for za0001 := range z.ObjSizes {
-				z.ObjSizes[za0001], err = dc.ReadUint64()
-				if err != nil {
-					err = msgp.WrapError(err, "ObjSizes", za0001)
-					return
-				}
-			}
-		case "vh":
-			var zb0003 uint32
-			zb0003, err = dc.ReadArrayHeader()
-			if err != nil {
-				err = msgp.WrapError(err, "ObjVersions")
-				return
-			}
-			if zb0003 != uint32(dataUsageVersionLen) {
-				err = msgp.ArrayError{Wanted: uint32(dataUsageVersionLen), Got: zb0003}
-				return
-			}
-			for za0002 := range z.ObjVersions {
-				z.ObjVersions[za0002], err = dc.ReadUint64()
-				if err != nil {
-					err = msgp.WrapError(err, "ObjVersions", za0002)
-					return
-				}
-			}
-		case "ats":
-			if dc.IsNil() {
-				err = dc.ReadNil()
-				if err != nil {
-					err = msgp.WrapError(err, "AllTierStats")
-					return
-				}
-				z.AllTierStats = nil
-			} else {
-				if z.AllTierStats == nil {
-					z.AllTierStats = new(allTierStats)
-				}
-				var zb0004 uint32
-				zb0004, err = dc.ReadMapHeader()
-				if err != nil {
-					err = msgp.WrapError(err, "AllTierStats")
-					return
-				}
-				for zb0004 > 0 {
-					zb0004--
-					field, err = dc.ReadMapKeyPtr()
-					if err != nil {
-						err = msgp.WrapError(err, "AllTierStats")
-						return
-					}
-					switch msgp.UnsafeString(field) {
-					case "ts":
-						var zb0005 uint32
-						zb0005, err = dc.ReadMapHeader()
-						if err != nil {
-							err = msgp.WrapError(err, "AllTierStats", "Tiers")
-							return
-						}
-						if z.AllTierStats.Tiers == nil {
-							z.AllTierStats.Tiers = make(map[string]tierStats, zb0005)
-						} else if len(z.AllTierStats.Tiers) > 0 {
-							clear(z.AllTierStats.Tiers)
-						}
-						for zb0005 > 0 {
-							zb0005--
-							var za0003 string
-							za0003, err = dc.ReadString()
-							if err != nil {
-								err = msgp.WrapError(err, "AllTierStats", "Tiers")
-								return
-							}
-							var za0004 tierStats
-							var zb0006 uint32
-							zb0006, err = dc.ReadMapHeader()
-							if err != nil {
-								err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
-								return
-							}
-							for zb0006 > 0 {
-								zb0006--
-								field, err = dc.ReadMapKeyPtr()
-								if err != nil {
-									err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
-									return
-								}
-								switch msgp.UnsafeString(field) {
-								case "ts":
-									za0004.TotalSize, err = dc.ReadUint64()
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
-										return
-									}
-								case "nv":
-									za0004.NumVersions, err = dc.ReadInt()
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
-										return
-									}
-								case "no":
-									za0004.NumObjects, err = dc.ReadInt()
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
-										return
-									}
-								default:
-									err = dc.Skip()
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
-										return
-									}
-								}
-							}
-							z.AllTierStats.Tiers[za0003] = za0004
-						}
-					default:
-						err = dc.Skip()
-						if err != nil {
-							err = msgp.WrapError(err, "AllTierStats")
-							return
-						}
-					}
-				}
-			}
-			zb0001Mask |= 0x1
-		case "c":
-			z.Compacted, err = dc.ReadBool()
-			if err != nil {
-				err = msgp.WrapError(err, "Compacted")
-				return
-			}
-		default:
-			err = dc.Skip()
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	// Clear omitted fields.
-	if (zb0001Mask & 0x1) == 0 {
-		z.AllTierStats = nil
-	}
-
-	return
-}
-
-// UnmarshalMsg implements msgp.Unmarshaler
-func (z *dataUsageEntryV8) UnmarshalMsg(bts []byte) (o []byte, err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	var zb0001Mask uint8 /* 1 bits */
-	_ = zb0001Mask
-	for zb0001 > 0 {
-		zb0001--
-		field, bts, err = msgp.ReadMapKeyZC(bts)
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "ch":
-			bts, err = z.Children.UnmarshalMsg(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Children")
-				return
-			}
-		case "sz":
-			z.Size, bts, err = msgp.ReadInt64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Size")
-				return
-			}
-		case "os":
-			z.Objects, bts, err = msgp.ReadUint64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Objects")
-				return
-			}
-		case "vs":
-			z.Versions, bts, err = msgp.ReadUint64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Versions")
-				return
-			}
-		case "dms":
-			z.DeleteMarkers, bts, err = msgp.ReadUint64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "DeleteMarkers")
-				return
-			}
-		case "szs":
-			var zb0002 uint32
-			zb0002, bts, err = msgp.ReadArrayHeaderBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "ObjSizes")
-				return
-			}
-			if zb0002 != uint32(dataUsageBucketLen) {
-				err = msgp.ArrayError{Wanted: uint32(dataUsageBucketLen), Got: zb0002}
-				return
-			}
-			for za0001 := range z.ObjSizes {
-				z.ObjSizes[za0001], bts, err = msgp.ReadUint64Bytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "ObjSizes", za0001)
-					return
-				}
-			}
-		case "vh":
-			var zb0003 uint32
-			zb0003, bts, err = msgp.ReadArrayHeaderBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "ObjVersions")
-				return
-			}
-			if zb0003 != uint32(dataUsageVersionLen) {
-				err = msgp.ArrayError{Wanted: uint32(dataUsageVersionLen), Got: zb0003}
-				return
-			}
-			for za0002 := range z.ObjVersions {
-				z.ObjVersions[za0002], bts, err = msgp.ReadUint64Bytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "ObjVersions", za0002)
-					return
-				}
-			}
-		case "ats":
-			if msgp.IsNil(bts) {
-				bts, err = msgp.ReadNilBytes(bts)
-				if err != nil {
-					return
-				}
-				z.AllTierStats = nil
-			} else {
-				if z.AllTierStats == nil {
-					z.AllTierStats = new(allTierStats)
-				}
-				var zb0004 uint32
-				zb0004, bts, err = msgp.ReadMapHeaderBytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "AllTierStats")
-					return
-				}
-				for zb0004 > 0 {
-					zb0004--
-					field, bts, err = msgp.ReadMapKeyZC(bts)
-					if err != nil {
-						err = msgp.WrapError(err, "AllTierStats")
-						return
-					}
-					switch msgp.UnsafeString(field) {
-					case "ts":
-						var zb0005 uint32
-						zb0005, bts, err = msgp.ReadMapHeaderBytes(bts)
-						if err != nil {
-							err = msgp.WrapError(err, "AllTierStats", "Tiers")
-							return
-						}
-						if z.AllTierStats.Tiers == nil {
-							z.AllTierStats.Tiers = make(map[string]tierStats, zb0005)
-						} else if len(z.AllTierStats.Tiers) > 0 {
-							clear(z.AllTierStats.Tiers)
-						}
-						for zb0005 > 0 {
-							var za0004 tierStats
-							zb0005--
-							var za0003 string
-							za0003, bts, err = msgp.ReadStringBytes(bts)
-							if err != nil {
-								err = msgp.WrapError(err, "AllTierStats", "Tiers")
-								return
-							}
-							var zb0006 uint32
-							zb0006, bts, err = msgp.ReadMapHeaderBytes(bts)
-							if err != nil {
-								err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
-								return
-							}
-							for zb0006 > 0 {
-								zb0006--
-								field, bts, err = msgp.ReadMapKeyZC(bts)
-								if err != nil {
-									err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
-									return
-								}
-								switch msgp.UnsafeString(field) {
-								case "ts":
-									za0004.TotalSize, bts, err = msgp.ReadUint64Bytes(bts)
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
-										return
-									}
-								case "nv":
-									za0004.NumVersions, bts, err = msgp.ReadIntBytes(bts)
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
-										return
-									}
-								case "no":
-									za0004.NumObjects, bts, err = msgp.ReadIntBytes(bts)
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
-										return
-									}
-								default:
-									bts, err = msgp.Skip(bts)
-									if err != nil {
-										err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
-										return
-									}
-								}
-							}
-							z.AllTierStats.Tiers[za0003] = za0004
-						}
-					default:
-						bts, err = msgp.Skip(bts)
-						if err != nil {
-							err = msgp.WrapError(err, "AllTierStats")
-							return
-						}
-					}
-				}
-			}
-			zb0001Mask |= 0x1
-		case "c":
-			z.Compacted, bts, err = msgp.ReadBoolBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Compacted")
-				return
-			}
-		default:
-			bts, err = msgp.Skip(bts)
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	// Clear omitted fields.
-	if (zb0001Mask & 0x1) == 0 {
-		z.AllTierStats = nil
-	}
-
-	o = bts
-	return
-}
-
-// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
-func (z *dataUsageEntryV8) Msgsize() (s int) {
-	s = 1 + 3 + z.Children.Msgsize() + 3 + msgp.Int64Size + 3 + msgp.Uint64Size + 3 + msgp.Uint64Size + 4 + msgp.Uint64Size + 4 + msgp.ArrayHeaderSize + (dataUsageBucketLen * (msgp.Uint64Size)) + 3 + msgp.ArrayHeaderSize + (dataUsageVersionLen * (msgp.Uint64Size)) + 4
-	if z.AllTierStats == nil {
-		s += msgp.NilSize
-	} else {
-		s += 1 + 3 + msgp.MapHeaderSize
-		if z.AllTierStats.Tiers != nil {
-			for za0003, za0004 := range z.AllTierStats.Tiers {
-				_ = za0004
-				s += msgp.StringPrefixSize + len(za0003) + 1 + 3 + msgp.Uint64Size + 3 + msgp.IntSize + 3 + msgp.IntSize
-			}
-		}
-	}
-	s += 2 + msgp.BoolSize
-	return
-}
-
 // DecodeMsg implements msgp.Decodable
 func (z *dataUsageHash) DecodeMsg(dc *msgp.Reader) (err error) {
 	{

cmd/data-usage-utils.go

@@ -46,8 +46,7 @@ type BucketTargetUsageInfo struct {
 // - total objects in a bucket
 // - object size histogram per bucket
 type BucketUsageInfo struct {
-	Size        uint64 `json:"size"`
+	Size uint64 `json:"size"`
-	HotTierSize uint64 `json:"hotTierSize,omitempty"`
 	// Following five fields suffixed with V1 are here for backward compatibility
 	// Total Size for objects that have not yet been replicated
 	ReplicationPendingSizeV1 uint64 `json:"objectsPendingReplicationTotalSize"`
@@ -79,10 +78,6 @@ type DataUsageInfo struct {
 	// LastUpdate is the timestamp of when the data usage info was last updated.
 	// This does not indicate a full scan.
 	LastUpdate time.Time `json:"lastUpdate"`
-	// ScannerCycle changes only after a complete scanner pass. Background
-	// consumers use it to distinguish a partial cache update from a baseline
-	// that has visited every bucket and server pool.
-	ScannerCycle uint32 `json:"scannerCycle,omitempty"`
 
 	// Objects total count across all buckets
 	ObjectsTotalCount uint64 `json:"objectsCount"`

cmd/erasure-multipart.go

@@ -1179,7 +1179,7 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
 		switch {
 		case gerr == nil:
 			reconcileStoredObjectLock(fi.Metadata, storedObjectLockState(curr.UserDefined))
-			reconcileStoredObjectTags(fi.Metadata, curr.UserDefined)
+			reconcileStoredObjectTags(fi.Metadata, curr.UserTags, curr.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
 		case isErrVersionNotFound(gerr) || isErrObjectNotFound(gerr):
 			// No existing version to order against: keep the upload's own accepted
 			// lock, including a pre-upgrade upload that persisted values without

cmd/erasure-object.go

@@ -135,7 +135,7 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
 
 	if dstOpts.ReplicaLockReconcile {
 		reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(fi.Metadata))
-		reconcileStoredObjectTags(srcInfo.UserDefined, fi.Metadata)
+		reconcileStoredObjectTags(srcInfo.UserDefined, fi.Metadata[xhttp.AmzObjectTagging], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
 	}
 
 	filterOnlineDisksInplace(fi, metaArr, onlineDisks)
@@ -1311,7 +1311,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
 		// existing version contributes independently ordered lock and tags.
 		if opts.ReplicaLockReconcile && err == nil {
 			reconcileStoredObjectLock(opts.UserDefined, storedObjectLockState(obj.UserDefined))
-			reconcileStoredObjectTags(opts.UserDefined, obj.UserDefined)
+			reconcileStoredObjectTags(opts.UserDefined, obj.UserTags, obj.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
 		}
 	}
 

cmd/erasure-server-pool-consistency.go

@@ -126,10 +126,9 @@ func mergedPoolObjectInfo(copies []PoolObjInfo) ObjectInfo {
 		stamp, _ := time.Parse(time.RFC3339Nano, ts)
 		if olderThan(oi.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], stamp) {
 			oi.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = ts
-			oi.UserDefined[xhttp.AmzObjectTagging] = copy.ObjInfo.UserDefined[xhttp.AmzObjectTagging]
+			oi.UserTags = copy.ObjInfo.UserTags
 		}
 	}
-	oi.UserTags = oi.UserDefined[xhttp.AmzObjectTagging]
 	return oi
 }
 
@@ -190,6 +189,12 @@ func (z *erasureServerPools) updatePoolMetadata(ctx context.Context, bucket, obj
 			changes[key] = ""
 		}
 	}
+	// Metadata callbacks may explicitly replace tags in the raw write map.
+	// Otherwise retain the merged value from the ObjectInfo read model.
+	tags, ok := updated.UserDefined[xhttp.AmzObjectTagging]
+	if !ok {
+		tags = updated.UserTags
+	}
 	state := storedObjectLockState(updated.UserDefined)
 	opts.VersionID = updated.VersionID
 	if opts.VersionID == "" {
@@ -199,11 +204,13 @@ func (z *erasureServerPools) updatePoolMetadata(ctx context.Context, bucket, obj
 	opts.EvalMetadataFn = func(oi *ObjectInfo, _ error) (ReplicateDecision, error) {
 		maps.Copy(oi.UserDefined, changes)
 		replaceObjectLockMetadata(oi.UserDefined, state)
-		for _, key := range []string{xhttp.AmzObjectTagging, ReservedMetadataPrefixLower + TaggingTimestamp} {
+		// Reassemble the tag value and its ordering timestamp for storage.
-			value, exists := updated.UserDefined[key]
+		if tags != "" || oi.UserTags != "" {
-			if exists || oi.UserDefined[key] != "" {
+			oi.UserDefined[xhttp.AmzObjectTagging] = tags
-				oi.UserDefined[key] = value
+		}
-			}
+		key := ReservedMetadataPrefixLower + TaggingTimestamp
+		if value, exists := updated.UserDefined[key]; exists || oi.UserDefined[key] != "" {
+			oi.UserDefined[key] = value
 		}
 		return ReplicateDecision{}, nil
 	}
@@ -220,16 +227,18 @@ func (z *erasureServerPools) updatePoolMetadata(ctx context.Context, bucket, obj
 	return primary, nil
 }
 
-func reconcileStoredObjectTags(metadata, stored map[string]string) {
+// Pass the stored tag value explicitly: ObjectInfo.UserDefined excludes it,
+// whereas FileInfo.Metadata retains the raw storage key.
+func reconcileStoredObjectTags(metadata map[string]string, storedTags, storedTimestamp string) {
 	key := ReservedMetadataPrefixLower + TaggingTimestamp
-	stamp, err := time.Parse(time.RFC3339Nano, stored[key])
+	stamp, err := time.Parse(time.RFC3339Nano, storedTimestamp)
 	if err != nil {
 		return
 	}
 	incoming, err := time.Parse(time.RFC3339Nano, metadata[key])
 	if err != nil || !stamp.Before(incoming) {
-		metadata[key] = stored[key]
+		metadata[key] = storedTimestamp
-		metadata[xhttp.AmzObjectTagging] = stored[xhttp.AmzObjectTagging]
+		metadata[xhttp.AmzObjectTagging] = storedTags
 	}
 }
 
@@ -287,33 +296,51 @@ func (z *erasureServerPools) retireReplicaCopies(ctx context.Context, bucket, ob
 	return nil
 }
 
-// deleteObjectConditional evaluates the condition once against the logical
+// deleteObjectReconciled evaluates any condition once against the logical
 // version, then removes all its copies under the same lock as pooled writers.
-func (z *erasureServerPools) deleteObjectConditional(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
+func (z *erasureServerPools) deleteObjectReconciled(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
 	copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
 	if err != nil {
 		return ObjectInfo{}, err
 	}
 	primary := copies[0]
-	if opts.CheckPrecondFn(primary.ObjInfo) {
+	if opts.CheckPrecondFn != nil && opts.CheckPrecondFn(primary.ObjInfo) {
 		return ObjectInfo{}, PreConditionFailed{}
 	}
 	opts.CheckPrecondFn = nil
 	opts.NoLock = true
 	if opts.EvalRetentionBypassFn != nil || opts.EvalMetadataFn != nil {
-		versions, err := z.metadataPoolInfos(ctx, bucket, object, opts)
+		logical := primary.ObjInfo
-		if err != nil {
+		var gerr error
-			return ObjectInfo{}, err
+		switch {
+		case logical.DeleteMarker:
+			// Markers can be deleted by version ID. Match the set layer's
+			// callback inputs instead of rejecting them as metadata updates.
+			gerr = toObjectErr(errMethodNotAllowed, bucket, object)
+			if opts.VersionID == "" || opts.DeleteMarker {
+				gerr = toObjectErr(errFileNotFound, bucket, object)
+			}
+		case opts.VersionID != "":
+			// An addressed version already resolved every copy above.
+			logical = mergedPoolObjectInfo(copies)
+		default:
+			versions, err := z.metadataPoolInfos(ctx, bucket, object, opts)
+			if err != nil {
+				return ObjectInfo{}, err
+			}
+			logical = mergedPoolObjectInfo(versions)
 		}
-		logical := mergedPoolObjectInfo(versions)
+		// Keep the retention gate first. These callbacks independently evaluate
+		// the logical version and run once before any deletion; the handler only
+		// sweeps metadata's transition state after a successful delete.
 		if opts.EvalRetentionBypassFn != nil {
-			if err := opts.EvalRetentionBypassFn(logical, nil); err != nil {
+			if err := opts.EvalRetentionBypassFn(logical, gerr); err != nil {
 				return ObjectInfo{}, err
 			}
 			opts.EvalRetentionBypassFn = nil
 		}
 		if opts.EvalMetadataFn != nil {
-			decision, err := opts.EvalMetadataFn(&logical, nil)
+			decision, err := opts.EvalMetadataFn(&logical, gerr)
 			if err != nil {
 				return ObjectInfo{}, err
 			}

cmd/erasure-server-pool-consistency_test.go

@@ -24,12 +24,16 @@ import (
 	"fmt"
 	"io"
 	"maps"
+	"net/http"
+	"net/http/httptest"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"testing"
 	"time"
 
 	madmin "github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/bucket/replication"
 	xhttp "github.com/minio/minio/internal/http"
 )
 
@@ -62,6 +66,490 @@ func putConsistencyObject(t *testing.T, z *erasureServerPools, bucket, object st
 	return oi
 }
 
+func consistencyRequest(t *testing.T, router http.Handler, method, bucket, object, version string) *httptest.ResponseRecorder {
+	t.Helper()
+	url := getGetObjectURL("", bucket, object)
+	if version != "" {
+		url += "?versionId=" + version
+	}
+	req, err := newTestSignedRequestV4(method, url, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	rec := httptest.NewRecorder()
+	router.ServeHTTP(rec, req)
+	return rec
+}
+
+// Exercise the handler's real replication/retention callbacks, including the
+// MethodNotAllowed metadata returned for an explicitly addressed delete marker.
+func TestPoolsDeleteVersionAPI(t *testing.T) {
+	z, _ := consistencyPools(t)
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{VersioningEnabled: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, kind := range []string{"uuid", "null", "marker"} {
+		for primary := range 2 {
+			t.Run(fmt.Sprintf("%s/primary=%d", kind, primary), func(t *testing.T) {
+				object := fmt.Sprintf("%s-%d", kind, primary)
+				opts := ObjectOptions{Versioned: kind != "null", MTime: UTCNow().Add(-time.Hour)}
+				var addressed ObjectInfo
+				if kind == "marker" {
+					opts.VersionID, opts.DeleteMarker = mustGetUUID(), true
+					addressed, err = z.serverPools[primary].DeleteObject(ctx, bucket, object, opts)
+					if err != nil {
+						t.Fatal(err)
+					}
+				} else {
+					addressed = putConsistencyObject(t, z, bucket, object, primary, "payload", opts)
+				}
+				version := addressed.VersionID
+				if version == "" {
+					version = nullVersionID
+				}
+				opts.VersionID = version
+				opts.MTime = addressed.ModTime.Add(-time.Minute)
+				if kind == "marker" {
+					opts.DeleteMarker = true
+					if _, err := z.serverPools[1-primary].DeleteObject(ctx, bucket, object, opts); err != nil {
+						t.Fatal(err)
+					}
+				} else {
+					putConsistencyObject(t, z, bucket, object, 1-primary, "payload", opts)
+				}
+				latest := putConsistencyObject(t, z, bucket, object, 1-primary, "keep-latest", ObjectOptions{Versioned: true})
+				rec := consistencyRequest(t, router, http.MethodDelete, bucket, object, version)
+				if rec.Code != http.StatusNoContent {
+					t.Fatalf("DELETE: %d %s", rec.Code, rec.Body.String())
+				}
+				if kind == "marker" && (strings.Join(rec.Header()[xhttp.AmzDeleteMarker], "") != "true" || strings.Join(rec.Header()[xhttp.AmzVersionID], "") != version) {
+					t.Errorf("lost deleted marker response headers: %v", rec.Header())
+				}
+				for i, pool := range z.serverPools {
+					if _, err := pool.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: version}); !isErrVersionNotFound(err) {
+						t.Errorf("pool %d retained addressed version: %v", i, err)
+					}
+				}
+				for _, method := range []string{http.MethodGet, http.MethodHead} {
+					if rec := consistencyRequest(t, router, method, bucket, object, version); rec.Code != http.StatusNotFound {
+						t.Errorf("%s after DELETE: %d %s", method, rec.Code, rec.Body.String())
+					}
+				}
+				if got, err := z.GetObjectInfo(ctx, bucket, object, ObjectOptions{}); err != nil || got.VersionID != latest.VersionID {
+					t.Errorf("DELETE changed another version: %+v, %v", got, err)
+				}
+				if rec := consistencyRequest(t, router, http.MethodDelete, bucket, object, version); rec.Code != http.StatusNoContent {
+					t.Errorf("idempotent retry: %d %s", rec.Code, rec.Body.String())
+				}
+			})
+		}
+	}
+	if rec := consistencyRequest(t, router, http.MethodDelete, bucket, "absent-key", mustGetUUID()); rec.Code != http.StatusNoContent {
+		t.Errorf("absent key: %d %s", rec.Code, rec.Body.String())
+	}
+}
+
+type consistencyReadFaultDisk struct {
+	StorageAPI
+	bucket, object string
+}
+
+func (d consistencyReadFaultDisk) ReadVersion(ctx context.Context, origvolume, volume, path, version string, opts ReadOptions) (FileInfo, error) {
+	if volume == d.bucket && path == d.object {
+		return FileInfo{}, errDiskNotFound
+	}
+	return d.StorageAPI.ReadVersion(ctx, origvolume, volume, path, version, opts)
+}
+
+func (d consistencyReadFaultDisk) ReadXL(ctx context.Context, volume, path string, readData bool) (RawFileInfo, error) {
+	if volume == d.bucket && path == d.object {
+		return RawFileInfo{}, errDiskNotFound
+	}
+	return d.StorageAPI.ReadXL(ctx, volume, path, readData)
+}
+
+func TestPoolsDeleteVersionUnreadablePool(t *testing.T) {
+	z, _ := consistencyPools(t)
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{VersioningEnabled: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, holding := range []bool{false, true} {
+		for failed := range 2 {
+			t.Run(fmt.Sprintf("holding=%t/pool=%d", holding, failed), func(t *testing.T) {
+				object := fmt.Sprintf("unreadable-%t-%d", holding, failed)
+				oi := putConsistencyObject(t, z, bucket, object, 1-failed, "payload", ObjectOptions{Versioned: true})
+				if holding {
+					putConsistencyObject(t, z, bucket, object, failed, "payload", ObjectOptions{Versioned: true, VersionID: oi.VersionID, MTime: oi.ModTime})
+				}
+				set := z.serverPools[failed].getHashedSet(object)
+				getDisks := set.getDisks
+				faulty := append([]StorageAPI(nil), getDisks()...)
+				for i := range faulty {
+					faulty[i] = consistencyReadFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object}
+				}
+				set.getDisks = func() []StorageAPI { return faulty }
+				defer func() { set.getDisks = getDisks }()
+				if rec := consistencyRequest(t, router, http.MethodDelete, bucket, object, oi.VersionID); rec.Code != http.StatusServiceUnavailable {
+					t.Errorf("unreadable pool DELETE: %d %s", rec.Code, rec.Body.String())
+				}
+				if _, err := z.serverPools[1-failed].GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: oi.VersionID}); err != nil {
+					t.Errorf("DELETE lost the readable copy: %v", err)
+				}
+				set.getDisks = getDisks
+				if rec := consistencyRequest(t, router, http.MethodDelete, bucket, object, oi.VersionID); rec.Code != http.StatusNoContent {
+					t.Errorf("recovered pool retry: %d %s", rec.Code, rec.Body.String())
+				}
+				for i, pool := range z.serverPools {
+					if _, err := pool.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: oi.VersionID}); !isErrVersionNotFound(err) {
+						t.Errorf("pool %d retained version after retry: %v", i, err)
+					}
+				}
+			})
+		}
+	}
+}
+
+type consistencyDeleteCountDisk struct {
+	StorageAPI
+	bucket, object         string
+	metadataReads, deletes *atomic.Int32
+}
+
+func (d consistencyDeleteCountDisk) ReadVersion(ctx context.Context, origvolume, volume, path, version string, opts ReadOptions) (FileInfo, error) {
+	if volume == d.bucket && path == d.object {
+		d.metadataReads.Add(1)
+	}
+	return d.StorageAPI.ReadVersion(ctx, origvolume, volume, path, version, opts)
+}
+
+func (d consistencyDeleteCountDisk) DeleteVersion(ctx context.Context, volume, path string, fi FileInfo, force bool, opts DeleteOptions) error {
+	if volume == d.bucket && path == d.object {
+		d.deletes.Add(1)
+	}
+	return d.StorageAPI.DeleteVersion(ctx, volume, path, fi, force, opts)
+}
+
+func TestPoolsDeleteVersionSingleCopy(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	for primary := range 2 {
+		t.Run(fmt.Sprintf("pool=%d", primary), func(t *testing.T) {
+			object := fmt.Sprintf("single-copy-%d", primary)
+			oi := putConsistencyObject(t, z, bucket, object, primary, "payload", ObjectOptions{Versioned: true})
+			var metadataReads, deletes [2]atomic.Int32
+			for i, pool := range z.serverPools {
+				set := pool.getHashedSet(object)
+				getDisks := set.getDisks
+				disks := append([]StorageAPI(nil), getDisks()...)
+				for j := range disks {
+					disks[j] = consistencyDeleteCountDisk{StorageAPI: disks[j], bucket: bucket, object: object, metadataReads: &metadataReads[i], deletes: &deletes[i]}
+				}
+				set.getDisks = func() []StorageAPI { return disks }
+				defer func() { set.getDisks = getDisks }()
+			}
+			metadata, retention := 0, 0
+			_, err := z.DeleteObject(t.Context(), bucket, object, ObjectOptions{
+				Versioned: true, VersionID: oi.VersionID,
+				EvalMetadataFn: func(current *ObjectInfo, err error) (ReplicateDecision, error) {
+					metadata++
+					if err != nil || current.VersionID != oi.VersionID {
+						t.Errorf("metadata callback: %+v, %v", current, err)
+					}
+					// Count preflight reads before the physical delete reads its own metadata.
+					for i := range metadataReads {
+						if got := metadataReads[i].Load(); got != 16 {
+							t.Errorf("pool %d metadata reads before callbacks = %d; want once per disk (16)", i, got)
+						}
+					}
+					return ReplicateDecision{}, nil
+				},
+				EvalRetentionBypassFn: func(current ObjectInfo, err error) error {
+					retention++
+					return err
+				},
+			})
+			if err != nil || metadata != 1 || retention != 1 {
+				t.Fatalf("DELETE: %v, metadata=%d retention=%d", err, metadata, retention)
+			}
+			if deletes[primary].Load() != 16 || deletes[1-primary].Load() != 0 {
+				t.Errorf("physical deletes per pool = %d, %d; want once per holding disk only", deletes[0].Load(), deletes[1].Load())
+			}
+		})
+	}
+}
+
+func TestPoolsDeleteVersionReplicationPurge(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	for _, marker := range []bool{false, true} {
+		t.Run(fmt.Sprintf("marker=%t", marker), func(t *testing.T) {
+			object := fmt.Sprintf("local-replication-purge-%t", marker)
+			version := mustGetUUID()
+			for _, pool := range z.serverPools {
+				opts := ObjectOptions{Versioned: true, VersionID: version, DeleteMarker: marker}
+				var err error
+				if marker {
+					_, err = pool.DeleteObject(t.Context(), bucket, object, opts)
+				} else {
+					_, err = pool.PutObject(t.Context(), bucket, object, mustGetPutObjReader(t, strings.NewReader("payload"), 7, "", ""), opts)
+				}
+				if err != nil {
+					t.Fatal(err)
+				}
+			}
+			dsc := ReplicateDecision{}
+			dsc.Set(newReplicateTargetDecision("arn1", true, false))
+			opts := ObjectOptions{Versioned: true, VersionID: version}
+			for attempt := range 2 {
+				calls := 0
+				opts.EvalMetadataFn = func(current *ObjectInfo, gerr error) (ReplicateDecision, error) {
+					calls++
+					wantMethodNotAllowed := marker || attempt > 0
+					if (wantMethodNotAllowed && !isErrMethodNotAllowed(gerr)) || (!wantMethodNotAllowed && gerr != nil) {
+						t.Errorf("pending purge callback lost set-layer read error: %v", gerr)
+					}
+					return dsc, nil
+				}
+				got, err := z.DeleteObject(t.Context(), bucket, object, opts)
+				if err != nil || calls != 1 || got.VersionPurgeStatus != replication.VersionPurgePending || got.replicationDecision != dsc.String() {
+					t.Fatalf("replication scheduling result: %+v, %v, calls=%d", got, err, calls)
+				}
+				for i, pool := range z.serverPools {
+					got, err := pool.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: version})
+					if (err != nil && (!got.DeleteMarker || !isErrMethodNotAllowed(err))) || got.VersionPurgeStatus != replication.VersionPurgePending {
+						t.Errorf("pool %d lost pending purge: %+v, %v", i, got, err)
+					}
+				}
+			}
+			// The local worker completes the purge without ReplicationRequest.
+			// Both pending copies must be removed, including a pending marker purge.
+			opts.EvalMetadataFn = nil
+			opts.DeleteReplication = ReplicationState{
+				VersionPurgeStatusInternal: "arn1=COMPLETED;",
+				PurgeTargets:               map[string]VersionPurgeStatusType{"arn1": replication.VersionPurgeComplete},
+			}
+			if _, err := z.DeleteObject(t.Context(), bucket, object, opts); err != nil {
+				t.Fatal(err)
+			}
+			for i, pool := range z.serverPools {
+				if _, err := pool.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: version}); !isErrVersionNotFound(err) {
+					t.Errorf("pool %d retained completed replication purge: %v", i, err)
+				}
+			}
+		})
+	}
+}
+
+func TestPoolsDeleteVersionCleanupFailure(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	for primary := range 2 {
+		t.Run(fmt.Sprintf("primary=%d", primary), func(t *testing.T) {
+			object := fmt.Sprintf("cleanup-failure-%d", primary)
+			oi := putConsistencyObject(t, z, bucket, object, primary, "payload", ObjectOptions{Versioned: true})
+			putConsistencyObject(t, z, bucket, object, 1-primary, "payload", ObjectOptions{
+				Versioned: true, VersionID: oi.VersionID, MTime: oi.ModTime.Add(-time.Minute),
+			})
+			set := z.serverPools[1-primary].getHashedSet(object)
+			getDisks := set.getDisks
+			faulty := append([]StorageAPI(nil), getDisks()...)
+			for i := range faulty {
+				faulty[i] = consistencyDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object, version: oi.VersionID}
+			}
+			set.getDisks = func() []StorageAPI { return faulty }
+			defer func() { set.getDisks = getDisks }()
+			opts := ObjectOptions{Versioned: true, VersionID: oi.VersionID}
+			if _, err := z.DeleteObject(t.Context(), bucket, object, opts); err == nil {
+				t.Error("acknowledged DELETE despite failed secondary cleanup")
+			}
+			if got, err := z.serverPools[primary].GetObjectInfo(t.Context(), bucket, object, opts); err != nil || got.ETag != oi.ETag {
+				t.Errorf("lost authoritative copy after secondary failure: %+v, %v", got, err)
+			}
+			set.getDisks = getDisks
+			if _, err := z.DeleteObject(t.Context(), bucket, object, opts); err != nil {
+				t.Fatal(err)
+			}
+			for i, pool := range z.serverPools {
+				if _, err := pool.GetObjectInfo(t.Context(), bucket, object, opts); !isErrVersionNotFound(err) {
+					t.Errorf("pool %d retained version after retry: %v", i, err)
+				}
+			}
+		})
+	}
+}
+
+func TestPoolsDeleteVersionCallbacks(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	for _, marker := range []bool{false, true} {
+		t.Run(fmt.Sprintf("marker=%t", marker), func(t *testing.T) {
+			object := fmt.Sprintf("callbacks-%t", marker)
+			old, recent := "2026-09-09T09:00:00Z", "2026-09-09T10:00:00Z"
+			oi := putConsistencyObject(t, z, bucket, object, 0, "payload", ObjectOptions{
+				Versioned: true, UserDefined: poolLockMetadata("GOVERNANCE", "OFF", recent, old),
+			})
+			putConsistencyObject(t, z, bucket, object, 1, "payload", ObjectOptions{
+				Versioned: true, VersionID: oi.VersionID, MTime: oi.ModTime,
+				UserDefined: poolLockMetadata("", "ON", old, recent),
+			})
+			if marker {
+				markerOpts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow()}
+				for _, pool := range z.serverPools {
+					var err error
+					oi, err = pool.DeleteObject(t.Context(), bucket, object, markerOpts)
+					if err != nil {
+						t.Fatal(err)
+					}
+				}
+			}
+			check := func(current ObjectInfo, err error) {
+				t.Helper()
+				if current.VersionID != oi.VersionID || current.DeleteMarker != marker || (marker && !isErrMethodNotAllowed(err)) || (!marker && err != nil) {
+					t.Errorf("wrong callback version or read error: %+v, %v", current, err)
+				}
+				if !marker {
+					state := storedObjectLockState(current.UserDefined)
+					if state.mode != "GOVERNANCE" || state.legalHold != "ON" {
+						t.Errorf("callback did not reconcile independently ordered lock state: %+v", state)
+					}
+				}
+			}
+			for _, reject := range []string{"retention", "metadata", "none"} {
+				metadata, retention := 0, 0
+				denied := errors.New("callback denied deletion")
+				opts := ObjectOptions{
+					Versioned: true, VersionID: oi.VersionID,
+					EvalRetentionBypassFn: func(current ObjectInfo, err error) error {
+						retention++
+						check(current, err)
+						if reject == "retention" {
+							return denied
+						}
+						return nil
+					},
+					EvalMetadataFn: func(current *ObjectInfo, err error) (ReplicateDecision, error) {
+						metadata++
+						check(*current, err)
+						if reject == "metadata" {
+							return ReplicateDecision{}, denied
+						}
+						return ReplicateDecision{}, nil
+					},
+				}
+				_, err := z.DeleteObject(t.Context(), bucket, object, opts)
+				if reject == "none" {
+					if err != nil || metadata != 1 || retention != 1 {
+						t.Fatalf("DELETE: %v, metadata=%d retention=%d", err, metadata, retention)
+					}
+				} else if !errors.Is(err, denied) {
+					t.Fatalf("%s rejection was lost: %v", reject, err)
+				}
+				for i, pool := range z.serverPools {
+					current, err := pool.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: oi.VersionID})
+					if reject == "none" {
+						if !isErrVersionNotFound(err) {
+							t.Errorf("pool %d retained deleted version: %v", i, err)
+						}
+					} else if current.VersionID != oi.VersionID || (err != nil && (!marker || !isErrMethodNotAllowed(err))) {
+						t.Errorf("pool %d changed despite %s rejection: %+v, %v", i, reject, current, err)
+					}
+				}
+			}
+		})
+	}
+}
+
+func TestPoolsDeleteVersionSpecialCalls(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	t.Run("incoming-marker-replication", func(t *testing.T) {
+		const object = "incoming-marker"
+		opts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, ReplicationRequest: true}
+		opts.SetReplicaStatus(replication.Replica)
+		if _, err := z.DeleteObject(t.Context(), bucket, object, opts); err != nil {
+			t.Fatalf("replication could not create an absent marker: %v", err)
+		}
+		if got, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: opts.VersionID}); !isErrMethodNotAllowed(err) || !got.DeleteMarker {
+			t.Fatalf("replicated marker missing: %+v, %v", got, err)
+		}
+	})
+	for _, rebalance := range []bool{false, true} {
+		t.Run(fmt.Sprintf("movement/rebalance=%t", rebalance), func(t *testing.T) {
+			object := fmt.Sprintf("movement-%t", rebalance)
+			putConsistencyObject(t, z, bucket, object, 1, "destination", ObjectOptions{Versioned: true})
+			opts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow()}
+			if _, err := z.serverPools[0].DeleteObject(t.Context(), bucket, object, opts); err != nil {
+				t.Fatal(err)
+			}
+			if rebalance {
+				z.rebalMu.Lock()
+				z.rebalMeta = &rebalanceMeta{PoolStats: []*rebalanceStats{{Participating: true, Info: rebalanceInfo{Status: rebalStarted}}, {}}}
+				z.rebalMu.Unlock()
+				defer func() { z.rebalMu.Lock(); z.rebalMeta = nil; z.rebalMu.Unlock() }()
+			} else {
+				z.poolMetaMutex.Lock()
+				z.poolMeta.Pools[0].Decommission = &PoolDecommissionInfo{}
+				z.poolMetaMutex.Unlock()
+				defer func() { z.poolMetaMutex.Lock(); z.poolMeta.Pools[0].Decommission = nil; z.poolMetaMutex.Unlock() }()
+			}
+			// These are the marker-copy options used by rebalance/decommission;
+			// Source cleanup belongs to the mover.
+			opts.DataMovement, opts.SrcPoolIdx = true, 0
+			opts.SkipRebalancing, opts.SkipDecommissioned = rebalance, !rebalance
+			if _, err := z.DeleteObject(t.Context(), bucket, object, opts); err != nil {
+				t.Fatal(err)
+			}
+			for i, pool := range z.serverPools {
+				if got, err := pool.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: opts.VersionID}); !isErrMethodNotAllowed(err) || !got.DeleteMarker {
+					t.Errorf("movement lost marker in pool %d: %+v, %v", i, got, err)
+				}
+			}
+		})
+	}
+	for _, expiration := range []bool{false, true} {
+		t.Run(fmt.Sprintf("scanner/expiration=%t", expiration), func(t *testing.T) {
+			object := fmt.Sprintf("scanner-%t", expiration)
+			oi := putConsistencyObject(t, z, bucket, object, 0, "payload", ObjectOptions{Versioned: true})
+			set := z.serverPools[1].getHashedSet(object)
+			getDisks := set.getDisks
+			faulty := append([]StorageAPI(nil), getDisks()...)
+			for i := range faulty {
+				faulty[i] = consistencyReadFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object}
+			}
+			set.getDisks = func() []StorageAPI { return faulty }
+			defer func() { set.getDisks = getDisks }()
+			opts := ObjectOptions{Versioned: true, VersionID: oi.VersionID, InclFreeVersions: !expiration, Expiration: ExpirationOptions{Expire: expiration}}
+			_, err := z.DeleteObject(t.Context(), bucket, object, opts)
+			if expiration {
+				// No lifecycle rule authorizes expiration. Preserve its existing
+				// version-not-found result, even with an unrelated unreadable pool.
+				if !isErrVersionNotFound(err) {
+					t.Errorf("expiration changed its scanner contract: %v", err)
+				}
+			} else if err != nil {
+				t.Errorf("free-version cleanup was forced through all-pool resolution: %v", err)
+			}
+		})
+	}
+}
+
+func TestPoolsDeleteUnversionedFanout(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	const object = "unversioned-fanout"
+	for i := range z.serverPools {
+		putConsistencyObject(t, z, bucket, object, i, "payload", ObjectOptions{})
+	}
+	if _, err := z.DeleteObject(t.Context(), bucket, object, ObjectOptions{}); err != nil {
+		t.Fatal(err)
+	}
+	for i, pool := range z.serverPools {
+		if _, err := pool.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); !isErrObjectNotFound(err) {
+			t.Errorf("unversioned fanout left pool %d readable: %v", i, err)
+		}
+	}
+}
+
 func TestPoolsConditionalDeleteVersionSelection(t *testing.T) {
 	z, bucket := consistencyPools(t)
 	const object = "split-versions"
@@ -108,7 +596,7 @@ func TestPoolsConditionalDeleteReportsOtherPoolFailure(t *testing.T) {
 	getDisks := set.getDisks
 	faulty := append([]StorageAPI(nil), getDisks()...)
 	for i := range faulty {
-		faulty[i] = accessMoveDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object}
+		faulty[i] = consistencyDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object}
 	}
 	set.getDisks = func() []StorageAPI { return faulty }
 	defer func() { set.getDisks = getDisks }()
@@ -135,9 +623,8 @@ func testPoolsConditionalDeleteWriter(t *testing.T, multipart bool) {
 	ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
 	defer cancel()
 	reader := mustGetPutObjReader(t, bytes.NewBufferString("after"), 5, "", "")
-	destination := 1
 	write := func() error {
-		_, err := z.PutObject(ctx, bucket, object, reader, ObjectOptions{DataMovement: true, SrcPoolIdx: 0, DstPoolIdx: &destination})
+		_, err := z.PutObject(ctx, bucket, object, reader, ObjectOptions{})
 		return err
 	}
 	if multipart {
@@ -548,7 +1035,7 @@ func TestPoolsReplicaCleanupFailureCanRetry(t *testing.T) {
 	getDisks := set.getDisks
 	faulty := append([]StorageAPI(nil), getDisks()...)
 	for i := range faulty {
-		faulty[i] = accessMoveDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object, version: oi.VersionID}
+		faulty[i] = consistencyDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object, version: oi.VersionID}
 	}
 	set.getDisks = func() []StorageAPI { return faulty }
 	defer func() { set.getDisks = getDisks }()
@@ -582,6 +1069,8 @@ func TestPoolsRetiringCopyPreservesSharedTierObject(t *testing.T) {
 		failPrimaryDelete bool
 		differentRemote   bool
 		restored          bool
+		unconditional     bool
+		skipFreeVersion   bool
 	}{
 		{name: "metadata-copy"},
 		{name: "restored-metadata-copy", restored: true},
@@ -589,6 +1078,9 @@ func TestPoolsRetiringCopyPreservesSharedTierObject(t *testing.T) {
 		{name: "failed-primary-delete", deleting: true, failPrimaryDelete: true},
 		{name: "failed-primary-delete-distinct-reference", deleting: true, failPrimaryDelete: true, differentRemote: true},
 		{name: "successful-delete", deleting: true},
+		{name: "ordinary-failed-primary-delete", deleting: true, unconditional: true, failPrimaryDelete: true},
+		{name: "ordinary-successful-delete", deleting: true, unconditional: true},
+		{name: "ordinary-skip-free-version", deleting: true, unconditional: true, skipFreeVersion: true},
 	} {
 		t.Run(test.name, func(t *testing.T) {
 			z, bucket := consistencyPools(t)
@@ -622,16 +1114,20 @@ func TestPoolsRetiringCopyPreservesSharedTierObject(t *testing.T) {
 				getDisks := set.getDisks
 				faulty := append([]StorageAPI(nil), getDisks()...)
 				for i := range faulty {
-					faulty[i] = accessMoveDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object, version: oi.VersionID}
+					faulty[i] = consistencyDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object, version: oi.VersionID}
 				}
 				set.getDisks = func() []StorageAPI { return faulty }
 				defer func() { set.getDisks = getDisks }()
 			}
 			if test.deleting {
-				_, err := z.DeleteObject(t.Context(), bucket, object, ObjectOptions{
+				opts := ObjectOptions{
-					Versioned: true, VersionID: oi.VersionID,
+					Versioned: true, VersionID: oi.VersionID, SkipFreeVersion: test.skipFreeVersion,
 					CheckPrecondFn: func(info ObjectInfo) bool { return info.ETag != current.ETag },
-				})
+				}
+				if test.unconditional {
+					opts.CheckPrecondFn = nil
+				}
+				_, err := z.DeleteObject(t.Context(), bucket, object, opts)
 				if (err != nil) != test.failPrimaryDelete {
 					t.Fatalf("unexpected authoritative delete result: %v", err)
 				}
@@ -656,6 +1152,7 @@ func TestPoolsRetiringCopyPreservesSharedTierObject(t *testing.T) {
 				t.Fatalf("lost retained authoritative copy: %v", err)
 			}
 			for pool, wantFree := range []bool{primaryDeleted, test.differentRemote} {
+				wantFree = wantFree && !test.skipFreeVersion
 				for _, disk := range z.serverPools[pool].getHashedSet(object).getDisks() {
 					data, err := disk.ReadAll(t.Context(), bucket, pathJoin(object, xlStorageFormatFile))
 					if errors.Is(err, errFileNotFound) && !wantFree {
@@ -680,3 +1177,132 @@ func TestPoolsRetiringCopyPreservesSharedTierObject(t *testing.T) {
 		})
 	}
 }
+
+// Fault injection shared by general multi-pool regressions.
+type consistencyDeleteFaultDisk struct {
+	StorageAPI
+	bucket, object, version string
+}
+
+func (d consistencyDeleteFaultDisk) DeleteVersion(ctx context.Context, volume, path string, fi FileInfo, forceDelMarker bool, opts DeleteOptions) error {
+	if volume == d.bucket && path == d.object && fi.VersionID == d.version {
+		return errDiskFull
+	}
+	return d.StorageAPI.DeleteVersion(ctx, volume, path, fi, forceDelMarker, opts)
+}
+
+// Exercise the surviving production rebalance copy, then interrupt the workflow
+// before its later source cleanup. Repeated versions are reachable without the
+// retired access-tier mover, and an API delete must remove both copies.
+func TestPoolsDeleteVersionAfterInterruptedRebalance(t *testing.T) {
+	z, _ := consistencyPools(t)
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{VersioningEnabled: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for source := range 2 {
+		t.Run(fmt.Sprintf("source=%d", source), func(t *testing.T) {
+			object := fmt.Sprintf("interrupted-rebalance-%d", source)
+			original := putConsistencyObject(t, z, bucket, object, source, "original", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
+			stats := []*rebalanceStats{{}, {}}
+			stats[source] = &rebalanceStats{Participating: true, Info: rebalanceInfo{Status: rebalStarted}}
+			z.rebalMu.Lock()
+			z.rebalMeta = &rebalanceMeta{PoolStats: stats}
+			z.rebalMu.Unlock()
+			defer func() { z.rebalMu.Lock(); z.rebalMeta = nil; z.rebalMu.Unlock() }()
+			gr, err := z.serverPools[source].GetObjectNInfo(ctx, bucket, object, nil, nil, ObjectOptions{VersionID: original.VersionID, NoLock: true})
+			if err != nil {
+				t.Fatal(err)
+			}
+			if err := z.rebalanceObject(ctx, source, bucket, gr); err != nil {
+				t.Fatalf("rebalance copy: %v", err)
+			}
+			// Simulate interruption after rebalanceObject, before the enclosing loop
+			// removes the source version stack. Stop rebalance before the API request.
+			z.rebalMu.Lock()
+			z.rebalMeta = nil
+			z.rebalMu.Unlock()
+			for i, pool := range z.serverPools {
+				got, err := pool.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: original.VersionID})
+				if err != nil || got.VersionID != original.VersionID || !got.ModTime.Equal(original.ModTime) {
+					t.Fatalf("copy missing/changed in pool %d: %+v, %v", i, got, err)
+				}
+			}
+			later, err := z.PutObject(ctx, bucket, object, mustGetPutObjReader(t, bytes.NewBufferString("later"), 5, "", ""), ObjectOptions{Versioned: true})
+			if err != nil {
+				t.Fatal(err)
+			}
+			rec := consistencyRequest(t, router, http.MethodDelete, bucket, object, original.VersionID)
+			if rec.Code != http.StatusNoContent {
+				t.Fatalf("DELETE: %d %s", rec.Code, rec.Body.String())
+			}
+			for i, pool := range z.serverPools {
+				if _, err := pool.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: original.VersionID}); !isErrVersionNotFound(err) {
+					t.Errorf("pool %d retained old version: %v", i, err)
+				}
+			}
+			if got, err := z.GetObjectInfo(ctx, bucket, object, ObjectOptions{}); err != nil || got.VersionID != later.VersionID {
+				t.Fatalf("other version changed: %+v, %v", got, err)
+			}
+			for _, method := range []string{http.MethodGet, http.MethodHead} {
+				if rec := consistencyRequest(t, router, method, bucket, object, original.VersionID); rec.Code != http.StatusNotFound {
+					t.Errorf("%s returned %d", method, rec.Code)
+				}
+			}
+		})
+	}
+}
+
+func TestPoolsDeleteDirectoryMarker(t *testing.T) {
+	z, _ := consistencyPools(t)
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{VersioningEnabled: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for holding := range 2 {
+		for _, unreadable := range []bool{false, true} {
+			t.Run(fmt.Sprintf("holding=%d/unreadable=%t", holding, unreadable), func(t *testing.T) {
+				object := fmt.Sprintf("directory-%d-%t/", holding, unreadable)
+				encoded := encodeDirObject(object)
+				putConsistencyObject(t, z, bucket, encoded, holding, "", ObjectOptions{})
+				set := z.serverPools[1-holding].getHashedSet(encoded)
+				getDisks := set.getDisks
+				defer func() { set.getDisks = getDisks }()
+				if unreadable {
+					disks := append([]StorageAPI(nil), getDisks()...)
+					for i := range disks {
+						disks[i] = consistencyReadFaultDisk{StorageAPI: disks[i], bucket: bucket, object: encoded}
+					}
+					set.getDisks = func() []StorageAPI { return disks }
+				}
+				// No explicit versionId: delOpts permanently deletes the null version.
+				rec := consistencyRequest(t, router, http.MethodDelete, bucket, object, "")
+				if unreadable {
+					if rec.Code != http.StatusServiceUnavailable {
+						t.Fatalf("unreadable pool: %d %s", rec.Code, rec.Body.String())
+					}
+					if _, err := z.serverPools[holding].GetObjectInfo(ctx, bucket, encoded, ObjectOptions{VersionID: nullVersionID}); err != nil {
+						t.Fatalf("readable directory marker lost: %v", err)
+					}
+					set.getDisks = getDisks
+					rec = consistencyRequest(t, router, http.MethodDelete, bucket, object, "")
+				}
+				if rec.Code != http.StatusNoContent {
+					t.Fatalf("DELETE: %d %s", rec.Code, rec.Body.String())
+				}
+				for i, pool := range z.serverPools {
+					if _, err := pool.GetObjectInfo(ctx, bucket, encoded, ObjectOptions{VersionID: nullVersionID}); !isErrVersionNotFound(err) {
+						t.Errorf("pool %d retained null version: %v", i, err)
+					}
+				}
+				if rec := consistencyRequest(t, router, http.MethodHead, bucket, object, ""); rec.Code != http.StatusNotFound {
+					t.Errorf("directory still visible: %d", rec.Code)
+				}
+			})
+		}
+	}
+}

cmd/erasure-server-pool-multipart-cond_test.go

@@ -0,0 +1,172 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"bytes"
+	"testing"
+	"time"
+)
+
+// A multipart completion's If-Match must be evaluated against the logical
+// latest object across pools, not against the copy local to a pool.
+//
+// Multi-pool write placement is not sticky (getPoolIdx picks by available
+// space even for existing objects), so an upload and a newer overwrite of
+// the same name routinely end up in different pools. Uploads are pinned to
+// their pools directly: routing through z.NewMultipartUpload would make the
+// placement depend on the space-weighted random choice.
+func TestPoolsMultipartConditionalUsesLogicalLatest(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	ctx := t.Context()
+
+	ifMatch := func(etag string) (opts ObjectOptions) {
+		return ObjectOptions{
+			HasIfMatch: true,
+			CheckPrecondFn: func(oi ObjectInfo) bool {
+				return oi.ETag != etag
+			},
+		}
+	}
+
+	uploadPart := func(t *testing.T, bucket, object, uploadID string) []CompletePart {
+		t.Helper()
+		pi, err := z.PutObjectPart(ctx, bucket, object, uploadID, 1,
+			mustGetPutObjReader(t, bytes.NewBufferString("part"), 4, "", ""), ObjectOptions{})
+		if err != nil {
+			t.Fatal(err)
+		}
+		return []CompletePart{{PartNumber: 1, ETag: pi.ETag}}
+	}
+
+	// Scenario A: the uploaded If-Match carries the stale ETag of the pool-0
+	// copy while the logical latest object lives in pool 1. The completion
+	// must fail with 412 instead of shadowing the newer logical state.
+	objectA := "cond-mp-stale-etag"
+	base := time.Now()
+	oldA := putConsistencyObject(t, z, bucket, objectA, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
+	mpA, err := z.serverPools[0].NewMultipartUpload(ctx, bucket, objectA, ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	newerA := putConsistencyObject(t, z, bucket, objectA, 1, "new", ObjectOptions{
+		MTime: base.Add(-time.Minute),
+	})
+
+	latest, _, err := z.getLatestObjectInfoWithIdx(ctx, bucket, objectA, ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if latest.ETag != newerA.ETag {
+		t.Fatalf("logical latest should be the pool-1 copy: got %s want %s", latest.ETag, newerA.ETag)
+	}
+
+	if _, err = z.CompleteMultipartUpload(ctx, bucket, objectA, mpA.UploadID,
+		uploadPart(t, bucket, objectA, mpA.UploadID), ifMatch(oldA.ETag)); err == nil {
+		t.Fatal("If-Match with the stale pool-0 ETag must not complete over the newer pool-1 object")
+	} else if _, ok := err.(PreConditionFailed); !ok {
+		t.Fatalf("expected PreconditionFailed, got %v", err)
+	}
+
+	if latest, _, err = z.getLatestObjectInfoWithIdx(ctx, bucket, objectA, ObjectOptions{}); err != nil {
+		t.Fatal(err)
+	}
+	if latest.ETag != newerA.ETag {
+		t.Fatalf("the newer pool-1 object must remain the logical latest, got %s", latest.ETag)
+	}
+
+	// Scenario B: the uploaded If-Match carries the logical latest ETag (the
+	// pool-1 copy) while the upload sits next to the stale pool-0 copy. The
+	// precondition is satisfied, so completion must succeed and its result
+	// must become the logical latest.
+	objectB := "cond-mp-latest-etag"
+	putConsistencyObject(t, z, bucket, objectB, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
+	mpB, err := z.serverPools[0].NewMultipartUpload(ctx, bucket, objectB, ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	newerB := putConsistencyObject(t, z, bucket, objectB, 1, "new", ObjectOptions{
+		MTime: base.Add(-time.Minute),
+	})
+	oiB, err := z.CompleteMultipartUpload(ctx, bucket, objectB, mpB.UploadID,
+		uploadPart(t, bucket, objectB, mpB.UploadID), ifMatch(newerB.ETag))
+	if err != nil {
+		t.Fatalf("If-Match with the logical latest ETag must complete, got %v", err)
+	}
+	if oiB.ETag == "" {
+		t.Fatal("completion returned an empty ETag")
+	}
+	if latest, _, err = z.getLatestObjectInfoWithIdx(ctx, bucket, objectB, ObjectOptions{}); err != nil {
+		t.Fatal(err)
+	}
+	if latest.ETag != oiB.ETag {
+		t.Fatalf("the completed object must be the logical latest: got %s want %s", latest.ETag, oiB.ETag)
+	}
+
+	// Scenario C: the upload lives in pool 1 with the newer copy while pool 0
+	// holds the stale one. A set-local evaluation order would let pool 0's
+	// stale copy fail the request before pool 1 is reached; the logical
+	// latest ETag must complete.
+	objectC := "cond-mp-upload-other-pool"
+	putConsistencyObject(t, z, bucket, objectC, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
+	mpC, err := z.serverPools[1].NewMultipartUpload(ctx, bucket, objectC, ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	newerC := putConsistencyObject(t, z, bucket, objectC, 1, "new", ObjectOptions{
+		MTime: base.Add(-time.Minute),
+	})
+	if _, err = z.CompleteMultipartUpload(ctx, bucket, objectC, mpC.UploadID,
+		uploadPart(t, bucket, objectC, mpC.UploadID), ifMatch(newerC.ETag)); err != nil {
+		t.Fatalf("If-Match with the logical latest ETag must complete regardless of upload pool, got %v", err)
+	}
+
+	// Scenario D: pool 1 holds the newer copy but cannot be read. An
+	// unreadable pool may contain the newest state, so the unverifiable
+	// condition must fail the request rather than pass it against pool 0's
+	// stale ETag.
+	objectD := "cond-mp-unreadable-pool"
+	oldD := putConsistencyObject(t, z, bucket, objectD, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
+	mpD, err := z.serverPools[0].NewMultipartUpload(ctx, bucket, objectD, ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	newerD := putConsistencyObject(t, z, bucket, objectD, 1, "new", ObjectOptions{
+		MTime: base.Add(-time.Minute),
+	})
+	if latest, _, err = z.getLatestObjectInfoWithIdx(ctx, bucket, objectD, ObjectOptions{}); err != nil {
+		t.Fatal(err)
+	}
+	if latest.ETag != newerD.ETag {
+		t.Fatalf("logical latest before faulting pool 1 should be its copy: got %s want %s", latest.ETag, newerD.ETag)
+	}
+	set := z.serverPools[1].getHashedSet(objectD)
+	getDisks := set.getDisks
+	faulty := append([]StorageAPI(nil), getDisks()...)
+	for i := range faulty {
+		faulty[i] = consistencyReadFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: objectD}
+	}
+	set.getDisks = func() []StorageAPI { return faulty }
+	defer func() { set.getDisks = getDisks }()
+
+	_, err = z.CompleteMultipartUpload(ctx, bucket, objectD, mpD.UploadID,
+		uploadPart(t, bucket, objectD, mpD.UploadID), ifMatch(oldD.ETag))
+	if !isErrReadQuorum(err) {
+		t.Fatalf("expected an insufficient read quorum error, got %v", err)
+	}
+}

cmd/erasure-server-pool-multipart-http_test.go

@@ -0,0 +1,341 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"bytes"
+	"encoding/xml"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	xhttp "github.com/minio/minio/internal/http"
+)
+
+func multipartConditionRequest(t *testing.T, router http.Handler, method, target, body string, headers map[string]string) *httptest.ResponseRecorder {
+	t.Helper()
+	req, err := newTestSignedRequestV4(method, target, int64(len(body)), strings.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, headers)
+	if err != nil {
+		t.Fatal(err)
+	}
+	rec := httptest.NewRecorder()
+	router.ServeHTTP(rec, req)
+	return rec
+}
+
+// The server writes the wire spelling ETag directly into Header; unlike a
+// network response, httptest's map has not canonicalized it to Etag.
+func multipartConditionResponseETag(rec *httptest.ResponseRecorder) string {
+	for key, values := range rec.Header() {
+		if strings.EqualFold(key, xhttp.ETag) && len(values) > 0 {
+			return strings.Trim(values[0], "\"")
+		}
+	}
+	return ""
+}
+
+func multipartConditionUpload(t *testing.T, z *erasureServerPools, bucket, object string, owner int, opts ObjectOptions) (string, []CompletePart) {
+	t.Helper()
+	mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, object, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	part, err := z.serverPools[owner].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	return mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}
+}
+
+func multipartConditionCompleteBody(parts []CompletePart) string {
+	data, err := xml.Marshal(CompleteMultipartUpload{Parts: parts})
+	if err != nil {
+		panic(err)
+	}
+	return string(data)
+}
+
+func multipartConditionError(t *testing.T, rec *httptest.ResponseRecorder, code string) {
+	t.Helper()
+	decoder := xml.NewDecoder(strings.NewReader(rec.Body.String()))
+	var response APIErrorResponse
+	if err := decoder.Decode(&response); err != nil || response.Code != code {
+		t.Fatalf("expected %s error, got %q: %v", code, rec.Body.String(), err)
+	}
+	if err := decoder.Decode(&response); err != io.EOF {
+		t.Fatalf("expected exactly one error response, got %q: %v", rec.Body.String(), err)
+	}
+}
+
+// State is deliberately placed per pool; the final request uses signed HTTP
+// and the real handler, precondition callback, erasure metadata and rename.
+func TestPoolsMultipartConditionalHTTPMatrix(t *testing.T) {
+	z, _ := consistencyPools(t)
+	bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for owner := range 2 {
+		for _, localOld := range []bool{false, true} {
+			for _, condition := range []string{"match-old", "match-current", "none-match"} {
+				t.Run(fmt.Sprintf("owner=%d/old=%t/%s", owner, localOld, condition), func(t *testing.T) {
+					object := fmt.Sprintf("http-%d-%t-%s", owner, localOld, condition)
+					oldETag := "old-does-not-exist"
+					if localOld {
+						oldETag = putConsistencyObject(t, z, bucket, object, owner, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)}).ETag
+					}
+					id, parts := multipartConditionUpload(t, z, bucket, object, owner, ObjectOptions{})
+					current := putConsistencyObject(t, z, bucket, object, 1-owner, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
+					head := multipartConditionRequest(t, router, http.MethodHead, getGetObjectURL("", bucket, object), "", nil)
+					if head.Code != 200 || multipartConditionResponseETag(head) != current.ETag {
+						t.Fatalf("bad HEAD: %d %v", head.Code, head.Header())
+					}
+					h := map[string]string{xhttp.IfMatch: "\"" + oldETag + "\""}
+					want := 412
+					if condition == "match-current" {
+						h[xhttp.IfMatch] = "\"" + current.ETag + "\""
+						want = 200
+					}
+					if condition == "none-match" {
+						h = map[string]string{xhttp.IfNoneMatch: "*"}
+					}
+					rec := multipartConditionRequest(t, router, http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, id), multipartConditionCompleteBody(parts), h)
+					t.Logf("HEAD current=%s, requested=%v, complete HTTP=%d", current.ETag, h, rec.Code)
+					if rec.Code != want {
+						t.Errorf("want HTTP %d, got %d: %s", want, rec.Code, rec.Body.String())
+					}
+					if want == 412 {
+						multipartConditionError(t, rec, "PreconditionFailed")
+						got := multipartConditionRequest(t, router, http.MethodGet, getGetObjectURL("", bucket, object), "", nil)
+						if got.Code != 200 || got.Body.String() != "current" {
+							t.Errorf("rejected request must preserve current data: %d %q", got.Code, got.Body.String())
+						}
+						if _, err := z.serverPools[owner].ListObjectParts(t.Context(), bucket, object, id, 0, 10, ObjectOptions{}); err != nil {
+							t.Errorf("rejected request consumed upload: %v", err)
+						}
+					}
+				})
+			}
+		}
+	}
+}
+
+func TestPoolsMultipartConditionalHTTPAbsentObject(t *testing.T) {
+	for _, deleted := range []bool{false, true} {
+		for _, match := range []bool{false, true} {
+			t.Run(fmt.Sprintf("delete-marker=%t/if-match=%t", deleted, match), func(t *testing.T) {
+				z, _ := consistencyPools(t)
+				bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: deleted})
+				if err != nil {
+					t.Fatal(err)
+				}
+				object := "http-absent"
+				if deleted {
+					putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
+					_, err = z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)})
+					if err != nil {
+						t.Fatal(err)
+					}
+				}
+				id, parts := multipartConditionUpload(t, z, bucket, object, 0, ObjectOptions{Versioned: deleted})
+				headers := map[string]string{xhttp.IfNoneMatch: "*"}
+				want := http.StatusOK
+				if match {
+					headers = map[string]string{xhttp.IfMatch: "\"missing\""}
+					want = http.StatusNotFound
+				}
+				rec := multipartConditionRequest(t, router, http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, id), multipartConditionCompleteBody(parts), headers)
+				if rec.Code != want {
+					t.Fatalf("expected HTTP %d, got %d: %s", want, rec.Code, rec.Body.String())
+				}
+				if match {
+					multipartConditionError(t, rec, "NoSuchKey")
+					if _, err := z.serverPools[0].ListObjectParts(t.Context(), bucket, object, id, 0, 10, ObjectOptions{}); err != nil {
+						t.Errorf("rejected request consumed upload: %v", err)
+					}
+				}
+			})
+		}
+	}
+}
+
+// All writes below use ordinary signed S3 requests. The result must be correct
+// for every placement; the matrix above deterministically covers split pools.
+func TestPoolsMultipartConditionalHTTPNormalRouting(t *testing.T) {
+	z, _ := consistencyPools(t)
+	bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	object := "normal-routing"
+	url := getPutObjectURL("", bucket, object)
+	old := multipartConditionRequest(t, router, http.MethodPut, url, "old-data", nil)
+	if old.Code != http.StatusOK {
+		t.Fatalf("initial PUT %d: %s", old.Code, old.Body.String())
+	}
+	init := multipartConditionRequest(t, router, http.MethodPost, url+"?uploads", "", nil)
+	if init.Code != http.StatusOK {
+		t.Fatalf("init %d: %s", init.Code, init.Body.String())
+	}
+	var mp InitiateMultipartUploadResponse
+	if err := xml.Unmarshal(init.Body.Bytes(), &mp); err != nil {
+		t.Fatal(err)
+	}
+	part := multipartConditionRequest(t, router, http.MethodPut, getPutObjectPartURL("", bucket, object, mp.UploadID, "1"), "replacement", nil)
+	if part.Code != http.StatusOK {
+		t.Fatalf("part %d: %s", part.Code, part.Body.String())
+	}
+	parts := []CompletePart{{PartNumber: 1, ETag: multipartConditionResponseETag(part)}}
+	newer := multipartConditionRequest(t, router, http.MethodPut, url, "newer-data", nil)
+	if newer.Code != http.StatusOK {
+		t.Fatalf("new PUT %d: %s", newer.Code, newer.Body.String())
+	}
+	head := multipartConditionRequest(t, router, http.MethodHead, url, "", nil)
+	if head.Code != http.StatusOK || multipartConditionResponseETag(head) != multipartConditionResponseETag(newer) {
+		t.Fatalf("HEAD did not pick new object: %d %v", head.Code, head.Header())
+	}
+	rec := multipartConditionRequest(t, router, http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, mp.UploadID), multipartConditionCompleteBody(parts), map[string]string{xhttp.IfMatch: "\"" + multipartConditionResponseETag(old) + "\""})
+	if rec.Code != http.StatusPreconditionFailed {
+		t.Errorf("stale If-Match should be HTTP 412, got %d: %s", rec.Code, rec.Body.String())
+	}
+	get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
+	if get.Code != http.StatusOK || get.Body.String() != "newer-data" {
+		t.Errorf("conditional completion changed newer data: %d %q", get.Code, get.Body.String())
+	}
+}
+
+func TestPoolsMultipartConditionalUnreadablePool(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	object := "quorum-with-readable-copy"
+	old := putConsistencyObject(t, z, bucket, object, 0, "readable", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
+	putConsistencyObject(t, z, bucket, object, 1, "hidden-newer", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
+	id, parts := multipartConditionUpload(t, z, bucket, object, 0, ObjectOptions{})
+	set := z.serverPools[1].getHashedSet(object)
+	original := set.getDisks
+	disks := append([]StorageAPI(nil), original()...)
+	for i := range disks {
+		disks[i] = consistencyReadFaultDisk{StorageAPI: disks[i], bucket: bucket, object: object}
+	}
+	set.getDisks = func() []StorageAPI { return disks }
+	defer func() { set.getDisks = original }()
+	read, readErr := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
+	t.Logf("ordinary GET lookup: etag=%s err=%v", read.ETag, readErr)
+	called := 0
+	_, err := z.CompleteMultipartUpload(t.Context(), bucket, object, id, parts, ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { called++; return oi.ETag != old.ETag }})
+	if !isErrReadQuorum(err) {
+		t.Errorf("conditional write must fail on unreadable pool, got %v", err)
+	}
+	if called != 0 {
+		t.Errorf("callback evaluated without complete state: %d calls", called)
+	}
+}
+
+func TestPoolsMultipartConditionalLatestVersionAndCallbackOnce(t *testing.T) {
+	for _, explicit := range []bool{false, true} {
+		t.Run(fmt.Sprintf("explicit=%t", explicit), func(t *testing.T) {
+			z, bucket := consistencyPools(t)
+			object := "tie-version"
+			opts := ObjectOptions{MTime: UTCNow().Add(-time.Hour), Versioned: explicit}
+			if explicit {
+				opts.VersionID = mustGetUUID()
+			}
+			current := putConsistencyObject(t, z, bucket, object, 0, "first", opts)
+			putConsistencyObject(t, z, bucket, object, 1, "second", opts)
+			if explicit {
+				current = putConsistencyObject(t, z, bucket, object, 1, "latest-other-version", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
+			}
+			id, parts := multipartConditionUpload(t, z, bucket, object, 1, opts)
+			called := 0
+			opts.CheckPrecondFn = func(oi ObjectInfo) bool { called++; return oi.ETag != current.ETag }
+			opts.MTime = time.Time{}
+			opts.HasIfMatch = true
+			_, err := z.CompleteMultipartUpload(t.Context(), bucket, object, id, parts, opts)
+			if err != nil {
+				t.Errorf("logical current object should match: %v", err)
+			}
+			if called != 1 {
+				t.Errorf("condition evaluated %d times; want exactly once", called)
+			}
+		})
+	}
+}
+
+func TestPoolsMultipartConditionalConcurrentCompletes(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	object := "concurrent-completes"
+	old := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
+	putConsistencyObject(t, z, bucket, object, 1, "old", ObjectOptions{MTime: old.ModTime})
+	ids := make([]string, 2)
+	parts := make([][]CompletePart, 2)
+	for i := range 2 {
+		ids[i], parts[i] = multipartConditionUpload(t, z, bucket, object, i, ObjectOptions{})
+	}
+	entered := make(chan struct{})
+	release := make(chan struct{})
+	var gate, releaseOnce sync.Once
+	defer releaseOnce.Do(func() { close(release) })
+	errs := make([]error, 2)
+	var wg sync.WaitGroup
+	// Let pool 1's completion hold the object lock before pool 0's starts.
+	// Once pool 1 commits, a set-local read in pool 0 would still see the
+	// old ETag and incorrectly accept the second completion.
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		_, errs[1] = z.CompleteMultipartUpload(t.Context(), bucket, object, ids[1], parts[1], ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool {
+			gate.Do(func() { close(entered); <-release })
+			return oi.ETag != old.ETag
+		}})
+	}()
+	select {
+	case <-entered:
+	case <-time.After(10 * time.Second):
+		t.Fatal("first completion did not enter its condition callback")
+	}
+	started := make(chan struct{})
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		close(started)
+		_, errs[0] = z.CompleteMultipartUpload(t.Context(), bucket, object, ids[0], parts[0], ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { return oi.ETag != old.ETag }})
+	}()
+	<-started
+	releaseOnce.Do(func() { close(release) })
+	wg.Wait()
+	success, failed := 0, 0
+	for _, err := range errs {
+		var p PreConditionFailed
+		switch {
+		case err == nil:
+			success++
+		case errors.As(err, &p):
+			failed++
+		default:
+			t.Errorf("unexpected completion error %v", err)
+		}
+	}
+	if success != 1 || failed != 1 {
+		t.Errorf("CAS writers: success=%d conditional failures=%d errors=%v", success, failed, errs)
+	}
+}

cmd/erasure-server-pool-multipart-preconditions_test.go

@@ -0,0 +1,135 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"testing"
+	"time"
+)
+
+func TestPoolsMultipartConditionMatrix(t *testing.T) {
+	for owner := range 2 {
+		for _, withOld := range []bool{false, true} {
+			for _, condition := range []string{"match-current", "match-old", "none-match-any"} {
+				t.Run(fmt.Sprintf("upload-pool=%d/old-copy=%t/%s", owner, withOld, condition), func(t *testing.T) {
+					z, bucket := consistencyPools(t)
+					object := "conditional-multipart"
+					oldETag := "arbitrary-old"
+					if withOld {
+						old := putConsistencyObject(t, z, bucket, object, owner, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
+						oldETag = old.ETag
+					}
+					mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, object, ObjectOptions{})
+					if err != nil {
+						t.Fatal(err)
+					}
+					part, err := z.serverPools[owner].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), ObjectOptions{})
+					if err != nil {
+						t.Fatal(err)
+					}
+					current := putConsistencyObject(t, z, bucket, object, 1-owner, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
+					visible, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
+					if err != nil || visible.ETag != current.ETag {
+						t.Fatalf("invalid current state: %v", err)
+					}
+					opts := ObjectOptions{HasIfMatch: condition != "none-match-any", CheckPrecondFn: func(oi ObjectInfo) bool {
+						switch condition {
+						case "match-current":
+							return oi.ETag != current.ETag
+						case "match-old":
+							return oi.ETag != oldETag
+						default:
+							return true
+						}
+					}}
+					_, err = z.CompleteMultipartUpload(t.Context(), bucket, object, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, opts)
+					if condition == "match-current" {
+						if err != nil {
+							t.Errorf("correct logical If-Match rejected: %v", err)
+						}
+					} else {
+						var expected PreConditionFailed
+						if !errors.As(err, &expected) {
+							t.Errorf("logical condition must fail, got %v", err)
+						}
+					}
+				})
+			}
+		}
+	}
+}
+
+func TestPoolsMultipartConditionBoundaries(t *testing.T) {
+	for _, state := range []string{"missing", "latest-delete-marker", "unreadable-other-pool"} {
+		for _, match := range []bool{false, true} {
+			t.Run(fmt.Sprintf("%s/if-match=%t", state, match), func(t *testing.T) {
+				z, bucket := consistencyPools(t)
+				object := "conditional-boundary"
+				versioned := state == "latest-delete-marker"
+				if versioned {
+					putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
+					if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)}); err != nil {
+						t.Fatal(err)
+					}
+				}
+				mp, err := z.serverPools[0].NewMultipartUpload(t.Context(), bucket, object, ObjectOptions{Versioned: versioned})
+				if err != nil {
+					t.Fatal(err)
+				}
+				part, err := z.serverPools[0].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("new"), 3, "", ""), ObjectOptions{})
+				if err != nil {
+					t.Fatal(err)
+				}
+				if state == "unreadable-other-pool" {
+					set := z.serverPools[1].getHashedSet(object)
+					getDisks := set.getDisks
+					faulty := append([]StorageAPI(nil), getDisks()...)
+					for i := range faulty {
+						faulty[i] = consistencyReadFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object}
+					}
+					set.getDisks = func() []StorageAPI { return faulty }
+					defer func() { set.getDisks = getDisks }()
+				}
+				opts := ObjectOptions{Versioned: versioned, HasIfMatch: match, CheckPrecondFn: func(ObjectInfo) bool { return true }}
+				_, err = z.CompleteMultipartUpload(t.Context(), bucket, object, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, opts)
+				switch {
+				case state == "unreadable-other-pool":
+					if !isErrReadQuorum(err) {
+						t.Errorf("unreadable pool must not mean absence: %v", err)
+					}
+				case match:
+					if !isErrObjectNotFound(err) {
+						t.Errorf("If-Match against logical absence should report absence: %v", err)
+					}
+				default:
+					if err != nil {
+						t.Errorf("If-None-Match against logical absence must succeed: %v", err)
+					}
+				}
+				if err != nil {
+					if _, lerr := z.serverPools[0].ListObjectParts(t.Context(), bucket, object, mp.UploadID, 0, 10, ObjectOptions{}); lerr != nil {
+						t.Errorf("failed condition consumed upload: %v", lerr)
+					}
+				}
+			})
+		}
+	}
+}

cmd/erasure-server-pool-tags_test.go

@@ -0,0 +1,459 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"context"
+	"fmt"
+	"maps"
+	"strings"
+	"testing"
+
+	xhttp "github.com/minio/minio/internal/http"
+)
+
+// The fixture places copies directly in real erasure pools. It models a
+// duplicated version; it does not claim to exercise a rebalance workflow.
+func TestPoolsMetadataUpdatePreservesTags(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	const (
+		old       = "2026-09-09T09:00:00Z"
+		recent    = "2026-09-09T10:00:00Z"
+		timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
+	)
+	for _, test := range []struct {
+		name         string
+		tags, stamps [2]string
+		winner       int
+		single       bool
+	}{
+		{name: "newer-secondary", tags: [2]string{"key=old", "key=new"}, stamps: [2]string{old, recent}, winner: 1},
+		{name: "newer-primary", tags: [2]string{"key=new", "key=old"}, stamps: [2]string{recent, old}},
+		{name: "empty-secondary", tags: [2]string{"key=old", ""}, stamps: [2]string{old, recent}, winner: 1},
+		{name: "empty-primary", tags: [2]string{"", "key=old"}, stamps: [2]string{recent, old}},
+		{name: "single-copy-primary", tags: [2]string{"key=only", ""}, stamps: [2]string{recent, ""}, single: true},
+		{name: "single-copy-secondary", tags: [2]string{"", "key=only"}, stamps: [2]string{"", recent}, winner: 1, single: true},
+		{name: "legacy-single-copy", tags: [2]string{"key=legacy", ""}, single: true},
+		{name: "legacy-duplicates", tags: [2]string{"key=legacy", "key=legacy"}},
+	} {
+		t.Run(test.name, func(t *testing.T) {
+			object := test.name
+			var original ObjectInfo
+			for pool := range 2 {
+				if test.single && pool != test.winner {
+					continue
+				}
+				metadata := map[string]string{
+					xhttp.AmzObjectTagging: test.tags[pool],
+					"copy-local":           fmt.Sprint(pool),
+				}
+				if test.stamps[pool] != "" {
+					metadata[timestamp] = test.stamps[pool]
+				}
+				original = putConsistencyObject(t, z, bucket, object, pool, "data", ObjectOptions{
+					Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, UserDefined: metadata,
+				})
+				got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: original.VersionID})
+				if err != nil || got.UserTags != test.tags[pool] || got.UserDefined[timestamp] != test.stamps[pool] {
+					t.Fatalf("pool %d fixture: tags=%q timestamp=%q err=%v", pool, got.UserTags, got.UserDefined[timestamp], err)
+				}
+				if _, exists := got.UserDefined[xhttp.AmzObjectTagging]; exists {
+					t.Fatal("fixture must use the cleaned ObjectInfo representation")
+				}
+			}
+			wantTags, wantStamp := test.tags[test.winner], test.stamps[test.winner]
+			called := 0
+			got, err := z.PutObjectMetadata(t.Context(), bucket, object, ObjectOptions{
+				VersionID: original.VersionID, MTime: original.ModTime,
+				EvalMetadataFn: func(current *ObjectInfo, _ error) (ReplicateDecision, error) {
+					called++
+					if current.UserTags != wantTags || current.UserDefined[timestamp] != wantStamp {
+						t.Errorf("callback tags=%q timestamp=%q; want %q %q", current.UserTags, current.UserDefined[timestamp], wantTags, wantStamp)
+					}
+					current.UserDefined["unrelated-update"] = "preserved"
+					return ReplicateDecision{}, nil
+				},
+			})
+			if err != nil || called != 1 {
+				t.Fatalf("metadata update: %v, callbacks=%d", err, called)
+			}
+			if got.UserTags != wantTags || got.UserDefined[timestamp] != wantStamp {
+				t.Errorf("response tags=%q timestamp=%q; want %q %q", got.UserTags, got.UserDefined[timestamp], wantTags, wantStamp)
+			}
+			for pool := range 2 {
+				got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: original.VersionID})
+				if test.single && pool != test.winner {
+					if !isErrVersionNotFound(err) {
+						t.Errorf("metadata update created another copy: %v", err)
+					}
+					continue
+				}
+				if err != nil {
+					t.Fatal(err)
+				}
+				t.Logf("pool %d persisted tags=%q timestamp=%q", pool, got.UserTags, got.UserDefined[timestamp])
+				if got.UserTags != wantTags || got.UserDefined[timestamp] != wantStamp {
+					t.Errorf("pool %d persisted tags=%q timestamp=%q; want %q %q", pool, got.UserTags, got.UserDefined[timestamp], wantTags, wantStamp)
+				}
+				if got.UserDefined["unrelated-update"] != "preserved" || got.UserDefined["copy-local"] != fmt.Sprint(pool) {
+					t.Errorf("pool %d lost unrelated metadata: %v", pool, got.UserDefined)
+				}
+			}
+		})
+	}
+}
+
+func TestReplicaWritesPreserveTagOrdering(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	// Pool allocation checks the host's used-space percentage. Present only
+	// its free space as fixture capacity; all reads and writes still use the
+	// real disks. This keeps unrelated host disk usage out of the tag test.
+	for _, pool := range z.serverPools {
+		for _, set := range pool.sets {
+			getDisks := set.getDisks
+			disks := append([]StorageAPI(nil), getDisks()...)
+			for i := range disks {
+				disks[i] = tagTestCapacityDisk{StorageAPI: disks[i]}
+			}
+			set.getDisks = func() []StorageAPI { return disks }
+			t.Cleanup(func() { set.getDisks = getDisks })
+		}
+	}
+	const (
+		old       = "2026-09-09T09:00:00Z"
+		recent    = "2026-09-09T10:00:00Z"
+		timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
+	)
+	for _, path := range []struct {
+		name, operation string
+		direct          bool
+		winner          int
+	}{
+		{name: "set-put", operation: "put", direct: true},
+		{name: "set-multipart", operation: "multipart", direct: true},
+		{name: "set-copy", operation: "copy", direct: true},
+		{name: "pools-put", operation: "put", winner: 1},
+		{name: "pools-multipart", operation: "multipart", winner: 1},
+		{name: "pools-copy-primary", operation: "copy"},
+		{name: "pools-copy-secondary", operation: "copy", winner: 1},
+	} {
+		for _, test := range []struct {
+			name, storedTags, incomingTags, storedStamp, incomingStamp, wantTags string
+		}{
+			{"stored-newer", "key=stored", "key=incoming", recent, old, "key=stored"},
+			{"stored-deleted", "", "key=incoming", recent, old, ""},
+			{"incoming-newer", "key=stored", "key=incoming", old, recent, "key=incoming"},
+			{"incoming-deleted", "key=stored", "", old, recent, ""},
+		} {
+			t.Run(path.name+"/"+test.name, func(t *testing.T) {
+				object := path.name + "-" + test.name
+				var original ObjectInfo
+				for pool := range 2 {
+					if path.direct && pool != 0 {
+						continue
+					}
+					metadata := map[string]string{
+						xhttp.AmzObjectTagging: "key=older-copy",
+						timestamp:              "2026-09-09T08:00:00Z",
+					}
+					if pool == path.winner {
+						metadata[xhttp.AmzObjectTagging] = test.storedTags
+						metadata[timestamp] = test.storedStamp
+					}
+					original = putConsistencyObject(t, z, bucket, object, pool, "data", ObjectOptions{
+						Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, UserDefined: metadata,
+					})
+				}
+				opts := ObjectOptions{
+					Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, ReplicaLockReconcile: true,
+					UserDefined: map[string]string{
+						xhttp.AmzObjectTagging: test.incomingTags,
+						timestamp:              test.incomingStamp,
+					},
+				}
+				var got ObjectInfo
+				var err error
+				switch path.operation {
+				case "put":
+					put := z.PutObject
+					if path.direct {
+						put = z.serverPools[0].PutObject
+					}
+					got, err = put(t.Context(), bucket, object, mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), opts)
+				case "multipart":
+					// Persist the incoming tags with the upload, before completion
+					// reconciles the destination version through its real resolver.
+					mp, err := z.serverPools[0].NewMultipartUpload(t.Context(), bucket, object, opts)
+					if err != nil {
+						t.Fatal(err)
+					}
+					part, err := z.serverPools[0].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1,
+						mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{})
+					if err != nil {
+						t.Fatal(err)
+					}
+					complete := z.CompleteMultipartUpload
+					if path.direct {
+						complete = z.serverPools[0].CompleteMultipartUpload
+					}
+					got, err = complete(t.Context(), bucket, object, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{
+						Versioned: true, MTime: original.ModTime, ReplicaLockReconcile: true,
+					})
+					if err != nil {
+						t.Fatal(err)
+					}
+				case "copy":
+					src := original
+					src.metadataOnly = true
+					src.UserDefined = maps.Clone(opts.UserDefined)
+					copyObject := z.CopyObject
+					if path.direct {
+						copyObject = z.serverPools[0].CopyObject
+					}
+					got, err = copyObject(t.Context(), bucket, object, bucket, object, src, ObjectOptions{VersionID: original.VersionID}, opts)
+				}
+				if err != nil {
+					t.Fatal(err)
+				}
+				if got.UserTags != test.wantTags || got.UserDefined[timestamp] != recent {
+					t.Errorf("response tags=%q timestamp=%q; want %q %q", got.UserTags, got.UserDefined[timestamp], test.wantTags, recent)
+				}
+				copies := 0
+				for pool := range 2 {
+					got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: original.VersionID})
+					if isErrVersionNotFound(err) {
+						continue
+					}
+					if err != nil {
+						t.Fatal(err)
+					}
+					copies++
+					if got.UserTags != test.wantTags || got.UserDefined[timestamp] != recent {
+						t.Errorf("pool %d persisted tags=%q timestamp=%q; want %q %q", pool, got.UserTags, got.UserDefined[timestamp], test.wantTags, recent)
+					}
+				}
+				if copies != 1 {
+					t.Errorf("replacement left %d copies; want 1", copies)
+				}
+			})
+		}
+	}
+}
+
+type tagTestCapacityDisk struct{ StorageAPI }
+
+func (d tagTestCapacityDisk) DiskInfo(ctx context.Context, opts DiskInfoOptions) (DiskInfo, error) {
+	info, err := d.StorageAPI.DiskInfo(ctx, opts)
+	info.Total, info.Used = info.Free, 0
+	return info, err
+}
+
+func TestMergedPoolObjectInfoTagOrdering(t *testing.T) {
+	const (
+		old       = "2026-09-09T09:00:00Z"
+		recent    = "2026-09-09T10:00:00Z"
+		timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
+	)
+	for _, test := range []struct {
+		name, firstStamp, secondStamp, secondTags string
+		winner                                    int
+	}{
+		{"newer", old, recent, "key=second", 1},
+		{"newer-removal", old, recent, "", 1},
+		{"equal", recent, recent, "key=second", 0},
+		{"unordered", "", "", "key=second", 0},
+		{"missing-first", "", recent, "key=second", 1},
+		{"missing-second", recent, "", "key=second", 0},
+		{"invalid-first", "invalid", recent, "key=second", 1},
+		{"invalid-second", recent, "invalid", "key=second", 0},
+	} {
+		t.Run(test.name, func(t *testing.T) {
+			tagValues := []string{"key=first", test.secondTags}
+			stamps := []string{test.firstStamp, test.secondStamp}
+			copies := make([]PoolObjInfo, 2)
+			before := make([]map[string]string, 2)
+			for i := range copies {
+				fi := FileInfo{Metadata: map[string]string{xhttp.AmzObjectTagging: tagValues[i]}}
+				if stamps[i] != "" {
+					fi.Metadata[timestamp] = stamps[i]
+				}
+				copies[i] = PoolObjInfo{Index: i, ObjInfo: fi.ToObjectInfo("bucket", "object", true)}
+				before[i] = maps.Clone(copies[i].ObjInfo.UserDefined)
+			}
+			got := mergedPoolObjectInfo(copies)
+			if got.UserTags != tagValues[test.winner] || got.UserDefined[timestamp] != stamps[test.winner] {
+				t.Errorf("merged tags=%q timestamp=%q; want %q %q", got.UserTags, got.UserDefined[timestamp], tagValues[test.winner], stamps[test.winner])
+			}
+			if _, exists := got.UserDefined[xhttp.AmzObjectTagging]; exists {
+				t.Error("merged ObjectInfo leaked the raw tagging key into UserDefined")
+			}
+			for i := range copies {
+				if !maps.Equal(copies[i].ObjInfo.UserDefined, before[i]) || copies[i].ObjInfo.UserTags != tagValues[i] {
+					t.Errorf("merge mutated input copy %d", i)
+				}
+			}
+		})
+	}
+}
+
+func TestPoolsMetadataCallbackReplacesTags(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	const timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
+	for _, test := range []struct{ name, tags string }{
+		{"replace", "key=callback"},
+		{"remove", ""},
+	} {
+		t.Run(test.name, func(t *testing.T) {
+			var original ObjectInfo
+			for pool := range 2 {
+				original = putConsistencyObject(t, z, bucket, test.name, pool, "data", ObjectOptions{
+					Versioned: true, VersionID: original.VersionID, MTime: original.ModTime,
+					UserDefined: map[string]string{
+						xhttp.AmzObjectTagging: []string{"key=old", "key=new"}[pool],
+						timestamp:              []string{"2026-09-09T09:00:00Z", "2026-09-09T10:00:00Z"}[pool],
+					},
+				})
+			}
+			const updatedStamp = "2026-09-09T11:00:00Z"
+			got, err := z.PutObjectMetadata(t.Context(), bucket, test.name, ObjectOptions{
+				VersionID: original.VersionID, MTime: original.ModTime,
+				EvalMetadataFn: func(current *ObjectInfo, _ error) (ReplicateDecision, error) {
+					if current.UserTags != "key=new" {
+						t.Errorf("callback read tags=%q; want key=new", current.UserTags)
+					}
+					if _, exists := current.UserDefined[xhttp.AmzObjectTagging]; exists {
+						t.Error("callback received the raw tagging key")
+					}
+					current.UserDefined[xhttp.AmzObjectTagging] = test.tags
+					current.UserDefined[timestamp] = updatedStamp
+					return ReplicateDecision{}, nil
+				},
+			})
+			if err != nil {
+				t.Fatal(err)
+			}
+			if got.UserTags != test.tags || got.UserDefined[timestamp] != updatedStamp {
+				t.Errorf("callback update response tags=%q timestamp=%q", got.UserTags, got.UserDefined[timestamp])
+			}
+			for pool := range 2 {
+				got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, test.name, ObjectOptions{VersionID: original.VersionID})
+				if err != nil || got.UserTags != test.tags || got.UserDefined[timestamp] != updatedStamp {
+					t.Errorf("pool %d did not persist callback tags: tags=%q timestamp=%q err=%v", pool, got.UserTags, got.UserDefined[timestamp], err)
+				}
+			}
+		})
+	}
+}
+
+func TestReconcileStoredObjectTagOrdering(t *testing.T) {
+	const (
+		old       = "2026-09-09T09:00:00Z"
+		recent    = "2026-09-09T10:00:00Z"
+		timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
+	)
+	for _, test := range []struct {
+		name, storedStamp, incomingStamp, storedTags string
+		wantStored                                   bool
+	}{
+		{"stored-newer", recent, old, "key=stored", true},
+		{"incoming-newer", old, recent, "key=stored", false},
+		{"equal", recent, recent, "key=stored", true},
+		{"equal-removal", recent, recent, "", true},
+		{"missing-stored", "", recent, "key=stored", false},
+		{"missing-incoming", recent, "", "key=stored", true},
+		{"invalid-stored", "invalid", recent, "key=stored", false},
+		{"invalid-incoming", recent, "invalid", "key=stored", true},
+	} {
+		t.Run(test.name, func(t *testing.T) {
+			metadata := map[string]string{
+				xhttp.AmzObjectTagging: "key=incoming",
+				timestamp:              test.incomingStamp,
+				"unrelated":            "preserved",
+			}
+			reconcileStoredObjectTags(metadata, test.storedTags, test.storedStamp)
+			wantTags, wantStamp := "key=incoming", test.incomingStamp
+			if test.wantStored {
+				wantTags, wantStamp = test.storedTags, test.storedStamp
+			}
+			if metadata[xhttp.AmzObjectTagging] != wantTags || metadata[timestamp] != wantStamp || metadata["unrelated"] != "preserved" {
+				t.Errorf("reconciled metadata=%v; want tags=%q timestamp=%q and unrelated field preserved", metadata, wantTags, wantStamp)
+			}
+		})
+	}
+}
+
+func TestPoolsMetadataUpdatePreservesAbsentTags(t *testing.T) {
+	z, bucket := consistencyPools(t)
+	const (
+		old       = "2026-09-09T09:00:00Z"
+		recent    = "2026-09-09T10:00:00Z"
+		timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
+	)
+	for _, removal := range []bool{false, true} {
+		t.Run(fmt.Sprintf("timestamp-only-removal=%t", removal), func(t *testing.T) {
+			object := fmt.Sprintf("absent-tags-%t", removal)
+			var original ObjectInfo
+			checkStored := func(pool int, wantKey bool, wantTags, wantStamp string) {
+				t.Helper()
+				infos, errs := readAllFileInfo(t.Context(), z.serverPools[pool].getHashedSet(object).getDisks(), "", bucket, object, original.VersionID, false, false)
+				for disk, info := range infos {
+					if errs[disk] != nil {
+						t.Fatal(errs[disk])
+					}
+					tags, exists := info.Metadata[xhttp.AmzObjectTagging]
+					if exists != wantKey || tags != wantTags || info.Metadata[timestamp] != wantStamp {
+						t.Errorf("pool %d disk %d raw tagging key=%t value=%q stamp=%q; want %t %q %q", pool, disk, exists, tags, info.Metadata[timestamp], wantKey, wantTags, wantStamp)
+					}
+				}
+			}
+			for pool := range 2 {
+				metadata := map[string]string{}
+				if removal {
+					metadata[timestamp] = recent
+					if pool == 1 {
+						metadata[xhttp.AmzObjectTagging] = "key=old"
+						metadata[timestamp] = old
+					}
+				}
+				original = putConsistencyObject(t, z, bucket, object, pool, "data", ObjectOptions{
+					Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, UserDefined: metadata,
+				})
+				checkStored(pool, removal && pool == 1, metadata[xhttp.AmzObjectTagging], metadata[timestamp])
+			}
+			_, err := z.PutObjectMetadata(t.Context(), bucket, object, ObjectOptions{
+				VersionID: original.VersionID, MTime: original.ModTime,
+				EvalMetadataFn: func(current *ObjectInfo, _ error) (ReplicateDecision, error) {
+					current.UserDefined["unrelated-update"] = "preserved"
+					return ReplicateDecision{}, nil
+				},
+			})
+			if err != nil {
+				t.Fatal(err)
+			}
+			wantStamp := ""
+			if removal {
+				wantStamp = recent
+			}
+			for pool := range 2 {
+				// A previously non-empty key needs an explicit empty value to
+				// propagate deletion; an absent key should remain absent.
+				checkStored(pool, removal && pool == 1, "", wantStamp)
+			}
+		})
+	}
+}

cmd/erasure-server-pool.go

@@ -615,17 +615,7 @@ func (z *erasureServerPools) getPoolIdxExistingNoLock(ctx context.Context, bucke
 	})
 }
 
-func (z *erasureServerPools) getPoolIdxNoLock(ctx context.Context, bucket, object string, size int64, dstPoolIdx *int) (idx int, err error) {
+func (z *erasureServerPools) getPoolIdxNoLock(ctx context.Context, bucket, object string, size int64) (idx int, err error) {
-	if dstPoolIdx != nil {
-		if *dstPoolIdx < 0 || *dstPoolIdx >= len(z.serverPools) {
-			return -1, errInvalidArgument
-		}
-		if z.IsSuspended(*dstPoolIdx) || z.IsPoolRebalancing(*dstPoolIdx) {
-			return -1, toObjectErr(errDiskFull)
-		}
-		return *dstPoolIdx, nil
-	}
-
 	idx, err = z.getPoolIdxExistingNoLock(ctx, bucket, object)
 	if err != nil && !isErrObjectNotFound(err) {
 		return idx, err
@@ -644,23 +634,13 @@ func (z *erasureServerPools) getPoolIdxNoLock(ctx context.Context, bucket, objec
 // getPoolIdx returns the found previous object and its corresponding pool idx,
 // if none are found falls back to most available space pool, this function is
 // designed to be only used by PutObject, CopyObject (newObject creation) and NewMultipartUpload.
-func (z *erasureServerPools) getPoolIdx(ctx context.Context, bucket, object string, size int64, dstPoolIdx *int) (idx int, err error) {
+func (z *erasureServerPools) getPoolIdx(ctx context.Context, bucket, object string, size int64) (idx int, err error) {
-	return z.getWritePoolIdx(ctx, bucket, object, size, dstPoolIdx, false)
+	return z.getWritePoolIdx(ctx, bucket, object, size, false)
 }
 
 // getWritePoolIdx keeps the write-allocation policy when the caller already
 // holds the pools-layer object lock and must not reacquire a set read lock.
-func (z *erasureServerPools) getWritePoolIdx(ctx context.Context, bucket, object string, size int64, dstPoolIdx *int, noLock bool) (idx int, err error) {
+func (z *erasureServerPools) getWritePoolIdx(ctx context.Context, bucket, object string, size int64, noLock bool) (idx int, err error) {
-	if dstPoolIdx != nil {
-		if *dstPoolIdx < 0 || *dstPoolIdx >= len(z.serverPools) {
-			return -1, errInvalidArgument
-		}
-		if z.IsSuspended(*dstPoolIdx) || z.IsPoolRebalancing(*dstPoolIdx) {
-			return -1, toObjectErr(errDiskFull)
-		}
-		return *dstPoolIdx, nil
-	}
-
 	pinfo, _, err := z.getPoolInfoExistingWithOpts(ctx, bucket, object, ObjectOptions{
 		NoLock:             noLock,
 		SkipDecommissioned: true,
@@ -683,13 +663,6 @@ func (z *erasureServerPools) getWritePoolIdx(ctx context.Context, bucket, object
 	return idx, nil
 }
 
-func dataMovementDstPool(opts ObjectOptions) *int {
-	if !opts.DataMovement {
-		return nil
-	}
-	return opts.DstPoolIdx
-}
-
 func (z *erasureServerPools) Shutdown(ctx context.Context) error {
 	g := errgroup.WithNErrs(len(z.serverPools))
 
@@ -1158,16 +1131,10 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
 
 	object = encodeDirObject(object)
 	if z.SinglePool() {
-		idx, err := z.getPoolIdx(ctx, bucket, object, data.Size(), dataMovementDstPool(opts))
+		_, err := z.getPoolIdx(ctx, bucket, object, data.Size())
 		if err != nil {
 			return ObjectInfo{}, err
 		}
-		if dataMovementDstPool(opts) != nil && idx == opts.SrcPoolIdx {
-			return ObjectInfo{}, DataMovementOverwriteErr{
-				Bucket: bucket, Object: object, VersionID: opts.VersionID,
-				Err: errDataMovementSrcDstPoolSame,
-			}
-		}
 		return z.serverPools[0].PutObject(ctx, bucket, object, data, opts)
 	}
 
@@ -1182,7 +1149,7 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
 	}
 	opts.NoLock = true
 
-	idx, err := z.getWritePoolIdx(ctx, bucket, object, data.Size(), dataMovementDstPool(opts), true)
+	idx, err := z.getWritePoolIdx(ctx, bucket, object, data.Size(), true)
 	if err != nil {
 		return ObjectInfo{}, err
 	}
@@ -1238,30 +1205,11 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
 		return ObjectInfo{}, z.deletePrefix(ctx, bucket, object)
 	}
 
-	// Access-tier moves must recreate delete markers on the explicitly
+	// Reconcile ordinary addressed-version deletes independently of pool movement.
-	// selected destination. The regular data-movement path discovers a pool
+	reconcileVersion := opts.VersionID != "" && !opts.DataMovement &&
-	// from existing object state, which is ambiguous while both source and
+		!opts.ReplicationRequest && !opts.Expiration.Expire && !opts.InclFreeVersions
-	// destination temporarily contain the version stack.
+	if !z.SinglePool() && (opts.CheckPrecondFn != nil || reconcileVersion) {
-	if dstPoolIdx := dataMovementDstPool(opts); dstPoolIdx != nil {
+		return z.deleteObjectReconciled(ctx, bucket, object, opts)
-		if *dstPoolIdx < 0 || *dstPoolIdx >= len(z.serverPools) {
-			return ObjectInfo{}, errInvalidArgument
-		}
-		if *dstPoolIdx == opts.SrcPoolIdx {
-			return ObjectInfo{}, DataMovementOverwriteErr{
-				Bucket: bucket, Object: decodeDirObject(object), VersionID: opts.VersionID,
-				Err: errDataMovementSrcDstPoolSame,
-			}
-		}
-		if z.IsSuspended(*dstPoolIdx) || z.IsPoolRebalancing(*dstPoolIdx) {
-			return ObjectInfo{}, toObjectErr(errDiskFull)
-		}
-		objInfo, err = z.serverPools[*dstPoolIdx].DeleteObject(ctx, bucket, object, opts)
-		objInfo.Name = decodeDirObject(object)
-		return objInfo, err
-	}
-
-	if !z.SinglePool() && opts.CheckPrecondFn != nil {
-		return z.deleteObjectConditional(ctx, bucket, object, opts)
 	}
 
 	gopts := opts
@@ -1499,7 +1447,7 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
 		}
 		stored := mergedPoolObjectInfo(copies)
 		reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(stored.UserDefined))
-		reconcileStoredObjectTags(srcInfo.UserDefined, stored.UserDefined)
+		reconcileStoredObjectTags(srcInfo.UserDefined, stored.UserTags, stored.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
 		idx := copies[0].Index
 		oi, err := z.serverPools[idx].CopyObject(ctx, srcBucket, srcObject, dstBucket, dstObject, srcInfo, srcOpts, dstOpts)
 		if err == nil {
@@ -1508,16 +1456,10 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
 		return oi, err
 	}
 
-	poolIdx, err := z.getPoolIdxNoLock(ctx, dstBucket, dstObject, srcInfo.Size, dataMovementDstPool(dstOpts))
+	poolIdx, err := z.getPoolIdxNoLock(ctx, dstBucket, dstObject, srcInfo.Size)
 	if err != nil {
 		return objInfo, err
 	}
-	if dataMovementDstPool(dstOpts) != nil && poolIdx == dstOpts.SrcPoolIdx {
-		return ObjectInfo{}, DataMovementOverwriteErr{
-			Bucket: dstBucket, Object: dstObject, VersionID: dstOpts.VersionID,
-			Err: errDataMovementSrcDstPoolSame,
-		}
-	}
 
 	if !z.SinglePool() && dstOpts.ReplicaLockReconcile {
 		dstOpts.replicaObjectInfo = z.replicaObjectInfo
@@ -1977,41 +1919,29 @@ func (z *erasureServerPools) NewMultipartUpload(ctx context.Context, bucket, obj
 	}()
 
 	if z.SinglePool() {
-		idx, err := z.getPoolIdx(ctx, bucket, object, -1, dataMovementDstPool(opts))
-		if err != nil {
-			return nil, err
-		}
-		if dataMovementDstPool(opts) != nil && idx == opts.SrcPoolIdx {
-			return nil, DataMovementOverwriteErr{
-				Bucket: bucket, Object: object, VersionID: opts.VersionID,
-				Err: errDataMovementSrcDstPoolSame,
-			}
-		}
 		return z.serverPools[0].NewMultipartUpload(ctx, bucket, object, opts)
 	}
 
-	if dataMovementDstPool(opts) == nil {
+	for idx, pool := range z.serverPools {
-		for idx, pool := range z.serverPools {
+		if z.IsSuspended(idx) || z.IsPoolRebalancing(idx) {
-			if z.IsSuspended(idx) || z.IsPoolRebalancing(idx) {
+			continue
-				continue
+		}
-			}
 
-			result, err := pool.ListMultipartUploads(ctx, bucket, object, "", "", "", maxUploadsList)
+		result, err := pool.ListMultipartUploads(ctx, bucket, object, "", "", "", maxUploadsList)
-			if err != nil {
+		if err != nil {
-				return nil, err
+			return nil, err
-			}
+		}
-			// If there is a multipart upload with the same bucket/object name,
+		// If there is a multipart upload with the same bucket/object name,
-			// create the new multipart in the same pool, this will avoid
+		// create the new multipart in the same pool, this will avoid
-			// creating two multiparts uploads in two different pools.
+		// creating two multiparts uploads in two different pools
-			if len(result.Uploads) != 0 {
+		if len(result.Uploads) != 0 {
-				return z.serverPools[idx].NewMultipartUpload(ctx, bucket, object, opts)
+			return z.serverPools[idx].NewMultipartUpload(ctx, bucket, object, opts)
-			}
 		}
 	}
 
 	// any parallel writes on the object will block for this poolIdx
 	// to return since this holds a read lock on the namespace.
-	idx, err := z.getPoolIdx(ctx, bucket, object, -1, dataMovementDstPool(opts))
+	idx, err := z.getPoolIdx(ctx, bucket, object, -1)
 	if err != nil {
 		return nil, err
 	}
@@ -2045,7 +1975,7 @@ func (z *erasureServerPools) PutObjectPart(ctx context.Context, bucket, object,
 	}
 
 	if z.SinglePool() {
-		_, err := z.getPoolIdx(ctx, bucket, object, data.Size(), dataMovementDstPool(opts))
+		_, err := z.getPoolIdx(ctx, bucket, object, data.Size())
 		if err != nil {
 			return PartInfo{}, err
 		}
@@ -2224,6 +2154,47 @@ func (z *erasureServerPools) CompleteMultipartUpload(ctx context.Context, bucket
 			defer lk.Unlock(lkctx)
 		}
 		opts.NoLock = true
+
+		// A conditional completion must be evaluated against the logical
+		// latest object across pools, under the object write lock held for
+		// this operation. The pool hosting the upload may only hold a stale
+		// duplicate, so its set-local check would both accept an outdated
+		// ETag and reject the current one. An unreadable pool is not
+		// absence: it may hold the newest copy, so a read that cannot be
+		// verified fails the request instead of passing the condition.
+		// Once satisfied, the callback is cleared so the set layer does not
+		// re-evaluate it against its local copy.
+		if opts.CheckPrecondFn != nil {
+			copies, lerr := z.objectPoolInfos(ctx, bucket, encodeDirObject(object), ObjectOptions{
+				// Conditions always compare the logical current object,
+				// independently of the completion's destination version.
+				VersionID:        "",
+				Versioned:        opts.Versioned,
+				VersionSuspended: opts.VersionSuspended,
+				NoAuditLog:       true,
+			})
+			var latest ObjectInfo
+			if lerr == nil {
+				latest = copies[0].ObjInfo
+				if latest.DeleteMarker {
+					// A delete-marker latest reads as an absent key, matching
+					// the set layer's getObjectInfo.
+					lerr = toObjectErr(errFileNotFound, bucket, object)
+				}
+			}
+			if lerr == nil && opts.CheckPrecondFn(latest) {
+				return ObjectInfo{}, PreConditionFailed{}
+			}
+			if lerr != nil && !isErrVersionNotFound(lerr) && !isErrObjectNotFound(lerr) {
+				return ObjectInfo{}, lerr
+			}
+			// if object doesn't exist return error for If-Match conditional requests
+			// If-None-Match should be allowed to proceed for non-existent objects
+			if lerr != nil && opts.HasIfMatch && (isErrObjectNotFound(lerr) || isErrVersionNotFound(lerr)) {
+				return ObjectInfo{}, lerr
+			}
+			opts.CheckPrecondFn = nil
+		}
 	}
 
 	// Hold write locks to verify uploaded parts, also disallows any
@@ -3219,7 +3190,7 @@ func (z *erasureServerPools) DecomTieredObject(ctx context.Context, bucket, obje
 		defer ns.Unlock(lkctx)
 		opts.NoLock = true
 	}
-	idx, err := z.getPoolIdxNoLock(ctx, bucket, object, fi.Size, dataMovementDstPool(opts))
+	idx, err := z.getPoolIdxNoLock(ctx, bucket, object, fi.Size)
 	if err != nil {
 		return err
 	}

cmd/handler-utils.go

@@ -246,16 +246,6 @@ func extractMetadata(ctx context.Context, mimesHeader ...textproto.MIMEHeader) (
 
 // extractMetadata extracts metadata from map values.
 func extractMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
-	return extractMetadataFromMimeWithReplication(ctx, v, m, false)
-}
-
-// extractReplicationMetadataFromMime restores replication-only metadata after the
-// caller has validated that the request is a trusted replication write.
-func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
-	return extractMetadataFromMimeWithReplication(ctx, v, m, true)
-}
-
-func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIMEHeader, m map[string]string, allowReplication bool) error {
 	if v == nil {
 		bugLogIf(ctx, errInvalidArgument)
 		return errInvalidArgument
@@ -267,18 +257,14 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
 		nv[http.CanonicalHeaderKey(k)] = kv
 	}
 
-	// Save all supported headers.
+	// Save ordinary object metadata. Replication-only headers are restored only
+	// after the request has been validated as a trusted replication write.
 	for _, supportedHeader := range supportedHeaders {
-		value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]
+		if _, ok := replicationToInternalHeaders[supportedHeader]; ok {
-		if ok {
+			continue
-			if v, ok := replicationToInternalHeaders[supportedHeader]; ok {
+		}
-				if !allowReplication {
+		if value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]; ok {
-					continue
+			m[supportedHeader] = strings.Join(value, ",")
-				}
-				m[v] = strings.Join(value, ",")
-			} else {
-				m[supportedHeader] = strings.Join(value, ",")
-			}
 		}
 	}
 
@@ -287,8 +273,7 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
 			if !stringsHasPrefixFold(key, prefix) {
 				continue
 			}
-			value, ok := nv[http.CanonicalHeaderKey(key)]
+			if value, ok := nv[http.CanonicalHeaderKey(key)]; ok {
-			if ok {
 				m[key] = strings.Join(value, ",")
 				break
 			}
@@ -297,6 +282,31 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
 	return nil
 }
 
+// extractReplicationMetadataFromMime restores replication-only metadata after the
+// caller has validated that the request is a trusted replication write.
+func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
+	if v == nil {
+		bugLogIf(ctx, errInvalidArgument)
+		return errInvalidArgument
+	}
+
+	nv := make(textproto.MIMEHeader, len(v))
+	for k, kv := range v {
+		// Canonicalize all headers, to remove any duplicates.
+		nv[http.CanonicalHeaderKey(k)] = kv
+	}
+
+	// Ordinary object metadata belongs to the caller. Re-extracting it would
+	// undo normalization (such as removing aws-chunked) or copy an outer
+	// Snowball archive's metadata onto its individual entries.
+	for header, internalHeader := range replicationToInternalHeaders {
+		if value, ok := nv[http.CanonicalHeaderKey(header)]; ok {
+			m[internalHeader] = strings.Join(value, ",")
+		}
+	}
+	return nil
+}
+
 // Returns access credentials in the request Authorization header.
 func getReqAccessCred(r *http.Request, region string) (cred auth.Credentials) {
 	cred, _, _ = getReqAccessKeyV4(r, region, serviceS3)

cmd/handler-utils_test.go

@@ -254,6 +254,9 @@ func TestExtractMetadataFromRequestKeepsQueryCompatibility(t *testing.T) {
 
 func TestExtractReplicationMetadataHeaders(t *testing.T) {
 	header := http.Header{
+		"Content-Type":      []string{"application/wasm"},
+		"Content-Encoding":  []string{"aws-chunked"},
+		"X-Amz-Meta-Source": []string{"client"},
 		"X-Minio-Replication-Server-Side-Encryption-Sealed-Key":     []string{"sealed-key"},
 		"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
 		"X-Minio-Replication-Server-Side-Encryption-Iv":             []string{"iv"},
@@ -262,12 +265,17 @@ func TestExtractReplicationMetadataHeaders(t *testing.T) {
 		ReplicationSsecChecksumHeader:                               []string{"checksum"},
 	}
 
-	metadata := make(map[string]string)
+	metadata := map[string]string{
+		"content-type":      "application/wasm",
+		"x-amz-meta-source": "client",
+	}
 	if err := extractReplicationMetadataFromMime(t.Context(), textproto.MIMEHeader(header), metadata); err != nil {
 		t.Fatalf("failed to extract replication metadata: %v", err)
 	}
 
 	expected := map[string]string{
+		"content-type":      "application/wasm",
+		"x-amz-meta-source": "client",
 		"X-Minio-Internal-Server-Side-Encryption-Sealed-Key":     "sealed-key",
 		"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "DAREv2-HMAC-SHA256",
 		"X-Minio-Internal-Server-Side-Encryption-Iv":             "iv",
@@ -279,6 +287,9 @@ func TestExtractReplicationMetadataHeaders(t *testing.T) {
 	if !reflect.DeepEqual(metadata, expected) {
 		t.Fatalf("unexpected replication metadata: expected %#v, got %#v", expected, metadata)
 	}
+	if _, ok := metadata["content-encoding"]; ok {
+		t.Fatalf("replication metadata restored transport content-encoding: %#v", metadata)
+	}
 }
 
 func TestGetCopyObjectMetadataFromHeaderReplication(t *testing.T) {

cmd/iam-credential-retention_test.go

@@ -0,0 +1,111 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/minio/minio/internal/auth"
+)
+
+func TestIAMCredentialRetention(t *testing.T) {
+	for _, backend := range []string{"object", "etcd"} {
+		t.Run(backend, func(t *testing.T) {
+			ctx, sys, _ := prepareIAMRevisionFixture(t, backend)
+			secret, err := getTokenSigningKey()
+			mustIAM(t, err)
+			parent := "external-idp-parent"
+			credential := func(exp time.Time) auth.Credentials {
+				cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": exp.Unix(), parentClaim: parent}, secret)
+				mustIAM(t, err)
+				cred.ParentUser = parent
+				return cred
+			}
+			// Disablement of an external identity must include cached STS,
+			// which are kept separately from regular and service accounts.
+			cred := credential(UTCNow().Add(time.Hour))
+			_, err = sys.SetTempUser(ctx, cred.AccessKey, cred, "")
+			mustIAM(t, err)
+			mustIAM(t, sys.store.DeleteUsers(ctx, []string{parent}))
+			r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath(cred.AccessKey, stsUser))
+			mustIAM(t, err)
+			if !r.Deleted || !r.ExpiresAt.Equal(cred.Expiration.Add(globalMaxSkewTime)) || r.Credentials.SessionToken != "" || r.Credentials.SecretKey != "" {
+				t.Fatal("early STS revocation lost its retention boundary or retained a secret")
+			}
+			if _, ok := sys.store.GetUser(cred.AccessKey); ok {
+				t.Fatal("external disablement left the STS cache live")
+			}
+			_, err = sys.SetTempUser(withIAMReplicationTime(ctx, UTCNow().Add(time.Minute)), cred.AccessKey, cred, "")
+			if !errors.Is(err, errIAMStaleUpdate) {
+				t.Fatalf("same revoked token was reissued by replay: %v", err)
+			}
+			var mp MappedPolicy
+			err = sys.store.loadIAMConfig(ctx, &mp, getMappedPolicyPath(cred.AccessKey, stsUser, false))
+			if !errors.Is(err, errConfigNotFound) {
+				t.Fatalf("random STS key produced a permanent mapping: %v", err)
+			}
+
+			// Seed genuinely expired immutable tokens, as an ordinary startup
+			// loader sees them. Natural expiry leaves no permanent tombstone.
+			expired := credential(UTCNow().Add(-time.Hour))
+			path := getUserIdentityPath(expired.AccessKey, stsUser)
+			mustIAM(t, sys.store.saveIAMConfig(ctx, &UserIdentity{Version: 1, Credentials: expired, UpdatedAt: UTCNow().Add(-2 * time.Hour)}, path))
+			_ = sys.store.loadUser(ctx, expired.AccessKey, stsUser, make(map[string]UserIdentity))
+			var u UserIdentity
+			if err := sys.store.loadIAMConfig(ctx, &u, path); !errors.Is(err, errConfigNotFound) {
+				t.Fatalf("natural expiration retained a random key: %v", err)
+			}
+			// A retained early-revocation record is collectable only after the
+			// immutable token's expiration plus the skew allowance.
+			tomb := UserIdentity{Version: 1, Deleted: true, UpdatedAt: UTCNow().Add(-2 * time.Hour), ExpiresAt: expired.Expiration.Add(globalMaxSkewTime)}
+			mustIAM(t, sys.store.saveIAMConfig(ctx, &tomb, path))
+			_ = sys.store.loadUser(ctx, expired.AccessKey, stsUser, make(map[string]UserIdentity))
+			if err := sys.store.loadIAMConfig(ctx, &u, path); !errors.Is(err, errConfigNotFound) {
+				t.Fatalf("expired STS revocation not collected: %v", err)
+			}
+			if _, ok := sys.store.revisionIndex().snapshot()[path]; ok {
+				t.Fatal("expired STS retained an index entry")
+			}
+		})
+	}
+}
+
+func TestIAMPolicyDeletionRemainsExplicit(t *testing.T) {
+	for _, backend := range []string{"object", "etcd"} {
+		t.Run(backend, func(t *testing.T) {
+			ctx, sys, _ := prepareIAMRevisionFixture(t, backend)
+			mustIAM(t, sys.DeletePolicy(ctx, "misspelled-policy", true))
+			r, err := loadIAMRevision(ctx, sys.store, getPolicyDocPath("misspelled-policy"))
+			mustIAM(t, err)
+			if r.Deleted {
+				t.Fatal("local nonexistent policy created a tombstone")
+			}
+			p, err := sys.store.GetPolicy("readwrite")
+			mustIAM(t, err)
+			if err := sys.DeletePolicy(ctx, "readwrite", true); err == nil {
+				t.Fatal("local pristine builtin policy became deletable")
+			}
+			_, err = sys.SetPolicy(ctx, "readwrite", p)
+			mustIAM(t, err)
+			mustIAM(t, sys.DeletePolicy(ctx, "readwrite", true))
+			mustIAM(t, sys.store.LoadIAMCache(ctx, false))
+			if _, err := sys.store.GetPolicy("readwrite"); !errors.Is(err, errNoSuchPolicy) {
+				t.Fatalf("reload restored an explicitly deleted override: %v", err)
+			}
+			_, err = sys.SetPolicy(ctx, "readwrite", p)
+			mustIAM(t, err)
+			if _, err := sys.store.GetPolicy("readwrite"); err != nil {
+				t.Fatal("explicit policy recreation failed", err)
+			}
+			mustIAM(t, globalSiteReplicationSys.PeerAddPolicyHandler(ctx, "remote-unknown-policy", nil, UTCNow()))
+			r, err = loadIAMRevision(ctx, sys.store, getPolicyDocPath("remote-unknown-policy"))
+			mustIAM(t, err)
+			if !r.Deleted {
+				t.Fatal("replicated unknown deletion lost its version")
+			}
+		})
+	}
+}

cmd/iam-etcd-store.go

@@ -26,7 +26,6 @@ import (
 	"sync"
 	"time"
 
-	jsoniter "github.com/json-iterator/go"
 	"github.com/minio/minio-go/v7/pkg/set"
 	"github.com/minio/minio/internal/config"
 	"github.com/minio/minio/internal/kms"
@@ -62,6 +61,7 @@ type IAMEtcdStore struct {
 	sync.RWMutex
 
 	*iamCache
+	index iamRevisionIndex
 
 	usersSysType UsersSysType
 
@@ -69,13 +69,17 @@ type IAMEtcdStore struct {
 }
 
 func newIAMEtcdStore(client *etcd.Client, usersSysType UsersSysType) *IAMEtcdStore {
-	return &IAMEtcdStore{
+	store := &IAMEtcdStore{
 		iamCache:     newIamCache(),
 		client:       client,
 		usersSysType: usersSysType,
 	}
+	store.revisions = &store.index
+	return store
 }
 
+func (ies *IAMEtcdStore) revisionIndex() *iamRevisionIndex { return &ies.index }
+
 func (ies *IAMEtcdStore) rlock() *iamCache {
 	ies.RLock()
 	return ies.iamCache
@@ -103,6 +107,7 @@ func (ies *IAMEtcdStore) saveIAMConfig(ctx context.Context, item any, itemPath s
 	if err != nil {
 		return err
 	}
+	plain := data
 	if GlobalKMS != nil {
 		data, err = config.EncryptBytes(GlobalKMS, data, kms.Context{
 			minioMetaBucket: path.Join(minioMetaBucket, itemPath),
@@ -111,24 +116,28 @@ func (ies *IAMEtcdStore) saveIAMConfig(ctx context.Context, item any, itemPath s
 			return err
 		}
 	}
-	return saveKeyEtcd(ctx, ies.client, itemPath, data, opts...)
+	if err := saveKeyEtcd(ctx, ies.client, itemPath, data, opts...); err != nil {
+		return err
+	}
+	ies.index.observe(itemPath, plain)
+	return nil
 }
 
-func getIAMConfig(item any, data []byte, itemPath string) error {
+func (ies *IAMEtcdStore) decodeIAMConfig(item any, data []byte, path string) error {
-	data, err := decryptData(data, itemPath)
+	data, err := decryptData(data, path)
 	if err != nil {
 		return err
 	}
-	json := jsoniter.ConfigCompatibleWithStandardLibrary
+	ies.index.observe(path, data)
 	return json.Unmarshal(data, item)
 }
 
 func (ies *IAMEtcdStore) loadIAMConfig(ctx context.Context, item any, path string) error {
-	data, err := readKeyEtcd(ctx, ies.client, path)
+	data, err := ies.loadIAMConfigBytes(ctx, path)
 	if err != nil {
 		return err
 	}
-	return getIAMConfig(item, data, path)
+	return json.Unmarshal(data, item)
 }
 
 func (ies *IAMEtcdStore) loadIAMConfigBytes(ctx context.Context, path string) ([]byte, error) {
@@ -136,11 +145,19 @@ func (ies *IAMEtcdStore) loadIAMConfigBytes(ctx context.Context, path string) ([
 	if err != nil {
 		return nil, err
 	}
-	return decryptData(data, path)
+	data, err = decryptData(data, path)
+	if err == nil {
+		ies.index.observe(path, data)
+	}
+	return data, err
 }
 
 func (ies *IAMEtcdStore) deleteIAMConfig(ctx context.Context, path string) error {
-	return deleteKeyEtcd(ctx, ies.client, path)
+	if err := deleteKeyEtcd(ctx, ies.client, path); err != nil {
+		return err
+	}
+	ies.index.forget(path)
+	return nil
 }
 
 func (ies *IAMEtcdStore) loadPolicyDocWithRetry(ctx context.Context, policy string, m map[string]PolicyDoc, _ int) error {
@@ -162,6 +179,9 @@ func (ies *IAMEtcdStore) loadPolicyDoc(ctx context.Context, policy string, m map
 		return err
 	}
 
+	if p.Deleted {
+		return errNoSuchPolicy
+	}
 	m[policy] = p
 	return nil
 }
@@ -181,7 +201,11 @@ func (ies *IAMEtcdStore) getPolicyDocKV(ctx context.Context, kvs *mvccpb.KeyValu
 		return err
 	}
 
+	ies.index.observe(string(kvs.Key), data)
 	policy := extractPathPrefixAndSuffix(string(kvs.Key), iamConfigPoliciesPrefix, path.Base(string(kvs.Key)))
+	if p.Deleted {
+		return errNoSuchPolicy
+	}
 	m[policy] = p
 	return nil
 }
@@ -207,7 +231,7 @@ func (ies *IAMEtcdStore) loadPolicyDocs(ctx context.Context, m map[string]Policy
 
 func (ies *IAMEtcdStore) getUserKV(ctx context.Context, userkv *mvccpb.KeyValue, userType IAMUserType, m map[string]UserIdentity, basePrefix string) error {
 	var u UserIdentity
-	err := getIAMConfig(&u, userkv.Value, string(userkv.Key))
+	err := ies.decodeIAMConfig(&u, userkv.Value, string(userkv.Key))
 	if err != nil {
 		if err == errConfigNotFound {
 			return errNoSuchUser
@@ -219,10 +243,14 @@ func (ies *IAMEtcdStore) getUserKV(ctx context.Context, userkv *mvccpb.KeyValue,
 }
 
 func (ies *IAMEtcdStore) addUser(ctx context.Context, user string, userType IAMUserType, u UserIdentity, m map[string]UserIdentity) error {
+	if u.Deleted {
+		if userType == stsUser && !u.ExpiresAt.IsZero() && UTCNow().After(u.ExpiresAt) {
+			bestEffortIAMExpiration(ctx, ies, getUserIdentityPath(user, userType))
+		}
+		return errNoSuchUser
+	}
 	if u.Credentials.IsExpired() {
-		// Delete expired identity.
+		bestEffortIAMExpiration(ctx, ies, getUserIdentityPath(user, userType))
-		deleteKeyEtcd(ctx, ies.client, getUserIdentityPath(user, userType))
-		deleteKeyEtcd(ctx, ies.client, getMappedPolicyPath(user, userType, false))
 		return nil
 	}
 	if u.Credentials.AccessKey == "" {
@@ -231,16 +259,17 @@ func (ies *IAMEtcdStore) addUser(ctx context.Context, user string, userType IAMU
 	if u.Credentials.SessionToken != "" {
 		jwtClaims, err := extractJWTClaims(u)
 		if err != nil {
-			if u.Credentials.IsTemp() {
+			// A temporarily unavailable signing key is not proof of expiration.
-				// We should delete such that the client can re-request
-				// for the expiring credentials.
-				deleteKeyEtcd(ctx, ies.client, getUserIdentityPath(user, userType))
-				deleteKeyEtcd(ctx, ies.client, getMappedPolicyPath(user, userType, false))
-			}
 			return nil
 		}
 		u.Credentials.Claims = jwtClaims.Map()
 	}
+	if err := checkIAMParentRevision(ctx, ies, u.Credentials); err != nil {
+		if errors.Is(err, errIAMStaleUpdate) {
+			return errNoSuchUser
+		}
+		return err
+	}
 	if u.Credentials.Description == "" {
 		u.Credentials.Description = u.Credentials.Comment
 	}
@@ -258,6 +287,9 @@ func (ies *IAMEtcdStore) loadSecretKey(ctx context.Context, user string, userTyp
 		}
 		return "", err
 	}
+	if u.Deleted {
+		return "", errNoSuchUser
+	}
 	return u.Credentials.SecretKey, nil
 }
 
@@ -274,6 +306,7 @@ func (ies *IAMEtcdStore) loadUser(ctx context.Context, user string, userType IAM
 }
 
 func (ies *IAMEtcdStore) loadUsers(ctx context.Context, userType IAMUserType, m map[string]UserIdentity) error {
+	ctx = withIAMExpirationCleanup(ctx)
 	var basePrefix string
 	switch userType {
 	case svcUser:
@@ -312,6 +345,9 @@ func (ies *IAMEtcdStore) loadGroup(ctx context.Context, group string, m map[stri
 		}
 		return err
 	}
+	if gi.Deleted {
+		return errNoSuchGroup
+	}
 	m[group] = gi
 	return nil
 }
@@ -349,13 +385,16 @@ func (ies *IAMEtcdStore) loadMappedPolicy(ctx context.Context, name string, user
 		}
 		return err
 	}
+	if !ies.index.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), p) {
+		return errNoSuchPolicy
+	}
 	m.Store(name, p)
 	return nil
 }
 
-func getMappedPolicy(kv *mvccpb.KeyValue, m *xsync.MapOf[string, MappedPolicy], basePrefix string) error {
+func (ies *IAMEtcdStore) getMappedPolicy(kv *mvccpb.KeyValue, m *xsync.MapOf[string, MappedPolicy], basePrefix string) error {
 	var p MappedPolicy
-	err := getIAMConfig(&p, kv.Value, string(kv.Key))
+	err := ies.decodeIAMConfig(&p, kv.Value, string(kv.Key))
 	if err != nil {
 		if err == errConfigNotFound {
 			return errNoSuchPolicy
@@ -363,6 +402,9 @@ func getMappedPolicy(kv *mvccpb.KeyValue, m *xsync.MapOf[string, MappedPolicy],
 		return err
 	}
 	name := extractPathPrefixAndSuffix(string(kv.Key), basePrefix, ".json")
+	if !ies.index.mappingAllowed(string(kv.Key), p) {
+		return errNoSuchPolicy
+	}
 	m.Store(name, p)
 	return nil
 }
@@ -392,61 +434,13 @@ func (ies *IAMEtcdStore) loadMappedPolicies(ctx context.Context, userType IAMUse
 
 	// Parse all policies mapping to create the proper data model
 	for _, kv := range r.Kvs {
-		if err = getMappedPolicy(kv, m, basePrefix); err != nil && !errors.Is(err, errNoSuchPolicy) {
+		if err = ies.getMappedPolicy(kv, m, basePrefix); err != nil && !errors.Is(err, errNoSuchPolicy) {
 			return err
 		}
 	}
 	return nil
 }
 
-func (ies *IAMEtcdStore) savePolicyDoc(ctx context.Context, policyName string, p PolicyDoc) error {
-	return ies.saveIAMConfig(ctx, &p, getPolicyDocPath(policyName))
-}
-
-func (ies *IAMEtcdStore) saveMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool, mp MappedPolicy, opts ...options) error {
-	return ies.saveIAMConfig(ctx, mp, getMappedPolicyPath(name, userType, isGroup), opts...)
-}
-
-func (ies *IAMEtcdStore) saveUserIdentity(ctx context.Context, name string, userType IAMUserType, u UserIdentity, opts ...options) error {
-	return ies.saveIAMConfig(ctx, u, getUserIdentityPath(name, userType), opts...)
-}
-
-func (ies *IAMEtcdStore) saveGroupInfo(ctx context.Context, name string, gi GroupInfo) error {
-	return ies.saveIAMConfig(ctx, gi, getGroupInfoPath(name))
-}
-
-func (ies *IAMEtcdStore) deletePolicyDoc(ctx context.Context, name string) error {
-	err := ies.deleteIAMConfig(ctx, getPolicyDocPath(name))
-	if err == errConfigNotFound {
-		err = errNoSuchPolicy
-	}
-	return err
-}
-
-func (ies *IAMEtcdStore) deleteMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool) error {
-	err := ies.deleteIAMConfig(ctx, getMappedPolicyPath(name, userType, isGroup))
-	if err == errConfigNotFound {
-		err = errNoSuchPolicy
-	}
-	return err
-}
-
-func (ies *IAMEtcdStore) deleteUserIdentity(ctx context.Context, name string, userType IAMUserType) error {
-	err := ies.deleteIAMConfig(ctx, getUserIdentityPath(name, userType))
-	if err == errConfigNotFound {
-		err = errNoSuchUser
-	}
-	return err
-}
-
-func (ies *IAMEtcdStore) deleteGroupInfo(ctx context.Context, name string) error {
-	err := ies.deleteIAMConfig(ctx, getGroupInfoPath(name))
-	if err == errConfigNotFound {
-		err = errNoSuchGroup
-	}
-	return err
-}
-
 func (ies *IAMEtcdStore) watch(ctx context.Context, keyPath string) <-chan iamWatchEvent {
 	ch := make(chan iamWatchEvent)
 

cmd/iam-group-revision.go

@@ -0,0 +1,164 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"maps"
+	"slices"
+	"time"
+
+	"github.com/minio/minio-go/v7/pkg/set"
+)
+
+type (
+	iamGroupGrantsKey   struct{}
+	iamGroupMutationKey struct{}
+	iamGroupMutation    struct {
+		Members    []string
+		Remove     bool
+		StatusOnly bool
+	}
+)
+
+// Merge the intended mutation with the record read under the distributed
+// revision lock, not the older cache used to prepare the request.
+func mergeIAMGroupMutation(ctx context.Context, previous GroupInfo, next *GroupInfo) {
+	op, ok := ctx.Value(iamGroupMutationKey{}).(iamGroupMutation)
+	if !ok || previous.Deleted || previous.Version == 0 {
+		return
+	}
+	members := set.CreateStringSet(previous.Members...)
+	grants := maps.Clone(previous.MemberGrants)
+	if grants == nil {
+		grants = make(map[string]time.Time)
+	}
+	switch {
+	case op.StatusOnly:
+		// Only the status changes.
+	case op.Remove:
+		for _, member := range op.Members {
+			members.Remove(member)
+			delete(grants, member)
+		}
+		next.Status = previous.Status
+	default:
+		requested := set.CreateStringSet(next.Members...)
+		for _, member := range op.Members {
+			if !requested.Contains(member) {
+				continue
+			}
+			at := next.MemberGrants[member]
+			if at.Before(grants[member]) {
+				continue
+			}
+			members.Add(member)
+			grants[member] = at
+		}
+		next.Status = previous.Status
+	}
+	next.Members, next.MemberGrants = members.ToSlice(), grants
+	slices.Sort(next.Members)
+}
+
+// A non-nil map is supplied by the versioned peer envelope, including for
+// snapshots. Missing times are unknown, never the snapshot's newer timestamp.
+func withIAMGroupGrants(ctx context.Context, grants map[string]time.Time) context.Context {
+	return context.WithValue(ctx, iamGroupGrantsKey{}, grants)
+}
+
+func (c *iamCache) effectiveGroupMembers(gi GroupInfo) []string {
+	var members []string
+	for _, member := range gi.Members {
+		if c.groupMemberAllowed(member, gi.MemberGrants[member], gi.RevokedBefore) {
+			members = append(members, member)
+		}
+	}
+	return members
+}
+
+func (c *iamCache) effectiveUserGroups(user string) []string {
+	var groups []string
+	for group := range c.iamUserGroupMemberships[user] {
+		gi, ok := c.iamGroupsMap[group]
+		r := c.revisions.get(getGroupInfoPath(group))
+		if r.RevokedBefore.After(gi.RevokedBefore) {
+			gi.RevokedBefore = r.RevokedBefore
+		}
+		if ok && !r.Deleted && c.groupMemberAllowed(user, gi.MemberGrants[user], gi.RevokedBefore) {
+			groups = append(groups, group)
+		}
+	}
+	return groups
+}
+
+func (c *iamCache) addGroupMembers(ctx context.Context, gi GroupInfo, members []string) (GroupInfo, error) {
+	grants, versioned := ctx.Value(iamGroupGrantsKey{}).(map[string]time.Time)
+	if boundary, ok := ctx.Value(iamRecordBoundaryKey{}).(time.Time); ok && boundary.After(gi.RevokedBefore) {
+		gi.RevokedBefore = boundary
+	}
+	origin, replicated := iamReplicationTime(ctx)
+	gi.Members = slices.Clone(gi.Members)
+	gi.MemberGrants = maps.Clone(gi.MemberGrants)
+	if gi.MemberGrants == nil {
+		gi.MemberGrants = make(map[string]time.Time)
+	}
+	current := set.CreateStringSet(gi.Members...)
+	gi.UpdatedAt = UTCNow()
+	if replicated {
+		gi.UpdatedAt = origin
+	}
+	for _, member := range members {
+		at := gi.UpdatedAt
+		r := c.userRevocation(member)
+		if replicated {
+			switch {
+			case versioned:
+				at = grants[member]
+				if at.After(origin) {
+					return gi, errInvalidArgument
+				}
+			case !r.RevokedBefore.IsZero() || r.Deleted || !gi.RevokedBefore.IsZero():
+				// Legacy snapshots cannot prove a post-revocation grant.
+				continue
+			case current.Contains(member):
+				continue
+			}
+			if !c.groupMemberAllowed(member, at, gi.RevokedBefore) {
+				continue
+			}
+		} else {
+			if current.Contains(member) && c.groupMemberAllowed(member, gi.MemberGrants[member], gi.RevokedBefore) {
+				continue // Editing the group is not reissuing every grant.
+			}
+			if !at.After(gi.RevokedBefore) {
+				at = gi.RevokedBefore.Add(time.Nanosecond)
+			}
+			if !at.After(r.RevokedBefore) {
+				at = r.RevokedBefore.Add(time.Nanosecond)
+			}
+			if !at.After(gi.MemberGrants[member]) {
+				at = gi.MemberGrants[member].Add(time.Nanosecond)
+			}
+			if at.After(gi.UpdatedAt) {
+				gi.UpdatedAt = at
+			}
+		}
+		u, ok := c.iamUsersMap[member]
+		if !ok {
+			return gi, errNoSuchUser
+		}
+		if u.Credentials.IsTemp() || u.Credentials.IsServiceAccount() {
+			return gi, errIAMActionNotAllowed
+		}
+		if previous := gi.MemberGrants[member]; previous.After(at) {
+			continue
+		}
+		current.Add(member)
+		gi.MemberGrants[member] = at
+	}
+	gi.Members = current.ToSlice()
+	slices.Sort(gi.Members)
+	return gi, nil
+}

cmd/iam-healing-leadership_test.go

@@ -0,0 +1,75 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"testing"
+	"time"
+)
+
+func TestIAMHealingResumesAfterLeadershipLoss(t *testing.T) {
+	previous := globalLeaderLock
+	locks := make(chan LockContext)
+	globalLeaderLock = &sharedLock{lockContext: locks}
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	c := &SiteReplicationSys{}
+	go func() { c.startHealRoutine(ctx, nil); close(done) }()
+	t.Cleanup(func() {
+		cancel()
+		select {
+		case <-done:
+		case locks <- LockContext{ctx: ctx}:
+		}
+		<-done
+		globalLeaderLock = previous
+	})
+	first, loseFirst := context.WithCancel(ctx)
+	defer loseFirst()
+	select {
+	case locks <- LockContext{ctx: first}:
+	case <-time.After(time.Second):
+		t.Fatal("healer did not acquire its first leader context")
+	}
+	loseFirst() // A transient quorum loss cancels the distributed lease.
+	select {
+	case <-done:
+		t.Fatal("healer permanently exited after temporary leadership loss")
+	case locks <- LockContext{ctx: ctx}:
+	case <-time.After(time.Second):
+		t.Fatal("healer did not wait for reacquired leadership")
+	}
+	// Shutdown must also interrupt the wait for leadership after lease loss.
+	cancel()
+	select {
+	case <-done:
+	case <-time.After(time.Second):
+		t.Fatal("healer did not stop with its owning context")
+	}
+}
+
+func TestIAMHealingLeadershipWaitCancels(t *testing.T) {
+	previous := globalLeaderLock
+	locks := make(chan LockContext)
+	globalLeaderLock = &sharedLock{lockContext: locks}
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	c := &SiteReplicationSys{}
+	go func() { c.startHealRoutine(ctx, nil); close(done) }()
+	cancel()
+	t.Cleanup(func() {
+		select {
+		case <-done:
+		case locks <- LockContext{ctx: ctx}:
+		}
+		<-done
+		globalLeaderLock = previous
+	})
+	select {
+	case <-done:
+	case <-time.After(time.Second):
+		t.Fatal("healer ignored shutdown while waiting for leadership")
+	}
+}

cmd/iam-healing-performance_test.go

@@ -0,0 +1,60 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+)
+
+// Measures steady-state index traversal, sorting and the capability request.
+// The network peer acknowledges real batches but performs no disk I/O; this
+// benchmark deliberately does not claim durable catch-up throughput.
+func BenchmarkIAMRevisionConvergedHealing(b *testing.B) {
+	for _, n := range []int{1000, 10000} {
+		b.Run(fmt.Sprint(n), func(b *testing.B) {
+			ctx, sys, _ := prepareIAMRevisionFixture(b)
+			_, err := sys.CreateUser(ctx, "benchmark-sync", madmin.AddOrUpdateUserReq{SecretKey: "valid-sync-password", Status: madmin.AccountEnabled})
+			mustIAM(b, err)
+			for i := range n {
+				at := UTCNow().Add(time.Duration(i) * time.Nanosecond)
+				data, err := json.Marshal(iamRevision{Deleted: true, UpdatedAt: at, RevokedBefore: at})
+				mustIAM(b, err)
+				sys.store.revisionIndex().observe(getUserIdentityPath(fmt.Sprintf("deleted-%06d", i), regUser), data)
+			}
+			var puts atomic.Int64
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				if r.URL.Path == "/minio/health/live" {
+					w.WriteHeader(http.StatusOK)
+					return
+				}
+				if r.Method == http.MethodPut {
+					puts.Add(1)
+				}
+				_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "node-1", Instance: "benchmark-peer", Digest: "constant"}})
+			}))
+			defer server.Close()
+			c := &SiteReplicationSys{enabled: true, state: srState{ServiceAccountAccessKey: "benchmark-sync", Peers: map[string]madmin.PeerInfo{globalDeploymentID(): {DeploymentID: globalDeploymentID(), Name: "local"}, "remote": {DeploymentID: "remote", Name: "remote", Endpoint: server.URL}}}}
+			mustIAM(b, c.healIAMDeletions(ctx))
+			before := puts.Load()
+			b.ReportAllocs()
+			b.ResetTimer()
+			for b.Loop() {
+				mustIAM(b, c.healIAMDeletions(ctx))
+			}
+			b.StopTimer()
+			b.ReportMetric(float64(puts.Load()-before)/float64(b.N), "PUT/op")
+			if puts.Load() != before {
+				b.Fatal("steady-state healing replayed acknowledged records")
+			}
+		})
+	}
+}

cmd/iam-object-store.go

@@ -45,6 +45,7 @@ type IAMObjectStore struct {
 	sync.RWMutex
 
 	*iamCache
+	index iamRevisionIndex
 
 	usersSysType UsersSysType
 
@@ -52,13 +53,17 @@ type IAMObjectStore struct {
 }
 
 func newIAMObjectStore(objAPI ObjectLayer, usersSysType UsersSysType) *IAMObjectStore {
-	return &IAMObjectStore{
+	store := &IAMObjectStore{
 		iamCache:     newIamCache(),
 		objAPI:       objAPI,
 		usersSysType: usersSysType,
 	}
+	store.revisions = &store.index
+	return store
 }
 
+func (iamOS *IAMObjectStore) revisionIndex() *iamRevisionIndex { return &iamOS.index }
+
 func (iamOS *IAMObjectStore) rlock() *iamCache {
 	iamOS.RLock()
 	return iamOS.iamCache
@@ -87,6 +92,7 @@ func (iamOS *IAMObjectStore) saveIAMConfig(ctx context.Context, item any, objPat
 	if err != nil {
 		return err
 	}
+	plain := data
 	if GlobalKMS != nil {
 		data, err = config.EncryptBytes(GlobalKMS, data, kms.Context{
 			minioMetaBucket: path.Join(minioMetaBucket, objPath),
@@ -95,7 +101,11 @@ func (iamOS *IAMObjectStore) saveIAMConfig(ctx context.Context, item any, objPat
 			return err
 		}
 	}
-	return saveConfig(ctx, iamOS.objAPI, objPath, data)
+	if err := saveConfig(ctx, iamOS.objAPI, objPath, data); err != nil {
+		return err
+	}
+	iamOS.index.observe(objPath, plain)
+	return nil
 }
 
 func decryptData(data []byte, objPath string) ([]byte, error) {
@@ -133,6 +143,7 @@ func (iamOS *IAMObjectStore) loadIAMConfigBytesWithMetadata(ctx context.Context,
 	if err != nil {
 		return nil, meta, err
 	}
+	iamOS.index.observe(objPath, data)
 	return data, meta, nil
 }
 
@@ -146,7 +157,11 @@ func (iamOS *IAMObjectStore) loadIAMConfig(ctx context.Context, item any, objPat
 }
 
 func (iamOS *IAMObjectStore) deleteIAMConfig(ctx context.Context, path string) error {
-	return deleteConfig(ctx, iamOS.objAPI, path)
+	if err := deleteConfig(ctx, iamOS.objAPI, path); err != nil {
+		return err
+	}
+	iamOS.index.forget(path)
+	return nil
 }
 
 func (iamOS *IAMObjectStore) loadPolicyDocWithRetry(ctx context.Context, policy string, m map[string]PolicyDoc, retries int) error {
@@ -171,6 +186,10 @@ func (iamOS *IAMObjectStore) loadPolicyDocWithRetry(ctx context.Context, policy
 			return err
 		}
 
+		if p.Deleted {
+			return errNoSuchPolicy
+		}
+
 		if p.Version == 0 {
 			// This means that policy was in the old version (without any
 			// timestamp info). We fetch the mod time of the file and save
@@ -200,6 +219,10 @@ func (iamOS *IAMObjectStore) loadPolicy(ctx context.Context, policy string) (Pol
 		return p, err
 	}
 
+	if p.Deleted {
+		return PolicyDoc{}, errNoSuchPolicy
+	}
+
 	if p.Version == 0 {
 		// This means that policy was in the old version (without any
 		// timestamp info). We fetch the mod time of the file and save
@@ -245,6 +268,9 @@ func (iamOS *IAMObjectStore) loadSecretKey(ctx context.Context, user string, use
 		}
 		return "", err
 	}
+	if u.Deleted {
+		return "", errNoSuchUser
+	}
 	return u.Credentials.SecretKey, nil
 }
 
@@ -258,10 +284,15 @@ func (iamOS *IAMObjectStore) loadUserIdentity(ctx context.Context, user string,
 		return u, err
 	}
 
+	if u.Deleted {
+		if userType == stsUser && !u.ExpiresAt.IsZero() && UTCNow().After(u.ExpiresAt) {
+			bestEffortIAMExpiration(ctx, iamOS, getUserIdentityPath(user, userType))
+		}
+		return UserIdentity{}, errNoSuchUser
+	}
+
 	if u.Credentials.IsExpired() {
-		// Delete expired identity - ignoring errors here.
+		bestEffortIAMExpiration(ctx, iamOS, getUserIdentityPath(user, userType))
-		iamOS.deleteIAMConfig(ctx, getUserIdentityPath(user, userType))
-		iamOS.deleteIAMConfig(ctx, getMappedPolicyPath(user, userType, false))
 		return u, errNoSuchUser
 	}
 
@@ -272,16 +303,18 @@ func (iamOS *IAMObjectStore) loadUserIdentity(ctx context.Context, user string,
 	if u.Credentials.SessionToken != "" {
 		jwtClaims, err := extractJWTClaims(u)
 		if err != nil {
-			if u.Credentials.IsTemp() {
+			// During startup the site signing key may not be available yet.
-				// We should delete such that the client can re-request
+			// Reject this load without deleting a credential that has not expired.
-				// for the expiring credentials.
+			return UserIdentity{}, errNoSuchUser
-				iamOS.deleteIAMConfig(ctx, getUserIdentityPath(user, userType))
-				iamOS.deleteIAMConfig(ctx, getMappedPolicyPath(user, userType, false))
-			}
-			return u, errNoSuchUser
 		}
 		u.Credentials.Claims = jwtClaims.Map()
 	}
+	if err := checkIAMParentRevision(ctx, iamOS, u.Credentials); err != nil {
+		if errors.Is(err, errIAMStaleUpdate) {
+			return UserIdentity{}, errNoSuchUser
+		}
+		return UserIdentity{}, err
+	}
 
 	if u.Credentials.Description == "" {
 		u.Credentials.Description = u.Credentials.Comment
@@ -320,6 +353,7 @@ func (iamOS *IAMObjectStore) loadUser(ctx context.Context, user string, userType
 }
 
 func (iamOS *IAMObjectStore) loadUsers(ctx context.Context, userType IAMUserType, m map[string]UserIdentity) error {
+	ctx = withIAMExpirationCleanup(ctx)
 	var basePrefix string
 	switch userType {
 	case svcUser:
@@ -354,6 +388,9 @@ func (iamOS *IAMObjectStore) loadGroup(ctx context.Context, group string, m map[
 		}
 		return err
 	}
+	if g.Deleted {
+		return errNoSuchGroup
+	}
 	m[group] = g
 	return nil
 }
@@ -391,6 +428,9 @@ func (iamOS *IAMObjectStore) loadMappedPolicyWithRetry(ctx context.Context, name
 			goto retry
 		}
 
+		if !iamOS.index.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), p) {
+			return errNoSuchPolicy
+		}
 		m.Store(name, p)
 		return nil
 	}
@@ -405,6 +445,9 @@ func (iamOS *IAMObjectStore) loadMappedPolicyInternal(ctx context.Context, name
 		}
 		return p, err
 	}
+	if !iamOS.index.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), p) {
+		return MappedPolicy{}, errNoSuchPolicy
+	}
 	return p, nil
 }
 
@@ -824,54 +867,6 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 	return nil
 }
 
-func (iamOS *IAMObjectStore) savePolicyDoc(ctx context.Context, policyName string, p PolicyDoc) error {
-	return iamOS.saveIAMConfig(ctx, &p, getPolicyDocPath(policyName))
-}
-
-func (iamOS *IAMObjectStore) saveMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool, mp MappedPolicy, opts ...options) error {
-	return iamOS.saveIAMConfig(ctx, mp, getMappedPolicyPath(name, userType, isGroup), opts...)
-}
-
-func (iamOS *IAMObjectStore) saveUserIdentity(ctx context.Context, name string, userType IAMUserType, u UserIdentity, opts ...options) error {
-	return iamOS.saveIAMConfig(ctx, u, getUserIdentityPath(name, userType), opts...)
-}
-
-func (iamOS *IAMObjectStore) saveGroupInfo(ctx context.Context, name string, gi GroupInfo) error {
-	return iamOS.saveIAMConfig(ctx, gi, getGroupInfoPath(name))
-}
-
-func (iamOS *IAMObjectStore) deletePolicyDoc(ctx context.Context, name string) error {
-	err := iamOS.deleteIAMConfig(ctx, getPolicyDocPath(name))
-	if err == errConfigNotFound {
-		err = errNoSuchPolicy
-	}
-	return err
-}
-
-func (iamOS *IAMObjectStore) deleteMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool) error {
-	err := iamOS.deleteIAMConfig(ctx, getMappedPolicyPath(name, userType, isGroup))
-	if err == errConfigNotFound {
-		err = errNoSuchPolicy
-	}
-	return err
-}
-
-func (iamOS *IAMObjectStore) deleteUserIdentity(ctx context.Context, name string, userType IAMUserType) error {
-	err := iamOS.deleteIAMConfig(ctx, getUserIdentityPath(name, userType))
-	if err == errConfigNotFound {
-		err = errNoSuchUser
-	}
-	return err
-}
-
-func (iamOS *IAMObjectStore) deleteGroupInfo(ctx context.Context, name string) error {
-	err := iamOS.deleteIAMConfig(ctx, getGroupInfoPath(name))
-	if err == errConfigNotFound {
-		err = errNoSuchGroup
-	}
-	return err
-}
-
 // Lists objects in the minioMetaBucket at the given path prefix. All returned
 // items have the pathPrefix removed from their names.
 func listIAMConfigItems(ctx context.Context, objAPI ObjectLayer, pathPrefix string) <-chan itemOrErr[string] {

cmd/iam-peer-reload_test.go

@@ -0,0 +1,133 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/auth"
+	"github.com/minio/minio/internal/grid"
+	"github.com/pgsty/silo-pkg/v3/policy"
+)
+
+// Two independent IAM caches share the same real object backend, as sibling
+// nodes do. Deliver the actual peer handler only after the source committed.
+func TestIAMPeerDeleteNotificationReloadsCommittedState(t *testing.T) {
+	for _, name := range []string{"deleted", "recreated", "recreated_without_grant"} {
+		recreate := name != "deleted"
+		t.Run(name, func(t *testing.T) {
+			resetTestGlobals()
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+			obj, disk, err := prepareFS(ctx)
+			if err != nil {
+				t.Fatal(err)
+			}
+			defer os.RemoveAll(disk)
+			defer obj.Shutdown(ctx)
+			defer resetTestGlobals()
+			globalObjLayerMutex.Lock()
+			globalObjectAPI = obj
+			globalObjLayerMutex.Unlock()
+			must := func(err error) {
+				t.Helper()
+				if err != nil {
+					t.Fatal(err)
+				}
+			}
+			source := globalIAMSys
+			const user = "peer-reload-user"
+			req := madmin.AddOrUpdateUserReq{SecretKey: "original-test-password", Status: madmin.AccountEnabled}
+			_, err = source.CreateUser(ctx, user, req)
+			must(err)
+			_, err = source.PolicyDBSet(ctx, user, "readwrite", regUser, false)
+			must(err)
+			_, err = source.AddUsersToGroup(ctx, "peer-reload-group", []string{user})
+			must(err)
+			_, err = source.PolicyDBSet(ctx, "peer-reload-group", "readwrite", regUser, true)
+			must(err)
+			svc, _, err := source.NewServiceAccount(ctx, user, nil, newServiceAccountOpts{accessKey: "peer-reload-service", secretKey: "service-test-password"})
+			must(err)
+			signingKey, err := getTokenSigningKey()
+			must(err)
+			sts, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: user}, signingKey)
+			must(err)
+			sts.ParentUser = user
+			_, err = source.SetTempUser(ctx, sts.AccessKey, sts, "")
+			must(err)
+
+			siblingStore := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+			must(siblingStore.LoadIAMCache(ctx, true))
+			must(siblingStore.UserNotificationHandler(ctx, sts.AccessKey, stsUser))
+			for _, key := range []string{user, svc.AccessKey, sts.AccessKey} {
+				if _, ok := siblingStore.GetUser(key); !ok {
+					t.Fatalf("fixture did not load %s", key)
+				}
+			}
+			must(source.DeleteUser(ctx, user, false))
+			if recreate {
+				req.SecretKey = "recreated-test-password"
+				_, err = source.CreateUser(ctx, user, req)
+				must(err)
+				if name == "recreated" {
+					_, err = source.PolicyDBSet(ctx, user, "readonly", regUser, false)
+					must(err)
+				}
+			}
+
+			sibling := &IAMSys{store: siblingStore, usersSysType: MinIOUsersSysType}
+			globalIAMSys = sibling
+			defer func() { globalIAMSys = source }()
+			server := &peerRESTServer{}
+			for range 2 {
+				_, remoteErr := server.DeleteUserHandler(grid.NewMSSWith(map[string]string{peerRESTUser: user}))
+				if remoteErr != nil {
+					t.Fatal(remoteErr)
+				}
+			}
+			if recreate {
+				u, ok := siblingStore.GetUser(user)
+				if !ok || u.Credentials.SecretKey != req.SecretKey {
+					t.Fatal("delayed deletion notification removed the recreated user")
+				}
+				loaded := make(map[string]UserIdentity)
+				must(source.store.loadUser(ctx, user, regUser, loaded))
+				if loaded[user].Credentials.SecretKey != req.SecretKey {
+					t.Fatal("notification changed the persisted recreated identity")
+				}
+				if allowed := sibling.IsAllowed(policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}); allowed != (name == "recreated") {
+					t.Fatal("notification did not load the recreated user's current grant")
+				}
+			}
+			if sibling.IsAllowed(policy.Args{AccountName: user, Action: policy.PutObjectAction, BucketName: "bucket", ObjectName: "object"}) {
+				t.Fatal("notification retained an old direct or group grant")
+			}
+			for _, key := range []string{svc.AccessKey, sts.AccessKey} {
+				if _, ok := siblingStore.GetUser(key); ok {
+					t.Fatalf("notification retained a revoked child: %s", key)
+				}
+			}
+			if !recreate {
+				for _, key := range []string{user} {
+					if _, ok := siblingStore.GetUser(key); ok {
+						t.Fatalf("notification retained a revoked cached identity: %s", key)
+					}
+				}
+				if sibling.IsAllowed(policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}) {
+					t.Fatal("notification retained the user's old grant")
+				}
+				cache := siblingStore.rlock()
+				member := cache.iamUserGroupMemberships[user].Contains("peer-reload-group")
+				siblingStore.runlock()
+				if member {
+					t.Fatal("notification retained the deleted user's group membership")
+				}
+			}
+		})
+	}
+}

cmd/iam-performance_test.go

@@ -0,0 +1,131 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/auth"
+)
+
+// Uses APIs shared with the pre-revision tree so the same benchmark can be
+// overlaid on that tree for a comparable local baseline.
+func prepareIAMPerformanceFixture(b *testing.B) (context.Context, *IAMSys) {
+	b.Helper()
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	disks, err := getRandomDisks(1)
+	if err != nil {
+		b.Fatal(err)
+	}
+	obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
+	if err != nil {
+		b.Fatal(err)
+	}
+	initAllSubsystems(ctx)
+	globalIAMSys.initStore(obj, nil)
+	if err := globalIAMSys.Load(ctx, true); err != nil {
+		b.Fatal(err)
+	}
+	b.Cleanup(func() { cancel(); obj.Shutdown(context.Background()); os.RemoveAll(disks[0]); resetTestGlobals() })
+	return ctx, globalIAMSys
+}
+
+func BenchmarkIAMCachedCredential(b *testing.B) {
+	for _, kind := range []string{"user", "service", "sts"} {
+		b.Run(kind, func(b *testing.B) {
+			ctx, sys := prepareIAMPerformanceFixture(b)
+			const parent = "benchmark-parent"
+			_, err := sys.CreateUser(ctx, parent, madmin.AddOrUpdateUserReq{SecretKey: "benchmark-user-password", Status: madmin.AccountEnabled})
+			if err != nil {
+				b.Fatal(err)
+			}
+			key := parent
+			if kind == "service" {
+				c, _, err := sys.NewServiceAccount(ctx, parent, nil, newServiceAccountOpts{accessKey: "benchmark-service", secretKey: "benchmark-service-password"})
+				if err != nil {
+					b.Fatal(err)
+				}
+				key = c.AccessKey
+			}
+			if kind == "sts" {
+				secret, err := getTokenSigningKey()
+				if err != nil {
+					b.Fatal(err)
+				}
+				c, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
+				if err != nil {
+					b.Fatal(err)
+				}
+				c.ParentUser = parent
+				if _, err := sys.SetTempUser(ctx, c.AccessKey, c, ""); err != nil {
+					b.Fatal(err)
+				}
+				key = c.AccessKey
+			}
+			b.ReportAllocs()
+			b.ResetTimer()
+			for b.Loop() {
+				if _, ok := sys.store.GetUser(key); !ok {
+					b.Fatal("credential missing")
+				}
+			}
+		})
+	}
+}
+
+func BenchmarkIAMSetTempUser(b *testing.B) {
+	ctx, sys := prepareIAMPerformanceFixture(b)
+	const parent = "benchmark-sts-parent"
+	_, err := sys.CreateUser(ctx, parent, madmin.AddOrUpdateUserReq{SecretKey: "benchmark-user-password", Status: madmin.AccountEnabled})
+	if err != nil {
+		b.Fatal(err)
+	}
+	secret, err := getTokenSigningKey()
+	if err != nil {
+		b.Fatal(err)
+	}
+	cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
+	if err != nil {
+		b.Fatal(err)
+	}
+	cred.ParentUser = parent
+	b.ReportAllocs()
+	b.ResetTimer()
+	for b.Loop() {
+		if _, err := sys.SetTempUser(ctx, cred.AccessKey, cred, "readwrite"); err != nil {
+			b.Fatal(err)
+		}
+	}
+}
+
+// Run with -benchtime=1x. Preparation is outside the timer; each measured load
+// sees a fresh set of expired reusable service-account records.
+func BenchmarkIAMColdLoadExpiredServices(b *testing.B) {
+	for _, count := range []int{100, 1000} {
+		b.Run(fmt.Sprint(count), func(b *testing.B) {
+			ctx, sys := prepareIAMPerformanceFixture(b)
+			b.ReportAllocs()
+			for i := 0; i < b.N; i++ {
+				b.StopTimer()
+				for j := 0; j < count; j++ {
+					key := fmt.Sprintf("expired-benchmark-%d-%d", i, j)
+					u := UserIdentity{Version: 1, UpdatedAt: UTCNow().Add(-2 * time.Hour), Credentials: auth.Credentials{AccessKey: key, SecretKey: "expired-benchmark-password", ParentUser: "absent-idp-parent", Expiration: UTCNow().Add(-time.Hour), Status: auth.AccountOn}}
+					if err := sys.store.saveIAMConfig(ctx, &u, getUserIdentityPath(key, svcUser)); err != nil {
+						b.Fatal(err)
+					}
+				}
+				b.StartTimer()
+				if err := sys.store.LoadIAMCache(ctx, true); err != nil {
+					b.Fatal(err)
+				}
+			}
+		})
+	}
+}

cmd/iam-replay_test.go

@@ -0,0 +1,168 @@
+package cmd
+
+import (
+	"context"
+	"errors"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/pgsty/silo-pkg/v3/policy"
+)
+
+func TestReviewIAMRevokedUserReplay(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	obj, disk, err := prepareFS(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	user := "review-revoked-user"
+	req := madmin.AddOrUpdateUserReq{SecretKey: "review-valid-password", Status: madmin.AccountEnabled}
+	created, err := globalIAMSys.CreateUser(ctx, user, req)
+	if err != nil {
+		t.Fatal(err)
+	}
+	policyAt, err := globalIAMSys.PolicyDBSet(ctx, user, "readwrite", regUser, false)
+	if err != nil {
+		t.Fatal(err)
+	}
+	args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "review-bucket", ObjectName: "review-object"}
+	if !globalIAMSys.IsAllowed(args) {
+		t.Fatal("seed must allow object read")
+	}
+	if err := globalIAMSys.DeleteUser(ctx, user, false); err != nil {
+		t.Fatal(err)
+	}
+	if err := globalIAMSys.store.LoadIAMCache(ctx, false); err != nil {
+		t.Fatal(err)
+	}
+	if globalIAMSys.IsAllowed(args) {
+		t.Fatal("deletion did not remove initial permission")
+	}
+	if err := globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, created); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := globalIAMSys.GetUserInfo(ctx, user); !errors.Is(err, errNoSuchUser) {
+		t.Errorf("revoked user restored by an older replicated create, GetUserInfo error = %v", err)
+	}
+	if err := globalSiteReplicationSys.PeerPolicyMappingHandler(ctx, &madmin.SRPolicyMapping{UserOrGroup: user, UserType: int(regUser), Policy: "readwrite"}, policyAt); err != nil {
+		t.Fatal(err)
+	}
+	if globalIAMSys.IsAllowed(args) {
+		t.Error("older replicated identity and policy events restored revoked S3 read permission")
+	}
+}
+
+func TestReviewIAMSourceTimestampOrder(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	obj, disk, err := prepareFS(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	user := "review-ordered-user"
+	req := madmin.AddOrUpdateUserReq{SecretKey: "review-valid-password", Status: madmin.AccountEnabled}
+	origin := UTCNow().Add(-time.Hour)
+	if err := globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin); err != nil {
+		t.Fatal(err)
+	}
+	if err := globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(time.Minute)); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := globalIAMSys.GetUserInfo(ctx, user); !errors.Is(err, errNoSuchUser) {
+		t.Fatalf("newer source deletion skipped after delayed creation, GetUserInfo error = %v", err)
+	}
+}
+
+// A user's old group grant must not return after deletion and deliberate recreation.
+func TestR3CandidateOldGroupReplayAfterRecreation(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	obj, disk, err := prepareFS(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	must := func(err error) {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	user, group := "r3-group-member", "r3-granting-group"
+	origin := UTCNow().Add(-time.Hour)
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-r3-user-password", Status: madmin.AccountEnabled}
+	peer := &globalSiteReplicationSys
+	must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin))
+	add := &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}
+	must(peer.PeerGroupInfoChangeHandler(ctx, add, origin.Add(time.Minute)))
+	must(peer.PeerPolicyMappingHandler(ctx, &madmin.SRPolicyMapping{UserOrGroup: group, IsGroup: true, UserType: int(regUser), Policy: "readwrite"}, origin.Add(time.Minute)))
+	args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "r3-bucket", ObjectName: "probe"}
+	if !globalIAMSys.IsAllowed(args) {
+		t.Fatal("fixture must grant through group")
+	}
+	must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(2*time.Minute)))
+	must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin.Add(3*time.Minute)))
+	must(globalIAMSys.store.LoadIAMCache(ctx, false))
+	if globalIAMSys.IsAllowed(args) {
+		t.Fatal("recreation must start without deleted group membership")
+	}
+	must(peer.PeerGroupInfoChangeHandler(ctx, add, origin.Add(time.Minute)))
+	must(globalIAMSys.store.LoadIAMCache(ctx, false))
+	if globalIAMSys.IsAllowed(args) {
+		t.Fatal("old group event restored the deleted user's read grant after recreation and durable reload")
+	}
+}
+
+// The user delete is also a revocation of its earlier group memberships.
+func TestR3CandidateLateDeleteRetainsOldGroupGrant(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	obj, disk, err := prepareFS(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	must := func(err error) {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	user, group := "r3-late-member", "r3-late-group"
+	origin := UTCNow().Add(-time.Hour)
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-r3-user-password", Status: madmin.AccountEnabled}
+	peer := &globalSiteReplicationSys
+	must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin))
+	must(peer.PeerGroupInfoChangeHandler(ctx, &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}, origin.Add(time.Minute)))
+	must(peer.PeerPolicyMappingHandler(ctx, &madmin.SRPolicyMapping{UserOrGroup: group, IsGroup: true, UserType: int(regUser), Policy: "readwrite"}, origin.Add(time.Minute)))
+	args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "r3-bucket", ObjectName: "probe"}
+	if !globalIAMSys.IsAllowed(args) {
+		t.Fatal("fixture must grant through group")
+	}
+	must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin.Add(3*time.Minute)))
+	must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(2*time.Minute)))
+	must(globalIAMSys.store.LoadIAMCache(ctx, false))
+	if _, ok := globalIAMSys.GetUser(ctx, user); !ok {
+		t.Fatal("newer identity must survive")
+	}
+	if globalIAMSys.IsAllowed(args) {
+		t.Fatal("late user deletion retained the older group grant on the recreated identity")
+	}
+}

cmd/iam-replication-protocol.go

@@ -0,0 +1,321 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"sort"
+	"strings"
+	"sync/atomic"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	xhttp "github.com/minio/minio/internal/http"
+	"github.com/pgsty/silo-pkg/v3/policy"
+)
+
+const (
+	iamRevisionProtocol  = 1
+	iamRevisionPeerPath  = "/v3/site-replication/peer/iam-revisions"
+	iamUserBoundaryType  = "silo-user-revocation"
+	iamGroupBoundaryType = "silo-group-revocation"
+	maxIAMRevisionBatch  = 128
+)
+
+var iamRevisionInstance = mustGetUUID()
+
+type iamUserBoundary struct {
+	User   string    `json:"user"`
+	Before time.Time `json:"before"`
+}
+
+type iamGroupBoundary struct {
+	Group  string    `json:"group"`
+	Before time.Time `json:"before"`
+}
+
+// The server owns this additive protocol, without changing the client SDK or
+// overloading a policy/document field. Old servers reject the dedicated route
+// before applying any change that would lose revocation or member metadata.
+type iamReplicationItem struct {
+	madmin.SRIAMItem
+	GroupGrants     map[string]time.Time `json:"groupGrants,omitempty"`
+	GroupSnapshot   bool                 `json:"groupSnapshot,omitempty"`
+	UserRevocation  *iamUserBoundary     `json:"userRevocation,omitempty"`
+	GroupRevocation *iamGroupBoundary    `json:"groupRevocation,omitempty"`
+	RevokedBefore   time.Time            `json:"revokedBefore,omitempty"`
+}
+
+type iamRevisionBatch struct {
+	Version int                  `json:"version"`
+	Items   []iamReplicationItem `json:"items"`
+}
+
+type iamRevisionStatus struct {
+	Version  int    `json:"version"`
+	Node     string `json:"node"`
+	Instance string `json:"instance"`
+	Digest   string `json:"digest"`
+}
+
+type iamRevisionResponse struct {
+	iamRevisionStatus
+	Errors []string `json:"errors,omitempty"`
+}
+
+type iamRevisionBatchError struct{ failures []string }
+
+func (e *iamRevisionBatchError) Error() string {
+	return "IAM revision batch: " + strings.Join(e.failures, "; ")
+}
+
+type iamRevisionProgress struct {
+	Instances    map[string]string
+	Acknowledged map[string]string
+}
+
+type iamRevisionMetrics struct {
+	healFailures       atomic.Uint64
+	healLastSuccess    atomic.Int64
+	healDurationMillis atomic.Int64
+}
+
+func iamRevisionDigest(items map[string]iamRevision) string {
+	paths := make([]string, 0, len(items))
+	for path := range items {
+		paths = append(paths, path)
+	}
+	sort.Strings(paths)
+	h := sha256.New()
+	for _, path := range paths {
+		r := items[path]
+		fmt.Fprintf(h, "%q %s %t %s\n", path, r.timestamp().UTC().Format(time.RFC3339Nano), r.Deleted, r.RevokedBefore.UTC().Format(time.RFC3339Nano))
+	}
+	return hex.EncodeToString(h.Sum(nil))
+}
+
+func (store *IAMStoreSys) iamRevisionStatus() iamRevisionStatus {
+	node := globalLocalNodeName
+	if node == "" {
+		node = "local"
+	}
+	return iamRevisionStatus{Version: iamRevisionProtocol, Node: node, Instance: iamRevisionInstance, Digest: store.revisionIndex().digest()}
+}
+
+func executeIAMRevisionRequest(ctx context.Context, client *madmin.AdminClient, method string, batch *iamRevisionBatch) (status iamRevisionStatus, err error) {
+	var content []byte
+	if batch != nil {
+		content, err = json.Marshal(batch)
+		if err != nil {
+			return status, err
+		}
+	}
+	resp, err := client.ExecuteMethod(ctx, method, madmin.RequestData{RelPath: iamRevisionPeerPath, QueryValues: url.Values{"api-version": {madmin.SiteReplAPIVersion}}, Content: content})
+	if resp != nil {
+		defer xhttp.DrainBody(resp.Body)
+	}
+	if err != nil {
+		return status, err
+	}
+	if resp.StatusCode != http.StatusOK {
+		var remote madmin.ErrorResponse
+		if json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&remote) == nil && remote.Code != "" {
+			return status, remote
+		}
+		return status, fmt.Errorf("IAM revision protocol requires upgraded peers: %s", resp.Status)
+	}
+	var response iamRevisionResponse
+	if err = json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&response); err != nil {
+		return status, err
+	}
+	status = response.iamRevisionStatus
+	if status.Version != iamRevisionProtocol || status.Node == "" || status.Instance == "" || status.Digest == "" {
+		return status, errors.New("peer did not acknowledge the IAM revision protocol")
+	}
+	if len(response.Errors) != 0 {
+		return status, &iamRevisionBatchError{failures: response.Errors}
+	}
+	return status, nil
+}
+
+type (
+	iamRecordBoundaryKey struct{}
+	iamGroupSnapshotKey  struct{}
+)
+
+func (c *SiteReplicationSys) replicationItem(ctx context.Context, item madmin.SRIAMItem) (iamReplicationItem, error) {
+	out := iamReplicationItem{SRIAMItem: item}
+	if item.Type == madmin.SRIAMItemSvcAcc && item.SvcAccChange != nil {
+		var key string
+		if item.SvcAccChange.Create != nil {
+			key = item.SvcAccChange.Create.AccessKey
+		} else if item.SvcAccChange.Update != nil {
+			key = item.SvcAccChange.Update.AccessKey
+		}
+		if key != "" {
+			r, err := loadIAMRevision(ctx, globalIAMSys.store, getUserIdentityPath(key, svcUser))
+			if err != nil {
+				return out, err
+			}
+			if r.Deleted || r.timestamp().After(item.UpdatedAt) {
+				return out, errIAMStaleUpdate
+			}
+			out.RevokedBefore = r.RevokedBefore
+		}
+	}
+	if item.Type == madmin.SRIAMItemGroupInfo && item.GroupInfo != nil && !item.GroupInfo.UpdateReq.IsRemove {
+		out.GroupSnapshot = true
+		var gi GroupInfo
+		if err := globalIAMSys.store.loadIAMConfig(ctx, &gi, getGroupInfoPath(item.GroupInfo.UpdateReq.Group)); err != nil {
+			return out, err
+		}
+		// The matching persisted snapshot carries member grant times. If a
+		// later write won before sending, propagate that whole newer state.
+		if gi.Deleted {
+			return out, errIAMStaleUpdate
+		}
+		out.UpdatedAt = gi.UpdatedAt
+		out.RevokedBefore = gi.RevokedBefore
+		out.GroupInfo = &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: item.GroupInfo.UpdateReq.Group, Status: madmin.GroupStatus(gi.Status)}}
+		cache := globalIAMSys.store.rlock()
+		out.GroupInfo.UpdateReq.Members = cache.effectiveGroupMembers(gi)
+		globalIAMSys.store.runlock()
+		out.GroupGrants = make(map[string]time.Time, len(out.GroupInfo.UpdateReq.Members))
+		for _, member := range out.GroupInfo.UpdateReq.Members {
+			out.GroupGrants[member] = gi.MemberGrants[member]
+		}
+	}
+	if item.Type == madmin.SRIAMItemGroupInfo && item.GroupInfo != nil && item.GroupInfo.UpdateReq.IsRemove && len(item.GroupInfo.UpdateReq.Members) == 0 {
+		r, err := loadIAMRevision(ctx, globalIAMSys.store, getGroupInfoPath(item.GroupInfo.UpdateReq.Group))
+		if err != nil {
+			return out, err
+		}
+		if !r.Deleted && !r.RevokedBefore.IsZero() {
+			out.Type, out.GroupInfo = iamGroupBoundaryType, nil
+			out.GroupRevocation = &iamGroupBoundary{Group: item.GroupInfo.UpdateReq.Group, Before: r.RevokedBefore}
+			out.UpdatedAt = r.RevokedBefore
+		}
+	}
+	if item.Type == madmin.SRIAMItemIAMUser && item.IAMUser != nil {
+		r, err := loadIAMRevision(ctx, globalIAMSys.store, getUserIdentityPath(item.IAMUser.AccessKey, regUser))
+		if err != nil {
+			return out, err
+		}
+		if item.IAMUser.IsDeleteReq && !r.Deleted && !r.RevokedBefore.IsZero() {
+			out.Type = iamUserBoundaryType
+			out.IAMUser = nil
+			out.UserRevocation = &iamUserBoundary{User: item.IAMUser.AccessKey, Before: r.RevokedBefore}
+			out.UpdatedAt = r.RevokedBefore
+		} else if !item.IAMUser.IsDeleteReq {
+			if r.Deleted || r.timestamp().After(item.UpdatedAt) {
+				return out, errIAMStaleUpdate
+			}
+			out.RevokedBefore = r.RevokedBefore
+		}
+	}
+	return out, nil
+}
+
+func applyIAMReplicationItem(ctx context.Context, item iamReplicationItem) error {
+	if item.GroupInfo != nil {
+		if item.GroupSnapshot {
+			ctx = context.WithValue(ctx, iamGroupSnapshotKey{}, true)
+		}
+		// A nil map also explicitly denotes unknown legacy grants. Do not
+		// turn an unrelated group edit into a new grant after a revocation.
+		ctx = withIAMGroupGrants(ctx, item.GroupGrants)
+	}
+	if !item.RevokedBefore.IsZero() {
+		if item.RevokedBefore.After(item.UpdatedAt) {
+			return errSRInvalidRequest(errInvalidArgument)
+		}
+		ctx = context.WithValue(ctx, iamRecordBoundaryKey{}, item.RevokedBefore)
+	}
+	switch item.Type {
+	case iamUserBoundaryType:
+		if item.UserRevocation == nil || item.UserRevocation.User == "" || item.UserRevocation.Before.IsZero() {
+			return errSRInvalidRequest(errInvalidArgument)
+		}
+		return iamReplicationError(globalIAMSys.DeleteUser(withIAMReplicationTime(ctx, item.UserRevocation.Before), item.UserRevocation.User, true))
+	case iamGroupBoundaryType:
+		if item.GroupRevocation == nil || item.GroupRevocation.Group == "" || item.GroupRevocation.Before.IsZero() {
+			return errSRInvalidRequest(errInvalidArgument)
+		}
+		_, err := globalIAMSys.RemoveUsersFromGroup(withIAMReplicationTime(ctx, item.GroupRevocation.Before), item.GroupRevocation.Group, nil)
+		return iamReplicationError(err)
+	case madmin.SRIAMItemPolicy:
+		if len(item.Policy) == 0 {
+			return globalSiteReplicationSys.PeerAddPolicyHandler(ctx, item.Name, nil, item.UpdatedAt)
+		}
+		p, err := policy.ParseConfig(bytes.NewReader(item.Policy))
+		if err != nil {
+			return err
+		}
+		if p.IsEmpty() {
+			p = nil
+		}
+		return globalSiteReplicationSys.PeerAddPolicyHandler(ctx, item.Name, p, item.UpdatedAt)
+	case madmin.SRIAMItemSvcAcc:
+		return globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, item.SvcAccChange, item.UpdatedAt)
+	case madmin.SRIAMItemPolicyMapping:
+		return globalSiteReplicationSys.PeerPolicyMappingHandler(ctx, item.PolicyMapping, item.UpdatedAt)
+	case madmin.SRIAMItemSTSAcc:
+		return globalSiteReplicationSys.PeerSTSAccHandler(ctx, item.STSCredential, item.UpdatedAt)
+	case madmin.SRIAMItemIAMUser:
+		return globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, item.IAMUser, item.UpdatedAt)
+	case madmin.SRIAMItemGroupInfo:
+		return globalSiteReplicationSys.PeerGroupInfoChangeHandler(ctx, item.GroupInfo, item.UpdatedAt)
+	default:
+		return errSRInvalidRequest(errInvalidArgument)
+	}
+}
+
+func (a adminAPIHandlers) SRPeerIAMRevisions(w http.ResponseWriter, r *http.Request) {
+	ctx := r.Context()
+	if obj, _ := validateAdminReq(ctx, w, r, policy.SiteReplicationOperationAction); obj == nil {
+		return
+	}
+	var failures []string
+	if r.Method == http.MethodPut {
+		var batch iamRevisionBatch
+		if err := parseJSONBody(ctx, r.Body, &batch, ""); err != nil {
+			writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
+			return
+		}
+		if batch.Version != iamRevisionProtocol || len(batch.Items) == 0 || len(batch.Items) > maxIAMRevisionBatch {
+			writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, errSRInvalidRequest(errInvalidArgument)), r.URL)
+			return
+		}
+		for i, item := range batch.Items {
+			if err := applyIAMReplicationItem(ctx, item); err != nil {
+				failures = append(failures, fmt.Sprintf("item %d (%s): %v", i, item.Type, err))
+			}
+		}
+	}
+	w.Header().Set("Content-Type", "application/json")
+	_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: globalIAMSys.store.iamRevisionStatus(), Errors: failures})
+}
+
+// A site endpoint can balance requests across nodes sharing durable IAM state.
+// Switching between known node incarnations preserves ACKs; a new incarnation
+// conservatively invalidates them so restoring an old backend cannot inherit
+// acknowledgements from before the restore.
+func (p *iamRevisionProgress) observePeer(status iamRevisionStatus) {
+	if p.Instances == nil {
+		p.Instances = make(map[string]string)
+	}
+	if p.Instances[status.Node] != status.Instance || p.Acknowledged == nil {
+		p.Instances[status.Node] = status.Instance
+		p.Acknowledged = make(map[string]string)
+	}
+}

cmd/iam-replication-protocol_test.go

@@ -0,0 +1,161 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+)
+
+func TestIAMRevisionProtocolDoesNotFallBackToLegacy(t *testing.T) {
+	var requests atomic.Int32
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		requests.Add(1)
+		if r.URL.Path != "/minio/admin/v3/site-replication/peer/iam-revisions" {
+			t.Errorf("unsafe fallback path: %s", r.URL.Path)
+		}
+		w.WriteHeader(http.StatusNotFound)
+		_, _ = w.Write([]byte(`{"Code":"NotImplemented","Message":"old server"}`))
+	}))
+	defer server.Close()
+	client, err := madmin.New(strings.TrimPrefix(server.URL, "http://"), "test-access", "valid-test-secret", false)
+	mustIAM(t, err)
+	_, err = executeIAMRevisionRequest(context.Background(), client, http.MethodPut, &iamRevisionBatch{Version: iamRevisionProtocol, Items: []iamReplicationItem{{SRIAMItem: madmin.SRIAMItem{Type: iamUserBoundaryType}, UserRevocation: &iamUserBoundary{User: "recreated", Before: UTCNow()}}}})
+	if err == nil || requests.Load() != 1 {
+		t.Fatalf("old peer must reject without fallback, err=%v requests=%d", err, requests.Load())
+	}
+}
+
+type iamNoHealingScanStore struct{ IAMStorageAPI }
+
+func (s *iamNoHealingScanStore) listIAMConfigPaths(context.Context) ([]string, error) {
+	panic("healing must use the loaded revision index")
+}
+
+func TestIAMRevisionHealingAcknowledgements(t *testing.T) {
+	for _, balanced := range []bool{false, true} {
+		t.Run(fmt.Sprintf("load_balanced_%t", balanced), func(t *testing.T) { testIAMRevisionHealingAcknowledgements(t, balanced) })
+	}
+}
+
+func testIAMRevisionHealingAcknowledgements(t *testing.T, balanced bool) {
+	ctx, sys, _ := prepareIAMRevisionFixture(t)
+	_, err := sys.CreateUser(ctx, "ack-sync", madmin.AddOrUpdateUserReq{SecretKey: "valid-sync-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	for i := range maxIAMRevisionBatch*2 + 1 {
+		at := UTCNow().Add(time.Duration(i) * time.Nanosecond)
+		mustIAM(t, sys.store.saveIAMConfig(ctx, &UserIdentity{Version: 1, Deleted: true, UpdatedAt: at, RevokedBefore: at}, getUserIdentityPath(fmt.Sprintf("ack-%04d", i), regUser)))
+	}
+	sys.store.IAMStorageAPI = &iamNoHealingScanStore{IAMStorageAPI: sys.store.IAMStorageAPI}
+	var mu sync.Mutex
+	var puts, gets int
+	var applied int
+	instance := "boot-1"
+	failSecondBatch := true
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/minio/health/live" {
+			w.WriteHeader(http.StatusOK)
+			return
+		}
+		mu.Lock()
+		defer mu.Unlock()
+		if r.URL.Path != "/minio/admin/v3/site-replication/peer/iam-revisions" {
+			t.Errorf("unexpected request: %s", r.URL.Path)
+			w.WriteHeader(404)
+			return
+		}
+		var failures []string
+		if r.Method == http.MethodGet {
+			gets++
+		} else {
+			puts++
+			var batch iamRevisionBatch
+			if err := json.NewDecoder(r.Body).Decode(&batch); err != nil {
+				t.Error(err)
+				w.WriteHeader(400)
+				return
+			}
+			if len(batch.Items) > maxIAMRevisionBatch {
+				t.Error("batch exceeds limit")
+			}
+			if failSecondBatch && puts == 2 {
+				failures = []string{"injected item error"}
+			} else {
+				applied += len(batch.Items)
+			}
+		}
+		node := "node-1"
+		if balanced {
+			node = fmt.Sprintf("node-%d", (gets+puts)%2+1)
+		}
+		_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: node, Instance: node + instance, Digest: fmt.Sprintf("%d", applied)}, Errors: failures})
+	}))
+	defer server.Close()
+	c := &SiteReplicationSys{enabled: true, state: srState{ServiceAccountAccessKey: "ack-sync", Peers: map[string]madmin.PeerInfo{globalDeploymentID(): {DeploymentID: globalDeploymentID(), Name: "local"}, "remote": {DeploymentID: "remote", Name: "remote", Endpoint: server.URL}}}}
+	if err := c.healIAMDeletions(ctx); err == nil {
+		t.Fatal("item failure was hidden")
+	}
+	mu.Lock()
+	if puts != 3 || applied != maxIAMRevisionBatch+1 {
+		t.Errorf("failed middle batch blocked later revocations: puts=%d applied=%d", puts, applied)
+	}
+	failSecondBatch = false
+	applied++ // Unrelated remote mutation changes its digest.
+	mu.Unlock()
+	at := UTCNow()
+	mustIAM(t, sys.store.saveIAMConfig(ctx, &UserIdentity{Version: 1, Deleted: true, UpdatedAt: at, RevokedBefore: at}, getUserIdentityPath("ack-new-local", regUser)))
+	mustIAM(t, c.healIAMDeletions(ctx))
+	mu.Lock()
+	if puts != 5 {
+		t.Errorf("did not resume at unacknowledged batch: puts=%d", puts)
+	}
+	mu.Unlock()
+	mustIAM(t, c.healIAMDeletions(ctx))
+	mu.Lock()
+	if puts != 5 || gets != 3 {
+		t.Errorf("converged records were replayed: puts=%d gets=%d", puts, gets)
+	}
+	instance = "boot-2"
+	mu.Unlock()
+	mustIAM(t, c.healIAMDeletions(ctx))
+	mu.Lock()
+	defer mu.Unlock()
+	if puts != 8 {
+		t.Fatalf("peer restart reused an old acknowledgement: puts=%d", puts)
+	}
+}
+
+func TestIAMRevisionIndexRebuildsFromStorage(t *testing.T) {
+	ctx, sys, obj := prepareIAMRevisionFixture(t)
+	const user = "index-parent"
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-parent-password", Status: madmin.AccountEnabled}
+	_, err := sys.CreateUser(ctx, user, req)
+	mustIAM(t, err)
+	mustIAM(t, sys.DeleteUser(ctx, user, false))
+	before := sys.store.revisionIndex().snapshot()
+	store := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+	mustIAM(t, store.LoadIAMCache(ctx, true))
+	if iamRevisionDigest(before) != iamRevisionDigest(store.revisionIndex().snapshot()) {
+		t.Fatal("ordinary IAM loading did not restore the deletion index")
+	}
+	_, err = store.AddUser(ctx, user, req)
+	mustIAM(t, err)
+	r := store.revisionIndex().get(getUserIdentityPath(user, regUser))
+	if r.Deleted || r.RevokedBefore.IsZero() {
+		t.Fatal("recreation discarded the retained boundary")
+	}
+	if r.Credentials.SecretKey != "" || r.Credentials.SessionToken != "" {
+		t.Fatal("index retained credentials")
+	}
+}

cmd/iam-revision-boundary_test.go

@@ -0,0 +1,393 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/grid"
+	xnet "github.com/pgsty/silo-pkg/v3/net"
+	"github.com/pgsty/silo-pkg/v3/policy"
+	etcd "go.etcd.io/etcd/client/v3"
+	"go.etcd.io/etcd/client/v3/namespace"
+)
+
+func prepareIAMRevisionFixture(t testing.TB, backend ...string) (context.Context, *IAMSys, ObjectLayer) {
+	t.Helper()
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	disks, err := getRandomDisks(1)
+	mustIAM(t, err)
+	obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
+	mustIAM(t, err)
+	initAllSubsystems(ctx)
+	// Deliberately omit the periodic refresh goroutine. Fault injection can
+	// replace this fixture's storage interface without racing initialization.
+	var client *etcd.Client
+	if len(backend) != 0 && backend[0] == "etcd" {
+		endpoint := os.Getenv("SILO_TEST_IAM_REVOCATION_ETCD")
+		if endpoint == "" {
+			cancel()
+			obj.Shutdown(context.Background())
+			os.RemoveAll(disks[0])
+			t.Skip("set SILO_TEST_IAM_REVOCATION_ETCD to a disposable etcd endpoint")
+		}
+		client, err = etcd.New(etcd.Config{Endpoints: strings.Split(endpoint, ","), DialTimeout: 5 * time.Second})
+		mustIAM(t, err)
+		prefix := fmt.Sprintf("/silo-boundary-test/%d/", time.Now().UnixNano())
+		client.KV = namespace.NewKV(client.KV, prefix)
+		client.Watcher = namespace.NewWatcher(client.Watcher, prefix)
+		t.Cleanup(func() { client.Delete(context.Background(), "", etcd.WithPrefix()); client.Close() })
+	}
+	globalIAMSys.initStore(obj, client)
+	mustIAM(t, globalIAMSys.Load(ctx, true))
+	t.Cleanup(func() { cancel(); obj.Shutdown(context.Background()); os.RemoveAll(disks[0]); resetTestGlobals() })
+	return ctx, globalIAMSys, obj
+}
+
+func mustIAM(t testing.TB, err error) {
+	t.Helper()
+	if err != nil {
+		t.Fatal(err)
+	}
+}
+
+var errIAMInjectedWrite = errors.New("injected IAM persistence failure")
+
+type iamFailingCleanupStore struct {
+	IAMStorageAPI
+	parentPath   string
+	beforeCommit bool
+}
+
+func (s *iamFailingCleanupStore) saveIAMConfig(ctx context.Context, item any, path string, opts ...options) error {
+	if s.beforeCommit || path != s.parentPath {
+		return errIAMInjectedWrite
+	}
+	return s.IAMStorageAPI.saveIAMConfig(ctx, item, path, opts...)
+}
+
+func TestIAMRevocationCommitBoundary(t *testing.T) {
+	for _, before := range []bool{true, false} {
+		name := "after_identity_commit"
+		if before {
+			name = "before_identity_commit"
+		}
+		t.Run(name, func(t *testing.T) {
+			ctx, sys, obj := prepareIAMRevisionFixture(t)
+			const user = "commit-boundary-user"
+			origin := UTCNow().Add(-time.Hour)
+			req := madmin.AddOrUpdateUserReq{SecretKey: "valid-test-password", Status: madmin.AccountEnabled}
+			_, err := sys.CreateUser(withIAMReplicationTime(ctx, origin), user, req)
+			mustIAM(t, err)
+			_, err = sys.PolicyDBSet(withIAMReplicationTime(ctx, origin.Add(time.Minute)), user, "readwrite", regUser, false)
+			mustIAM(t, err)
+			_, err = sys.AddUsersToGroup(withIAMReplicationTime(ctx, origin.Add(time.Minute)), "commit-group", []string{user})
+			mustIAM(t, err)
+			_, err = sys.PolicyDBSet(ctx, "commit-group", "readwrite", regUser, true)
+			mustIAM(t, err)
+			child, _, err := sys.NewServiceAccount(withIAMReplicationTime(ctx, origin), user, nil, newServiceAccountOpts{accessKey: "commit-child", secretKey: "valid-child-password"})
+			mustIAM(t, err)
+			args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}
+			if !sys.IsAllowed(args) {
+				t.Fatal("fixture has no grant")
+			}
+			siblingStore := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+			mustIAM(t, siblingStore.LoadIAMCache(ctx, true))
+			sibling := &IAMSys{store: siblingStore, usersSysType: MinIOUsersSysType}
+			tg, err := grid.SetupTestGrid(2)
+			mustIAM(t, err)
+			defer tg.Cleanup()
+			var notifications atomic.Int32
+			mustIAM(t, deleteUserRPC.Register(tg.Managers[1], func(r *grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
+				notifications.Add(1)
+				if err := sibling.LoadUserAfterDelete(ctx, r.Get(peerRESTUser)); err != nil {
+					return grid.NoPayload{}, grid.NewRemoteErr(err)
+				}
+				return grid.NoPayload{}, nil
+			}))
+			host, err := xnet.ParseHost(strings.TrimPrefix(tg.Hosts[1], "http://"))
+			mustIAM(t, err)
+			globalNotificationSys = &NotificationSys{peerClients: []*peerRESTClient{{host: host, gridConn: func() *grid.Connection { return tg.Managers[0].Connection(tg.Hosts[1]) }}}}
+			original := sys.store.IAMStorageAPI
+			sys.store.IAMStorageAPI = &iamFailingCleanupStore{IAMStorageAPI: original, parentPath: getUserIdentityPath(user, regUser), beforeCommit: before}
+			boundary := origin.Add(2 * time.Minute)
+			err = sys.DeleteUser(withIAMReplicationTime(ctx, boundary), user, true)
+			if !errors.Is(err, errIAMInjectedWrite) {
+				t.Fatalf("expected write failure, got %v", err)
+			}
+			sys.store.IAMStorageAPI = original
+			r, err := loadIAMRevision(ctx, original, getUserIdentityPath(user, regUser))
+			mustIAM(t, err)
+			if before {
+				if r.Deleted || !sys.IsAllowed(args) || !sibling.IsAllowed(args) || notifications.Load() != 0 {
+					t.Fatal("failure before commit changed the identity or grant")
+				}
+				return
+			}
+			if !r.Deleted || !r.RevokedBefore.Equal(boundary) {
+				t.Fatal("cleanup failure lost durable revocation")
+			}
+			if sys.IsAllowed(args) || sibling.IsAllowed(args) || notifications.Load() != 1 {
+				t.Fatal("cleanup failure retained old permission")
+			}
+			// Subsequent fixture writes need no additional RPC handlers.
+			globalNotificationSys = &NotificationSys{}
+			// Recreate after the partial cleanup. The old mapping, group member
+			// and child still exist in storage; none may authorize this identity.
+			_, err = sys.CreateUser(withIAMReplicationTime(ctx, origin.Add(3*time.Minute)), user, req)
+			mustIAM(t, err)
+			reloaded := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+			mustIAM(t, reloaded.LoadIAMCache(ctx, true))
+			fresh := &IAMSys{store: reloaded, usersSysType: MinIOUsersSysType}
+			if fresh.IsAllowed(args) {
+				t.Fatal("cold reload restored partially cleaned-up grants")
+			}
+			if _, ok := reloaded.GetUser(child.AccessKey); ok {
+				t.Fatal("cold reload restored the old child")
+			}
+			gd, err := reloaded.GetGroupDescription("commit-group")
+			mustIAM(t, err)
+			if len(gd.Members) != 0 {
+				t.Fatalf("listing exposed a revoked group relation: %v", gd.Members)
+			}
+			_, err = sys.AddUsersToGroup(ctx, "commit-group", []string{user})
+			mustIAM(t, err)
+			if !sys.IsAllowed(args) {
+				t.Fatal("explicit new group grant was not accepted")
+			}
+		})
+	}
+}
+
+func TestIAMGroupGrantVersionsSurviveSnapshotsAndRecreation(t *testing.T) {
+	ctx, sys, _ := prepareIAMRevisionFixture(t)
+	origin := UTCNow().Add(-time.Hour)
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-test-password", Status: madmin.AccountEnabled}
+	for _, user := range []string{"grant-alice", "grant-bob"} {
+		_, err := sys.CreateUser(withIAMReplicationTime(ctx, origin), user, req)
+		mustIAM(t, err)
+	}
+	grant := origin.Add(time.Minute)
+	_, err := sys.AddUsersToGroup(withIAMReplicationTime(ctx, grant), "grant-group", []string{"grant-alice"})
+	mustIAM(t, err)
+	_, err = sys.PolicyDBSet(ctx, "grant-group", "readwrite", regUser, true)
+	mustIAM(t, err)
+	boundary := origin.Add(2 * time.Minute)
+	mustIAM(t, sys.DeleteUser(withIAMReplicationTime(ctx, boundary), "grant-alice", false))
+	_, err = sys.CreateUser(withIAMReplicationTime(ctx, origin.Add(3*time.Minute)), "grant-alice", req)
+	mustIAM(t, err)
+	_, err = sys.AddUsersToGroup(withIAMReplicationTime(ctx, origin.Add(4*time.Minute)), "grant-group", []string{"grant-bob"})
+	mustIAM(t, err)
+	_, err = sys.SetGroupStatus(withIAMReplicationTime(ctx, origin.Add(5*time.Minute)), "grant-group", true)
+	mustIAM(t, err)
+	var gi GroupInfo
+	mustIAM(t, sys.store.loadIAMConfig(ctx, &gi, getGroupInfoPath("grant-group")))
+	if !gi.MemberGrants["grant-alice"].Equal(grant) {
+		t.Fatal("unrelated group edits refreshed an old grant")
+	}
+	args := policy.Args{AccountName: "grant-alice", Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}
+	for _, stale := range []time.Time{grant, boundary, {}} {
+		item := iamReplicationItem{SRIAMItem: madmin.SRIAMItem{Type: madmin.SRIAMItemGroupInfo, UpdatedAt: origin.Add(6 * time.Minute), GroupInfo: &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: "grant-group", Members: []string{"grant-alice", "grant-bob"}}}}, GroupSnapshot: true, GroupGrants: map[string]time.Time{"grant-alice": stale, "grant-bob": origin.Add(4 * time.Minute)}}
+		mustIAM(t, applyIAMReplicationItem(ctx, item))
+		mustIAM(t, sys.store.LoadIAMCache(ctx, false))
+		if sys.IsAllowed(args) {
+			t.Fatalf("snapshot restored revoked grant %s", stale)
+		}
+		gd, err := sys.GetGroupDescription("grant-group")
+		mustIAM(t, err)
+		if len(gd.Members) != 1 || gd.Members[0] != "grant-bob" {
+			t.Fatalf("inconsistent effective members: %v", gd.Members)
+		}
+	}
+	// Only an explicit post-revocation grant restores access.
+	freshAt, err := sys.AddUsersToGroup(ctx, "grant-group", []string{"grant-alice"})
+	mustIAM(t, err)
+	if !sys.IsAllowed(args) {
+		t.Fatal("explicit regrant rejected")
+	}
+	mustIAM(t, sys.store.LoadIAMCache(ctx, false))
+	mustIAM(t, sys.store.loadIAMConfig(ctx, &gi, getGroupInfoPath("grant-group")))
+	if !gi.MemberGrants["grant-alice"].Equal(freshAt) {
+		t.Fatal("new grant version was not persisted")
+	}
+	if !gi.MemberGrants["grant-bob"].Equal(origin.Add(4 * time.Minute)) {
+		t.Fatal("regranting Alice changed Bob's grant")
+	}
+}
+
+func TestIAMGroupRevocationCommitAndRecreation(t *testing.T) {
+	for _, backend := range []string{"object", "etcd"} {
+		t.Run(backend, func(t *testing.T) { testIAMGroupRevocationCommitAndRecreation(t, backend) })
+	}
+}
+
+func testIAMGroupRevocationCommitAndRecreation(t *testing.T, backend string) {
+	ctx, sys, obj := prepareIAMRevisionFixture(t, backend)
+	origin := UTCNow().Add(-time.Hour)
+	user, group := "group-boundary-user", "group-boundary"
+	_, err := sys.CreateUser(withIAMReplicationTime(ctx, origin), user, madmin.AddOrUpdateUserReq{SecretKey: "valid-user-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	grant, boundary := origin.Add(time.Minute), origin.Add(2*time.Minute)
+	_, err = sys.AddUsersToGroup(withIAMReplicationTime(ctx, grant), group, []string{user})
+	mustIAM(t, err)
+	// A newer mapping must not veto the authoritative group deletion.
+	_, err = sys.PolicyDBSet(withIAMReplicationTime(ctx, origin.Add(3*time.Minute)), group, "readwrite", regUser, true)
+	mustIAM(t, err)
+	_, err = sys.RemoveUsersFromGroup(withIAMReplicationTime(ctx, boundary), group, nil)
+	mustIAM(t, err)
+	r, err := loadIAMRevision(ctx, sys.store, getGroupInfoPath(group))
+	mustIAM(t, err)
+	if !r.Deleted || !r.RevokedBefore.Equal(boundary) {
+		t.Fatal("newer mapping swallowed group deletion")
+	}
+	_, err = sys.AddUsersToGroup(withIAMReplicationTime(ctx, origin.Add(4*time.Minute)), group, nil)
+	mustIAM(t, err)
+	for _, at := range []time.Time{grant, boundary, {}} {
+		item := iamReplicationItem{SRIAMItem: madmin.SRIAMItem{Type: madmin.SRIAMItemGroupInfo, UpdatedAt: origin.Add(5 * time.Minute), GroupInfo: &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}}, GroupSnapshot: true, GroupGrants: map[string]time.Time{user: at}}
+		mustIAM(t, applyIAMReplicationItem(ctx, item))
+		gd, err := sys.GetGroupDescription(group)
+		mustIAM(t, err)
+		if len(gd.Members) != 0 {
+			t.Fatalf("group recreation restored grant %s", at)
+		}
+	}
+	_, err = sys.AddUsersToGroup(ctx, group, []string{user})
+	mustIAM(t, err)
+	args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}
+	if !sys.IsAllowed(args) {
+		t.Fatal("explicit group regrant was rejected")
+	}
+	// The newer live snapshot may arrive before an older group deletion.
+	lateBoundary := origin.Add(6 * time.Minute)
+	_, err = sys.RemoveUsersFromGroup(withIAMReplicationTime(ctx, lateBoundary), group, nil)
+	mustIAM(t, err)
+	r, err = loadIAMRevision(ctx, sys.store, getGroupInfoPath(group))
+	mustIAM(t, err)
+	if r.Deleted || !r.RevokedBefore.Equal(lateBoundary) {
+		t.Fatal("late deletion lost the live group's revocation boundary")
+	}
+	// The old mapping is now revoked; a new explicit mapping restores access.
+	if sys.IsAllowed(args) {
+		t.Fatal("late group boundary retained an old mapping")
+	}
+	_, err = sys.PolicyDBSet(ctx, group, "readwrite", regUser, true)
+	mustIAM(t, err)
+	store := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+	if es, ok := sys.store.IAMStorageAPI.(*IAMEtcdStore); ok {
+		store.IAMStorageAPI = newIAMEtcdStore(es.client, MinIOUsersSysType)
+	}
+	mustIAM(t, store.LoadIAMCache(ctx, true))
+	fresh := &IAMSys{store: store, usersSysType: MinIOUsersSysType}
+	if !fresh.IsAllowed(args) {
+		t.Fatal("reload lost explicit grants after a retained group boundary")
+	}
+	item, err := globalSiteReplicationSys.replicationItem(ctx, madmin.SRIAMItem{Type: madmin.SRIAMItemGroupInfo, GroupInfo: &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group}}, UpdatedAt: r.timestamp()})
+	mustIAM(t, err)
+	if !item.RevokedBefore.Equal(lateBoundary) || !item.GroupGrants[user].After(lateBoundary) {
+		t.Fatal("group snapshot lost revision metadata")
+	}
+}
+
+// A committed revision is observable before all cached dependents have been
+// cleaned up. Every authorization read must apply that boundary in this window.
+func TestIAMCachedMappingHonorsCommittedRevision(t *testing.T) {
+	ctx, sys, _ := prepareIAMRevisionFixture(t)
+	origin := UTCNow().Add(-time.Hour)
+	parent := "cached-external-parent"
+	_, err := sys.PolicyDBSet(withIAMReplicationTime(ctx, origin), parent, "readwrite", stsUser, false)
+	mustIAM(t, err)
+	policies, err := sys.PolicyDBGet(parent)
+	mustIAM(t, err)
+	if len(policies) == 0 {
+		t.Fatal("fixture has no STS-parent mapping")
+	}
+	mustIAM(t, sys.store.saveIAMConfig(ctx, &MappedPolicy{Version: 1, Deleted: true, UpdatedAt: origin.Add(time.Minute)}, getMappedPolicyPath(parent, stsUser, false)))
+	policies, err = sys.PolicyDBGet(parent)
+	mustIAM(t, err)
+	if len(policies) != 0 {
+		t.Fatal("cached STS mapping ignored its own namespace tombstone")
+	}
+
+	user, group := "cached-group-user", "cached-group"
+	_, err = sys.CreateUser(withIAMReplicationTime(ctx, origin), user, madmin.AddOrUpdateUserReq{SecretKey: "valid-user-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	grant := origin.Add(5 * time.Minute)
+	_, err = sys.AddUsersToGroup(withIAMReplicationTime(ctx, grant), group, []string{user})
+	mustIAM(t, err)
+	_, err = sys.PolicyDBSet(withIAMReplicationTime(ctx, origin), group, "readwrite", regUser, true)
+	mustIAM(t, err)
+	args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}
+	if !sys.IsAllowed(args) {
+		t.Fatal("fixture has no group grant")
+	}
+	// A late deletion preserves the newer member grant but revokes the older
+	// policy mapping. Simulate the interval before mapping cleanup completes.
+	gi := GroupInfo{Version: 1, Status: statusEnabled, Members: []string{user}, MemberGrants: map[string]time.Time{user: grant}, UpdatedAt: grant, RevokedBefore: origin.Add(2 * time.Minute)}
+	mustIAM(t, sys.store.saveIAMConfig(ctx, &gi, getGroupInfoPath(group)))
+	if sys.IsAllowed(args) {
+		t.Fatal("cached group mapping ignored the committed group boundary")
+	}
+	gd, err := sys.GetGroupDescription(group)
+	mustIAM(t, err)
+	if gd.Policy != "" {
+		t.Fatal("group listing exposed a revoked mapping")
+	}
+}
+
+type (
+	iamExpiryLockFailure struct {
+		ObjectLayer
+		path string
+	}
+	iamFailedExpiryLock struct{ RWLocker }
+)
+
+func (o *iamExpiryLockFailure) NewNSLock(bucket string, objects ...string) RWLocker {
+	lock := o.ObjectLayer.NewNSLock(bucket, objects...)
+	if bucket == minioMetaBucket && len(objects) == 1 && objects[0] == o.path+".revision-lock" {
+		return &iamFailedExpiryLock{RWLocker: lock}
+	}
+	return lock
+}
+
+func (l *iamFailedExpiryLock) GetLock(context.Context, *dynamicTimeout) (LockContext, error) {
+	return LockContext{}, errIAMInjectedWrite
+}
+
+func TestIAMExpiredCredentialCleanupDoesNotBlockLoading(t *testing.T) {
+	ctx, sys, obj := prepareIAMRevisionFixture(t)
+	_, err := sys.CreateUser(ctx, "healthy-user", madmin.AddOrUpdateUserReq{SecretKey: "healthy-user-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	_, err = sys.PolicyDBSet(ctx, "healthy-user", "readwrite", regUser, false)
+	mustIAM(t, err)
+	c, _, err := sys.NewServiceAccount(ctx, "healthy-user", nil, newServiceAccountOpts{accessKey: "expired-service", secretKey: "expired-service-password"})
+	mustIAM(t, err)
+	c.Expiration = UTCNow().Add(-time.Hour)
+	path := getUserIdentityPath(c.AccessKey, svcUser)
+	mustIAM(t, sys.store.saveIAMConfig(ctx, &UserIdentity{Version: 1, Credentials: c, UpdatedAt: UTCNow()}, path))
+	// A cold loader sees the existing version but cannot acquire the cleanup
+	// write lock. Healthy users must still load; the expired one stays denied.
+	fresh := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(&iamExpiryLockFailure{ObjectLayer: obj, path: path}, MinIOUsersSysType)}
+	mustIAM(t, fresh.LoadIAMCache(ctx, true))
+	if _, ok := fresh.GetUser("healthy-user"); !ok {
+		t.Fatal("cleanup failure prevented healthy IAM state from loading")
+	}
+	if _, ok := fresh.GetUser(c.AccessKey); ok {
+		t.Fatal("cleanup failure admitted an expired service account")
+	}
+	r, err := loadIAMRevision(ctx, fresh, path)
+	mustIAM(t, err)
+	if r.Deleted || !r.Credentials.IsExpired() {
+		t.Fatal("failed cleanup lost the existing expired revision")
+	}
+}

cmd/iam-revision-index.go

@@ -0,0 +1,220 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"maps"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/minio/minio/internal/auth"
+)
+
+// This index is rebuilt by the existing IAM loaders and updated by successful
+// storage operations. It avoids a second full IAM walk during every heal pass.
+// It is an optimization of the durable records, never a reason to delete them.
+// The index contains no secrets or grants.
+type iamParentRevision struct {
+	deleted bool
+	before  time.Time
+}
+
+type iamRevisionIndex struct {
+	mu         sync.RWMutex
+	items      map[string]iamRevision
+	parents    map[string]iamParentRevision
+	floors     map[string]time.Time
+	generation uint64
+}
+
+func (idx *iamRevisionIndex) observe(path string, data []byte) {
+	if !strings.HasPrefix(path, iamConfigPrefix+"/") {
+		return
+	}
+	var r iamRevision
+	if json.Unmarshal(data, &r) != nil {
+		return // The caller reports malformed data using its normal decoder.
+	}
+	r.Credentials = auth.Credentials{ParentUser: r.Credentials.ParentUser, Expiration: r.Credentials.Expiration}
+	idx.mu.Lock()
+	defer idx.mu.Unlock()
+	if strings.HasPrefix(path, iamConfigUsersPrefix) {
+		// Keep a compact name-keyed view for the authentication hot path;
+		// constructing a config path on every S3 request allocates needlessly.
+		defer func() {
+			name := strings.TrimSuffix(strings.TrimPrefix(path, iamConfigUsersPrefix), "/"+iamIdentityFile)
+			if current, ok := idx.items[path]; ok {
+				if idx.parents == nil {
+					idx.parents = make(map[string]iamParentRevision)
+				}
+				idx.parents[name] = iamParentRevision{deleted: current.Deleted, before: current.RevokedBefore}
+			} else {
+				delete(idx.parents, name)
+			}
+		}()
+	}
+	if floor, ok := idx.floors[path]; ok && r.timestamp().Before(floor) {
+		return
+	}
+	if previous, ok := idx.items[path]; ok {
+		// A concurrent read that began before a write must not roll it back.
+		if previous.timestamp().After(r.timestamp()) || (previous.Deleted && !r.Deleted && !r.timestamp().After(previous.timestamp())) {
+			return
+		}
+		if previous.RevokedBefore.After(r.RevokedBefore) {
+			r.RevokedBefore = previous.RevokedBefore
+		}
+		if previous.timestamp().Equal(r.timestamp()) && previous.Deleted == r.Deleted && previous.RevokedBefore.Equal(r.RevokedBefore) {
+			return
+		}
+	}
+	if r.Deleted && !r.ExpiresAt.IsZero() && UTCNow().After(r.ExpiresAt) {
+		if _, tracked := idx.items[path]; tracked {
+			delete(idx.items, path)
+			idx.generation++
+		}
+		delete(idx.floors, path)
+		return
+	}
+	if !r.Deleted && r.RevokedBefore.IsZero() {
+		_, tracked := idx.items[path]
+		_, hasFloor := idx.floors[path]
+		if tracked || hasFloor {
+			if idx.floors == nil {
+				idx.floors = make(map[string]time.Time)
+			}
+			idx.floors[path] = r.timestamp()
+		}
+		if tracked {
+			delete(idx.items, path)
+			idx.generation++
+		}
+		return
+	}
+	if idx.items == nil {
+		idx.items = make(map[string]iamRevision)
+	}
+	idx.items[path] = r
+	delete(idx.floors, path)
+	idx.generation++
+}
+
+func (idx *iamRevisionIndex) get(path string) iamRevision {
+	if idx == nil {
+		return iamRevision{}
+	}
+	idx.mu.RLock()
+	defer idx.mu.RUnlock()
+	return idx.items[path]
+}
+
+func (idx *iamRevisionIndex) snapshot() map[string]iamRevision {
+	idx.mu.Lock()
+	defer idx.mu.Unlock()
+	for path, r := range idx.items {
+		if r.Deleted && !r.ExpiresAt.IsZero() && UTCNow().After(r.ExpiresAt) {
+			delete(idx.items, path)
+			delete(idx.floors, path)
+			idx.generation++
+		}
+	}
+	return maps.Clone(idx.items)
+}
+
+func (idx *iamRevisionIndex) count() int {
+	idx.mu.RLock()
+	defer idx.mu.RUnlock()
+	return len(idx.items)
+}
+
+// A process-local generation plus the protocol's instance ID is sufficient
+// for acknowledgements. Avoid hashing the entire index on every IAM write.
+func (idx *iamRevisionIndex) digest() string {
+	idx.mu.RLock()
+	defer idx.mu.RUnlock()
+	return fmt.Sprintf("%x:%x", idx.generation, len(idx.items))
+}
+
+func (idx *iamRevisionIndex) forget(path string) {
+	idx.mu.Lock()
+	if _, ok := idx.items[path]; ok {
+		delete(idx.items, path)
+		idx.generation++
+	}
+	delete(idx.floors, path)
+	if strings.HasPrefix(path, iamConfigUsersPrefix) {
+		delete(idx.parents, strings.TrimSuffix(strings.TrimPrefix(path, iamConfigUsersPrefix), "/"+iamIdentityFile))
+	}
+	idx.mu.Unlock()
+}
+
+func (c *iamCache) userRevocation(user string) iamRevision {
+	r := c.revisions.parentRevision(user)
+	if u, ok := c.iamUsersMap[user]; ok && u.RevokedBefore.After(r.RevokedBefore) {
+		r.RevokedBefore = u.RevokedBefore
+	}
+	return r
+}
+
+func (c *iamCache) groupMemberAllowed(member string, grantedAt, groupBoundary time.Time) bool {
+	r := c.userRevocation(member)
+	return !r.Deleted && (r.RevokedBefore.IsZero() || grantedAt.After(r.RevokedBefore)) && (groupBoundary.IsZero() || grantedAt.After(groupBoundary))
+}
+
+func iamMappingParentPath(path string) string {
+	kind, name, ok := strings.Cut(strings.TrimPrefix(path, iamConfigPolicyDBPrefix), "/")
+	if !ok {
+		return ""
+	}
+	name = strings.TrimSuffix(name, ".json")
+	switch kind {
+	case "users", "sts-users":
+		return getUserIdentityPath(name, regUser)
+	case "service-accounts":
+		return getUserIdentityPath(name, svcUser)
+	case "groups":
+		return getGroupInfoPath(name)
+	}
+	return ""
+}
+
+func (idx *iamRevisionIndex) mappingAllowed(path string, mp MappedPolicy) bool {
+	if mp.Deleted || idx.get(path).Deleted {
+		return false
+	}
+	r := idx.get(iamMappingParentPath(path))
+	return !r.Deleted && (r.RevokedBefore.IsZero() || mp.UpdatedAt.After(r.RevokedBefore))
+}
+
+// Apply the persisted commit boundary even before dependent cache cleanup has
+// completed. The map namespace is part of the authorization record's identity.
+func (c *iamCache) cachedMappedPolicy(name string, userType IAMUserType, isGroup bool) (MappedPolicy, bool) {
+	var mp MappedPolicy
+	var ok bool
+	switch {
+	case isGroup:
+		mp, ok = c.iamGroupPolicyMap.Load(name)
+	case userType == stsUser:
+		mp, ok = c.iamSTSPolicyMap.Load(name)
+	default:
+		mp, ok = c.iamUserPolicyMap.Load(name)
+	}
+	if !ok || !c.revisions.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), mp) {
+		return MappedPolicy{}, false
+	}
+	return mp, true
+}
+
+func (idx *iamRevisionIndex) parentRevision(user string) iamRevision {
+	if idx == nil {
+		return iamRevision{}
+	}
+	idx.mu.RLock()
+	p := idx.parents[user]
+	idx.mu.RUnlock()
+	return iamRevision{Deleted: p.deleted, RevokedBefore: p.before}
+}

cmd/iam-revision-lock_test.go

@@ -0,0 +1,305 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"crypto/sha256"
+	"fmt"
+	"os"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	etcd "go.etcd.io/etcd/client/v3"
+	"go.etcd.io/etcd/client/v3/concurrency"
+	"go.etcd.io/etcd/client/v3/namespace"
+)
+
+type iamRevisionLockObserver struct {
+	ObjectLayer
+	path    string
+	waiting chan struct{}
+	once    sync.Once
+}
+
+func (o *iamRevisionLockObserver) NewNSLock(bucket string, objects ...string) RWLocker {
+	lock := o.ObjectLayer.NewNSLock(bucket, objects...)
+	if bucket == minioMetaBucket && len(objects) == 1 && objects[0] == o.path {
+		return &iamRevisionObservedLock{RWLocker: lock, observe: func() { o.once.Do(func() { close(o.waiting) }) }}
+	}
+	return lock
+}
+
+type iamRevisionObservedLock struct {
+	RWLocker
+	observe func()
+}
+
+func (l *iamRevisionObservedLock) GetLock(ctx context.Context, timeout *dynamicTimeout) (LockContext, error) {
+	l.observe()
+	return l.RWLocker.GetLock(ctx, timeout)
+}
+
+type iamRevisionWatchObserver struct {
+	etcd.Watcher
+	waiting chan struct{}
+	once    sync.Once
+}
+
+func (w *iamRevisionWatchObserver) Watch(ctx context.Context, key string, opts ...etcd.OpOption) etcd.WatchChan {
+	w.once.Do(func() { close(w.waiting) })
+	return w.Watcher.Watch(ctx, key, opts...)
+}
+
+// Simulate an unavailable cleanup RPC. Mutex.Lock calls Delete after its wait
+// is canceled; that RPC must inherit a deadline too, not Client.Ctx() forever.
+type iamRevisionCleanupBlocker struct {
+	etcd.KV
+	release chan struct{}
+}
+
+func (b *iamRevisionCleanupBlocker) Delete(ctx context.Context, key string, opts ...etcd.OpOption) (*etcd.DeleteResponse, error) {
+	if strings.Contains(key, "/iam-revision-locks/") {
+		select {
+		case <-ctx.Done():
+			return nil, ctx.Err()
+		case <-b.release:
+		}
+	}
+	return b.KV.Delete(ctx, key, opts...)
+}
+
+type iamRevisionReadBlocker struct {
+	IAMStorageAPI
+	path    string
+	after   int
+	waiting chan struct{}
+}
+
+func (b *iamRevisionReadBlocker) loadIAMConfig(ctx context.Context, item any, path string) error {
+	if path == b.path {
+		b.after--
+		if b.after == 0 {
+			close(b.waiting)
+			<-ctx.Done()
+			return ctx.Err()
+		}
+	}
+	return b.IAMStorageAPI.loadIAMConfig(ctx, item, path)
+}
+
+func TestIAMRevisionReadDoesNotBlockAuthentication(t *testing.T) {
+	for _, stage := range []struct {
+		name   string
+		offset time.Duration
+	}{{"deletion", time.Minute}, {"retained_revocation", -time.Minute}} {
+		t.Run(stage.name, func(t *testing.T) {
+			resetTestGlobals()
+			t.Cleanup(resetTestGlobals)
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+			disks, err := getRandomDisks(1)
+			if err != nil {
+				t.Fatal(err)
+			}
+			obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
+			if err != nil {
+				t.Fatal(err)
+			}
+			t.Cleanup(func() {
+				obj.Shutdown(context.Background())
+				os.RemoveAll(disks[0])
+			})
+			store := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+			const user = "read-blocked-parent"
+			created, err := store.AddUser(ctx, user, madmin.AddOrUpdateUserReq{SecretKey: "original-password", Status: madmin.AccountEnabled})
+			if err != nil {
+				t.Fatal(err)
+			}
+			blocked := &iamRevisionReadBlocker{IAMStorageAPI: store.IAMStorageAPI, path: getUserIdentityPath(user, regUser), after: 1, waiting: make(chan struct{})}
+			store.IAMStorageAPI = blocked
+			done := make(chan error, 1)
+			go func() {
+				done <- store.DeleteUser(withIAMReplicationTime(ctx, created.Add(stage.offset)), user, regUser)
+			}()
+			defer func() { cancel(); <-done }()
+			select {
+			case <-blocked.waiting:
+			case <-time.After(5 * time.Second):
+				t.Fatal("revision read was not attempted")
+			}
+			read := make(chan bool, 1)
+			go func() {
+				u, ok := store.GetUser(user)
+				read <- ok && u.Credentials.SecretKey == "original-password"
+			}()
+			select {
+			case ok := <-read:
+				if !ok {
+					t.Fatal("pending revision read changed the cached identity")
+				}
+			case <-time.After(time.Second):
+				t.Fatal("revision read blocked cached authentication")
+			}
+		})
+	}
+}
+
+func TestIAMRevisionLockContention(t *testing.T) {
+	for _, backend := range []string{"object", "etcd"} {
+		t.Run(backend, func(t *testing.T) {
+			endpoint := os.Getenv("SILO_TEST_IAM_REVOCATION_ETCD")
+			if backend == "etcd" && endpoint == "" {
+				t.Skip("set SILO_TEST_IAM_REVOCATION_ETCD to a disposable etcd endpoint")
+			}
+			for _, outcome := range []string{"release", "cancel", "default_timeout"} {
+				t.Run(outcome, func(t *testing.T) {
+					resetTestGlobals()
+					t.Cleanup(resetTestGlobals)
+					ctx, cancel := context.WithCancel(context.Background())
+					defer cancel()
+					oldTimeout := defaultContextTimeout
+					defaultContextTimeout = 2 * time.Second
+					t.Cleanup(func() { defaultContextTimeout = oldTimeout })
+					must := func(err error) {
+						t.Helper()
+						if err != nil {
+							t.Fatal(err)
+						}
+					}
+					const user = "contended-user"
+					path := getUserIdentityPath(user, regUser)
+					waiting := make(chan struct{})
+					var store *IAMStoreSys
+					var hold func() func()
+					unblockCleanup := func() {}
+					if backend == "object" {
+						disks, err := getRandomDisks(1)
+						must(err)
+						obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
+						must(err)
+						t.Cleanup(func() {
+							obj.Shutdown(context.Background())
+							os.RemoveAll(disks[0])
+						})
+						observed := &iamRevisionLockObserver{ObjectLayer: obj, path: path + ".revision-lock", waiting: waiting}
+						store = &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
+						hold = func() func() {
+							lock := obj.NewNSLock(minioMetaBucket, observed.path)
+							lc, err := lock.GetLock(ctx, newDynamicTimeout(time.Second, time.Second))
+							must(err)
+							store.IAMStorageAPI.(*IAMObjectStore).objAPI = observed
+							return func() { lock.Unlock(lc) }
+						}
+					} else {
+						client, err := etcd.New(etcd.Config{Endpoints: strings.Split(endpoint, ","), DialTimeout: time.Second})
+						must(err)
+						t.Cleanup(func() { client.Close() })
+						prefix := fmt.Sprintf("/silo-lock-test/%d/", time.Now().UnixNano())
+						client.KV = namespace.NewKV(client.KV, prefix)
+						client.Watcher = namespace.NewWatcher(client.Watcher, prefix)
+						store = &IAMStoreSys{IAMStorageAPI: newIAMEtcdStore(client, MinIOUsersSysType)}
+						hold = func() func() {
+							session, err := concurrency.NewSession(client, concurrency.WithContext(ctx))
+							must(err)
+							lock := concurrency.NewMutex(session, fmt.Sprintf("%s/iam-revision-locks/%x", minioConfigPrefix, sha256.Sum256([]byte(path))))
+							must(lock.Lock(ctx))
+							client.Watcher = &iamRevisionWatchObserver{Watcher: client.Watcher, waiting: waiting}
+							blocker := &iamRevisionCleanupBlocker{KV: client.KV, release: make(chan struct{})}
+							client.KV = blocker
+							unblockCleanup = sync.OnceFunc(func() { close(blocker.release) })
+							t.Cleanup(unblockCleanup)
+							return func() { session.Close() }
+						}
+					}
+					request := func(secret string) madmin.AddOrUpdateUserReq {
+						return madmin.AddOrUpdateUserReq{SecretKey: secret, Status: madmin.AccountEnabled}
+					}
+					_, err := store.AddUser(ctx, user, request("original-password"))
+					must(err)
+					release := sync.OnceFunc(hold())
+					t.Cleanup(release)
+					writeCtx, cancelWrite := context.WithCancel(ctx)
+					defer cancelWrite()
+					first, second := make(chan error, 1), make(chan error, 1)
+					var writers sync.WaitGroup
+					t.Cleanup(func() {
+						cancelWrite()
+						unblockCleanup()
+						release()
+						writers.Wait()
+					})
+					writers.Go(func() {
+						_, err := store.AddUser(writeCtx, user, request("first-password"))
+						first <- err
+					})
+					select {
+					case <-waiting:
+					case <-time.After(5 * time.Second):
+						t.Fatal("writer did not attempt the held revision lock")
+					}
+					// A second writer must queue without taking the cache's RWMutex:
+					// Go's writer preference would otherwise block every new reader.
+					writers.Go(func() {
+						_, err := store.AddUser(ctx, user, request("second-password"))
+						second <- err
+					})
+					select {
+					case err := <-second:
+						t.Fatalf("second writer bypassed the first: %v", err)
+					case <-time.After(50 * time.Millisecond):
+					}
+					read := make(chan UserIdentity, 1)
+					go func() {
+						u, _ := store.GetUser(user)
+						read <- u
+					}()
+					select {
+					case u := <-read:
+						if u.Credentials.SecretKey != "original-password" {
+							t.Fatal("pending write changed the cached credential")
+						}
+					case <-time.After(time.Second):
+						t.Fatal("distributed lock contention blocked cached authentication")
+					}
+					switch outcome {
+					case "release":
+						release()
+					case "cancel":
+						cancelWrite()
+					}
+					select {
+					case err := <-first:
+						if outcome == "release" {
+							must(err)
+						} else if err == nil {
+							t.Fatal("canceled or timed-out write succeeded")
+						}
+					case <-time.After(5 * time.Second):
+						t.Fatal("lock wait or cancellation cleanup exceeded its deadline")
+					}
+					release()
+					select {
+					case err := <-second:
+						must(err)
+					case <-time.After(5 * time.Second):
+						t.Fatal("queued writer did not recover after the first completed")
+					}
+					cached, ok := store.GetUser(user)
+					if !ok || cached.Credentials.SecretKey != "second-password" {
+						t.Fatal("cached write order was lost")
+					}
+					var persisted UserIdentity
+					must(store.loadIAMConfig(ctx, &persisted, path))
+					if persisted.Credentials.SecretKey != cached.Credentials.SecretKey || !persisted.UpdatedAt.Equal(cached.UpdatedAt) {
+						t.Fatal("persistent and cached revisions differ")
+					}
+				})
+			}
+		})
+	}
+}

cmd/iam-revision.go

@@ -0,0 +1,749 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"crypto/sha256"
+	"errors"
+	"fmt"
+	"net/http"
+	"sort"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/auth"
+	etcd "go.etcd.io/etcd/client/v3"
+	"go.etcd.io/etcd/client/v3/concurrency"
+)
+
+var errIAMStaleUpdate = errors.New("IAM update predates a stored revision or revocation")
+
+// The parent is still live; callers must not broadcast a user deletion when
+// only its revocation boundary was retained.
+var errIAMRevocationRetained = errors.New("IAM revocation recorded without deleting the record")
+
+// A revocation advances the boundary even when a newer identity already
+// exists. Keep this operation distinct from replacing/deleting that identity.
+type iamUserRevocation struct {
+	UserIdentity
+	retained bool
+}
+
+type iamGroupRevocation struct {
+	GroupInfo
+	retained     bool
+	requireEmpty bool
+}
+
+// Natural expiration is distinct from revoking a live credential. An expired
+// immutable STS token can be removed; a reusable service-account key retains
+// its revision so an older non-expiring credential cannot return.
+type iamExpireIdentity struct{}
+
+// The authoritative revocation is durable even if dependent cleanup fails.
+// Callers must publish it to sibling caches before returning the error.
+type iamCommittedCleanupError struct {
+	err      error
+	retained bool
+}
+
+func (e *iamCommittedCleanupError) Error() string {
+	return "IAM revocation committed; cleanup failed: " + e.err.Error()
+}
+func (e *iamCommittedCleanupError) Unwrap() error { return e.err }
+
+type iamReplicationTimeKey struct{}
+
+func withIAMReplicationTime(ctx context.Context, at time.Time) context.Context {
+	return context.WithValue(ctx, iamReplicationTimeKey{}, at)
+}
+
+func iamReplicationTime(ctx context.Context) (time.Time, bool) {
+	at, ok := ctx.Value(iamReplicationTimeKey{}).(time.Time)
+	return at, ok
+}
+
+func iamReplicationError(err error) error {
+	if errors.Is(err, errIAMStaleUpdate) {
+		// Retrying an obsolete event cannot change the result.
+		return nil
+	}
+	return wrapSRErr(err)
+}
+
+// Deletions occupy the original IAM config path. They contain no secret or
+// grant and are hidden by the normal loaders, but remain available to heal
+// and to timestamp comparisons after a restart. Do not age them out: a peer
+// can be offline indefinitely.
+type iamRevision struct {
+	UpdatedAt     time.Time        `json:"updatedAt"`
+	UpdateDate    time.Time        `json:"UpdateDate"`
+	Deleted       bool             `json:"deleted"`
+	RevokedBefore time.Time        `json:"revokedBefore"`
+	ExpiresAt     time.Time        `json:"expiresAt,omitempty"`
+	Credentials   auth.Credentials `json:"credentials"`
+}
+
+func (r iamRevision) timestamp() time.Time {
+	if r.UpdateDate.After(r.UpdatedAt) {
+		return r.UpdateDate
+	}
+	return r.UpdatedAt
+}
+
+func loadIAMRevision(ctx context.Context, store IAMStorageAPI, path string) (iamRevision, error) {
+	var r iamRevision
+	err := store.loadIAMConfig(ctx, &r, path)
+	if errors.Is(err, errConfigNotFound) {
+		err = nil
+	}
+	return r, err
+}
+
+func (store *IAMStoreSys) checkIAMRevision(ctx context.Context, path string, deleting bool) error {
+	at, replicated := iamReplicationTime(ctx)
+	if !replicated {
+		return nil
+	}
+	return store.withIAMStorage(ctx, func(ctx context.Context) error {
+		r, err := loadIAMRevision(ctx, store.IAMStorageAPI, path)
+		if err != nil {
+			return err
+		}
+		if r.timestamp().After(at) || (r.Deleted && !deleting && !at.After(r.timestamp())) {
+			return errIAMStaleUpdate
+		}
+		return nil
+	})
+}
+
+// This signed claim records the parent's revocation boundary at issuance.
+// Unlike UpdatedAt, it cannot advance when an offline site edits an old child.
+// It travels in the existing service-account Claims and STS SessionToken fields.
+const iamParentRevocationClaim = "siloParentRevocation"
+
+func setIAMParentRevocationClaim(ctx context.Context, store IAMStorageAPI, parent string, claims map[string]any) error {
+	delete(claims, iamParentRevocationClaim)
+	if parent == "" || parent == globalActiveCred.AccessKey {
+		return nil
+	}
+	r, err := loadIAMRevision(ctx, store, getUserIdentityPath(parent, regUser))
+	if err != nil {
+		return err
+	}
+	if r.Deleted {
+		return errIAMStaleUpdate
+	}
+	if !r.RevokedBefore.IsZero() {
+		claims[iamParentRevocationClaim] = r.RevokedBefore.Format(time.RFC3339Nano)
+	}
+	return nil
+}
+
+func iamCredentialSurvivesRevocation(cred auth.Credentials, at time.Time) bool {
+	if at.IsZero() {
+		return true
+	}
+	s, _ := cred.Claims[iamParentRevocationClaim].(string)
+	issuedAfter, err := time.Parse(time.RFC3339Nano, s)
+	return err == nil && !issuedAfter.Before(at)
+}
+
+// Parent revocations delete old children even if an offline peer has edited
+// them later. Preserve children that prove issuance after this revocation.
+func iamChildDeletionContext(ctx context.Context, child UserIdentity) (context.Context, bool) {
+	if at, replicated := iamReplicationTime(ctx); replicated {
+		if !at.IsZero() && iamCredentialSurvivesRevocation(child.Credentials, at) {
+			return ctx, false
+		}
+		if child.UpdatedAt.After(at) {
+			ctx = withIAMReplicationTime(ctx, child.UpdatedAt)
+		}
+	}
+	return ctx, true
+}
+
+// A delayed service account or STS event must not outlive deletion of its
+// built-in parent. The caller must populate Claims from the verified token.
+func checkIAMParentRevision(ctx context.Context, store IAMStorageAPI, cred auth.Credentials) error {
+	parent := cred.ParentUser
+	if parent == "" || parent == globalActiveCred.AccessKey {
+		return nil
+	}
+	r, err := loadIAMRevision(ctx, store, getUserIdentityPath(parent, regUser))
+	if err != nil {
+		return err
+	}
+	if r.Deleted || !iamCredentialSurvivesRevocation(cred, r.RevokedBefore) {
+		return errIAMStaleUpdate
+	}
+	return nil
+}
+
+// Called with the IAM writer mutex and cache lock held. Persistence only
+// touches the caller's record, not the cache. Keep writers serialized while
+// allowing cached authentication reads throughout storage and lock waits.
+func (store *IAMStoreSys) withIAMStorage(ctx context.Context, fn func(context.Context) error) error {
+	store.IAMStorageAPI.unlock()
+	defer store.IAMStorageAPI.lock()
+	ctx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
+	defer cancel()
+	return fn(ctx)
+}
+
+func (store *IAMStoreSys) saveIAMRevision(ctx context.Context, path string, item any, opts ...options) error {
+	return store.withIAMStorage(ctx, func(ctx context.Context) error {
+		return saveIAMRevision(ctx, store.IAMStorageAPI, path, item, opts...)
+	})
+}
+
+func (store *IAMStoreSys) checkIAMParentRevision(ctx context.Context, cred auth.Credentials) error {
+	return store.withIAMStorage(ctx, func(ctx context.Context) error {
+		return checkIAMParentRevision(ctx, store.IAMStorageAPI, cred)
+	})
+}
+
+// Update the caller's record with the persisted revision before it is cached.
+func saveIAMRevision(ctx context.Context, store IAMStorageAPI, path string, item any, opts ...options) error {
+	ctx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
+	defer cancel()
+
+	// Serialize compare-and-write across nodes, as well as goroutines. Use a
+	// separate lock name so saving the config does not reacquire this lock.
+	switch s := store.(type) {
+	case *IAMObjectStore:
+		lock := s.objAPI.NewNSLock(minioMetaBucket, path+".revision-lock")
+		lc, err := lock.GetLock(ctx, globalOperationTimeout)
+		if err != nil {
+			return err
+		}
+		defer lock.Unlock(lc)
+		ctx = lc.Context()
+	case *IAMEtcdStore:
+		// Mutex.Lock also uses Client.Ctx() for cleanup after cancellation.
+		// Borrow the existing services with the operation's bounded context;
+		// never close this facade, which does not own those services.
+		client := etcd.NewCtxClient(ctx, etcd.WithZapLogger(s.client.GetLogger()))
+		client.KV, client.Lease, client.Watcher = s.client.KV, s.client.Lease, s.client.Watcher
+		session, err := concurrency.NewSession(client, concurrency.WithContext(ctx))
+		if err != nil {
+			return err
+		}
+		defer func() {
+			session.Orphan()
+			// A canceled operation must still release its lease when etcd is
+			// reachable. If it is unavailable, stop waiting and let it expire.
+			cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultContextTimeout)
+			defer cancel()
+			_, _ = s.client.Revoke(cleanupCtx, session.Lease())
+		}()
+		lock := concurrency.NewMutex(session, fmt.Sprintf("%s/iam-revision-locks/%x", minioConfigPrefix, sha256.Sum256([]byte(path))))
+		if err = lock.Lock(ctx); err != nil {
+			return err
+		}
+		// Revoking the session lease releases the lock, including on cancellation.
+	}
+	previous, err := loadIAMRevision(ctx, store, path)
+	if err != nil {
+		return err
+	}
+	if _, expiring := item.(*iamExpireIdentity); expiring {
+		sts := strings.HasPrefix(path, iamConfigSTSPrefix)
+		if previous.Deleted {
+			if sts && !previous.ExpiresAt.IsZero() && UTCNow().After(previous.ExpiresAt) {
+				return expireIAMSTSConfig(ctx, store, path)
+			}
+			return nil
+		}
+		if previous.timestamp().IsZero() || !previous.Credentials.IsExpired() {
+			return nil
+		}
+		if sts {
+			return expireIAMSTSConfig(ctx, store, path)
+		}
+		item = &UserIdentity{Version: 1, Deleted: true}
+		ctx = withIAMReplicationTime(ctx, previous.timestamp())
+	}
+	var revocation *iamUserRevocation
+	if op, ok := item.(*iamUserRevocation); ok {
+		revocation = op
+		op.UserIdentity = UserIdentity{Version: 1, Deleted: true}
+		if origin, replicated := iamReplicationTime(ctx); replicated && previous.timestamp().After(origin) {
+			if previous.Deleted || !origin.After(previous.RevokedBefore) {
+				return errIAMStaleUpdate
+			}
+			op.retained = true
+			op.UserIdentity = UserIdentity{Version: 1, Credentials: previous.Credentials, UpdatedAt: previous.timestamp(), RevokedBefore: origin}
+			ctx = withIAMReplicationTime(ctx, previous.timestamp())
+		}
+		item = &op.UserIdentity
+	}
+	var groupRevocation *iamGroupRevocation
+	if op, ok := item.(*iamGroupRevocation); ok {
+		groupRevocation = op
+		var group GroupInfo
+		if err := store.loadIAMConfig(ctx, &group, path); err != nil && !errors.Is(err, errConfigNotFound) {
+			return err
+		}
+		if op.requireEmpty && !group.Deleted {
+			for _, member := range group.Members {
+				r := store.revisionIndex().get(getUserIdentityPath(member, regUser))
+				at := group.MemberGrants[member]
+				if !r.Deleted && (r.RevokedBefore.IsZero() || at.After(r.RevokedBefore)) && (group.RevokedBefore.IsZero() || at.After(group.RevokedBefore)) {
+					return errGroupNotEmpty
+				}
+			}
+		}
+		op.GroupInfo = GroupInfo{Version: 1, Deleted: true}
+		if origin, replicated := iamReplicationTime(ctx); replicated && previous.timestamp().After(origin) {
+			if previous.Deleted || !origin.After(previous.RevokedBefore) {
+				return errIAMStaleUpdate
+			}
+			op.retained = true
+			op.GroupInfo = group
+			op.RevokedBefore = origin
+			ctx = withIAMReplicationTime(ctx, previous.timestamp())
+		}
+		item = &op.GroupInfo
+	}
+	var at *time.Time
+	var deleted bool
+	switch v := item.(type) {
+	case *UserIdentity:
+		at, deleted = &v.UpdatedAt, v.Deleted
+		if boundary, ok := ctx.Value(iamRecordBoundaryKey{}).(time.Time); ok && boundary.After(v.RevokedBefore) {
+			v.RevokedBefore = boundary
+		}
+		if previous.RevokedBefore.After(v.RevokedBefore) {
+			v.RevokedBefore = previous.RevokedBefore
+		}
+	case *GroupInfo:
+		at, deleted = &v.UpdatedAt, v.Deleted
+		if boundary, ok := ctx.Value(iamRecordBoundaryKey{}).(time.Time); ok && boundary.After(v.RevokedBefore) {
+			v.RevokedBefore = boundary
+		}
+		if !deleted {
+			var group GroupInfo
+			if err := store.loadIAMConfig(ctx, &group, path); err != nil && !errors.Is(err, errConfigNotFound) {
+				return err
+			}
+			mergeIAMGroupMutation(ctx, group, v)
+		}
+		if previous.RevokedBefore.After(v.RevokedBefore) {
+			v.RevokedBefore = previous.RevokedBefore
+		}
+	case *MappedPolicy:
+		at, deleted = &v.UpdatedAt, v.Deleted
+	case *PolicyDoc:
+		at, deleted = &v.UpdateDate, v.Deleted
+	default:
+		return errInvalidArgument
+	}
+	if strings.HasPrefix(path, iamConfigSTSPrefix) && previous.Deleted && !deleted {
+		// STS access keys identify immutable tokens, not reusable user names.
+		return errIAMStaleUpdate
+	}
+	if origin, replicated := iamReplicationTime(ctx); replicated {
+		*at = origin
+		if previous.timestamp().After(origin) || (previous.Deleted && !deleted && !origin.After(previous.timestamp())) {
+			return errIAMStaleUpdate
+		}
+		if strings.HasPrefix(path, iamConfigServiceAccountsPrefix) && !deleted && previous.Credentials.AccessKey != "" && previous.timestamp().Equal(origin) {
+			// Duplicate service snapshots are acknowledgements, not new creates
+			// or edits. Reload the winner without writing, so even a stale
+			// sibling cache is refreshed by the retry before acknowledging it.
+			return store.loadIAMConfig(ctx, item, path)
+		}
+		if previous.Deleted && deleted && !origin.After(previous.timestamp()) {
+			// An already-applied tombstone needs no further persistent write.
+			if v, ok := item.(*UserIdentity); ok {
+				v.RevokedBefore = previous.RevokedBefore
+			}
+			if v, ok := item.(*GroupInfo); ok {
+				v.RevokedBefore = previous.RevokedBefore
+			}
+			return nil
+		}
+	} else {
+		if previous.Deleted && deleted {
+			// A peer notification without an originating revision must not
+			// advance a tombstone past a subsequent deliberate recreation.
+			*at = previous.timestamp()
+			if v, ok := item.(*UserIdentity); ok {
+				v.RevokedBefore = previous.RevokedBefore
+			}
+			if v, ok := item.(*GroupInfo); ok {
+				v.RevokedBefore = previous.RevokedBefore
+			}
+			return nil
+		}
+		if at.IsZero() {
+			*at = UTCNow()
+		}
+		if !at.After(previous.timestamp()) {
+			*at = previous.timestamp().Add(time.Nanosecond)
+		}
+	}
+	if v, ok := item.(*UserIdentity); ok {
+		if deleted {
+			// Retain only the parent name for root-account exclusion during heal.
+			v.Credentials = auth.Credentials{ParentUser: previous.Credentials.ParentUser}
+			v.RevokedBefore = *at
+			if strings.HasPrefix(path, iamConfigSTSPrefix) && !previous.Credentials.Expiration.IsZero() && !previous.Credentials.Expiration.Equal(timeSentinel) {
+				// The signed STS token cannot authorize beyond this time, even
+				// if an offline site replays it with a newer event timestamp.
+				v.ExpiresAt = previous.Credentials.Expiration.Add(globalMaxSkewTime)
+				opts = []options{{ttl: max(1, int64(time.Until(v.ExpiresAt).Seconds())+1)}}
+			}
+		} else {
+			if v.Credentials.SessionToken != "" && v.Credentials.Claims == nil {
+				claims, err := extractJWTClaims(*v)
+				if err != nil {
+					return err
+				}
+				v.Credentials.Claims = claims.Map()
+			}
+			if err = checkIAMParentRevision(ctx, store, v.Credentials); err != nil {
+				return err
+			}
+		}
+	}
+	if v, ok := item.(*GroupInfo); ok && deleted {
+		v.RevokedBefore = *at
+		v.Members, v.MemberGrants = nil, nil
+	}
+	if _, ok := item.(*MappedPolicy); ok && !deleted {
+		if parentPath := iamMappingParentPath(path); parentPath != "" {
+			parent, err := loadIAMRevision(ctx, store, parentPath)
+			if err != nil {
+				return err
+			}
+			if parent.Deleted {
+				return errIAMStaleUpdate
+			}
+			if !parent.RevokedBefore.IsZero() && !at.After(parent.RevokedBefore) {
+				if _, replicated := iamReplicationTime(ctx); replicated {
+					return errIAMStaleUpdate
+				}
+				*at = parent.RevokedBefore.Add(time.Nanosecond)
+			}
+		}
+	}
+	if err := store.saveIAMConfig(ctx, item, path, opts...); err != nil {
+		return err
+	}
+	if revocation != nil && revocation.retained {
+		return errIAMRevocationRetained
+	}
+	if groupRevocation != nil && groupRevocation.retained {
+		return errIAMRevocationRetained
+	}
+	return nil
+}
+
+func (iamOS *IAMObjectStore) listIAMConfigPaths(ctx context.Context) ([]string, error) {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+	var paths []string
+	for item := range listIAMConfigItems(ctx, iamOS.objAPI, iamConfigPrefix+"/") {
+		if item.Err != nil {
+			return nil, item.Err
+		}
+		paths = append(paths, iamConfigPrefix+"/"+item.Item)
+	}
+	return paths, nil
+}
+
+func (ies *IAMEtcdStore) listIAMConfigPaths(ctx context.Context) ([]string, error) {
+	ctx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
+	defer cancel()
+	r, err := ies.client.Get(ctx, iamConfigPrefix+"/", etcd.WithPrefix(), etcd.WithKeysOnly())
+	if err != nil {
+		return nil, err
+	}
+	paths := make([]string, 0, len(r.Kvs))
+	for _, kv := range r.Kvs {
+		paths = append(paths, string(kv.Key))
+	}
+	return paths, nil
+}
+
+func iamDeletionItem(path string, r iamRevision) (item madmin.SRIAMItem, ok bool) {
+	if (strings.HasPrefix(path, iamConfigUsersPrefix) || strings.HasPrefix(path, iamConfigGroupsPrefix)) && !r.RevokedBefore.IsZero() {
+		// Recreating a parent does not cancel its older revocation of derived
+		// credentials. Replay this boundary even after the parent is live again.
+		r.Deleted = true
+		r.UpdatedAt, r.UpdateDate = r.RevokedBefore, time.Time{}
+	}
+	if !r.Deleted {
+		return item, false
+	}
+	item.UpdatedAt = r.timestamp()
+	switch {
+	case strings.HasPrefix(path, iamConfigUsersPrefix):
+		name := strings.TrimSuffix(strings.TrimPrefix(path, iamConfigUsersPrefix), "/"+iamIdentityFile)
+		item.Type = madmin.SRIAMItemIAMUser
+		item.IAMUser = &madmin.SRIAMUser{AccessKey: name, IsDeleteReq: true}
+	case strings.HasPrefix(path, iamConfigServiceAccountsPrefix):
+		name := strings.TrimSuffix(strings.TrimPrefix(path, iamConfigServiceAccountsPrefix), "/"+iamIdentityFile)
+		if name == siteReplicatorSvcAcc || r.Credentials.ParentUser == globalActiveCred.AccessKey {
+			return item, false
+		}
+		item.Type = madmin.SRIAMItemSvcAcc
+		item.SvcAccChange = &madmin.SRSvcAccChange{Delete: &madmin.SRSvcAccDelete{AccessKey: name}}
+	case strings.HasPrefix(path, iamConfigGroupsPrefix):
+		name := strings.TrimSuffix(strings.TrimPrefix(path, iamConfigGroupsPrefix), "/"+iamGroupMembersFile)
+		item.Type = madmin.SRIAMItemGroupInfo
+		item.GroupInfo = &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: name, IsRemove: true}}
+	case strings.HasPrefix(path, iamConfigPoliciesPrefix):
+		item.Type = madmin.SRIAMItemPolicy
+		item.Name = strings.TrimSuffix(strings.TrimPrefix(path, iamConfigPoliciesPrefix), "/"+iamPolicyFile)
+	case strings.HasPrefix(path, iamConfigPolicyDBPrefix):
+		prefix, name, found := strings.Cut(strings.TrimPrefix(path, iamConfigPolicyDBPrefix), "/")
+		if !found {
+			return item, false
+		}
+		typ := regUser
+		switch prefix {
+		case "sts-users":
+			typ = stsUser
+		case "service-accounts":
+			typ = svcUser
+		}
+		item.Type = madmin.SRIAMItemPolicyMapping
+		item.PolicyMapping = &madmin.SRPolicyMapping{UserOrGroup: strings.TrimSuffix(name, ".json"), UserType: int(typ), IsGroup: prefix == "groups"}
+	default:
+		// Expired STS credentials are not replayed. Parent revocations and
+		// their retained timestamp reject delayed copies of derived tokens.
+		return item, false
+	}
+	return item, true
+}
+
+func iamDeletionPath(item madmin.SRIAMItem) string {
+	switch item.Type {
+	case madmin.SRIAMItemIAMUser:
+		if item.IAMUser != nil && item.IAMUser.IsDeleteReq {
+			return getUserIdentityPath(item.IAMUser.AccessKey, regUser)
+		}
+	case madmin.SRIAMItemSvcAcc:
+		if item.SvcAccChange != nil && item.SvcAccChange.Delete != nil {
+			return getUserIdentityPath(item.SvcAccChange.Delete.AccessKey, svcUser)
+		}
+	case madmin.SRIAMItemGroupInfo:
+		if item.GroupInfo != nil && item.GroupInfo.UpdateReq.IsRemove && len(item.GroupInfo.UpdateReq.Members) == 0 {
+			return getGroupInfoPath(item.GroupInfo.UpdateReq.Group)
+		}
+	case madmin.SRIAMItemPolicy:
+		if len(item.Policy) == 0 {
+			return getPolicyDocPath(item.Name)
+		}
+	case madmin.SRIAMItemPolicyMapping:
+		if p := item.PolicyMapping; p != nil && p.Policy == "" {
+			return getMappedPolicyPath(p.UserOrGroup, IAMUserType(p.UserType), p.IsGroup)
+		}
+	}
+	return ""
+}
+
+func (c *SiteReplicationSys) healIAMDeletions(ctx context.Context) (err error) {
+	started := time.Now()
+	defer func() {
+		c.iamRevisionMetrics.healDurationMillis.Store(time.Since(started).Milliseconds())
+		if err != nil {
+			c.iamRevisionMetrics.healFailures.Add(1)
+		} else {
+			c.iamRevisionMetrics.healLastSuccess.Store(time.Now().Unix())
+		}
+	}()
+	c.iamHealMu.Lock()
+	defer c.iamHealMu.Unlock()
+	c.RLock()
+	defer c.RUnlock()
+	if !c.enabled {
+		return nil
+	}
+	snapshot := globalIAMSys.store.revisionIndex().snapshot()
+	paths := make([]string, 0, len(snapshot))
+	for path := range snapshot {
+		paths = append(paths, path)
+	}
+	sort.Strings(paths)
+	byType := make(map[string][]iamReplicationItem)
+	for _, path := range paths {
+		r := snapshot[path]
+		item, ok := iamDeletionItem(path, r)
+		if !ok {
+			continue
+		}
+		out := iamReplicationItem{SRIAMItem: item}
+		if item.Type == madmin.SRIAMItemIAMUser && !r.Deleted {
+			out.Type, out.IAMUser = iamUserBoundaryType, nil
+			out.UserRevocation = &iamUserBoundary{User: item.IAMUser.AccessKey, Before: r.RevokedBefore}
+		}
+		if item.Type == madmin.SRIAMItemGroupInfo && !r.Deleted {
+			out.Type, out.GroupInfo = iamGroupBoundaryType, nil
+			out.GroupRevocation = &iamGroupBoundary{Group: item.GroupInfo.UpdateReq.Group, Before: r.RevokedBefore}
+		}
+		byType[item.Type] = append(byType[item.Type], out)
+	}
+	var items []iamReplicationItem
+	for _, typ := range []string{madmin.SRIAMItemPolicyMapping, madmin.SRIAMItemIAMUser, madmin.SRIAMItemSvcAcc, madmin.SRIAMItemGroupInfo, madmin.SRIAMItemPolicy} {
+		items = append(items, byType[typ]...)
+	}
+	if len(items) == 0 {
+		return nil
+	}
+	if c.iamRevisionProgress == nil {
+		c.iamRevisionProgress = make(map[string]iamRevisionProgress)
+	}
+	for id := range c.iamRevisionProgress {
+		if _, present := c.state.Peers[id]; !present {
+			delete(c.iamRevisionProgress, id)
+		}
+	}
+	var progressMu sync.Mutex
+	cerr := c.concDo(nil, func(id string, p madmin.PeerInfo) error {
+		// Bound each pass, but retain acknowledgements independently of the
+		// pass deadline or unrelated changes at either site.
+		peerCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
+		defer cancel()
+		client, err := c.getAdminClient(peerCtx, id)
+		if err != nil {
+			return err
+		}
+		remote, err := executeIAMRevisionRequest(peerCtx, client, http.MethodGet, nil)
+		if err != nil {
+			return err
+		}
+		progressMu.Lock()
+		progress := c.iamRevisionProgress[id]
+		progressMu.Unlock()
+		progress.observePeer(remote)
+		defer func() {
+			progressMu.Lock()
+			c.iamRevisionProgress[id] = progress
+			progressMu.Unlock()
+		}()
+		var pending []iamReplicationItem
+		for _, item := range items {
+			path, version := iamReplicationMarker(item)
+			if progress.Acknowledged[path] != version {
+				pending = append(pending, item)
+			}
+		}
+		// Acknowledgements are only a replay optimization, never GC proof.
+		for path := range progress.Acknowledged {
+			if _, retained := snapshot[path]; !retained {
+				delete(progress.Acknowledged, path)
+			}
+		}
+		var failures []error
+		for next := 0; next < len(pending); {
+			end := min(next+maxIAMRevisionBatch, len(pending))
+			batch := pending[next:end]
+			remote, err = executeIAMRevisionRequest(peerCtx, client, http.MethodPut, &iamRevisionBatch{Version: iamRevisionProtocol, Items: batch})
+			if err != nil {
+				var batchErr *iamRevisionBatchError
+				if !errors.As(err, &batchErr) {
+					return errors.Join(append(failures, err)...)
+				}
+				failures = append(failures, err)
+			} else {
+				progress.observePeer(remote)
+				for _, item := range batch {
+					path, version := iamReplicationMarker(item)
+					progress.Acknowledged[path] = version
+				}
+			}
+			next = end
+		}
+		return errors.Join(failures...)
+	}, "IAM revision convergence")
+	return errors.Unwrap(cerr)
+}
+
+func iamReplicationMarker(item iamReplicationItem) (path, version string) {
+	path = iamDeletionPath(item.SRIAMItem)
+	if item.UserRevocation != nil {
+		path = getUserIdentityPath(item.UserRevocation.User, regUser)
+	}
+	if item.GroupRevocation != nil {
+		path = getGroupInfoPath(item.GroupRevocation.Group)
+	}
+	return path, item.Type + ":" + item.UpdatedAt.UTC().Format(time.RFC3339Nano)
+}
+
+func (store *IAMStoreSys) savePolicyDoc(ctx context.Context, policyName string, p *PolicyDoc) error {
+	return store.saveIAMRevision(ctx, getPolicyDocPath(policyName), p)
+}
+
+func (store *IAMStoreSys) saveMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool, mp *MappedPolicy, opts ...options) error {
+	return store.saveIAMRevision(ctx, getMappedPolicyPath(name, userType, isGroup), mp, opts...)
+}
+
+func (store *IAMStoreSys) saveUserIdentity(ctx context.Context, name string, userType IAMUserType, u *UserIdentity, opts ...options) error {
+	return store.saveIAMRevision(ctx, getUserIdentityPath(name, userType), u, opts...)
+}
+
+func (store *IAMStoreSys) saveGroupInfo(ctx context.Context, name string, gi *GroupInfo) error {
+	return store.saveIAMRevision(ctx, getGroupInfoPath(name), gi)
+}
+
+func (store *IAMStoreSys) deletePolicyDoc(ctx context.Context, name string) error {
+	return store.saveIAMRevision(ctx, getPolicyDocPath(name), &PolicyDoc{Version: 1, Deleted: true})
+}
+
+func (store *IAMStoreSys) deleteMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool) error {
+	return store.saveIAMRevision(ctx, getMappedPolicyPath(name, userType, isGroup), &MappedPolicy{Version: 1, Deleted: true})
+}
+
+func (store *IAMStoreSys) deleteUserIdentity(ctx context.Context, name string, userType IAMUserType) error {
+	return store.saveIAMRevision(ctx, getUserIdentityPath(name, userType), &UserIdentity{Version: 1, Deleted: true})
+}
+
+// Called under the identity's distributed revision lock, after verifying that
+// its immutable STS token (or early-revocation retention) has expired. Only the
+// old token-key mapping is removed; the reusable parent mapping is unaffected.
+func expireIAMSTSConfig(ctx context.Context, store IAMStorageAPI, path string) error {
+	key := strings.TrimSuffix(strings.TrimPrefix(path, iamConfigSTSPrefix), "/"+iamIdentityFile)
+	if err := store.deleteIAMConfig(ctx, getMappedPolicyPath(key, stsUser, false)); err != nil && !errors.Is(err, errConfigNotFound) {
+		return err
+	}
+	return store.deleteIAMConfig(ctx, path)
+}
+
+type (
+	iamExpirationCleanupKey   struct{}
+	iamExpirationCleanupState struct{ failed atomic.Bool }
+)
+
+func withIAMExpirationCleanup(ctx context.Context) context.Context {
+	if _, ok := ctx.Value(iamExpirationCleanupKey{}).(*iamExpirationCleanupState); ok {
+		return ctx
+	}
+	return context.WithValue(ctx, iamExpirationCleanupKey{}, &iamExpirationCleanupState{})
+}
+
+func bestEffortIAMExpiration(ctx context.Context, store IAMStorageAPI, path string) {
+	state, _ := ctx.Value(iamExpirationCleanupKey{}).(*iamExpirationCleanupState)
+	if ctx.Err() != nil || (state != nil && state.failed.Load()) {
+		return
+	}
+	ctx, cancel := context.WithTimeout(ctx, time.Second)
+	defer cancel()
+	// Failure leaves the expired record and its existing version intact.
+	// Stop optional reclamation for this load, while still loading healthy
+	// users. Healthy cleanup has no per-scan quota that could build a backlog.
+	if err := saveIAMRevision(ctx, store, path, &iamExpireIdentity{}); err != nil {
+		if state != nil {
+			state.failed.Store(true)
+		}
+		iamLogIf(ctx, err)
+	}
+}

cmd/iam-revision_test.go

@@ -0,0 +1,330 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"os"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/grid"
+	xnet "github.com/pgsty/silo-pkg/v3/net"
+)
+
+// Count physical saves: comparing timestamps alone would miss identical
+// tombstones being rewritten on every heal pass.
+type iamRevisionWriteCounter struct {
+	IAMStorageAPI
+	data   []byte
+	writes int
+}
+
+func (s *iamRevisionWriteCounter) loadIAMConfig(_ context.Context, item any, _ string) error {
+	return json.Unmarshal(s.data, item)
+}
+
+func (s *iamRevisionWriteCounter) saveIAMConfig(_ context.Context, item any, _ string, _ ...options) error {
+	data, err := json.Marshal(item)
+	if err == nil {
+		s.data = data
+		s.writes++
+	}
+	return err
+}
+
+func TestIAMRevocationTombstoneReplayIsIdempotent(t *testing.T) {
+	at := time.Date(2026, 9, 14, 12, 0, 0, 0, time.UTC)
+	for _, record := range []struct {
+		name string
+		new  func(bool) any
+	}{
+		{"user", func(deleted bool) any { return &UserIdentity{Version: 1, Deleted: deleted} }},
+		{"group", func(deleted bool) any { return &GroupInfo{Version: 1, Deleted: deleted} }},
+		{"policy", func(deleted bool) any { return &PolicyDoc{Version: 1, Deleted: deleted} }},
+		{"mapping", func(deleted bool) any { return &MappedPolicy{Version: 1, Deleted: deleted} }},
+	} {
+		t.Run(record.name, func(t *testing.T) {
+			data, err := json.Marshal(iamRevision{Deleted: true, UpdatedAt: at, RevokedBefore: at})
+			if err != nil {
+				t.Fatal(err)
+			}
+			store := &iamRevisionWriteCounter{data: data}
+			ctx := context.Background()
+			for range 3 {
+				// Site heal carries the original timestamp. Sibling notifications
+				// have no timestamp; both must leave an applied deletion untouched.
+				for _, replay := range []context.Context{withIAMReplicationTime(ctx, at), ctx} {
+					if err := saveIAMRevision(replay, store, record.name, record.new(true)); err != nil {
+						t.Fatal(err)
+					}
+				}
+			}
+			if store.writes != 0 || string(store.data) != string(data) {
+				t.Fatalf("replayed tombstone changed storage: writes=%d, record=%s", store.writes, store.data)
+			}
+			for _, deleted := range []bool{false, true} {
+				err := saveIAMRevision(withIAMReplicationTime(ctx, at.Add(-time.Second)), store, record.name, record.new(deleted))
+				if !errors.Is(err, errIAMStaleUpdate) {
+					t.Fatalf("older event accepted, deleted=%t: %v", deleted, err)
+				}
+			}
+			if err := saveIAMRevision(withIAMReplicationTime(ctx, at), store, record.name, record.new(false)); !errors.Is(err, errIAMStaleUpdate) {
+				t.Fatalf("equal-time recreation accepted: %v", err)
+			}
+			newer := at.Add(time.Minute)
+			if err := saveIAMRevision(withIAMReplicationTime(ctx, newer), store, record.name, record.new(true)); err != nil {
+				t.Fatal(err)
+			}
+			r, err := loadIAMRevision(ctx, store, record.name)
+			if err != nil || store.writes != 1 || !r.timestamp().Equal(newer) || !r.Deleted {
+				t.Fatalf("newer deletion did not advance storage: writes=%d, revision=%+v, error=%v", store.writes, r, err)
+			}
+			if err := saveIAMRevision(withIAMReplicationTime(ctx, newer.Add(time.Minute)), store, record.name, record.new(false)); err != nil {
+				t.Fatalf("newer recreation rejected: %v", err)
+			}
+		})
+	}
+}
+
+// Run with both object storage and etcd through TestIAMRevocation*Lifecycle.
+// The object-store case uses the real peer RPC and deletion handler, so a
+// spurious notification actually destroys the parent instead of only counting it.
+func testIAMRevocationReplayAfterRecreation(ctx context.Context, t *testing.T, sys *IAMSys) {
+	t.Helper()
+	peer := &globalSiteReplicationSys
+	must := func(err error) {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	user := "heal-recreated-parent"
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-test-password", Status: madmin.AccountEnabled}
+	origin := UTCNow().Add(-time.Hour).Truncate(time.Millisecond)
+	deleted, recreated := origin.Add(time.Minute), origin.Add(3*time.Minute)
+	create := func(at time.Time) {
+		must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, at))
+	}
+	revoke := func(at time.Time) {
+		must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, at))
+	}
+	create(origin)
+	revoke(deleted)
+	create(recreated)
+	child, _, err := sys.NewServiceAccount(ctx, user, nil, newServiceAccountOpts{
+		accessKey: "heal-recreated-child", secretKey: "valid-service-password",
+	})
+	must(err)
+
+	tg, err := grid.SetupTestGrid(2)
+	must(err)
+	t.Cleanup(tg.Cleanup)
+	var deletes atomic.Int32
+	server := &peerRESTServer{}
+	must(deleteUserRPC.Register(tg.Managers[1], func(req *grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
+		deletes.Add(1)
+		return server.DeleteUserHandler(req)
+	}))
+	// Future user updates still use the normal peer reload notification.
+	must(loadUserRPC.Register(tg.Managers[1], server.LoadUserHandler))
+	host, err := xnet.ParseHost(strings.TrimPrefix(tg.Hosts[1], "http://"))
+	must(err)
+	previousNotifications := globalNotificationSys
+	globalNotificationSys = &NotificationSys{peerClients: []*peerRESTClient{{
+		host: host,
+		gridConn: func() *grid.Connection {
+			return tg.Managers[0].Connection(tg.Hosts[1])
+		},
+	}}}
+	t.Cleanup(func() { globalNotificationSys = previousNotifications })
+	assertLive := func(key string) {
+		t.Helper()
+		if _, ok := sys.GetUser(ctx, key); !ok {
+			t.Fatalf("live credential %s lost during deletion replay", key)
+		}
+	}
+	assertNoDelete := func() {
+		t.Helper()
+		if n := deletes.Load(); n != 0 {
+			t.Fatalf("retained revocation sent %d destructive sibling notifications", n)
+		}
+	}
+	for range 3 {
+		r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath(user, regUser))
+		must(err)
+		item, ok := iamDeletionItem(getUserIdentityPath(user, regUser), r)
+		if !ok || item.IAMUser == nil || !item.UpdatedAt.Equal(deleted) {
+			t.Fatal("recreated user lost its durable revocation replay")
+		}
+		must(peer.PeerIAMUserChangeHandler(ctx, item.IAMUser, item.UpdatedAt))
+		must(sys.store.LoadIAMCache(ctx, false))
+		assertLive(user)
+		assertLive(child.AccessKey)
+		assertNoDelete()
+	}
+
+	// A divergent site sends a previously unseen revocation between our old
+	// boundary and recreation. Retain it and revoke old children, but never
+	// turn it into an unversioned delete of the recreated parent.
+	delayed := deleted.Add(time.Minute)
+	revoke(delayed)
+	assertNoDelete()
+	must(sys.store.LoadIAMCache(ctx, false))
+	assertLive(user)
+	if _, ok := sys.GetUser(ctx, child.AccessKey); ok {
+		t.Fatal("child from before the delayed revocation remains usable")
+	}
+	r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath(user, regUser))
+	must(err)
+	if r.Deleted || !r.RevokedBefore.Equal(delayed) || !r.timestamp().Equal(recreated) {
+		t.Fatalf("retained revocation damaged the recreated identity: %+v", r)
+	}
+
+	// A genuinely newer deletion must still reach siblings and remove the
+	// parent plus credentials issued under its latest revocation boundary.
+	fresh, _, err := sys.NewServiceAccount(ctx, user, nil, newServiceAccountOpts{
+		accessKey: "heal-fresh-child", secretKey: "valid-service-password",
+	})
+	must(err)
+	latest := recreated.Add(time.Minute)
+	revoke(latest)
+	wantDeletes := int32(1)
+	if sys.HasWatcher() {
+		wantDeletes = 0
+	}
+	if n := deletes.Load(); n != wantDeletes {
+		t.Fatalf("new deletion notifications=%d, want %d", n, wantDeletes)
+	}
+	for _, key := range []string{user, fresh.AccessKey} {
+		if _, ok := sys.GetUser(ctx, key); ok {
+			t.Fatalf("newer deletion left credential %s usable", key)
+		}
+	}
+	// Exercise the actual sibling handler again against the already persisted
+	// tombstone. Its context has no revision; it must not re-stamp the record.
+	_, remoteErr := server.DeleteUserHandler(grid.NewMSSWith(map[string]string{peerRESTUser: user}))
+	if remoteErr != nil {
+		t.Fatal(remoteErr)
+	}
+	r, err = loadIAMRevision(ctx, sys.store, getUserIdentityPath(user, regUser))
+	must(err)
+	if !r.Deleted || !r.timestamp().Equal(latest) {
+		t.Fatalf("sibling re-stamped the tombstone: got %s, want %s", r.timestamp(), latest)
+	}
+	create(latest.Add(time.Minute))
+	_, remoteErr = server.DeleteUserHandler(grid.NewMSSWith(map[string]string{peerRESTUser: user}))
+	if remoteErr != nil {
+		t.Fatal(remoteErr)
+	}
+	assertLive(user)
+	revoke(latest)
+	must(sys.store.LoadIAMCache(ctx, false))
+	assertLive(user)
+}
+
+// Counts what a retained revocation actually sends to sibling nodes.
+func TestIAMRevocationRetainedReloadsSibling(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	disks, err := getRandomDisks(1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
+	if err != nil {
+		t.Fatal(err)
+	}
+	initAllSubsystems(ctx)
+	globalIAMSys.Init(ctx, obj, nil, 2*time.Second)
+	defer os.RemoveAll(disks[0])
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+
+	sys, peer := globalIAMSys, &globalSiteReplicationSys
+	must := func(err error) {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	user := "retained-parent"
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-test-password", Status: madmin.AccountEnabled}
+	origin := UTCNow().Add(-time.Hour).Truncate(time.Millisecond)
+	deleted, recreated := origin.Add(time.Minute), origin.Add(3*time.Minute)
+	create := func(at time.Time) {
+		must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, at))
+	}
+	revoke := func(at time.Time) {
+		must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, at))
+	}
+	create(origin)
+	revoke(deleted)
+	create(recreated)
+	child, _, err := sys.NewServiceAccount(ctx, user, nil, newServiceAccountOpts{
+		accessKey: "retained-child", secretKey: "valid-service-password",
+	})
+	must(err)
+
+	// A sibling shares persistent state but has an independent IAM cache.
+	sibling := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, sys.usersSysType)}
+	must(sibling.LoadIAMCache(ctx, false))
+	if _, ok := sibling.GetUser(child.AccessKey); !ok {
+		t.Fatal("sibling fixture did not load child")
+	}
+	tg, err := grid.SetupTestGrid(2)
+	must(err)
+	t.Cleanup(tg.Cleanup)
+	var deletes, loads atomic.Int32
+	server := &peerRESTServer{}
+	must(deleteUserRPC.Register(tg.Managers[1], func(r *grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
+		deletes.Add(1)
+		return server.DeleteUserHandler(r)
+	}))
+	must(loadUserRPC.Register(tg.Managers[1], func(r *grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
+		loads.Add(1)
+		// LoadUserHandler delegates to this same cache reload method.
+		if err := sibling.UserNotificationHandler(ctx, r.Get(peerRESTUser), regUser); err != nil {
+			return grid.NoPayload{}, grid.NewRemoteErr(err)
+		}
+		return grid.NoPayload{}, nil
+	}))
+	host, err := xnet.ParseHost(strings.TrimPrefix(tg.Hosts[1], "http://"))
+	must(err)
+	prev := globalNotificationSys
+	globalNotificationSys = &NotificationSys{peerClients: []*peerRESTClient{{
+		host:     host,
+		gridConn: func() *grid.Connection { return tg.Managers[0].Connection(tg.Hosts[1]) },
+	}}}
+	t.Cleanup(func() { globalNotificationSys = prev })
+
+	delayed := deleted.Add(time.Minute)
+	revoke(delayed)
+
+	t.Logf("sibling notifications after a retained revocation: destructive=%d reload=%d", deletes.Load(), loads.Load())
+	if deletes.Load() != 0 {
+		t.Errorf("destructive sibling delete sent: %d", deletes.Load())
+	}
+	if loads.Load() == 0 {
+		t.Errorf("retained revocation did not notify the sibling")
+	}
+	if _, ok := sibling.GetUser(child.AccessKey); ok {
+		t.Error("sibling still resolves revoked child")
+	}
+	if _, ok := sibling.GetUser(user); !ok {
+		t.Error("sibling lost live parent")
+	}
+	if _, ok := sys.store.GetUser(user); !ok {
+		t.Error("live parent lost")
+	}
+	if _, ok := sys.store.GetUser(child.AccessKey); ok {
+		t.Error("revoked child still resolves on the receiving node")
+	}
+}

cmd/iam-revocation_test.go

@@ -0,0 +1,535 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/auth"
+	etcd "go.etcd.io/etcd/client/v3"
+	"go.etcd.io/etcd/client/v3/namespace"
+)
+
+// Exercise the persisted IAM store and the same peer handler used by site heal.
+// A delete must survive a cache reload and an older create arriving afterwards.
+func TestIAMRevocationRejectsOfflineUser(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	obj, disk, err := prepareFS(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	user := "offline-revoked-user"
+	req := madmin.AddOrUpdateUserReq{SecretKey: "test-password-valid", Status: madmin.AccountEnabled}
+	created, err := globalIAMSys.CreateUser(ctx, user, req)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err = globalIAMSys.DeleteUser(ctx, user, false); err != nil {
+		t.Fatal(err)
+	}
+	if err = globalIAMSys.store.LoadIAMCache(ctx, false); err != nil {
+		t.Fatal(err)
+	}
+	if err = globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, created); err != nil {
+		t.Fatal(err)
+	}
+	if _, err = globalIAMSys.GetUserInfo(ctx, user); !errors.Is(err, errNoSuchUser) {
+		t.Fatalf("revoked user restored by old peer event: %v", err)
+	}
+}
+
+func TestIAMRevocationHealingContinuesAfterPeerRejectsDelete(t *testing.T) {
+	resetTestGlobals()
+	ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
+	defer cancel()
+	obj, disk, err := prepareFS(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-test-password", Status: madmin.AccountEnabled}
+	if _, err := globalIAMSys.CreateUser(ctx, "heal-sync", req); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := globalIAMSys.CreateUser(ctx, "heal-deleted", req); err != nil {
+		t.Fatal(err)
+	}
+	if err := globalIAMSys.DeleteUser(ctx, "heal-deleted", false); err != nil {
+		t.Fatal(err)
+	}
+	p, err := globalIAMSys.store.GetPolicy("readwrite")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := globalIAMSys.SetPolicy(ctx, "heal-new-policy", p); err != nil {
+		t.Fatal(err)
+	}
+	var liveUpdates atomic.Int32
+	peer := func(id string, rejectDelete bool) *httptest.Server {
+		return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			switch {
+			case strings.HasSuffix(r.URL.Path, "/metainfo"):
+				_ = json.NewEncoder(w).Encode(madmin.SRInfo{DeploymentID: id})
+			case r.URL.Path == "/minio/admin/v3/site-replication/peer/iam-revisions":
+				if r.Method == http.MethodGet {
+					_ = json.NewEncoder(w).Encode(iamRevisionStatus{Version: iamRevisionProtocol, Node: "node-1", Instance: id, Digest: "fixture"})
+					return
+				}
+				var batch iamRevisionBatch
+				if err := json.NewDecoder(r.Body).Decode(&batch); err != nil {
+					t.Error(err)
+					w.WriteHeader(http.StatusBadRequest)
+					return
+				}
+				for _, item := range batch.Items {
+					if rejectDelete && iamDeletionPath(item.SRIAMItem) != "" {
+						w.WriteHeader(http.StatusForbidden)
+						_, _ = w.Write([]byte(`{"Code":"AccessDenied","Message":"delete rejected"}`))
+						return
+					}
+					if id == "healthy" && item.Type == madmin.SRIAMItemPolicy && item.Name == "heal-new-policy" && len(item.Policy) > 0 {
+						liveUpdates.Add(1)
+					}
+				}
+				_ = json.NewEncoder(w).Encode(iamRevisionStatus{Version: iamRevisionProtocol, Node: "node-1", Instance: id, Digest: "fixture"})
+			default:
+				t.Errorf("unexpected peer request %s", r.URL.Path)
+				w.WriteHeader(http.StatusNotFound)
+			}
+		}))
+	}
+	healthy, rejected := peer("healthy", false), peer("rejected", true)
+	defer healthy.Close()
+	defer rejected.Close()
+	c := &SiteReplicationSys{enabled: true, state: srState{
+		ServiceAccountAccessKey: "heal-sync",
+		Peers: map[string]madmin.PeerInfo{
+			globalDeploymentID(): {Name: "local", DeploymentID: globalDeploymentID()},
+			"healthy":            {Name: "healthy", DeploymentID: "healthy", Endpoint: healthy.URL},
+			"rejected":           {Name: "rejected", DeploymentID: "rejected", Endpoint: rejected.URL},
+		},
+	}}
+	if err := c.healIAMSystem(ctx, obj); err == nil {
+		t.Fatal("deletion failure was not reported")
+	}
+	if liveUpdates.Load() == 0 {
+		t.Fatal("one peer rejecting a deletion blocked unrelated live IAM healing to a healthy peer")
+	}
+}
+
+func TestIAMRevocationLifecycle(t *testing.T) {
+	testIAMRevocationLifecycle(t, nil)
+}
+
+func TestIAMRevocationEtcdLifecycle(t *testing.T) {
+	endpoint := os.Getenv("SILO_TEST_IAM_REVOCATION_ETCD")
+	if endpoint == "" {
+		t.Skip("set SILO_TEST_IAM_REVOCATION_ETCD to a disposable etcd endpoint")
+	}
+	connection, err := etcd.New(etcd.Config{Endpoints: strings.Split(endpoint, ","), DialTimeout: 5 * time.Second})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer connection.Close()
+	// The facade borrows the connection's services. Close the owning client,
+	// not namespace.Watcher while IAM's canceled watch loop is winding down.
+	ctx, cancel := context.WithCancel(connection.Ctx())
+	defer cancel()
+	client := etcd.NewCtxClient(ctx, etcd.WithZapLogger(connection.GetLogger()))
+	prefix := fmt.Sprintf("/silo-revocation-test/%d/", time.Now().UnixNano())
+	client.KV = namespace.NewKV(connection.KV, prefix)
+	client.Watcher = namespace.NewWatcher(connection.Watcher, prefix)
+	client.Lease = connection.Lease
+	testIAMRevocationLifecycle(t, client)
+}
+
+func testIAMRevocationLifecycle(t *testing.T, client *etcd.Client) {
+	resetTestGlobals()
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	disks, err := getRandomDisks(1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	disk := disks[0]
+	obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
+	if err == nil {
+		initAllSubsystems(ctx)
+		globalIAMSys.Init(ctx, obj, client, 2*time.Second)
+	}
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer os.RemoveAll(disk)
+	defer obj.Shutdown(ctx)
+	defer resetTestGlobals()
+	sys, peer := globalIAMSys, &globalSiteReplicationSys
+	must := func(t *testing.T, err error) {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	reload := func(t *testing.T) { t.Helper(); must(t, sys.store.LoadIAMCache(ctx, false)) }
+	req := madmin.AddOrUpdateUserReq{SecretKey: "valid-test-password", Status: madmin.AccountEnabled}
+	origin := UTCNow().Add(-time.Hour).Truncate(time.Millisecond)
+	createUser := func(t *testing.T, name string) {
+		t.Helper()
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: name, UserReq: &req}, origin))
+	}
+	assertAbsent := func(t *testing.T, name string) {
+		t.Helper()
+		if _, ok := sys.GetUser(ctx, name); ok {
+			t.Fatalf("revoked credential %s is usable", name)
+		}
+	}
+
+	t.Run("replay after recreation", func(t *testing.T) {
+		testIAMRevocationReplayAfterRecreation(ctx, t, sys)
+	})
+
+	t.Run("origin timestamp and recreation", func(t *testing.T) {
+		name := "revocation-recreate"
+		createUser(t, name)
+		ui, ok := sys.store.GetUser(name)
+		if !ok || !ui.UpdatedAt.Equal(origin) {
+			t.Fatalf("origin time changed: %v", ui.UpdatedAt)
+		}
+		must(t, sys.DeleteUser(ctx, name, false))
+		reload(t)
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: name, UserReq: &req}, time.Time{}))
+		assertAbsent(t, name)
+		newTime := UTCNow().Add(time.Minute)
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: name, UserReq: &req}, newTime))
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: name, IsDeleteReq: true}, origin.Add(time.Second)))
+		reload(t)
+		ui, ok = sys.store.GetUser(name)
+		if !ok || !ui.UpdatedAt.Equal(newTime) || ui.RevokedBefore.IsZero() {
+			t.Fatalf("newer recreation lost, or deletion boundary missing: present=%v", ok)
+		}
+	})
+
+	t.Run("groups policies and mappings", func(t *testing.T) {
+		user, group, name := "revocation-member", "revocation-group", "revocation-policy"
+		createUser(t, user)
+		p, err := sys.store.GetPolicy("readwrite")
+		must(t, err)
+		must(t, peer.PeerAddPolicyHandler(ctx, name, &p, origin))
+		add := &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}
+		must(t, peer.PeerGroupInfoChangeHandler(ctx, add, origin))
+		for _, isGroup := range []bool{false, true} {
+			entity := user
+			if isGroup {
+				entity = group
+			}
+			mp := &madmin.SRPolicyMapping{UserOrGroup: entity, Policy: name, UserType: int(regUser), IsGroup: isGroup}
+			must(t, peer.PeerPolicyMappingHandler(ctx, mp, origin))
+			_, err = sys.PolicyDBSet(ctx, entity, "", regUser, isGroup)
+			must(t, err)
+			must(t, peer.PeerPolicyMappingHandler(ctx, mp, origin))
+			if _, ok := sys.store.GetMappedPolicy(entity, isGroup); ok {
+				t.Fatal("old grant restored")
+			}
+		}
+		// This receiver never saw the member-removal event preceding deletion.
+		must(t, peer.PeerGroupInfoChangeHandler(ctx, &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, IsRemove: true}}, UTCNow()))
+		must(t, sys.DeletePolicy(ctx, name, true))
+		reload(t)
+		must(t, peer.PeerGroupInfoChangeHandler(ctx, add, origin))
+		must(t, peer.PeerAddPolicyHandler(ctx, name, &p, origin))
+		if _, err = sys.GetGroupDescription(group); !errors.Is(err, errNoSuchGroup) {
+			t.Fatalf("group restored: %v", err)
+		}
+		if _, err = sys.store.GetPolicyDoc(name); !errors.Is(err, errNoSuchPolicy) {
+			t.Fatalf("policy restored: %v", err)
+		}
+		paths, err := sys.store.listIAMConfigPaths(ctx)
+		must(t, err)
+		found := make(map[string]bool)
+		for _, path := range paths {
+			r, err := loadIAMRevision(ctx, sys.store, path)
+			must(t, err)
+			if item, ok := iamDeletionItem(path, r); ok {
+				found[iamDeletionPath(item)] = true
+				if item.UpdatedAt.IsZero() {
+					t.Fatal("undated delete replay")
+				}
+			}
+		}
+		for _, path := range []string{getGroupInfoPath(group), getPolicyDocPath(name), getMappedPolicyPath(user, regUser, false), getMappedPolicyPath(group, regUser, true)} {
+			if !found[path] {
+				t.Errorf("deletion missing from heal: %s", path)
+			}
+		}
+	})
+
+	t.Run("parent revokes service accounts and STS", func(t *testing.T) {
+		parent := "revocation-parent"
+		createUser(t, parent)
+		svc, svcAt, err := sys.NewServiceAccount(withIAMReplicationTime(ctx, origin), parent, nil, newServiceAccountOpts{accessKey: "revocation-service", secretKey: "valid-service-password"})
+		must(t, err)
+		secret, err := getTokenSigningKey()
+		must(t, err)
+		sts, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
+		must(t, err)
+		sts.ParentUser = parent
+		_, err = sys.SetTempUser(withIAMReplicationTime(ctx, origin), sts.AccessKey, sts, "readwrite")
+		must(t, err)
+		must(t, sys.DeleteUser(ctx, parent, false))
+		reload(t)
+		assertAbsent(t, parent)
+		assertAbsent(t, svc.AccessKey)
+		assertAbsent(t, sts.AccessKey)
+		// Recreate the parent, then deliver old child events from the offline site.
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, UserReq: &req}, UTCNow()))
+		must(t, peer.PeerSvcAccChangeHandler(ctx, &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: parent, AccessKey: svc.AccessKey, SecretKey: svc.SecretKey}}, svcAt))
+		must(t, peer.PeerSTSAccHandler(ctx, &madmin.SRSTSCredential{AccessKey: sts.AccessKey, SecretKey: sts.SecretKey, ParentUser: parent, SessionToken: sts.SessionToken, ParentPolicyMapping: "readwrite"}, origin))
+		reload(t)
+		assertAbsent(t, svc.AccessKey)
+		assertAbsent(t, sts.AccessKey)
+		// A freshly issued credential is still supported after deliberate recreation.
+		_, _, err = sys.NewServiceAccount(ctx, parent, nil, newServiceAccountOpts{accessKey: "new-service", secretKey: "valid-service-password"})
+		must(t, err)
+		if _, ok := sys.GetUser(ctx, "new-service"); !ok {
+			t.Fatal("fresh service account rejected")
+		}
+	})
+
+	t.Run("delete before first create", func(t *testing.T) {
+		name := "revocation-unseen"
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: name, IsDeleteReq: true}, UTCNow()))
+		createUser(t, name)
+		assertAbsent(t, name)
+		svc := "unseen-service"
+		must(t, peer.PeerSvcAccChangeHandler(ctx, &madmin.SRSvcAccChange{Delete: &madmin.SRSvcAccDelete{AccessKey: svc}}, UTCNow()))
+		must(t, peer.PeerSvcAccChangeHandler(ctx, &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: "revocation-recreate", AccessKey: svc, SecretKey: "valid-service-password"}}, origin))
+		assertAbsent(t, svc)
+	})
+
+	t.Run("recreation arrives before revocation", func(t *testing.T) {
+		parent := "reordered-parent"
+		createUser(t, parent)
+		child, _, err := sys.NewServiceAccount(withIAMReplicationTime(ctx, origin), parent, nil, newServiceAccountOpts{accessKey: "reordered-child", secretKey: "valid-service-password"})
+		must(t, err)
+		newTime, deleteTime := UTCNow(), origin.Add(time.Minute)
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, UserReq: &req}, newTime))
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, IsDeleteReq: true}, deleteTime))
+		assertAbsent(t, child.AccessKey)
+		reload(t)
+		assertAbsent(t, child.AccessKey)
+		u, ok := sys.GetUser(ctx, parent)
+		if !ok || !u.UpdatedAt.Equal(newTime) || !u.RevokedBefore.Equal(deleteTime) {
+			t.Fatal("reordered revocation damaged the new parent or lost its boundary")
+		}
+		r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath(parent, regUser))
+		must(t, err)
+		item, ok := iamDeletionItem(getUserIdentityPath(parent, regUser), r)
+		if !ok || !item.UpdatedAt.Equal(deleteTime) {
+			t.Fatal("recreation erased deletion replay")
+		}
+	})
+
+	t.Run("user cleanup does not supersede group deletion", func(t *testing.T) {
+		user, group := "cascade-user", "cascade-group"
+		createUser(t, user)
+		must(t, peer.PeerGroupInfoChangeHandler(ctx, &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}, origin))
+		// On the origin site the group was removed before the user, but the
+		// recovering receiver processes those independent events in reverse.
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(2*time.Minute)))
+		must(t, peer.PeerGroupInfoChangeHandler(ctx, &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, IsRemove: true}}, origin.Add(time.Minute)))
+		reload(t)
+		if _, err := sys.GetGroupDescription(group); !errors.Is(err, errNoSuchGroup) {
+			t.Fatalf("deleted group survived reordered cleanup: %v", err)
+		}
+		groups, err := sys.ListGroups(ctx)
+		must(t, err)
+		for _, name := range groups {
+			if name == group {
+				t.Fatal("deleted group listed")
+			}
+		}
+	})
+
+	t.Run("parent revocation covers later updates to existing children", func(t *testing.T) {
+		parent, key := "late-update-parent", "late-update-child"
+		createUser(t, parent)
+		_, _, err := sys.NewServiceAccount(withIAMReplicationTime(ctx, origin), parent, nil, newServiceAccountOpts{accessKey: key, secretKey: "valid-service-password"})
+		must(t, err)
+		// This site missed the deletion and subsequently edited an old child.
+		_, err = sys.UpdateServiceAccount(withIAMReplicationTime(ctx, origin.Add(2*time.Minute)), key, updateServiceAccountOpts{description: "edited while the peer was offline"})
+		must(t, err)
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, IsDeleteReq: true}, origin.Add(time.Minute)))
+		assertAbsent(t, key)
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, UserReq: &req}, origin.Add(3*time.Minute)))
+		reload(t)
+		assertAbsent(t, key)
+	})
+
+	t.Run("old generation cannot return with a newer event timestamp", func(t *testing.T) {
+		parent := "generation-parent"
+		createUser(t, parent)
+		deleteTime := origin.Add(time.Minute)
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, IsDeleteReq: true}, deleteTime))
+		must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, UserReq: &req}, origin.Add(2*time.Minute)))
+		// Another offline site issued this child under the original parent,
+		// after this site's delete/recreate. Wall-clock ordering cannot identify it.
+		late := origin.Add(3 * time.Minute)
+		must(t, peer.PeerSvcAccChangeHandler(ctx, &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: parent, AccessKey: "old-gen-service", SecretKey: "valid-service-password"}}, late))
+		secret, err := getTokenSigningKey()
+		must(t, err)
+		sts, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
+		must(t, err)
+		must(t, peer.PeerSTSAccHandler(ctx, &madmin.SRSTSCredential{AccessKey: sts.AccessKey, SecretKey: sts.SecretKey, ParentUser: parent, SessionToken: sts.SessionToken}, late))
+		reload(t)
+		assertAbsent(t, "old-gen-service")
+		assertAbsent(t, sts.AccessKey)
+		// A local issuer knows the new boundary and signs it into both kinds
+		// of child. Untrusted inherited claims cannot select that boundary.
+		child, _, err := sys.NewServiceAccount(ctx, parent, nil, newServiceAccountOpts{accessKey: "new-gen-service", secretKey: "valid-service-password", claims: map[string]any{iamParentRevocationClaim: "forged"}})
+		must(t, err)
+		newClaims := map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}
+		must(t, setIAMParentRevocationClaim(ctx, sys.store, parent, newClaims))
+		fresh, err := auth.GetNewCredentialsWithMetadata(newClaims, secret)
+		must(t, err)
+		fresh.ParentUser = parent
+		_, err = sys.SetTempUser(ctx, fresh.AccessKey, fresh, "")
+		must(t, err)
+		reload(t)
+		// A periodic reload retains the STS cache. Explicitly clear it to
+		// exercise the cold credential load performed after process restart.
+		cache := sys.store.lock()
+		cache.iamSTSAccountsMap = make(map[string]UserIdentity)
+		sys.store.unlock()
+		for _, key := range []string{child.AccessKey, fresh.AccessKey} {
+			u, ok := sys.GetUser(ctx, key)
+			if !ok || !iamCredentialSurvivesRevocation(u.Credentials, deleteTime) {
+				t.Fatalf("new-generation credential %s rejected", key)
+			}
+		}
+	})
+
+	t.Run("late revocation preserves proven new-generation children", func(t *testing.T) {
+		for _, recreateFirst := range []bool{false, true} {
+			parent := fmt.Sprintf("gen-parent-%t", recreateFirst)
+			createUser(t, parent)
+			deleteTime, createTime := origin.Add(time.Minute), origin.Add(2*time.Minute)
+			if recreateFirst {
+				must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, UserReq: &req}, createTime))
+			}
+			key := fmt.Sprintf("gen-child-%t", recreateFirst)
+			must(t, peer.PeerSvcAccChangeHandler(ctx, &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: parent, AccessKey: key, SecretKey: "valid-service-password", Claims: map[string]any{iamParentRevocationClaim: deleteTime.Format(time.RFC3339Nano)}}}, createTime.Add(time.Second)))
+			must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, IsDeleteReq: true}, deleteTime))
+			if !recreateFirst {
+				assertAbsent(t, key)
+				must(t, peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: parent, UserReq: &req}, createTime))
+			}
+			reload(t)
+			if _, ok := sys.GetUser(ctx, key); !ok {
+				t.Fatal("late revocation deleted a child issued by the recreated parent")
+			}
+		}
+	})
+
+	t.Run("cold loading preserves site-signed STS", func(t *testing.T) {
+		parent := "cold-sts-parent"
+		createUser(t, parent)
+		secret := "site-signing-key-valid"
+		_, _, err := sys.NewServiceAccount(ctx, globalActiveCred.AccessKey, nil, newServiceAccountOpts{
+			accessKey: siteReplicatorSvcAcc, secretKey: secret, allowSiteReplicatorAccount: true,
+		})
+		must(t, err)
+		setReplication := func(enabled bool) {
+			globalSiteReplicationSys.Lock()
+			globalSiteReplicationSys.enabled = enabled
+			globalSiteReplicationSys.Unlock()
+			globalSiteReplicatorCred.Set("")
+		}
+		setReplication(true)
+		defer setReplication(false)
+		cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
+		must(t, err)
+		cred.ParentUser = parent
+		_, err = sys.SetTempUser(ctx, cred.AccessKey, cred, "")
+		must(t, err)
+		// IAM can load before the site replication manager during startup.
+		// A signing key that is not available yet must not delete live tokens.
+		setReplication(false)
+		for range 3 {
+			unverified := make(map[string]UserIdentity)
+			_ = sys.store.loadUser(ctx, cred.AccessKey, stsUser, unverified)
+			if _, ok := unverified[cred.AccessKey]; ok {
+				t.Fatal("accepted STS before the signing key became available")
+			}
+		}
+		r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath(cred.AccessKey, stsUser))
+		must(t, err)
+		if r.Credentials.SessionToken == "" {
+			t.Fatal("cold IAM load physically deleted a non-expired site-signed STS credential")
+		}
+		setReplication(true)
+		loaded := make(map[string]UserIdentity)
+		must(t, sys.store.loadUser(ctx, cred.AccessKey, stsUser, loaded))
+		if _, ok := loaded[cred.AccessKey]; !ok {
+			t.Fatal("STS credential did not recover when the signing key became available")
+		}
+	})
+
+	t.Run("unverifiable STS stay denied and expired STS are removed", func(t *testing.T) {
+		parent := "invalid-sts-parent"
+		createUser(t, parent)
+		// Keep the etcd watcher from cleaning half of the fixture before the
+		// second record is seeded; this subtest exercises the loader directly.
+		sys.store.lock()
+		defer sys.store.unlock()
+		for _, expired := range []bool{false, true} {
+			cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, "unavailable-test-signing-key")
+			must(t, err)
+			cred.ParentUser = parent
+			if expired {
+				cred.Expiration = UTCNow().Add(-time.Minute)
+			}
+			identityPath := getUserIdentityPath(cred.AccessKey, stsUser)
+			mappingPath := getMappedPolicyPath(cred.AccessKey, stsUser, false)
+			// Seed disk directly to exercise loading, including existing records
+			// whose key is unknown. The write API should not accept such tokens.
+			must(t, sys.store.saveIAMConfig(ctx, &UserIdentity{Version: 1, Credentials: cred, UpdatedAt: UTCNow()}, identityPath))
+			must(t, sys.store.saveIAMConfig(ctx, &MappedPolicy{Version: 1, Policies: "readwrite"}, mappingPath))
+			loaded := make(map[string]UserIdentity)
+			_ = sys.store.loadUser(ctx, cred.AccessKey, stsUser, loaded)
+			if _, ok := loaded[cred.AccessKey]; ok {
+				t.Fatalf("invalid STS accepted, expired=%t", expired)
+			}
+			for _, path := range []string{identityPath, mappingPath} {
+				var record map[string]any
+				err := sys.store.loadIAMConfig(ctx, &record, path)
+				if expired {
+					if !errors.Is(err, errConfigNotFound) {
+						t.Fatalf("expired STS data not cleaned up at %s: %v", path, err)
+					}
+				} else {
+					must(t, err)
+				}
+			}
+		}
+	})
+}

cmd/iam-service-recreation_test.go

@@ -0,0 +1,239 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/auth"
+)
+
+// The receiver missed a deletion and still has the previous service key.
+// A later full snapshot must replace it, including when the owner changed.
+func TestIAMServiceAccountRecreation(t *testing.T) {
+	for _, backend := range []string{"object", "etcd"} {
+		t.Run(backend, func(t *testing.T) {
+			ctx, sys, _ := prepareIAMRevisionFixture(t, backend)
+			origin := UTCNow().Add(-time.Hour)
+			for _, parent := range []string{"old-owner", "new-owner"} {
+				_, err := sys.CreateUser(withIAMReplicationTime(ctx, origin), parent, madmin.AddOrUpdateUserReq{SecretKey: "valid-owner-password", Status: madmin.AccountEnabled})
+				mustIAM(t, err)
+			}
+			const key = "reusable-service"
+			old := &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: "old-owner", AccessKey: key, SecretKey: "old-service-password"}}
+			mustIAM(t, globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, old, origin))
+			oldIdentity, _ := sys.store.GetUser(key)
+			_, err := sys.PolicyDBSet(withIAMReplicationTime(ctx, origin), key, "readwrite", svcUser, false)
+			mustIAM(t, err)
+			boundary, newer := origin.Add(time.Minute), origin.Add(2*time.Minute)
+			fresh := iamReplicationItem{SRIAMItem: madmin.SRIAMItem{Type: madmin.SRIAMItemSvcAcc, UpdatedAt: newer, SvcAccChange: &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: "new-owner", AccessKey: key, SecretKey: "new-service-password", Status: auth.AccountOff}}}, RevokedBefore: boundary}
+			mustIAM(t, applyIAMReplicationItem(ctx, fresh))
+			// A sibling may have missed the notification of the committed
+			// replacement. An equal-version retry must refresh that cache too.
+			staleCache := sys.store.lock()
+			staleCache.iamUsersMap[key] = oldIdentity
+			sys.store.unlock()
+			mustIAM(t, applyIAMReplicationItem(ctx, fresh)) // duplicate delivery is acknowledged
+			if current, _ := sys.store.GetUser(key); current.Credentials.SecretKey != fresh.SvcAccChange.Create.SecretKey {
+				t.Fatal("duplicate snapshot acknowledged without refreshing the stale cache")
+			}
+			mustIAM(t, globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, old, origin))
+			mustIAM(t, globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, &madmin.SRSvcAccChange{Delete: &madmin.SRSvcAccDelete{AccessKey: key}}, boundary))
+			mustIAM(t, sys.store.LoadIAMCache(ctx, false))
+			u, ok := sys.store.GetUser(key)
+			if !ok || u.Credentials.SecretKey != fresh.SvcAccChange.Create.SecretKey || u.Credentials.ParentUser != "new-owner" || u.Credentials.Status != auth.AccountOff || !u.UpdatedAt.Equal(newer) || !u.RevokedBefore.Equal(boundary) {
+				t.Fatal("recreation did not retain the new identity, disabled status, source version and revocation")
+			}
+			if _, ok := sys.GetUser(ctx, key); ok {
+				t.Fatal("replicated disabled service can authenticate")
+			}
+			cache := sys.store.rlock()
+			_, mapped := cache.cachedMappedPolicy(key, svcUser, false)
+			sys.store.runlock()
+			if mapped {
+				t.Fatal("recreated service inherited an older mapping")
+			}
+			_, err = sys.PolicyDBSet(withIAMReplicationTime(ctx, origin), key, "readwrite", svcUser, false)
+			if !errors.Is(err, errIAMStaleUpdate) {
+				t.Fatalf("old service mapping replay was accepted: %v", err)
+			}
+			_, _, err = sys.NewServiceAccount(ctx, "new-owner", nil, newServiceAccountOpts{accessKey: key, secretKey: "local-service-password"})
+			if !errors.Is(err, errIAMServiceAccountNotAllowed) {
+				t.Fatalf("local duplicate creation must remain rejected: %v", err)
+			}
+			// Outbound snapshots must carry the retained service boundary too.
+			out, err := globalSiteReplicationSys.replicationItem(ctx, fresh.SRIAMItem)
+			mustIAM(t, err)
+			if !out.RevokedBefore.Equal(boundary) {
+				t.Fatal("outbound service snapshot lost its revocation")
+			}
+		})
+	}
+}
+
+func TestIAMServiceAccountReplicationRejectsOtherCredentialKinds(t *testing.T) {
+	ctx, sys, _ := prepareIAMRevisionFixture(t)
+	_, err := sys.CreateUser(ctx, "builtin-collision", madmin.AddOrUpdateUserReq{SecretKey: "valid-user-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	secret, err := getTokenSigningKey()
+	mustIAM(t, err)
+	token, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: "builtin-collision"}, secret)
+	mustIAM(t, err)
+	token.ParentUser = "builtin-collision"
+	_, err = sys.SetTempUser(ctx, token.AccessKey, token, "")
+	mustIAM(t, err)
+	for _, key := range []string{"builtin-collision", token.AccessKey} {
+		_, _, err := sys.NewServiceAccount(withIAMReplicationTime(ctx, UTCNow().Add(time.Minute)), "another-owner", nil, newServiceAccountOpts{accessKey: key, secretKey: "valid-service-password"})
+		if !errors.Is(err, errIAMServiceAccountNotAllowed) {
+			t.Fatalf("service replication replaced another credential kind: %v", err)
+		}
+	}
+}
+
+// SR configuration can be temporarily unreadable even though a service token
+// is signed with its own valid secret. Do not acknowledge a failed cache load.
+func TestIAMServiceAccountRetryReportsClaimLoadFailure(t *testing.T) {
+	ctx, sys, _ := prepareIAMRevisionFixture(t)
+	globalSiteReplicatorCred.RLock()
+	previousSigningKey := globalSiteReplicatorCred.secretKey
+	globalSiteReplicatorCred.RUnlock()
+	globalSiteReplicatorCred.Set("")
+	t.Cleanup(func() { globalSiteReplicatorCred.Set(previousSigningKey) })
+	_, err := sys.CreateUser(ctx, "retry-owner", madmin.AddOrUpdateUserReq{SecretKey: "valid-owner-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	opts := newServiceAccountOpts{accessKey: "retry-service", secretKey: "valid-service-password"}
+	_, _, err = sys.NewServiceAccount(ctx, "retry-owner", nil, opts)
+	mustIAM(t, err)
+	old, _ := sys.store.GetUser(opts.accessKey)
+	opts.secretKey = "replacement-service-password"
+	at, err := sys.UpdateServiceAccount(ctx, opts.accessKey, updateServiceAccountOpts{secretKey: opts.secretKey})
+	mustIAM(t, err)
+	cache := sys.store.lock()
+	cache.iamUsersMap[opts.accessKey] = old // Missed sibling notification.
+	sys.store.unlock()
+	globalSiteReplicationSys.Lock()
+	globalSiteReplicationSys.enabled = true // No site-replicator credential is installed.
+	globalSiteReplicationSys.Unlock()
+	_, _, err = sys.NewServiceAccount(withIAMReplicationTime(ctx, at), "retry-owner", nil, opts)
+	if err == nil {
+		t.Fatal("acknowledged service retry despite failed claims loading")
+	}
+	if _, ok := sys.store.GetUser(opts.accessKey); ok {
+		t.Fatal("failed cache refresh retained the superseded service secret")
+	}
+	globalSiteReplicationSys.Lock()
+	globalSiteReplicationSys.enabled = false
+	globalSiteReplicationSys.Unlock()
+	_, _, err = sys.NewServiceAccount(withIAMReplicationTime(ctx, at), "retry-owner", nil, opts)
+	mustIAM(t, err)
+}
+
+// A delayed snapshot still has its original absolute expiration. Reapplying
+// the local minimum issuance lifetime would leave the old unexpired key alive.
+func TestIAMServiceAccountReplicationPreservesExpiration(t *testing.T) {
+	for _, backend := range []string{"object", "etcd"} {
+		for _, action := range []string{"create", "update"} {
+			for _, expired := range []bool{false, true} {
+				name := backend + "/" + action + "/near_expiry"
+				if expired {
+					name = backend + "/" + action + "/expired"
+				}
+				t.Run(name, func(t *testing.T) {
+					ctx, sys, _ := prepareIAMRevisionFixture(t, backend)
+					origin := UTCNow().Add(-time.Hour)
+					_, err := sys.CreateUser(ctx, "expiry-owner", madmin.AddOrUpdateUserReq{SecretKey: "valid-owner-password", Status: madmin.AccountEnabled})
+					mustIAM(t, err)
+					old := &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: "expiry-owner", AccessKey: "expiry-service", SecretKey: "old-service-password"}}
+					mustIAM(t, globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, old, origin))
+					expires := UTCNow().Add(time.Minute)
+					if expired {
+						expires = UTCNow().Add(-time.Minute)
+					}
+					change := &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{Parent: "expiry-owner", AccessKey: "expiry-service", SecretKey: "new-service-password", Expiration: &expires}}
+					if action == "update" {
+						change = &madmin.SRSvcAccChange{Update: &madmin.SRSvcAccUpdate{AccessKey: "expiry-service", SecretKey: "new-service-password", Expiration: &expires}}
+					}
+					mustIAM(t, globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, change, origin.Add(2*time.Minute)))
+					u, ok := sys.store.GetUser("expiry-service")
+					if !ok || u.Credentials.SecretKey != "new-service-password" || !u.Credentials.Expiration.Equal(expires) {
+						t.Fatal("delayed snapshot lost its new secret or absolute expiration")
+					}
+					_, allowed := sys.GetUser(ctx, "expiry-service")
+					if allowed == expired {
+						t.Fatal("credential validity disagrees with its absolute expiration")
+					}
+					mustIAM(t, sys.store.LoadIAMCache(ctx, false))
+					mustIAM(t, globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, old, origin))
+					r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath("expiry-service", svcUser))
+					mustIAM(t, err)
+					if r.Credentials.SecretKey == old.Create.SecretKey || (expired && !r.Deleted) {
+						t.Fatal("old non-expiring credential returned after reload")
+					}
+					_, _, err = sys.NewServiceAccount(ctx, "expiry-owner", nil, newServiceAccountOpts{accessKey: "local-expiry", secretKey: "valid-service-password", expiration: &expires})
+					if !errors.Is(err, errInvalidSvcAcctExpiration) {
+						t.Fatalf("local issuance lifetime check changed: %v", err)
+					}
+				})
+			}
+		}
+	}
+}
+
+// Status-only summaries intentionally omit secrets. Different revisions must
+// still trigger live healing, and disabled identities must be eligible sources.
+func TestIAMServiceAccountHealingNewerSnapshot(t *testing.T) {
+	ctx, sys, obj := prepareIAMRevisionFixture(t)
+	origin := UTCNow().Add(-time.Hour)
+	_, err := sys.CreateUser(ctx, "heal-svc-owner", madmin.AddOrUpdateUserReq{SecretKey: "valid-owner-password", Status: madmin.AccountEnabled})
+	mustIAM(t, err)
+	_, _, err = sys.NewServiceAccount(withIAMReplicationTime(ctx, origin), "heal-svc-owner", nil, newServiceAccountOpts{accessKey: "heal-service", secretKey: "valid-service-password"})
+	mustIAM(t, err)
+	at, err := sys.UpdateServiceAccount(ctx, "heal-service", updateServiceAccountOpts{status: auth.AccountOff})
+	mustIAM(t, err)
+	var sent atomic.Int32
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/minio/health/live" {
+			return
+		}
+		if r.URL.Path != "/minio/admin/v3/site-replication/peer/iam-revisions" || r.Method != http.MethodPut {
+			t.Errorf("unexpected request %s %s", r.Method, r.URL.Path)
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+		var batch iamRevisionBatch
+		if err := json.NewDecoder(r.Body).Decode(&batch); err != nil {
+			t.Error(err)
+			w.WriteHeader(http.StatusBadRequest)
+			return
+		}
+		for _, item := range batch.Items {
+			if item.SvcAccChange != nil && item.SvcAccChange.Create != nil && item.SvcAccChange.Create.AccessKey == "heal-service" && item.SvcAccChange.Create.Status == auth.AccountOff && item.UpdatedAt.Equal(at) {
+				sent.Add(1)
+			}
+		}
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "node", Instance: "boot", Digest: "fixture"}})
+	}))
+	defer server.Close()
+	peers := map[string]madmin.PeerInfo{globalDeploymentID(): {Name: "local", DeploymentID: globalDeploymentID()}, "remote": {Name: "remote", DeploymentID: "remote", Endpoint: server.URL}}
+	c := &SiteReplicationSys{enabled: true, state: srState{ServiceAccountAccessKey: "heal-svc-owner", Peers: peers}}
+	local := madmin.UserInfo{Status: madmin.AccountStatus(auth.AccountOff), UpdatedAt: at}
+	remote := local
+	remote.UpdatedAt = origin
+	if isUserInfoReplicated(2, 2, []madmin.UserInfo{local, remote}) {
+		t.Fatal("status-only summaries concealed different service revisions")
+	}
+	info := srStatusInfo{Sites: peers, UserStats: map[string]map[string]srUserStatsSummary{"heal-service": {globalDeploymentID(): {userInfo: srUserInfo{UserInfo: local}}, "remote": {SRUserStatsSummary: madmin.SRUserStatsSummary{UserInfoMismatch: true}, userInfo: srUserInfo{UserInfo: remote}}}}}
+	mustIAM(t, c.healUsers(ctx, obj, "heal-service", info))
+	if sent.Load() == 0 {
+		t.Fatal("disabled newer service snapshot was not healed")
+	}
+}

cmd/iam.go

@@ -162,6 +162,15 @@ func (sys *IAMSys) LoadUser(ctx context.Context, objAPI ObjectLayer, accessKey s
 	return sys.store.UserNotificationHandler(ctx, accessKey, userType)
 }
 
+// LoadUserAfterDelete reloads a parent's identity and cached dependents after a
+// sibling committed a deletion. Each record may already have been recreated.
+func (sys *IAMSys) LoadUserAfterDelete(ctx context.Context, accessKey string) error {
+	if !sys.Initialized() {
+		return errServerNotInitialized
+	}
+	return sys.store.UserDeletionNotificationHandler(ctx, accessKey)
+}
+
 // LoadServiceAccount - reloads a specific service account from backend disks or etcd.
 func (sys *IAMSys) LoadServiceAccount(ctx context.Context, accessKey string) error {
 	if !sys.Initialized() {
@@ -596,10 +605,22 @@ func (sys *IAMSys) DeletePolicy(ctx context.Context, policyName string, notifyPe
 		return errServerNotInitialized
 	}
 
-	for _, v := range policy.DefaultPolicies {
+	if _, replicated := iamReplicationTime(ctx); !replicated && notifyPeers {
-		if v.Name == policyName {
+		for _, v := range policy.DefaultPolicies {
-			if err := checkConfig(ctx, globalObjectAPI, getPolicyDocPath(policyName)); err != nil && err == errConfigNotFound {
+			if v.Name == policyName {
-				return fmt.Errorf("inbuilt policy `%s` not allowed to be deleted", policyName)
+				var err error
+				if objectStore, ok := sys.store.IAMStorageAPI.(*IAMObjectStore); ok {
+					err = checkConfig(ctx, objectStore.objAPI, getPolicyDocPath(policyName))
+				} else {
+					var r iamRevision
+					err = sys.store.loadIAMConfig(ctx, &r, getPolicyDocPath(policyName))
+				}
+				if errors.Is(err, errConfigNotFound) {
+					return fmt.Errorf("inbuilt policy `%s` not allowed to be deleted", policyName)
+				}
+				if err != nil {
+					return err
+				}
 			}
 		}
 	}
@@ -705,20 +726,30 @@ func (sys *IAMSys) DeleteUser(ctx context.Context, accessKey string, notifyPeers
 		return errServerNotInitialized
 	}
 
-	if err := sys.store.DeleteUser(ctx, accessKey, regUser); err != nil {
+	err := sys.store.DeleteUser(ctx, accessKey, regUser)
+	var cleanupErr *iamCommittedCleanupError
+	retained := errors.Is(err, errIAMRevocationRetained)
+	if errors.As(err, &cleanupErr) {
+		retained = cleanupErr.retained
+	} else if err != nil && !retained {
 		return err
 	}
-
+	// Publish the committed state even when dependent cleanup must be retried.
-	// Notify all other MinIO peers to delete user.
 	if notifyPeers && !sys.HasWatcher() {
-		for _, nerr := range globalNotificationSys.DeleteUser(ctx, accessKey) {
+		if retained {
-			if nerr.Err != nil {
+			sys.notifyForUser(ctx, accessKey, false)
-				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
+		} else {
-				iamLogIf(ctx, nerr.Err)
+			for _, nerr := range globalNotificationSys.DeleteUser(ctx, accessKey) {
+				if nerr.Err != nil {
+					logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
+					iamLogIf(ctx, nerr.Err)
+				}
 			}
 		}
 	}
-
+	if cleanupErr != nil {
+		return cleanupErr
+	}
 	return nil
 }
 
@@ -1052,6 +1083,7 @@ type newServiceAccountOpts struct {
 	sessionPolicy              *policy.Policy
 	accessKey                  string
 	secretKey                  string
+	status                     string // Used by replication snapshots; local creates default to enabled.
 	name, description          string
 	expiration                 *time.Time
 	allowSiteReplicatorAccount bool // allow creating internal service account for site-replication.
@@ -1116,6 +1148,11 @@ func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, gro
 			m[k] = v
 		}
 	}
+	if _, replicated := iamReplicationTime(ctx); !replicated {
+		if err := setIAMParentRevocationClaim(ctx, sys.store, parentUser, m); err != nil {
+			return auth.Credentials{}, time.Time{}, err
+		}
+	}
 
 	var accessKey, secretKey string
 	var err error
@@ -1134,12 +1171,19 @@ func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, gro
 	cred.ParentUser = parentUser
 	cred.Groups = groups
 	cred.Status = string(auth.AccountOn)
+	switch opts.status {
+	case "", auth.AccountOn, string(madmin.AccountEnabled):
+	case auth.AccountOff, string(madmin.AccountDisabled):
+		cred.Status = auth.AccountOff
+	default:
+		return auth.Credentials{}, time.Time{}, errInvalidArgument
+	}
 	cred.Name = opts.name
 	cred.Description = opts.description
 
 	if opts.expiration != nil {
 		expirationInUTC := opts.expiration.UTC()
-		if err := validateSvcExpirationInUTC(expirationInUTC); err != nil {
+		if err := validateSvcExpirationInUTC(ctx, expirationInUTC); err != nil {
 			return auth.Credentials{}, time.Time{}, err
 		}
 		cred.Expiration = expirationInUTC
@@ -1370,7 +1414,7 @@ func (sys *IAMSys) DeleteServiceAccount(ctx context.Context, accessKey string, n
 	}
 
 	sa, ok := sys.store.GetUser(accessKey)
-	if !ok || !sa.Credentials.IsServiceAccount() {
+	if _, replicated := iamReplicationTime(ctx); (!ok || !sa.Credentials.IsServiceAccount()) && !replicated {
 		return nil
 	}
 
@@ -1474,8 +1518,8 @@ func (sys *IAMSys) purgeExpiredCredentialsForExternalSSO(ctx context.Context) {
 		}
 	}
 
-	// We ignore any errors
+	// Keep failed revocations visible so the next purge can retry.
-	_ = sys.store.DeleteUsers(ctx, expiredUsers)
+	iamLogIf(ctx, sys.store.DeleteUsers(ctx, expiredUsers))
 }
 
 // purgeExpiredCredentialsForLDAP - validates if local credentials are still
@@ -1503,8 +1547,8 @@ func (sys *IAMSys) purgeExpiredCredentialsForLDAP(ctx context.Context) {
 		return
 	}
 
-	// We ignore any errors
+	// Keep failed revocations visible so the next purge can retry.
-	_ = sys.store.DeleteUsers(ctx, expiredUsers)
+	iamLogIf(ctx, sys.store.DeleteUsers(ctx, expiredUsers))
 }
 
 // updateGroupMembershipsForLDAP - updates the list of groups associated with the credential.
@@ -1925,12 +1969,12 @@ func (sys *IAMSys) RemoveUsersFromGroup(ctx context.Context, group string, membe
 	}
 
 	updatedAt, err = sys.store.RemoveUsersFromGroup(ctx, group, members)
-	if err != nil {
+	var cleanupErr *iamCommittedCleanupError
+	if err != nil && !errors.As(err, &cleanupErr) {
 		return updatedAt, err
 	}
-
 	sys.notifyForGroup(ctx, group)
-	return updatedAt, nil
+	return updatedAt, err
 }
 
 // SetGroupStatus - enable/disabled a group

cmd/ilm-access-move_test.go

@@ -1,529 +0,0 @@
-// Copyright 2026 PGSTY contributors.
-// SPDX-License-Identifier: AGPL-3.0-or-later
-
-package cmd
-
-import (
-	"bytes"
-	"context"
-	"crypto/md5"
-	"encoding/base64"
-	"encoding/xml"
-	"errors"
-	"fmt"
-	"io"
-	"maps"
-	"net/http"
-	"net/http/httptest"
-	"sync"
-	"testing"
-	"time"
-
-	"github.com/minio/minio/internal/dsync"
-	"github.com/minio/minio/internal/hash"
-	xhttp "github.com/minio/minio/internal/http"
-)
-
-func accessMovePools(t *testing.T) (*erasureServerPools, string) {
-	t.Helper()
-	ctx, cancel := context.WithCancel(t.Context())
-	dirs, err := getRandomDisks(32)
-	if err != nil {
-		t.Fatal(err)
-	}
-	endpoints := mustGetPoolEndpoints(0, dirs[:16]...)
-	endpoints = append(endpoints, mustGetPoolEndpoints(1, dirs[16:]...)...)
-	obj, _, err := initObjectLayer(ctx, endpoints)
-	if err != nil {
-		cancel()
-		removeRoots(dirs)
-		t.Fatal(err)
-	}
-	z := obj.(*erasureServerPools)
-	previous := newObjectLayerFn()
-	setObjectLayer(z)
-	t.Cleanup(func() { cancel(); z.Shutdown(context.Background()); removeRoots(dirs); setObjectLayer(previous) })
-	bucket := "access-move-test"
-	if err := z.MakeBucket(ctx, bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
-		t.Fatal(err)
-	}
-	return z, bucket
-}
-
-func putAccessMoveVersion(t *testing.T, z *erasureServerPools, bucket, object string, pool int, body string, moved bool) ObjectInfo {
-	t.Helper()
-	metadata := map[string]string{"test-value": body}
-	if moved {
-		metadata[accessTierMetadataKey] = accessTierStamp(pool, time.Now().UnixNano())
-	}
-	cs := hash.NewChecksumFromData(hash.ChecksumCRC32C, []byte(body))
-	oi, err := z.serverPools[pool].PutObject(t.Context(), bucket, object,
-		mustGetPutObjReader(t, bytes.NewBufferString(body), int64(len(body)), "", ""),
-		ObjectOptions{Versioned: true, UserDefined: metadata, WantChecksum: cs})
-	if err != nil {
-		t.Fatal(err)
-	}
-	return oi
-}
-
-func assertAccessMoveVersion(t *testing.T, z *erasureServerPools, bucket, object string, pool int, want ObjectInfo, body string) {
-	t.Helper()
-	gr, err := z.serverPools[pool].GetObjectNInfo(t.Context(), bucket, object, nil, http.Header{}, ObjectOptions{VersionID: want.VersionID})
-	if err != nil {
-		t.Errorf("pool %d version %s: %v", pool, want.VersionID, err)
-		return
-	}
-	got, err := io.ReadAll(gr)
-	gr.Close()
-	if err != nil || string(got) != body {
-		t.Errorf("pool %d payload = %q, err = %v, want %q", pool, got, err, body)
-	}
-	if gr.ObjInfo.ETag != want.ETag || !gr.ObjInfo.ModTime.Equal(want.ModTime) {
-		t.Error("move changed ETag or modification time")
-	}
-	if len(want.Checksum) == 0 {
-		t.Fatal("fixture has no checksum")
-	}
-	if !bytes.Equal(gr.ObjInfo.Checksum, want.Checksum) {
-		t.Error("move changed or dropped the stored checksum")
-	}
-	if gr.ObjInfo.UserDefined["test-value"] != body {
-		t.Error("move changed user metadata")
-	}
-}
-
-func TestAccessMoveVersionStack(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "versions"
-	old := putAccessMoveVersion(t, z, bucket, object, 1, "old", false)
-	dm, err := z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true})
-	if err != nil {
-		t.Fatal(err)
-	}
-	latest := putAccessMoveVersion(t, z, bucket, object, 1, "new", false)
-	for _, pair := range [][2]int{{1, 0}, {0, 1}} {
-		if _, err := moveObjectPool(t.Context(), z, bucket, object, pair[0], pair[1], nil); err != nil {
-			t.Fatal(err)
-		}
-		assertAccessMoveVersion(t, z, bucket, object, pair[1], old, "old")
-		assertAccessMoveVersion(t, z, bucket, object, pair[1], latest, "new")
-		versions, err := accessObjectVersions(t.Context(), z, pair[1], bucket, object)
-		if err != nil {
-			t.Fatal(err)
-		}
-		found := false
-		for _, version := range versions {
-			if version.VersionID == dm.VersionID && version.Deleted {
-				found = true
-			}
-		}
-		if !found || len(versions) != 3 {
-			t.Errorf("move lost a version or delete marker: %+v", versions)
-		}
-		if _, err := z.serverPools[pair[0]].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); !isErrObjectNotFound(err) {
-			t.Errorf("source remains after completed move: %v", err)
-		}
-	}
-}
-
-func TestAccessMovePreservesNestedObject(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	parent := putAccessMoveVersion(t, z, bucket, "parent", 1, "parent", false)
-	child := putAccessMoveVersion(t, z, bucket, "parent/child", 1, "child", false)
-	if _, err := moveObjectPool(t.Context(), z, bucket, "parent", 1, 0, nil); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, "parent", 0, parent, "parent")
-	assertAccessMoveVersion(t, z, bucket, "parent/child", 1, child, "child")
-}
-
-func TestAccessMoveNullVersion(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object, body = "null-version", "unversioned"
-	// A null version can remain in a bucket after versioning is enabled.
-	oi, err := z.serverPools[1].PutObject(t.Context(), bucket, object,
-		mustGetPutObjReader(t, bytes.NewBufferString(body), int64(len(body)), "", ""), ObjectOptions{
-			UserDefined:  map[string]string{"test-value": body},
-			WantChecksum: hash.NewChecksumFromData(hash.ChecksumCRC32C, []byte(body)),
-		})
-	if err != nil || oi.VersionID != "" {
-		t.Fatalf("null version fixture failed: %v %q", err, oi.VersionID)
-	}
-	if _, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, oi, body)
-	if _, err := z.serverPools[1].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); !isErrObjectNotFound(err) {
-		t.Fatalf("null version source was not removed: %v", err)
-	}
-}
-
-// Source removal can partially commit before a process or disk fails. The
-// destination can therefore hold the only remaining copy of an older version.
-func TestAccessMoveRetryPreservesDestinationVersions(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "retry"
-	onlyAtDestination := putAccessMoveVersion(t, z, bucket, object, 0, "unique-old", true)
-	source := putAccessMoveVersion(t, z, bucket, object, 1, "source-new", false)
-	if _, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, onlyAtDestination, "unique-old")
-	assertAccessMoveVersion(t, z, bucket, object, 0, source, "source-new")
-}
-
-type accessMoveFaultDisk struct {
-	StorageAPI
-	failVersion string
-}
-
-func (d accessMoveFaultDisk) RenameData(ctx context.Context, srcVolume, srcPath string, fi FileInfo, dstVolume, dstPath string, opts RenameOptions) (RenameDataResp, error) {
-	if fi.VersionID == d.failVersion {
-		return RenameDataResp{}, errDiskFull
-	}
-	return d.StorageAPI.RenameData(ctx, srcVolume, srcPath, fi, dstVolume, dstPath, opts)
-}
-
-func TestAccessMoveWriteFailureCanResume(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "write-failure"
-	old := putAccessMoveVersion(t, z, bucket, object, 1, "old", false)
-	latest := putAccessMoveVersion(t, z, bucket, object, 1, "new", false)
-	existing := putAccessMoveVersion(t, z, bucket, object, 0, "unique-destination", true)
-	set := z.serverPools[0].getHashedSet(object)
-	getDisks := set.getDisks
-	disks := getDisks()
-	faulty := make([]StorageAPI, len(disks))
-	for i, disk := range disks {
-		faulty[i] = accessMoveFaultDisk{StorageAPI: disk, failVersion: latest.VersionID}
-	}
-	set.getDisks = func() []StorageAPI { return faulty }
-	_, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil)
-	set.getDisks = getDisks
-	if err == nil {
-		t.Fatal("injected destination write failure was ignored")
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 1, old, "old")
-	assertAccessMoveVersion(t, z, bucket, object, 1, latest, "new")
-	assertAccessMoveVersion(t, z, bucket, object, 0, existing, "unique-destination")
-	if _, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, old, "old")
-	assertAccessMoveVersion(t, z, bucket, object, 0, latest, "new")
-	assertAccessMoveVersion(t, z, bucket, object, 0, existing, "unique-destination")
-}
-
-func TestAccessMoveExcludesSourceWriter(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "concurrent"
-	old := putAccessMoveVersion(t, z, bucket, object, 1, "old", false)
-	_, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, func(ObjectInfo, uint64) error {
-		ctx, cancel := context.WithTimeout(t.Context(), 200*time.Millisecond)
-		defer cancel()
-		_, err := z.serverPools[1].PutObject(ctx, bucket, object,
-			mustGetPutObjReader(t, bytes.NewBufferString("racing-write"), 12, "", ""), ObjectOptions{Versioned: true})
-		if err == nil {
-			t.Error("source writer committed while move was in progress")
-		}
-		if ctx.Err() == nil {
-			return errors.New("writer did not wait for the move lock")
-		}
-		return nil
-	})
-	if err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, old, "old")
-}
-
-func TestAccessMoveResumesPartialCopy(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "partial-copy"
-	old := putAccessMoveVersion(t, z, bucket, object, 1, "old", false)
-	latest := putAccessMoveVersion(t, z, bucket, object, 1, "new", false)
-	gr, err := z.serverPools[1].GetObjectNInfo(t.Context(), bucket, object, nil, http.Header{}, ObjectOptions{VersionID: old.VersionID, NoDecryption: true})
-	if err != nil {
-		t.Fatal(err)
-	}
-	// Model a process stopping after the first copy commits, before cleanup.
-	if err := moveAccessTierVersion(t.Context(), z, 1, 0, bucket, gr, time.Now().UnixNano()); err != nil {
-		t.Fatal(err)
-	}
-	if _, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, old, "old")
-	assertAccessMoveVersion(t, z, bucket, object, 0, latest, "new")
-}
-
-func TestAccessMoveRefusesConflictingVersion(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "conflicting-version"
-	source := putAccessMoveVersion(t, z, bucket, object, 1, "source", false)
-	destination, err := z.serverPools[0].PutObject(t.Context(), bucket, object,
-		mustGetPutObjReader(t, bytes.NewBufferString("target"), 6, "", ""), ObjectOptions{
-			VersionID: source.VersionID, MTime: source.ModTime,
-			UserDefined:  map[string]string{"test-value": "target"},
-			WantChecksum: hash.NewChecksumFromData(hash.ChecksumCRC32C, []byte("target")),
-		})
-	if err != nil {
-		t.Fatal(err)
-	}
-	if _, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil); !errors.Is(err, errAccessTierNotEligible) {
-		t.Fatalf("conflicting version should be left intact: %v", err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 1, source, "source")
-	assertAccessMoveVersion(t, z, bucket, object, 0, destination, "target")
-}
-
-type accessMoveDeleteFaultDisk struct {
-	StorageAPI
-	bucket, object, version string
-}
-
-func (d accessMoveDeleteFaultDisk) DeleteVersion(ctx context.Context, volume, path string, fi FileInfo, forceDelMarker bool, opts DeleteOptions) error {
-	if volume == d.bucket && path == d.object && fi.VersionID == d.version {
-		return errDiskFull
-	}
-	return d.StorageAPI.DeleteVersion(ctx, volume, path, fi, forceDelMarker, opts)
-}
-
-func TestAccessMoveSourceDeleteFailureCanResume(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "source-delete-failure"
-	old := putAccessMoveVersion(t, z, bucket, object, 1, "old", false)
-	latest := putAccessMoveVersion(t, z, bucket, object, 1, "new", false)
-	set := z.serverPools[1].getHashedSet(object)
-	getDisks := set.getDisks
-	faulty := append([]StorageAPI(nil), getDisks()...)
-	// Commit removal of the old version, then reject the latest version on
-	// every disk. This fixes the retry boundary without relying on how a
-	// partially deleted erasure quorum is subsequently resolved or healed.
-	for i := range faulty {
-		faulty[i] = accessMoveDeleteFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object, version: latest.VersionID}
-	}
-	set.getDisks = func() []StorageAPI { return faulty }
-	_, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil)
-	set.getDisks = getDisks
-	if err == nil {
-		t.Fatal("injected partial source purge was ignored")
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, old, "old")
-	assertAccessMoveVersion(t, z, bucket, object, 0, latest, "new")
-	if _, err := z.serverPools[1].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: old.VersionID}); !isErrVersionNotFound(err) {
-		t.Fatalf("old source version should already be removed: %v", err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 1, latest, "new")
-	if _, err := moveObjectPool(t.Context(), z, bucket, object, 1, 0, nil); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, old, "old")
-	assertAccessMoveVersion(t, z, bucket, object, 0, latest, "new")
-}
-
-// Exercise the public multipart API and compare whole-object and part reads
-// across both directions of a real pool move, including raw SSE-C ciphertext.
-func TestAccessMoveMultipartChecksums(t *testing.T) {
-	z, _ := accessMovePools(t)
-	ctx, cancel := context.WithCancel(t.Context())
-	defer cancel()
-	bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{VersioningEnabled: true})
-	if err != nil {
-		t.Fatal(err)
-	}
-	previousTLS := globalIsTLS
-	globalIsTLS = true
-	defer func() { globalIsTLS = previousTLS }()
-	partData, full := multipartChecksumTestData()
-	for _, tc := range []struct {
-		typ  hash.ChecksumType
-		ssec bool
-	}{{hash.ChecksumCRC32, false}, {hash.ChecksumCRC32C, false}, {hash.ChecksumCRC64NVME, false}, {hash.ChecksumCRC32C, true}} {
-		t.Run(fmt.Sprintf("%s/ssec=%t", tc.typ, tc.ssec), func(t *testing.T) {
-			object := getRandomObjectName()
-			headers := map[string]string{}
-			if tc.ssec {
-				key := bytes.Repeat([]byte{0x42}, 32)
-				keyMD5 := md5.Sum(key)
-				headers[xhttp.AmzServerSideEncryptionCustomerAlgorithm] = xhttp.AmzEncryptionAES
-				headers[xhttp.AmzServerSideEncryptionCustomerKey] = base64.StdEncoding.EncodeToString(key)
-				headers[xhttp.AmzServerSideEncryptionCustomerKeyMD5] = base64.StdEncoding.EncodeToString(keyMD5[:])
-			}
-			do := func(method, url string, body []byte, extra map[string]string) *httptest.ResponseRecorder {
-				t.Helper()
-				h := maps.Clone(headers)
-				maps.Copy(h, extra)
-				req, err := newTestSignedRequestV4(method, url, int64(len(body)), bytes.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, h)
-				if err != nil {
-					t.Fatal(err)
-				}
-				rec := httptest.NewRecorder()
-				router.ServeHTTP(rec, req)
-				if rec.Code != http.StatusOK && rec.Code != http.StatusPartialContent {
-					t.Fatalf("%s %s: %d %s", method, url, rec.Code, rec.Body.String())
-				}
-				return rec
-			}
-			init := do(http.MethodPost, getNewMultipartURL("", bucket, object), nil, map[string]string{
-				xhttp.AmzChecksumAlgo: tc.typ.String(), xhttp.AmzChecksumType: xhttp.AmzChecksumTypeFullObject,
-			})
-			var upload InitiateMultipartUploadResponse
-			if err := xml.Unmarshal(init.Body.Bytes(), &upload); err != nil {
-				t.Fatal(err)
-			}
-			parts := make([]CompletePart, len(partData))
-			for i, data := range partData {
-				rec := do(http.MethodPut, getPutObjectPartURL("", bucket, object, upload.UploadID, fmt.Sprint(i+1)), data,
-					map[string]string{tc.typ.Key(): mustChecksum(t, tc.typ, data)})
-				parts[i] = CompletePart{PartNumber: i + 1, ETag: canonicalizeETag(rec.Header()[xhttp.ETag][0])}
-			}
-			body, err := xml.Marshal(CompleteMultipartUpload{Parts: parts})
-			if err != nil {
-				t.Fatal(err)
-			}
-			do(http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, upload.UploadID), body,
-				map[string]string{tc.typ.Key(): mustChecksum(t, tc.typ, full), xhttp.AmzChecksumType: xhttp.AmzChecksumTypeFullObject})
-			source, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
-			if err != nil || len(source.Checksum) == 0 {
-				t.Fatalf("source checksum missing: %v", err)
-			}
-			src := 0
-			if _, err := z.serverPools[src].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); isErrObjectNotFound(err) {
-				src = 1
-			}
-			for i := 0; i < 2; i++ {
-				dst := 1 - src
-				if _, err := moveObjectPool(t.Context(), z, bucket, object, src, dst, nil); err != nil {
-					t.Fatal(err)
-				}
-				got, err := z.serverPools[dst].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
-				if err != nil || !bytes.Equal(got.Checksum, source.Checksum) || got.ETag != source.ETag || got.VersionID != source.VersionID || !got.ModTime.Equal(source.ModTime) {
-					t.Fatalf("move changed version metadata: %v checksumEqual=%t ETag=%q/%q version=%q/%q modTimeEqual=%t", err, bytes.Equal(got.Checksum, source.Checksum), got.ETag, source.ETag, got.VersionID, source.VersionID, got.ModTime.Equal(source.ModTime))
-				}
-				rec := do(http.MethodGet, getGetObjectURL("", bucket, object), nil, map[string]string{xhttp.AmzChecksumMode: "ENABLED"})
-				if !bytes.Equal(rec.Body.Bytes(), full) || rec.Header().Get(tc.typ.Key()) != mustChecksum(t, tc.typ, full) {
-					t.Fatal("GET payload or checksum changed after move")
-				}
-				for j, data := range partData {
-					rec := do(http.MethodGet, getGetObjectURL("", bucket, object)+fmt.Sprintf("?partNumber=%d", j+1), nil, nil)
-					if !bytes.Equal(rec.Body.Bytes(), data) {
-						t.Fatalf("part %d changed after move", j+1)
-					}
-				}
-				src = dst
-			}
-		})
-	}
-}
-
-type accessMoveNamedLocker struct {
-	*localLocker
-	address string
-}
-
-func (l accessMoveNamedLocker) String() string { return l.address }
-
-func TestAccessMoveSharedDistributedLockers(t *testing.T) {
-	// Three overlapping sets of real dsync lock servers. Taking independent
-	// quorum locks for the same object would contend with our own earlier lock.
-	peers := make([]dsync.NetLocker, 5)
-	for i := range peers {
-		peers[i] = accessMoveNamedLocker{localLocker: newLocker(), address: fmt.Sprint(i)}
-	}
-	z := &erasureServerPools{}
-	for i := 0; i < 3; i++ {
-		setPeers := peers[i : i+3]
-		set := &erasureObjects{nsMutex: &nsLockMap{isDistErasure: true}, getLockers: func() ([]dsync.NetLocker, string) {
-			return setPeers, "access-move-fixture"
-		}}
-		z.serverPools = append(z.serverPools, &erasureSets{sets: []*erasureObjects{set}, distributionAlgo: formatErasureVersionV3DistributionAlgoV3})
-	}
-	ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
-	defer cancel()
-	_, unlock, err := lockAccessTierObject(ctx, z, "bucket", "object", 1, 2)
-	if err != nil {
-		t.Fatal(err)
-	}
-	release := sync.OnceFunc(unlock)
-	defer release()
-	for i, pool := range z.serverPools {
-		ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
-		lock := pool.NewNSLock("bucket", "object")
-		lc, err := lock.GetLock(ctx, globalOperationTimeout)
-		cancel()
-		if err == nil {
-			lock.Unlock(lc)
-			t.Fatalf("pool %d writer acquired its quorum during the move", i)
-		}
-	}
-	release()
-	// Every acquired peer lock is released, so ordinary writes can resume.
-	for _, peer := range peers {
-		// Distributed Unlock sends releases asynchronously.
-		lock := (&nsLockMap{isDistErasure: true}).NewNSLock(func() ([]dsync.NetLocker, string) {
-			return []dsync.NetLocker{peer}, "access-move-fixture"
-		}, "bucket", "object")
-		ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
-		lc, err := lock.GetLock(ctx, globalOperationTimeout)
-		cancel()
-		if err != nil {
-			t.Fatalf("peer %s leaked a lock: %v", peer.String(), err)
-		}
-		lock.Unlock(lc)
-	}
-}
-
-func TestAccessMoveConfiguredPromotionAndDemotion(t *testing.T) {
-	z, bucket := accessMovePools(t)
-	const object = "configured-move"
-	version := putAccessMoveVersion(t, z, bucket, object, 1, "configured", false)
-	oldCfg, oldTracker := globalILMConfig.accessCfg(), globalAccessTracker
-	defer func() { globalILMConfig.update(oldCfg); globalAccessTracker = oldTracker }()
-	cfg := oldCfg
-	cfg.AccessTiering, cfg.AccessPools = true, []int{0, 1}
-	cfg.AccessMinResidency, cfg.AccessPromoteWatermark = 0, 99
-	globalILMConfig.update(cfg)
-	globalAccessTracker = newAccessTracker()
-	lc := accessLifecycleForTest(t)
-	data, err := xml.Marshal(lc)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketLifecycleConfig, data); err != nil {
-		t.Fatal(err)
-	}
-	state := newAccessTierState(t.Context())
-	state.z, state.objAPI = z, z
-	task := accessTierTask{ctx: t.Context(), bucket: bucket, object: object, src: 1, dst: 0, direction: accessTierPromote, bytes: uint64(version.Size)}
-	// A queued move is rechecked when configuration is disabled dynamically.
-	cfg.AccessTiering = false
-	globalILMConfig.update(cfg)
-	if err := state.processTask(task); !errors.Is(err, errAccessTierNotEligible) {
-		t.Fatalf("disabled mover = %v", err)
-	}
-	cfg.AccessTiering = true
-	globalILMConfig.update(cfg)
-	globalAccessTracker.merged.Store(&mergedAccess{binWidth: 60, entries: map[string]accessEntry{
-		accessKey(bucket, object): {Bins: []uint32{100}, LastAt: time.Now().Unix()},
-	}})
-	if reason, ok := state.reservePromotion(task, cfg, 0); !ok {
-		t.Fatalf("promotion reservation failed: %s", reason)
-	}
-	if err := state.processTask(task); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 0, version, "configured")
-	// Advance the counter view beyond the rule's idle period.
-	globalAccessTracker.merged.Store(&mergedAccess{binWidth: 60, entries: map[string]accessEntry{
-		accessKey(bucket, object): {Bins: []uint32{0}, LastAt: time.Now().Add(-2 * time.Hour).Unix()},
-	}})
-	task.src, task.dst, task.direction, task.bytes = 0, 1, accessTierDemote, 0
-	if err := state.processTask(task); err != nil {
-		t.Fatal(err)
-	}
-	assertAccessMoveVersion(t, z, bucket, object, 1, version, "configured")
-	if state.promotions.Load() != 1 || state.demotions.Load() != 1 || state.bytesMoved.Load() != 2*uint64(version.Size) || len(state.pending) != 0 || len(state.reserved) != 0 {
-		t.Fatal("promotion/demotion metrics or reservation cleanup are incorrect")
-	}
-}

cmd/ilm-access-tier_test.go

@@ -1,263 +0,0 @@
-// Copyright (c) 2015-2026 MinIO, Inc.
-
-package cmd
-
-import (
-	"bytes"
-	"context"
-	"errors"
-	"strconv"
-	"strings"
-	"testing"
-	"time"
-
-	"github.com/klauspost/compress/zstd"
-	"github.com/minio/minio/internal/bucket/lifecycle"
-	"github.com/minio/minio/internal/config/ilm"
-	"github.com/tinylib/msgp/msgp"
-)
-
-func accessLifecycleForTest(t *testing.T) *lifecycle.Lifecycle {
-	t.Helper()
-	xml := "<LifecycleConfiguration>" +
-		"<Rule><ID>access</ID><Status>Enabled</Status>" +
-		"<AccessTransition><Window>10m</Window><PromoteAfterAccesses>100</PromoteAfterAccesses>" +
-		"<DemoteAfterAccesses>5</DemoteAfterAccesses><DemoteAfterIdle>1h</DemoteAfterIdle></AccessTransition>" +
-		"</Rule></LifecycleConfiguration>"
-	lc, err := lifecycle.ParseLifecycleConfig(strings.NewReader(xml))
-	if err != nil {
-		t.Fatal(err)
-	}
-	return lc
-}
-
-func TestAccessTierStampAndDemoteEligibility(t *testing.T) {
-	oldTracker := globalAccessTracker
-	globalAccessTracker = newAccessTracker()
-	t.Cleanup(func() { globalAccessTracker = oldTracker })
-
-	now := time.Now()
-	cfg := ilm.Config{
-		AccessTiering: true, AccessPools: []int{0, 1},
-		AccessBinWidth: time.Minute, AccessBins: 12,
-		AccessMinResidency: 30 * time.Minute,
-	}
-	oi := ObjectInfo{
-		Bucket: "bucket", Name: "object", Size: 10, IsLatest: true,
-		UserDefined: map[string]string{
-			accessTierMetadataKey: "0:" + strconv.FormatInt(now.Add(-2*time.Hour).UnixNano(), 10),
-		},
-	}
-	if !accessDemoteEligible(accessLifecycleForTest(t), oi, 0, cfg, now) {
-		t.Fatal("cold, resident object should be demotion eligible")
-	}
-	oi.UserDefined[accessTierMetadataKey] = "0:" + strconv.FormatInt(now.Add(-time.Minute).UnixNano(), 10)
-	if accessDemoteEligible(accessLifecycleForTest(t), oi, 0, cfg, now) {
-		t.Fatal("object inside minimum residency was eligible")
-	}
-	oi.UserDefined[accessTierMetadataKey] = "broken"
-	if accessDemoteEligible(accessLifecycleForTest(t), oi, 0, cfg, now) {
-		t.Fatal("object with malformed marker was eligible")
-	}
-}
-
-func TestAccessTierReservationsEnforceCaps(t *testing.T) {
-	state := newAccessTierState(t.Context())
-	state.usageReady = true
-	state.baseUsage["a"] = 80
-	state.baseTotal = 180
-
-	cfg := ilm.Config{AccessMaxSize: 200}
-	task := accessTierTask{ctx: t.Context(), bucket: "a", object: "one", bytes: 30}
-	if reason, ok := state.reservePromotion(task, cfg, 0); ok || reason != "max-size" {
-		t.Fatalf("max-size reserve = %q/%v", reason, ok)
-	}
-
-	cfg.AccessMaxSize = 0
-	if reason, ok := state.reservePromotion(task, cfg, 100); ok || reason != "quota" {
-		t.Fatalf("quota reserve = %q/%v", reason, ok)
-	}
-
-	task.bytes = 20
-	if reason, ok := state.reservePromotion(task, cfg, 100); !ok || reason != "" {
-		t.Fatalf("valid reserve = %q/%v", reason, ok)
-	}
-	if got := state.bucketUsageLocked("a"); got != 100 {
-		t.Fatalf("reserved bucket usage = %d, want 100", got)
-	}
-	state.releasePending(task, true)
-}
-
-func TestDataMovementDestinationIsGated(t *testing.T) {
-	dst := 0
-	if got := dataMovementDstPool(ObjectOptions{DstPoolIdx: &dst}); got != nil {
-		t.Fatal("destination honored without DataMovement")
-	}
-	if got := dataMovementDstPool(ObjectOptions{DataMovement: true, DstPoolIdx: &dst}); got == nil || *got != 0 {
-		t.Fatalf("destination = %v", got)
-	}
-}
-
-func TestForcedDestinationPoolSelection(t *testing.T) {
-	z := &erasureServerPools{serverPools: make([]*erasureSets, 2)}
-	dst := 1
-	if got, err := z.getPoolIdx(context.Background(), "bucket", "object", 1, &dst); err != nil || got != dst {
-		t.Fatalf("destination = %d, err = %v", got, err)
-	}
-	bad := 2
-	if _, err := z.getPoolIdx(context.Background(), "bucket", "object", 1, &bad); !errors.Is(err, errInvalidArgument) {
-		t.Fatalf("out-of-range error = %v", err)
-	}
-	z.poolMeta.Pools = make([]PoolStatus, 2)
-	z.poolMeta.Pools[1].Decommission = &PoolDecommissionInfo{}
-	if _, err := z.getPoolIdx(context.Background(), "bucket", "object", 1, &dst); err == nil {
-		t.Fatal("suspended destination was accepted")
-	}
-}
-
-func TestHotTierAccounting(t *testing.T) {
-	entry := dataUsageEntry{}
-	entry.addSizes(sizeSummary{totalSize: 100, hotTierSize: 60, versions: 1})
-	entry.merge(dataUsageEntry{Size: 50, HotTierSize: 25})
-	if entry.Size != 150 || entry.HotTierSize != 85 {
-		t.Fatalf("usage = size:%d hot:%d", entry.Size, entry.HotTierSize)
-	}
-}
-
-func TestScannerCyclesApart(t *testing.T) {
-	tests := []struct {
-		current, previous uint32
-		want              uint32
-	}{
-		{current: 12, previous: 10, want: 2},
-		{current: 0, previous: ^uint32(0), want: 1},
-		// A cycle counter reset is not evidence that a complete pass covered
-		// recent moves, so it must not release conservative deltas.
-		{current: 1, previous: 100, want: 0},
-	}
-	for _, tt := range tests {
-		if got := scannerCyclesApart(tt.current, tt.previous); got != tt.want {
-			t.Fatalf("scannerCyclesApart(%d, %d) = %d, want %d", tt.current, tt.previous, got, tt.want)
-		}
-	}
-}
-
-func TestDataUsageCacheV8Migration(t *testing.T) {
-	var encoded bytes.Buffer
-	if err := encoded.WriteByte(dataUsageCacheVerV8); err != nil {
-		t.Fatal(err)
-	}
-	zw, err := zstd.NewWriter(&encoded)
-	if err != nil {
-		t.Fatal(err)
-	}
-	mw := msgp.NewWriter(zw)
-	if err = mw.WriteMapHeader(2); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteString("Info"); err != nil {
-		t.Fatal(err)
-	}
-	info := dataUsageCacheInfo{Name: dataUsageRoot, NextCycle: 17, LastUpdate: time.Now().UTC()}
-	if err = info.EncodeMsg(mw); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteString("Cache"); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteMapHeader(1); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteString("entry"); err != nil {
-		t.Fatal(err)
-	}
-	// A v8 entry has no hts field. Encoding only populated fields also
-	// verifies that its map decoder retains normal msgpack compatibility.
-	if err = mw.WriteMapHeader(2); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteString("sz"); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteInt64(123); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteString("os"); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.WriteUint64(2); err != nil {
-		t.Fatal(err)
-	}
-	if err = mw.Flush(); err != nil {
-		t.Fatal(err)
-	}
-	if err = zw.Close(); err != nil {
-		t.Fatal(err)
-	}
-
-	var migrated dataUsageCache
-	if err = migrated.deserialize(bytes.NewReader(encoded.Bytes())); err != nil {
-		t.Fatal(err)
-	}
-	entry := migrated.Cache["entry"]
-	if entry.Size != 123 || entry.Objects != 2 || entry.HotTierSize != 0 {
-		t.Fatalf("migrated entry = size:%d objects:%d hot:%d", entry.Size, entry.Objects, entry.HotTierSize)
-	}
-	if migrated.Info.NextCycle != 17 || !migrated.Info.LastUpdate.Equal(info.LastUpdate) {
-		t.Fatalf("migrated cache info = %+v", migrated.Info)
-	}
-}
-
-// The stamp written on every moved version and the stamp the rollback path
-// matches against must agree, otherwise rollback silently skips its own work.
-func TestAccessTierStampRoundTrip(t *testing.T) {
-	movedAt := time.Now().UnixNano()
-	stamp := accessTierStamp(3, movedAt)
-
-	pool, at, ok := parseAccessTierStamp(map[string]string{accessTierMetadataKey: stamp})
-	if !ok {
-		t.Fatalf("stamp %q did not parse", stamp)
-	}
-	if pool != 3 {
-		t.Fatalf("pool = %d, want 3", pool)
-	}
-	if at.UnixNano() != movedAt {
-		t.Fatalf("movedAt = %d, want %d", at.UnixNano(), movedAt)
-	}
-	// A different move of the same object must not match, so a concurrent
-	// client overwrite is never mistaken for our own copy.
-	if stamp == accessTierStamp(3, movedAt+1) {
-		t.Fatal("stamps from distinct moves collided")
-	}
-	if stamp == accessTierStamp(4, movedAt) {
-		t.Fatal("stamps from distinct pools collided")
-	}
-}
-
-func TestAccessHitsMightPromote(t *testing.T) {
-	lc := accessLifecycleForTest(t)
-	hot := accessEntry{Bins: []uint32{100}, HeadAt: 0}
-	cold := accessEntry{Bins: []uint32{1}, HeadAt: 0}
-	if !accessHitsMightPromote(lc, "object", hot, 60) {
-		t.Fatal("object above promote threshold was rejected")
-	}
-	if accessHitsMightPromote(lc, "object", cold, 60) {
-		t.Fatal("object below promote threshold was accepted")
-	}
-
-	xml := "<LifecycleConfiguration><Rule><ID>logs</ID><Status>Enabled</Status>" +
-		"<Filter><Prefix>logs/</Prefix></Filter>" +
-		"<AccessTransition><Window>10m</Window><PromoteAfterAccesses>100</PromoteAfterAccesses>" +
-		"<DemoteAfterAccesses>5</DemoteAfterAccesses><DemoteAfterIdle>1h</DemoteAfterIdle></AccessTransition>" +
-		"</Rule></LifecycleConfiguration>"
-	prefixed, err := lifecycle.ParseLifecycleConfig(strings.NewReader(xml))
-	if err != nil {
-		t.Fatal(err)
-	}
-	if accessHitsMightPromote(prefixed, "data/object", hot, 60) {
-		t.Fatal("object outside the rule prefix was accepted")
-	}
-	if !accessHitsMightPromote(prefixed, "logs/object", hot, 60) {
-		t.Fatal("object inside the rule prefix was rejected")
-	}
-}

cmd/ilm-access-tracker.go

@@ -1,598 +0,0 @@
-// Copyright (c) 2015-2026 MinIO, Inc.
-//
-// This file is part of MinIO Object Storage stack
-//
-// This program is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Affero General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// This program is distributed in the hope that it will be useful
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU Affero General Public License for more details.
-//
-// You should have received a copy of the GNU Affero General Public License
-// along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-package cmd
-
-import (
-	"cmp"
-	"context"
-	"fmt"
-	"slices"
-	"strings"
-	"sync"
-	"sync/atomic"
-	"time"
-
-	"github.com/minio/minio/internal/config/ilm"
-	"github.com/zeebo/xxh3"
-)
-
-//go:generate msgp -file=$GOFILE -unexported
-//msgp:ignore accessTracker mergedAccess
-
-const (
-	// accessTrackerPrefix is where each node publishes its own counters.
-	// One object per node, merged by every node on a timer.
-	accessTrackerPrefix = minioConfigPrefix + "/ilm/access"
-
-	// accessQueueSize bounds the GET -> tracker handoff. Overflow drops
-	// samples rather than slowing down reads.
-	accessQueueSize = 100000
-
-	// accessMaxDemoteCandidates bounds what the scanner may hand over in a
-	// single flush interval.
-	accessMaxDemoteCandidates = 100000
-
-	// accessShardStaleFactor multiplies the flush interval to decide when a
-	// peer's counters are too old to trust, e.g. after a node is removed.
-	accessShardStaleFactor = 5
-)
-
-func accessShardFresh(now int64, shard accessShard, stale int64) bool {
-	if shard.UpdatedAt <= 0 {
-		return false
-	}
-	if stale <= 0 {
-		return true
-	}
-	age := now - shard.UpdatedAt
-	return age >= -stale && age <= stale
-}
-
-// accessEntry is a rolling hit counter for one object. Bins[0] is the current
-// bin and each subsequent bin is one bin-width older, so a rule asking for
-// "100 hits in 10 minutes" sums the newest ceil(10m/binWidth) bins.
-//
-// A fixed window is used rather than an exponentially decayed score because
-// the rule is stated to operators in exactly those terms.
-type accessEntry struct {
-	Bins   []uint32 `msg:"b"`
-	HeadAt int64    `msg:"h"` // unix seconds at the start of Bins[0]
-	LastAt int64    `msg:"l"` // unix seconds of the most recent hit
-}
-
-// demoteCandidate is an object the scanner found sitting on a fast pool with
-// no recent reads. Candidates ride the node's own counter shard so they reach
-// the leader without a new peer RPC.
-type demoteCandidate struct {
-	Bucket string `msg:"b"`
-	Object string `msg:"o"`
-	Pool   int    `msg:"p"`
-}
-
-// accessShard is what one node publishes. BinWidth is carried so a peer that
-// has not yet picked up a configuration change is ignored rather than merged
-// with mismatched bins.
-type accessShard struct {
-	UpdatedAt int64                  `msg:"u"`
-	BinWidth  int64                  `msg:"bw"`
-	Entries   map[string]accessEntry `msg:"e"`
-	Demote    []demoteCandidate      `msg:"d"`
-}
-
-// binStart truncates a unix timestamp to the start of its bin.
-func binStart(now, binWidth int64) int64 {
-	if binWidth <= 0 {
-		return now
-	}
-	return now - now%binWidth
-}
-
-// rollTo advances the counter to now, zeroing the bins that elapsed since the
-// last update and resizing if the configured bin count changed.
-func (e *accessEntry) rollTo(now, binWidth int64, nbins int) {
-	if nbins <= 0 || binWidth <= 0 {
-		return
-	}
-	if len(e.Bins) != nbins {
-		resized := make([]uint32, nbins)
-		copy(resized, e.Bins)
-		e.Bins = resized
-	}
-	head := binStart(now, binWidth)
-	if e.HeadAt == 0 {
-		e.HeadAt = head
-		return
-	}
-	steps := (head - e.HeadAt) / binWidth
-	if steps <= 0 {
-		return
-	}
-	if steps >= int64(nbins) {
-		clear(e.Bins)
-	} else {
-		copy(e.Bins[steps:], e.Bins[:nbins-int(steps)])
-		clear(e.Bins[:steps])
-	}
-	e.HeadAt = head
-}
-
-// hits returns the number of accesses recorded over the newest bins covering
-// window. A window longer than the configured history is clamped to it.
-//
-// The newest bin is partial, so the covered span is between window-binWidth
-// and window. Operators tune resolution with ilm access_bin_width.
-func (e accessEntry) hits(window time.Duration, binWidth int64) uint64 {
-	if binWidth <= 0 || len(e.Bins) == 0 {
-		return 0
-	}
-	n := int((int64(window/time.Second) + binWidth - 1) / binWidth)
-	if n < 1 {
-		n = 1
-	}
-	if n > len(e.Bins) {
-		n = len(e.Bins)
-	}
-	var total uint64
-	for _, v := range e.Bins[:n] {
-		total += uint64(v)
-	}
-	return total
-}
-
-// total is the whole retained history, used to decide what to evict.
-func (e accessEntry) total() uint64 {
-	var t uint64
-	for _, v := range e.Bins {
-		t += uint64(v)
-	}
-	return t
-}
-
-// mergeFrom adds another node's counters for the same object. Both sides must
-// already be rolled to the same head.
-func (e *accessEntry) mergeFrom(o accessEntry) {
-	for i := range e.Bins {
-		if i < len(o.Bins) {
-			total := uint64(e.Bins[i]) + uint64(o.Bins[i])
-			if total > uint64(^uint32(0)) {
-				total = uint64(^uint32(0))
-			}
-			e.Bins[i] = uint32(total)
-		}
-	}
-	if o.LastAt > e.LastAt {
-		e.LastAt = o.LastAt
-	}
-}
-
-// mergedAccess is an immutable cluster-wide snapshot. Readers take it from an
-// atomic pointer, so the hot scanner and sweep paths never take a lock.
-type mergedAccess struct {
-	entries  map[string]accessEntry
-	binWidth int64
-	at       int64
-}
-
-func (m *mergedAccess) hits(key string, window time.Duration) uint64 {
-	if m == nil {
-		return 0
-	}
-	e, ok := m.entries[key]
-	if !ok {
-		return 0
-	}
-	return e.hits(window, m.binWidth)
-}
-
-func (m *mergedAccess) lastAccess(key string) int64 {
-	if m == nil {
-		return 0
-	}
-	return m.entries[key].LastAt
-}
-
-// accessTracker records how often each object is read.
-//
-// Ownership is deliberately narrow: the live counter map is touched only by
-// run()'s goroutine, so it needs no lock. Everything read from elsewhere goes
-// through the immutable merged snapshot.
-type accessTracker struct {
-	ch      chan string
-	enabled atomic.Bool
-	merged  atomic.Pointer[mergedAccess]
-
-	// Demote candidates arrive from scanner goroutines, so this one does
-	// need a lock. It is small: only objects we previously promoted.
-	demoteMu sync.Mutex
-	demote   map[string]demoteCandidate
-
-	dropped atomic.Uint64
-}
-
-var globalAccessTracker = newAccessTracker()
-
-func newAccessTracker() *accessTracker {
-	return &accessTracker{
-		ch:     make(chan string, accessQueueSize),
-		demote: make(map[string]demoteCandidate),
-	}
-}
-
-// accessKey is the tracker's map key. Bucket names cannot contain '/', so the
-// join is unambiguous.
-func accessKey(bucket, object string) string {
-	return bucket + "/" + object
-}
-
-func splitAccessKey(key string) (bucket, object string, ok bool) {
-	bucket, object, ok = strings.Cut(key, "/")
-	if !ok || bucket == "" || object == "" {
-		return "", "", false
-	}
-	return bucket, object, true
-}
-
-// note records one read. It is called from the GET path and must never block
-// or allocate meaningfully: on a full queue the sample is dropped.
-func (t *accessTracker) note(bucket, object string) {
-	if t == nil || !t.enabled.Load() {
-		return
-	}
-	select {
-	case t.ch <- accessKey(bucket, object):
-	default:
-		t.dropped.Add(1)
-	}
-}
-
-// noteDemoteCandidate is called by the scanner for an object it found on a
-// fast pool that has gone quiet. The leader picks these up on the next merge.
-func (t *accessTracker) noteDemoteCandidate(bucket, object string, pool int) {
-	if t == nil || !t.enabled.Load() {
-		return
-	}
-	t.demoteMu.Lock()
-	defer t.demoteMu.Unlock()
-	if len(t.demote) >= accessMaxDemoteCandidates {
-		return
-	}
-	t.demote[accessKey(bucket, object)] = demoteCandidate{Bucket: bucket, Object: object, Pool: pool}
-}
-
-// hits reports cluster-wide accesses to an object over window.
-func (t *accessTracker) hits(bucket, object string, window time.Duration) uint64 {
-	if t == nil {
-		return 0
-	}
-	return t.merged.Load().hits(accessKey(bucket, object), window)
-}
-
-// lastAccess reports the cluster-wide time an object was last read. A zero
-// time means "no read on record", which for demotion purposes is idle.
-func (t *accessTracker) lastAccess(bucket, object string) time.Time {
-	if t == nil {
-		return time.Time{}
-	}
-	sec := t.merged.Load().lastAccess(accessKey(bucket, object))
-	if sec == 0 {
-		return time.Time{}
-	}
-	return time.Unix(sec, 0)
-}
-
-// snapshot returns the current merged view, or nil if none has been published.
-func (t *accessTracker) snapshot() *mergedAccess {
-	if t == nil {
-		return nil
-	}
-	return t.merged.Load()
-}
-
-// takeDemoteCandidates drains and returns the pending candidates.
-func (t *accessTracker) takeDemoteCandidates() []demoteCandidate {
-	t.demoteMu.Lock()
-	defer t.demoteMu.Unlock()
-	if len(t.demote) == 0 {
-		return nil
-	}
-	out := make([]demoteCandidate, 0, len(t.demote))
-	for _, c := range t.demote {
-		out = append(out, c)
-	}
-	t.demote = make(map[string]demoteCandidate)
-	return out
-}
-
-// restoreDemoteCandidates puts candidates back when the publisher is still
-// busy. Dropping access samples is acceptable; dropping the only scanner
-// discovery of an idle promoted object would delay demotion by a full scan.
-func (t *accessTracker) restoreDemoteCandidates(candidates []demoteCandidate) {
-	if len(candidates) == 0 {
-		return
-	}
-	t.demoteMu.Lock()
-	defer t.demoteMu.Unlock()
-	for _, c := range candidates {
-		if len(t.demote) >= accessMaxDemoteCandidates {
-			return
-		}
-		t.demote[accessKey(c.Bucket, c.Object)] = c
-	}
-}
-
-// shardName is this node's counter object. The node name is hashed so that
-// host:port never has to be escaped into an object key.
-func (t *accessTracker) shardName() string {
-	return fmt.Sprintf("%s/%016x.bin", accessTrackerPrefix, xxh3.HashString(globalLocalNodeName))
-}
-
-// run owns the live counter map. It drains reads, and on every flush interval
-// hands a marshaled shard to a background publisher.
-//
-// Everything here is best effort: this is accounting for a background data
-// movement decision, not a durability path.
-func (t *accessTracker) run(ctx context.Context, objAPI ObjectLayer) {
-	cfg := globalILMConfig.accessCfg()
-	t.enabled.Store(cfg.AccessTiering)
-
-	live := make(map[string]accessEntry)
-	if cfg.AccessTiering {
-		if buf, err := readConfig(ctx, objAPI, t.shardName()); err == nil {
-			var previous accessShard
-			if _, err = previous.UnmarshalMsg(buf); err == nil && previous.BinWidth == int64(cfg.AccessBinWidth/time.Second) {
-				now := time.Now().Unix()
-				for key, entry := range previous.Entries {
-					entry.rollTo(now, previous.BinWidth, cfg.AccessBins)
-					if entry.total() != 0 {
-						live[key] = entry
-					}
-				}
-			}
-		}
-	}
-	ticker := time.NewTicker(cfg.AccessFlush)
-	defer ticker.Stop()
-
-	// One publisher goroutine keeps object-layer I/O off the drain loop.
-	type pub struct {
-		shard accessShard
-		cfg   ilm.Config
-	}
-	pubCh := make(chan pub, 1)
-	go func() {
-		for p := range pubCh {
-			t.publish(ctx, objAPI, p.shard, p.cfg)
-		}
-	}()
-	defer close(pubCh)
-
-	for {
-		select {
-		case <-ctx.Done():
-			return
-
-		case key := <-t.ch:
-			now := time.Now().Unix()
-			e := live[key]
-			e.rollTo(now, int64(cfg.AccessBinWidth/time.Second), cfg.AccessBins)
-			if len(e.Bins) > 0 {
-				if e.Bins[0] != ^uint32(0) {
-					e.Bins[0]++
-				}
-			}
-			e.LastAt = now
-			live[key] = e
-
-		case <-ticker.C:
-			newCfg := globalILMConfig.accessCfg()
-			t.enabled.Store(newCfg.AccessTiering)
-			if newCfg.AccessFlush != cfg.AccessFlush && newCfg.AccessFlush > 0 {
-				ticker.Reset(newCfg.AccessFlush)
-			}
-			cfg = newCfg
-			if !cfg.AccessTiering {
-				// Feature turned off: release the counters rather than
-				// holding a stale working set for the process lifetime.
-				clear(live)
-				t.merged.Store(nil)
-				continue
-			}
-
-			now := time.Now().Unix()
-			binWidth := int64(cfg.AccessBinWidth / time.Second)
-			t.evict(live, now, binWidth, cfg)
-
-			shard := accessShard{
-				UpdatedAt: now,
-				BinWidth:  binWidth,
-				Entries:   make(map[string]accessEntry, len(live)),
-				Demote:    t.takeDemoteCandidates(),
-			}
-			for k, e := range live {
-				e.Bins = slices.Clone(e.Bins)
-				shard.Entries[k] = e
-			}
-			select {
-			case pubCh <- pub{shard: shard, cfg: cfg}:
-			default:
-				// Previous publish still running; skip this round
-				// rather than queueing work we cannot keep up with.
-				t.restoreDemoteCandidates(shard.Demote)
-			}
-		}
-	}
-}
-
-// evict rolls every counter forward and drops the ones with no hits left in
-// the retained history, then enforces the tracked-object cap.
-//
-// ponytail: amortized O(n) sweep on the flush tick; a heap would only pay off
-// past ~10M tracked keys.
-func (t *accessTracker) evict(live map[string]accessEntry, now, binWidth int64, cfg ilm.Config) {
-	for k, e := range live {
-		e.rollTo(now, binWidth, cfg.AccessBins)
-		if e.total() == 0 {
-			delete(live, k)
-			continue
-		}
-		live[k] = e
-	}
-	if len(live) <= cfg.AccessMaxTracked {
-		return
-	}
-	// Over the cap: keep the hottest and drop the rest. Objects that fall
-	// out are cold by construction, which is exactly what the demotion path
-	// already assumes about anything missing from the map.
-	type kt struct {
-		key   string
-		total uint64
-	}
-	all := make([]kt, 0, len(live))
-	for k, e := range live {
-		all = append(all, kt{k, e.total()})
-	}
-	slices.SortFunc(all, func(a, b kt) int { return cmp.Compare(b.total, a.total) })
-	for _, x := range all[cfg.AccessMaxTracked:] {
-		delete(live, x.key)
-	}
-}
-
-// publish writes this node's shard and rebuilds the merged snapshot from every
-// node's shard. Both halves are best effort.
-func (t *accessTracker) publish(ctx context.Context, objAPI ObjectLayer, shard accessShard, cfg ilm.Config) {
-	buf, err := shard.MarshalMsg(nil)
-	if err != nil {
-		ilmLogIf(ctx, err)
-		return
-	}
-	if err := saveConfig(ctx, objAPI, t.shardName(), buf); err != nil {
-		ilmLogIf(ctx, err)
-		// Still rebuild the snapshot below: our own counters are already
-		// in hand and a stale peer view beats no view.
-	}
-	t.merged.Store(t.mergeShards(ctx, objAPI, shard, cfg))
-}
-
-// mergeShards sums every live node's counters, including our own in-memory
-// shard so this node's most recent reads are never a flush behind.
-func (t *accessTracker) mergeShards(ctx context.Context, objAPI ObjectLayer, own accessShard, cfg ilm.Config) *mergedAccess {
-	now := time.Now().Unix()
-	binWidth := int64(cfg.AccessBinWidth / time.Second)
-	out := &mergedAccess{
-		entries:  make(map[string]accessEntry, len(own.Entries)),
-		binWidth: binWidth,
-		at:       now,
-	}
-
-	add := func(s accessShard) {
-		if s.BinWidth != binWidth {
-			// A peer has not yet picked up a bin-width change; merging
-			// its bins would silently mis-scale the counts.
-			return
-		}
-		for k, e := range s.Entries {
-			e.rollTo(now, binWidth, cfg.AccessBins)
-			cur, ok := out.entries[k]
-			if !ok {
-				cur = accessEntry{Bins: make([]uint32, cfg.AccessBins)}
-			}
-			cur.mergeFrom(e)
-			out.entries[k] = cur
-		}
-	}
-
-	add(own)
-
-	ownName := t.shardName()
-	stale := int64(cfg.AccessFlush/time.Second) * accessShardStaleFactor
-	for _, name := range t.listShards(ctx, objAPI) {
-		if name == ownName {
-			continue
-		}
-		buf, err := readConfig(ctx, objAPI, name)
-		if err != nil {
-			continue
-		}
-		var s accessShard
-		if _, err := s.UnmarshalMsg(buf); err != nil {
-			continue
-		}
-		if !accessShardFresh(now, s, stale) {
-			continue // node is gone or wedged
-		}
-		add(s)
-	}
-	return out
-}
-
-func (t *accessTracker) listShards(ctx context.Context, objAPI ObjectLayer) []string {
-	res, err := objAPI.ListObjects(ctx, minioMetaBucket, accessTrackerPrefix+"/", "", "", maxObjectList)
-	if err != nil {
-		return nil
-	}
-	names := make([]string, 0, len(res.Objects))
-	for _, o := range res.Objects {
-		if strings.HasSuffix(o.Name, ".bin") {
-			names = append(names, o.Name)
-		}
-	}
-	return names
-}
-
-// collectDemoteCandidates returns every node's pending demote candidates. Only
-// the leader calls this, right before running a demotion pass.
-//
-// Our own shard is read back rather than skipped: run() drains the local map
-// into the published shard, so anything found since the last leader pass lives
-// there, not in memory. The local map is still drained here to pick up
-// candidates recorded since that publish.
-func (t *accessTracker) collectDemoteCandidates(ctx context.Context, objAPI ObjectLayer, cfg ilm.Config) []demoteCandidate {
-	seen := make(map[string]struct{})
-	var out []demoteCandidate
-
-	for _, c := range t.takeDemoteCandidates() {
-		key := accessKey(c.Bucket, c.Object)
-		seen[key] = struct{}{}
-		out = append(out, c)
-	}
-
-	now := time.Now().Unix()
-	stale := int64(cfg.AccessFlush/time.Second) * accessShardStaleFactor
-	for _, name := range t.listShards(ctx, objAPI) {
-		buf, err := readConfig(ctx, objAPI, name)
-		if err != nil {
-			continue
-		}
-		var s accessShard
-		if _, err := s.UnmarshalMsg(buf); err != nil {
-			continue
-		}
-		if !accessShardFresh(now, s, stale) {
-			continue
-		}
-		for _, c := range s.Demote {
-			key := accessKey(c.Bucket, c.Object)
-			if _, dup := seen[key]; dup {
-				continue
-			}
-			seen[key] = struct{}{}
-			out = append(out, c)
-		}
-	}
-	return out
-}

cmd/ilm-access-tracker_gen.go

@@ -1,742 +0,0 @@
-// Code generated by github.com/tinylib/msgp DO NOT EDIT.
-
-package cmd
-
-import (
-	"github.com/tinylib/msgp/msgp"
-)
-
-// DecodeMsg implements msgp.Decodable
-func (z *accessEntry) DecodeMsg(dc *msgp.Reader) (err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, err = dc.ReadMapHeader()
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, err = dc.ReadMapKeyPtr()
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "b":
-			var zb0002 uint32
-			zb0002, err = dc.ReadArrayHeader()
-			if err != nil {
-				err = msgp.WrapError(err, "Bins")
-				return
-			}
-			if cap(z.Bins) >= int(zb0002) {
-				z.Bins = (z.Bins)[:zb0002]
-			} else {
-				z.Bins = make([]uint32, zb0002)
-			}
-			for za0001 := range z.Bins {
-				z.Bins[za0001], err = dc.ReadUint32()
-				if err != nil {
-					err = msgp.WrapError(err, "Bins", za0001)
-					return
-				}
-			}
-		case "h":
-			z.HeadAt, err = dc.ReadInt64()
-			if err != nil {
-				err = msgp.WrapError(err, "HeadAt")
-				return
-			}
-		case "l":
-			z.LastAt, err = dc.ReadInt64()
-			if err != nil {
-				err = msgp.WrapError(err, "LastAt")
-				return
-			}
-		default:
-			err = dc.Skip()
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	return
-}
-
-// EncodeMsg implements msgp.Encodable
-func (z *accessEntry) EncodeMsg(en *msgp.Writer) (err error) {
-	// map header, size 3
-	// write "b"
-	err = en.Append(0x83, 0xa1, 0x62)
-	if err != nil {
-		return
-	}
-	err = en.WriteArrayHeader(uint32(len(z.Bins)))
-	if err != nil {
-		err = msgp.WrapError(err, "Bins")
-		return
-	}
-	for za0001 := range z.Bins {
-		err = en.WriteUint32(z.Bins[za0001])
-		if err != nil {
-			err = msgp.WrapError(err, "Bins", za0001)
-			return
-		}
-	}
-	// write "h"
-	err = en.Append(0xa1, 0x68)
-	if err != nil {
-		return
-	}
-	err = en.WriteInt64(z.HeadAt)
-	if err != nil {
-		err = msgp.WrapError(err, "HeadAt")
-		return
-	}
-	// write "l"
-	err = en.Append(0xa1, 0x6c)
-	if err != nil {
-		return
-	}
-	err = en.WriteInt64(z.LastAt)
-	if err != nil {
-		err = msgp.WrapError(err, "LastAt")
-		return
-	}
-	return
-}
-
-// MarshalMsg implements msgp.Marshaler
-func (z *accessEntry) MarshalMsg(b []byte) (o []byte, err error) {
-	o = msgp.Require(b, z.Msgsize())
-	// map header, size 3
-	// string "b"
-	o = append(o, 0x83, 0xa1, 0x62)
-	o = msgp.AppendArrayHeader(o, uint32(len(z.Bins)))
-	for za0001 := range z.Bins {
-		o = msgp.AppendUint32(o, z.Bins[za0001])
-	}
-	// string "h"
-	o = append(o, 0xa1, 0x68)
-	o = msgp.AppendInt64(o, z.HeadAt)
-	// string "l"
-	o = append(o, 0xa1, 0x6c)
-	o = msgp.AppendInt64(o, z.LastAt)
-	return
-}
-
-// UnmarshalMsg implements msgp.Unmarshaler
-func (z *accessEntry) UnmarshalMsg(bts []byte) (o []byte, err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, bts, err = msgp.ReadMapKeyZC(bts)
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "b":
-			var zb0002 uint32
-			zb0002, bts, err = msgp.ReadArrayHeaderBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Bins")
-				return
-			}
-			if cap(z.Bins) >= int(zb0002) {
-				z.Bins = (z.Bins)[:zb0002]
-			} else {
-				z.Bins = make([]uint32, zb0002)
-			}
-			for za0001 := range z.Bins {
-				z.Bins[za0001], bts, err = msgp.ReadUint32Bytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "Bins", za0001)
-					return
-				}
-			}
-		case "h":
-			z.HeadAt, bts, err = msgp.ReadInt64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "HeadAt")
-				return
-			}
-		case "l":
-			z.LastAt, bts, err = msgp.ReadInt64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "LastAt")
-				return
-			}
-		default:
-			bts, err = msgp.Skip(bts)
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	o = bts
-	return
-}
-
-// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
-func (z *accessEntry) Msgsize() (s int) {
-	s = 1 + 2 + msgp.ArrayHeaderSize + (len(z.Bins) * (msgp.Uint32Size)) + 2 + msgp.Int64Size + 2 + msgp.Int64Size
-	return
-}
-
-// DecodeMsg implements msgp.Decodable
-func (z *accessShard) DecodeMsg(dc *msgp.Reader) (err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, err = dc.ReadMapHeader()
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, err = dc.ReadMapKeyPtr()
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "u":
-			z.UpdatedAt, err = dc.ReadInt64()
-			if err != nil {
-				err = msgp.WrapError(err, "UpdatedAt")
-				return
-			}
-		case "bw":
-			z.BinWidth, err = dc.ReadInt64()
-			if err != nil {
-				err = msgp.WrapError(err, "BinWidth")
-				return
-			}
-		case "e":
-			var zb0002 uint32
-			zb0002, err = dc.ReadMapHeader()
-			if err != nil {
-				err = msgp.WrapError(err, "Entries")
-				return
-			}
-			if z.Entries == nil {
-				z.Entries = make(map[string]accessEntry, zb0002)
-			} else if len(z.Entries) > 0 {
-				clear(z.Entries)
-			}
-			for zb0002 > 0 {
-				zb0002--
-				var za0001 string
-				za0001, err = dc.ReadString()
-				if err != nil {
-					err = msgp.WrapError(err, "Entries")
-					return
-				}
-				var za0002 accessEntry
-				err = za0002.DecodeMsg(dc)
-				if err != nil {
-					err = msgp.WrapError(err, "Entries", za0001)
-					return
-				}
-				z.Entries[za0001] = za0002
-			}
-		case "d":
-			var zb0003 uint32
-			zb0003, err = dc.ReadArrayHeader()
-			if err != nil {
-				err = msgp.WrapError(err, "Demote")
-				return
-			}
-			if cap(z.Demote) >= int(zb0003) {
-				z.Demote = (z.Demote)[:zb0003]
-			} else {
-				z.Demote = make([]demoteCandidate, zb0003)
-			}
-			for za0003 := range z.Demote {
-				var zb0004 uint32
-				zb0004, err = dc.ReadMapHeader()
-				if err != nil {
-					err = msgp.WrapError(err, "Demote", za0003)
-					return
-				}
-				for zb0004 > 0 {
-					zb0004--
-					field, err = dc.ReadMapKeyPtr()
-					if err != nil {
-						err = msgp.WrapError(err, "Demote", za0003)
-						return
-					}
-					switch msgp.UnsafeString(field) {
-					case "b":
-						z.Demote[za0003].Bucket, err = dc.ReadString()
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003, "Bucket")
-							return
-						}
-					case "o":
-						z.Demote[za0003].Object, err = dc.ReadString()
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003, "Object")
-							return
-						}
-					case "p":
-						z.Demote[za0003].Pool, err = dc.ReadInt()
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003, "Pool")
-							return
-						}
-					default:
-						err = dc.Skip()
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003)
-							return
-						}
-					}
-				}
-			}
-		default:
-			err = dc.Skip()
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	return
-}
-
-// EncodeMsg implements msgp.Encodable
-func (z *accessShard) EncodeMsg(en *msgp.Writer) (err error) {
-	// map header, size 4
-	// write "u"
-	err = en.Append(0x84, 0xa1, 0x75)
-	if err != nil {
-		return
-	}
-	err = en.WriteInt64(z.UpdatedAt)
-	if err != nil {
-		err = msgp.WrapError(err, "UpdatedAt")
-		return
-	}
-	// write "bw"
-	err = en.Append(0xa2, 0x62, 0x77)
-	if err != nil {
-		return
-	}
-	err = en.WriteInt64(z.BinWidth)
-	if err != nil {
-		err = msgp.WrapError(err, "BinWidth")
-		return
-	}
-	// write "e"
-	err = en.Append(0xa1, 0x65)
-	if err != nil {
-		return
-	}
-	err = en.WriteMapHeader(uint32(len(z.Entries)))
-	if err != nil {
-		err = msgp.WrapError(err, "Entries")
-		return
-	}
-	for za0001, za0002 := range z.Entries {
-		err = en.WriteString(za0001)
-		if err != nil {
-			err = msgp.WrapError(err, "Entries")
-			return
-		}
-		err = za0002.EncodeMsg(en)
-		if err != nil {
-			err = msgp.WrapError(err, "Entries", za0001)
-			return
-		}
-	}
-	// write "d"
-	err = en.Append(0xa1, 0x64)
-	if err != nil {
-		return
-	}
-	err = en.WriteArrayHeader(uint32(len(z.Demote)))
-	if err != nil {
-		err = msgp.WrapError(err, "Demote")
-		return
-	}
-	for za0003 := range z.Demote {
-		// map header, size 3
-		// write "b"
-		err = en.Append(0x83, 0xa1, 0x62)
-		if err != nil {
-			return
-		}
-		err = en.WriteString(z.Demote[za0003].Bucket)
-		if err != nil {
-			err = msgp.WrapError(err, "Demote", za0003, "Bucket")
-			return
-		}
-		// write "o"
-		err = en.Append(0xa1, 0x6f)
-		if err != nil {
-			return
-		}
-		err = en.WriteString(z.Demote[za0003].Object)
-		if err != nil {
-			err = msgp.WrapError(err, "Demote", za0003, "Object")
-			return
-		}
-		// write "p"
-		err = en.Append(0xa1, 0x70)
-		if err != nil {
-			return
-		}
-		err = en.WriteInt(z.Demote[za0003].Pool)
-		if err != nil {
-			err = msgp.WrapError(err, "Demote", za0003, "Pool")
-			return
-		}
-	}
-	return
-}
-
-// MarshalMsg implements msgp.Marshaler
-func (z *accessShard) MarshalMsg(b []byte) (o []byte, err error) {
-	o = msgp.Require(b, z.Msgsize())
-	// map header, size 4
-	// string "u"
-	o = append(o, 0x84, 0xa1, 0x75)
-	o = msgp.AppendInt64(o, z.UpdatedAt)
-	// string "bw"
-	o = append(o, 0xa2, 0x62, 0x77)
-	o = msgp.AppendInt64(o, z.BinWidth)
-	// string "e"
-	o = append(o, 0xa1, 0x65)
-	o = msgp.AppendMapHeader(o, uint32(len(z.Entries)))
-	for za0001, za0002 := range z.Entries {
-		o = msgp.AppendString(o, za0001)
-		o, err = za0002.MarshalMsg(o)
-		if err != nil {
-			err = msgp.WrapError(err, "Entries", za0001)
-			return
-		}
-	}
-	// string "d"
-	o = append(o, 0xa1, 0x64)
-	o = msgp.AppendArrayHeader(o, uint32(len(z.Demote)))
-	for za0003 := range z.Demote {
-		// map header, size 3
-		// string "b"
-		o = append(o, 0x83, 0xa1, 0x62)
-		o = msgp.AppendString(o, z.Demote[za0003].Bucket)
-		// string "o"
-		o = append(o, 0xa1, 0x6f)
-		o = msgp.AppendString(o, z.Demote[za0003].Object)
-		// string "p"
-		o = append(o, 0xa1, 0x70)
-		o = msgp.AppendInt(o, z.Demote[za0003].Pool)
-	}
-	return
-}
-
-// UnmarshalMsg implements msgp.Unmarshaler
-func (z *accessShard) UnmarshalMsg(bts []byte) (o []byte, err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, bts, err = msgp.ReadMapKeyZC(bts)
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "u":
-			z.UpdatedAt, bts, err = msgp.ReadInt64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "UpdatedAt")
-				return
-			}
-		case "bw":
-			z.BinWidth, bts, err = msgp.ReadInt64Bytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "BinWidth")
-				return
-			}
-		case "e":
-			var zb0002 uint32
-			zb0002, bts, err = msgp.ReadMapHeaderBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Entries")
-				return
-			}
-			if z.Entries == nil {
-				z.Entries = make(map[string]accessEntry, zb0002)
-			} else if len(z.Entries) > 0 {
-				clear(z.Entries)
-			}
-			for zb0002 > 0 {
-				var za0002 accessEntry
-				zb0002--
-				var za0001 string
-				za0001, bts, err = msgp.ReadStringBytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "Entries")
-					return
-				}
-				bts, err = za0002.UnmarshalMsg(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "Entries", za0001)
-					return
-				}
-				z.Entries[za0001] = za0002
-			}
-		case "d":
-			var zb0003 uint32
-			zb0003, bts, err = msgp.ReadArrayHeaderBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Demote")
-				return
-			}
-			if cap(z.Demote) >= int(zb0003) {
-				z.Demote = (z.Demote)[:zb0003]
-			} else {
-				z.Demote = make([]demoteCandidate, zb0003)
-			}
-			for za0003 := range z.Demote {
-				var zb0004 uint32
-				zb0004, bts, err = msgp.ReadMapHeaderBytes(bts)
-				if err != nil {
-					err = msgp.WrapError(err, "Demote", za0003)
-					return
-				}
-				for zb0004 > 0 {
-					zb0004--
-					field, bts, err = msgp.ReadMapKeyZC(bts)
-					if err != nil {
-						err = msgp.WrapError(err, "Demote", za0003)
-						return
-					}
-					switch msgp.UnsafeString(field) {
-					case "b":
-						z.Demote[za0003].Bucket, bts, err = msgp.ReadStringBytes(bts)
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003, "Bucket")
-							return
-						}
-					case "o":
-						z.Demote[za0003].Object, bts, err = msgp.ReadStringBytes(bts)
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003, "Object")
-							return
-						}
-					case "p":
-						z.Demote[za0003].Pool, bts, err = msgp.ReadIntBytes(bts)
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003, "Pool")
-							return
-						}
-					default:
-						bts, err = msgp.Skip(bts)
-						if err != nil {
-							err = msgp.WrapError(err, "Demote", za0003)
-							return
-						}
-					}
-				}
-			}
-		default:
-			bts, err = msgp.Skip(bts)
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	o = bts
-	return
-}
-
-// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
-func (z *accessShard) Msgsize() (s int) {
-	s = 1 + 2 + msgp.Int64Size + 3 + msgp.Int64Size + 2 + msgp.MapHeaderSize
-	if z.Entries != nil {
-		for za0001, za0002 := range z.Entries {
-			_ = za0002
-			s += msgp.StringPrefixSize + len(za0001) + za0002.Msgsize()
-		}
-	}
-	s += 2 + msgp.ArrayHeaderSize
-	for za0003 := range z.Demote {
-		s += 1 + 2 + msgp.StringPrefixSize + len(z.Demote[za0003].Bucket) + 2 + msgp.StringPrefixSize + len(z.Demote[za0003].Object) + 2 + msgp.IntSize
-	}
-	return
-}
-
-// DecodeMsg implements msgp.Decodable
-func (z *demoteCandidate) DecodeMsg(dc *msgp.Reader) (err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, err = dc.ReadMapHeader()
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, err = dc.ReadMapKeyPtr()
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "b":
-			z.Bucket, err = dc.ReadString()
-			if err != nil {
-				err = msgp.WrapError(err, "Bucket")
-				return
-			}
-		case "o":
-			z.Object, err = dc.ReadString()
-			if err != nil {
-				err = msgp.WrapError(err, "Object")
-				return
-			}
-		case "p":
-			z.Pool, err = dc.ReadInt()
-			if err != nil {
-				err = msgp.WrapError(err, "Pool")
-				return
-			}
-		default:
-			err = dc.Skip()
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	return
-}
-
-// EncodeMsg implements msgp.Encodable
-func (z demoteCandidate) EncodeMsg(en *msgp.Writer) (err error) {
-	// map header, size 3
-	// write "b"
-	err = en.Append(0x83, 0xa1, 0x62)
-	if err != nil {
-		return
-	}
-	err = en.WriteString(z.Bucket)
-	if err != nil {
-		err = msgp.WrapError(err, "Bucket")
-		return
-	}
-	// write "o"
-	err = en.Append(0xa1, 0x6f)
-	if err != nil {
-		return
-	}
-	err = en.WriteString(z.Object)
-	if err != nil {
-		err = msgp.WrapError(err, "Object")
-		return
-	}
-	// write "p"
-	err = en.Append(0xa1, 0x70)
-	if err != nil {
-		return
-	}
-	err = en.WriteInt(z.Pool)
-	if err != nil {
-		err = msgp.WrapError(err, "Pool")
-		return
-	}
-	return
-}
-
-// MarshalMsg implements msgp.Marshaler
-func (z demoteCandidate) MarshalMsg(b []byte) (o []byte, err error) {
-	o = msgp.Require(b, z.Msgsize())
-	// map header, size 3
-	// string "b"
-	o = append(o, 0x83, 0xa1, 0x62)
-	o = msgp.AppendString(o, z.Bucket)
-	// string "o"
-	o = append(o, 0xa1, 0x6f)
-	o = msgp.AppendString(o, z.Object)
-	// string "p"
-	o = append(o, 0xa1, 0x70)
-	o = msgp.AppendInt(o, z.Pool)
-	return
-}
-
-// UnmarshalMsg implements msgp.Unmarshaler
-func (z *demoteCandidate) UnmarshalMsg(bts []byte) (o []byte, err error) {
-	var field []byte
-	_ = field
-	var zb0001 uint32
-	zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
-	if err != nil {
-		err = msgp.WrapError(err)
-		return
-	}
-	for zb0001 > 0 {
-		zb0001--
-		field, bts, err = msgp.ReadMapKeyZC(bts)
-		if err != nil {
-			err = msgp.WrapError(err)
-			return
-		}
-		switch msgp.UnsafeString(field) {
-		case "b":
-			z.Bucket, bts, err = msgp.ReadStringBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Bucket")
-				return
-			}
-		case "o":
-			z.Object, bts, err = msgp.ReadStringBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Object")
-				return
-			}
-		case "p":
-			z.Pool, bts, err = msgp.ReadIntBytes(bts)
-			if err != nil {
-				err = msgp.WrapError(err, "Pool")
-				return
-			}
-		default:
-			bts, err = msgp.Skip(bts)
-			if err != nil {
-				err = msgp.WrapError(err)
-				return
-			}
-		}
-	}
-	o = bts
-	return
-}
-
-// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
-func (z demoteCandidate) Msgsize() (s int) {
-	s = 1 + 2 + msgp.StringPrefixSize + len(z.Bucket) + 2 + msgp.StringPrefixSize + len(z.Object) + 2 + msgp.IntSize
-	return
-}

cmd/ilm-access-tracker_gen_test.go

@@ -1,349 +0,0 @@
-// Code generated by github.com/tinylib/msgp DO NOT EDIT.
-
-package cmd
-
-import (
-	"bytes"
-	"testing"
-
-	"github.com/tinylib/msgp/msgp"
-)
-
-func TestMarshalUnmarshalaccessEntry(t *testing.T) {
-	v := accessEntry{}
-	bts, err := v.MarshalMsg(nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-	left, err := v.UnmarshalMsg(bts)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(left) > 0 {
-		t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
-	}
-
-	left, err = msgp.Skip(bts)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(left) > 0 {
-		t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
-	}
-}
-
-func BenchmarkMarshalMsgaccessEntry(b *testing.B) {
-	v := accessEntry{}
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		v.MarshalMsg(nil)
-	}
-}
-
-func BenchmarkAppendMsgaccessEntry(b *testing.B) {
-	v := accessEntry{}
-	bts := make([]byte, 0, v.Msgsize())
-	bts, _ = v.MarshalMsg(bts[0:0])
-	b.SetBytes(int64(len(bts)))
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		bts, _ = v.MarshalMsg(bts[0:0])
-	}
-}
-
-func BenchmarkUnmarshalaccessEntry(b *testing.B) {
-	v := accessEntry{}
-	bts, _ := v.MarshalMsg(nil)
-	b.ReportAllocs()
-	b.SetBytes(int64(len(bts)))
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		_, err := v.UnmarshalMsg(bts)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func TestEncodeDecodeaccessEntry(t *testing.T) {
-	v := accessEntry{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-
-	m := v.Msgsize()
-	if buf.Len() > m {
-		t.Log("WARNING: TestEncodeDecodeaccessEntry Msgsize() is inaccurate")
-	}
-
-	vn := accessEntry{}
-	err := msgp.Decode(&buf, &vn)
-	if err != nil {
-		t.Error(err)
-	}
-
-	buf.Reset()
-	msgp.Encode(&buf, &v)
-	err = msgp.NewReader(&buf).Skip()
-	if err != nil {
-		t.Error(err)
-	}
-}
-
-func BenchmarkEncodeaccessEntry(b *testing.B) {
-	v := accessEntry{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-	b.SetBytes(int64(buf.Len()))
-	en := msgp.NewWriter(msgp.Nowhere)
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		v.EncodeMsg(en)
-	}
-	en.Flush()
-}
-
-func BenchmarkDecodeaccessEntry(b *testing.B) {
-	v := accessEntry{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-	b.SetBytes(int64(buf.Len()))
-	rd := msgp.NewEndlessReader(buf.Bytes(), b)
-	dc := msgp.NewReader(rd)
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		err := v.DecodeMsg(dc)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func TestMarshalUnmarshalaccessShard(t *testing.T) {
-	v := accessShard{}
-	bts, err := v.MarshalMsg(nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-	left, err := v.UnmarshalMsg(bts)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(left) > 0 {
-		t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
-	}
-
-	left, err = msgp.Skip(bts)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(left) > 0 {
-		t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
-	}
-}
-
-func BenchmarkMarshalMsgaccessShard(b *testing.B) {
-	v := accessShard{}
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		v.MarshalMsg(nil)
-	}
-}
-
-func BenchmarkAppendMsgaccessShard(b *testing.B) {
-	v := accessShard{}
-	bts := make([]byte, 0, v.Msgsize())
-	bts, _ = v.MarshalMsg(bts[0:0])
-	b.SetBytes(int64(len(bts)))
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		bts, _ = v.MarshalMsg(bts[0:0])
-	}
-}
-
-func BenchmarkUnmarshalaccessShard(b *testing.B) {
-	v := accessShard{}
-	bts, _ := v.MarshalMsg(nil)
-	b.ReportAllocs()
-	b.SetBytes(int64(len(bts)))
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		_, err := v.UnmarshalMsg(bts)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func TestEncodeDecodeaccessShard(t *testing.T) {
-	v := accessShard{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-
-	m := v.Msgsize()
-	if buf.Len() > m {
-		t.Log("WARNING: TestEncodeDecodeaccessShard Msgsize() is inaccurate")
-	}
-
-	vn := accessShard{}
-	err := msgp.Decode(&buf, &vn)
-	if err != nil {
-		t.Error(err)
-	}
-
-	buf.Reset()
-	msgp.Encode(&buf, &v)
-	err = msgp.NewReader(&buf).Skip()
-	if err != nil {
-		t.Error(err)
-	}
-}
-
-func BenchmarkEncodeaccessShard(b *testing.B) {
-	v := accessShard{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-	b.SetBytes(int64(buf.Len()))
-	en := msgp.NewWriter(msgp.Nowhere)
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		v.EncodeMsg(en)
-	}
-	en.Flush()
-}
-
-func BenchmarkDecodeaccessShard(b *testing.B) {
-	v := accessShard{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-	b.SetBytes(int64(buf.Len()))
-	rd := msgp.NewEndlessReader(buf.Bytes(), b)
-	dc := msgp.NewReader(rd)
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		err := v.DecodeMsg(dc)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func TestMarshalUnmarshaldemoteCandidate(t *testing.T) {
-	v := demoteCandidate{}
-	bts, err := v.MarshalMsg(nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-	left, err := v.UnmarshalMsg(bts)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(left) > 0 {
-		t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
-	}
-
-	left, err = msgp.Skip(bts)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(left) > 0 {
-		t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
-	}
-}
-
-func BenchmarkMarshalMsgdemoteCandidate(b *testing.B) {
-	v := demoteCandidate{}
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		v.MarshalMsg(nil)
-	}
-}
-
-func BenchmarkAppendMsgdemoteCandidate(b *testing.B) {
-	v := demoteCandidate{}
-	bts := make([]byte, 0, v.Msgsize())
-	bts, _ = v.MarshalMsg(bts[0:0])
-	b.SetBytes(int64(len(bts)))
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		bts, _ = v.MarshalMsg(bts[0:0])
-	}
-}
-
-func BenchmarkUnmarshaldemoteCandidate(b *testing.B) {
-	v := demoteCandidate{}
-	bts, _ := v.MarshalMsg(nil)
-	b.ReportAllocs()
-	b.SetBytes(int64(len(bts)))
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		_, err := v.UnmarshalMsg(bts)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func TestEncodeDecodedemoteCandidate(t *testing.T) {
-	v := demoteCandidate{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-
-	m := v.Msgsize()
-	if buf.Len() > m {
-		t.Log("WARNING: TestEncodeDecodedemoteCandidate Msgsize() is inaccurate")
-	}
-
-	vn := demoteCandidate{}
-	err := msgp.Decode(&buf, &vn)
-	if err != nil {
-		t.Error(err)
-	}
-
-	buf.Reset()
-	msgp.Encode(&buf, &v)
-	err = msgp.NewReader(&buf).Skip()
-	if err != nil {
-		t.Error(err)
-	}
-}
-
-func BenchmarkEncodedemoteCandidate(b *testing.B) {
-	v := demoteCandidate{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-	b.SetBytes(int64(buf.Len()))
-	en := msgp.NewWriter(msgp.Nowhere)
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		v.EncodeMsg(en)
-	}
-	en.Flush()
-}
-
-func BenchmarkDecodedemoteCandidate(b *testing.B) {
-	v := demoteCandidate{}
-	var buf bytes.Buffer
-	msgp.Encode(&buf, &v)
-	b.SetBytes(int64(buf.Len()))
-	rd := msgp.NewEndlessReader(buf.Bytes(), b)
-	dc := msgp.NewReader(rd)
-	b.ReportAllocs()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		err := v.DecodeMsg(dc)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}

cmd/ilm-access-tracker_test.go

@@ -1,95 +0,0 @@
-// Copyright (c) 2015-2026 MinIO, Inc.
-
-package cmd
-
-import (
-	"math"
-	"testing"
-	"time"
-
-	"github.com/minio/minio/internal/config/ilm"
-)
-
-func TestAccessEntryRollAndHits(t *testing.T) {
-	entry := accessEntry{Bins: []uint32{3, 2, 1}, HeadAt: 100, LastAt: 107}
-	entry.rollTo(120, 10, 3)
-	want := []uint32{0, 0, 3}
-	for i := range want {
-		if entry.Bins[i] != want[i] {
-			t.Fatalf("bins = %v, want %v", entry.Bins, want)
-		}
-	}
-	if entry.HeadAt != 120 || entry.LastAt != 107 {
-		t.Fatalf("head/last = %d/%d", entry.HeadAt, entry.LastAt)
-	}
-
-	entry = accessEntry{Bins: []uint32{10, 20, 30}, HeadAt: 120}
-	if got := entry.hits(20*time.Second, 10); got != 30 {
-		t.Fatalf("20s hits = %d, want 30", got)
-	}
-	if got := entry.hits(21*time.Second, 10); got != 60 {
-		t.Fatalf("21s hits = %d, want 60", got)
-	}
-	entry.rollTo(200, 10, 3)
-	if got := entry.total(); got != 0 {
-		t.Fatalf("expired total = %d, want 0", got)
-	}
-}
-
-func TestAccessEntryMergeSaturates(t *testing.T) {
-	entry := accessEntry{Bins: []uint32{math.MaxUint32 - 1}}
-	entry.mergeFrom(accessEntry{Bins: []uint32{10}, LastAt: 50})
-	if entry.Bins[0] != math.MaxUint32 || entry.LastAt != 50 {
-		t.Fatalf("merged entry = %+v", entry)
-	}
-}
-
-func TestAccessTrackerEvictsColdest(t *testing.T) {
-	tracker := newAccessTracker()
-	live := map[string]accessEntry{
-		"a": {Bins: []uint32{1}, HeadAt: 100},
-		"b": {Bins: []uint32{5}, HeadAt: 100},
-		"c": {Bins: []uint32{3}, HeadAt: 100},
-	}
-	tracker.evict(live, 100, 10, ilm.Config{AccessBins: 1, AccessMaxTracked: 2})
-	if len(live) != 2 {
-		t.Fatalf("len = %d, want 2", len(live))
-	}
-	if _, ok := live["a"]; ok {
-		t.Fatal("coldest entry was retained")
-	}
-}
-
-func TestAccessKeyRoundTrip(t *testing.T) {
-	key := accessKey("bucket", "a/b/c")
-	bucket, object, ok := splitAccessKey(key)
-	if !ok || bucket != "bucket" || object != "a/b/c" {
-		t.Fatalf("split %q = %q/%q/%v", key, bucket, object, ok)
-	}
-}
-
-func TestAccessShardFresh(t *testing.T) {
-	const now = int64(1000)
-	if !accessShardFresh(now, accessShard{UpdatedAt: 950}, 100) {
-		t.Fatal("fresh shard rejected")
-	}
-	if accessShardFresh(now, accessShard{UpdatedAt: 899}, 100) {
-		t.Fatal("stale shard accepted")
-	}
-	if accessShardFresh(now, accessShard{UpdatedAt: 1101}, 100) {
-		t.Fatal("far-future shard accepted")
-	}
-	if accessShardFresh(now, accessShard{}, 100) {
-		t.Fatal("zero timestamp accepted")
-	}
-}
-
-func TestAccessTrackerRestoresDemoteCandidates(t *testing.T) {
-	tracker := newAccessTracker()
-	candidate := demoteCandidate{Bucket: "bucket", Object: "object", Pool: 1}
-	tracker.restoreDemoteCandidates([]demoteCandidate{candidate})
-	got := tracker.takeDemoteCandidates()
-	if len(got) != 1 || got[0] != candidate {
-		t.Fatalf("restored candidates = %+v, want %+v", got, candidate)
-	}
-}

cmd/ilm-config.go

@@ -19,7 +19,6 @@ package cmd
 
 import (
 	"sync"
-	"time"
 
 	"github.com/minio/minio/internal/config/ilm"
 )
@@ -28,16 +27,6 @@ var globalILMConfig = ilmConfig{
 	cfg: ilm.Config{
 		ExpirationWorkers: 100,
 		TransitionWorkers: 100,
-		// Access tiering stays off until configured, but the counter
-		// geometry must be sane from the start: the tracker divides by
-		// AccessBinWidth before any config is loaded.
-		AccessPromoteWatermark: 85,
-		AccessBinWidth:         time.Minute,
-		AccessBins:             12,
-		AccessFlush:            time.Minute,
-		AccessMinResidency:     24 * time.Hour,
-		AccessWorkers:          10,
-		AccessMaxTracked:       1000000,
 	},
 }
 
@@ -60,25 +49,6 @@ func (c *ilmConfig) getTransitionWorkers() int {
 	return c.cfg.TransitionWorkers
 }
 
-// accessCfg returns a copy of the access tiering settings. Callers take the
-// whole struct rather than one getter per field because the promotion and
-// demotion paths need a consistent view of several knobs at once.
-func (c *ilmConfig) accessCfg() ilm.Config {
-	c.mu.RLock()
-	defer c.mu.RUnlock()
-
-	return c.cfg
-}
-
-// accessTieringEnabled is the cheap gate used on the scanner path.
-func (c *ilmConfig) accessTieringEnabled() bool {
-	c.mu.RLock()
-	defer c.mu.RUnlock()
-
-	_, ok := c.cfg.HotPool()
-	return ok
-}
-
 func (c *ilmConfig) update(cfg ilm.Config) {
 	c.mu.Lock()
 	defer c.mu.Unlock()

cmd/lceventsrc_string.go

@@ -19,12 +19,11 @@ func _() {
 	_ = x[lcEventSrc_s3PutObject-8]
 	_ = x[lcEventSrc_s3CopyObject-9]
 	_ = x[lcEventSrc_s3CompleteMultipartUpload-10]
-	_ = x[lcEventSrc_AccessTier-11]
 }
 
-const _lcEventSrc_name = "NoneHealScannerDecomRebals3HeadObjects3GetObjects3ListObjectss3PutObjects3CopyObjects3CompleteMultipartUploadAccessTier"
+const _lcEventSrc_name = "NoneHealScannerDecomRebals3HeadObjects3GetObjects3ListObjectss3PutObjects3CopyObjects3CompleteMultipartUpload"
 
-var _lcEventSrc_index = [...]uint8{0, 4, 8, 15, 20, 25, 37, 48, 61, 72, 84, 109, 119}
+var _lcEventSrc_index = [...]uint8{0, 4, 8, 15, 20, 25, 37, 48, 61, 72, 84, 109}
 
 func (i lcEventSrc) String() string {
 	idx := int(i) - 0

cmd/metrics-v3-cluster-iam.go

@@ -34,6 +34,10 @@ const (
 	sinceLastSyncMillis                    = "since_last_sync_millis"
 	syncFailures                           = "sync_failures"
 	syncSuccesses                          = "sync_successes"
+	revocationRecords                      = "revocation_records"
+	revocationHealFailures                 = "revocation_heal_failures"
+	revocationHealDurationMillis           = "revocation_heal_duration_millis"
+	revocationHealLastSuccess              = "revocation_heal_last_success_timestamp_seconds"
 )
 
 var (
@@ -47,10 +51,20 @@ var (
 	sinceLastSyncMillisMD                    = NewCounterMD(sinceLastSyncMillis, "Time (in milliseconds) since last successful IAM data sync.")
 	syncFailuresMD                           = NewCounterMD(syncFailures, "Number of failed IAM data syncs since server start.")
 	syncSuccessesMD                          = NewCounterMD(syncSuccesses, "Number of successful IAM data syncs since server start.")
+	revocationRecordsMD                      = NewGaugeMD(revocationRecords, "Retained IAM deletion records and revocation boundaries in this node's index.")
+	revocationHealFailuresMD                 = NewCounterMD(revocationHealFailures, "Failed IAM revocation convergence passes since server start.")
+	revocationHealDurationMillisMD           = NewGaugeMD(revocationHealDurationMillis, "Duration of the last IAM revocation convergence pass in milliseconds.")
+	revocationHealLastSuccessMD              = NewGaugeMD(revocationHealLastSuccess, "Unix timestamp of the last successful IAM revocation convergence pass.")
 )
 
 // loadClusterIAMMetrics - `MetricsLoaderFn` for cluster IAM metrics.
 func loadClusterIAMMetrics(_ context.Context, m MetricValues, _ *metricsCache) error {
+	if globalIAMSys.Initialized() {
+		m.Set(revocationRecords, float64(globalIAMSys.store.revisionIndex().count()))
+	}
+	m.Set(revocationHealFailures, float64(globalSiteReplicationSys.iamRevisionMetrics.healFailures.Load()))
+	m.Set(revocationHealDurationMillis, float64(globalSiteReplicationSys.iamRevisionMetrics.healDurationMillis.Load()))
+	m.Set(revocationHealLastSuccess, float64(globalSiteReplicationSys.iamRevisionMetrics.healLastSuccess.Load()))
 	m.Set(lastSyncDurationMillis, float64(atomic.LoadUint64(&globalIAMSys.LastRefreshDurationMilliseconds)))
 	pluginAuthNMetrics := globalAuthNPlugin.Metrics()
 	m.Set(pluginAuthnServiceFailedRequestsMinute, float64(pluginAuthNMetrics.FailedRequests))

cmd/metrics-v3-ilm.go

@@ -26,17 +26,6 @@ const (
 	transitionActiveTasks          = "transition_active_tasks"
 	transitionPendingTasks         = "transition_pending_tasks"
 	transitionMissedImmediateTasks = "transition_missed_immediate_tasks"
-	accessTierActiveTasks          = "access_tier_active_tasks"
-	accessTierPendingTasks         = "access_tier_pending_tasks"
-	accessTierPromotionsTotal      = "access_tier_promotions_total"
-	accessTierDemotionsTotal       = "access_tier_demotions_total"
-	accessTierBytesMovedTotal      = "access_tier_bytes_moved_total"
-	accessTierFailuresTotal        = "access_tier_failures_total"
-	accessTierSkippedWatermark     = "access_tier_skipped_watermark_total"
-	accessTierSkippedMaxSize       = "access_tier_skipped_max_size_total"
-	accessTierSkippedQuota         = "access_tier_skipped_bucket_quota_total"
-	accessTierHotBytes             = "access_tier_hot_bytes"
-	accessTierSamplesDropped       = "access_tier_samples_dropped_total"
 	versionsScanned                = "versions_scanned"
 )
 
@@ -45,17 +34,6 @@ var (
 	ilmTransitionActiveTasksMD          = NewGaugeMD(transitionActiveTasks, "Number of active ILM transition tasks")
 	ilmTransitionPendingTasksMD         = NewGaugeMD(transitionPendingTasks, "Number of pending ILM transition tasks in the queue")
 	ilmTransitionMissedImmediateTasksMD = NewCounterMD(transitionMissedImmediateTasks, "Number of missed immediate ILM transition tasks")
-	ilmAccessTierActiveTasksMD          = NewGaugeMD(accessTierActiveTasks, "Number of active access-tier pool moves")
-	ilmAccessTierPendingTasksMD         = NewGaugeMD(accessTierPendingTasks, "Number of pending access-tier pool moves")
-	ilmAccessTierPromotionsTotalMD      = NewCounterMD(accessTierPromotionsTotal, "Total objects promoted by access-tier ILM")
-	ilmAccessTierDemotionsTotalMD       = NewCounterMD(accessTierDemotionsTotal, "Total objects demoted by access-tier ILM")
-	ilmAccessTierBytesMovedTotalMD      = NewCounterMD(accessTierBytesMovedTotal, "Total logical bytes moved by access-tier ILM")
-	ilmAccessTierFailuresTotalMD        = NewCounterMD(accessTierFailuresTotal, "Total failed access-tier ILM moves")
-	ilmAccessTierSkippedWatermarkMD     = NewCounterMD(accessTierSkippedWatermark, "Promotions skipped because the hot pool reached its watermark")
-	ilmAccessTierSkippedMaxSizeMD       = NewCounterMD(accessTierSkippedMaxSize, "Promotions skipped because the cluster hot-tier size cap was reached")
-	ilmAccessTierSkippedQuotaMD         = NewCounterMD(accessTierSkippedQuota, "Promotions skipped because the bucket hot-tier quota was reached")
-	ilmAccessTierHotBytesMD             = NewGaugeMD(accessTierHotBytes, "Logical bytes currently accounted to the hot tier", "bucket")
-	ilmAccessTierSamplesDroppedMD       = NewCounterMD(accessTierSamplesDropped, "GET samples dropped because the access tracker queue was full")
 	ilmVersionsScannedMD                = NewCounterMD(versionsScanned, "Total number of object versions checked for ILM actions since server start")
 )
 
@@ -69,21 +47,6 @@ func loadILMMetrics(_ context.Context, m MetricValues, _ *metricsCache) error {
 		m.Set(transitionPendingTasks, float64(globalTransitionState.PendingTasks()))
 		m.Set(transitionMissedImmediateTasks, float64(globalTransitionState.MissedImmediateTasks()))
 	}
-	if globalAccessTierState != nil {
-		m.Set(accessTierActiveTasks, float64(globalAccessTierState.ActiveTasks()))
-		m.Set(accessTierPendingTasks, float64(globalAccessTierState.PendingTasks()))
-		m.Set(accessTierPromotionsTotal, float64(globalAccessTierState.promotions.Load()))
-		m.Set(accessTierDemotionsTotal, float64(globalAccessTierState.demotions.Load()))
-		m.Set(accessTierBytesMovedTotal, float64(globalAccessTierState.bytesMoved.Load()))
-		m.Set(accessTierFailuresTotal, float64(globalAccessTierState.failures.Load()))
-		m.Set(accessTierSkippedWatermark, float64(globalAccessTierState.skippedWatermark.Load()))
-		m.Set(accessTierSkippedMaxSize, float64(globalAccessTierState.skippedMaxSize.Load()))
-		m.Set(accessTierSkippedQuota, float64(globalAccessTierState.skippedQuota.Load()))
-		for bucket, bytes := range globalAccessTierState.hotUsageSnapshot() {
-			m.Set(accessTierHotBytes, float64(bytes), "bucket", bucket)
-		}
-	}
-	m.Set(accessTierSamplesDropped, float64(globalAccessTracker.dropped.Load()))
 	m.Set(versionsScanned, float64(globalScannerMetrics.lifetime(scannerMetricILM)))
 
 	return nil

cmd/metrics-v3.go

@@ -323,6 +323,10 @@ func newMetricGroups(r *prometheus.Registry) *metricsV3Collection {
 			sinceLastSyncMillisMD,
 			syncFailuresMD,
 			syncSuccessesMD,
+			revocationRecordsMD,
+			revocationHealFailuresMD,
+			revocationHealDurationMillisMD,
+			revocationHealLastSuccessMD,
 		},
 		loadClusterIAMMetrics,
 	)
@@ -392,17 +396,6 @@ func newMetricGroups(r *prometheus.Registry) *metricsV3Collection {
 			ilmTransitionActiveTasksMD,
 			ilmTransitionPendingTasksMD,
 			ilmTransitionMissedImmediateTasksMD,
-			ilmAccessTierActiveTasksMD,
-			ilmAccessTierPendingTasksMD,
-			ilmAccessTierPromotionsTotalMD,
-			ilmAccessTierDemotionsTotalMD,
-			ilmAccessTierBytesMovedTotalMD,
-			ilmAccessTierFailuresTotalMD,
-			ilmAccessTierSkippedWatermarkMD,
-			ilmAccessTierSkippedMaxSizeMD,
-			ilmAccessTierSkippedQuotaMD,
-			ilmAccessTierHotBytesMD,
-			ilmAccessTierSamplesDroppedMD,
 			ilmVersionsScannedMD,
 		},
 		loadILMMetrics,

cmd/object-api-interface.go

@@ -122,10 +122,6 @@ type ObjectOptions struct {
 	SkipRebalancing bool
 
 	SrcPoolIdx int // set by PutObject/CompleteMultipart operations due to rebalance; used to prevent rebalance src, dst pools to be the same
-	// DstPoolIdx forces a data-movement write onto a specific server pool.
-	// It is ignored unless DataMovement is true; a pointer keeps pool zero
-	// distinguishable from the unset value.
-	DstPoolIdx *int
 
 	DataMovement bool // indicates an going decommisionning or rebalacing
 

cmd/object-api-options-replication_test.go

@@ -0,0 +1,113 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"encoding/base64"
+	"net/http"
+	"reflect"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	xhttp "github.com/minio/minio/internal/http"
+)
+
+func TestPutOptsFromHeadersReplicationTimestamps(t *testing.T) {
+	stamp := time.Date(2026, 9, 15, 1, 2, 3, 123456789, time.UTC)
+	context := base64.StdEncoding.EncodeToString([]byte(`{"purpose":"tag-replication"}`))
+	for _, encryption := range []struct {
+		name    string
+		headers map[string]string
+	}{
+		{name: "none"},
+		{name: "SSE-S3", headers: map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionAES}},
+		{name: "SSE-KMS", headers: map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionKMS}},
+		{name: "SSE-KMS-context", headers: map[string]string{
+			xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionKMS, xhttp.AmzServerSideEncryptionKmsID: "tag-replication-key",
+			xhttp.AmzServerSideEncryptionKmsContext: context,
+		}},
+		{name: "SSE-C", headers: ssecKeyHeaders([]byte("01234567890123456789012345678901"), false)},
+	} {
+		t.Run(encryption.name, func(t *testing.T) {
+			for _, trusted := range []bool{false, true} {
+				t.Run("trusted="+strconv.FormatBool(trusted), func(t *testing.T) {
+					for _, tagging := range []struct {
+						name, header string
+						want         time.Time
+						invalid      bool
+					}{
+						{name: "absent"},
+						{name: "nanoseconds", header: stamp.Format(time.RFC3339Nano), want: stamp},
+						{name: "offset-whitespace", header: " " + stamp.In(time.FixedZone("UTC+8", 8*60*60)).Format(time.RFC3339Nano) + " ", want: stamp},
+						{name: "invalid", header: "not-a-timestamp", invalid: true},
+					} {
+						t.Run(tagging.name, func(t *testing.T) {
+							for _, metadata := range []map[string]string{nil, {"x-amz-meta-test": "kept"}} {
+								hdr := make(http.Header)
+								wantEncryption := make(http.Header)
+								for key, value := range encryption.headers {
+									hdr.Set(key, value)
+									wantEncryption.Set(key, value)
+								}
+								hdr.Set(xhttp.MinIOSourceTaggingTimestamp, tagging.header)
+								hdr.Set(xhttp.MinIOSourceMTime, stamp.Add(-time.Hour).Format(time.RFC3339Nano))
+								hdr.Set(xhttp.MinIOSourceObjectRetentionTimestamp, stamp.Add(-time.Minute).Format(time.RFC3339Nano))
+								hdr.Set(xhttp.MinIOSourceObjectLegalHoldTimestamp, stamp.Add(-time.Second).Format(time.RFC3339Nano))
+								hdr.Set(xhttp.MinIOSourceETag, "source-etag")
+								opts, err := putOptsFromHeaders(t.Context(), hdr, metadata, trusted)
+								if trusted && tagging.invalid {
+									if err == nil || !strings.Contains(err.Error(), xhttp.MinIOSourceTaggingTimestamp) {
+										t.Fatalf("malformed trusted timestamp: got %v", err)
+									}
+									continue
+								}
+								if err != nil {
+									t.Fatal(err)
+								}
+								wantTag, wantMTime, wantRetention, wantLegalhold, wantETag := time.Time{}, time.Time{}, time.Time{}, time.Time{}, ""
+								if trusted {
+									wantTag, wantMTime = tagging.want, stamp.Add(-time.Hour)
+									wantRetention, wantLegalhold, wantETag = stamp.Add(-time.Minute), stamp.Add(-time.Second), "source-etag"
+								}
+								if !opts.ReplicationSourceTaggingTimestamp.Equal(wantTag) {
+									t.Errorf("tag timestamp=%s, want %s", opts.ReplicationSourceTaggingTimestamp, wantTag)
+								}
+								if !opts.MTime.Equal(wantMTime) || !opts.ReplicationSourceRetentionTimestamp.Equal(wantRetention) ||
+									!opts.ReplicationSourceLegalholdTimestamp.Equal(wantLegalhold) || opts.PreserveETag != wantETag || opts.ReplicationRequest != trusted {
+									t.Error("other source fields did not preserve the replication trust boundary")
+								}
+								if opts.UserDefined == nil || (metadata != nil && !reflect.DeepEqual(opts.UserDefined, metadata)) {
+									t.Errorf("metadata=%v, want nonnil map preserving %v", opts.UserDefined, metadata)
+								}
+								gotEncryption := make(http.Header)
+								if opts.ServerSideEncryption != nil {
+									opts.ServerSideEncryption.Marshal(gotEncryption)
+								}
+								if !reflect.DeepEqual(gotEncryption, wantEncryption) {
+									t.Errorf("SSE headers=%v, want %v", gotEncryption, wantEncryption)
+								}
+							}
+						})
+					}
+				})
+			}
+		})
+	}
+}

cmd/object-api-options.go

@@ -452,11 +452,11 @@ func putOptsFromHeaders(ctx context.Context, hdr http.Header, metadata map[strin
 			MTime:                mtime,
 			PreserveETag:         etag,
 			ReplicationRequest:   trustedReplication,
-			// The Object Lock timestamps order replicated retention and legal
+			// These timestamps order replicated retention, legal hold and tagging
-			// hold updates. Dropping them here would leave every update on an
+			// updates on an SSE-KMS destination.
-			// SSE-KMS destination unordered.
 			ReplicationSourceLegalholdTimestamp: lholdtimestmp,
 			ReplicationSourceRetentionTimestamp: retaintimestmp,
+			ReplicationSourceTaggingTimestamp:   taggingtimestmp,
 		}
 		return op, nil
 	}

cmd/object-copy-replication-tagging_test.go

@@ -0,0 +1,143 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"github.com/minio/minio/internal/auth"
+	"github.com/minio/minio/internal/crypto"
+	xhttp "github.com/minio/minio/internal/http"
+	"github.com/minio/minio/internal/kms"
+)
+
+// TestAPICopyObjectReplicaTaggingTimestampUnderKMS covers signed replica COPY
+// requests through encryption, metadata replacement and disk persistence. Both
+// the single-disk and 16-disk fixtures are single-pool backends.
+func TestAPICopyObjectReplicaTaggingTimestampUnderKMS(t *testing.T) {
+	defer DetectTestLeak(t)()
+	ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testAPICopyObjectReplicaTaggingTimestampUnderKMS})
+}
+
+func testAPICopyObjectReplicaTaggingTimestampUnderKMS(obj ObjectLayer, instance, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
+	// Ignore the host free-space percentage while retaining real disk I/O.
+	for _, pool := range obj.(*erasureServerPools).serverPools {
+		for _, set := range pool.sets {
+			original := set.getDisks
+			disks := append([]StorageAPI(nil), original()...)
+			for i := range disks {
+				disks[i] = tagTestCapacityDisk{StorageAPI: disks[i]}
+			}
+			set.getDisks = func() []StorageAPI { return disks }
+			defer func() { set.getDisks = original }()
+		}
+	}
+	oldKMS, oldAuto := GlobalKMS, globalAutoEncryption
+	GlobalKMS = kms.NewStub("replica-tags-key")
+	globalAutoEncryption = false
+	defer func() { GlobalKMS, globalAutoEncryption = oldKMS, oldAuto }()
+	if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
+		t.Fatal(err)
+	}
+	const tsKey = ReservedMetadataPrefixLower + TaggingTimestamp
+	stamp := time.Date(2026, 9, 15, 1, 0, 0, 123456789, time.UTC)
+	for _, mode := range []string{"none", "explicit-sse-s3", "explicit-kms", "auto-kms", "bucket-kms"} {
+		t.Run(instance+"/"+mode, func(t *testing.T) {
+			globalAutoEncryption = mode == "auto-kms"
+			if mode == "bucket-kms" {
+				sseXML := []byte(`<ServerSideEncryptionConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>aws:kms</SSEAlgorithm><KMSMasterKeyID>replica-tags-key</KMSMasterKeyID></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>`)
+				if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketSSEConfig, sseXML); err != nil {
+					t.Fatal(err)
+				}
+			}
+			const data = "encrypted replica copy remains readable"
+			oi, err := obj.PutObject(t.Context(), bucket, mode, mustGetPutObjReader(t, bytes.NewReader([]byte(data)), int64(len(data)), "", ""), ObjectOptions{
+				Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "key=old", tsKey: stamp.Format(time.RFC3339Nano)},
+			})
+			if err != nil {
+				t.Fatal(err)
+			}
+			for _, event := range []struct {
+				name, tags, wantTags string
+				delta, wantDelta     time.Duration
+				missingTimestamp     bool
+			}{
+				{"newer", "key=new", "key=new", 2, 2, false},
+				{"stale", "key=stale", "key=new", 1, 2, false},
+				{"duplicate", "key=new", "key=new", 2, 2, false},
+				{"newer-again", "key=latest", "key=latest", 3, 3, false},
+				{"missing-timestamp", "key=unordered", "key=latest", 0, 3, true},
+			} {
+				headers := map[string]string{
+					xhttp.AmzCopySource:        "/" + bucket + "/" + mode + "?versionId=" + oi.VersionID,
+					xhttp.AmzMetadataDirective: "REPLACE", xhttp.AmzTagDirective: "REPLACE",
+					xhttp.AmzObjectTagging: event.tags, xhttp.MinIOSourceReplicationRequest: "true",
+					xhttp.AmzBucketReplicationStatus: "REPLICA", xhttp.MinIOSourceTaggingTimestamp: stamp.Add(event.delta).Format(time.RFC3339Nano),
+					xhttp.MinIOSourceMTime: oi.ModTime.Format(time.RFC3339Nano), xhttp.MinIOSourceETag: oi.ETag,
+				}
+				if event.missingTimestamp {
+					delete(headers, xhttp.MinIOSourceTaggingTimestamp)
+				}
+				if mode == "explicit-sse-s3" {
+					headers[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionAES
+				}
+				if mode == "explicit-kms" {
+					headers[xhttp.AmzServerSideEncryption] = "aws:kms"
+					headers[xhttp.AmzServerSideEncryptionKmsID] = "replica-tags-key"
+				}
+				req, err := newTestSignedRequestV4(http.MethodPut, "/"+bucket+"/"+mode+"?versionId="+oi.VersionID, 0, nil, creds.AccessKey, creds.SecretKey, headers)
+				if err != nil {
+					t.Fatal(err)
+				}
+				w := httptest.NewRecorder()
+				router.ServeHTTP(w, req)
+				if w.Code != http.StatusOK {
+					t.Fatalf("%s: COPY %d %s", event.name, w.Code, w.Body.String())
+				}
+				got, err := obj.GetObjectInfo(t.Context(), bucket, mode, ObjectOptions{VersionID: oi.VersionID})
+				if err != nil {
+					t.Fatal(err)
+				}
+				t.Logf("%s: tags=%q timestamp=%q kms=%v", event.name, got.UserTags, got.UserDefined[tsKey], crypto.S3KMS.IsEncrypted(got.UserDefined))
+				if got.UserTags != event.wantTags || got.UserDefined[tsKey] != stamp.Add(event.wantDelta).Format(time.RFC3339Nano) {
+					t.Errorf("%s: incorrect persisted tags/timestamp", event.name)
+				}
+				wantKMS := mode == "explicit-kms" || mode == "auto-kms" || mode == "bucket-kms"
+				if crypto.S3KMS.IsEncrypted(got.UserDefined) != wantKMS || crypto.S3.IsEncrypted(got.UserDefined) != (mode == "explicit-sse-s3") {
+					t.Errorf("%s: unexpected destination encryption", event.name)
+				}
+				if got.VersionID != oi.VersionID {
+					t.Errorf("%s: version=%q, want %q", event.name, got.VersionID, oi.VersionID)
+				}
+				req, err = newTestSignedRequestV4(http.MethodGet, "/"+bucket+"/"+mode+"?versionId="+oi.VersionID, 0, nil, creds.AccessKey, creds.SecretKey, nil)
+				if err != nil {
+					t.Fatal(err)
+				}
+				w = httptest.NewRecorder()
+				router.ServeHTTP(w, req)
+				if w.Code != http.StatusOK || w.Body.String() != data {
+					t.Fatalf("%s: GET %d %q", event.name, w.Code, w.Body.String())
+				}
+			}
+		})
+	}
+}

cmd/object-handlers.go

@@ -576,8 +576,6 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 		return
 	}
 
-	globalAccessTracker.note(bucket, object)
-
 	// Notify object accessed via a GET request.
 	sendEvent(eventArgs{
 		EventName:    event.ObjectAccessedGet,

cmd/peer-rest-server.go

@@ -204,7 +204,9 @@ func (s *peerRESTServer) DeleteServiceAccountHandler(mss *grid.MSS) (np grid.NoP
 		return np, grid.NewRemoteErr(errors.New("service account name is missing"))
 	}
 
-	if err := globalIAMSys.DeleteServiceAccount(context.Background(), accessKey, false); err != nil {
+	ctx, cancel := context.WithTimeout(GlobalContext, defaultContextTimeout)
+	defer cancel()
+	if err := globalIAMSys.LoadServiceAccount(ctx, accessKey); err != nil {
 		return np, grid.NewRemoteErr(err)
 	}
 
@@ -230,7 +232,8 @@ func (s *peerRESTServer) LoadServiceAccountHandler(mss *grid.MSS) (np grid.NoPay
 	return np, nerr
 }
 
-// DeleteUserHandler - deletes a user on the server.
+// DeleteUserHandler reloads the state committed by another node. A delayed
+// notification must not delete an identity recreated since that commit.
 func (s *peerRESTServer) DeleteUserHandler(mss *grid.MSS) (np grid.NoPayload, nerr *grid.RemoteErr) {
 	objAPI := newObjectLayerFn()
 	if objAPI == nil {
@@ -242,7 +245,9 @@ func (s *peerRESTServer) DeleteUserHandler(mss *grid.MSS) (np grid.NoPayload, ne
 		return np, grid.NewRemoteErr(errors.New("username is missing"))
 	}
 
-	if err := globalIAMSys.DeleteUser(context.Background(), accessKey, false); err != nil {
+	ctx, cancel := context.WithTimeout(GlobalContext, defaultContextTimeout)
+	defer cancel()
+	if err := globalIAMSys.LoadUserAfterDelete(ctx, accessKey); err != nil {
 		return np, grid.NewRemoteErr(err)
 	}
 
@@ -271,7 +276,9 @@ func (s *peerRESTServer) LoadUserHandler(mss *grid.MSS) (np grid.NoPayload, nerr
 		userType = stsUser
 	}
 
-	if err = globalIAMSys.LoadUser(context.Background(), objAPI, accessKey, userType); err != nil {
+	ctx, cancel := context.WithTimeout(GlobalContext, defaultContextTimeout)
+	defer cancel()
+	if err = globalIAMSys.LoadUser(ctx, objAPI, accessKey, userType); err != nil {
 		return np, grid.NewRemoteErr(err)
 	}
 

cmd/replication-content-encoding_test.go

@@ -0,0 +1,276 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"encoding/xml"
+	"maps"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"reflect"
+	"strings"
+	"testing"
+
+	"github.com/minio/minio/internal/auth"
+	xhttp "github.com/minio/minio/internal/http"
+)
+
+// Exercise authenticated handlers and actual disk metadata, including the
+// response headers consumers see after replication has completed.
+func TestAPIReplicaContentEncoding(t *testing.T) {
+	defer DetectTestLeak(t)()
+	ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testAPIReplicaContentEncoding})
+}
+
+func testAPIReplicaContentEncoding(obj ObjectLayer, instance, bucket string, router http.Handler, owner auth.Credentials, t *testing.T) {
+	ordinary := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject","s3:GetObject"`)
+	replicator := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject","s3:GetObject","s3:ReplicateObject"`)
+	for _, mode := range []string{"ordinary", "untrusted-marker", "replica"} {
+		for _, tc := range []struct{ name, wire, want string }{
+			{"bare", "aws-chunked", ""}, {"mixed", "aws-chunked,gzip", "gzip"}, {"gzip", "gzip", "gzip"},
+		} {
+			for _, operation := range []string{"put", "copy-replace", "multipart"} {
+				t.Run(instance+"/"+mode+"/"+tc.name+"/"+operation, func(t *testing.T) {
+					object := mode + "/" + tc.name + "/" + operation
+					payload := replicaEncodingPayload(t, tc.want)
+					creds := ordinary
+					headers := map[string]string{xhttp.ContentEncoding: tc.wire, xhttp.ContentType: "application/octet-stream", "X-Amz-Meta-Source": "encoding-test"}
+					if mode != "ordinary" {
+						headers[xhttp.MinIOSourceReplicationRequest] = "true"
+					}
+					if mode == "replica" {
+						creds = replicator
+						headers[xhttp.AmzBucketReplicationStatus] = "REPLICA"
+					}
+					send := func(method, target string, data []byte, hdrs map[string]string) *httptest.ResponseRecorder {
+						t.Helper()
+						req, err := newTestSignedRequestV4(method, target, int64(len(data)), bytes.NewReader(data), creds.AccessKey, creds.SecretKey, hdrs)
+						if err != nil {
+							t.Fatal(err)
+						}
+						return replicaEncodingServe(t, router, req, http.StatusOK)
+					}
+					switch operation {
+					case "put":
+						if strings.Contains(tc.wire, "aws-chunked") {
+							req := replicaEncodingStream(t, getPutObjectURL("", bucket, object), payload, creds, headers)
+							replicaEncodingServe(t, router, req, http.StatusOK)
+						} else {
+							send(http.MethodPut, getPutObjectURL("", bucket, object), payload, headers)
+						}
+					case "copy-replace":
+						source := object + "-source"
+						if _, err := obj.PutObject(t.Context(), bucket, source, mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""), ObjectOptions{}); err != nil {
+							t.Fatal(err)
+						}
+						headers[xhttp.AmzCopySource] = url.QueryEscape("/" + bucket + "/" + source)
+						headers[xhttp.AmzMetadataDirective] = replaceDirective
+						send(http.MethodPut, getCopyObjectURL("", bucket, object), nil, headers)
+					case "multipart":
+						rec := send(http.MethodPost, getNewMultipartURL("", bucket, object), nil, headers)
+						var init InitiateMultipartUploadResponse
+						if err := xml.Unmarshal(rec.Body.Bytes(), &init); err != nil {
+							t.Fatal(err)
+						}
+						// Part/completion metadata must not replace the encoding saved at initiation.
+						partHeaders := map[string]string{xhttp.ContentEncoding: "br"}
+						if mode == "replica" {
+							partHeaders[xhttp.MinIOSourceReplicationRequest] = "true"
+							partHeaders[xhttp.AmzBucketReplicationStatus] = "REPLICA"
+						}
+						part := send(http.MethodPut, getPutObjectPartURL("", bucket, object, init.UploadID, "1"), payload, partHeaders)
+						partETags := part.Header()[xhttp.ETag]
+						if len(partETags) != 1 {
+							t.Fatalf("missing part ETag: %#v", part.Header())
+						}
+						complete, err := xml.Marshal(CompleteMultipartUpload{Parts: []CompletePart{{PartNumber: 1, ETag: canonicalizeETag(partETags[0])}}})
+						if err != nil {
+							t.Fatal(err)
+						}
+						send(http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, init.UploadID), complete, partHeaders)
+					}
+					assertReplicaEncodingObject(t, obj, router, owner, bucket, object, tc.want, payload)
+					info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
+					if err != nil {
+						t.Fatal(err)
+					}
+					if got := info.UserDefined[xhttp.AmzBucketReplicationStatus]; (got == "REPLICA") != (mode == "replica") {
+						t.Errorf("replica status %q for mode %s", got, mode)
+					}
+					if info.ContentType != "application/octet-stream" {
+						t.Errorf("content-type=%q", info.ContentType)
+					}
+					if value, ok := caseInsensitiveMap(info.UserDefined).Lookup("x-amz-meta-source"); !ok || value != "encoding-test" {
+						t.Errorf("user metadata lost: %#v", info.UserDefined)
+					}
+				})
+			}
+		}
+	}
+	t.Run(instance+"/unauthorized-replica", func(t *testing.T) {
+		object := "denied-replica"
+		req := replicaEncodingStream(t, getPutObjectURL("", bucket, object), []byte("denied"), ordinary, map[string]string{xhttp.ContentEncoding: "aws-chunked", xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA"})
+		rec := replicaEncodingServe(t, router, req, http.StatusForbidden)
+		var response APIErrorResponse
+		if err := xml.Unmarshal(rec.Body.Bytes(), &response); err != nil {
+			t.Fatal(err)
+		}
+		if response.Code != "AccessDenied" {
+			t.Fatalf("expected permission denial, got %s", response.Code)
+		}
+		if _, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); err == nil {
+			t.Error("denied replica created an object")
+		}
+	})
+}
+
+func replicaEncodingPayload(t *testing.T, encoding string) []byte {
+	t.Helper()
+	data := bytes.Repeat([]byte("replica encoding payload\n"), 128)
+	if encoding != "gzip" {
+		return data
+	}
+	var b bytes.Buffer
+	w := gzip.NewWriter(&b)
+	if _, err := w.Write(data); err != nil {
+		t.Fatal(err)
+	}
+	if err := w.Close(); err != nil {
+		t.Fatal(err)
+	}
+	return b.Bytes()
+}
+
+func replicaEncodingStream(t *testing.T, target string, data []byte, creds auth.Credentials, headers map[string]string) *http.Request {
+	t.Helper()
+	const chunkSize = 64
+	body := bytes.NewReader(data)
+	req, err := newTestStreamingRequest(http.MethodPut, target, int64(len(data)), chunkSize, body)
+	if err != nil {
+		t.Fatal(err)
+	}
+	for k, v := range headers {
+		req.Header.Set(k, v)
+	}
+	now := UTCNow()
+	signature, err := signStreamingRequest(req, creds.AccessKey, creds.SecretKey, now)
+	if err != nil {
+		t.Fatal(err)
+	}
+	req, err = assembleStreamingChunks(req, body, chunkSize, creds.SecretKey, signature, now)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return req
+}
+
+func replicaEncodingServe(t *testing.T, router http.Handler, req *http.Request, want int) *httptest.ResponseRecorder {
+	t.Helper()
+	rec := httptest.NewRecorder()
+	router.ServeHTTP(rec, req)
+	if rec.Code != want {
+		t.Fatalf("%s %s: status=%d want=%d body=%s", req.Method, req.URL, rec.Code, want, rec.Body.String())
+	}
+	return rec
+}
+
+func assertReplicaEncodingObject(t *testing.T, obj ObjectLayer, router http.Handler, creds auth.Credentials, bucket, object, encoding string, data []byte) {
+	t.Helper()
+	info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if info.ContentEncoding != encoding {
+		t.Errorf("persisted content-encoding=%q want=%q", info.ContentEncoding, encoding)
+	}
+	if encoding == "" {
+		if _, present := info.UserDefined["content-encoding"]; present {
+			t.Error("transport-only content-encoding key persisted")
+		}
+	}
+	for _, method := range []string{http.MethodGet, http.MethodHead} {
+		req, err := newTestSignedRequestV4(method, getPutObjectURL("", bucket, object), 0, nil, creds.AccessKey, creds.SecretKey, nil)
+		if err != nil {
+			t.Fatal(err)
+		}
+		rec := replicaEncodingServe(t, router, req, http.StatusOK)
+		if got := rec.Header().Get(xhttp.ContentEncoding); got != encoding {
+			t.Errorf("%s content-encoding=%q want=%q", method, got, encoding)
+		}
+		if encoding == "" {
+			if _, present := rec.Header()[xhttp.ContentEncoding]; present {
+				t.Errorf("%s sent an empty/transport encoding header", method)
+			}
+		}
+		if method == http.MethodGet && !bytes.Equal(rec.Body.Bytes(), data) {
+			t.Errorf("GET body differs: got %d bytes want %d", rec.Body.Len(), len(data))
+		}
+	}
+}
+
+func TestAPISnowballReplicaContentEncoding(t *testing.T) {
+	defer DetectTestLeak(t)()
+	ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
+		for _, tc := range []struct {
+			name string
+			pax  map[string]string
+			want string
+		}{
+			{name: "no-pax"},
+			{name: "pax-without-encoding", pax: map[string]string{"minio.metadata.Content-Type": "application/octet-stream"}},
+			{name: "pax-bare", pax: map[string]string{"minio.metadata.Content-Encoding": "aws-chunked"}},
+			{name: "pax-mixed", pax: map[string]string{"minio.metadata.Content-Encoding": "aws-chunked,gzip"}, want: "gzip"},
+		} {
+			t.Run(instance+"/"+tc.name, func(t *testing.T) {
+				object := "snowball/" + tc.name
+				data := replicaEncodingPayload(t, tc.want)
+				var archive bytes.Buffer
+				tw := tar.NewWriter(&archive)
+				if err := tw.WriteHeader(&tar.Header{Name: object, Mode: 0o600, Size: int64(len(data)), PAXRecords: tc.pax}); err != nil {
+					t.Fatal(err)
+				}
+				if _, err := tw.Write(data); err != nil {
+					t.Fatal(err)
+				}
+				if err := tw.Close(); err != nil {
+					t.Fatal(err)
+				}
+				var ordinaryMetadata map[string]string
+				// An unauthorized entry in a REPLICA request is rejected. Compare the
+				// same archive across ordinary and authorized replica requests instead.
+				for _, replica := range []bool{false, true} {
+					headers := map[string]string{
+						xhttp.ContentEncoding: "aws-chunked", xhttp.AmzSnowballExtract: "true",
+						xhttp.ContentType: "application/x-tar", xhttp.CacheControl: "max-age=123",
+						"X-Amz-Meta-Archive": "outer-request",
+					}
+					if replica {
+						headers[xhttp.MinIOSourceReplicationRequest] = "true"
+						headers[xhttp.AmzBucketReplicationStatus] = "REPLICA"
+					}
+					req := replicaEncodingStream(t, getPutObjectURL("", bucket, "archive.tar"), archive.Bytes(), creds, headers)
+					replicaEncodingServe(t, router, req, http.StatusOK)
+					assertReplicaEncodingObject(t, obj, router, creds, bucket, object, tc.want, data)
+					info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
+					if err != nil {
+						t.Fatal(err)
+					}
+					metadata := maps.Clone(info.UserDefined)
+					for _, key := range []string{xhttp.AmzBucketReplicationStatus, ReservedMetadataPrefixLower + ReplicaStatus, ReservedMetadataPrefixLower + ReplicaTimestamp, "etag"} {
+						delete(metadata, key)
+					}
+					if !replica {
+						ordinaryMetadata = metadata
+					} else if !reflect.DeepEqual(metadata, ordinaryMetadata) {
+						t.Errorf("replica inherited ordinary archive metadata: got %#v want %#v", metadata, ordinaryMetadata)
+					}
+				}
+			})
+		}
+	}})
+}

cmd/replication-metadata_test.go

@@ -0,0 +1,121 @@
+// Copyright (c) 2026 PGSTY
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package cmd
+
+import (
+	"maps"
+	"net/http"
+	"net/textproto"
+	"reflect"
+	"strings"
+	"testing"
+
+	xhttp "github.com/minio/minio/internal/http"
+)
+
+func TestExtractReplicationMetadataPreservesNormalizedMetadata(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		wire []string
+		want string
+	}{
+		{name: "absent"},
+		{name: "transport-only", wire: []string{"aws-chunked"}},
+		{name: "mixed", wire: []string{"aws-chunked,gzip"}, want: "gzip"},
+		{name: "gzip", wire: []string{"gzip"}, want: "gzip"},
+		{name: "transport-last", wire: []string{"gzip,aws-chunked"}, want: "gzip"},
+		{name: "multiple-values", wire: []string{"aws-chunked", "gzip"}, want: "gzip"},
+		// Preserve the existing exact-token grammar; whitespace is not normalized here.
+		{name: "space-before-gzip", wire: []string{"aws-chunked, gzip"}, want: " gzip"},
+		{name: "space-before-transport", wire: []string{"gzip, aws-chunked"}, want: "gzip, aws-chunked"},
+	} {
+		for _, lowercase := range []bool{false, true} {
+			name := tc.name + "/canonical"
+			if lowercase {
+				name = tc.name + "/lowercase"
+			}
+			t.Run(name, func(t *testing.T) {
+				header := http.Header{
+					"Content-Type":      []string{"application/octet-stream"},
+					"X-Amz-Meta-Source": []string{"raw"},
+					"X-Minio-Replication-Server-Side-Encryption-Sealed-Key":     []string{"sealed-key"},
+					"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
+					"X-Minio-Replication-Server-Side-Encryption-Iv":             []string{"iv"},
+					"X-Minio-Replication-Encrypted-Multipart":                   []string{""},
+					"X-Minio-Replication-Actual-Object-Size":                    []string{"1"},
+					ReplicationSsecChecksumHeader:                               []string{"checksum"},
+					xhttp.AmzMetaUnencryptedContentLength:                       []string{"injected-length"},
+					xhttp.AmzMetaUnencryptedContentMD5:                          []string{"injected-md5"},
+				}
+				if tc.wire != nil {
+					header[xhttp.ContentEncoding] = tc.wire
+				}
+				if lowercase {
+					h := make(http.Header, len(header))
+					for k, v := range header {
+						h[strings.ToLower(k)] = v
+					}
+					header = h
+				}
+				metadata, err := extractMetadata(t.Context(), textproto.MIMEHeader(header))
+				if err != nil {
+					t.Fatal(err)
+				}
+				if metadata["content-encoding"] != tc.want {
+					t.Fatalf("ordinary encoding=%q want=%q", metadata["content-encoding"], tc.want)
+				}
+				for _, internal := range replicationToInternalHeaders {
+					if _, ok := metadata[internal]; ok {
+						t.Fatalf("ordinary request accepted internal field %s", internal)
+					}
+				}
+				// Callers own ordinary metadata and may transform it after extraction.
+				metadata["content-type"] = "application/wasm"
+				for k := range metadata {
+					if strings.EqualFold(k, "x-amz-meta-source") {
+						metadata[k] = "caller"
+					}
+				}
+				want := maps.Clone(metadata)
+				maps.Copy(want, map[string]string{
+					"X-Minio-Internal-Server-Side-Encryption-Sealed-Key":     "sealed-key",
+					"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "DAREv2-HMAC-SHA256",
+					"X-Minio-Internal-Server-Side-Encryption-Iv":             "iv",
+					"X-Minio-Internal-Encrypted-Multipart":                   "",
+					"X-Minio-Internal-Actual-Object-Size":                    "1",
+					ReplicationSsecChecksumHeader:                            "checksum",
+				})
+				for range 2 {
+					if err := extractReplicationMetadataFromMime(t.Context(), textproto.MIMEHeader(header), metadata); err != nil {
+						t.Fatal(err)
+					}
+					if !reflect.DeepEqual(metadata, want) {
+						t.Errorf("restoration changed normalized metadata: got %#v want %#v", metadata, want)
+					}
+				}
+				if tc.want == "" {
+					if _, present := metadata["content-encoding"]; present {
+						t.Error("transport-only content-encoding key restored")
+					}
+				}
+				for _, key := range []string{xhttp.AmzMetaUnencryptedContentLength, xhttp.AmzMetaUnencryptedContentMD5} {
+					if _, present := caseInsensitiveMap(metadata).Lookup(key); present {
+						t.Errorf("redacted metadata restored: %s", key)
+					}
+				}
+			})
+		}
+	}
+}
+
+func TestExtractReplicationMetadataNilHeader(t *testing.T) {
+	metadata := map[string]string{"content-type": "application/wasm"}
+	want := maps.Clone(metadata)
+	if err := extractReplicationMetadataFromMime(t.Context(), nil, metadata); err != errInvalidArgument {
+		t.Fatalf("nil header: got %v want %v", err, errInvalidArgument)
+	}
+	if !reflect.DeepEqual(metadata, want) {
+		t.Fatalf("nil input changed metadata: %#v", metadata)
+	}
+}

cmd/server-main.go

@@ -501,7 +501,6 @@ func initAllSubsystems(ctx context.Context) {
 	globalTierConfigMgr = NewTierConfigMgr()
 
 	globalTransitionState = newTransitionState(GlobalContext)
-	globalAccessTierState = newAccessTierState(GlobalContext)
 	globalSiteResyncMetrics = newSiteResyncMetrics(GlobalContext)
 }
 
@@ -1070,10 +1069,6 @@ func serverMain(ctx *cli.Context) {
 		bootstrapTrace("globalTransitionState.Init", func() {
 			globalTransitionState.Init(newObject)
 		})
-		bootstrapTrace("globalAccessTierState.Init", func() {
-			globalAccessTierState.Init(newObject)
-			go globalAccessTracker.run(GlobalContext, newObject)
-		})
 
 		go func() {
 			// Initialize transition tier configuration manager

cmd/site-replication-metadata-gate_test.go

@@ -274,6 +274,10 @@ func TestBucketMetadataInitialSyncPhysicalCreated(t *testing.T) {
 					}
 					events = append(events, event)
 				}
+				if r.URL.Path == "/minio/admin/v3/site-replication/peer/iam-revisions" {
+					_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "initial-peer", Instance: "initial-boot", Digest: "ack"}})
+					return
+				}
 				w.WriteHeader(http.StatusOK)
 			}))
 			defer peer.Close()

cmd/site-replication.go

@@ -28,6 +28,7 @@ import (
 	"fmt"
 	"maps"
 	"math/rand"
+	"net/http"
 	"net/url"
 	"reflect"
 	"runtime"
@@ -209,7 +210,11 @@ type SiteReplicationSys struct {
 	// In-memory and persisted multi-site replication state.
 	state srState
 
-	iamMetaCache srIAMCache
+	iamMetaCache        srIAMCache
+	healOnce            sync.Once // Configuration reloads must not spawn more healing loops.
+	iamHealMu           sync.Mutex
+	iamRevisionProgress map[string]iamRevisionProgress
+	iamRevisionMetrics  iamRevisionMetrics
 }
 
 type srState srStateV1
@@ -234,7 +239,7 @@ type srStateData struct {
 
 // Init - initialize the site replication manager.
 func (c *SiteReplicationSys) Init(ctx context.Context, objAPI ObjectLayer) error {
-	go c.startHealRoutine(ctx, objAPI)
+	c.healOnce.Do(func() { go c.startHealRoutine(ctx, objAPI) })
 	r := rand.New(rand.NewSource(time.Now().UnixNano()))
 	for {
 		err := c.loadFromDisk(ctx, objAPI)
@@ -1257,13 +1262,36 @@ func (c *SiteReplicationSys) IAMChangeHook(ctx context.Context, item madmin.SRIA
 		return nil
 	}
 
+	if path := iamDeletionPath(item); path != "" {
+		r, err := loadIAMRevision(ctx, globalIAMSys.store, path)
+		if err != nil {
+			return err
+		}
+		if !r.Deleted && ((item.Type != madmin.SRIAMItemIAMUser && item.Type != madmin.SRIAMItemGroupInfo) || r.RevokedBefore.IsZero()) {
+			// A concurrent recreation has already superseded this delete.
+			return nil
+		}
+		item.UpdatedAt = r.timestamp()
+		if !r.Deleted {
+			item.UpdatedAt = r.RevokedBefore
+		}
+	}
+
+	versioned, err := c.replicationItem(ctx, item)
+	if errors.Is(err, errIAMStaleUpdate) {
+		return nil
+	}
+	if err != nil {
+		return err
+	}
 	cerr := c.concDo(nil, func(d string, p madmin.PeerInfo) error {
 		admClient, err := c.getAdminClient(ctx, d)
 		if err != nil {
 			return wrapSRErr(err)
 		}
 
-		return c.annotatePeerErr(p.Name, replicateIAMItem, admClient.SRPeerReplicateIAMItem(ctx, item))
+		_, err = executeIAMRevisionRequest(ctx, admClient, http.MethodPut, &iamRevisionBatch{Version: iamRevisionProtocol, Items: []iamReplicationItem{versioned}})
+		return c.annotatePeerErr(p.Name, replicateIAMItem, err)
 	},
 		replicateIAMItem,
 	)
@@ -1273,6 +1301,7 @@ func (c *SiteReplicationSys) IAMChangeHook(ctx context.Context, item madmin.SRIA
 // PeerAddPolicyHandler - copies IAM policy to local. A nil policy argument,
 // causes the named policy to be deleted.
 func (c *SiteReplicationSys) PeerAddPolicyHandler(ctx context.Context, policyName string, p *policy.Policy, updatedAt time.Time) error {
+	ctx = withIAMReplicationTime(ctx, updatedAt)
 	var err error
 	// skip overwrite of local update if peer sent stale info
 	if !updatedAt.IsZero() {
@@ -1286,18 +1315,19 @@ func (c *SiteReplicationSys) PeerAddPolicyHandler(ctx context.Context, policyNam
 		_, err = globalIAMSys.SetPolicy(ctx, policyName, *p)
 	}
 	if err != nil {
-		return wrapSRErr(err)
+		return iamReplicationError(err)
 	}
 	return nil
 }
 
 // PeerIAMUserChangeHandler - copies IAM user to local.
 func (c *SiteReplicationSys) PeerIAMUserChangeHandler(ctx context.Context, change *madmin.SRIAMUser, updatedAt time.Time) error {
+	ctx = withIAMReplicationTime(ctx, updatedAt)
 	if change == nil {
 		return errSRInvalidRequest(errInvalidArgument)
 	}
 	// skip overwrite of local update if peer sent stale info
-	if !updatedAt.IsZero() {
+	if !change.IsDeleteReq && !updatedAt.IsZero() {
 		if ui, err := globalIAMSys.GetUserInfo(ctx, change.AccessKey); err == nil && ui.UpdatedAt.After(updatedAt) {
 			return nil
 		}
@@ -1326,13 +1356,14 @@ func (c *SiteReplicationSys) PeerIAMUserChangeHandler(ctx context.Context, chang
 		}
 	}
 	if err != nil {
-		return wrapSRErr(err)
+		return iamReplicationError(err)
 	}
 	return nil
 }
 
 // PeerGroupInfoChangeHandler - copies group changes to local.
 func (c *SiteReplicationSys) PeerGroupInfoChangeHandler(ctx context.Context, change *madmin.SRGroupInfo, updatedAt time.Time) error {
+	ctx = withIAMReplicationTime(ctx, updatedAt)
 	if change == nil {
 		return errSRInvalidRequest(errInvalidArgument)
 	}
@@ -1340,7 +1371,7 @@ func (c *SiteReplicationSys) PeerGroupInfoChangeHandler(ctx context.Context, cha
 	var err error
 
 	// skip overwrite of local update if peer sent stale info
-	if !updatedAt.IsZero() {
+	if !updatedAt.IsZero() && (!updReq.IsRemove || len(updReq.Members) != 0) {
 		if gd, err := globalIAMSys.GetGroupDescription(updReq.Group); err == nil && gd.UpdatedAt.After(updatedAt) {
 			return nil
 		}
@@ -1349,7 +1380,8 @@ func (c *SiteReplicationSys) PeerGroupInfoChangeHandler(ctx context.Context, cha
 	if updReq.IsRemove {
 		_, err = globalIAMSys.RemoveUsersFromGroup(ctx, updReq.Group, updReq.Members)
 	} else {
-		if updReq.Status != "" && len(updReq.Members) == 0 {
+		snapshot, _ := ctx.Value(iamGroupSnapshotKey{}).(bool)
+		if !snapshot && updReq.Status != "" && len(updReq.Members) == 0 {
 			_, err = globalIAMSys.SetGroupStatus(ctx, updReq.Group, updReq.Status == madmin.GroupEnabled)
 		} else {
 			if globalIAMSys.LDAPConfig.Enabled() {
@@ -1365,13 +1397,14 @@ func (c *SiteReplicationSys) PeerGroupInfoChangeHandler(ctx context.Context, cha
 		}
 	}
 	if err != nil && !errors.Is(err, errNoSuchGroup) {
-		return wrapSRErr(err)
+		return iamReplicationError(err)
 	}
 	return nil
 }
 
 // PeerSvcAccChangeHandler - copies service-account change to local.
 func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change *madmin.SRSvcAccChange, updatedAt time.Time) error {
+	ctx = withIAMReplicationTime(ctx, updatedAt)
 	if change == nil {
 		return errSRInvalidRequest(errInvalidArgument)
 	}
@@ -1382,7 +1415,7 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 		if len(change.Create.SessionPolicy) > 0 {
 			sp, err = policy.ParseConfig(bytes.NewReader(change.Create.SessionPolicy))
 			if err != nil {
-				return wrapSRErr(err)
+				return iamReplicationError(err)
 			}
 		}
 		// skip overwrite of local update if peer sent stale info
@@ -1394,6 +1427,7 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 		opts := newServiceAccountOpts{
 			accessKey:     change.Create.AccessKey,
 			secretKey:     change.Create.SecretKey,
+			status:        change.Create.Status,
 			sessionPolicy: sp,
 			claims:        change.Create.Claims,
 			name:          change.Create.Name,
@@ -1402,7 +1436,7 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 		}
 		_, _, err = globalIAMSys.NewServiceAccount(ctx, change.Create.Parent, change.Create.Groups, opts)
 		if err != nil {
-			return wrapSRErr(err)
+			return iamReplicationError(err)
 		}
 
 	case change.Update != nil:
@@ -1411,7 +1445,7 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 		if len(change.Update.SessionPolicy) > 0 {
 			sp, err = policy.ParseConfig(bytes.NewReader(change.Update.SessionPolicy))
 			if err != nil {
-				return wrapSRErr(err)
+				return iamReplicationError(err)
 			}
 		}
 		// skip overwrite of local update if peer sent stale info
@@ -1431,7 +1465,7 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 
 		_, err = globalIAMSys.UpdateServiceAccount(ctx, change.Update.AccessKey, opts)
 		if err != nil {
-			return wrapSRErr(err)
+			return iamReplicationError(err)
 		}
 
 	case change.Delete != nil:
@@ -1442,7 +1476,7 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 			}
 		}
 		if err := globalIAMSys.DeleteServiceAccount(ctx, change.Delete.AccessKey, true); err != nil {
-			return wrapSRErr(err)
+			return iamReplicationError(err)
 		}
 	}
 
@@ -1451,24 +1485,17 @@ func (c *SiteReplicationSys) PeerSvcAccChangeHandler(ctx context.Context, change
 
 // PeerPolicyMappingHandler - copies policy mapping to local.
 func (c *SiteReplicationSys) PeerPolicyMappingHandler(ctx context.Context, mapping *madmin.SRPolicyMapping, updatedAt time.Time) error {
+	ctx = withIAMReplicationTime(ctx, updatedAt)
 	if mapping == nil {
 		return errSRInvalidRequest(errInvalidArgument)
 	}
-	// skip overwrite of local update if peer sent stale info
-	if !updatedAt.IsZero() {
-		mp, ok := globalIAMSys.store.GetMappedPolicy(mapping.Policy, mapping.IsGroup)
-		if ok && mp.UpdatedAt.After(updatedAt) {
-			return nil
-		}
-	}
-
 	// When LDAP is enabled, we verify that the user or group exists in LDAP and
 	// use the normalized form of the entityName (which will be an LDAP DN).
 	userType := IAMUserType(mapping.UserType)
 	isGroup := mapping.IsGroup
 	entityName := mapping.UserOrGroup
 
-	if globalIAMSys.GetUsersSysType() == LDAPUsersSysType && userType == stsUser {
+	if mapping.Policy != "" && globalIAMSys.GetUsersSysType() == LDAPUsersSysType && userType == stsUser {
 		// Validate that the user or group exists in LDAP and use the normalized
 		// form of the entityName (which will be an LDAP DN).
 		var err error
@@ -1491,19 +1518,20 @@ func (c *SiteReplicationSys) PeerPolicyMappingHandler(ctx context.Context, mappi
 			entityName = foundUserDN.NormDN
 		}
 		if err != nil {
-			return wrapSRErr(err)
+			return iamReplicationError(err)
 		}
 	}
 
 	_, err := globalIAMSys.PolicyDBSet(ctx, entityName, mapping.Policy, userType, isGroup)
 	if err != nil {
-		return wrapSRErr(err)
+		return iamReplicationError(err)
 	}
 	return nil
 }
 
 // PeerSTSAccHandler - replicates STS credential locally.
 func (c *SiteReplicationSys) PeerSTSAccHandler(ctx context.Context, stsCred *madmin.SRSTSCredential, updatedAt time.Time) error {
+	ctx = withIAMReplicationTime(ctx, updatedAt)
 	if stsCred == nil {
 		return errSRInvalidRequest(errInvalidArgument)
 	}
@@ -1555,7 +1583,7 @@ func (c *SiteReplicationSys) PeerSTSAccHandler(ctx context.Context, stsCred *mad
 
 	// Set these credentials to IAM.
 	if _, err := globalIAMSys.SetTempUser(ctx, cred.AccessKey, cred, stsCred.ParentPolicyMapping); err != nil {
-		return fmt.Errorf("unable to save STS credential and/or parent policy mapping: %w", err)
+		return iamReplicationError(fmt.Errorf("unable to save STS credential and/or parent policy mapping: %w", err))
 	}
 
 	return nil
@@ -4193,7 +4221,8 @@ func (c *SiteReplicationSys) SiteReplicationMetaInfo(ctx context.Context, objAPI
 				}
 
 				info.UserInfoMap[k] = madmin.UserInfo{
-					Status: madmin.AccountStatus(v.Credentials.Status),
+					Status:    madmin.AccountStatus(v.Credentials.Status),
+					UpdatedAt: v.UpdatedAt,
 				}
 			}
 		}
@@ -4549,9 +4578,25 @@ func (c *SiteReplicationSys) PeerStateEditReq(ctx context.Context, arg madmin.SR
 const siteHealTimeInterval = 30 * time.Second
 
 func (c *SiteReplicationSys) startHealRoutine(ctx context.Context, objAPI ObjectLayer) {
-	ctx, cancel := globalLeaderLock.GetLock(ctx)
+	for ctx.Err() == nil {
-	defer cancel()
+		var leadership LockContext
+		select {
+		case <-ctx.Done():
+			return
+		case leadership = <-globalLeaderLock.lockContext:
+		}
+		if leadership.Context().Err() != nil {
+			continue
+		}
+		leaderCtx, cancel := mergeContext(leadership.Context(), ctx)
+		c.healWithLeadership(leaderCtx, objAPI)
+		cancel()
+		// Quorum loss cancels a leadership lease, not the subsystem. Wait
+		// for a new lease so revocations can still reach offline sites.
+	}
+}
 
+func (c *SiteReplicationSys) healWithLeadership(ctx context.Context, objAPI ObjectLayer) {
 	healTimer := time.NewTimer(siteHealTimeInterval)
 	defer healTimer.Stop()
 
@@ -5170,13 +5215,16 @@ func (c *SiteReplicationSys) healBucketReplicationConfig(ctx context.Context, ob
 }
 
 func (c *SiteReplicationSys) healIAMSystem(ctx context.Context, objAPI ObjectLayer) error {
+	// A peer rejecting a deletion must not stop unrelated live updates from
+	// reaching healthy peers. Retain and report the error for the next retry.
+	deletionErr := c.healIAMDeletions(ctx)
 	info, err := c.siteReplicationStatus(ctx, objAPI, madmin.SRStatusOptions{
 		Users:    true,
 		Policies: true,
 		Groups:   true,
 	})
 	if err != nil {
-		return err
+		return errors.Join(deletionErr, err)
 	}
 	for policy := range info.PolicyStats {
 		c.healPolicies(ctx, objAPI, policy, info)
@@ -5194,7 +5242,7 @@ func (c *SiteReplicationSys) healIAMSystem(ctx context.Context, objAPI ObjectLay
 		c.healGroupPolicies(ctx, objAPI, group, info)
 	}
 
-	return nil
+	return deletionErr
 }
 
 // heal iam policies present on this site to peers, provided current cluster has the most recent update.
@@ -5429,8 +5477,10 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 
 		peerName := info.Sites[dID].Name
 
-		u, ok := globalIAMSys.GetUser(ctx, user)
+		// Disabled identities are valid replication sources. CheckKey returns
-		if !ok {
+		// their stored record even though authentication is denied.
+		u, _, err := globalIAMSys.CheckKey(ctx, user)
+		if err != nil || u.Credentials.AccessKey == "" || u.Credentials.IsExpired() {
 			continue
 		}
 		creds := u.Credentials
@@ -5629,7 +5679,10 @@ func isGroupDescEqual(g1, g2 madmin.GroupDesc) bool {
 }
 
 func isUserInfoEqual(u1, u2 madmin.UserInfo) bool {
-	if u1.PolicyName != u2.PolicyName ||
+	// Full-site summaries omit secrets and claims. Equal status alone cannot
+	// distinguish a recreated identity or an edited service-account policy.
+	if !u1.UpdatedAt.Equal(u2.UpdatedAt) ||
+		u1.PolicyName != u2.PolicyName ||
 		u1.Status != u2.Status ||
 		u1.SecretKey != u2.SecretKey {
 		return false

cmd/sts-handlers.go

@@ -624,6 +624,10 @@ func (sts *stsAPIHandlers) AssumeRole(w http.ResponseWriter, r *http.Request) {
 
 	claims[expClaim] = UTCNow().Add(duration).Unix()
 	claims[parentClaim] = user.AccessKey
+	if err := setIAMParentRevocationClaim(ctx, globalIAMSys.store, user.AccessKey, claims); err != nil {
+		writeSTSErrorResponse(ctx, w, ErrSTSInternalError, err)
+		return
+	}
 
 	tokenRevokeType := r.Form.Get(stsRevokeTokenType)
 	if tokenRevokeType != "" {

cmd/testdata/data-usage-v9/README.md

@@ -0,0 +1,27 @@
+# Data-usage v9 retirement fixture
+
+Generated using unmodified `scanDataFolder` and `dataUsageCache.serializeTo` from
+Server commit `89637554d60c27cfc51d2281d0a4fe15e415f06d` (Go 1.27.1, darwin/arm64).
+The scanner visits three real local files; its size callback supplies synthetic
+version/delete-marker/remote-tier summaries, following `TestDataUsageCacheSerialize`.
+It is a scanner/cache compatibility fixture, not a distributed object-store test.
+
+The old scanner counts 74,962 bytes, 3 objects, 5 versions and 2 delete markers.
+Its nonzero retired `hts` totals 9,426 bytes. Remote tier `COLD` contains 65,536
+bytes, one version and one object. The cache includes nested children, both
+histograms, a fixed timestamp and scanner cycle 42. The JSON is the old scanner's
+complete expected cache (including `HotTierSize`, which the new reader ignores).
+
+Binary SHA-256: `4c9c7e94cea7758fe9b498b19f639188764b2787582783e6cc950f2c44d52a8b`.
+
+To regenerate, create a detached worktree at that exact source commit, copy
+`generate.go.txt` to `cmd/retirement-fixture_test.go`, and run:
+
+```sh
+SILO_RETIRE_FIXTURE_DIR=/absolute/output/directory go test ./cmd -run '^TestGenerateAccessRetirementV9Fixture$' -count=1 -v
+```
+
+The historical production writer adds the version byte, compresses with zstd
+and encodes msgp, including the nonzero `hts` field. Do not regenerate using the
+retired implementation or by changing the header of a v8 payload. Map order may
+change serialized bytes across regeneration; compare the decoded full cache.

cmd/testdata/data-usage-v9/data-usage-v9.json

@@ -0,0 +1,118 @@
+{
+  "Info": {
+    "Name": "/",
+    "NextCycle": 42,
+    "LastUpdate": "2026-09-15T00:00:00Z",
+    "SkipHealing": true
+  },
+  "Cache": {
+    "/": {
+      "Children": {
+        "v9-bucket": {}
+      },
+      "Size": 0,
+      "HotTierSize": 0,
+      "Objects": 0,
+      "Versions": 0,
+      "DeleteMarkers": 0,
+      "ObjSizes": [
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0
+      ],
+      "ObjVersions": [
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0
+      ],
+      "AllTierStats": null,
+      "Compacted": false
+    },
+    "v9-bucket": {
+      "Children": {
+        "v9-bucket/nested": {}
+      },
+      "Size": 1234,
+      "HotTierSize": 1234,
+      "Objects": 1,
+      "Versions": 1,
+      "DeleteMarkers": 0,
+      "ObjSizes": [
+        0,
+        1,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0
+      ],
+      "ObjVersions": [
+        0,
+        1,
+        0,
+        0,
+        0,
+        0,
+        0
+      ],
+      "AllTierStats": null,
+      "Compacted": false
+    },
+    "v9-bucket/nested": {
+      "Children": null,
+      "Size": 73728,
+      "HotTierSize": 8192,
+      "Objects": 2,
+      "Versions": 4,
+      "DeleteMarkers": 2,
+      "ObjSizes": [
+        0,
+        1,
+        1,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0,
+        0
+      ],
+      "ObjVersions": [
+        0,
+        1,
+        1,
+        0,
+        0,
+        0,
+        0
+      ],
+      "AllTierStats": {
+        "Tiers": {
+          "COLD": {
+            "TotalSize": 65536,
+            "NumVersions": 1,
+            "NumObjects": 1
+          }
+        }
+      },
+      "Compacted": true
+    }
+  }
+}

cmd/testdata/data-usage-v9/generate.go.txt

@@ -0,0 +1,82 @@
+// Copyright (c) 2026 Feng Ruohang
+//
+// This file is part of Silo Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/minio/minio/internal/cachevalue"
+)
+
+// Run only on 89637554d, before access-tiering is removed. This uses the real
+// folder scanner and v9 serializer with synthetic file-size/tier summaries,
+// following TestDataUsageCacheSerialize; it does not relabel a v8 payload.
+func TestGenerateAccessRetirementV9Fixture(t *testing.T) {
+ if dataUsageCacheVerCurrent != 9 { t.Fatal("requires the historical v9 writer") }
+ base := t.TempDir()
+ const bucket = "v9-bucket"
+ createUsageTestFiles(t, base, bucket, []usageTestFile{
+  {name: "root", size: 1234},
+  {name: "nested/versions", size: 8192},
+  {name: "nested/remote", size: 65536},
+ })
+ getSize := func(item scannerItem) (s sizeSummary, err error) {
+  if item.Typ&os.ModeDir != 0 { return s, nil }
+  info, err := os.Stat(item.Path)
+  if err != nil { return s, err }
+  s.totalSize, s.hotTierSize, s.versions = info.Size(), info.Size(), 1
+  if filepath.Base(item.Path) == "versions" { s.versions, s.deleteMarkers = 3, 2 }
+  if filepath.Base(item.Path) == "remote" {
+   s.hotTierSize = 0
+   s.tiers = map[string]tierStats{"COLD": {TotalSize: uint64(info.Size()), NumVersions: 1, NumObjects: 1}}
+  }
+  return s, nil
+ }
+ xls := xlStorage{drivePath: base, diskInfoCache: cachevalue.New[DiskInfo]()}
+ xls.diskInfoCache.InitOnce(time.Second, cachevalue.Opts{}, func(context.Context) (DiskInfo,error) {
+  return DiskInfo{Total: 1<<40, Free: 1<<40}, nil
+ })
+ cache, err := scanDataFolder(t.Context(), nil, &xls, dataUsageCache{Info:dataUsageCacheInfo{Name:bucket, SkipHealing:true}}, getSize, 0, func()bool{return false})
+ if err != nil { t.Fatal(err) }
+ root := *cache.find(bucket)
+ cache.replace(dataUsageRoot, "", dataUsageEntry{})
+ cache.replace(bucket, dataUsageRoot, root)
+ cache.Info.Name = dataUsageRoot
+ cache.Info.LastUpdate = time.Date(2026, 9, 15, 0, 0, 0, 0, time.UTC)
+ cache.Info.NextCycle = 42
+ flat := cache.flatten(*cache.root())
+ if flat.Size != 74962 || flat.Objects != 3 || flat.Versions != 5 || flat.DeleteMarkers != 2 || flat.HotTierSize != 9426 {
+  t.Fatalf("unexpected scanner fixture: %+v", flat)
+ }
+ var buf bytes.Buffer
+ if err := cache.serializeTo(&buf); err != nil { t.Fatal(err) }
+ dir := os.Getenv("SILO_RETIRE_FIXTURE_DIR")
+ if dir == "" { t.Fatal("SILO_RETIRE_FIXTURE_DIR required") }
+ if err := os.MkdirAll(dir, 0755); err != nil { t.Fatal(err) }
+ if err := os.WriteFile(filepath.Join(dir,"data-usage-v9.bin"),buf.Bytes(),0644);err != nil{t.Fatal(err)}
+ expected, err := json.MarshalIndent(cache,"","  ")
+ if err != nil { t.Fatal(err) }
+ if err := os.WriteFile(filepath.Join(dir,"data-usage-v9.json"),append(expected,'\n'),0644);err != nil{t.Fatal(err)}
+ t.Logf("old scanner/v9 writer: bytes=%d size=%d objects=%d versions=%d markers=%d hts=%d",buf.Len(),flat.Size,flat.Objects,flat.Versions,flat.DeleteMarkers,flat.HotTierSize)
+}

cmd/xl-storage.go

@@ -583,7 +583,6 @@ func (s *xlStorage) NSScanner(ctx context.Context, cache dataUsageCache, updates
 	}
 
 	poolIdx, setIdx, _ := s.GetDiskLoc()
-	hotPool, hotPoolOK := globalILMConfig.accessCfg().HotPool()
 
 	disks, err := objAPI.GetDisks(poolIdx, setIdx)
 	if err != nil {
@@ -593,7 +592,6 @@ func (s *xlStorage) NSScanner(ctx context.Context, cache dataUsageCache, updates
 	cache.Info.updates = updates
 
 	dataUsageInfo, err := scanDataFolder(ctx, disks, s, cache, func(item scannerItem) (sizeSummary, error) {
-		item.poolIdx = poolIdx
 		// Look for `xl.meta/xl.json' at the leaf.
 		if !strings.HasSuffix(item.Path, SlashSeparator+xlStorageFormatFile) &&
 			!strings.HasSuffix(item.Path, SlashSeparator+xlStorageFormatFileV1) {
@@ -657,9 +655,6 @@ func (s *xlStorage) NSScanner(ctx context.Context, cache dataUsageCache, updates
 				sizeS.versions++
 			}
 			sizeS.totalSize += sz
-			if hotPoolOK && poolIdx == hotPool {
-				sizeS.hotTierSize += sz
-			}
 
 			// Skip tier accounting if object version is a delete-marker or a free-version
 			// tracking deleted transitioned objects

docs/bucket/lifecycle/README.md

@@ -226,149 +226,11 @@ aws s3api restore-object --bucket srcbucket \
 
 Note that transition event notification is a Silo extension.
 
-## 5. Access-based tiering between server pools
-
-Silo can move frequently read objects to a faster server pool and return them
-to a slower pool after they become idle. This is different from remote ILM
-transition: the object remains a native local object and all versions move
-together.
-
-Access tiering requires at least two server pools. Pool indices follow the
-order on the server command line:
-
-~~~sh
-silo server /srv/nvme{1...4} /srv/hdd{1...8}
-#             pool 0 (fast)   pool 1 (slow)
-~~~
-
-Server pools are erasure-coding expansion units, not individual drives. Each
-pool should consist of internally homogeneous media.
-
-The feature is disabled by default. Configure the topology and safety limits
-with `mc admin config set`; ILM is a dynamic subsystem, so this applies without
-a restart. Environment variables (`MINIO_ILM_ACCESS_TIERING`,
-`MINIO_ILM_ACCESS_POOLS`, `MINIO_ILM_ACCESS_MAX_SIZE`,
-`MINIO_ILM_ACCESS_PROMOTE_WATERMARK`, `MINIO_ILM_ACCESS_BIN_WIDTH`,
-`MINIO_ILM_ACCESS_BINS`, `MINIO_ILM_ACCESS_FLUSH`,
-`MINIO_ILM_ACCESS_MIN_RESIDENCY`, `MINIO_ILM_ACCESS_WORKERS`,
-`MINIO_ILM_ACCESS_MAX_TRACKED`) are read at process start and override the
-stored config.
-
-~~~sh
-mc admin config set local ilm \
-  access_tiering=on \
-  access_pools="0,1" \
-  access_max_size="2TiB" \
-  access_promote_watermark=85 \
-  access_bin_width=1m \
-  access_bins=12 \
-  access_flush=1m \
-  access_min_residency=24h \
-  access_workers=10 \
-  access_max_tracked=1000000
-~~~
-
-The pool list is ordered hottest to coldest. With three or more pools,
-promotion always targets the first index and demotion always targets the last;
-intermediate pools are not hop targets. An access_max_size value of zero means
-no cluster-wide logical-byte cap. Promotion also stops when the hottest pool
-reaches access_promote_watermark.
-
-New PUTs still land via the usual free-space pool picker; they are not steered
-onto the cold pool. Size the capacity pool larger than the hot pool so new
-objects tend to land there.
-
-Add an AccessTransition to the bucket lifecycle XML:
-
-~~~xml
-<LifecycleConfiguration>
-  <AccessTierQuota>500GiB</AccessTierQuota>
-  <Rule>
-    <ID>hot-logs</ID>
-    <Status>Enabled</Status>
-    <Filter>
-      <And>
-        <Prefix>logs/</Prefix>
-        <ObjectSizeGreaterThan>65536</ObjectSizeGreaterThan>
-      </And>
-    </Filter>
-    <AccessTransition>
-      <Window>10m</Window>
-      <PromoteAfterAccesses>100</PromoteAfterAccesses>
-      <DemoteAfterAccesses>5</DemoteAfterAccesses>
-      <DemoteAfterIdle>24h</DemoteAfterIdle>
-    </AccessTransition>
-  </Rule>
-</LifecycleConfiguration>
-~~~
-
-This promotes a matching object after 100 successful GETs in 10 minutes. An
-object becomes eligible to return to the coldest configured pool only after
-access tiering has already moved it (the `x-minio-internal-ilm-atier` stamp),
-it has stayed put for the server-wide minimum residency, it has been idle at
-least 24 hours, and it has no more than 5 GETs in the window. Objects that
-landed on the hot pool via a normal PUT never demote. Prefix, tag, and
-object-size lifecycle filters are honored.
-
-Access-based moves are a parallel path: they are not lifecycle `Eval` actions
-and do not appear in S3 prediction headers. If the same object is also due
-for age-based remote `Transition` or expiry, that scanner action wins and
-demotion discovery is skipped for that pass; promotions still run from the
-GET tracker. Site replication copies expiry rules only, same as remote
-Transition, so AccessTransition stays local to the cluster.
-
-AccessTierQuota is an optional bucket-wide cap. Promotion checks, in order:
-
-1. hot-pool used percentage;
-2. cluster-wide access_max_size;
-3. bucket AccessTierQuota.
-
-Demotion is not blocked by these caps and is processed before promotion.
-Access moves pause during rebalance or decommission, never target a suspended
-pool, skip remotely transitioned objects and objects with excessive version
-counts, and recheck eligibility while holding the object namespace lock.
-Moves also lock the source and destination write locations. All lock servers
-for those locations must be reachable; otherwise the background move is
-deferred. Ordinary S3 requests keep their existing quorum requirements.
-
-Each move preserves version IDs, delete markers, ETags, checksums, encryption
-and user metadata. A retry copies only missing versions and retains versions
-already at the destination. If the same version ID has conflicting metadata
-in the two pools, the move is skipped and both copies are left intact. Source
-data is removed only after the complete version stack exists at the destination.
-
-The hit counter is intentionally best effort. Only successfully served GET
-requests count; HEAD requests do not. Counters are merged across nodes and
-bounded by access_max_tracked. A rule window longer than
-access_bin_width multiplied by access_bins is clamped to retained history.
-
-AccessTransition and AccessTierQuota are Silo lifecycle extensions. A stock
-AWS SDK that reads and rewrites the lifecycle configuration may discard
-unknown fields. Use a raw signed S3 PUT lifecycle request, such as
-[setup_ilm_access_tiering.sh](setup_ilm_access_tiering.sh), when installing
-the rule. Save the XML above as `rule.xml`, then run:
-
-~~~sh
-AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \
-  ./setup_ilm_access_tiering.sh http://127.0.0.1:9000 testbucket us-east-1 rule.xml
-~~~
-
-Access-tier activity is exposed under /minio/metrics/v3/ilm, including move
-counts, moved bytes, queue depth, hot bytes per bucket, failed moves, dropped
-GET samples, and separate skip counters for each capacity limit.
-
-To stop scheduling moves, set `access_tiering=off` (and remove any environment
-override). Objects already moved remain in their current pools and stay
-accessible; disabling the feature does not move them back. Before downgrading
-to a release without access tiering, disable it and let active moves finish.
-The object storage format is unchanged. The data-usage cache advances from
-v8 to v9: this release reads both, but an older binary discards v9 caches and
-rebuilds usage statistics through the scanner. Usage and quota statistics can
-therefore take time to repopulate after a downgrade. Keep a copy of lifecycle
-XML containing Silo extensions, since an older binary may omit those fields
-when rewriting a lifecycle rule.
-
 ## Explore Further
 
 - [MinIO Go client API reference (S3-compatible SDK)](https://pkg.go.dev/github.com/minio/minio-go/v7)
 - [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
+
+## Access-frequency tiering removal
+
+The opt-in cross-pool access-frequency extension has been removed. For upgrades from a main/snapshot build that included it, see [the migration notes](access-tiering-removal.md). Ordinary lifecycle expiration and remote-tier transitions remain supported.

docs/bucket/lifecycle/access-tiering-removal.md

@@ -0,0 +1,36 @@
+# Removing access-frequency pool tiering
+
+PR #60 introduced an opt-in scheduler that moved objects between local server pools according to GET frequency. It and its feature-specific fixes have been removed. This does not remove ordinary lifecycle expiration, transitions to remote tiers, rebalance, decommission, or the general multi-pool correctness fixes from PR #178.
+
+The [introduction and rollback record](../../investigations/access-tiering-revert.md) documents the commit history, scope decision, review corrections and unresolved validation findings.
+
+The published Server 20260903 predates this feature. These instructions concern main/snapshot deployments that included #60; upgrading from the published version does not require access-tier configuration cleanup.
+
+## Before upgrading a build with access tiering
+
+1. Save a copy of the server ILM configuration and each affected bucket's lifecycle XML. Use an API client that preserves the nonstandard XML; do not rely on a client model that silently omits unknown elements.
+2. On the old server, set `ilm access_tiering=off` and remove or disable any access-tier environment overrides. Allow in-progress moves to finish before replacing nodes. This reduces movement intermediate states; the removal itself changes no storage RPC protocol.
+3. Remove top-level `AccessTierQuota` and rule-level `AccessTransition` elements. **Delete rules whose only action was `AccessTransition`**. For a mixed rule, retain its filter, status, ID and ordinary expiration/transition actions. An access-only rule loads harmlessly after upgrade, but becomes an actionless rule and fails validation on the next lifecycle edit. If no rules remain, delete the lifecycle configuration through the S3 API.
+4. Use a coordinated maintenance window: stop the deployment, install the same new binary on every node, then restart all nodes. The existing bootstrap check compares binary checksums; in a four-node test the first new node could not finish starting among three old nodes. Do not assume that an unchanged RPC protocol permits replacing one node at a time and waiting for it to become ready. This removal does not relax that check. Apply the same environment changes on every node: bootstrap also compares server environment settings, so removing an old override on only some nodes can block startup even with matching binaries. The check runs only during startup and is not a safety guarantee for nodes already running different binaries.
+5. After restarting, verify object reads, bucket listing, ILM worker settings and a lifecycle edit. Check storage access from every request-serving node to each pool's drives: successful reads or bucket listing establish less than complete drive reachability, and admin disk summaries aggregate server-local state. Retirement acceptance used unique probes and storage trace to confirm every node-to-drive path before and after version deletion. Startup connection times varied, so a fixed sleep is insufficient. Complete distributed upgrade acceptance for the exact binaries before production rollout.
+
+## What happens to stored state
+
+| State | Behavior after removal |
+| --- | --- |
+| Ten old ILM keys | `access_tiering`, `access_pools`, `access_max_size`, `access_promote_watermark`, `access_bin_width`, `access_bins`, `access_flush`, `access_min_residency`, `access_workers`, `access_max_tracked` are accepted but ignored. Existing transition/expiration worker settings are preserved. |
+| Admin configuration | Deprecated keys may still appear in `mcli admin config get ilm`; setting them may succeed but has no effect, even with `access_tiering=on`. Remove obsolete environment settings from deployment manifests. |
+| Lifecycle XML | `AccessTierQuota` and `AccessTransition` are ignored when read and omitted when re-encoded. The same parser handles new PUT requests, so these extensions are also silently discarded there; access-only rules still fail action validation. |
+| Data-usage cache | Both v8 and v9 caches are read, preserving ordinary counts, sizes, histograms and remote-tier statistics. The retired hot-tier byte count is discarded; subsequent writes use v8. No feature-driven full statistics rebuild is required. |
+| Objects already moved | Remain in their current pools with the same versions and timestamps. There is no bulk move-back or object metadata rewrite. |
+| Internal leftovers | `x-minio-internal-ilm-atier` and `.minio.sys/config/ilm/access/` counter objects may remain unused. They do not require a cleanup service or an object scan. |
+
+Interrupted rebalance/decommission can leave the same version in more than one pool independently of access tiering. Removing the scheduler does not remove such existing copies. General Object Lock, conditional-delete, metadata reconciliation and shared remote-tier reference protections remain in place.
+
+## Version deletion scope
+
+Ordinary single-object `DELETE ?versionId=...` reconciles the addressed UUID, null version or delete marker across pools. Unqualified DELETE of a directory marker (a key ending in `/`) also addresses its null version and uses this path. A successful request applies the deletion to every resolved pool copy under the existing per-pool quorum rules; other version IDs remain. If outbound delete replication is pending, copies retain `VersionPurgePending` until the existing replication worker completes the purge. Success does not guarantee immediate physical removal from every drive.
+
+If a pool is unreadable, these requests can return 503 even when another pool has a readable copy. Insufficient read quorum returns `503 SlowDownRead`; other failures retain their corresponding error codes. This extends an existing failure surface: previously the result could depend on whether the unreadable pool preceded the successful pool in traversal order; it now fails consistently. Retry after recovery. Cleanup failures also return an error. Ordinary unqualified DELETE retains its existing semantics. Batch `DeleteObjects` already fans out across pools.
+
+Incoming replicated deletes, lifecycle expiration, free-version cleanup and movement-internal calls retain their existing contracts. In particular, an incoming replicated version delete can leave movement duplicates in other pools; this change does not solve that separate case. Expiration scanners process their own pools and may remove duplicate expired copies in later cycles; free-version cleanup remains local to a pool. Do not treat the ordinary DELETE repair as a guarantee for every source of deletion.

docs/bucket/lifecycle/setup_ilm_access_tiering.sh

@@ -1,37 +0,0 @@
-#!/bin/sh
-
-# Install a lifecycle XML document without an SDK normalizing away Silo's
-# AccessTransition and AccessTierQuota extension elements.
-set -eu
-
-if [ "$#" -ne 4 ]; then
-	echo "usage: AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... $0 ENDPOINT BUCKET REGION LIFECYCLE_XML" >&2
-	exit 2
-fi
-
-endpoint=$1
-bucket=$2
-region=$3
-lifecycle_file=$4
-
-: "$AWS_ACCESS_KEY_ID"
-: "$AWS_SECRET_ACCESS_KEY"
-
-if [ ! -r "$lifecycle_file" ]; then
-	echo "cannot read lifecycle document: $lifecycle_file" >&2
-	exit 2
-fi
-
-content_md5=$(openssl dgst -md5 -binary "$lifecycle_file" | openssl base64)
-endpoint=$(printf '%s' "$endpoint" | sed 's:/*$::')
-
-curl --fail-with-body --silent --show-error \
-	--request PUT \
-	--aws-sigv4 "aws:amz:$region:s3" \
-	--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
-	--header "Content-MD5: $content_md5" \
-	--header "Content-Type: application/xml" \
-	--data-binary "@$lifecycle_file" \
-	"$endpoint/$bucket?lifecycle"
-
-echo "installed lifecycle configuration on $bucket"

docs/investigations/access-tiering-revert.md

@@ -0,0 +1,193 @@
+# 访问频率分层：引入、修复与回退记录
+
+记录日期：2026-09-15。本记录说明 PR #60 的设计、合入后的取舍，以及此次回退为什么同时保留并补齐通用多池正确性修复。操作步骤见[退役迁移说明](../bucket/lifecycle/access-tiering-removal.md)。合并、正式发布和生产部署是不同状态；本记录随回退变更交付，不代表已经发布。
+
+初稿审查时（2026-09-15 03:20 UTC），最终候选尚未移植到远端基线、尚未冻结 PR head，合并前全量检查与三次有效 Linux 运行尚未执行，本变更尚未合并。后续执行状态以承载本记录的 PR 及其绑定提交的验收记录为准；下文的历史实验不替代这些检查。
+
+**当前验收状态：** 回退、普通版本 DELETE 调和、扫描复用及文档由 [PR #188](https://github.com/pgsty/silo/pull/188) 交付。本地与 CI 检查通过；首次 Linux 验收因 DELETE204 后的 HEAD/GET503 停止。随后完成可控机制实验、匹配写入负载对照，并修正准备检查；独立的新轮次 R-Upgrade-2 三次完整升级验收均通过。旧失败没有改判，原单次请求的逐盘状态仍不可追溯。最终合并状态以 PR 为准，正式发布与部署另行验收。详情见第 9 至 11 节。
+
+## 1. 引入的目标和实际范围
+
+[@mrjavadseydi](https://github.com/mrjavadseydi) 在 [PR #60](https://github.com/pgsty/silo/pull/60) 提出了基于 GET 频率的本地池间分层。成功 GET 更新有界滚动计数，后台调度器把热对象提升到配置中的首个池，把已经迁移且变冷的对象降到末个池。功能默认关闭，至少需要两个池；它与普通生命周期过期、远端对象存储 transition、rebalance 和 decommission 是不同机制。
+
+实现不只是一个后台任务：它增加了 GET 计数入口、leader 调度、跨池版本栈复制、来源清理及失败恢复、热池配额、生命周期 XML 扩展、配置项、指标和扫描统计。访问热度统计使 data-usage cache 从 v8 升到 v9。对象本体的存储格式没有因此改变。
+
+搬移要保持完整版本历史、删除标记、null version、时间戳、ETag、校验和和加密元数据；同时需要处理并发写入、目的端已有版本、来源部分删除和远端 tier 引用。合入前的修补和测试针对并覆盖了这些边界，不应把回退解释成贡献无效。贡献者署名继续保留，独立的其他贡献也不回退。
+
+## 2. 可追溯时间线
+
+下表的 PR/Issue 时间使用 UTC；提交链接对应具体代码，不把“报告时间”当作“缺陷首次出现时间”。
+
+| 时间 | 事件 | 本次处理 |
+| --- | --- | --- |
+| 2026-08-15 10:15 | #60 创建；原始实现 [`7a060cab1`](https://github.com/pgsty/silo/commit/7a060cab1edd5bbc17da7f703bbd1ab7415b6f7c) | 随特性撤销 |
+| 2026-09-06 00:09 | [#133](https://github.com/pgsty/silo/issues/133) 报告多池副本写不能权威调和 Object Lock 状态 | 保留解决它的通用修复 |
+| 2026-09-06 15:12 | [#144](https://github.com/pgsty/silo/issues/144) 报告条件 DELETE 原子性仅限单个纠删码集合 | 保留解决它的通用修复 |
+| 2026-09-08 05:18 | [`9a6e1477f`](https://github.com/pgsty/silo/commit/9a6e1477f45067559def8423d431ee177795134f) 补访问分层兼容标识清单 | 删除功能专属标识，保留有依据的退役兼容 |
+| 2026-09-08 07:08 | [`374de0fa3`](https://github.com/pgsty/silo/commit/374de0fa32aa1d6eda57dbba6ac522ebf793b6be) 修复搬移的版本保全、写隔离和删除范围 | 随专属搬移器撤销 |
+| 2026-09-08 07:28 | [`5ac33e158`](https://github.com/pgsty/silo/commit/5ac33e1583e838ade3f56c7d60e80e7a854f9a88) 确定性覆盖搬移失败恢复 | 随已删除搬移器的专属测试撤销 |
+| 2026-09-08 07:42 | #60 以 [`a3df317ae`](https://github.com/pgsty/silo/commit/a3df317ae0725eb650e4d3e21551154f69be6229) 合入 | 以该 merge 的第一父差异确定功能边界 |
+| 2026-09-11 12:34 | [#178](https://github.com/pgsty/silo/pull/178) 合入通用多池写入、元数据与条件删除调和 | 保留，解除测试对访问搬移器的依赖 |
+| 2026-09-13 | [`2dd1e00da`](https://github.com/pgsty/silo/commit/2dd1e00da49faf995f2db29807fc6211b8376d7d) 的 CHANGELOG 同时汇总访问分层和通用多池修复 | 拆开表述，不整条删除独立修复历史 |
+| 2026-09-15 | 维护者决定收缩访问分层；完成来源分析、三种候选反证、退役兼容、普通版本 DELETE 修补与外部评审 | 形成此次选择性回退 |
+
+#178 中的 [`e59a3d938`](https://github.com/pgsty/silo/commit/e59a3d938ed25c1bcd51efbb4ad6955073d195f7) 提供池级串行化、字段调和和条件删除；[`ccb676e60`](https://github.com/pgsty/silo/commit/ccb676e60cb7441ee65ff7c35f3b7828979101fd) 保护仍被其他副本使用的远端 tier 引用；[`51d41345f`](https://github.com/pgsty/silo/commit/51d41345f7ac532f9c6fea2b7dba5f29da930b8f) 保存 Linux 重启及 OIDC 验收记录。这三项不属于仅为访问频率调度而存在的代码。
+
+截至回退评估时，公开 Server `RELEASE.2026-09-03T13-18-01Z` 早于 #60 合入。退役迁移主要针对运行过后续 main、自行构建或快照版本的实例；不能据此声称正式 release 用户普遍启用过此特性。
+
+## 3. 为什么回退，为什么不能整批撤销后续修复
+
+维护者的取舍是：默认关闭的可选调度能力，对配置、生命周期、缓存、统计和核心多池写入路径带来了过大的维护面。此次移除的是该能力及专属实现，没有测量并宣称吞吐量提升、延迟降低或固定减少一次分布式锁往返。
+
+“后来改过同一文件”不等于“由 #60 引发”。#133/#144 的报告早于 #60 合入；更关键的是，原有 rebalance、decommission 和复制写入也会使同一版本暂时存在于多个池。访问分层消失后，这些状态仍然合法存在。移除调和与锁纪律会重新允许旧副本遮蔽新元数据、条件删除选错版本、清理错误被吞掉等问题。
+
+评估在隔离工作树中实际比较了三条路线：
+
+| 候选 | 通用多池回归 | 普通指定版本 DELETE |
+| --- | --- | --- |
+| A：撤销 #60，保留 #178 | 原有 13 组通过 | 仍能成功返回后留下可读副本 |
+| B：同时撤销 #60 和 #178 的存储修补 | 相同 13 组中 10 组失败 | 问题仍在 |
+| C：A 加普通版本 DELETE 调和 | 13 组原有及当时新增的 8 组通过 | 同一复现通过 |
+
+这些是 2026-09-15 的历史对照结果，不是最终 PR head 的发布验收。后续补上目录标记、真实 rebalance 中断等覆盖后，通用多池测试达到 23 组。测试通过不能替代来源分析，来源分析也不能替代最终候选的运行验证。
+
+## 4. 最终保留与删除的边界
+
+- 删除访问 tracker、调度/搬移器、GET 和 scanner 钩子、热池配额、专属配置帮助、生命周期动作、指标及专属测试。
+- 保留普通过期、远端 transition、rebalance/decommission、复制写入，以及 #178 的 Object Lock、标签、条件删除、元数据调和与远端引用保护。
+- 原有 data-usage 生成解码器恢复到 #60 前的实现；允许读取 v8/v9，利用字段编码跳过已退役热度字段，继续写 v8。普通字段保真由历史真实 v9 样本测试覆盖。
+- 仅容忍准确的十个退役 ILM 键；读取生命周期时丢弃退役扩展。纯访问动作的规则需先清理才能再次编辑；混合规则保留普通动作。
+- 已经搬移的对象留在当前池；没有全量搬回、自动删除所有重复版本、后台清理服务或对象元数据重写。
+
+曾对本地审查提交 `d06f1c614` 做过声明来源复核：消失的 357 个声明均不在 #60 之前，删除的十个文件均由 #60 引入；#60 原先删换的 41 行旧文本按忽略空白比较有 40 行恢复，剩下一行保留 #178 在已持有池锁时调用 `getWritePoolIdx(..., true)` 的修正，避免对同一对象再次取锁。生成缓存解码器与功能前逐字节一致，原有 13 组通用测试没有删除。这是该审查版本的保全证据，不能把数字脱离 SHA 当作未来所有版本的保证。
+
+## 5. 普通版本 DELETE 是独立补洞
+
+功能移除不会自动消除历史重复副本。普通单对象指定版本 DELETE 因而复用既有调和路径：在池级对象锁内读取每个池的目标版本，计算一次条件及回调，先删除非权威副本，再处理权威副本；任何不可读池或清理错误都不能当作成功。
+
+范围包括 UUID、null version、delete marker，以及原先就被解析为 null version 的未指定版本目录标记 DELETE。入站复制、搬移内部调用、生命周期过期和 free-version 清理保留各自语义；批量 `DeleteObjects` 原本就会向池并发扇出，不是此次遗漏。
+
+删除标记需要向 retention/metadata 回调传入与 set 层相同的 `MethodNotAllowed` 或 `ObjectNotFound` 语义。直接复用拒绝 marker 的元数据更新入口会错误地拒绝合法版本删除。回调从所有副本合并独立更新的 Object Lock 和标签，不能随意只采用一个池的状态。
+
+存在两项明确的成功/失败边界：
+
+1. 读法定多数不足时返回 `503 SlowDownRead`，即使另一个池有可读副本。旧路径的结果会受池遍历顺序影响；新路径把失败语义统一。它是正确性与可用性的取舍，需要恢复后重试。
+2. 出站删除复制尚未完成时，成功响应可以表示各副本进入 `VersionPurgePending`，由既有 worker 完成清理。原有每池 quorum 规则也继续适用；不能把成功响应等同于每一块盘立即物理删除。
+
+## 6. 审查如何改变了方案
+
+本地先形成三个线性审查提交：`8fdfdabd9` 移除特性，`d06f1c614` 补普通 DELETE，`6e3fdca97` 去除重复扫描。它们记录审查演进，最终 PR 在独立远端基线上重放，提交 ID 会改变；不应把线性演进误认为三份同时维护的实现。
+
+Claude Code Opus 5 / max 的五轮实现评审要求补齐退役兼容、说明协调停机及环境一致性、验证真实池故障，并纠正运行证据措辞。随后 Claude 与 ZCode 的独立复核再次确认了回退边界和普通 DELETE 语义；最终两轮计划商榷收束了交付流程。
+
+| 意见 | 裁定与处理 |
+| --- | --- |
+| 指定版本 DELETE 重复扫描所有池 | 接受。第一次扫描已在同一锁内得到目标副本，直接合并其结果。16 盘池在回调前的读取计数从 32 降为 16；这不等于总 I/O 或延迟减半。 |
+| N 个副本产生 N 条 DELETE 审计 | 反驳。底层调用只追加上下文标签，HTTP 层向每个配置审计目标发送一次请求事件。成功完成调和时池标签最终指向 primary；不增加额外 NoAuditLog 修改。 |
+| retention/metadata 顺序与 set 层不同 | 差异存在，但已有明确注释。保留 retention 优先，拒绝后不删除、不调度删除复制、不 Sweep；不宣称任意自定义回调都能交换。 |
+| 把优化 amend 进旧提交并直接丢弃 main 脏修改 | 不 amend 已审历史。先保存完整文件、补丁、哈希及可达恢复引用，核对覆盖后受控恢复。八组新增通用测试承接，访问搬移专用测试由真实 rebalance 中断覆盖替代。 |
+| 从当前本地分支直接开 PR | 调整。其祖先含另一个任务的 IAM/超时提交 `ebc9937d9`；从远端 `89637554d` 仅移植本次变更，不夹带或删除独立工作。 |
+| 合并之后再跑全量与 Linux 验收 | 不接受。先完成文档和移植，冻结 PR head，再验收；不能把旧 SHA 的测试直接提升为新基线的通过记录。 |
+| 不同基线 diff 必须逐字节一致 | 改为每提交 stable patch-id、路径与完整树等价性核验。blob hash 和行号随基线变化，不应成为错误的拒绝依据。 |
+
+`objectPoolInfos` 的并行查询作为独立性能跟进，本次不增加并发实现。Contributor 署名、#132 配额指标、#77 桶元数据、federated COPY、IAM、超时及其他独立修复不因本次取舍被整体撤销。
+
+## 7. 历史运行证据与未定案事项
+
+以下是移植前 `d06f1c614` 的历史实验，不是最终 PR head 的验收替身。
+
+| 运行 | 实际观察 | 不应推导的结论 |
+| --- | --- | --- |
+| `f590538f`，四节点双池 | 旧版真实 rebalance 中断留下 9 个重复 UUID；协调停机换新后对象四节点可读，旧配置与普通规则可编辑，真实缓存头 v9→v8；删除一个 addressed UUID 后四节点 HEAD/GET 404，另一个版本仍可读 | 没有验证全部 9 个不同 UUID 收敛；物理元数据仅每池抽查一盘；没有整份运行时统计守恒测量 |
+| `fd93cd37`，三节点双池 | 停掉一个池所在节点，保留 namespace 锁的 2/3 法定多数；DELETE 返回 503 SlowDownRead，源全部四盘保留目标版本；恢复后 DELETE204，各节点 HEAD/GET404 | 不能推广为跨主机网络、持久盘及压力验收 |
+| 被弃用的四节点停池拓扑 | 同时丢失 namespace 锁法定多数，发生客户端超时，记录脚本还遇到 NoneType 错误 | 不是“池读取返回503”的证明 |
+| `83676ca2` | 升级后配置/生命周期编辑之后一次 HeadObject 返回503；artifact 没有记录 DELETE 自身响应 | “DELETE之后”仅来自脚本顺序，不能写成已证实 DELETE204 后异常，也不能归类为已修复、既有问题或暂态 |
+
+**开放项：`83676ca2` 的 HEAD503 仍未定案。** 可直接比较的目标阶段历史运行是一失败、一成功；一次未复现不足以关闭问题。仅凭 `SlowDownWrite` 等错误名字也不能给其他失败确定容量或环境根因。
+
+最终候选的合并前复核采用有界规则：要求三次有效升级后 DELETE/HEAD 运行，最多五次总尝试；每次记录 DELETE 码/耗时、失败 HEAD 的节点与版本、GET 错误码、两池全盘元数据、固定间隔重试时序和旧版同拓扑对照。旧版可能保留副本返回200，不要求它满足新增跨池删除契约。
+
+有证据证明在目标操作前失败的 harness 尝试才能不计入有效运行，但仍计入总尝试。任何目标阶段的新503或数据不变量失败都不能通过补跑抹掉，必须暂停合并并定位。三次通过也只满足这项工程检查，不证明历史异常已消失；开放项在合并和正式发布评估时仍须可见。
+
+## 8. 交付与恢复纪律
+
+最终候选使用独立分支；main 的五个旧修改先保存完整内容、二进制补丁、SHA-256 和具名 Git 恢复引用，再核对原 HEAD/哈希及测试覆盖，只恢复这五个文件。禁止用整树 reset 或 clean 代替受控归一；若用户已有新增编辑，应保留并重新核对。
+
+全量 cmd/internal、相关 race、构建、vet、lint、生成文件和兼容检查，以及上述 Linux 验收，绑定最终候选的实际 SHA。若纳入新的远端提交，重新记录基线与 head，复核变更并重跑受影响验收。文档与原始测试记录各自保留其对应版本，不篡改旧失败、不用新通过覆盖旧记录。
+
+逐节点滚动升级未通过既有二进制校验检查，采用[协调停机方案](../bucket/lifecycle/access-tiering-removal.md#before-upgrading-a-build-with-access-tiering)。实验使用单个 Docker Linux VM 和 tmpfs；正式 tag、包、镜像、跨主机及生产部署仍是独立交付。未验证全部重复 UUID 或运行时统计守恒应如实披露，不能反向引入自动搬回/清理需求或无关重构。
+
+## 9. 执行后记：最终基线、恢复窗口与验收
+
+本次实际交付由 [PR #188](https://github.com/pgsty/silo/pull/188) 承载。选择的远端基线为 `89637554d60c27cfc51d2281d0a4fe15e415f06d`，移植没有包含本地独立 IAM/超时提交 `ebc9937d9`。前三项实现和历史文档逐提交通过 stable patch-id 对照；虚拟补回独立 IAM 差异后，完整树与原审查分支一致。
+
+首次本地 lint 发现新增回调选择分支触发 `gocritic/ifElseChain`，因此追加等价的无表达式 `switch` 改写，保持 marker、指定版本和普通元数据查找的条件顺序及分支体。重新固定的代码候选为 [`41aa84609`](https://github.com/pgsty/silo/commit/41aa84609754769cfb1861d7fd060c2e84182b98)。这一提交上，全量 cmd/internal 得到 6,428 个测试及子测试通过、166 个跳过，50 个有测试的包通过；相关 race 得到 283 个测试及子测试通过。`make build`、全包构建、vet、lint、生成文件及 rebrand/compat 检查均通过。[Go CI](https://github.com/pgsty/silo/actions/runs/34925534139)、[DCO](https://github.com/pgsty/silo/actions/runs/34925534110)、[VulnCheck](https://github.com/pgsty/silo/actions/runs/34925534142) 和[发布流水线的测试运行](https://github.com/pgsty/silo/actions/runs/34925534198)共 11 项检查通过；后者没有发布正式制品。
+
+第一次最终候选停池实验 `dcd5c2e5` 在源版本保全断言后失败：停掉另一池返回 `503 SlowDownRead`，源四盘版本保留；恢复后 DELETE 成功，节点 0/2 的 HEAD/GET 返回 404，节点 1 返回 503。DELETE 成功由脚本已通过的 204 断言确定，原输出没有单独保存该次 DELETE 响应。此次失败如实保留，不能把后来的成功写回原记录。
+
+复核发现，`ListBuckets` 以及位于源池的现存版本 GET，不能证明每个协调节点对另一池的读取连接已经恢复。随后进行了旧基线与候选的同拓扑对照，探测的是从未写入过的随机 UUID，并且在这些探测之前没有执行任何 DELETE：
+
+| 高时间分辨率对照 | 桶列表和现存版本 | 从未写入版本的 HEAD/GET | 观察到的恢复窗口 |
+| --- | --- | --- | --- |
+| `20502a2f`，旧基线 `89637554d` | 三节点均为 200 | 节点 0/2 为 404，节点 1 为 503 SlowDownRead，连续 16 组 | 从重启后的观察循环起算约 1.60–2.50 秒 |
+| `246aaf77`，候选 `41aa84609` | 三节点均为 200 | 同样是节点 1 的 503，连续 11 组 | 约 1.67–2.30 秒 |
+
+两边在缺失版本全节点连续三轮返回 404 后执行 DELETE，均得到 204、全节点 HEAD/GET 404、目标 UUID 在八盘均不存在且其他版本可读。另有两次较低时间分辨率诊断未捕捉到窗口，同样保留；这四次诊断不计入三次升级验收。
+
+这给出了基线在零 DELETE 下的正向复现，证明原恢复条件不足。`getLatestObjectInfoWithIdx` 的读选择函数与基线文本相同：现存副本可以遮蔽另一池的读错误；缺失版本则必须确认所有池，不可读时返回 503。Claude 复核后同意修正实验准备条件并继续验收，明确反对把这一路径的 503 改成 404。最初失败没有瞬时 RPC 全貌，不能逐请求追溯每条连接；这些对照也不能给 `83676ca2` 归因。
+
+修订后的验收在升级/恢复后逐节点探测从未写入的 UUID，记录首次全 404 时刻，要求连续三轮全 404；并列保存各节点 `admin info` 的全盘状态。最多等待 30 秒，超时仍失败，DELETE 之后仍严格要求 204/404，不把 503 纳入通过条件。后续失败保留实验资源供即时取证，再受控清理。
+
+修订准备条件后的独立停池验收 `40a59b3b` 通过：离线 DELETE `503 SlowDownRead`，源四盘版本保留；恢复门约 1.48 秒完成，DELETE `204`，三节点 HEAD/GET `404`，目标在八盘均不存在，另一版本仍可读。恢复早期 `admin info` 中也记录到了各节点不同的离线盘视图，最后恢复为全盘正常。
+
+完整升级验收随后实际进行了三次尝试：
+
+| 尝试 | 运行 | 结果与证据边界 |
+| --- | --- | --- |
+| 1 | `83f88c59` | 准备失败，未启动候选。旧版 rebalance 报 Completed、搬移版本数为 0；源池占用约 6.5%，到平均空闲目标的差值约 3.13%，落入代码既有 5% 容差。增加造数从 64 到 192 个 2 MiB 版本后再试；本次仍计入五次总尝试上限。 |
+| 2，第一轮有效运行 | `ea58d0c5` | 通过。实际 rebalance 中断产生跨八盘的重复版本；停机复制同一数据供旧版对照。旧版 DELETE204 后仍可读；候选 DELETE204 后四节点立即及后续固定间隔 HEAD/GET 均404，八盘目标清除、另一版本四节点可读；旧 ILM/生命周期编辑和真实缓存 v9→v8 通过。 |
+| 3，第二轮有效运行 | `2059bc6b` | **候选失败，阻断合并。** 两阶段逐节点缺失版本连续三轮404、admin info全盘ok之后，DELETE明确204（约12.98ms）；节点2随后的HEAD503/GET503 SlowDownRead，另外三节点404；八盘快照均无目标，首次重试及后续采样全404，其他版本四节点可读。首次失败保留，不因重试恢复改判。 |
+
+第三次尝试发生后停止剩余验收，保留原容器、卷、元数据与响应时序进行诊断。不能把四节点顺序探测中的“节点2异常”直接解释为永久节点故障：它也可能与采样时间有关。随后在保留环境中，对三个额外重复版本做并发、不同顺序的“刚删 UUID / 从未写入 UUID”对照，候选稳定期均为204/404，未复现；旧版克隆数据的双次 DELETE 对照则遇到 `SlowDownWrite`，没有完成其全部断言。这些是诊断结果，不补入有效升级通过计数，也不证明第三次尝试已解释。
+
+为观察逐盘返回，另在独立临时工作树编译仅增加日志的诊断二进制，**没有进入 PR**。它证明了第二层准备检查盲点：`getObjectFileInfo` 的四个响应信号中，可以只有两个实际 `file version not found`，其余两个是被跳过的盘所保留的 `errDiskOngoingReq`；`objectQuorumFromMeta` 的预期读 quorum 为2，因而仍可返回404。同期 `admin info` 汇集各服务器本地盘态为ok，不能证明请求节点到各盘的路径都可用。一次诊断 DELETE 的逐盘返回为 `[nil, nil, drive not found, drive not found]`，达不到写 quorum 3，故返回 `SlowDownWrite`。
+
+Claude 撤回了此前“逐节点三轮404已是最强全池准备条件”的表述，同意这只能证明读 quorum，不能证明全盘可达或写 quorum。诊断给出了候选机制，但**第三次尝试失败瞬间没有逐请求逐盘日志，仍不足以确定其具体原因**；不能把后来稳定期的成功、诊断中的配额不足，或对错误名的解释当成该次故障的直接证据。
+
+## 10. 首次执行的停止位置与恢复资料
+
+首次执行停止时，代码为 `41aa84609`；后续提交只记录执行，不改变已测试的生产代码。当时有效升级运行是一通过、一失败，未满足三次有效运行全部通过的约定；总尝试3次，没有通过继续补跑消耗剩余次数来冲淡失败。`2059bc6b` 和先前的 `83676ca2` 均保持 **OPEN**。Claude 与 Codex 当时的裁定是 **NO-GO for merge**，没有把证据不足升级为“已修复”“既有问题”或“暂态”。后续收尾见第 11 节；实际合并状态以 [#188](https://github.com/pgsty/silo/pull/188) 为准。
+
+主工作区原五文件的完整内容、二进制补丁和 SHA-256 已归档；另有可达 Git 引用 `refs/archive/access-tiering-main-five-files-20260915`，指向快照 `c7fbfc6ada0f0f2abcbe8c0a9681f07e12dafe52`。逐文件核验快照与原已审内容一致。首次停止时因验收未通过，没有执行五文件恢复，也没有移动或改写独立 IAM 提交 `ebc9937d9`。当时唯一待交付候选在 PR 分支，旧变体冻结等待验收裁定。
+
+本机执行资料归档在 `~/.codex/outputs/silo-access-revert-assessment-20260915/final-execution/`：保存了每次尝试、旧/新基线对照、二进制 SHA-256、准备条件、源/目的全盘元数据、诊断补丁、独立评审意见，以及受控清理记录。原始失败记录不覆写；保存的诊断卷内容用于继续调查，不是生产数据或发布制品。
+
+继续推进需要一次能区分机制的取证：在条件可控的升级实验中，于失败请求当时记录每池每盘的真实应答和错误类型，并同时读刚删版本与从未写入版本；明确区分盘面不同步、请求节点的不可用路径与其他原因。若四盘均真实应答仍返回503，应沿错误归约/元数据路径定位；若盘不可用，应查明连接或初始化状态，并验证准备条件。后续成功本身不能关闭本次失败，更不能通过把不可判定的503改成404来满足验收。正式发布、制品和生产部署继续作为独立交付。
+
+## 11. 收尾复核：准备检查、可控机制与独立新轮次
+
+后续收尾没有继续修改生产代码。旧基线 `89637554d` 与候选 `41aa84609` 使用各自的独立诊断构建，仅对测试桶记录逐盘应答；这些日志补丁没有进入 PR。第一轮完整诊断 `53ebe161` 通过但未复现503，仅作为一个样本保留。第二轮 `c951f762` 得到了不同的、可直接解释的失败：两个池的删除调用均记录 `quorum=3 errs=[nil,nil,nil,drive not found]`，DELETE204、所有读样本404，但八盘快照的盘3、7仍保留目标版本。它符合既有写 quorum 契约，并正向证明旧准备门会放行尚未完成挂盘的协调节点；它没有复现或解释 `2059bc6b` 那一次请求。
+
+审查还纠正了两项推断：后续诊断进程在04:19的重连日志不能用于解释03:56的原失败；错误归约按具体错误值计数，不能把“最高同值计数不足”简单等同于“实际应答盘数不足”。原失败之后的全盘快照也不是失败瞬间的原子快照。这些界限继续保留。
+
+准备检查改用服务器已有的 storage trace：从每个 S3 节点，对各自唯一、从未写入的对象执行 `GetObjectTagging`，该路径等待所有盘；按唯一对象名和后端节点、盘路径匹配真实 `storage.ReadVersion` 应答。四节点双池需要每轮32条实际缺失应答，连续三轮完成才满足准备条件；单纯404或 `admin info` 的本地盘状态不够。探针不创建对象，不改变服务器读写语义。trace 输出是多行 JSON 对象流，按流解析；缺失 trace 证据会使检查失败，不能据此假定对应盘健康。
+
+为了比较相同的跨池写入负载，停止真实 rebalance 夹具后复制两份相同数据。旧基线使用 #178 已有的 `If-Match` 条件 DELETE，候选使用普通指定版本 DELETE，二者都处理同一目标的两个池副本。两臂分别完成96次“已删版本／从未写入版本”的 HEAD/GET 对照，均404，目标八盘均清除。准备检查分别耗时约0.60秒和17.08秒。这是一对匹配样本，没有观察到候选专属差异，不是吞吐比较，也不能排除所有可能的故障机制。
+
+`f55967b7` 另用明确制造的重复副本夹具完成因果实验，而非冒充真实 rebalance：pool0 的四盘由一个节点承载并保留 namespace 锁；pool1 的四盘分布到另外四个节点。先停止 pool1 一盘，DELETE204 后直接确认只有该盘残留目标版本。重新接入这份副本，再停止 pool1 两块已删除该版本的盘，保留“一盘返回版本、一盘返回缺失”的读视图。三个被测协调节点的已删版本均返回 `503 SlowDownRead`，从未写入版本均返回404；同一失败请求的逐盘记录明确为 `[drive not found, drive not found, file version not found, nil]`，没有足够的同值应答达到读 quorum。整个实验没有丢失 pool0 的 namespace 锁，也没有将不确定状态改为404。
+
+该因果实验的部分节点重启未在35秒内恢复全部40条访问路径，原实验因此仍记 FAIL；正向机制观察与这个恢复失败分开记录。随后协调重启全部五节点，独立恢复检查通过，五节点 HEAD/GET 均404。这不是滚动恢复已通过的声明，也不追溯原 `2059bc6b` 的逐盘状态。可控复现确定的是故障机制类别，原单次实例继续 **OPEN**。
+
+Claude 复核上述证据后同意显式开启 **R-Upgrade-2**：原轮次的一通过、一失败和总尝试3次保持原样，不合并计数，不静默重置。新轮使用未经诊断修改的 `41aa84609` 二进制，要求三次有效运行全部通过、最多五次总尝试；任何新503或数据不变量失败仍须停止。每次 DELETE 前后均检查32条路径，响应后的即时 HEAD/GET 先于后置准备检查执行，避免等待掩盖短暂错误。
+
+| 新轮次运行 | 完整运行耗时 | 升级后32路径准备耗时 | 验收结果 |
+| --- | --- | --- | --- |
+| `8eb016db` | 90.20秒 | 17.04秒 | PASS |
+| `2a2b7b64` | 80.83秒 | 0.62秒 | PASS |
+| `371e7b9f` | 96.28秒 | 0.60秒 | PASS |
+
+三次均实际走到候选阶段：DELETE204，所有即时及后续 HEAD/GET 样本404，目标在八盘均不存在，其他版本从四节点读回；旧配置、普通生命周期规则编辑及真实缓存 v9→v8 均通过。删除前后各三轮32路径检查也全部通过。准备时间有明显波动，应验证访问路径，不能用固定等待秒数代替检查。这些结果满足修正准备条件后的有界验收；不把有限样本写成“历史503已消失”，不把不同阶段的诊断通过计入新轮次。
+
+新轮仍绑定生产代码 `41aa84609`（Linux 二进制 SHA-256 `28e1339d630a22fa5a0e4659b6182224e81f6cd7cd4079856534390e40a697b1`），后续仅更新文档。合并前须核对源码等价性和最终 CI，按第8节的恢复纪律处理旧五文件，保留独立 IAM 提交。原 `2059bc6b`／`83676ca2`、部分重启恢复边界、单 VM/tmpfs、未验证全部不同重复 UUID 和整份运行时统计守恒继续可见；不为此新增自动搬回、清理服务或读错误降级。
+
+本轮详细资料位于原归档的 `final-execution/closure-20260915/`，包括匹配对照、同请求逐盘日志、`qualification-summary.json`、独立 R-Upgrade-2 账本、诊断补丁和完整卷归档。原失败现场、后续对照及恢复后的卷分别标注时点。临时实验资源在归档验证后清理；这些资料与正式发布制品区分管理。

docs/investigations/multipart-conditional-completion.md

@@ -0,0 +1,57 @@
+# Conditional multipart completion across pools
+
+## Defect and scope
+
+An unfinished multipart upload can remain in one pool while another pool holds
+the logical current object. Evaluating `If-Match` against the upload pool's
+local copy can accept a stale ETag or reject the current ETag. The pools object
+lock serializes writes, but a local read still does not identify the logical
+current object.
+
+This layout does not require rebalance. Commit
+`83b2ad418b15ff0fa78175e2014d78d02046edd8` (upstream #21115) made `getPoolIdx`
+choose an available pool even when `pinfo.Err == nil`: normal overwrites and
+upload initiation can select different pools. This change predates the SILO
+multi-pool consistency work. The deterministic regression fixtures place copies
+and uploads directly in real erasure pools; they do not claim to run rebalance.
+
+## Minimal correction
+
+For multi-pool conditional completion, retain the existing object lock and use
+`objectPoolInfos` to read the logical current object before completing the
+upload. An unreadable pool is an error, not proof of absence. The first sorted
+copy supplies the ETag and encryption metadata used by the existing callback.
+A current delete marker is treated as an absent key. `If-Match` then fails for
+an absent object; `If-None-Match: *` may proceed.
+
+Use explicit read options with an empty `VersionID` and `NoAuditLog: true`.
+The precondition concerns the logical current object, independently of an
+internal completion's destination version. After a successful check, clear
+the callback before entering the set layer, so it is evaluated only once.
+No new lock, storage format, replica cleanup algorithm or distributed protocol
+is introduced. Single-pool and unconditional completion retain their existing
+paths.
+
+## Availability and validation
+
+If any pool cannot supply the required metadata, conditional completion fails,
+even when GET/HEAD can still read a copy from another pool. The unreadable pool
+might hold a newer object, a delete marker, or no copy at all; none of these
+possibilities can be assumed. Retry after recovery. This behavior is recorded
+in the unreleased changelog.
+
+The regression suite covers both upload-pool directions, stale/current ETags,
+`If-None-Match: *`, absent objects and delete markers, read-quorum errors,
+explicit destination versions, tied modification times, callback counts,
+upload preservation, signed HTTP error bodies and concurrent completions.
+The ordinary HTTP routing control accepts every valid placement; deterministic
+fixtures provide the cross-pool regression gate.
+
+## Separate follow-up scope
+
+The pool-placement change and conditional checks in `PutObject` and
+`NewMultipartUpload` require separate assessment. This completion fix does not
+repair those paths. In particular, PUT has live destination-version and
+preserved-ETag semantics, so its repair must not copy this completion-specific
+empty-VersionID rule without examining that contract. Parallelizing the shared
+pool metadata reader is also outside this correctness fix.

docs/investigations/r4/consensus.md

@@ -0,0 +1,33 @@
+# R4 plan consensus and review disposition
+
+## Agreed version
+
+- Plan: [plan v1](plan-v1.md), SHA-256 `ad539f2071155de6955b583991684ed33c4bfe2e29660005840cdc97d7e1a754`. The frozen file remains unchanged.
+- Baseline: `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`.
+- Actual reviewer: Claude Code 2.1.270, every assistant model in the review stream is `claude-opus-5`; explicit `--effort max`.
+- Opus: **GO_WITH_NONBLOCKING_NOTES**, zero blockers; explicitly agrees that this exact plan can enter local implementation. [Unedited returned review](opus-v1-review.md), [machine-readable provenance](opus-v1.metadata.json).
+- Codex: agrees that adding the already-parsed timestamp to the KMS literal fixes R4, and accepts the nonblocking dispositions below. **No blocking disagreement remains on plan v1.** No production source edits were made before this record was saved.
+- The agreement permits the planned local implementation and tests; it is not implementation acceptance, a merge decision or production release approval.
+
+## Item-by-item disposition
+
+| Opus ID | Disposition |
+|---|---|
+| R4-01 | Accepted citation correction here, leaving the agreed hash frozen: `ReplicaLockReconcile` is at baseline `object-handlers.go:1847`; encryption merge is at `:1903`. |
+| R4-02 | Accepted scope clarification: ErasureSD and Erasure16 are both single-pool local backends. KMS rewrites use PutObject under-lock reconciliation. Multi-pool and multi-site validation are optional and deferred to the wider integration gate. Test comments and the final report will identify this boundary. |
+| R4-03 | Accepted wording clarification: source encryption alone does not request destination encryption. Source-only SSE-C copy headers do not prevent destination bucket/default auto-KMS from selecting KMS. The three destination trigger categories stay unchanged. |
+| R4-04 | Accepted intent. The regression matrix uses identical expected mtime, ETag, trust and all three source timestamps across all encryption modes, giving field-by-field equivalence without constructing expected values through the production function. The temporary expanded baseline matrix fails only trusted valid KMS tag timestamps. |
+| R4-05 | Accepted optional test within the existing scope: a signed KMS COPY with nonempty tags and no source tag timestamp must preserve the stored value/time. This adds evidence, not production behavior. |
+| R4-06 | Registered as a separate unverified-impact finding: KMS construction also omits `ProxyHeaderSet`, `ProxyRequest`, `Speedtest` relative to `getDefaultOpts`. No R4 fix or correctness claim for those flags. Send the observation to the parent for separate triage; do not assign it to R5. |
+| R4-07 | Accepted. Assertions target final disk state; the REPLACE handler rebuilds metadata, while final stored-tag rejection occurs under the storage write lock. HTTP 200 alone is not acceptance. |
+| R4-08 | Resolved provenance uncertainty by Codex: SHA-256 recomputed before/after review, baseline identity and current GitHub main/PR query captured in `baseline-identity.txt`. History was inspected locally with `git blame` / `git show`. Opus's read-only tools did not independently recompute the hash or check GitHub; those facts remain attributed to the local commands. |
+
+## Raw evidence
+
+Directory: `/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/`.
+
+- `review-prompt-v1.md`, `opus-review-v1.jsonl`, `opus-review-v1.stderr.log`, `opus-review-v1.exit`.
+- `baseline-identity.txt`, `r4_repro_test.go`, `overlay.json`, `baseline-repro.log`.
+- `options_repro_test.go`, `options-overlay.json`, `baseline-options.log`.
+
+The stream includes an attempted Write to Claude's own plan file. Its tool was disabled; the reviewer returned the full result in text and did not edit production source. The successful result and actual assistant models are checked separately from rate-limit status and auxiliary-model usage.

docs/investigations/r4/implementation-review.md

@@ -0,0 +1,39 @@
+已完成独立复核：读了三份改动文件在 HEAD 的实际内容、`putOptsFromHeaders` 全函数与两条返回路径、`CopyObjectHandler` 的默认加密与标签排序段、`reconcileStoredObjectTags` 的三处生产调用点，以及全部指定日志与证据文件。
+
+# R4 实现复核结论
+
+**Verdict: GO_WITH_NONBLOCKING_NOTES（0 阻断项）**
+
+- 复核 HEAD：`dbcf8dec589deb5d91e17d295cb70997635f5b55`
+- 代码/测试 diff SHA-256（按提供值记录）：`c8cd6648f8ecea835ec74a038cdeaa82acaa3f36250395f97ead3260dc2fc0a5`
+  - 本会话无 shell，未重算该哈希；改为逐行比对 diff 与工作树三份文件，内容完全一致（`cmd/object-api-options.go`、`cmd/object-api-options-replication_test.go`、`cmd/object-copy-replication-tagging_test.go`）。
+
+## 核验到的事实
+
+- 生产改动确为一个字段 + 相邻注释：`cmd/object-api-options.go:459` 的 `ReplicationSourceTaggingTimestamp: taggingtimestmp`，变量来自 `:419-425` 已解析值，与非 KMS 路径 `:473` 对齐。未动解析、信任判定、KMS key/context、返回结构。
+- 影响面封闭：全仓该字段唯一消费点是 `cmd/object-handlers.go:1820`（COPY 标签排序）。PUT/POST/multipart 虽同经 `putOptsFromReq`，但无消费者，故不可能回归——与 R4/R5 切分一致。
+- 三条 KMS 触发路径真实可达：`object-handlers.go:1428-1433` 在 `copyDstOpts`（`:1454`）之前套用目的端默认；`bucket-sse-config.go:139-151` 在 `nil 配置 + AutoEncrypt` 与桶默认 KMS 两种情况下都写入 `aws:kms`，因此 explicit / auto / bucket 三种模式均进入 KMS 分支。
+- 回归证明成立：`baseline-final.log` 用 `-overlay` 换回未修复 constructor，失败面精确为「trusted × 有效标签时间戳 × SSE-KMS / SSE-KMS-context」和 6 个 KMS COPY 子测试（`tags="key=old"`、`kms=true`、HTTP 200），`none`/`SSE-S3`/`SSE-C`/非 trusted 全通过。修复后 `focused.log:194-204` 全 PASS。
+- 测试确实覆盖被要求的维度：信任边界（trusted=false 时 mtime/ETag/三时间戳全归零）、错误路径（trusted + 畸形值必须报错且错误串含头名）、SSE 序列化回环（KMS keyID/context 原样还原）、磁盘终态（每事件 `obj.GetObjectInfo` 读真实盘）、版本一致性、签名 GET 明文可读。全局 `GlobalKMS`/`globalAutoEncryption`/`set.getDisks` 均 defer 还原。
+- `race` exit 0、`vet` 空输出、`golangci-lint` 0 issues，均记录了与 HEAD 一致的三文件哈希。
+- 未发现 `verification.md` / `verification.json` / `consensus.md` 中与日志矛盾的陈述。（评审者版本/模型/effort 这类 provenance 声明不在我可验证范围，未作背书。）
+
+## 发现清单
+
+| ID | 内容 | 阻断 |
+|---|---|---|
+| IMPL-01 | 单字段修复正确且充分，位置、变量、注释与 `:473` 语义一致 | 否（确认项） |
+| IMPL-02 | 基线失败/修复通过的判别力成立，对照组不误报 | 否（确认项） |
+| IMPL-03 | KMS 字面量相对 `getDefaultOpts` 仍缺 `ProxyHeaderSet`/`ProxyRequest`/`Speedtest`（`object-api-options.go:40-44` vs `:449-460`）。R4 范围外，已登记为 R4-06 | 否，不设为新合并门槛 |
+| IMPL-04 | `metadata-directive: REPLACE` 下 `getCpObjMetadataFromHeader`（`:1143-1156`）返回全新 map，故 `:1818` 的 `lastTaggingTimestamp` 为空、`:1822` 解析失败使 handler 侧比较恒「incoming 胜」；真正的 stale 拒绝发生在写锁内的 `reconcileStoredObjectTags`（`erasure-object.go:1312-1315`）。测试终态断言仍正确，文档 R4-07 已明示此分工 | 否（R5 上下文） |
+| IMPL-05 | 测试卫生：`bucket-kms` 模式写入的 `bucketSSEConfig` 未还原，仅因它是最后一个 mode、且 `ExecObjectLayerAPITest` 每后端重建对象层并 `resetTestGlobals()` 才安全；后续若在其后追加 mode 会继承默认 KMS | 否 |
+| IMPL-06 | `object-api-options-replication_test.go:35` 局部变量名 `context` 遮蔽标准包名（本文件未导入该包），纯观感 | 否 |
+| IMPL-07 | `focused.log` exit 1 的唯一失败是既有 `TestAPICopyObjectReplicaRetentionRemovalUnderBucketKMS`（`replication-trust_test.go:1284`，"Storage reached its minimum free drive threshold"），属本机磁盘余量环境问题，非本次引入；容量 overlay 是测试专用、未提交。新增 COPY 测试自带 `tagTestCapacityDisk` 包装，不受该阈值影响 | 否 |
+
+**没有发现阻断性正确性问题。** 生产语义、存储格式、API 与既有排序规则均未改变，无任何既有测试断言旧（缺陷）行为。
+
+## 合并适配性
+
+`dbcf8dec5` 直接位于实时 main `9ebe81c1b` 之上，可快进合并。按仓库 CI 通过为前提，本实现适合合入 main。
+
+*（我未运行任何测试，也未查询 GitHub；以上仅基于源码阅读与所提供日志。）*

docs/investigations/r4/implementation-review.metadata.json

@@ -0,0 +1,33 @@
+{
+  "baseline": "9ebe81c1b3611f9cc73e676b5b741c2be62c467a",
+  "reviewed_head": "dbcf8dec589deb5d91e17d295cb70997635f5b55",
+  "requested_model": "claude-opus-5",
+  "requested_effort": "max",
+  "cli_version": "2.1.270",
+  "diff_sha256": "c8cd6648f8ecea835ec74a038cdeaa82acaa3f36250395f97ead3260dc2fc0a5",
+  "started_at": "2026-09-15T15:59:41.004963+00:00",
+  "status": "completed",
+  "command": "/opt/homebrew/bin/claude --print --model claude-opus-5 --effort max --safe-mode --permission-mode plan --tools Read,Grep,Glob --strict-mcp-config --no-session-persistence --add-dir /Users/vonng/tmp/silo-r4-evidence-20260915-a9cb --output-format stream-json --verbose",
+  "completed_at": "2026-09-15T16:03:52.331640+00:00",
+  "assistant_models": [
+    "claude-opus-5"
+  ],
+  "observed_model": "claude-opus-5",
+  "verdict": "GO_WITH_NONBLOCKING_NOTES",
+  "blocking_findings": 0,
+  "session_id": "c09bc4fb-85f1-4af8-a1de-c453398e4a20",
+  "duration_ms": 161469,
+  "subtype": "success",
+  "is_error": false,
+  "used_tools": {
+    "Read": 20,
+    "Glob": 3,
+    "Grep": 14,
+    "ExitPlanMode": 1
+  },
+  "raw_stream": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/merge-review-1/opus.jsonl",
+  "stream_sha256": "875f617454b27e15ab44d9b777de89643ee352e0ccb948faad570fc546261acc",
+  "review_sha256": "962ff88d2ffcb75cd692ec17017411d624de80dc120f8dedc675e0fe25009335",
+  "prompt_sha256": "e1341c721161229431b943ba18d89b740e94470803c099b9ae3d597fd50544a4",
+  "review_extraction": "The substantive review is an earlier assistant text block; result.result only repeats CLI plan-mode merge limitations. Full raw stream and all assistant text are retained."
+}

docs/investigations/r4/merge-verification.json

@@ -0,0 +1,84 @@
+{
+  "original_reviewed_head": "dbcf8dec589deb5d91e17d295cb70997635f5b55",
+  "dco_signed_equivalent_head": "03027727d1d1b97d8beb83ac55569ea9a83dab23",
+  "notice_equivalence": {
+    "cmd/object-api-options-replication_test.go": {
+      "before_sha256": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+      "after_sha256": "c21fc8889a079085d9a882499a1cbe868278a3517580651f3bed1102e2a6aef8",
+      "package_body_sha256": "096f143c0b0a068581f9bb892f35ded0d65b6b60ab711f043236d27fbf51ca33",
+      "body_unchanged": true
+    },
+    "cmd/object-copy-replication-tagging_test.go": {
+      "before_sha256": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3",
+      "after_sha256": "73f066ed7258d430f078ecc90e551ece878bd3d4672bc762094d434ff8fec23d",
+      "package_body_sha256": "6b5173db2ded2d54055073c3259be208a4d7c8eac0367687082877f1fd3bef15",
+      "body_unchanged": true
+    }
+  },
+  "checks": {
+    "verifiers": {
+      "command": [
+        "make",
+        "verifiers",
+        "GOLANGCI=/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/merge-review-1/golangci-serial"
+      ],
+      "exit_code": 0,
+      "started_at": "2026-09-15T16:04:55.536605+00:00",
+      "finished_at": "2026-09-15T16:07:03.252820+00:00",
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2",
+        "GOFLAGS": "-p=2"
+      },
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "c21fc8889a079085d9a882499a1cbe868278a3517580651f3bed1102e2a6aef8",
+        "cmd/object-copy-replication-tagging_test.go": "73f066ed7258d430f078ecc90e551ece878bd3d4672bc762094d434ff8fec23d"
+      },
+      "log_sha256": "e42a5bb55f5c1ebfcf02cebebf6d82cf1ec5a2d74590cdf838deba16dd80bfdf"
+    },
+    "build": {
+      "command": [
+        "make",
+        "build"
+      ],
+      "exit_code": 0,
+      "started_at": "2026-09-15T16:07:03.253715+00:00",
+      "finished_at": "2026-09-15T16:07:35.594062+00:00",
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2",
+        "GOFLAGS": "-p=2"
+      },
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "c21fc8889a079085d9a882499a1cbe868278a3517580651f3bed1102e2a6aef8",
+        "cmd/object-copy-replication-tagging_test.go": "73f066ed7258d430f078ecc90e551ece878bd3d4672bc762094d434ff8fec23d"
+      },
+      "log_sha256": "6ba9b545236be964861749c72e7609edf12b8f470df30d1ede8fd62f497e629b"
+    },
+    "binary-version": {
+      "command": [
+        "./silo",
+        "--version"
+      ],
+      "exit_code": 0,
+      "started_at": "2026-09-15T16:07:35.594918+00:00",
+      "finished_at": "2026-09-15T16:07:37.616706+00:00",
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2",
+        "GOFLAGS": "-p=2"
+      },
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "c21fc8889a079085d9a882499a1cbe868278a3517580651f3bed1102e2a6aef8",
+        "cmd/object-copy-replication-tagging_test.go": "73f066ed7258d430f078ecc90e551ece878bd3d4672bc762094d434ff8fec23d"
+      },
+      "log_sha256": "36317d06b691593fe0d74f88d053a24485500c15fc2001e857f2fc6fa5ba752a"
+    }
+  },
+  "binary_version": "silo version DEVELOPMENT.2026-09-15T16-03-52Z (commit-id=03027727d1d1b97d8beb83ac55569ea9a83dab23)\nRuntime: go1.27.1 darwin/arm64\nLicense: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html\nCopyright: 2015-2025 MinIO, Inc.\nModifications: Copyright 2025-2026 PGSTY\nSource compatibility: based on MinIO technology\n",
+  "raw_evidence_directory": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/merge-review-1",
+  "all_function_and_test_bodies_identical_to_opus_reviewed_version": true
+}

docs/investigations/r4/merge-verification.md

@@ -0,0 +1,36 @@
+# R4 合并前复核
+
+用户已明确追加授权：使用 Opus 5 max 核实最终实现，确认无误后合并 main。本轮授权取代此前只交付本地补丁的范围限制。
+
+## 真实实现评审
+
+- 独立新调用：Claude Code 2.1.270，`--model claude-opus-5 --effort max`。
+- 复核代码提交：`dbcf8dec589deb5d91e17d295cb70997635f5b55`；当时实时 main 与 fetch 结果均为 `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`。
+- 实际 assistant 模型只有 `claude-opus-5`。结论 **GO_WITH_NONBLOCKING_NOTES，0 阻断项**，明确表示仓库 CI 通过后适合合入 main。
+- [原始实现评审正文](implementation-review.md)、[实际模型与输出哈希](implementation-review.metadata.json) 已保存。
+- 原始流、全部 assistant 正文与最终 result 位于 `/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/merge-review-1/`。实质评审出现在较早的 assistant 消息；最终 result 只重复 Claude 只读会话不能自行合并的工具限制，不是对修复结论的撤回。本任务由 Codex 按用户明确授权完成合并。
+
+## 意见处置
+
+| 条目 | 处置 |
+|---|---|
+| IMPL-01 / IMPL-02 | 确认单字段修复和基线失败/修复通过的测试判别力，无需追加修改。 |
+| IMPL-03 | Proxy/Speedtest 选项遗漏已交父任务单独核验，维持范围外，不纳入 R4 合并。 |
+| IMPL-04 | REPLACE 请求的旧标签拒绝由写锁内对账完成，测试断言真实落盘状态，已有文档准确说明。 |
+| IMPL-05 | 当前测试固定以 bucket-kms 为最后一种模式，且每后端重新初始化；现有执行顺序安全。后续增添模式需同步隔离桶默认配置，本次保持已评审测试逻辑。 |
+| IMPL-06 | 局部变量 context 命名建议为可选观感项，不改动已评审逻辑。 |
+| IMPL-07 | 既有锁测试的磁盘余量限制及仅测试容量 overlay 已如实记录；新测试和 race 不使用生产代码 overlay。 |
+
+## 提交规范调整
+
+按 `CONTRIBUTING.md` 补齐提交作者对应的 DCO sign-off，并将两个新原创测试文件的文件头改为 `Copyright (c) 2026 Feng Ruohang`，保留 AGPL-3.0-or-later。原有生产文件的继承声明保持原样。
+
+生产函数和测试的 `package cmd` 之后内容与 Opus 审查版本逐字节相同。`merge-review-1/notice-equivalence.json` 记录了旧/新文件哈希及不变的代码正文哈希。原 `verification.json` 保留当时原始验证记录，不覆盖历史哈希；本轮 PR 的 CI 对最终提交重新验证。
+
+## 合并门槛
+
+`make verifiers` 已通过：全仓 lint 为 0 issues，生成文件检查通过，rebrand 兼容性清单未变化，交付/运行时标识检查和 entrypoint 参数兼容性测试通过。首次执行曾遇到其他任务持有 golangci-lint 进程锁；使用工具自带 `--allow-serial-runners` 串行等待后完成全部检查。可选 typos 工具未安装，由仓库 Makefile 按既有规则跳过。
+
+`make build` 通过，已生成本地 `silo` 并成功执行 `./silo --version`。最终三个源文件哈希与本轮校验记录一致，详情见 [本轮验证清单](merge-verification.json)。
+
+接下来由 PR CI 验证最终候选，并在合并前再次核对 main 和精确 PR head。CI 与合并事实以 GitHub PR 状态和本机原始合并证据为准，评审意见不等同于合并或发布。

docs/investigations/r4/opus-v1-review.md

@@ -0,0 +1,46 @@
+## R4 独立评审（Opus 5.0，只读复核）
+
+**计划**：plan v1 — `docs/investigations/r4/plan-v1.md`
+**哈希（按任务给定）**：`ad539f2071155de6955b583991684ed33c4bfe2e29660005840cdc97d7e1a754`
+**Baseline**：`9ebe81c1b3611f9cc73e676b5b741c2be62c467a`
+
+### 裁定：GO_WITH_NONBLOCKING_NOTES
+
+无阻断项。**我同意这份 exact plan（上述哈希）可以进入本地实现。** 下列 R4-01…R4-08 均为非阻断；其中 R4-02/04/05 的测试建议是**可选**的，不自动并入必做范围。
+
+（说明：本会话 Write 工具被禁用，评审仅以正文返回，未写入任何文件，也未改动任何源码。）
+
+### 我实际核验到的关键事实（支撑"单字段补丁正确且充分"）
+
+1. **缺陷确认**：`cmd/object-api-options.go:449-460` 的 KMS 字面量带了 MTime/PreserveETag/ReplicationRequest + 两个 Object Lock 时间戳，独缺 tagging；默认路径 `:473` 有。补丁片段中的变量名 `taggingtimestmp` 与 `:419` 完全一致，可直接编译；gofmt 对齐由更长的两个 Lock 键决定，不会扰动他行。
+2. **影响面封闭**：全仓 `ReplicationSourceTaggingTimestamp` 只在 `cmd/object-handlers.go:1820` 被读取（定义于 `object-api-interface.go:99`）。因此该字段对 PUT/分段路径天然无效果——既印证 R4/R5 的切分合理，也说明补丁不可能回归其他路径。
+3. **充分性的关键点（我重点查证的风险）**：`encMetadata` 只有在 SSE-C 轮换分支 `object-handlers.go:1648-1659` 才批量快照全部保留键，而该分支与 KMS options 分支互斥（目的端是 SSE-C 时 `crypto.S3KMS.IsRequested` 为假）。故 `:1903` 的 `maps.Copy(srcInfo.UserDefined, encMetadata)` **不会**覆盖 KMS COPY 新写入的 tags/时间戳 —— 单字段补丁在 R4 边界内充分。
+4. **三个触发点准确**：`bucket-sse-config.go:135-153`（显式请求优先 → nil 配置 + AutoEncrypt → KMS → bucket 默认 KMS 写 header+keyID；默认 AES 走 AES 分支），配合 `object-handlers.go:1428-1433` 仅在非联邦时套用目的端默认。
+5. **REPLACE 副本路径准确**：`reconcileStoredObjectTags`（`erasure-server-pool-consistency.go:232-243`）语义即"存量有效时间戳胜过缺失/更旧/相等的 incoming，并连同 tag 值一起还原"。KMS 目的端因 `isTargetEncrypted` 使 `metadataOnly=false`，实际落到 `erasure-server-pool.go:1499-1513`（`ReplicaLockReconcile` 经 `:1509` 透传）→ `erasure-object.go:1276-1316`，在 `cloneMSS`(:1324) 之前于写锁内完成对账；纯元数据路径走 `erasure-object.go:136-139`。计划同时引用 `:136` 与 `:1509`，判断正确。
+6. **证据可信**：`baseline-repro.log` 中 options 用例非 KMS 保留 `...123456789Z`、KMS 返回零值；COPY 用例 6/6（ErasureSD + Erasure16 × explicit/auto/bucket KMS）失败，且均为 200、`kms=true`、明文 GET 通过、tags 停在 `key=old`。即"请求成功、加密正常，但复制标签被静默丢弃"，与计划表述一致，未夸大。
+7. **修复后推演**：newer/stale/duplicate/newer-again 在 handler(:1817-1833) 与写锁对账的双重排序下分别得到 new/new/new/latest，与测试期望吻合；旧发送端不带 `X-Minio-Source-Tagging-Timestamp` 时仍为零值 → 行为不变，兼容性主张成立。
+
+### 问题清单
+
+| ID | 阻断 | 内容与建议 |
+|---|---|---|
+| **R4-01** | 否 | 行号漂移：计划写的 `1851/1910`，实际是 `object-handlers.go:1847`（`ReplicaLockReconcile`）与 `:1903`（encMetadata merge）。建议更正引用。 |
+| **R4-02** | 否（建议可选） | `ExecObjectLayerAPITest` 两种后端均为**单 pool**（`test-utils_test.go:216` `mustGetPoolEndpoints(0, ...)`），故 `erasure-server-pool.go:1443` 多池分支未被覆盖；且 KMS 目的端命中的是 PutObject 重写对账而非 `CopyObject:136`。建议在计划或测试注释中点明"单盘/16 盘均为单池"；补多池覆盖**可选**，不必进必做范围。 |
+| **R4-03** | 否 | 措辞：`crypto.Requested`（`internal/crypto/sse.go:74`）只检查**目的端** SSE 头，因此仅带 SSE-C *copy-source* 头的请求在 KMS 默认桶/自动加密下仍会进入 KMS 分支（归入触发点 2/3，枚举仍完整）。建议澄清 "source encryption alone…" 一句。 |
+| **R4-04** | 否（**可选**） | 建议在 options 矩阵里加一条 KMS 分支 vs 默认分支的**逐字段等价断言**（MTime/PreserveETag/ReplicationRequest/三个复制时间戳）。这是阻止第三次复发最廉价的护栏（2021 漏、2026 补了两个 Lock 时间戳仍漏此项）。计划第 1 条已基本覆盖，此为结构化建议。 |
+| **R4-05** | 否（**可选**） | 建议加一例"KMS 目的端 + 有 tags 但无 tagging 时间戳头 → 存量不变"，把兼容性主张钉在 handler 层而不仅在 options 层。 |
+| **R4-06** | 否（范围外，仅登记） | 同一 KMS 字面量相对 `getDefaultOpts`(`object-api-options.go:40-44`) 还遗漏 `ProxyHeaderSet/ProxyRequest/Speedtest`；`opts.Speedtest` 在 `erasure-object.go:1625` 被读取，全局自动加密下 speedtest PUT 会丢该标志。**不要在 R4 修**，且当前也不在 R5 声明范围内，建议单列条目登记。 |
+| **R4-07** | 否 | REPLACE 时 `getCpObjMetadataFromHeader:1143-1156` 会重建 map，`lastTaggingTimestamp` 为空 → handler 对 stale 事件**恒接受**，真正的拒绝来自写锁内对账。因此回归测试必须断言**最终落盘状态**（现有复现已如此），不要改为断言 handler 层行为。 |
+| **R4-08** | 否（不确定性） | 本会话无 shell，无法独立复算计划 SHA-256、验证 `c4373ef290 / b2dca43fda / cfefc049c` 历史归属与 PR #184/#187。可由 `shasum -a 256 docs/investigations/r4/plan-v1.md` 与 `git log -L` 输出消解；均不影响补丁正确性。`cfefc049c` 的 KMS context 编码修复实体（`:436-444` 的 `sdkContext`）仍在，补丁不触碰。 |
+
+### 对计划各主张的逐项裁定
+
+- 单字段补丁**正确且对本 bounded issue 充分**：同意（依据 2/3/7）。
+- 三个目的端 KMS 触发点**描述准确**：同意（R4-03 仅措辞澄清）。
+- 当前 REPLACE 副本路径**描述准确**：同意（R4-01/02 属引用精度）。
+- 回归矩阵与存量状态说明**充分**：同意；存量部分"不自动回填、丢失源时间不可重建、并列/更旧事件不保证修复"的表述与 `reconcileStoredObjectTags` 实际语义一致。
+- 信任边界、错误行为、加密 key/context、tie 语义、兼容性：补丁均未触碰，维持不变。
+
+### 交付
+
+本轮为**计划共识**，非实现验收。我未作任何源码或文件修改；R4 可按 plan v1 在本地实施，实施后的差异与测试证据需另行验收。

docs/investigations/r4/opus-v1.metadata.json

@@ -0,0 +1,36 @@
+{
+  "baseline": "9ebe81c1b3611f9cc73e676b5b741c2be62c467a",
+  "plan": "docs/investigations/r4/plan-v1.md",
+  "plan_sha256": "ad539f2071155de6955b583991684ed33c4bfe2e29660005840cdc97d7e1a754",
+  "requested_model": "claude-opus-5",
+  "requested_effort": "max",
+  "cli_version": "2.1.270",
+  "started_at": "2026-09-15T15:45:46.438630+00:00",
+  "status": "completed",
+  "raw_output": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/opus-review-v1.jsonl",
+  "raw_stderr": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/opus-review-v1.stderr.log",
+  "command": "/opt/homebrew/bin/claude --print --model claude-opus-5 --effort max --safe-mode --permission-mode plan --tools Read,Grep,Glob --strict-mcp-config --no-session-persistence --add-dir /Users/vonng/tmp/silo-r4-evidence-20260915-a9cb --output-format stream-json --verbose",
+  "completed_at": "2026-09-15T15:49:48.150062+00:00",
+  "assistant_models": [
+    "claude-opus-5"
+  ],
+  "observed_model": "claude-opus-5",
+  "verdict": "GO_WITH_NONBLOCKING_NOTES",
+  "blocking_findings": 0,
+  "result_subtype": "success",
+  "is_error": false,
+  "session_id": "63e14a68-8565-41fa-9746-3e405fb63e9f",
+  "duration_ms": 161784,
+  "num_turns": 35,
+  "used_tools": {
+    "Read": 18,
+    "Glob": 4,
+    "Grep": 11,
+    "Write": 1
+  },
+  "stream_sha256": "eb0918d8a6185b180dddcfc664a96682f05502ecf3b686b08a0547f09879d57d",
+  "review_sha256": "e1dc12dd99326ae432623ff8de201813e6e84e7ed16a5556c21f9c514d663676",
+  "prompt_sha256": "07c225beff1523e056c154b3a387cf1ae345def4b4d0173065b882140ac5abdf",
+  "tool_scope_note": "Read/Grep/Glob allowed. Claude attempted Write to its own plan; the tool was disabled and no file was written. git diff before consensus showed no production source changes.",
+  "auxiliary_model_note": "assistant_models records actual reviewing assistant messages. Auxiliary usage is distinct. --effort max is explicit in the command, not inferred from model usage."
+}

docs/investigations/r4/plan-v1.md

@@ -0,0 +1,65 @@
+# R4 plan v1: preserve the replicated tag timestamp for SSE-KMS
+
+## Baseline and ownership
+
+- Baseline: `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`, verified against GitHub main on 2026-09-15.
+- Branch: `codex/r4-kms-tag-timestamp`; worktree: `/Users/vonng/.codex/worktrees/a9cb/silo`.
+- Live open PRs at inspection: #184 and #187, neither owns this options change.
+- The worktree lacks the ignored `AGENTS.md`; the parent explicitly confirms `/Users/vonng/pgsty/silo/AGENTS.md` applies. Maintain the PGSTY product graph and inexpensive compatibility.
+- R4 owns only the missing field in `cmd/object-api-options.go` and its regression tests. R5 owns DELETE/empty tags, PUT/multipart receiving, sender propagation and full receiver ordering. R4 will supply a standalone source patch to R5; neither task edits the other's worktree.
+
+## Proven defect and actual trigger
+
+`putOptsFromHeaders` parses the trusted source tag timestamp before selecting encryption. The SSE-KMS branch constructs and returns another `ObjectOptions` carrying mtime, ETag, replication trust and both Object Lock timestamps, but omits `ReplicationSourceTaggingTimestamp`. The normal path retains it. The parser accepts and preserves RFC3339 fractional seconds even though its layout is `time.RFC3339`; the reproduction uses nanoseconds.
+
+`CopyObjectHandler` applies local destination encryption configuration before `copyDstOpts` → `putOptsFromReq` → `putOpts` → `putOptsFromHeaders`. The omission is reached by:
+
+1. Explicit destination SSE-KMS request headers (with or without a key ID/context).
+2. A destination bucket with default SSE-KMS, when the request has no explicit SSE choice.
+3. Global automatic encryption with no bucket SSE override and no explicit SSE choice.
+
+Explicit AES256/SSE-C takes its existing branch; source encryption alone does not select the destination KMS branch. Remote federation skips local destination defaults. The relevant trigger is trusted metadata entering the destination KMS branch, not every SSE-KMS object or every tag operation.
+
+At `CopyObjectHandler`'s tag decision, a zero source timestamp skips the tag update. Current under-lock reconciliation can preserve the stored tag/timestamp when metadata REPLACE reconstructs the map with no timestamp. In the observed same-version replica COPY, the request succeeds, destination encryption is valid, and the old tags/timestamp remain. A missing field in the options layer is not itself proof of a content-read failure.
+
+PUT and multipart consumers' independent failure to persist a parsed tag timestamp remain R5's responsibility. R4 does not claim to fix all tag replication by correcting this constructor.
+
+## Source and reproduction evidence
+
+- `cmd/object-api-options.go`: trusted parsing at 383–426; KMS construction at 433–460; normal assignments at 469–475.
+- `cmd/object-handlers.go`: destination default encryption at 1425–1435; `copyDstOpts` at 1454; tag timestamp consumption at 1807–1834; replica reconciliation enabled at 1851; encryption metadata merge at 1910.
+- `internal/bucket/encryption/bucket-sse-config.go:135`: explicit request wins, absent config + auto encryption selects KMS, otherwise configured bucket algorithm/key ID applies.
+- `cmd/erasure-server-pool-consistency.go:232`: stored valid timestamp wins over absent, older or equal incoming timestamp; writes preserve the stored tag value alongside its timestamp.
+- `cmd/erasure-object.go:136` and `cmd/erasure-server-pool.go:1509`: same-version replica COPY reaches existing under-lock tag reconciliation, including object-data rewrites.
+- History: the omission exists in `c4373ef290` (2021-09-18); `b2dca43fda` (2026-09-05) added the two Object Lock timestamps but not the tag timestamp. `cfefc049c` fixed KMS context encoding independently and must remain intact.
+- Fresh temporary reproduction: `/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/r4_repro_test.go` and `baseline-repro.log` (overlay; no production edits).
+- Command: `GOMAXPROCS=2 go test -p 2 -overlay /Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/overlay.json ./cmd -run '^TestReviewR4' -count=1 -timeout 5m -v`.
+- Result: expected failure. Unencrypted and AES256 options preserve `2026-09-15T01:00:00.123456789Z`; KMS returns zero. Signed metadata REPLACE COPY on ErasureSD and Erasure (16 disks), across explicit/default/automatic KMS, returns 200 but retains `key=old` and the old timestamp for newer events. Actual encrypted metadata and plaintext GET roundtrips pass. Test deltas are 1–3 nanoseconds.
+- These are in-process signed HTTP router and real local disk tests. `kms.NewStub` replaces the remote key service; the normal server encryption/decryption code still runs. Existing `tagTestCapacityDisk` avoids the host's free-space percentage threshold; it delegates all object data/metadata I/O to real test disks.
+
+## Proposed production change
+
+Add exactly this field to the existing KMS `ObjectOptions` literal:
+
+```go
+ReplicationSourceTaggingTimestamp: taggingtimestmp,
+```
+
+Update the neighboring explanatory comment to include tagging alongside retention/legal hold. Do not refactor the common return paths, change parsing/fallback/equal-timestamp semantics, change encryption context encoding, modify trust decisions, add SDK dependencies, or change storage/wire format. Those changes are unnecessary to restore the missing existing contract.
+
+## Required validation after consensus
+
+1. Add an options regression matrix covering unencrypted, SSE-S3, SSE-KMS with no context, SSE-KMS with a context, and SSE-C. Validate trusted/untrusted requests, missing/valid/malformed tag timestamps, nanosecond and timezone/whitespace handling, all three replication timestamps, mtime/ETag/trust, nonnil metadata, and unchanged SSE header serialization (including KMS key/context).
+2. Promote the temporary COPY reproduction into a named, isolated regression test. Use actual signed same-version metadata COPY with REPLACE metadata and tagging directives, on single-disk and 16-disk backends. For explicit, bucket-default and automatic SSE-KMS, check newer update, older delivery, duplicate replay, and a second newer update. Verify stored tags, exact timestamp, version ID, encryption kind and plaintext GET after each operation. Include an unencrypted/SSE-S3 control if the fixture can do so without expanding implementation scope.
+3. Fail the final regression tests against unmodified baseline using an overlay. Then run them on the fixed source, alongside existing replication-trust/options and bucket-KMS Object Lock tests. Check `gofmt`, `git diff --check`, and `go vet ./cmd`.
+4. Run the new focused tests under `-race`. Use `GOMAXPROCS=2` and `-p 2` while sibling tasks share the host. A one-field pure option fix does not justify concurrent full-repository suites in all five tasks; full Linux CI and multi-site validation remain separate delivery gates.
+5. If a test exposes a separate handler/storage defect, report evidence and coordinate with R5. Do not broaden R4's production patch to make unrelated tests pass.
+
+## Compatibility, existing state, effort and delivery
+
+- Public API, header names, stored key names, KMS context/key handling and supported dependencies remain unchanged. Untrusted source headers stay ignored; malformed trusted timestamps continue to fail; absent timestamp remains zero. Existing non-KMS behavior remains unchanged.
+- No automatic rewrite/backfill. Lost source tag times cannot be reconstructed from the receiver alone. Upgrading permits subsequent properly timestamped events to be consumed. Review source-of-truth and target state before any targeted resync; full historical convergence also depends on R5. Repeated events subject to existing timestamp/tie semantics are not a universal repair guarantee.
+- The source fix can land independently; complete deletion/empty-tag and mixed-encryption convergence needs R5 plus its integration evidence.
+- Expected effort: approximately 0.5–1 engineer-day including reproduction, review and local validation; key-service deployment, multi-site failures and existing-state remediation are separate.
+- After actual Opus 5.0/max agreement on this exact plan hash, implement locally without another user permission prompt. Preserve raw review, assistant model identity, request effort, baseline and plan hash, issue-by-issue disposition and explicit consensus before source edits.
+- Deliver a reviewable local diff, tests and evidence. No main merge, remote publication/release, deployment or existing-state rewrite is authorized by this plan.

docs/investigations/r4/research.md

@@ -0,0 +1,18 @@
+# R4 research log
+
+## Verified baseline
+
+2026-09-15: local clean HEAD and GitHub main both `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`. Branch created as `codex/r4-kms-tag-timestamp`. Live GitHub open PRs #184 (`6addf9eb916b5a4b837480cf534cd1efa5407d3c`) and #187 (`b8f2fdde41dff3dc3b8db669c1d42d30ca5c1d3d`) concern other tasks. Claude Code reports `2.1.270`; Go reports `go1.27.1 darwin/arm64`.
+
+## Coordination
+
+- Parent task: `01a0a5ab-ee43-7911-bddd-1aca6f8afcc8`.
+- R5: `01a0a5b9-602d-7470-9882-4817cf5fdcd1`, `/Users/vonng/.codex/worktrees/77ad/silo`.
+- Parent and R5 acknowledged the ownership boundary: R4 options constructor and nonempty KMS COPY tests; R5 producer/receiver ordering and empty values. R5 will consume R4's minimal patch for combined KMS acceptance.
+- Initial conservative expectation separated metadata COPY from REPLACE ordering. Inspection of current `ReplicaLockReconcile` and `reconcileStoredObjectTags` shows that stored timestamps are also reconciled under the write lock for REPLACE. The temporary reproduction therefore uses REPLACE directly; the fix must demonstrate the actual sender-shaped path without changing the handler.
+
+## Baseline reproduction
+
+Temporary overlay test source and output are in `/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/`. The options and HTTP/disk reproductions fail for the expected missing timestamp. Every KMS HTTP request completed with 200; newer tags remained old; encrypted object metadata and subsequent ordinary plaintext GET succeeded. The result is narrower than claiming all KMS replication fails, and stronger than merely comparing options.
+
+The temporary source is not a production implementation. See [plan v1](plan-v1.md) for exact scope and required acceptance. The plan is frozen by SHA-256 before invoking real Opus.

docs/investigations/r4/verification.json

@@ -0,0 +1,358 @@
+{
+  "baseline": "9ebe81c1b3611f9cc73e676b5b741c2be62c467a",
+  "branch": "codex/r4-kms-tag-timestamp",
+  "source_sha256": {
+    "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+    "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+    "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+  },
+  "source_files_match_all_test_runs": true,
+  "checks": [
+    {
+      "name": "baseline-final",
+      "command": [
+        "go",
+        "test",
+        "-p",
+        "2",
+        "-overlay",
+        "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/baseline-final-overlay.json",
+        "./cmd",
+        "-run",
+        "^Test(PutOptsFromHeadersReplicationTimestamps|APICopyObjectReplicaTaggingTimestampUnderKMS)$",
+        "-count=1",
+        "-timeout=5m",
+        "-v"
+      ],
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2"
+      },
+      "started_at": "2026-09-15T15:50:49.430860+00:00",
+      "finished_at": "2026-09-15T15:51:20.637630+00:00",
+      "exit_code": 1,
+      "log": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/baseline-final.log",
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+        "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+      },
+      "log_sha256": "f849c2082235213764e3db7a314d52af75c4859102478a1e8f837afcaa8f1ea8",
+      "overlay_sha256": "f4cbc16e4ffccaf63191de2e8476162876055796adb4c38cda2d8c569a3bbabc",
+      "overlay_sources": {
+        "/Users/vonng/.codex/worktrees/a9cb/silo/cmd/object-api-options.go": {
+          "path": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/object-api-options.baseline.go",
+          "sha256": "16a560d0990ae929393f682f22b32ecd2e7f4d9390484b03b54e176fcd00cff5"
+        }
+      },
+      "assessment": "Expected baseline regression failure; KMS timestamp loss. Non-KMS controls pass."
+    },
+    {
+      "name": "focused",
+      "command": [
+        "go",
+        "test",
+        "-p",
+        "2",
+        "./cmd",
+        "-run",
+        "^Test(PutOptsFromHeadersReplicationTimestamps|APICopyObjectReplicaTaggingTimestampUnderKMS|ReplicationTrustControlsInternalOptionsAndEvents|GetAndValidateAttributesOpts.*|APICopyObjectReplicaRetentionRemovalUnderBucketKMS)$",
+        "-count=1",
+        "-timeout=5m",
+        "-v"
+      ],
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2"
+      },
+      "started_at": "2026-09-15T15:51:20.638633+00:00",
+      "finished_at": "2026-09-15T15:51:48.382212+00:00",
+      "exit_code": 1,
+      "log": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/focused.log",
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+        "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+      },
+      "log_sha256": "9a64cbc46e9fd6186b1d3031034b720857851ce70b1466b1da5da6d1a52b76ba",
+      "assessment": "New tests and options/trust pass; pre-existing KMS lock fixture blocked by host disk free-space percentage."
+    },
+    {
+      "name": "focused-capacity-adapted",
+      "command": [
+        "go",
+        "test",
+        "-p",
+        "2",
+        "-overlay",
+        "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/capacity-overlay.json",
+        "./cmd",
+        "-run",
+        "^Test(PutOptsFromHeadersReplicationTimestamps|APICopyObjectReplicaTaggingTimestampUnderKMS|ReplicationTrustControlsInternalOptionsAndEvents|GetAndValidateAttributesOpts.*|APICopyObjectReplicaRetentionRemovalUnderBucketKMS)$",
+        "-count=1",
+        "-timeout=5m",
+        "-v"
+      ],
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2"
+      },
+      "started_at": "2026-09-15T15:52:20.113475+00:00",
+      "finished_at": "2026-09-15T15:52:50.842773+00:00",
+      "exit_code": 0,
+      "log": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/focused-capacity-adapted.log",
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+        "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+      },
+      "log_sha256": "13f7aa54564a433bef4dddcf2c5ad1fe46fa03869527fb902a255ce4c3263bc9",
+      "overlay_sha256": "76a7e6fb364bdaaa3469b1dc79f9ea318059f287a4888f322639920c76dfe53a",
+      "overlay_sources": {
+        "/Users/vonng/.codex/worktrees/a9cb/silo/cmd/replication-trust_test.go": {
+          "path": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/replication-trust-capacity_test.go",
+          "sha256": "c81b526ae51983881bbf464199e6b90f074fa695300fa8ff005e427e4d3c8208"
+        }
+      },
+      "assessment": "PASS"
+    },
+    {
+      "name": "race",
+      "command": [
+        "go",
+        "test",
+        "-p",
+        "2",
+        "-race",
+        "./cmd",
+        "-run",
+        "^Test(PutOptsFromHeadersReplicationTimestamps|APICopyObjectReplicaTaggingTimestampUnderKMS)$",
+        "-count=1",
+        "-timeout=5m",
+        "-v"
+      ],
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2"
+      },
+      "started_at": "2026-09-15T15:52:50.843898+00:00",
+      "finished_at": "2026-09-15T15:53:43.789325+00:00",
+      "exit_code": 0,
+      "log": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/race.log",
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+        "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+      },
+      "log_sha256": "ae66a8c9e569a5c4b57ae56e75afc85c06a8b76f1567187519da0726605a4de4",
+      "assessment": "PASS"
+    },
+    {
+      "name": "vet",
+      "command": [
+        "go",
+        "vet",
+        "-p",
+        "2",
+        "./cmd"
+      ],
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2"
+      },
+      "started_at": "2026-09-15T15:53:43.790197+00:00",
+      "finished_at": "2026-09-15T15:53:51.129773+00:00",
+      "exit_code": 0,
+      "log": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/vet.log",
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+        "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+      },
+      "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+      "assessment": "PASS"
+    },
+    {
+      "name": "lint",
+      "command": [
+        "/Users/vonng/pgsty/silo/.bin/golangci/v2.13.1/golangci-lint",
+        "run",
+        "--build-tags",
+        "kqueue",
+        "--timeout=10m",
+        "--config",
+        "./.golangci.yml",
+        "./cmd/..."
+      ],
+      "cwd": "/Users/vonng/.codex/worktrees/a9cb/silo",
+      "env_override": {
+        "GOMAXPROCS": "2"
+      },
+      "started_at": "2026-09-15T15:53:51.130456+00:00",
+      "finished_at": "2026-09-15T15:55:39.114303+00:00",
+      "exit_code": 0,
+      "log": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/lint.log",
+      "source_sha256": {
+        "cmd/object-api-options.go": "25e2e9484fafd94d1b2c857b94373758e481893ad93fb1a063edf7746277accc",
+        "cmd/object-api-options-replication_test.go": "1ea2a060987e32a4c76fce96ee974df475944c2d6ab482a4893e33daf7bca849",
+        "cmd/object-copy-replication-tagging_test.go": "5437a77e68736b4ce69de9c777675251fef24b0352dfe30bd8a836fc7ee810e3"
+      },
+      "log_sha256": "e92606b0bf483111dff0a120c315ea165821348f31365020e2468a0059095c47",
+      "assessment": "PASS"
+    }
+  ],
+  "format_checks": [
+    {
+      "command": [
+        "gofmt",
+        "-l",
+        "cmd/object-api-options.go",
+        "cmd/object-api-options-replication_test.go",
+        "cmd/object-copy-replication-tagging_test.go"
+      ],
+      "exit_code": 0,
+      "output": ""
+    },
+    {
+      "command": [
+        "git",
+        "diff",
+        "--check"
+      ],
+      "exit_code": 0,
+      "output": ""
+    }
+  ],
+  "evidence_directory": "/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb",
+  "evidence_files": {
+    "baseline-final-overlay.json": {
+      "size": 177,
+      "sha256": "f4cbc16e4ffccaf63191de2e8476162876055796adb4c38cda2d8c569a3bbabc"
+    },
+    "baseline-final.json": {
+      "size": 1584,
+      "sha256": "5bf7be51e5a0eb41a40dc5fc5d3a1aba5df7733ad5dcb001f8d870a01c4233ba"
+    },
+    "baseline-final.log": {
+      "size": 22493,
+      "sha256": "f849c2082235213764e3db7a314d52af75c4859102478a1e8f837afcaa8f1ea8"
+    },
+    "baseline-identity.txt": {
+      "size": 1676,
+      "sha256": "9b21841e19a0cbb8ded18c2597488a527a27bedc65109d05d4ff598103073b68"
+    },
+    "baseline-options.log": {
+      "size": 11395,
+      "sha256": "d692f0a4bc9c58e2ac0087afa356ddf48f86e1040ea8138d68ffc4d0992bf3cb"
+    },
+    "baseline-repro.log": {
+      "size": 5693,
+      "sha256": "aa89b76f4723c6a3ce224faa7796403628d978a8707544bd97848b8887de2113"
+    },
+    "capacity-fixture.diff": {
+      "size": 870,
+      "sha256": "8d01e0b0068441f37ecee37125b81424d1f30d7c4fb37d435ea0cfe2e4617e5e"
+    },
+    "capacity-overlay.json": {
+      "size": 185,
+      "sha256": "76a7e6fb364bdaaa3469b1dc79f9ea318059f287a4888f322639920c76dfe53a"
+    },
+    "final_copy_repro_test.go": {
+      "size": 5942,
+      "sha256": "8949e07d96d2949a79f5a9e83c7a7c0473733d77b407e9c51da477d4ab74f1a8"
+    },
+    "focused-capacity-adapted.json": {
+      "size": 1661,
+      "sha256": "1a599b41caabfc5eb44db8d89c8b7e4f4f84f5f036d008369155fd337f65bdf9"
+    },
+    "focused-capacity-adapted.log": {
+      "size": 20384,
+      "sha256": "13f7aa54564a433bef4dddcf2c5ad1fe46fa03869527fb902a255ce4c3263bc9"
+    },
+    "focused.json": {
+      "size": 1253,
+      "sha256": "d9bb4979ea8aeaabb809cdc6e400a8673530bc83abf3dc2b2a06853a8523d0d9"
+    },
+    "focused.log": {
+      "size": 20475,
+      "sha256": "9a64cbc46e9fd6186b1d3031034b720857851ce70b1466b1da5da6d1a52b76ba"
+    },
+    "format-checks.json": {
+      "size": 352,
+      "sha256": "7569260900a799d5efdfb39db1f575ab1dadbbb04ace222e036968e66b6b59e7"
+    },
+    "lint.json": {
+      "size": 991,
+      "sha256": "a7144713b069f470a94b1ebe6fca6683a4b866a891a2756e15f28c280666ca14"
+    },
+    "lint.log": {
+      "size": 10,
+      "sha256": "e92606b0bf483111dff0a120c315ea165821348f31365020e2468a0059095c47"
+    },
+    "object-api-options.baseline.go": {
+      "size": 16653,
+      "sha256": "16a560d0990ae929393f682f22b32ecd2e7f4d9390484b03b54e176fcd00cff5"
+    },
+    "options-overlay.json": {
+      "size": 185,
+      "sha256": "5506b9c3b998b32f01c45af3cf01605eae9e4fb262c9ff3a6b0040abe719d4d9"
+    },
+    "options_repro_test.go": {
+      "size": 4192,
+      "sha256": "1a57a47bdd370042fa0f0d2d90efe447abedee9b9ef48a938d4bed631d83ec0b"
+    },
+    "opus-review-v1.exit": {
+      "size": 2,
+      "sha256": "9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa"
+    },
+    "opus-review-v1.jsonl": {
+      "size": 410466,
+      "sha256": "eb0918d8a6185b180dddcfc664a96682f05502ecf3b686b08a0547f09879d57d"
+    },
+    "opus-review-v1.stderr.log": {
+      "size": 0,
+      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+    },
+    "overlay.json": {
+      "size": 158,
+      "sha256": "95f7c3f7e206fe36731e6d7e4a48f90c7403f07ae8155c125c6b86d2f1c2d487"
+    },
+    "r4-kms-tag-timestamp.patch": {
+      "size": 874,
+      "sha256": "2d4806d986bbd94ba4bc3951f3aeee48401ee1921c28ded0988fa09ca76ca26f"
+    },
+    "r4_repro_test.go": {
+      "size": 5508,
+      "sha256": "9bcefb6da2416b577b58085485cad60f677c2265e9dfa84d02e465e1b203766b"
+    },
+    "race.json": {
+      "size": 1026,
+      "sha256": "50b73e4acbc2426f3dcfadde78d0f0a86f10702345d2939a30204600bc750a13"
+    },
+    "race.log": {
+      "size": 18566,
+      "sha256": "ae66a8c9e569a5c4b57ae56e75afc85c06a8b76f1567187519da0726605a4de4"
+    },
+    "replication-trust-capacity_test.go": {
+      "size": 61271,
+      "sha256": "c81b526ae51983881bbf464199e6b90f074fa695300fa8ff005e427e4d3c8208"
+    },
+    "review-prompt-v1.md": {
+      "size": 2770,
+      "sha256": "07c225beff1523e056c154b3a387cf1ae345def4b4d0173065b882140ac5abdf"
+    },
+    "run-checks.py": {
+      "size": 2266,
+      "sha256": "ddecea5220bc9c286df18c0e9eca101f3d8ab731c307cd4acef937f3ecd11e65"
+    },
+    "vet.json": {
+      "size": 853,
+      "sha256": "e0964444bc91640ed6cf78229050bad5b94af4210bdf44b11d1a848f1ee930a6"
+    },
+    "vet.log": {
+      "size": 0,
+      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+    }
+  },
+  "scope": "Darwin arm64; real signed HTTP + disk I/O; KMS service stub; single-pool single/16-disk fixtures; no remote CI, multi-site, release or deployment."
+}

docs/investigations/r4/verification.md

@@ -0,0 +1,52 @@
+# R4 修复与本地验收
+
+这是 2026-09-15 的本地验收快照。用户后续授权的实现级复核、提交规范调整与合并流程见 [合并前复核](merge-verification.md)；以下原始测试记录及哈希保留当时状态。
+
+## 结果
+
+在 `putOptsFromHeaders` 的 SSE-KMS 选项构造中补齐 `ReplicationSourceTaggingTimestamp`。目的端使用显式 SSE-KMS、桶默认 KMS 或自动加密时，可信复制 COPY 现在能消费来源标签时间戳，并在现有存储锁内完成排序。
+
+生产修改只有一个字段和相邻注释。API、存储格式、KMS key/context、信任判断和既有排序规则保持兼容。R5 的删除/空标签及 PUT/multipart 时间戳传播独立交付。
+
+## 方案与 Opus 共识
+
+- 基线：`9ebe81c1b3611f9cc73e676b5b741c2be62c467a`，已重新查询 GitHub main。
+- 分支：`codex/r4-kms-tag-timestamp`。
+- [冻结方案 v1](plan-v1.md)：SHA-256 `ad539f2071155de6955b583991684ed33c4bfe2e29660005840cdc97d7e1a754`。
+- 真实评审为本机 Claude Code 2.1.270，实际 assistant 模型 `claude-opus-5`，显式 `--effort max`。结论 **GO_WITH_NONBLOCKING_NOTES，0 个阻断项**。
+- [逐条意见处置与双方共识](consensus.md)、[原始返回评审正文](opus-v1-review.md)、[模型与哈希记录](opus-v1.metadata.json) 已保存。先保存共识，再修改生产源码。
+
+## 变更与测试
+
+| 文件 | 内容 |
+|---|---|
+| `cmd/object-api-options.go` | 在 KMS 字面量中保留已解析的来源标签时间戳。 |
+| `cmd/object-api-options-replication_test.go` | 无加密、SSE-S3、SSE-KMS、带 key/context 的 KMS、SSE-C；可信/非可信；缺失、有效、无效标签时间；纳秒、时区与空格；mtime/ETag/三个时间戳、metadata 与 SSE 序列化。 |
+| `cmd/object-copy-replication-tagging_test.go` | 两种单池后端 × 五种目的端加密模式 × 五个有序事件，共 50 次签名 COPY 和 50 次普通 GET。每步检查最终标签、精确时间戳、对象版本、加密类型及明文内容。 |
+
+COPY 使用 `metadata=REPLACE`、`tagging=REPLACE` 和可信复制身份。事件为较新更新、乱序旧更新、重复事件、再次更新，以及不带来源标签时间戳的请求。更新间隔仅 1–3 纳秒，防止时间精度退化被秒级测试掩盖。无加密与 AES256 是对照；KMS 覆盖显式、桶默认和自动加密入口。
+
+## 验证状态
+
+| 检查 | 结果 | 证据文件 |
+|---|---|---|
+| 最终测试 + 未修复基线 constructor overlay | 预期失败；只有可信 KMS 有效标签时间戳及 KMS COPY 更新失败，对照通过 | `baseline-final.log/json` |
+| 修复后最终新增测试与既有 trust/options 测试 | 通过；未使用生产源码 overlay | `focused.log` |
+| 既有 KMS Object Lock 回归 | 首次受宿主机磁盘余量阈值阻挡；仅适配测试容量报告后，与上述定向测试一起通过 | `focused-capacity-adapted.log/json`、`capacity-fixture.diff` |
+| 新增测试 `-race` | 通过 | `race.log/json` |
+| `go vet -p 2 ./cmd` | 通过 | `vet.log/json` |
+| 仓库配置的 golangci-lint，范围 `./cmd/...`、`kqueue` build tag | 通过，0 issues | `lint.log/json` |
+| gofmt、git diff --check | 通过 | `format-checks.json` |
+
+原始日志和每条命令的运行记录位于 `/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/`。每份检查 JSON 都记录命令、退出码、时间和三个源码/测试文件的 SHA-256；最终交付已逐一确认文件哈希一致。[验证清单](verification.json) 另记录 overlay 的实际替换文件哈希，避免混淆基线与修复版执行代码。
+
+测试使用真实签名 HTTP 路由、实际本地对象数据/元数据读写、服务器加解密代码；远程密钥服务由 `kms.NewStub` 代替。ErasureSD 与 16 盘 Erasure 均为单池。容量适配只使用已有 `tagTestCapacityDisk`，避免本机磁盘使用比例触发防写阈值，所有对象 I/O 仍由真实测试磁盘承担；未调整生产容量保护。
+
+## 交付与剩余边界
+
+- 本地实现和要求的定向验证均已完成，将源码、回归、研究、共识和验收记录作为一个本地提交交付。
+- R4 的独立生产补丁已提供给 R5：`/Users/vonng/tmp/silo-r4-evidence-20260915-a9cb/r4-kms-tag-timestamp.patch`，SHA-256 `2d4806d986bbd94ba4bc3951f3aeee48401ee1921c28ded0988fa09ca76ca26f`。
+- Opus 共识为方案级共识；本地测试结论来自实际运行，不把它记作 Opus 执行了测试。
+- 多池/多站点故障恢复、外部 KMS 服务、完整 Linux CI、主干合并、远端发布和部署尚未执行。
+- 没有改写存量。丢失的来源时间戳不能仅从接收端推导；后续重放/重同步须核对来源权威性及 R5 的全链路处理，不保证旧事件重放可以修复全部历史状态。
+- 另登记 KMS 字面量缺少 Proxy/Speedtest 标志的范围外观察，已交父任务单独核验，本次未扩大修复。

docs/investigations/r7/README.md

@@ -0,0 +1,74 @@
+# R7：可信复制 Content-Encoding 修复
+
+## 交付摘要
+
+生产修复只改 `cmd/handler-utils.go`：可信复制只恢复六个复制专用字段，保留调用方已规范化的普通元数据。采用 [PR #187](https://github.com/pgsty/silo/pull/187) 的生产逻辑，增加准确说明 Snowball 调用方的注释。PR 原作者：Mikhail Khadarenka；本地新增回归与调查记录由本任务提供。
+
+- `aws-chunked`：对象元数据和 GET/HEAD 不包含 Content-Encoding。
+- `aws-chunked,gzip`：只保留 `gzip`，原始 gzip 字节不变。
+- `gzip`：保持原值与原字节。
+- 六个复制字段保留，包括空 multipart 标记和 SSE-C checksum；认证/权限门控保持原语义。
+- 普通提取删除的旧 unencrypted length/MD5 用户元数据不会被复制恢复阶段重新注入。
+
+## 方案和真实 Opus 共识
+
+- [调查与基线复现](research.md)
+- [冻结方案 v1](plan-v1.md) 与 [逐项处置附录](plan-v1.dispositions.md)
+- [最终共识](consensus.md)：真实 Claude Code 2.1.270，两轮显式 `claude-opus-5 --effort max`；所有实际评审 assistant 消息均为 `claude-opus-5`。第二轮 `APPROVE`，阻断 0。
+- [首轮原文](review/opus-v1.md)、[第二轮原文](review/opus-v1-confirmation.md)；相邻 metadata 文件记录模型、命令、源 SHA、方案/原文哈希与原始 JSONL 路径。
+
+共识在产品代码修改前记录；Opus 审阅代码和方案，测试由本任务执行，二者分别留证。
+
+## 最终实现复核与合并准备
+
+用户随后追加授权：使用 Opus 5 Max 核实，确认无误后合并 main。已对提交 `4fcdf37ce656152b32ad0f615d47f5e3f9748c3a`（基线 `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`）执行新的独立实现复核，实际评审模型为 `claude-opus-5`，显式 `--effort max`。结论为 **APPROVE_WITH_NONBLOCKING_NOTES，阻塞 0**；适用 PR CI 实际通过后才可合并。
+
+[实现复核原文](implementation-review/opus-implementation.md)、[身份与哈希](implementation-review/opus-implementation.metadata.json)、[逐项处置](implementation-review/dispositions.md) 分别记录审阅与执行方验证边界。本次补充仅为文档；生产代码与测试维持已审阅、已本地验证版本。
+
+## 兼容性与边界
+
+Snowball 无 PAX 的可信复制条目不再继承外层归档的 content-type/cache-control/用户元数据，与普通 Snowball 一致。外层的六个复制专用字段仍可按既有规则作用于已授权条目。相同 tar 的普通和 replica 写入已纳入条目元数据一致性回归。
+
+现有精确 token 裁剪规则保持不变：`aws-chunked, gzip` 留下带前导空格的 ` gzip`；`gzip, aws-chunked` 中带空格的 token 仍不会被去掉。这两条记录现状的断言不代表它们已被修复。POST 表单低层元数据提取行为也保持原样。
+
+旧对象不会因升级自动修正；普通 COPY 保留来源已有元数据。若权威来源仍受污染，后续对账可能继续认为目标不一致并再次选择元数据复制。先核实来源版本、再协调副本的操作提案见 [存量处理设计](stored-metadata-remediation.md)。本任务未扫描或改写现网对象。
+
+## 复验命令
+
+在有充足空闲比例的普通测试机器上，正式测试不需要容量 overlay：
+
+```sh
+go test ./cmd -run '^Test(ExtractReplicationMetadata.*|APIReplicaContentEncoding|APISnowballReplicaContentEncoding)$' -count=1
+go test -race ./cmd -run '^Test(ExtractReplicationMetadata.*|APIReplicaContentEncoding|APISnowballReplicaContentEncoding|APISnowballReplicationTrustIsPerEntry|APISSECReplicaSkipsDestinationTransforms|APISSECMultipartReplicaRoundTripWithCompression)$' -count=1
+make verifiers
+make build
+```
+
+本机实际命令见下述每次运行的 JSON；其中包含容量 overlay、并行度和使用的本地 golangci-lint 路径。
+
+## 验证证据
+
+完整命令、源文件/方案哈希、构建身份和检查结果汇总于 [verification.json](verification.json)。构建发生在本地提交前，二进制嵌入基线提交号；代码内容以验证清单中的文件哈希为准，不作为发布制品。
+
+原始材料目录：`/Users/vonng/tmp/silo-r7-20260915-ad51/`。除原始发现阶段外，每次正式验证的 `.json` 记录命令、退出码、时间、日志哈希和四个代码文件的 SHA-256。
+
+| 验证 | 状态与材料 |
+| --- | --- |
+| 原始 helper 基线 | `baseline.log`：裸/混合可信恢复失败，gzip 控制通过 |
+| 原始 HTTP 基线 | `http-baseline-v2.log`：单盘及 16 盘，44 个控制通过，20 个已知缺陷失败 |
+| 最终测试回退原始 helper | `exact-baseline-regression.{json,log}`：测试不变，只覆盖回基线产品文件；44 控制通过、36 预期失败（20 HTTP + 16 helper） |
+| 修复后的定向测试 | `fixed-targeted.{json,log}`：9 个顶层测试、80 个具名子用例全部通过 |
+| 既有 SSE 与信任边界 | `fixed-sse-trust.{json,log}`：SSE-C 单段/多段、SSE multipart trust、PUT/COPY 投毒、普通/复制权限、Snowball per-entry、默认桶加密、streaming trailer 等全部通过 |
+| Race | `fixed-race.{json,log}`：新增 helper/HTTP/Snowball、既有 Snowball per-entry 与 SSE-C 单段/多段全部通过 |
+| 仓库 verifiers | `verifiers.{json,log}`：make verifiers 通过，golangci-lint 0 issues，生成文件与兼容标识检查通过；typos 未安装，按 Makefile 跳过 |
+| 构建 | `build.{json,log}`：make build 通过；本地 silo --version 已核对，二进制身份见 verification.json |
+
+### 本机容量条件
+
+未调整的 HTTP 夹具返回 507 / XMinioStorageFull，原始日志为 `http-baseline-unadapted.log`。宿主 APFS 接近满盘，触发相对空闲阈值。HTTP/既有 SSE/race 验证使用临时 Go overlay 复用仓库的 `tagTestCapacityDisk`，只改变 API 测试夹具看到的容量比率，实际对象和元数据仍读写测试磁盘。该临时文件在仓库外，不进入交付；生产容量策略没有变化。
+
+证据为本机认证请求处理链路及实际存储、读取和既有 SSE 往返，不是双站点调度器、进程重启、网络故障或线上验收。
+
+## 初始交付状态
+
+初始提交形成时，研究、真实 Opus 方案共识、本地实现与验证均完成，结果保存在 `codex/r7-replication-content-encoding` 分支；当时尚未推送或合并。随后按用户追加授权进行上述最终实现复核与合并准备。实际 PR、CI 与主干合并状态以对应远端记录和 `/Users/vonng/tmp/silo-r7-merge-20260916-ad51/` 中的执行回执为准。

docs/investigations/r7/consensus.md

@@ -0,0 +1,25 @@
+# R7 最终方案共识
+
+记录时间：2026-09-15T15:53:09.813317+00:00。此记录写入时产品代码仍为基线，只有调查文件和仓库外的临时测试。
+
+## 同一版方案
+
+- 基线：`9ebe81c1b3611f9cc73e676b5b741c2be62c467a`。
+- [plan-v1.md](plan-v1.md)：`7af5705ebbfb0a375956d38dba059095dc16e24558290b1bd35a6ce85b9e1f96`。
+- [plan-v1.dispositions.md](plan-v1.dispositions.md)：`d32d30f8a420f904da6ee039f0461d6281e7da44bd805d416b4970d071f0af32`。
+- Codex 重新计算并确认上述两个哈希未变。Opus 只读源码，明确未计算哈希、未运行测试。
+
+## 实际讨论结果
+
+两轮均使用 Claude Code 2.1.270，显式 `--model claude-opus-5 --effort max`。原始记录中两轮所有评审 assistant 消息均为 `claude-opus-5`；辅助模型用量与主评审模型分开记录。
+
+1. [首轮独立评审](review/opus-v1.md)：APPROVE_WITH_NONBLOCKING_NOTES，阻断 0，9 项非阻断意见。
+2. Codex 逐项核验：采纳验证/文档建议；纠正 N1 的单包权限比较方式、收窄 N5 的重试风险表述、以源码反驳 N6 的容量适配器不存在判断。详见绑定处置附录。
+3. [第二轮确认](review/opus-v1-confirmation.md)：**APPROVE，阻断 0**；Opus 明确接受 N1/N5 的纠正，撤回 N6 的事实判断，并同意这两个哈希所标识的 v1 组合直接进入实现。
+4. Codex 同意该方案及全部最终处置。没有剩余阻断分歧；共识完成，现在开始本地实现与验证。
+
+评审原始 JSONL、stderr、实际模型、命令、耗时及输出哈希均由 `review/*.metadata.json` 指向 `/Users/vonng/tmp/silo-r7-20260915-ad51/` 中的原始记录。首轮 Claude plan 模式尝试写自己的 plan 文件但 Write 工具被禁用，最后只以文本返回评审；未写产品文件。没有把失败、限流或别的模型当成通过。
+
+## 授权及证据边界
+
+共识是源代码与修复方案的认可。实现、测试、合并、发布和部署仍分别记录。本地常规修复已获工作流授权，无需再次询问；主干合并、远端发布、部署和现网存量改写不在此次范围。

docs/investigations/r7/implementation-review/dispositions.md

@@ -0,0 +1,27 @@
+# R7 最终实现复核意见处置
+
+## 审阅身份与范围
+
+用户追加指令：使用 Opus 5 Max 核实，确认无误后合并 main。该指令授权此次推送、PR 与主干合并。
+
+候选提交 `4fcdf37ce656152b32ad0f615d47f5e3f9748c3a`，基线 `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`。实际评审 assistant 消息均为 `claude-opus-5`，命令显式指定 `--effort max`；辅助 Haiku 用量与评审模型分开记录。结果为 **APPROVE_WITH_NONBLOCKING_NOTES，阻塞 0**。
+
+完整独立意见见 [Opus 原文](opus-implementation.md)，模型、源文件、diff 和原始日志哈希见 [metadata](opus-implementation.metadata.json)。原始 JSONL 保存在 `/Users/vonng/tmp/silo-r7-merge-20260916-ad51/opus-implementation.jsonl`。
+
+## 七项非阻塞意见
+
+| 意见 | 处置与证据 |
+| --- | --- |
+| 1. 必须等待本分支自己的真实 CI | 接受。合并前核验适用检查全部通过，尤其是不使用本机容量 overlay 的完整 `cmd` 测试。现有 main CI 通过不能替代候选 PR 的 CI。 |
+| 2. 自检提交对象、署名和 diff | 已核对实际提交：树 `d5cc904bf2a8b57aea1aac53999b5007522279fc`，包含 Mikhail Khadarenka 的 Co-authored-by 与提交作者匹配的 DCO Signed-off-by。相对基线仅一个生产文件、三个测试文件及调查文档变化；四个 Go 文件哈希与通过的验证日志一致。 |
+| 3. POST 表单路径仍未归一化 | 确认是原有低层调用路径，本修复不改变它。作为独立后续研究项记录；没有把本次结果宣称为所有上传方式的编码归一化。 |
+| 4. 普通请求还可断言六个 wire 字段不泄漏 | 现有 `TestExtractMetadataHeaders` 已输入全部六个 wire 字段，仅期望 `content-type`，并用 `reflect.DeepEqual` 比较完整 metadata map，任何 wire 或 internal 字段泄漏都会失败。该测试已包含在通过的 `fixed-targeted` 验证中；无需增加重复断言。新增 canonical/lowercase 矩阵进一步覆盖恢复行为。 |
+| 5. 可补充 Snowball untrusted-marker 对照 | 保留为可选增强。当前测试含同一归档 ordinary/replica 元数据一致性，以及既有 Snowball 逐条目权限回归；此次调用点与授权门控未改。 |
+| 6. GET/HEAD 使用 getPutObjectURL 命名不够直观 | 确认 URL 构造等价，不影响方法、签名或断言。无需为命名改动已经通过的测试。 |
+| 7. PR 描述必须说明存量与来源污染限制 | 接受并写入 PR 描述。旧对象不会自动修复；来源仍受污染时，后续 heal/resync/比较可能反复选择元数据复制。参见既有存量处理设计；本次没有现网扫描或改写。 |
+
+## 合并条件
+
+此次处置只增加审阅文档，生产与测试代码维持 Opus 审阅版本。推送前再次核对源文件哈希、DCO 和 main 基线；main 若前进，先检查集成增量，相关行为改变时重新验证和评审。通过正常 PR 合并流程保留 #187 作者署名，不修改贡献者分支。
+
+本记录形成时尚未发布本分支的 PR，不能作为 CI 通过或已合并的证据。实际 PR、CI 与合并回执另行核验并保存于 `/Users/vonng/tmp/silo-r7-merge-20260916-ad51/`。发布、部署、线上对象改写和双站点故障验收不在此次执行范围。

docs/investigations/r7/implementation-review/opus-implementation.md

@@ -0,0 +1,82 @@
+## R7 实现与可合并性独立复审（Opus 5，只读会话）
+
+### 1. 审阅对象
+
+| 项 | 值 |
+| --- | --- |
+| 候选提交 | `4fcdf37ce656152b32ad0f615d47f5e3f9748c3a` |
+| 候选树 | `d5cc904bf2a8b57aea1aac53999b5007522279fc` |
+| 现场 main/base | `9ebe81c1b3611f9cc73e676b5b741c2be62c467a` |
+| 审阅 diff | `/Users/vonng/tmp/silo-r7-merge-20260916-ad51/reviewed-code.diff` |
+| diff SHA-256（按调用方声明采信） | `5af32dc2cb4acc34ac91a3e48c5decfcac0d25c70bfa7d6399934c66f629df2a` |
+| 原始 PR | #187，head `b8f2fdde41dff3dc3b8db669c1d42d30ca5c1d3d`，未合并 |
+
+本会话无 shell，未运行任何测试、未计算任何哈希、未做任何网络操作，未修改任何文件。
+
+### 2. 结论
+
+**APPROVE_WITH_NONBLOCKING_NOTES。阻塞问题 0。**
+
+**这份代码在其自身 PR 的 CI 实际变绿后可以合入 main。** 合并前提是 CI 真实通过，而不是"预期会通过"（理由见 §5.1）。
+
+### 3. 我实际核验的内容（非采信）
+
+逐行对读了 diff 与工作区当前源码，确认两者一致（`cmd/handler-utils.go:247-308` 就是 diff 后的内容；三个测试文件的 grep 行号与 diff 偏移量精确对应，如 `replication-content-encoding_test.go:108` = diff:246−138，`replication-metadata_test.go:68/90` = diff:488/510−420）。
+
+**3.1 等价性（普通路径零行为变化）**
+基线 `allowReplication=false` 分支是"命中 supportedHeader → 若属复制映射则 continue"；新代码是"若属复制映射则 continue → 再查表写入"。对 `nv` 的查找、`strings.Join(value, ",")` 的拼接、user-metadata 前缀循环全部逐字保留（`handler-utils.go:262-281`）。普通提取路径字节级等价。
+
+**3.2 六个 SSE 字段与空标记**
+`supportedHeaders` 的后六项（`handler-utils.go:86-91`）与 `replicationToInternalHeaders` 的键集完全相同（`:106-114`），且六→六为单射（含 `ReplicationSsecChecksumHeader` 的恒等映射，`bucket-replication.go:81`），因此由"遍历 slice"改为"遍历 map"后迭代顺序无关，无覆盖歧义。空标记 `X-Minio-Replication-Encrypted-Multipart: ""` 走 `ok=true` 分支、`Join([""])==""`，与基线一致，且 `internal/crypto` 按键存在性消费——测试 `replication-metadata_test.go:82-86` 的 want 映射显式固定了这一点。
+
+**3.3 信任边界（PUT/COPY/MPU/Snowball）**
+diff **未触碰任何调用点与信任判定**。我复核了全部五处调用与门控：
+- PUT：`object-handlers.go:2199` 先提取 → `:2263` `evaluateReplicationTrust` → `:2278-2280` `applyReplicationTrust` → `:2281-2282` 仅 `replicaTrusted` 恢复。注意提取发生在剥离之前，因此普通提取器**必须**无条件跳过复制专用头——新代码正是无条件 `continue`（`:263-265`），这道纵深防御被完整保留。
+- COPY：`object-handlers.go:1144-1152`，仅 REPLACE 分支且 `allowReplication` 为真时恢复；COPY 指令分支语义不变。
+- MPU：`object-multipart-handlers.go:233` + `:246`，仅初始化阶段恢复；Part/Complete 不再提取元数据，测试里的 `Content-Encoding: br` 因此是未来回归护栏。
+- Snowball：`replication-trust.go:78-90` 的整包判定 + `object-handlers.go:2786-2793` 的逐条目 `ReplicateObjectAction` 复核未变；`replicationRequestHeaders`（`replication-trust.go:96-112`）仍覆盖六个 wire 头与 REPLICA 状态。
+无新增投毒面。
+
+**3.4 Snowball 普通/replica parity（最实质的行为变化，判定为修复而非回归）**
+`object-handlers.go:2802-2804` 的 `metadata` 只含 storage class。基线在 `:2836` 的恢复调用会把**外层 archive 请求**的 supportedHeaders + 用户元数据整体灌进每个条目（普通条目则完全没有），补丁后只剩六个映射；PAX 分支 `:2864-2872` 同理不再用原始 wire 值覆盖 `extractMetadata` 已归一化的 `m`。我另行确认仓库内**没有生产代码发送** `X-Amz-Meta-Snowball-Auto-Extract`（只有 `api-router.go:391` 接收），即不存在依赖旧继承行为的内部生产者。
+
+**3.5 空格 token 行为**
+`trimAwsChunkedContentEncoding`（`handler-utils.go:367-378`）按 `,` 切分后做**精确等值**比较、不做 TrimSpace。因此 `"aws-chunked, gzip" → " gzip"`（保留前导空格）、`"gzip, aws-chunked" → "gzip, aws-chunked"`（整串不变）。测试 `replication-metadata_test.go:44-51` 的两条期望与代码一致，属于**现状固化**，不得对外宣称为本次修复。
+
+**3.6 原始普通元数据的删除（GHSA 相关，正面收益）**
+`extractMetadata:219-223` 删除 `X-Amz-Meta-X-Amz-Unencrypted-Content-Length/-Md5`。基线的恢复函数会通过 `x-amz-meta-` 前缀循环把它们**重新注入**，这是对该 advisory 缓解的实际回退（仅限可信 replica 写）。补丁消除了该路径，测试在 canonical/lowercase 两种写法下都做了断言（`:100-105`）。
+
+**3.7 GET/HEAD 与原始字节**
+`ObjectInfo.ContentEncoding` 来自 `fi.Metadata["content-encoding"]`（`erasure-metadata.go:138`），`setObjectHeaders` 仅在非空时下发（`api-headers.go:129-131`）。所以测试同时断言"落盘 UserDefined 无该键"和"响应头不存在该键"是有意义且互相独立的。GET 分支比较原始字节，replica/gzip 用例写入的是真实 gzip 字节。
+
+**3.8 签名与请求体**
+`replicaEncodingStream` 的顺序正确：`newTestStreamingRequest` → 设置全部头 → `signStreamingRequest` → `assembleStreamingChunks`，chunk 签名逐块校验；非 chunked 分支 `newTestSignedRequestV4` 对 payload 计算 `x-amz-content-sha256` 并走 `authTypeSigned` 校验。测试确实经过签名验证链路，不是绕过。无权 replica 用例断言 XML `Code == AccessDenied` 并复核对象未被创建，能区分"签名失败"与"授权拒绝"。
+
+**3.9 生产代码与 PR #187 的关系**
+逐行比对 `pr187.diff` 与候选 diff：`cmd/handler-utils.go` 与 `cmd/handler-utils_test.go` **完全一致**，唯一差异是恢复函数的三行注释（pr187.diff:76-78 vs 候选:76-78），候选版补充了 Snowball 语义。"仅澄清注释"的说法属实。`commit-message.txt` 含 `Co-authored-by: Mikhail Khadarenka` 与 PR #187 归属声明。
+
+**3.10 容量夹具是否削弱证据：不削弱**
+`capacity-overlay.json` 只替换 `cmd/test-utils_test.go`；夹具唯一的功能性改动是 `ExecObjectLayerAPITest` 开头用仓库**既有**的 `tagTestCapacityDisk`（`cmd/erasure-server-pool-tags_test.go:258-264`，`DiskInfo` 返回 `Total=Free, Used=0`）包装 set 磁盘。对象字节与 xl.meta 仍写真实临时盘。未适配时的 `507/XMinioStorageFull` 原始日志保留（`http-baseline-unadapted.log`）。该文件在仓库外，**不在交付 diff 中**。
+反事实同样成立：`exact-baseline-overlay.json` 只额外把**生产文件**换成 `baseline-handler-utils.go`（我核对该文件确含 `extractMetadataFromMimeWithReplication`/`allowReplication` 布尔开关），测试文件一字未改。失败点精确落在 `replica/bare`、`replica/mixed`、全部 8 个 Snowball 用例与全部 16 个 helper 子用例，`ordinary`/`untrusted-marker`/`gzip`/`unauthorized-replica` 全通过——44 通过 / 36 失败与声明吻合。
+
+### 4. 我采信而未独立验证的部分
+
+- 四个源文件与各日志的 SHA-256、diff SHA-256、提交/树对象哈希（无 shell，无法计算）。
+- 所有测试的**执行事实**：`fixed-targeted`（80 叶子）、`fixed-sse-trust`、`fixed-race`、`make verifiers`、`make build` 的通过是读日志所得（我确认了 `fixed-targeted.log:179-180` 的 `PASS/ok`、`fixed-race.log:209-210`、`fixed-sse-trust.log:125-126`，以及四个 fixed 日志中无 `--- FAIL`），但这些进程不是本会话运行的。
+- 提交对象实际携带的 author/trailer（`commit-message.txt` 内容正确，但我无法确认它就是 `4fcdf37ce` 的提交信息）。
+- PR #187 远端当前状态（读取的是本地缓存 `pr187.json` / `pr187.diff`）。
+- 无任何两站点调度器 / 进程重启 / 网络故障 / 线上验收；本次结论只覆盖本机认证 handler 接收链路与既有 SSE 回归。
+
+### 5. 非阻塞意见
+
+1. **CI 是真正的门，不是形式。** 这两个 HTTP 级测试从未在"无容量 overlay"的环境里跑过（本机盘满，未适配时直接 507）。CI 机器盘空间正常时不需要 overlay，预期可过，但这一点**本地无证据**。请等该分支自己的 CI 变绿再合，不要用本地日志替代。
+2. **提交对象自检**（推送前，一条命令）：`git log -1 --format='%H %T %an %ae%n%b' 4fcdf37ce` 与 `git diff --stat 9ebe81c1b..4fcdf37ce`，确认树哈希、`Co-authored-by` 与"只动 2 个产品/测试文件 + 2 个新测试 + docs"。
+3. **POST 表单路径仍未归一化**：`bucket-handlers.go:1262` 直接调用低层 `extractMetadataFromMime`，不走 `extractMetadata` 的 `aws-chunked` 裁剪与默认 content-type。本补丁按 N9 刻意不动它，但这意味着 POST policy 上传仍可能落盘原始 `aws-chunked`。建议单独开 issue，不要在本 PR 顺手统一。
+4. **测试可加一条断言**：`replication-metadata_test.go:68-72` 只断言普通路径不出现六个 **internal** 键；再断言六个 **wire** 键名也不在 metadata 中，可完整封死这一面（当前实现确实不会写入，属加固）。
+5. **Snowball 可选对照**：现有 parity 只比 ordinary vs replica，可再加一条 untrusted-marker archive 对照（Opus v1 的可选建议，非必须）。
+6. **小瑕疵**：`assertReplicaEncodingObject` 用 `getPutObjectURL` 构造 GET/HEAD 目标，功能等价但读起来别扭，`getGetObjectURL` 更清楚。
+7. **存量限制需在 PR 描述里写明**：本补丁只阻止目的端再次污染，不修复既有对象；若来源端仍存错误编码，`bucket-replication.go` 的逐字符串比较会在后续 heal/resync/比较时反复选择元数据复制。措辞按 N5 收窄（"可能反复选中"，不是"不间断热循环"）。设计见 `stored-metadata-remediation.md`，本次不授权任何现网扫描或改写。
+
+### 6. 明确许可声明
+
+**本次审阅的这份代码（候选 `4fcdf37ce…`，基线 `9ebe81c1b…`，即上述 diff）在其 PR 的适用 CI 实际通过后，允许合入 main。** 不得修改原贡献者的 #187 分支；不授权发布、部署或线上对象改写。若在此期间 main 前进，需重新查看集成增量，并在相关行为改变时重跑相应测试并重新评审。

docs/investigations/r7/implementation-review/opus-implementation.metadata.json

@@ -0,0 +1,39 @@
+{
+  "baseline": "9ebe81c1b3611f9cc73e676b5b741c2be62c467a",
+  "candidate_commit": "4fcdf37ce656152b32ad0f615d47f5e3f9748c3a",
+  "candidate_tree": "d5cc904bf2a8b57aea1aac53999b5007522279fc",
+  "code_diff_sha256": "5af32dc2cb4acc34ac91a3e48c5decfcac0d25c70bfa7d6399934c66f629df2a",
+  "source_sha256": {
+    "cmd/handler-utils.go": "76d6f98a8c9b04fcf1ae7c79d5832396bbf234bd5b5efc32a7f80dec64179b1a",
+    "cmd/handler-utils_test.go": "f8a53dd29170280eb007c4e43773187710f027e2aa49aadda91d2777bd0e034e",
+    "cmd/replication-content-encoding_test.go": "c71c4cf79a3ea6336febaf14bcb8ddfd68fcbef7625ff18830c14e9143749ee7",
+    "cmd/replication-metadata_test.go": "2cd3c45687d7333a9466906a278d7953fe4eeb008e07e2c7e2d167480c8b64bb"
+  },
+  "prior_validation_log_hashes_verified": true,
+  "requested_model": "claude-opus-5",
+  "requested_effort": "max",
+  "cli_version": "2.1.270",
+  "started_at": "2026-09-15T16:03:50.465537+00:00",
+  "raw_output": "/Users/vonng/tmp/silo-r7-merge-20260916-ad51/opus-implementation.jsonl",
+  "stderr": "/Users/vonng/tmp/silo-r7-merge-20260916-ad51/opus-implementation.stderr.log",
+  "status": "completed",
+  "command": "claude --print --model claude-opus-5 --effort max --safe-mode --permission-mode plan --tools Read,Grep,Glob --strict-mcp-config --no-session-persistence --output-format stream-json --verbose --add-dir /Users/vonng/tmp/silo-r7-20260915-ad51 --add-dir /Users/vonng/tmp/silo-r7-merge-20260916-ad51",
+  "exit_code": 0,
+  "actual_review_models": [
+    "claude-opus-5"
+  ],
+  "auxiliary_usage_models": [
+    "claude-haiku-4-5-20251001"
+  ],
+  "session_id": "80cab374-7884-47ab-a83f-4cc6d4aa4303",
+  "duration_ms": 179439,
+  "num_turns": 40,
+  "verdict": "APPROVE_WITH_NONBLOCKING_NOTES",
+  "blocking_issues": 0,
+  "merge_condition": "The exact candidate must pass its own applicable PR CI. Inspect main integration changes before merge.",
+  "raw_sha256": "d77c1f1ad308640aefa035a6ebefbe0e15f376613561aa38aa8890cded320485",
+  "stderr_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+  "review_sha256": "cf0f5d4c19c0477b3742dd2de24387886cdd77123b1ca8186768319674e41573",
+  "prompt_sha256": "c95202843857c6b4d079820189473119d8911b09715d012fae4e2862c9a2f780",
+  "permission_denials": []
+}

docs/investigations/r7/implementation-review/opus-implementation.prompt.md

@@ -0,0 +1,17 @@
+Perform a new independent IMPLEMENTATION and merge-readiness review for R7, not another plan approval. User explicitly requests real "opus 5 max" verification and authorizes merging main only if correct. Do not edit files or attempt Write, do not make any network mutations or claim tests you did not run. Return review text only.
+
+Exact candidate commit: 4fcdf37ce656152b32ad0f615d47f5e3f9748c3a
+Exact live main/base fetched now: 9ebe81c1b3611f9cc73e676b5b741c2be62c467a
+Candidate git tree: d5cc904bf2a8b57aea1aac53999b5007522279fc
+Reviewed implementation diff: /Users/vonng/tmp/silo-r7-merge-20260916-ad51/reviewed-code.diff
+Diff SHA-256: 5af32dc2cb4acc34ac91a3e48c5decfcac0d25c70bfa7d6399934c66f629df2a
+Source hashes and full local test evidence: /Users/vonng/.codex/worktrees/ad51/silo/docs/investigations/r7/verification.json
+Caller just recomputed all four code hashes and previous validation log hashes: all match. No source changes since tests passed. The prior evidence includes 80 focused cases, existing SSE-C and trust/poisoning/trailer/Snowball regressions, race, make verifiers and build. Capacity adaptation is explicit, temporary, and real storage I/O still runs. Exact baseline counterfactual: same final tests, old product extractor, 44 controls pass/36 fail. Review logs are available at /Users/vonng/tmp/silo-r7-20260915-ad51/ and permitted via add-dir.
+
+Read the complete actual diff and relevant current call sites/source/tests. Prior agreed plan: docs/investigations/r7/plan-v1.md; prior dispositions and final Opus agreement: plan-v1.dispositions.md and review/opus-v1-confirmation.md. Treat them as context, not a reason to rubber-stamp implementation. Production change adopts PR #187, with only a clarified Snowball comment; candidate also adds stronger tests and documents. PR #187 head remains b8f2fdde41dff3dc3b8db669c1d42d30ca5c1d3d and unmerged; its checks were awaiting outside-contributor workflow approval, not successful CI.
+
+Assess correctness, unintended regressions, trust boundary (PUT/COPY/MPU/Snowball), six SSE metadata fields including empty marker/checksum, request-body signature handling, GET/HEAD/raw-byte expectations, whether tests actually exercise promised paths, and whether capacity fixtures or evidence fail to justify the narrow fix. Inspect especially Snowball ordinary-vs-replica parity, source polluted-metadata limitation, exact token whitespace behavior, and raw ordinary metadata redaction. Do not broaden to R4/R5/R8 or claim full distributed testing. Identify real merge blockers with file/line/evidence and smallest repair; separate optional improvements.
+
+Intended integration: publish this complete reviewed local branch to a normal pgsty/silo PR, wait for its applicable CI, and merge into main. This includes the originally credited PR #187 implementation plus local tests/docs. Do not mutate the original contributor's branch. If main advances first, inspect integration delta, revalidate necessary tests and review again if relevant behavior changed. Merely adding this review report changes documentation only. Merging main is now explicitly authorized by the user, superseding the earlier workflow's no-merge boundary; no release/deploy/live object rewrite is authorized.
+
+Return in Chinese: exact reviewed candidate/base/hash; APPROVE / APPROVE_WITH_NONBLOCKING_NOTES / REQUEST_CHANGES; each blocking finding and optional note; what you inspected vs trusted as supplied test evidence; and explicit whether this exact code may merge once CI is verified. Be concise but evidence-based. No model substitution, no fake test execution.

docs/investigations/r7/plan-v1.dispositions.md

@@ -0,0 +1,23 @@
+# R7 v1 评审意见处置与验收补充
+
+- 冻结方案仍为 `plan-v1.md`，SHA-256 `7af5705ebbfb0a375956d38dba059095dc16e24558290b1bd35a6ce85b9e1f96`。
+- Opus 首轮：`APPROVE_WITH_NONBLOCKING_NOTES`，阻断 0；原始评审见 `review/opus-v1.md`。
+- 本文件只澄清兼容边界与验收，不改变生产补丁范围；作为 v1 的绑定附录交给 Opus 再确认。确认前仍不修改产品代码。
+
+| 意见 | Codex 处置与证据 |
+| --- | --- |
+| N1：Snowball 无 PAX 行为变化与 parity | 接受。可信 replica 的无 PAX 条目不再继承外层 archive 的 ordinary content-type/content-encoding/cache-control/user metadata；这一可见变化使它与普通 Snowball 一致，纳入报告。六个复制专用字段仍能由外层传给每个已授权条目，PAX 的专用字段可按现有次序覆盖。parity 测试用相同 tar 分别执行 ordinary 与 replica 请求并比较条目元数据（排除 replica 状态/时间/ETag）；不能按建议字面在一个 REPLICA 请求中混入无 ReplicateObject 权限的条目并期待它成功，因为 `object-handlers.go:2788-2791` 会拒绝该条目。现有 per-entry 权限回归另行保持。 |
+| N2：HTTP baseline 回归护栏 | 接受，已实测。`/Users/vonng/tmp/silo-r7-20260915-ad51/http-baseline-v2.log` 包含单盘及 16 盘的真实 streaming PUT -> ObjectInfo -> GET/HEAD 失败，同期 ordinary/gzip 控制通过；还覆盖 COPY/multipart/Snowball。64 个叶子：44 控制通过，20 缺陷失败。 |
+| N3：签名前注入所有头 | 接受，已落实在临时测试 `replicaEncodingStream`：先 newTestStreamingRequest、设置所有头，再 signStreamingRequest 和 assembleStreamingChunks。拒绝测试还应断言 XML 错误码为 AccessDenied，区分签名失败。 |
+| N4：空格 token 现状 | 接受。helper 增加 `aws-chunked, gzip -> " gzip"`、`gzip, aws-chunked -> "gzip, aws-chunked"`，仅固定现有精确 token 规则，生产 normalizer 不改。后一例属于既有 token 语法限制，不能宣传为本次已修复。 |
+| N5：历史污染来源反复不一致 | 接受风险并限定措辞。`bucket-replication.go:987-997` 的逐字符串比较可让仍有错误编码的来源与修正后目的对象持续不一致；在再次 heal/resync/比较时可再次选择 metadata 复制。源码证据不单独证明一个不间断热循环。存量提案应先确认并处理权威源版本，再协调各副本；记录重复元数据复制/不一致，而不是只修目的端。自动清理历史来源不进入本次生产补丁。 |
+| N6：容量 adapter 不存在 | 不采纳此事实判断，但接受“临时调整不入交付”的要求。请直接读取基线 `cmd/erasure-server-pool-tags_test.go:258-265`：`type tagTestCapacityDisk struct{ StorageAPI }` 的 DiskInfo 返回当前 Free 作为 Total、Used=0；该文件 :131 有现有调用。此前按字符串 adapter 搜索漏掉了该类型。临时 `capacity-test-utils_test.go` 仅复用它来包装 API 夹具；原始 507 和适配日志均保留，产品容量策略不改。 |
+| N7：map 迭代等价性 | 接受。现有六个 wire key 到六个 internal key 为单射，遍历顺序无关；重复不同大小写头的 canonical map 碰撞行为沿用基线，不引入新的解析规则。 |
+| N8：被删除的旧加密用户字段 | 接受，根因和测试均包含 `X-Amz-Meta-X-Amz-Unencrypted-Content-Length/-Md5` 的再次注入。历史污染可能继续从来源传来；此次目标端普通提取删除后不会恢复这些字段。helper 对 canonical/lowercase 均断言不存在。 |
+| N9：POST 表单路径 | 接受边界说明。POST 表单直接调用低层 extractMetadataFromMime，原本就不执行 extractMetadata 的完整归一化；本补丁保持它的现状，不顺带统一逻辑。 |
+
+## 最终验收范围补充
+
+正式回归保留真实分块签名、有效 gzip 字节、GET 原始字节比较；没有两站点调度器/进程重启/网络故障验收时，就只报告本地复制接收链路和既有 SSE 测试的结论。
+
+存量修复有单独可审阅文件 `stored-metadata-remediation.md`，须按 N5 增补“权威来源优先”的顺序；没有扫描/改写现网对象的授权或动作。

docs/investigations/r7/plan-v1.md

@@ -0,0 +1,71 @@
+# R7 plan v1: preserve normalized replica object metadata
+
+## Frozen scope and source
+
+- Date: 2026-09-15. Worktree: `/Users/vonng/.codex/worktrees/ad51/silo`.
+- Local branch: `codex/r7-replication-content-encoding`.
+- Baseline: `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`, also returned by current `gh api repos/pgsty/silo/commits/main` and fetched `origin/main`.
+- PR [#187](https://github.com/pgsty/silo/pull/187): OPEN, unmerged, no reviews/checks returned; head `b8f2fdde41dff3dc3b8db669c1d42d30ca5c1d3d`. GraphQL's PR baseRefOid is `89637554d60c27cfc51d2281d0a4fe15e415f06d`; it is not the live main checked above. Snapshot and exact diff: `/Users/vonng/tmp/silo-r7-20260915-ad51/pr187.{json,diff}`.
+- Introduction: `56fa63bfd155154157cd7e1fb6dc295a3b3104ed` (2026-04-15), replication-header injection hardening. Keep its trust protections intact.
+- Governing scope: PGSTY maintained stack, minimal compatible fix. No dependency, wire-format, credential, encryption algorithm or API changes.
+
+## Root cause and observable contract
+
+`extractMetadata` calls the ordinary extractor, removes disallowed unencrypted-length/MD5 user metadata, and trims the exact `aws-chunked` transport token from `content-encoding`. The trusted-replica restoration currently calls the same broad extractor with `allowReplication=true`. That replays all supported headers and user metadata, reversing normalization and redaction.
+
+Expected mappings are `aws-chunked` -> absent Content-Encoding, `aws-chunked,gzip` -> `gzip`, and `gzip` -> `gzip`. Object bytes are not transformed by this fix. AWS documents this behavior in [SigV4 streaming](https://docs.aws.amazon.com/AmazonS3/latest/developerguide/sigv4-streaming.html). The helper reproduction is `/Users/vonng/tmp/silo-r7-20260915-ad51/baseline_test.go` with Go overlay and `baseline.log`; its failing expectations are evidence of the current defect, not implementation validation.
+
+Persistence/read path: `erasure-metadata.go` reads `fi.Metadata["content-encoding"]` into ObjectInfo.ContentEncoding; `api-headers.go` exposes it on GET/HEAD. Outbound `putReplicationOpts` and metadata-only replication copy also use the object's content encoding. Preventing raw request metadata from being replayed at ingress is sufficient for this defect and avoids read-path masking.
+
+## Input and trust boundary audit
+
+The helper does not authenticate; callers own authentication and authorization. `evaluateReplicationTrust` requires an authenticated principal, the exact single replication marker `true`, and `s3:ReplicateObject`; restoring replica-only metadata also requires `REPLICA`. Unauthorized declared replicas are rejected; marker-only/untrusted requests retain their existing sanitized behavior. Do not move restoration earlier or make headers themselves establish trust.
+
+| Actual caller | Metadata before restoration | Trust gate and intended result |
+| --- | --- | --- |
+| PutObjectHandler | extractMetadataFromReq before trust evaluation | after successful signature verification, evaluate/apply trust; only replicaTrusted restores six fields |
+| CopyObjectHandler via getCpObjMetadataFromHeader | REPLACE calls extractMetadataFromReq; COPY uses source metadata | authenticated source/destination checks; allowReplicationMetadata=replicaTrusted; REPLACE preserves normalization, COPY retains existing semantics |
+| NewMultipartUploadHandler | extractMetadataFromReq after trust/sanitization | replicaTrusted restores six fields into initiation metadata; parts and completion reuse saved metadata |
+| PutObjectExtractHandler, outer Snowball headers | only storage class and per-entry transform metadata, not generic extractMetadata | per-entry PutObject and ReplicateObject authorization; only replicaTrusted restores six fields. No-PAX entries must not inherit ordinary outer archive metadata |
+| PutObjectExtractHandler, PAX entry metadata | extractMetadata on minio.metadata.* records | reuse per-entry trust; merge normalized entry metadata plus six allowed fields. Outer ordinary archive encoding must not leak even if the PAX map omits it |
+
+PutObjectPart, CopyObjectPart and CompleteMultipartUpload do not call this helper; no additional restoration is needed there. Validate completion persistence to catch assumptions at this boundary. POST form upload does not restore replication metadata. Metadata COPY does not normalize historical source metadata; that is deliberately outside this preventive fix.
+
+## Proposed production patch
+
+Adopt the production change in PR #187, adjusted only if current-context application requires it:
+
+1. Remove the `extractMetadataFromMimeWithReplication` boolean-mode helper.
+2. Ordinary `extractMetadataFromMime` keeps header canonicalization and supported/user metadata extraction, always skips the replication-only mapping keys.
+3. `extractReplicationMetadataFromMime` keeps nil-input error behavior and canonical header lookup; loops only over `replicationToInternalHeaders` and joins multi-values exactly as before.
+4. Never re-read ordinary supported or user metadata in the restoration helper. Preserve keys already normalized, defaulted, redacted, or set by the caller.
+5. Preserve all six mappings: sealed SSE-C key, seal algorithm, IV, encrypted-multipart marker (including its empty value), actual object size, and ReplicationSsecChecksumHeader (identity mapping). Preserve canonicalized/lowercase input header compatibility.
+6. Clarify the comment to cover Snowball: ordinary metadata is owned by the caller; the common normalizing path runs before restoration, while outer archive metadata is not per-entry object metadata.
+
+No normalizer/token grammar rewrite. The current exact-token trimming semantics, malformed duplicate-cased headers, and validation of SSE field payloads are outside this bug; retain existing behavior rather than expanding accepted formats or validation rules.
+
+## Verification matrix and acceptance
+
+Use temporary overlay reproductions before consensus. Promote focused regressions only after recorded Opus agreement. Run targeted tests with bounded Go parallelism because sibling tasks share this host.
+
+1. Helper pipeline: absent encoding, bare aws-chunked, aws-chunked,gzip, gzip, gzip,aws-chunked, multi-valued encoding; ordinary vs restoration; key absence for bare encoding; legitimate gzip retained; ordinary/user metadata sentinel values and redacted unencrypted metadata not restored. Exact expected six-field map, canonical/lowercase headers, empty multipart marker, nil input handling. Repeat restoration should not change ordinary metadata.
+2. Real signed HTTP PUT -> persisted ObjectInfo -> GET and HEAD on the existing single-drive and 16-drive erasure fixtures. Use real streaming chunk signatures for transport cases and a valid gzip payload for gzip cases. Compare raw response bytes and content encoding; bare transport must have no header. Test authenticated ordinary, trusted replica, and marker without replication permission; declared replica without permission returns 403 and creates nothing.
+3. Signed COPY REPLACE and multipart initiate/part/complete -> persisted ObjectInfo -> GET/HEAD for bare, mixed, and plain gzip. Include ordinary controls. Initiation carries object metadata; part/completion carry contrasting content encoding to prove they cannot replace it. COPY preserves existing source metadata semantics.
+4. Snowball trusted entry tests with and without PAX, including PAX no Content-Encoding and PAX mixed encoding; outer aws-chunked never leaks. Existing per-entry trust test stays green.
+5. Existing SSE-C single PUT and multipart replication round trips plus replication-header poisoning regressions. Helper matrix verifies all six field mappings; actual SSE-C tests verify readable ciphertext replicas, encryption metadata, checksum and multipart layout. Preserve bucket default encryption/compression behavior. Run relevant SSE-KMS/SSE-S3 option/replica tests if available without broadening R4 scope.
+6. Targeted package tests, focused race run, build and repository verifiers. If environmental failures (e.g. disk free-space threshold) prevent existing tests from reaching the path, retain the original failure and use an explicitly documented temporary capacity adapter already used by the repository, keeping actual object I/O on test disks. Do not mislabel that as an unmodified pass.
+7. Baseline regression must fail for raw/mixed trusted metadata and fixed code must pass identical expectations. Record exact commands, exit status, baseline/diff hashes and fixture limits. No full distributed sites/deployment acceptance claim from local handler tests.
+
+## Stored-object remediation proposal (separate, no execution)
+
+Upgrade only prevents new pollution. Existing source/COPY metadata can remain wrong, and rollback reopens ingress pollution without undoing repairs. Do not rewrite production objects or private xl.meta files.
+
+A separate operator-reviewed job must inventory bucket/key/version, original Content-Encoding and complete metadata, source/replica provenance, version/ETag/size/checksum and encryption/retention settings. Identify exact aws-chunked tokens and preserve other encodings/order. Verify source bytes/encoding before deciding; gzip must not be guessed or decompressed merely from the broken label. Keep an immutable manifest and metadata backup. Test a version-preserving supported metadata operation on a local replica of the relevant setup; ordinary S3 self-COPY can create a new version/change metadata timestamps and is not a universal version-preserving repair. Resolve object-lock, SSE-C keys, concurrent changes and replication ordering before approving the concrete write plan. Apply a small approved batch with concurrency guards, re-read exact versions, verify GET/HEAD and raw bytes/checksums, then reconcile replicas. Skips/conflicts need explicit reporting and a tested rollback. This task supplies the reviewable design only.
+
+## Effort, delivery and gate
+
+Expected 0.5-1 engineer-day for patch, targeted tests and evidence on a familiar checkout; stored-object repair and release are separate work. Production patch is about 30 added/20 removed lines in one helper file; tests provide most of the new code.
+
+Before implementation: actual Claude Code `--model claude-opus-5 --effort max`, read-only tools, same frozen plan hash + baseline + PR snapshot. Record raw review, actual assistant model(s), objections and dispositions. Any model mismatch/error/rate-limit is not consensus. Resolve substantive findings and get explicit approval of the same plan version before production changes. After consensus, implement and verify without another user permission request.
+
+Deliver research, versioned plan, consensus/dispositions, minimal production diff, tests and verification summary. Local commit may package the reviewable result. No main merge, remote PR mutation, push, release, deployment, or existing-object rewrite is included.

docs/investigations/r7/research.md

@@ -0,0 +1,43 @@
+# R7 调查与复现
+
+## 结论
+
+基线 `9ebe81c1b3611f9cc73e676b5b741c2be62c467a` 中，可信复制恢复过程把已规范化的普通元数据从原始请求中重新提取。`aws-chunked` 是传输编码，重新落盘后会被 GET/HEAD 返回。SILO 在 2026-04-15 的 `56fa63bfd155154157cd7e1fb6dc295a3b3104ed` 中引入此回归；该提交修复的复制头信任边界仍需保留。
+
+实时核对 [PR #187](https://github.com/pgsty/silo/pull/187)：OPEN、未合并，head `b8f2fdde41dff3dc3b8db669c1d42d30ca5c1d3d`；只恢复复制专用字段的方向与根因吻合。没有将 PR 自报测试当成本轮验证。
+
+## 当前调用链
+
+五处调用：PUT、COPY REPLACE、NewMultipartUpload、Snowball 外层请求、Snowball PAX 条目。
+
+- PUT/COPY/multipart 使用规范化普通元数据；可信分支不应再覆盖它。
+- Snowball 外层只有 storage class/条目转换信息，没有通用提取；归档外层 Content-Type/Content-Encoding 不是条目元数据。
+- PAX 元数据经过普通提取；无 Content-Encoding 的 PAX 映射也不能留下先前泄漏的外层编码。
+- multipart 的 Part/CopyPart/Complete 不调用该恢复函数；完成后必须检验初始化元数据确实被保留。
+- 所有可信恢复均需通过认证、精确复制标记、ReplicateObject 权限和 REPLICA 状态的组合判断；Snowball 对每个条目分别鉴权。
+
+`erasure-metadata.go` 将落盘 `content-encoding` 读入 ObjectInfo；`api-headers.go` 在 GET/HEAD 返回该值。对象字节并非因此一定受损。
+
+AWS [SigV4 streaming 规范](https://docs.aws.amazon.com/AmazonS3/latest/developerguide/sigv4-streaming.html) 要求保存对象时去掉 aws-chunked，只保留实际的内容编码；只有 aws-chunked 时读取响应不应有 Content-Encoding。
+
+## 实测记录
+
+原始证据根目录：`/Users/vonng/tmp/silo-r7-20260915-ad51/`。
+
+| 记录 | 结果与边界 |
+| --- | --- |
+| `baseline_test.go` / `baseline-overlay.json` / `baseline.log` | 原始产品代码，临时 Go 测试覆盖：ordinary bare/mixed 正常，trusted bare/mixed 重新污染，纯 gzip 正常 |
+| `http-baseline-unadapted.log` | 未调整夹具的本机单盘 HTTP 上传被 507 / XMinioStorageFull 拒绝；不是 R7 结果 |
+| `http-baseline.log` | 首个容量适配 HTTP 运行；PUT/COPY/Snowball 可复现；multipart 夹具错误用 Header.Get 读取了仓库直接写入的 ETag 键，完成时 InvalidPart，不能用于 multipart 结论 |
+| `http_test.go` / `http-capacity-overlay.json` / `http-baseline-v2.log` | 修正 ETag 读取后：PUT、COPY REPLACE、multipart 的普通和 untrusted-marker 对照通过；trusted bare/mixed 在落盘和 GET/HEAD 中失败；trusted gzip 通过；无权 replica 返回 403 且不建对象 |
+| 同一最终 HTTP 基线日志的 Snowball 部分 | no-PAX、PAX 无编码、PAX 裸编码、PAX 混合编码均复现 GET/HEAD 污染 |
+
+最终 HTTP 基线总计 64 个叶子用例：44 个控制通过，20 个预期缺陷失败。每个用例均使用认证 HTTP handler + 实际对象磁盘。PUT 传输案例和 Snowball 使用真实流式分块签名，gzip 案例使用实际 gzip 字节；读取比较原始字节并检查 Content-Encoding。
+
+## 容量适配的准确含义
+
+宿主 APFS 报告约 100% 使用率，触发 SILO 的相对空闲阈值。临时 `capacity-test-utils_test.go` 使用仓库既有 `tagTestCapacityDisk` 包装 API 测试夹具，将总量/已用量报告为当前空闲量/0；实际对象写入、读取和元数据持久化仍落在测试磁盘。该 overlay 不进入产品 diff，也不调整生产容量策略。结果属于明确容量适配后的本地验证，不是未改夹具的通过。
+
+## 方案与存量
+
+冻结方案见 [plan-v1.md](plan-v1.md)。存量对象处理见 [stored-metadata-remediation.md](stored-metadata-remediation.md)，只提供设计，本任务不执行现网对象改写。

docs/investigations/r7/review/opus-v1-confirmation.md

@@ -0,0 +1,28 @@
+本轮只做回应核验，未重复独立审计；无 shell，未计算哈希、未运行任何测试。
+
+# R7 v1 第二轮确认（Opus 5）
+
+## 1. 版本与哈希
+
+- 冻结方案：`docs/investigations/r7/plan-v1.md`，任务声明 SHA-256 **`7af5705ebbfb0a375956d38dba059095dc16e24558290b1bd35a6ce85b9e1f96`**。
+- 处置附录：`docs/investigations/r7/plan-v1.dispositions.md`，任务声明 SHA-256 **`d32d30f8a420f904da6ee039f0461d6281e7da44bd805d416b4970d071f0af32`**。
+- 两个哈希均按任务声明采纳，本会话无法计算，校验仍属调用方责任。产品基线 `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`，工作区确认未改产品代码（`docs/investigations/r7/` 为 untracked）。生产补丁仍读 `pr187.diff`，作为提议代码而非我运行的证据。
+
+## 2. 三个争议点的裁定
+
+**N6：我错了，撤回。** `cmd/erasure-server-pool-tags_test.go:258-264` 确有 `type tagTestCapacityDisk struct{ StorageAPI }`，其 `DiskInfo` 把 `Total=Free, Used=0`，并已在同文件 `:131` 被 `TestReplicaWritesPreserveTagOrdering` 使用。我上轮按 “adapter” 字面 grep 漏掉了该类型，所以 plan 第 56 行 “repository already used 的临时容量适配器” 属实。撤回 N6 的事实判断，保留其被接受的部分：临时 `capacity-test-utils_test.go` 不进交付 diff，原始 507 与适配过程需记录。补充一句非阻塞：若正式回归仍需容量适配，直接复用同包内已有类型即可（不算临时旁路），产品容量策略不得改动。
+
+**N1：接受纠正。** `object-handlers.go:2740` 的 `rawReplica` 是整包级判定，`:2788-2791` 对任何缺 `ReplicateObject` 的条目直接 `ErrAccessDenied` 并中止该条目；因此我建议的“同一 REPLICA 包内 trusted 与 untrusted 条目对比”在机制上不可能成立。Codex 的做法正确：同一 tar 分别发 ordinary 与 replica 两次请求，比较条目元数据（排除 replica 状态/时间戳/ETag）。实质结论不变且已被证据坐实——补丁前 `:2835-2838` 的恢复分支会把外层 `supportedHeaders` + 用户元数据整体灌进仅含 storage class 的 `metadata`（`:2802-2804`），补丁后只剩六个映射；`http-baseline-v2.log:195-201` 的 `Erasure/no-pax` 正是外层 `aws-chunked` 泄漏。可选增补（非必须）：再加一条 untrusted-marker 归档对照。
+
+**N5：接受收窄措辞。** 源码只能支撑到：`bucket-replication.go:987-997` 的逐字符串比较 → `replicationActionForTarget:1131` → 仅在 `replicateObjectToTarget:1598` 的复制任务里求值，且成功后置 `Completed`、不自我重排队。所以“每次 heal/resync/重放对账都会再次选中 `replicateMetadata`” 成立，“不间断热循环” 我上轮说过头，撤回该措辞。补救顺序“先确认并修权威源版本、再协调副本”，以及把重复元数据复制/不一致记为已知影响，均予保留。
+
+## 3. 其余处置确认
+
+- **N2 已用实测兑现**：`http-baseline-v2.log` 是 handler 级证据，单盘 `ErasureSD` 与 16 盘 `Erasure` 均覆盖 put/copy-replace/multipart，失败精确落在 `replica/bare`（persisted/GET/HEAD 均为 `aws-chunked`，期望空）与 `replica/mixed`（`aws-chunked,gzip` vs `gzip`），ordinary、untrusted-marker、gzip、unauthorized-replica 全通过；连同 Snowball 8 例，64 叶 = 44 通过 / 20 失败，与附录计数一致。护栏诉求已满足（该日志由本会话之外产生，我只读未跑）。
+- **N3 / N7 / N8 / N9**：接受无异议。N3 额外要求断言 XML 码为 `AccessDenied` 以区分签名失败，正确。
+- **N4**：接受，且两条期望值与代码一致——`handler-utils.go:357-368` 按 `,` 精确等值比较，故 `"aws-chunked, gzip" → " gzip"`（保留前导空格）、`"gzip, aws-chunked" → "gzip, aws-chunked"`。仅作现状记录，不得宣传为本次修复。
+- **最终验收范围补充**：同意。无双站点调度器/重启/网络故障验收时只报告本地接收链路与既有 SSE 结论；存量修复只出设计文件，不授权扫描或改写现网对象。
+
+## 4. 结论
+
+**APPROVE。阻塞问题 0。** plan-v1 + 本处置附录构成的 v1 组合可直接进入实现，无需 v2；N1/N5 采用 Codex 的修正表述，N6 以我撤回告结。实现时请把 N1（无 PAX 行为变化）、N4（空格现状）、N5（收敛风险与补救顺序）落为断言或文档，并确保临时容量适配文件不出现在交付 diff 中。

docs/investigations/r7/review/opus-v1-confirmation.metadata.json

@@ -0,0 +1,36 @@
+{
+  "requested_model": "claude-opus-5",
+  "requested_effort": "max",
+  "cli_version": "2.1.270",
+  "baseline": "9ebe81c1b3611f9cc73e676b5b741c2be62c467a",
+  "plan": "plan-v1.md",
+  "plan_sha256": "7af5705ebbfb0a375956d38dba059095dc16e24558290b1bd35a6ce85b9e1f96",
+  "addendum": "plan-v1.dispositions.md",
+  "addendum_sha256": "d32d30f8a420f904da6ee039f0461d6281e7da44bd805d416b4970d071f0af32",
+  "status": "completed",
+  "started_at": "2026-09-15T15:51:25.514239+00:00",
+  "raw_output": "/Users/vonng/tmp/silo-r7-20260915-ad51/opus-v1-confirmation.jsonl",
+  "stderr": "/Users/vonng/tmp/silo-r7-20260915-ad51/opus-v1-confirmation.stderr.log",
+  "command": "claude --print --model claude-opus-5 --effort max --safe-mode --permission-mode plan --tools Read,Grep,Glob --strict-mcp-config --no-session-persistence --output-format stream-json --verbose --add-dir /Users/vonng/tmp/silo-r7-20260915-ad51",
+  "actual_assistant_models": [
+    "claude-opus-5"
+  ],
+  "subtype": "success",
+  "is_error": false,
+  "duration_ms": 57910,
+  "num_turns": 18,
+  "session_id": "9b4140a8-140f-4d94-9a59-9983346709fd",
+  "raw_sha256": "2e2db44411a0b367607b73a2f7fe38c5125f49294f496d603c8ed4975c549bba",
+  "review_sha256": "a471ca03b6e68d77d01dede8c6d1a8f989bceed9f7eeb99fc36634c025eb3541",
+  "verdict": "APPROVE",
+  "blocking_findings": 0,
+  "completed_at": "2026-09-15T15:53:09.806502+00:00",
+  "auxiliary_model_ids": [
+    "claude-haiku-4-5-20251001",
+    "claude-opus-5"
+  ],
+  "observed_tool_attempts": [
+    "Grep",
+    "Read"
+  ]
+}

docs/investigations/r7/review/opus-v1-confirmation.prompt.md

@@ -0,0 +1,7 @@
+This is round 2 of the actual R7 Opus review discussion. Product baseline remains 9ebe81c1b3611f9cc73e676b5b741c2be62c467a and NO product code has been changed. Your first actual review is /Users/vonng/.codex/worktrees/ad51/silo/docs/investigations/r7/review/opus-v1.md. It approved v1 with 9 nonblocking notes and zero blockers.
+
+Please read the SAME frozen plan /Users/vonng/.codex/worktrees/ad51/silo/docs/investigations/r7/plan-v1.md (SHA-256 7af5705ebbfb0a375956d38dba059095dc16e24558290b1bd35a6ce85b9e1f96) and Codex's numbered replies / binding acceptance addendum /Users/vonng/.codex/worktrees/ad51/silo/docs/investigations/r7/plan-v1.dispositions.md (SHA-256 d32d30f8a420f904da6ee039f0461d6281e7da44bd805d416b4970d071f0af32). The plan+addendum is the exact v1 consensus bundle; production patch remains /Users/vonng/tmp/silo-r7-20260915-ad51/pr187.diff.
+
+Focus this round on replies, not repeating the whole independent audit. Check the factual disagreement N6 by reading cmd/erasure-server-pool-tags_test.go lines 258-265 (tagTestCapacityDisk does exist; Grep for adapter missed it). Check N1's parity testing correction: a single REPLICA archive with an unauthorized entry rejects that entry, so equal ordinary vs replica archive uploads is the correct comparison. Check N5's narrower statement about repeated reconciliation opportunities rather than claiming an automatic continuous hot loop. N2 is now backed by /Users/vonng/tmp/silo-r7-20260915-ad51/http-baseline-v2.log. The other notes are accepted and will become assertions/docs.
+
+Return a concise Chinese confirmation that (a) explicitly names both hashes, (b) agrees/disagrees with each disputed point N1/N5/N6 and remaining dispositions, and (c) states APPROVE or REQUEST_CHANGES and whether there are any blockers to implementing this exact v1 bundle. Do not simulate tests. No shell is available: hash checking remains the caller's responsibility. Do not attempt Write or write a Claude plan file; tools are restricted to Read/Grep/Glob and your review is the final text.