astelm/minio
1
0
Fork 0
mirror of https://github.com/pgsty/minio.git synced 2026-09-12 21:44:04 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(replication): gate metadata tombstone export during rolling upgrades

Browse Source 
Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
2026-09-12 11:40:04 +08:00
parent 01aaef2b50
commit 1ee64a8d89
12 changed files with 435 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/site-replication/README.md
+57 -1
View File
@@ -60,4 +60,60 @@ mc admin replicate info silo1
** Note **
Previously, site replication required the root credentials of peer sites to be identical. This is no longer necessary because STS tokens are now signed with the site replicator service account credentials, thus allowing flexibility in the independent management of root accounts across sites and the ability to disable root accounts eventually.
However, this means that STS tokens signed previously by root credentials will no longer be valid upon upgrading to the latest version with this change. Please re-generate them as you usually do. Additionally, if site replication is ever removed - the STS tokens will become invalid, regenerate them as you usually do.
However, this means that STS tokens signed previously by root credentials will no longer be valid upon upgrading to the latest version with this change. Please re-generate them as you usually do. Additionally, if site replication is ever removed - the STS tokens will become invalid, regenerate them as you usually do.
## Bucket metadata source times and deletion recovery
Policy, tags, encryption, quota, versioning and Object Lock use the originating
field timestamp. Peer apply and healing compare under the bucket metadata lock;
duplicates and older events do not rewrite the field. Real changes win over
creation-time defaults. Equal-time deletions win over live values; equal-time
live conflicts use a deterministic content key. An empty creation-time default
is never a deletion. Versioning and Object Lock cannot be deleted by an empty
replication event.
`MINIO_SITE_REPLICATION_METADATA_TOMBSTONES=off` is the startup default. After
**all nodes at every participating site** run the fixed server, have consistent
settings within each site, and old requests have drained, restart them with
`MINIO_SITE_REPLICATION_METADATA_TOMBSTONES=on`. This exports the saved deletion
times for absent tags, encryption and quota, and includes real Policy/tag/SSE/
quota deletions in initial synchronization. It lets healing recover deletions
missed during an outage. The setting does not detect remote capabilities.
With the setting off, the new timestamp ordering still applies. Ordinary delete
events still replicate, and Policy deletion times remain exported as before.
Only the newly exposed deletion information is withheld. Hidden tag/SSE/quota
tombstones can cause repeated stale heal RPCs that the fixed receiver rejects;
zero RPCs on a subsequent heal is only expected when full state is visible.
Do not enable the setting while old nodes remain: old quota heal can leave a
stale parsed quota in memory after deletion.
Before rolling back to an older server, turn the setting off on every fixed
node and restart it, then downgrade. This removes the newly exposed deletion
information; it does not repair bugs in the older software. The maintained and
release-tested target is the coordinated PGSTY stack. Compatibility with
unmodified upstream MinIO is best effort.
Legacy events without a source timestamp are still accepted and assigned a
monotonic local time, independently of this setting. In particular, older
servers sent tag heal events without `UpdatedAt`. Their arrival-time pollution,
previously polluted timestamps, and genuine differences in bucket creation
identity cannot be reconstructed automatically. Inspect the sites, resolve
bucket identity conflicts first, then resubmit the intended configuration or
delete at the authoritative site. A local write advances beyond an existing
future field timestamp. Source times before the target bucket's creation are
ignored; an unknown creation time is recovered from the physical bucket, or the
operation fails without writing.
The server emits bounded diagnostics for `legacy-zero`, `before-created` and
`indeterminate`. Keys and error messages remain stable for each bucket/field/
reason; timestamps and peer details are log attributes. Existing hourly logger
cleanup applies. Normal duplicates, older events and resolved ties are quiet.
A local PUT of a policy whose parsed statements are empty now consistently
means deletion: PUT succeeds and GET returns the existing NotFound response.
This matches the established peer-event interpretation. Zero-value quota
JSON (`{}`, `null`, or a valid zero quota document) remains a live document;
it is not silently sent as a deletion. Bulk omission preserves a field,
whereas an explicit Policy JSON `null` deletes it. These rules use the existing
wire fields and on-disk metadata format.