A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
remotePeerS3Client.HealBucket sent only the bucket name and the remove
flag, and HealBucketHandler built its heal options from those two
values. During a dry-run bucket heal every remote node therefore ran a
real one: it created the bucket directory on drives that lacked it, or
removed empty bucket directories when the bucket was missing from
every pool. Only the coordinating node honored DryRun.
Add a "dry-run" key to the peer message and read it into
HealOpts.DryRun on the receiving side, so healBucketLocal returns
before MakeVol and DeleteVol on every node. The message is a string
map: an older node ignores the key, and a missing key reads as false,
so a mixed-version cluster behaves as before until every node is
upgraded.
The regression calls the real client over a test grid connection to
the real handler. It fails on main, and it fails if either side of the
change is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Retain source-ordered tombstones and parent grant boundaries across both IAM backends, cache reloads, and deliberate identity recreation. Reconcile deletions through a versioned, bounded replication protocol with restart-aware acknowledgements.
Cover inherited group grants, STS retention, same-key service recreation, absolute expiration, and failures after the durable commit. Document coordinated upgrades and the remaining consistency boundaries.
Signed-off-by: Feng Ruohang <rh@vonng.com>
The #105 T3 change (PR #156) tried to keep the resident metadata cache
monotonic by guarding peer-reload publication on lastUpdate(). But
lastUpdate() is the max of per-config timestamps and cannot order whole
records: a node caching {policy@20, CORS@10} that receives a newer CORS@15
still has lastUpdate()==20, so the guard rejects the legitimately-newer
record and the periodic refresh (same comparator) cannot repair it. A
paused reload could also resurrect deleted resident state.
Per the maintainer decision, revert the reload publication to its original
unconditional (acceptable-until-refresh) behavior:
- remove setReloaded and restore the plain Set plus notification/target
registry updates in LoadBucketMetadataHandler;
- restore refreshBucketsMetadataLoop's own lastUpdate() staleness check and
globalEventNotifier.set / globalBucketTargetSys.set publication;
- restore the unconditional GetConfig cache-miss publication;
- document the known freshness limitation at the reload site (the periodic
refresh is best-effort and cannot repair an equal-maximum-timestamp
divergence).
The T1 lifecycle merge-under-lock (UpdateExpiryLCConfig) and both T2 fixes
(DeleteBucket takes metadata.lock before deleting; saveMetadata and
loadBucketMetadataParseUnderLock recheck physical bucket existence) are
kept fully intact.
Tests:
- drop the T3 reproductions (overlapping-reload resident-cache test and the
peer-reload-preserves-current-targets publication test);
- add lockBucketMetadataAcquireHook, a nil-in-production atomic test hook in
the shared metadata.lock path, so tests can deterministically observe a
caller (notably DeleteBucket, whose lock is taken through its
erasureServerPools receiver and is invisible to an injected object layer)
reaching the lock;
- rewrite the T2 delete-race ghost test to hold metadata.lock MID-SAVE (past
saveMetadata's existence recheck) and synchronize on the delete's actual
lock attempt via the hook, so it isolates the lock-before-delete fix:
removing only DeleteBucket's metadata.lock (recheck kept) now fails it;
- rewrite the cancellation test to observe the delete's actual lock attempt,
then cancel and await its error while still holding the lock, so a
scheduling-delayed delete stopped by the canceled context can no longer
pass on a broken tree.
Refs #105. Follows #156.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Audit of the three deferred #105 follow-ups. Each reproduces with a
deterministic red test in cmd/bucket-metadata-race_test.go, and each fix is
the minimal change that turns its test green while preserving the
<bucket>.lck -> metadata.lock -> .metadata.bin lock order established by #103.
1. Lifecycle expiry merge lost update (persistent). PeerBucketLCConfigHandler
and healBucketILMExpiry read the current lifecycle with an unlocked
GetConfigFromDisk, merged the replicated expiry rules with the local
transition rules, then wrote the pre-computed blob via Update. Any lifecycle
transition change committed between the merge read and the merge write was
silently lost. New BucketMetadataSys.UpdateExpiryLCConfig performs the read,
merge, and save under one metadata.lock; mergeExpiryWithLCConfig now takes
the locked snapshot and validates object-lock retention from it instead of
re-reading (avoids a re-entrant metadata load under the lock).
2. DeleteBucket ghost .metadata.bin (persistent). DeleteBucket took only
<bucket>.lck while config writers take only metadata.lock, so a writer that
was mid-save could re-create .metadata.bin after the prefix purge. The purge
now runs under metadata.lock, with a best-effort unlocked fallback so a
delete is never blocked from completing.
3. Overlapping peer reloads publishing a stale resident cache (freshness only;
the persisted record stays correct). LoadBucketMetadataHandler and the
GetConfig cache-miss path published with an unconditional Set, so a reload
that read an older revision could overwrite a newer resident record until the
next refresh. New BucketMetadataSys.setReloaded (and a matching GetConfig
guard) refuses to regress a newer resident record, mirroring
refreshBucketsMetadataLoop.
Verification: go build -tags kqueue,dev ./...; go vet ./cmd; gofmt clean;
rebrand-guard baseline unchanged; go test -tags kqueue,dev ./cmd (207s) green;
new tests plus the #103 metadata suite green under -race.
Refs #105. Parent #102. Foundation #103.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
Two coupled changes that must land together, because the same files carry both:
the server now identifies itself as Silo, and every path that would have called
home to a MinIO-operated service is closed.
Product identity
- build-constants.go: store name, UA name and startup banner become Silo. The
Go identifiers (MinioStoreName, MinioBannerName, ...) keep their names on
purpose - renaming exported symbols would churn the compatibility surface for
a cosmetic gain, and the rebrand guard freezes that surface.
- main.go, server-startup-msg.go, ftp-server.go and the user-visible log, help
and error strings across cmd/ and internal/ switch to Silo. Original MinIO
copyright, LICENSE, NOTICE and CREDITS are untouched; --version now prints
the upstream copyright, the pgsty modification notice, and the trademark
policy's approved "based on MinIO technology" attribution.
- api-headers.go: the HTTP Server header becomes "Silo". This is the one
externally observable identity change, so TestCommonHeadersUseSiloProductName
pins it - probes that sniff for "MinIO" must move to capability detection.
- Prometheus metric HELP strings keep their MinIO wording. They are part of the
metrics contract the guard protects, not product copy.
Configuration directory
- config-dir.go: new installs use ~/.silo. If only ~/.minio exists it is still
read, with a one-time notice and no files moved. If both exist ~/.silo wins
and an ambiguity warning is emitted; an explicit --config-dir always wins.
Covered by TestSelectDefaultConfigDir. The internal .minio.sys layout is
never renamed - this rule applies to the user config directory only.
Upstream service lockdown
- globalInplaceUpdateDisabled is now true at initialization rather than being
set from MINIO_UPDATE. common-main.go still parses MINIO_UPDATE so upgrading
nodes do not fail on an unknown key, but warns that the value is ignored;
there is no way to re-enable the updater. TestInplaceUpdateCannotBeEnabled
guards that. Without this, an admin with mc could have overwritten
/usr/bin/silo with an upstream MinIO binary.
- verifyBinary and commitBinary refuse early; the ServerUpdate v1/v2 admin
routes and the peer-rest update endpoints stay registered and keep returning
the existing programmatic error, so clients see a stable failure rather than
a 404.
- MinioReleaseBaseURL and defaultMinisignPubkey are emptied: no dl.min.io
download root, and upstream's minisign key is no longer a trust root for
anything this fork ships.
- cmd/callhome.go is deleted and internal/config/subnet/ is reduced to parsing
its old keys and reporting that the integration is disabled. config-current.go
warns instead of failing when callhome or SUBNET settings are present, so an
upgraded node with those keys still starts.
- internal/config/errors.go replaces the MinIO Slack and support entry points
with Silo documentation and issue links. Error codes and programmatic fields
are unchanged.
Verified: the compatibility baseline is unchanged except for the deliberate
removal of the /api/health/upload SUBNET route; go build, go vet and the full
cmd/ and internal/ unit suites pass; a locally built binary starts, serves
S3/Admin/metrics on the unchanged /minio/* routes, answers with Server: Silo,
and falls back to a pre-existing ~/.minio with the expected notice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
this cache will be honored only when `prefix=""` while
performing ListMultipartUploads() operation.
This is mainly to satisfy applications like alluxio
for their underfs implementation and tests.
replaces https://github.com/minio/minio/pull/20181
Services are unfrozen before `initBackgroundReplication` is finished. This means that
the globalReplicationStats write is racy. Switch to an atomic pointer.
Provide the `ReplicationPool` with the stats, so it doesn't have to be grabbed
from the atomic pointer on every use.
All other loads and checks are nil, and calls return empty values when stats
still haven't been initialized.
When a drive is in a failed state when a single node multiple drives
deployment is started, a replacement of a fresh disk will not be
properly healed unless the user restarts the node.
Fix this by always adding the new fresh disk to globalLocalDrivesMap. Also
remove globalLocalDrives for simplification, a map to store local node
drives can still be used since the order of local drives of a node is
not defined.
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
It is expected that whoever is using the credentials which has
the proper set of permissions must be able to run.
`mc support perf object`
While the root login is disabled.
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
In handlers related to health diagnostics e.g. CPU, Network, Partitions,
etc, globalMinioHost was being passed as the addr, resulting in empty
value for the same in the health report.
Using globalLocalNodeName instead fixes the issue.
Metrics v3 is mainly a reorganization of metrics into smaller groups of
metrics and the removal of internal aggregation of metrics received from
peer nodes in a MinIO cluster.
This change adds the endpoint `/minio/metrics/v3` as the top-level metrics
endpoint and under this, various sub-endpoints are implemented. These
are currently documented in `docs/metrics/v3.md`
The handler will serve metrics at any path
`/minio/metrics/v3/PATH`, as follows:
when PATH is a sub-endpoint listed above => serves the group of
metrics under that path; or when PATH is a (non-empty) parent
directory of the sub-endpoints listed above => serves metrics
from each child sub-endpoint of PATH. otherwise, returns a no
resource found error
All available metrics are listed in the `docs/metrics/v3.md`. More will
be added subsequently.
globalLocalDrives seem to be not updated during the
HealFormat() leads to a requirement where the server
needs to be restarted for the healing to continue.
Depending on when the context cancelation is picked up the handler may return and close the channel before `SubscribeJSON` returns, causing:
```
Feb 05 17:12:00 s3-us-node11 minio[3973657]: panic: send on closed channel
Feb 05 17:12:00 s3-us-node11 minio[3973657]: goroutine 378007076 [running]:
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub.(*PubSub[...]).SubscribeJSON.func1()
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub/pubsub.go:139 +0x12d
Feb 05 17:12:00 s3-us-node11 minio[3973657]: created by github.com/minio/minio/internal/pubsub.(*PubSub[...]).SubscribeJSON in goroutine 378010884
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub/pubsub.go:124 +0x352
```
Wait explicitly for the goroutine to exit.
Bonus: Listen for doneCh when sending to not risk getting blocked there is channel isn't being emptied.
Allow internal types to support a `Recycler` interface, which will allow for sharing of common types across handlers.
This means that all `grid.MSS` (and similar) objects are shared across in a common pool instead of a per-handler pool.
Add internal request reuse of internal types. Add for safe (pointerless) types explicitly.
Only log params for internal types. Doing Sprint(obj) is just a bit too messy.
add new update v2 that updates per node, allows idempotent behavior
new API ensures that
- binary is correct and can be downloaded checksummed verified
- committed to actual path
- restart returns back the relevant waiting drives
- Move RenameFile to websockets
- Move ReadAll that is primarily is used
for reading 'format.json' to to websockets
- Optimize DiskInfo calls, and provide a way
to make a NoOp DiskInfo call.
Also limit the amount of concurrency when sending
binary updates to peers, avoid high network over
TX that can cause disconnection events for the
node sending updates.
New API now verifies any hung disks before restart/stop,
provides a 'per node' break down of the restart/stop results.
Provides also how many blocked syscalls are present on the
drives and what users must do about them.
Adds options to do pre-flight checks to provide information
to the user regarding any hung disks. Provides 'force' option
to forcibly attempt a restart() even with waiting syscalls
on the drives.
replace io.Discard usage to fix NUMA copy() latencies
On NUMA systems copying from 8K buffer allocated via
io.Discard leads to large latency build-up for every
```
copy(new8kbuf, largebuf)
```
can in-cur upto 1ms worth of latencies on NUMA systems
due to memory sharding across NUMA nodes.
Feng Ruohang