The unreleased native-restore fixes after RELEASE.2026-10-26T00-00-00Z
let the lock decide who owns a restore. Since fa42a0982 a single-pool
worker holds the object lock for the whole tier download, and since
7f38bf628 a busy lock means "a restore runs". 7f38bf628 also keeps the
restore's data in a marked multipart upload. Two regressions followed:
- While an old version restored, every GET, HEAD and write of the same
key waited for the download. On a single pool this is new; with
several pools the released code already held the lock that long.
- An ordinary upload whose completion failed after staging its metadata
became InvalidUploadID for every caller, and could not be aborted for
an hour. The marker that hid restore uploads could not tell them apart
once a completion had rewritten the upload's xl.meta.
A restore now runs in two phases:
- The worker erasure-codes the tier copy into the tmp volume of the
version's set, without the object lock, as heal does. A drive that
fails any part, or misses a part when CheckParts runs before the
commit, takes no part in the commit.
- A restore lock per version tells whether a restore runs. POST takes it
before the object lock and hands it to the worker. It is a liveness
hint: only the checks under the object lock decide what is written.
- The commit takes the object lock briefly, on one pool or several. It
reads the version again and checks the attempt's ownership, the
version identity, the tier copy, the part layout and the erasure
layout. It then publishes with RenameData drive by drive, never
through erasure renameData's undo, and needs the write quorum. The
replaced data directory goes only when every published drive reports
the same one.
- A publication that reached some drives but not the write quorum
converges under the same lock hold, in bounded retries after it: back
to the unrestored version on every drive that holds it, or to a
committed restore when the copy turns out to be on a write quorum.
Each pass first checks that every drive holds what the version held
before the commit, the commit wrote, or a convergence wrote, and stops
on anything else. It deletes no directory.
Multipart goes back to main: erasure-multipart.go and
bucket-object-lock.go are identical to 69fcf50cd. The commit lock, the
marker and the restore exceptions of multipart completion and of the
null-version Object Lock check are gone, and single-pool completion
again takes the upload lock first and the object lock right before it
commits. With several pools, abandoned-data cleanup now takes the pools
lock that writes take, as heal does, so it cannot remove a directory
that a commit moved in place and has not yet referenced. The restore
capability becomes protocol 2, and a queued worker checks it again.
Recovery after a coordinator crash during publication, and storage RPCs
that complete after their caller returned, stay excluded. Main has the
same exposure, and they are tracked separately.
Tests:
- New regression tests cover both problems. They fail on f85d4a5fa,
and the multipart one also passes on main.
- Fault-injection tests cover a publication below the write quorum,
including seven of twelve drives on 8+4; lost answers; convergence
retries and stops; the old data directory; a stale inline marker; a
recreated tmp volume; a published revision read across its expiry; a
second POST; a target deleted during the download; a tier 404; and
pools-locked abandoned cleanup.
- Thirteen tests of the upload marker from 7f38bf628 are dropped, and
so is TestNativeRestoreConcurrentCompletion from fa42a0982, which
completed the restore's upload from a client. Four tests are reframed
for the new semantics, including the lock-domain restore test, which
now gates the commit.
- Three self-recursive WriteMetadata test doubles are fixed, and the
helpers of the dropped tests are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Adopt silo-pkg v3.13.2, the 20260903 mcli release, and the latest validated SILO Console commit. Move maintained source imports to the pgsty/silo-pkg module path, refresh the resulting dependency closure and credits, and keep only documented legacy minio/pkg transitive residue.
Signed-off-by: Feng Ruohang <rh@vonng.com>
`go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...` executed.
`go generate ./...` ran afterwards to keep generated.
- Move VersionPurgeStatus into replication package
- ilm: Evaluate policy w/ obj retention/replication
- lifecycle: Use Evaluator to enforce ILM in scanner
- Unit tests covering ILM, replication and retention
- Simplify NewEvaluator constructor
move away from map[string]interface{} to map[string]string
to simplify the audit, and also provide concise information.
avoids large allocations under load(), reduces the amount
of audit information generated, as the current implementation
was a bit free-form. instead all datastructures must be
flattened.
Directory objects are used by applications that simulate the folder
structure of an on-disk filesystem. These are zero-byte objects with names
ending with '/'. They are only used to check whether a 'folder' exists in
the namespace.
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
When objects have more versions than their ILM policy expects to retain
via NewerNoncurrentVersions, but they don't qualify for expiry due to
NoncurrentDays are configured in that rule.
In this case, applyNewerNoncurrentVersionsLimit method was enqueuing empty
tasks, which lead to a panic (panic: runtime error: index out of range [0] with
length 0) in newerNoncurrentTask.OpHash method, which assumes the task
to contain at least one version to expire.
Remove api.expiration_workers config setting which was inadvertently left behind. Per review comment
https://github.com/minio/minio/pull/18926, expiration_workers can be configured via ilm.expiration_workers.
- Use a shared worker pool for all ILM expiry tasks
- Free version cleanup executes in a separate goroutine
- Add a free version only if removing the remote object fails
- Add ILM expiry metrics to the node namespace
- Move tier journal tasks to expiryState
- Remove unused on-disk journal for tiered objects pending deletion
- Distribute expiry tasks across workers such that the expiry of versions of
the same object serialized
- Ability to resize worker pool without server restart
- Make scaling down of expiryState workers' concurrency safe; Thanks
@klauspost
- Add error logs when expiryState and transition state are not
initialized (yet)
* metrics: Add missed tier journal entry tasks
* Initialize the ILM worker pool after the object layer
This PR fixes a bug that perhaps has been long introduced,
with no visible workarounds. In any deployment, if an entire
erasure set is deleted, there is no way the cluster recovers.
It also fixes a long-standing bug in expiring transitioned objects.
The expiration action was deleting the current version in the case'
of tiered objects instead of adding a delete marker.
minio_node_tier_ttlb_seconds - Distribution of time to last byte for streaming objects from warm tier
minio_node_tier_requests_success - Number of requests to download object from warm tier that were successful
minio_node_tier_requests_failure - Number of requests to download object from warm tier that failed
it is okay if the warm-tier cannot keep up, we should continue
to take I/O at hot-tier, only fail hot-tier or block it when
we are disk full.
Bonus: add metrics counter for these missed tasks, we will
know for sure if one of the node is lagging behind or is
losing too many tasks during transitioning.
AWS S3 closes keep-alive connections frequently
leading to frivolous logs filling up the MinIO
logs when the transition tier is an AWS S3 bucket.
Ignore such transient errors, let MinIO retry
it when it can.
Immediate transition use case and is mostly used to fill warm
backend with a lot of data when a new deployment is created
Currently, if the transition queue is complete, the transition will be
deferred to the scanner; change this behavior by blocking the PUT request
until the transition queue has a new place for a transition task.
globalDeploymentID was being read while it was being set.
Fixes race:
```
WARNING: DATA RACE
Write at 0x0000079605a0 by main goroutine:
github.com/minio/minio/cmd.connectLoadInitFormats()
github.com/minio/minio/cmd/prepare-storage.go:269 +0x14f0
github.com/minio/minio/cmd.waitForFormatErasure()
github.com/minio/minio/cmd/prepare-storage.go:294 +0x21d
...
Previous read at 0x0000079605a0 by goroutine 105:
github.com/minio/minio/cmd.newContext()
github.com/minio/minio/cmd/utils.go:817 +0x31e
github.com/minio/minio/cmd.adminMiddleware.func1()
github.com/minio/minio/cmd/admin-router.go:110 +0x96
net/http.HandlerFunc.ServeHTTP()
net/http/server.go:2136 +0x47
github.com/minio/minio/cmd.setBucketForwardingMiddleware.func1()
github.com/minio/minio/cmd/generic-handlers.go:460 +0xb1a
net/http.HandlerFunc.ServeHTTP()
net/http/server.go:2136 +0x47
...
```
* Reduce allocations
* Add stringsHasPrefixFold which can compare string prefixes, while ignoring case and not allocating.
* Reuse all msgp.Readers
* Reuse metadata buffers when not reading data.
* Make type safe. Make buffer 4K instead of 8.
* Unslice
Feng Ruohang