An ILM expiry that is refused, because the rule no longer applies when
it is checked again under the object lock or Object Lock protects the
version, or that fails, for example for lack of quorum, left almost no
trace. The bulk noncurrent purge skipped such entries without audit,
trace, log or count; the scanner traced a refusal exactly like a
successful expiry; and nothing counted either. Operators could not see
why a version they expected gone was still there.
- Count every such expiry by reason in expiry_incomplete_total
{reason="not_applicable"|"locked"|"history_unavailable"|"error"}, in
metrics v2 and v3; the counting unit is one expiry attempt, and no
bucket or object is a label.
- Trace it with result=refused or result=failed and the reason. A
failure is also audited and logged once per object; a refusal changes
nothing and is not audited, since a split key is refused on every scan.
- Never announce it: no ObjectRemoved notification for a version that is
still there.
- Name a completed expiry by what its delete did. An expiry that
addressed no version and whose result is a delete marker created that
marker, also with the null version ID of the version it replaced; any
other removed the version it addressed, also a delete marker. The
scanner and the transitioned expiry took the identity from the queued
object instead: the expiry of a latest transitioned version in a
versioned bucket announced the removal of the version it kept, and the
removal of an expired delete marker was announced as a marker created.
Tests: classification of each error; a bulk purge with a refused, a
failed and a removed entry counts and traces the first two, audits only
the failure and announces only the removal; the expiry events name the
created marker for a latest version (versioned, suspended null,
transitioned) and the removed version for a noncurrent version and an
expired delete marker; both metrics versions export the counter by
reason. The #278 side-effect tests now tell removal traces from
refusal and failure traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The migration follow-up for #265 (PR #276) made lifecycle-created
deletes replication-aware: the scanner, the expiry of a transitioned
version and the bulk noncurrent purge evaluated the delete replication
decision and queued a delete marker or a version purge for the targets.
That is the content of the still-open PR #240. It changes behavior for
every bucket whose replication rule replicates deletes or delete
markers, the mc default: a target with longer retention loses versions
on the source's schedule, and a purge keeps its space as a pending
tombstone until the target confirms. SILO, like S3, does not replicate
lifecycle deletes; an opt-in can come later. The release did not
replicate them.
Remove exactly the replication parts: the decision callbacks and the
post-delete scheduling in applyExpiryOnNonTransitionedObjects and
expireTransitionedObject, the scheduling in deleteObjectVersions, and
ilmPurgeToDelete; the scanner builds noncurrent entries as before. The
rest of #276 stays: the hot-tier delete before the remote one, the
free-version retry record, the lifecycle-change fence and the migration
completion checks. API deletes still replicate, and the replication heal
still handles persisted pending markers and purges. As in the release,
the replication heal also resyncs an empty-state marker when
existing-object replication applies to it.
The delete-all variant of the scheduling, which queued a task with an
empty decision, goes with it.
Tests: the scanner's noncurrent purge, run with replication configured,
deletes the version at once and queues nothing; an ILM marker carries
no replication state and the heal leaves it unless existing-object
replication applies, in which case it is resynced as before; an API
version delete still queues a purge. The cross-pool batch, dispatch and
decommission tests now assert no replication, and a decommission whose
expiry used to wait for the target completes in one run. Adding back the
scanner's decision makes the first two fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The unreleased native-restore fixes after RELEASE.2026-10-26T00-00-00Z
let the lock decide who owns a restore. Since fa42a0982 a single-pool
worker holds the object lock for the whole tier download, and since
7f38bf628 a busy lock means "a restore runs". 7f38bf628 also keeps the
restore's data in a marked multipart upload. Two regressions followed:
- While an old version restored, every GET, HEAD and write of the same
key waited for the download. On a single pool this is new; with
several pools the released code already held the lock that long.
- An ordinary upload whose completion failed after staging its metadata
became InvalidUploadID for every caller, and could not be aborted for
an hour. The marker that hid restore uploads could not tell them apart
once a completion had rewritten the upload's xl.meta.
A restore now runs in two phases:
- The worker erasure-codes the tier copy into the tmp volume of the
version's set, without the object lock, as heal does. A drive that
fails any part, or misses a part when CheckParts runs before the
commit, takes no part in the commit.
- A restore lock per version tells whether a restore runs. POST takes it
before the object lock and hands it to the worker. It is a liveness
hint: only the checks under the object lock decide what is written.
- The commit takes the object lock briefly, on one pool or several. It
reads the version again and checks the attempt's ownership, the
version identity, the tier copy, the part layout and the erasure
layout. It then publishes with RenameData drive by drive, never
through erasure renameData's undo, and needs the write quorum. The
replaced data directory goes only when every published drive reports
the same one.
- A publication that reached some drives but not the write quorum
converges under the same lock hold, in bounded retries after it: back
to the unrestored version on every drive that holds it, or to a
committed restore when the copy turns out to be on a write quorum.
Each pass first checks that every drive holds what the version held
before the commit, the commit wrote, or a convergence wrote, and stops
on anything else. It deletes no directory.
Multipart goes back to main: erasure-multipart.go and
bucket-object-lock.go are identical to 69fcf50cd. The commit lock, the
marker and the restore exceptions of multipart completion and of the
null-version Object Lock check are gone, and single-pool completion
again takes the upload lock first and the object lock right before it
commits. With several pools, abandoned-data cleanup now takes the pools
lock that writes take, as heal does, so it cannot remove a directory
that a commit moved in place and has not yet referenced. The restore
capability becomes protocol 2, and a queued worker checks it again.
Recovery after a coordinator crash during publication, and storage RPCs
that complete after their caller returned, stay excluded. Main has the
same exposure, and they are tracked separately.
Tests:
- New regression tests cover both problems. They fail on f85d4a5fa,
and the multipart one also passes on main.
- Fault-injection tests cover a publication below the write quorum,
including seven of twelve drives on 8+4; lost answers; convergence
retries and stops; the old data directory; a stale inline marker; a
recreated tmp volume; a published revision read across its expiry; a
second POST; a target deleted during the download; a tier 404; and
pools-locked abandoned cleanup.
- Thirteen tests of the upload marker from 7f38bf628 are dropped, and
so is TestNativeRestoreConcurrentCompletion from fa42a0982, which
completed the restore's upload from a client. Four tests are reframed
for the new semantics, including the lock-domain restore test, which
now gates the commit.
- Three self-recursive WriteMetadata test doubles are fixed, and the
helpers of the dropped tests are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Adopt silo-pkg v3.13.2, the 20260903 mcli release, and the latest validated SILO Console commit. Move maintained source imports to the pgsty/silo-pkg module path, refresh the resulting dependency closure and credits, and keep only documented legacy minio/pkg transitive residue.
Signed-off-by: Feng Ruohang <rh@vonng.com>
`go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...` executed.
`go generate ./...` ran afterwards to keep generated.
- Move VersionPurgeStatus into replication package
- ilm: Evaluate policy w/ obj retention/replication
- lifecycle: Use Evaluator to enforce ILM in scanner
- Unit tests covering ILM, replication and retention
- Simplify NewEvaluator constructor
move away from map[string]interface{} to map[string]string
to simplify the audit, and also provide concise information.
avoids large allocations under load(), reduces the amount
of audit information generated, as the current implementation
was a bit free-form. instead all datastructures must be
flattened.
Directory objects are used by applications that simulate the folder
structure of an on-disk filesystem. These are zero-byte objects with names
ending with '/'. They are only used to check whether a 'folder' exists in
the namespace.
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
When objects have more versions than their ILM policy expects to retain
via NewerNoncurrentVersions, but they don't qualify for expiry due to
NoncurrentDays are configured in that rule.
In this case, applyNewerNoncurrentVersionsLimit method was enqueuing empty
tasks, which lead to a panic (panic: runtime error: index out of range [0] with
length 0) in newerNoncurrentTask.OpHash method, which assumes the task
to contain at least one version to expire.
Remove api.expiration_workers config setting which was inadvertently left behind. Per review comment
https://github.com/minio/minio/pull/18926, expiration_workers can be configured via ilm.expiration_workers.
- Use a shared worker pool for all ILM expiry tasks
- Free version cleanup executes in a separate goroutine
- Add a free version only if removing the remote object fails
- Add ILM expiry metrics to the node namespace
- Move tier journal tasks to expiryState
- Remove unused on-disk journal for tiered objects pending deletion
- Distribute expiry tasks across workers such that the expiry of versions of
the same object serialized
- Ability to resize worker pool without server restart
- Make scaling down of expiryState workers' concurrency safe; Thanks
@klauspost
- Add error logs when expiryState and transition state are not
initialized (yet)
* metrics: Add missed tier journal entry tasks
* Initialize the ILM worker pool after the object layer
This PR fixes a bug that perhaps has been long introduced,
with no visible workarounds. In any deployment, if an entire
erasure set is deleted, there is no way the cluster recovers.
It also fixes a long-standing bug in expiring transitioned objects.
The expiration action was deleting the current version in the case'
of tiered objects instead of adding a delete marker.
minio_node_tier_ttlb_seconds - Distribution of time to last byte for streaming objects from warm tier
minio_node_tier_requests_success - Number of requests to download object from warm tier that were successful
minio_node_tier_requests_failure - Number of requests to download object from warm tier that failed
it is okay if the warm-tier cannot keep up, we should continue
to take I/O at hot-tier, only fail hot-tier or block it when
we are disk full.
Bonus: add metrics counter for these missed tasks, we will
know for sure if one of the node is lagging behind or is
losing too many tasks during transitioning.
AWS S3 closes keep-alive connections frequently
leading to frivolous logs filling up the MinIO
logs when the transition tier is an AWS S3 bucket.
Ignore such transient errors, let MinIO retry
it when it can.
Immediate transition use case and is mostly used to fill warm
backend with a lot of data when a new deployment is created
Currently, if the transition queue is complete, the transition will be
deferred to the scanner; change this behavior by blocking the PUT request
until the transition queue has a new place for a transition task.
globalDeploymentID was being read while it was being set.
Fixes race:
```
WARNING: DATA RACE
Write at 0x0000079605a0 by main goroutine:
github.com/minio/minio/cmd.connectLoadInitFormats()
github.com/minio/minio/cmd/prepare-storage.go:269 +0x14f0
github.com/minio/minio/cmd.waitForFormatErasure()
github.com/minio/minio/cmd/prepare-storage.go:294 +0x21d
...
Previous read at 0x0000079605a0 by goroutine 105:
github.com/minio/minio/cmd.newContext()
github.com/minio/minio/cmd/utils.go:817 +0x31e
github.com/minio/minio/cmd.adminMiddleware.func1()
github.com/minio/minio/cmd/admin-router.go:110 +0x96
net/http.HandlerFunc.ServeHTTP()
net/http/server.go:2136 +0x47
github.com/minio/minio/cmd.setBucketForwardingMiddleware.func1()
github.com/minio/minio/cmd/generic-handlers.go:460 +0xb1a
net/http.HandlerFunc.ServeHTTP()
net/http/server.go:2136 +0x47
...
```
* Reduce allocations
* Add stringsHasPrefixFold which can compare string prefixes, while ignoring case and not allocating.
* Reuse all msgp.Readers
* Reuse metadata buffers when not reading data.
* Make type safe. Make buffer 4K instead of 8.
* Unslice
Feng Ruohang