The implementation review of the lock fixes found test flaws, which are
corrected here; the production behavior is unchanged.
- The alias and merge tests replaced globalOperationTimeout while
background healing reads it, a data race under -race. They now bound
each request with its own context instead.
- The alias test now holds the cleaned name's write lock while the copy
runs, so it proves the names are checked before anything is locked;
it checks both objects before any later write, and covers a/./x as a
destination.
- The merge test picks aliases that hash to the clean name's set and,
where sets have their own lockers, ones that do not, and requires the
alias's own table to hold a read lock: readers admitted, writers
excluded.
- Release polling creates a fresh lock instance for every probe.
- The getNSLocks event hook and the bulk delete group limit are atomic,
so tests can set them while other goroutines read them.
Without the name check and the resource merge, both tests now fail in
every lock layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission or rebalance walk prechecks each due version of a key
with the versions of the key in all pools, to decide whether the version
moves or is left to its ILM expiry. Each precheck read and merged the
key's whole history from every pool, so a key with V due versions read
it V times in the walk alone; with long histories this grew roughly
quadratically.
Read the history once per key into a cache that the walk keeps for the
entry, and evaluate every version's precheck against it. The precheck
only chooses between moving a version and leaving it to the expiry; the
expiry reads the history again under the object lock before it deletes
anything, so a stale cache can only move a version or defer it to a
recheck. A read error is kept, so every due version of that key moves.
The expiry's own per-version reads are unchanged.
Test: decommission and rebalance of a key with 63 due noncurrent
versions read its history once, with the same outcome; without the
cache they read it 63 times.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
BatchJobExpire.Start checked the job's submitter before it looked at
the job's report, and refuse() clears Complete. A job that had completed
was therefore turned into a failed one on the next start: one that
completed before the upgrade records no submitter, and one completed
with a temporary credential that has expired since, a deleted user or a
changed password fails the binding. Such a job asked to be submitted
again, inviting an operator to run a destructive expiration twice.
Return before the submitter check when the report says the job is
complete, as replication and key rotation jobs already do.
Tests: a completed job without a submitter, and a completed job whose
temporary credential expired, keep their completed state and delete
nothing when started again; a job that did not complete is still
refused once its credential expired. The first two fail without the
change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A version can have copies in two pools, for example after an interrupted
decommission or rebalance. DeleteObject left ILM expiries out of the
reconciled delete of an addressed version, so a lifecycle version expiry
deleted only the copy that latest-key routing chose. The other copy stayed
readable; for a tiered version, expireTransitionedObject then deleted the
remote object that copy still referenced; and a migration of the pool
holding it left the expired version behind and failed its completion
check. With delete replication on, the replication completion's own
delete had reconciled the copies and hidden this.
Admit an ILM version expiry that the pools layer approved under the lock
with the versions in all pools (CheckedAllPools, and not a restored-copy
expiry) to deleteObjectReconciled. It removes the secondary copies before
the primary and stops at the first failure, so the remote object is only
deleted after every copy is gone. Each set still guards the protection
the pools approved. Marker creation, restored-copy expiry, free-version
cleanup and delete-all keep their own paths; the bulk noncurrent purge
already reconciled.
Tests: with copies in two pools, a hot and a tiered version expiry remove
both copies, and the tiered one deletes its remote object once; when the
secondary copy cannot be deleted the expiry fails, both copies and the
remote object stay; with replication configured the tiered expiry queues
nothing; decommissioning the pool that holds the second copy completes
and leaves no expired version. All fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The migration follow-up for #265 (PR #276) made lifecycle-created
deletes replication-aware: the scanner, the expiry of a transitioned
version and the bulk noncurrent purge evaluated the delete replication
decision and queued a delete marker or a version purge for the targets.
That is the content of the still-open PR #240. It changes behavior for
every bucket whose replication rule replicates deletes or delete
markers, the mc default: a target with longer retention loses versions
on the source's schedule, and a purge keeps its space as a pending
tombstone until the target confirms. SILO, like S3, does not replicate
lifecycle deletes; an opt-in can come later. The release did not
replicate them.
Remove exactly the replication parts: the decision callbacks and the
post-delete scheduling in applyExpiryOnNonTransitionedObjects and
expireTransitionedObject, the scheduling in deleteObjectVersions, and
ilmPurgeToDelete; the scanner builds noncurrent entries as before. The
rest of #276 stays: the hot-tier delete before the remote one, the
free-version retry record, the lifecycle-change fence and the migration
completion checks. API deletes still replicate, and the replication heal
still handles persisted pending markers and purges. As in the release,
the replication heal also resyncs an empty-state marker when
existing-object replication applies to it.
The delete-all variant of the scheduling, which queued a task with an
empty decision, goes with it.
Tests: the scanner's noncurrent purge, run with replication configured,
deletes the version at once and queues nothing; an ILM marker carries
no replication state and the heal leaves it unless existing-object
replication applies, in which case it is resynced as before; an API
version delete still queues a purge. The cross-pool batch, dispatch and
decommission tests now assert no replication, and a decommission whose
expiry used to wait for the target completes in one run. Adding back the
scanner's decision makes the first two fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The single-pool lock-domain change made a bulk delete lock each set's
names in that set and ran the groups one after another: a batch over S
sets cost about S lock round trips plus S set deletes, where the release
took one lock and then let erasureSets.DeleteObjects work on all sets at
once. Every distributed single pool with several sets paid for it,
including layouts where every set uses every host.
Run the groups concurrently, up to 16 at a time. Each group takes only
its own set's lock and waits for no other while it holds it, so the
groups cannot hold each other. A group checks for cancellation once it
is admitted, so a queued group of a canceled request does no storage
work. Each result is written only by the group of its object.
Tests: the first drive calls of two sets must overlap, which fails when
the groups run one after the other; with one worker, a group in flight
at a cancel keeps its own result, an earlier group keeps its success,
and a queued group makes no drive call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
getNSLocks takes the locks of a copy in one order and waited for a later
lock, such as the source, for up to two seconds while it held the earlier
ones, such as the destination; then it released them, paused up to 250 ms
and tried again until the operation timeout. When the source was write
locked for long (a multi-pool PUT for its whole upload, a heal, a
transition, another copy onto it), the copy held its destination write
lock most of the time: reads and writes of the destination waited for
seconds, their own backoff rarely hitting the gaps. In the release a copy
that waited for its source held nothing.
After a bounded wait for an atomic lock times out while earlier locks are
held, release them, wait for that lock alone with a fresh instance until
the deadline, release it once it is free, and start again from the first
lock in the same order. Holding nothing while waiting forms no cycle, and
the next attempt still takes every lock in one order with fresh
instances. A non-atomic request may hold some of its own names while it
waits, so it keeps the bounded retries.
A test hook reports the bounded timeout, the start of the idle wait and
its end, so the tests coordinate on events rather than timing: a lock
before the contended one is free for others for the whole wait, a cancel
or the deadline ends the idle wait and leaves nothing held, a non-atomic
request never waits alone, and HEAD of a copy's destination succeeds
while the copy waits for its source. All but the non-atomic control fail
without the idle wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
CopyObject locks its source and destination before it reads the source,
and lockObjects merged two requests only when their raw names were equal.
A lock is taken on the cleaned name, so an invalid source such as a//x or
a/./x and the destination a/x asked for a read lock and a write lock on
one resource: the copy waited on itself until the operation timeout and
answered 503, holding the key's lock in short windows meanwhile. Before
the lock-domain change the invalid source was refused at once, when the
source was read. This reached single nodes, multi-pool deployments and
single pools whose sets share hosts.
- Check the source with checkGetObjArgs before any lock, the first check
and the first error the release reported. Lock the destination only
when its name is valid. An invalid destination fails at the copy's own
destination check before it writes, after a missing source (404) or a
failed source precondition (412), as before.
- lockObjects merges requests by the resource the lockers see, the
cleaned bucket/object, within one lock table: the object's set where
the sets of a pool have their own lockers, else one table. This keeps
aliases on a shared local table from conflicting and never merges
requests of different distributed domains.
Tests: aliased sources and destinations are refused with 400 at once in
every lock layout and two pools; an invalid destination keeps the 404
and 412 precedence; lockObjects merges an alias with its clean name. The
alias and merge tests fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Drive healing evaluates lifecycle on the versions of one pool. For a
version it found due it enqueued the expiry and skipped the heal. The
expiry is decided again under the object lock, with the versions in all
pools and the Object Lock state it reads then, and can be refused: a
newer version in another pool, an unreadable history, a changed rule,
or a delete-all the exact-key purge refuses. Such a version was never
healed onto the replaced drive and kept reduced redundancy for as long
as the refusal lasted. The scanner heals only versions without a due
action, so it did not repair it either.
Keep the enqueue and heal the version. A version that the expiry then
removes costs one heal. Heal and expiry exclude each other through the
object lock in the writers' lock domain, which the single-pool lock
domain change provides; a version removed first reads as not found and
is skipped, and heal rebuilds only from read-quorum metadata, so it does
not bring back a version deleted on a write quorum.
Tests: a version refused across pools is healed onto the replaced drive
and survives the expiry; an unversioned object due is healed and then
removed from every drive; a versioned latest version due is healed and
hidden by a marker; a stale minority is not recreated. The first three
fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Three restore tests record the options of a POST's background restore
with a stub and restore with them later: TestRestoreAttemptGenerationFencing,
TestExpireRestoredUsesQuorumGeneration and TestRestoreHealNeedsRestoreQuorum.
The options carry the restore lock that POST hands to its worker. The
handler releases that lock only after the stub returns, while the test
proceeds as soon as the stub has signaled. When the test's restore
checked the lock before that release, it kept the lock, and the release
then canceled its context: every drive read failed with "context
canceled" and the restore reported insufficient read quorum.
Under -race this failed 11 times in 60 runs of the three tests on
69a4f938a, in the worker and old-reset modes of the fencing test and in
the multi-pool expiry test.
A new helper, restoreCaptureOptions, releases the lock before it records
the options, so a later restore takes it again, as a queued worker whose
lock ended does. TestRestoreSupersededAttempt already does this. With
the helper, the three tests passed 60 runs under -race.
Production code does not change: the handler's goroutine runs the
restore and releases the lock after it, and a lock lost during a
restore stops it as intended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A node that reaches only some drives of a set, for example right after
it restarts and before it reconnects to its peers' drives, could answer
404 NoSuchKey for an object that exists. It happened when the copies it
could see were split between two revisions and neither reached read
quorum: a restore, a transition or an unversioned overwrite committed on
a write quorum but not on every drive. S3 clients and caches treat 404
as final; the honest answer from such a view is a retryable 503.
Two paths turned that view into "not found":
- getObjectFileInfo converted InsufficientReadQuorum{RQInconsistentMeta}
to not found. Upstream added this for the case where every drive
answered, but never checked it.
- When the version merge found no version in quorum,
pickLatestQuorumFilesInfo marked every drive that answered as not
found, including drives that hold the object.
Both now require every drive to have answered: metadata, or a definite
file, version or volume not found, or corrupt metadata. Otherwise the
read keeps its read-quorum error. A key absent from every drive that
answered still gets 404, and so do inconsistent copies when every drive
answered. The dangling-object purge is unchanged.
Deletes, copies, metadata updates, decommission and rebalance that read
through such a view now fail and retry instead of treating the object
as absent. The change is local to the serving node: no storage RPC,
on-disk format or configuration changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestBatchExpirePurgeAuthorizesDriveRevisions writes a failed tag revision
to one drive of sixteen and needs the read quorum of the metadata to stay
below sixteen. Tests that run newTestConfig on a single-drive layer, such
as TestIsReqAuthenticated, leave a storage class of parity 0 behind, which
resetTestGlobals does not restore. With parity 0 the quorum history of the
key is unreadable, the purge correctly refuses, and the test failed in the
full ./cmd run while passing alone. Set the default parity for the test,
as other erasure tests do, and restore it afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A Walk that returned an error before any result cancelled the run like
an ordinary cancellation: the job was saved as neither complete nor
failed. The listing failure is now recorded first, and the run is saved
as failed with its cursor where it was. Start still returns the error,
so the job is left to a later run, which lists again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Key rotation authorized the version as the copies in all pools merge
it, but rewrote the copy that GetObjectInfo returned, routed by the
pools layer to the pool of the latest version of the key. Where the
copies differ, it rewrote metadata it had not authorized; where another
pool held another write of the same version ID, it replaced that
write's sealed key without re-encrypting its data, which no longer
decrypted. The rewrite also stored the metadata without the tags and
the expiry, which a read returns apart from it.
Rotation now authorizes both the copy it rewrites and the merged
version, and rewrites exactly that copy in its own pool. A version read
from a pool that a decommission or a rebalance is moving is left to a
later run, checked again right before the write, so that the source of
the move does not change under it. The tags and the expiry are kept.
A pool-state change during the commit itself affects every write of
the pools layer and is left to a fix there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions authorized the versions its
walk saw, and the purge then deleted the versions it read under its
lock. A version written after the walk, or tags changed on one since,
was deleted without being authorized for the submitter.
The purge takes an authorization that a batch purge must set and no
other delete may. Under the object lock, once it has approved the
history and checked every drive, and before its first delete, it
hands the writes it is about to delete to the job: every copy of every
approved version and every revision of one that a drive holds, one per
distinct version ID and tags, as they are then. The job evaluates the
expire actions on each for the submitter. A denial deletes nothing and
ends the job as refused; the time it takes counts against the purge's
budget. A locked version still refuses the whole key, counted as a
refused key. The checks on the walk's snapshot are removed.
Reviewed adversarially; the review's regressions are included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Submission and start with changed, expired or restricted credentials,
session policies with and without an authorization plugin, request
context in policy conditions, per-object denies, jobs without a
recorded submitter and describe output; expire, pull, push and
rotation over versions under retention or legal hold, on one and
sixteen drives; cursor and counter persistence; and the cases found in
review: destination versions and tags of pulls, encryption values,
S3 Tables statements, copies in several pools, claims, prefixes,
listing errors and retry accounting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch job ran with the server's object access, whoever submitted it,
and acted on versions that Object Lock protects.
Submitter: StartBatchJob binds the authenticated credential to the job
(type, access key, parent, identity revision and a keyed fingerprint).
Submission, every start and every object resolve the binding against
IAM and evaluate the action with the submitter's policies and session
policy. A credential that is gone, disabled, expired or changed fails
the job. A policy statement that would need request context the job
does not have refuses the action. With an authorization plugin, the
plugin and the session policy must both allow. A refusal is persisted
as the job's failure reason and ends the job. A job stored without a
binding fails and has to be submitted again; a definition holds one
job type.
Expire deletes each older version on its own, checking retention and
legal hold and authorizing it under the delete's lock; a locked version
is left and counted. Until the exact-key purge is integrated, a
retainVersions: 0 key is refused when a version the walk saw is locked.
Pull writes refuse to replace a locked destination version, use the
destination's versioning, validate copied retention and apply default
retention. Rotation leaves locked versions as they are.
The status reports versionsSkippedLocked, keysRefused and the failure
reason. Expire and rotation persist their cursor together with these
counts, so a resumed job counts each version once. Describe returns a
redacted copy of the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
IsAllowed returns the authorization plugin's decision when a plugin is
configured, and evaluates the local policies otherwise. Move the local
evaluation into isAllowedLocally so that a caller can combine it with
the plugin's decision. IsAllowed is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Add ObjectOptions.RefuseLockedOverwrite. With it set, PutObject (before
the rename) and CompleteMultipartUpload check, under the destination's
write lock, the stored version the write would replace: the version it
addresses, or the null version. DeleteObject checks the stored version
that a delete marker it writes names; in a multi-pool deployment that
version is resolved across pools. A version under retention or legal
hold is refused with ObjectLocked. The check reads the version's own
metadata, whatever the bucket configuration, and returns read errors.
Nothing sets the option yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Supersedes PR #279 and PR #282 and keeps their contributor commits. Native restores stage outside the object lock and commit under a short object lock (#284); ordinary multipart completion returns to main's behavior.
The unreleased native-restore fixes after RELEASE.2026-10-26T00-00-00Z
let the lock decide who owns a restore. Since fa42a0982 a single-pool
worker holds the object lock for the whole tier download, and since
7f38bf628 a busy lock means "a restore runs". 7f38bf628 also keeps the
restore's data in a marked multipart upload. Two regressions followed:
- While an old version restored, every GET, HEAD and write of the same
key waited for the download. On a single pool this is new; with
several pools the released code already held the lock that long.
- An ordinary upload whose completion failed after staging its metadata
became InvalidUploadID for every caller, and could not be aborted for
an hour. The marker that hid restore uploads could not tell them apart
once a completion had rewritten the upload's xl.meta.
A restore now runs in two phases:
- The worker erasure-codes the tier copy into the tmp volume of the
version's set, without the object lock, as heal does. A drive that
fails any part, or misses a part when CheckParts runs before the
commit, takes no part in the commit.
- A restore lock per version tells whether a restore runs. POST takes it
before the object lock and hands it to the worker. It is a liveness
hint: only the checks under the object lock decide what is written.
- The commit takes the object lock briefly, on one pool or several. It
reads the version again and checks the attempt's ownership, the
version identity, the tier copy, the part layout and the erasure
layout. It then publishes with RenameData drive by drive, never
through erasure renameData's undo, and needs the write quorum. The
replaced data directory goes only when every published drive reports
the same one.
- A publication that reached some drives but not the write quorum
converges under the same lock hold, in bounded retries after it: back
to the unrestored version on every drive that holds it, or to a
committed restore when the copy turns out to be on a write quorum.
Each pass first checks that every drive holds what the version held
before the commit, the commit wrote, or a convergence wrote, and stops
on anything else. It deletes no directory.
Multipart goes back to main: erasure-multipart.go and
bucket-object-lock.go are identical to 69fcf50cd. The commit lock, the
marker and the restore exceptions of multipart completion and of the
null-version Object Lock check are gone, and single-pool completion
again takes the upload lock first and the object lock right before it
commits. With several pools, abandoned-data cleanup now takes the pools
lock that writes take, as heal does, so it cannot remove a directory
that a commit moved in place and has not yet referenced. The restore
capability becomes protocol 2, and a queued worker checks it again.
Recovery after a coordinator crash during publication, and storage RPCs
that complete after their caller returned, stay excluded. Main has the
same exposure, and they are tracked separately.
Tests:
- New regression tests cover both problems. They fail on f85d4a5fa,
and the multipart one also passes on main.
- Fault-injection tests cover a publication below the write quorum,
including seven of twelve drives on 8+4; lost answers; convergence
retries and stops; the old data directory; a stale inline marker; a
recreated tmp volume; a published revision read across its expiry; a
second POST; a target deleted during the download; a tier 404; and
pools-locked abandoned cleanup.
- Thirteen tests of the upload marker from 7f38bf628 are dropped, and
so is TestNativeRestoreConcurrentCompletion from fa42a0982, which
completed the restore's upload from a client. Four tests are reframed
for the new semantics, including the lock-domain restore test, which
now gates the commit.
- Three self-recursive WriteMetadata test doubles are fixed, and the
helpers of the dropped tests are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The native-restore tests waited 10 seconds for steps that do real drive
work: metadata writes with synchronous persistence and renames on every
drive. A full suite run on a machine whose SSD was busy with another
decommission run failed TestRestoreUploadStagingFailsClosed/staged once
with "background restore did not finish"; about 430 reruns, under CPU and
fsync stress as well, passed.
The test uses local locks, and after its release the completion takes no
new namespace lock, so the bound is a liveness bound, not a latency
contract. All eight such waits in the native-restore tests now use one
restoreTestWait of one minute, and the background-restore helper prints
the goroutine stacks when it times out. Assertions that something must not
happen keep their own bounds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Pin the Console module to the v2.6.0 tag commit (f8ab14669). It carries
the bucket-list quota display fix and the bucket-creation quota and
retention validation (pgsty/silo-console#72, #73, PR #74), the follow-up
numeric fixes, and the usage-scan feature with its optional PostgreSQL
worker (PR #70), which brings github.com/jackc/pgx/v5 in as an indirect
dependency. golang.org/x/crypto and golang.org/x/net move with Console's
security dependency update.
The v2.6.0 release was published on 2026-11-04 from that commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Native restore chose its own single-pool lock in the object's erasure
set before the pools layer did. Now that every object write locks in
that domain through NewObjectNSLock, the restore lock is that lock.
Behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
In a single pool, DELETE in every form, bulk delete, the CopyObject
destination, SelectObjectContent, batch key rotation and the upload-ID
guards of complete, abort, get-info and list-parts locked through the
first erasure set, while PUT, multipart commit, PutObjectPart, tag,
retention, legal-hold and metadata updates, restore, transition and heal
lock in the set the object hashes to. When the sets of a pool use
different hosts, as every automatic layout of 18 or more hosts does, the
two locks ask different lockers and do not exclude each other, and
concurrent read-modify-writes of xl.meta can lose acknowledged versions
and delete markers or leave an object on a minority of drives.
Route these locks through NewObjectNSLock, which returns the lock of
the object's own set in a single pool and the pools lock otherwise.
Upload guards choose the set by their object and keep the name
object/uploadID. True prefix deletion and the locks of names that are
not objects keep their domain.
Bulk delete groups its names by set and takes one set's lock at a time,
in ascending set index; a failed group fails alone. A single node, whose
sets share one local lock table, keeps its one bulk lock.
The COPY handler takes its source and destination locks before it opens
the source, sorted, and transactionally: bounded waits while holding a
lock, release in reverse order and retry with new lock instances, one
deadline. It reads the source under them, passes NoLock to every nested
call, and runs that work in the locks' context. This also removes a
deadlock of copies in opposite directions wherever both locks were in
one domain, such as a single node or several pools.
Lock names and RPCs are unchanged, but a pool is protected only when
every node runs this change: affected pools must be upgraded and rolled
back with a drained restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions (ExpiredObjectAllVersions, DelMarkerExpiration,
and batch expiration with the default purge) asked the pools layer for a
prefix delete. The pools layer rebuilt the options, so the request ran as
a recursive delete of the key's directory in every erasure set of every
pool. Object a/b is stored inside the directory of a, so every key under
a/ was removed with it, however new, with one audit entry and one
notification for a alone. The rebuilt options also skipped the lifecycle
recheck, so on a single pool a version written after the decision was
deleted too, and tiered versions lost their metadata without free
versions.
The expiry worker, the migration expiry and batch expiration now ask for
a purge with the in-memory ObjectOptions.DeleteAllVersions:
- DeleteObject hands it to the purge under the object lock, before the
all-pools check. ILM decides with the unchanged lifecycleEventsAllPools,
on a single pool too; batch needs the version its walk matched to still
be the latest. A second read of each pool's authoritative history
records every copy, older writes of the same ID included, and must
equal the approval. Anything else deletes nothing.
- Before deleting, the purge reads every drive of each set that holds a
copy. It refuses the key when a drive cannot be read, when any revision
of an approved version there is transitioned, or, with Object Lock,
when retention or a legal hold protects one. It then deletes on exactly
the drive handles it checked.
- Versions are deleted by ID with the drives' version delete, which only
removes the object directory once it is empty: oldest first, for each
version older writes before the copy reads choose, and the latest last,
only once every older version is gone from every pool. No marker,
replication state or free version is written. The purge stops at the
first error, repeats a quorum-successful delete on the drives that
missed it while the lock is held, and stops before the latest version
at a budget of 1000 versions or one minute, reads included.
- The ILM audit entry records how many versions were removed. A purge
that failed after it began to delete is audited as partial and sends
no removal event. Batch counts come from the purge; a key whose latest
version changed is not expired, and a failed delete marker fails the
job.
The option is rejected with conflicting options and by every layer below
the pools. Force delete and internal prefix deletes keep DeletePrefix.
Delete-all of a transitioned latest version keeps its current path, and
a key with tiered older versions is refused. On a single pool the purge
relies on the lock-domain fix for writer exclusion.
Reviewed adversarially in two design rounds and four patch rounds; the
last approved with nits, which are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The xl.meta rename at the end of RenameData creates the object directory
the same way the data rename does, and its failure branch left that
directory behind for writes without a data directory, such as inline
and zero-byte objects. Remove it and its empty parents there too, with
the same trailing-slash call that PR #286 adds to the data branch for
issue #285.
Making the cleanup safe against concurrent writes needs two more
changes:
- deleteFile now walks up with a trailing slash, so each parent step
removes only an empty directory. Between two steps of a walk, another
walk can remove the rest of the chain and a writer can commit an
object whose xl.meta sits at the path the first walk removes next,
for example the object "parent" under the key "parent/xl.meta/child".
The walk used to delete that file. Ordinary deletes could already do
this; the new cleanup made it reachable with failed writes.
- The two renames of RenameData retry when the directory creation or
the rename reports "not found" while the source is still in place,
up to once per path level. A late, failed RenameData can now remove
the directory another write of the same key has just created, and
that write used to fail on the drive although its data was intact.
The retry decides on the error of the attempt itself, because another
write can recreate the directories in between. renameAll's error
mapping moves unchanged into renameAllErr so that the retry can see
the raw error.
The stagedDataGoneDisks test helper now swaps the drives behind the
set's disk list under the lock that the list takes, instead of
replacing the getDisks callback that background tasks read unlocked.
Tests:
- TestXLStorageRenameDataCommitMetaMissingRemovesEmptyDirs: zero-byte
and inline commit failures leave no empty directories, a sibling and
a moved data directory stay.
- TestXLStorageWalkUpKeepsCommittedObject: a committed object at a
parent path survives the walk, through DeleteVersion and both
cleanup branches.
- TestRenameDataSurvivesConcurrentEmptyDirCleanup: PutObject (large
and inline) and CompleteMultipartUpload succeed when a late failed
RenameData of the same key removes their fresh directory on all 16
drives.
- TestXLStorageRenameDataRetriesWhileCleanupWalksUp: a write survives
a cleanup that removes each level of its path in turn, also when a
nested-key write recreates the directories in between.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A runtime change to audit_webhook, logger_webhook or audit_kafka is
applied on every peer by serviceReloadDynamic, under a 30-second context
that is cancelled as soon as the handler returns. applyDynamicConfigForSubSys
passed that context to the logger target updates, and the targets tie
their queue worker and, with queue_dir, their disk replay to the Init
context. On every server except the one that served the admin request
the new targets therefore stopped right after they were created: records
filled the in-memory queue and were then dropped with "log buffer full"
until the server restarted.
- Build logger webhook, audit webhook and audit Kafka targets with
GlobalContext, as startup and the serving node already do. The
caller's context is still used for logging.
- Open the HTTP disk queue store before starting the queue worker, and
cancel the derived context when it cannot be opened. A worker started
before a failed Open dereferenced the nil store on the first record and
crashed the server; with the process context that would have happened
on every server.
- Let Cancel hand records still waiting in the channel to the shared
buffer of the replacement target. A target whose queue store never
opened has no worker to do this, so repairing queue_dir and applying
the configuration again used to discard the records it had accepted.
- TestDynamicWebhookOutlivesConfigContext applies each case through
applyDynamicConfigForSubSys under a short context, cancels it as the
peer handler does, and requires delivery: audit and logger webhooks
with a memory queue, a disk queue and an unopenable disk queue
(recovered by applying the configuration again), and audit Kafka with
and without queue_dir against a sarama mock broker. Each case runs in
a child process without logger and audit environment variables.
This completes earlier uncommitted maintainer work on this issue, which
covered the two HTTP call sites, the store-before-worker order and the
first version of the test.
Refs #259
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
remotePeerS3Client.HealBucket sent only the bucket name and the remove
flag, and HealBucketHandler built its heal options from those two
values. During a dry-run bucket heal every remote node therefore ran a
real one: it created the bucket directory on drives that lacked it, or
removed empty bucket directories when the bucket was missing from
every pool. Only the coordinating node honored DryRun.
Add a "dry-run" key to the peer message and read it into
HealOpts.DryRun on the receiving side, so healBucketLocal returns
before MakeVol and DeleteVol on every node. The message is a string
map: an older node ignores the key, and a missing key reads as false,
so a mixed-version cluster behaves as before until every node is
upgraded.
The regression calls the real client over a test grid connection to
the real handler. It fails on main, and it fails if either side of the
change is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
healObject handles a metadata read-quorum failure before its dry-run
return, so an admin heal with DryRun set still called deleteIfDangling.
When the drives without the version's xl.meta all answered "not found"
and outnumbered its parity blocks, the dry run deleted the version,
data directory included, from every drive that still held it. A
site-wide dry run reaches .minio.sys/config the same way.
When DryRun is set, return the quorum error with a set-level result
built from the observed read errors, before the purge. Real heals are
unchanged.
The production change is the first cmd/erasure-healing.go hunk of
PR #280, unchanged. The tests are the PR's dry-run tests that do not
depend on its parallel deep verification or its dry-run reporting
change; both stay out of this commit. The tests now assert the delete
counter first and the converted error for each quorum case, and drop
the CheckParts stub that only the excluded tests use.
Co-authored-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The merged all-pool check from #279 decided on the versions that a read
quorum of each set merged non-strictly. A drive that missed one metadata
update could therefore supply the revision used for the decision, and a
partially unreadable set could drop a committed version from the history.
Both let an expiry delete data the rule did not cover, including a version
under legal hold.
- objectVersions accepts a revision only when max(n/2+1, the version's
metadata read quorum) drives hold it, identifies revisions by the header
plus a sha256 over a canonical, length-prefixed encoding of the
normalized metadata, filters non-accepted revisions before the strict
merge, and fails closed on omitted versions, too many unreadable drives,
or cancellation.
- Lock fields are read with the evaluator's key precedence. Cross-pool
copies that disagree on hold or retention must be strictly ordered by
valid timestamps; copies with different ModTime are separate writes and
an older protected write refuses the object. The set safeguard re-checks
protection and pending purges on its fresh read for checked version
deletes.
- Duplicate copies with semantically different replication or purge state
refuse the object.
- Batch expiry announces and audits only completed or already-absent
removals.
Reviewed adversarially over seven rounds; the final round approved with no
remaining findings in scope. All-versions expiry through deletePrefix and
tiered delete-all dispatch are tracked separately.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expiry worker now asks DeleteObjects to recheck ILM against all
pools. A rejected entry keeps its version and returns
errLifecycleNotApplicable, but the worker still wrote an ILMExpiry
audit record, an ILM trace and an s3:ObjectRemoved:Delete event for it.
Event delivery drops the error element, so consumers saw an ordinary
removal, repeated on every scan while the versions stay split across
pools.
Skip rejected entries; eligible entries of the same batch are still
reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The POST handler marks a version ongoing before its background restore
takes the object lock, and only the restore clears that marker. Native
restore holds the lock through the tier download, so a restore of
another version of the same key waits behind it. Both restore entry
points gave up after one delete-operation timeout and returned before
any cleanup, leaving ongoing-request="true" with no owner. Every later
POST for that version was refused with 409 and nothing expired the
marker.
Retry the lock while the restore's context lives, with a dedicated
dynamic timeout so repeated waits do not lengthen delete timeouts.
Shutdown or a caller deadline still ends the wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep the specialized scanner routing test aligned with the explicit lifecycle rejection result, and verify the addressed version and payload remain readable. This distinguishes refused expiry from an idempotent missing version.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang