The native restore gate refused a POST restore and a queued restore
with one message, "unavailable during a rolling upgrade: not every
server of the cluster supports it yet", whether a server answered
another protocol or was simply not connected. A cluster with one server
down thus reported a rolling upgrade, and the operator looked for an
older binary instead of the server.
The gate now returns an error that wraps errNativeRestoreUnavailable
and says why: errNativeRestorePeerUnreachable names the server that is
not connected ("server host:port is not reachable, so it cannot be
shown to have stopped an older restore"), errNativeRestoreUnsupported
keeps the rolling-upgrade text for a server that reports another
protocol or reconnected while it was asked, and a check that its
deadline ended carries that error. The POST handler maps the error to
503 ServiceUnavailable with the message as before, the worker still
clears its marker and stops, and errors.Is(err,
errNativeRestoreUnavailable) holds for every refusal. The gate remains
closed in every case it was closed before.
TestNativeRestoreRefusalReasons posts a restore with a fake server that
is not connected and one that answers protocol 1; before this change
the first answered 503 with the rolling-upgrade text and without "not
reachable".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission walk removes a restored copy that is past its restore
deadline before it moves the version, as the scanner would. Since
0a546c7a7 a refused removal made filterLifecycle return the
DeleteRestored action: the walk counted the version as decommissioned
without moving it and without queuing an expiry, removeMovedVersions
then found a source copy with no retained copy and failed the key with
PreConditionFailed, the moved versions of the key stayed duplicated in
both pools, the bucket stayed queued and the decommission ended Failed;
the retry after a restart failed the same way while the refusal held.
The walk now moves the version like any other when the removal is
refused, as the release before 0a546c7a7 did: a copy past its deadline
reads as remote, so the version moves with its tier metadata and its
restore deadline through DecomTieredObject, the remote object stays,
the source copy retires with the key once the destination holds it, and
the scanner of the destination pool removes the restored copy by that
deadline. The rebalance walk never reaches this action for such a
version, which it skips as remote; its filterLifecycle now names the
same decision so neither walk counts a restored-copy expiry as a
deletion it performed.
TestDecommissionMovesRestoredCopyPastDeadline refuses the removal on
every drive of the set; before this change it failed with complete=false
failed=true, pool 0 holding the version and pool 1 nothing.
TestRebalanceLeavesRestoredCopyPastDeadline is the control for the
rebalance walk.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions purges each key as a whole,
and the purge refuses a key when one of its versions is transitioned to
a remote tier, as it refuses a key with a version Object Lock protects.
The job counted the Object Lock refusal as a refused key and completed,
but counted the tier refusal as failed versions and ended Failed, on
every run, with objectsFailed set to the versions the walk saw. ILM
expiry reports the same tier refusal as refused, not failed.
expireAllVersions now counts a key refused for a transitioned version as
refused, as it does a locked key: nothing was removed, the attempt is
not retried, and the job completes with keysRefused, which mcli reports.
A later run of the job checks the key again. A key whose history cannot
be read still fails the purge as before.
TestBatchExpirePurgeRefusesTieredKey runs a purge of a key with a
transitioned older version and requires a complete job with one refused
key and every version kept; TestPurgeAllVersionsRefusesTieredKeys now
requires refused keys rather than failed objects. Before this change
the job ended Failed with two failed objects.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Canceling a job ID this node does not own recorded a canceled owner so
that startup discovery, which may still admit the manifest of that ID,
would not run a job that was canceled before the node had loaded it.
The record was kept for any ID, at any time, and never removed: a caller
with admin:CancelBatchJob could grow the ownership table by one entry per
cancellation of an unknown ID, and the retry scheduler walks that table
every second.
Once discovery has finished, nothing admits a job this node does not own:
new submissions use new IDs, so the record has nothing left to fence.
canceler now records a canceled owner for an unknown ID only while
discovery is still running. Canceling an owned job, and the fencing of a
cancellation during discovery, are unchanged, and the cancel handler
still removes the manifest.
TestBatchCancelUnknownJobAfterDiscovery cancels an unknown ID during
discovery and requires its admission to be refused, then cancels a
thousand unknown IDs after discovery and requires the ownership table
not to grow. Before this change it held a thousand records.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Deleting a policy checks that no user or group mapping names it, but not
the STS mappings of OpenID parent users and LDAP user DNs. After such a
deletion the request path (MergePolicies) skips the confirmed-absent
policy and decides on the remaining mapped policies, while batch
authorization treated the absent document as a denial of the whole
action: jobs of such a submitter ended refused, although the same
principal's requests were allowed.
batchLocalPolicies now skips a policy document whose load reports that it
does not exist, as MergePolicies does, and still decides on the remaining
policies. Any other failure to read a document, and a load that returns
no document, still ends the attempt as unavailable, since it could hide a
Deny. With every mapped policy absent nothing allows the action, so the
job is refused as before.
TestBatchAuthorizeSkipsDeletedMappedPolicy maps an STS parent user to two
policies, deletes one, and requires the batch decision to agree with the
request decision, allowed with the other policy and denied once both are
deleted. Before this change the batch decision was a refusal while the
request was allowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Batch authorization evaluates the submitter's mapped policies from one IAM
cache revision. It took the cache write lock for the whole evaluation and
performed the lazy loads it needed under that lock: the mapping of a user
whose policies come from a group is never cached, so every authorization
of such a submitter (and of an LDAP or STS parent user) read storage twice
with the write lock held, and an evicted policy document was read under
it too. While a read lasted, every reader of the cache waited: request
authentication (GetUser, CheckKey) and PolicyDBGet on the whole node. An
expire job authorizes each version several times, so a large job with a
group-mapped submitter serialized node authentication behind IAM storage
latency, and a mapping load had no deadline at all.
batchPolicySnapshot now holds the store's writer mutex and the cache read
lock, and releases the read lock around each storage read, as
MergePolicies releases the cache lock around a document load. The writer
mutex still excludes cache writers, so the mapping and the documents of
one evaluation still come from one revision; the read lock is enough
because the batch path writes nothing to the cache except lazily loaded
mappings, which the request path stores under the read lock as well, and
taking the write lock back after each read would have made readers queue
behind it. Each read is bounded by the job's context and the same 5s
MergePolicies grants a document load. Concurrent reads of the group
fallback release the lock once and take it back after the last.
policyDBGet and policyDBGetGroups take the loads they need through the
iamMappingLoader interface, which *IAMStoreSys satisfies unchanged.
Unreadable policies still end the attempt as unavailable, and the error
classification is unchanged.
TestBatchAuthorizeSlowLoadKeepsCachedAuthorizationAvailable holds the
lazy load of a group-mapped user's mapping, and of an evicted document,
and requires that GetUser and IsAllowed of a cached user return within a
second and that a load ends with the job's context. Before this change
both cases timed out on the blocked cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A RenameData that fails after its caller gave up removes its empty
object directory and the empty parents (ed05d982a, 73025824a). Only
the two renames of RenameData itself survived that cleanup, through
renameDataAll. Any other write of the same key that creates the object
directory failed on the drive when the cleanup removed the directory
between the directory creation and the open: WriteMetadata for a
delete marker on an absent key or for a first version, through
openFile, returned the raw open error, and writeAllDirect returned
errFileNotFound. The version was then missing on that drive, and
DeleteObject queues only offline drives for MRF. Before ed05d982a the
failure path never removed the directory, so this window is new.
Move the retry policy of renameDataAll into the shared helpers. A write
below the volume directory retries once per path level when the
directory creation or the use of the directory reports "not found"
(retriesAgainstCleanup): a cleanup removes each directory at most once
and each failed attempt needs one such removal. reliableRename applies
it in place of its blind single rename retry, which never recreated the
directory, and only while the source still exists, so renameDataAll is
replaced by renameAll. openFileAll applies it to the directory creation
and open shared by openFile and writeAllDirect. mkdirAllErr is split
out of mkdirAll, like renameAllErr, so that the retry decision uses the
OS error: on unix, mkdirAll maps a vanished parent to
errFileAccessDenied through isSysErrPathNotFound.
Invariants kept: the cleanup still removes only empty directories and
never a committed xl.meta or data directory; the error mapping of
mkdirAll, renameAll, openFile and writeAllDirect is unchanged, including
the Windows isSysErrPathNotFound branch; retries are bounded by the
path depth; a rename whose source is gone still fails at once.
UpdateMetadata renames over an existing xl.meta, so its directory is
never empty and the cleanup cannot remove it; it needs no change.
TestXLStorageMetadataWriteSurvivesLateCleanup injects, right after the
writer's os.Mkdir of the object directory, either a late RenameData
whose staged data is gone or a cleanup that walks up the whole path one
directory per step, for a delete marker, a first version and a large
xl.meta. Before the fix all six cases failed with "no such file or
directory" or "file not found".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of the delete-all reporting found that no
committed test checked the error a purge returns on a drive whose
metadata does not decode: restoring the old read quorum error passed.
TestPurgeAllVersionsRefusesCorruptDrive now sends the purge through
DeleteObject and checks errDeleteAllHistoryUnavailable and its
SlowDownWrite mapping; with the old error it fails.
The HELP text of expiry_incomplete_total now says that the
tier_unsupported_skipped reason counts the scanner's per-drive
observations, not expiry attempts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of this round's public fixes found one gap in
the code and four in the tests.
- A refused expiry was traced with its reason but without its error,
so the trace did not say what refused it. It now carries the error, as
a failed one does.
- The restore MRF test replaced the live MRF channel, which the heal
routines of the test servers read: a race, and a consumer could take
the entry under test. addPartialOp now offers an atomic hook that
receives the operation in place of the queue. Production never sets
it.
- Two fixtures restored the storage class with Update, which marks an
uninitialized configuration initialized and so changes the default
parity (-1) into an explicit 0 for later tests.
setTestStandardParity restores the whole saved value under
storageclass.ConfigLock.
- The v3 metric test built its own descriptor map, so dropping the
counter from the registered ILM collector still passed. It gathers
through the collector the handler registers.
- The audit test read one extra entry or waited 500 ms. It drains every
entry of the bucket up to the removal audited last, and checks one
failure audit, one removal audit and no refusal audit per object.
Each corrected test fails with the corresponding mutation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions of a key purges exactly that key, and the
purge can refuse it or stop early. Those outcomes were hard to see, and
the scanner reported a key it had queued as if it were already gone.
- Refusals and failures of a purge are counted in
expiry_incomplete_total with their own reasons: tier_unsupported (a
version is in a remote tier), budget (stopped after progress; the next
expiry continues), changed (the latest version a batch matched
changed) and history_unavailable. They are traced with the reason and
send no removal event. A partial purge keeps its audit record, now
with the reason. A tier refusal is also logged, once per key and hour,
in words.
- A drive that cannot be read or decoded during the purge's own drive
check now returns errDeleteAllHistoryUnavailable, which names the
cause and stays a write quorum error (503) for API callers.
- The scanner does not queue an expiry of all versions whose latest
version is hot when the pool holds a version in a remote tier: the
purge would refuse it each cycle. It counts each such observation as
tier_unsupported_skipped, once per drive scanner and cycle. A tiered
version only another pool holds is unknown here, so the purge still
decides; a transitioned latest version keeps the transitioned expiry.
The tier predicate is shared with the purge's own check, for FileInfo
and ObjectInfo, and excludes free versions.
- The scanner accounts every version of a key under an expiry of all
versions at its size, queued or not, instead of as already removed.
Usage, and so hard quota, drops after a fresh scan of the subtree once
the purge succeeded. The alerts on many or large versions count the
versions of a key that is not queued.
- A lifecycle expiry never deletes a prefix: DeleteObject refuses
DeletePrefix with Expiration.Expire as InvalidArgument. No caller
sends it since expiries of all versions became purges; a force delete
of a prefix is unchanged.
Not done from the plan: queuing an MRF heal per version when a drive's
xl.meta fails to decode. Object heal does not rewrite metadata that
fails its CRC check (it reports the drive as unknown and skips it), so
those heals would not repair the drive. Such a key stays refused until
its metadata on that drive is repaired.
Tests: TestPurgeAllVersionsRefusalReasons,
TestPurgeAllVersionsRefusesCorruptDrive,
TestScannerSkipsDeleteAllForTieredKey and
TestPoolsDeleteObjectRejectsExpirePrefix are new. Tests that expected
the scanner to queue a refused purge, the predictive accounting, or
the old prefix options are converted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission stopped at the first bucket it could not finish. A free
version whose tier is no longer configured is enough: its remote object
cannot be removed, so the free version stays, the bucket's check for
versions left behind fails, and every bucket after it stayed in the pool.
decommissionInBackground now keeps the first error, continues with the
next bucket, and returns that error after the last one. A canceled
context still returns at once. A bucket is retired from the resumable
queue only when it is empty, as before, so the decommission still fails
and a retry resumes with the buckets that are left.
removeFreeVersions now names the free version, its remote object and the
tier when the tier is not configured, and says to configure the tier
again under the same name. The free version is the only record of its
remote object, so it is kept; nothing is dropped on inference. Until the
tier is configured again, such a free version blocks the decommission
of its pool.
Tests: TestPoolsDecommissionContinuesPastStuckBucket and
TestRemoveFreeVersionsMissingTierError fail on the base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of the lock fixes found two gaps in the merge
test of lockObjects:
- Where the sets share one local table, it only tried an alias of the
clean name's own set, so dropping the shared-table normalization of
the lock domain still passed. It now tries an alias of another set in
every mode; with that mutation the shared case times out.
- In a split distributed domain it probed readers before writers. The
reader probe releases asynchronously and could hold off the writer
probe right after it, so a missing alias read lock passed some runs.
Writers are probed first now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore commit that reaches a write quorum but not every drive
queues an MRF heal of the version. It passed the raw version ID, which is
empty for the null version, while the convergence path passes the
normalized "null". The heal boundary already reads an empty ID as the
null version, so the old entry healed the right version; this makes the
two paths queue the same identity.
The header of restore-capability.go still described the multipart upload
of the first native-restore design. It now describes what the capability
gate protects in the two-phase design: restore ownership through the
restore lock and the attempt marker, the staged commit under the object
lock, and the attempt fence. It also says why an unreachable server
blocks native restore: it is not proven stopped.
Test: TestRestorePartialCommitQueuesNullVersionHeal fails one drive's
commit on a noncurrent null version, checks the queued entry names
"null", and heals it back onto that drive. It fails on the base at the
identity check only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An ILM expiry that is refused, because the rule no longer applies when
it is checked again under the object lock or Object Lock protects the
version, or that fails, for example for lack of quorum, left almost no
trace. The bulk noncurrent purge skipped such entries without audit,
trace, log or count; the scanner traced a refusal exactly like a
successful expiry; and nothing counted either. Operators could not see
why a version they expected gone was still there.
- Count every such expiry by reason in expiry_incomplete_total
{reason="not_applicable"|"locked"|"history_unavailable"|"error"}, in
metrics v2 and v3; the counting unit is one expiry attempt, and no
bucket or object is a label.
- Trace it with result=refused or result=failed and the reason. A
failure is also audited and logged once per object; a refusal changes
nothing and is not audited, since a split key is refused on every scan.
- Never announce it: no ObjectRemoved notification for a version that is
still there.
- Name a completed expiry by what its delete did. An expiry that
addressed no version and whose result is a delete marker created that
marker, also with the null version ID of the version it replaced; any
other removed the version it addressed, also a delete marker. The
scanner and the transitioned expiry took the identity from the queued
object instead: the expiry of a latest transitioned version in a
versioned bucket announced the removal of the version it kept, and the
removal of an expired delete marker was announced as a marker created.
Tests: classification of each error; a bulk purge with a refused, a
failed and a removed entry counts and traces the first two, audits only
the failure and announces only the removal; the expiry events name the
created marker for a latest version (versioned, suspended null,
transitioned) and the removed version for a noncurrent version and an
expired delete marker; both metrics versions export the counter by
reason. The #278 side-effect tests now tell removal traces from
refusal and failure traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Earlier tests can leave a storage class of parity 0 behind (newTestConfig
on a single drive), and a version written without parity cannot be
rebuilt on the drive the tests heal. Set the parity of the test pools'
sets for the duration of each drive heal test, as other tests that
depend on parity do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of the lock fixes found test flaws, which are
corrected here; the production behavior is unchanged.
- The alias and merge tests replaced globalOperationTimeout while
background healing reads it, a data race under -race. They now bound
each request with its own context instead.
- The alias test now holds the cleaned name's write lock while the copy
runs, so it proves the names are checked before anything is locked;
it checks both objects before any later write, and covers a/./x as a
destination.
- The merge test picks aliases that hash to the clean name's set and,
where sets have their own lockers, ones that do not, and requires the
alias's own table to hold a read lock: readers admitted, writers
excluded.
- Release polling creates a fresh lock instance for every probe.
- The getNSLocks event hook and the bulk delete group limit are atomic,
so tests can set them while other goroutines read them.
Without the name check and the resource merge, both tests now fail in
every lock layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission or rebalance walk prechecks each due version of a key
with the versions of the key in all pools, to decide whether the version
moves or is left to its ILM expiry. Each precheck read and merged the
key's whole history from every pool, so a key with V due versions read
it V times in the walk alone; with long histories this grew roughly
quadratically.
Read the history once per key into a cache that the walk keeps for the
entry, and evaluate every version's precheck against it. The precheck
only chooses between moving a version and leaving it to the expiry; the
expiry reads the history again under the object lock before it deletes
anything, so a stale cache can only move a version or defer it to a
recheck. A read error is kept, so every due version of that key moves.
The expiry's own per-version reads are unchanged.
Test: decommission and rebalance of a key with 63 due noncurrent
versions read its history once, with the same outcome; without the
cache they read it 63 times.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
BatchJobExpire.Start checked the job's submitter before it looked at
the job's report, and refuse() clears Complete. A job that had completed
was therefore turned into a failed one on the next start: one that
completed before the upgrade records no submitter, and one completed
with a temporary credential that has expired since, a deleted user or a
changed password fails the binding. Such a job asked to be submitted
again, inviting an operator to run a destructive expiration twice.
Return before the submitter check when the report says the job is
complete, as replication and key rotation jobs already do.
Tests: a completed job without a submitter, and a completed job whose
temporary credential expired, keep their completed state and delete
nothing when started again; a job that did not complete is still
refused once its credential expired. The first two fail without the
change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A version can have copies in two pools, for example after an interrupted
decommission or rebalance. DeleteObject left ILM expiries out of the
reconciled delete of an addressed version, so a lifecycle version expiry
deleted only the copy that latest-key routing chose. The other copy stayed
readable; for a tiered version, expireTransitionedObject then deleted the
remote object that copy still referenced; and a migration of the pool
holding it left the expired version behind and failed its completion
check. With delete replication on, the replication completion's own
delete had reconciled the copies and hidden this.
Admit an ILM version expiry that the pools layer approved under the lock
with the versions in all pools (CheckedAllPools, and not a restored-copy
expiry) to deleteObjectReconciled. It removes the secondary copies before
the primary and stops at the first failure, so the remote object is only
deleted after every copy is gone. Each set still guards the protection
the pools approved. Marker creation, restored-copy expiry, free-version
cleanup and delete-all keep their own paths; the bulk noncurrent purge
already reconciled.
Tests: with copies in two pools, a hot and a tiered version expiry remove
both copies, and the tiered one deletes its remote object once; when the
secondary copy cannot be deleted the expiry fails, both copies and the
remote object stay; with replication configured the tiered expiry queues
nothing; decommissioning the pool that holds the second copy completes
and leaves no expired version. All fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The migration follow-up for #265 (PR #276) made lifecycle-created
deletes replication-aware: the scanner, the expiry of a transitioned
version and the bulk noncurrent purge evaluated the delete replication
decision and queued a delete marker or a version purge for the targets.
That is the content of the still-open PR #240. It changes behavior for
every bucket whose replication rule replicates deletes or delete
markers, the mc default: a target with longer retention loses versions
on the source's schedule, and a purge keeps its space as a pending
tombstone until the target confirms. SILO, like S3, does not replicate
lifecycle deletes; an opt-in can come later. The release did not
replicate them.
Remove exactly the replication parts: the decision callbacks and the
post-delete scheduling in applyExpiryOnNonTransitionedObjects and
expireTransitionedObject, the scheduling in deleteObjectVersions, and
ilmPurgeToDelete; the scanner builds noncurrent entries as before. The
rest of #276 stays: the hot-tier delete before the remote one, the
free-version retry record, the lifecycle-change fence and the migration
completion checks. API deletes still replicate, and the replication heal
still handles persisted pending markers and purges. As in the release,
the replication heal also resyncs an empty-state marker when
existing-object replication applies to it.
The delete-all variant of the scheduling, which queued a task with an
empty decision, goes with it.
Tests: the scanner's noncurrent purge, run with replication configured,
deletes the version at once and queues nothing; an ILM marker carries
no replication state and the heal leaves it unless existing-object
replication applies, in which case it is resynced as before; an API
version delete still queues a purge. The cross-pool batch, dispatch and
decommission tests now assert no replication, and a decommission whose
expiry used to wait for the target completes in one run. Adding back the
scanner's decision makes the first two fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The single-pool lock-domain change made a bulk delete lock each set's
names in that set and ran the groups one after another: a batch over S
sets cost about S lock round trips plus S set deletes, where the release
took one lock and then let erasureSets.DeleteObjects work on all sets at
once. Every distributed single pool with several sets paid for it,
including layouts where every set uses every host.
Run the groups concurrently, up to 16 at a time. Each group takes only
its own set's lock and waits for no other while it holds it, so the
groups cannot hold each other. A group checks for cancellation once it
is admitted, so a queued group of a canceled request does no storage
work. Each result is written only by the group of its object.
Tests: the first drive calls of two sets must overlap, which fails when
the groups run one after the other; with one worker, a group in flight
at a cancel keeps its own result, an earlier group keeps its success,
and a queued group makes no drive call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
getNSLocks takes the locks of a copy in one order and waited for a later
lock, such as the source, for up to two seconds while it held the earlier
ones, such as the destination; then it released them, paused up to 250 ms
and tried again until the operation timeout. When the source was write
locked for long (a multi-pool PUT for its whole upload, a heal, a
transition, another copy onto it), the copy held its destination write
lock most of the time: reads and writes of the destination waited for
seconds, their own backoff rarely hitting the gaps. In the release a copy
that waited for its source held nothing.
After a bounded wait for an atomic lock times out while earlier locks are
held, release them, wait for that lock alone with a fresh instance until
the deadline, release it once it is free, and start again from the first
lock in the same order. Holding nothing while waiting forms no cycle, and
the next attempt still takes every lock in one order with fresh
instances. A non-atomic request may hold some of its own names while it
waits, so it keeps the bounded retries.
A test hook reports the bounded timeout, the start of the idle wait and
its end, so the tests coordinate on events rather than timing: a lock
before the contended one is free for others for the whole wait, a cancel
or the deadline ends the idle wait and leaves nothing held, a non-atomic
request never waits alone, and HEAD of a copy's destination succeeds
while the copy waits for its source. All but the non-atomic control fail
without the idle wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
CopyObject locks its source and destination before it reads the source,
and lockObjects merged two requests only when their raw names were equal.
A lock is taken on the cleaned name, so an invalid source such as a//x or
a/./x and the destination a/x asked for a read lock and a write lock on
one resource: the copy waited on itself until the operation timeout and
answered 503, holding the key's lock in short windows meanwhile. Before
the lock-domain change the invalid source was refused at once, when the
source was read. This reached single nodes, multi-pool deployments and
single pools whose sets share hosts.
- Check the source with checkGetObjArgs before any lock, the first check
and the first error the release reported. Lock the destination only
when its name is valid. An invalid destination fails at the copy's own
destination check before it writes, after a missing source (404) or a
failed source precondition (412), as before.
- lockObjects merges requests by the resource the lockers see, the
cleaned bucket/object, within one lock table: the object's set where
the sets of a pool have their own lockers, else one table. This keeps
aliases on a shared local table from conflicting and never merges
requests of different distributed domains.
Tests: aliased sources and destinations are refused with 400 at once in
every lock layout and two pools; an invalid destination keeps the 404
and 412 precedence; lockObjects merges an alias with its clean name. The
alias and merge tests fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Drive healing evaluates lifecycle on the versions of one pool. For a
version it found due it enqueued the expiry and skipped the heal. The
expiry is decided again under the object lock, with the versions in all
pools and the Object Lock state it reads then, and can be refused: a
newer version in another pool, an unreadable history, a changed rule,
or a delete-all the exact-key purge refuses. Such a version was never
healed onto the replaced drive and kept reduced redundancy for as long
as the refusal lasted. The scanner heals only versions without a due
action, so it did not repair it either.
Keep the enqueue and heal the version. A version that the expiry then
removes costs one heal. Heal and expiry exclude each other through the
object lock in the writers' lock domain, which the single-pool lock
domain change provides; a version removed first reads as not found and
is skipped, and heal rebuilds only from read-quorum metadata, so it does
not bring back a version deleted on a write quorum.
Tests: a version refused across pools is healed onto the replaced drive
and survives the expiry; an unversioned object due is healed and then
removed from every drive; a versioned latest version due is healed and
hidden by a marker; a stale minority is not recreated. The first three
fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Three restore tests record the options of a POST's background restore
with a stub and restore with them later: TestRestoreAttemptGenerationFencing,
TestExpireRestoredUsesQuorumGeneration and TestRestoreHealNeedsRestoreQuorum.
The options carry the restore lock that POST hands to its worker. The
handler releases that lock only after the stub returns, while the test
proceeds as soon as the stub has signaled. When the test's restore
checked the lock before that release, it kept the lock, and the release
then canceled its context: every drive read failed with "context
canceled" and the restore reported insufficient read quorum.
Under -race this failed 11 times in 60 runs of the three tests on
69a4f938a, in the worker and old-reset modes of the fencing test and in
the multi-pool expiry test.
A new helper, restoreCaptureOptions, releases the lock before it records
the options, so a later restore takes it again, as a queued worker whose
lock ended does. TestRestoreSupersededAttempt already does this. With
the helper, the three tests passed 60 runs under -race.
Production code does not change: the handler's goroutine runs the
restore and releases the lock after it, and a lock lost during a
restore stops it as intended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A node that reaches only some drives of a set, for example right after
it restarts and before it reconnects to its peers' drives, could answer
404 NoSuchKey for an object that exists. It happened when the copies it
could see were split between two revisions and neither reached read
quorum: a restore, a transition or an unversioned overwrite committed on
a write quorum but not on every drive. S3 clients and caches treat 404
as final; the honest answer from such a view is a retryable 503.
Two paths turned that view into "not found":
- getObjectFileInfo converted InsufficientReadQuorum{RQInconsistentMeta}
to not found. Upstream added this for the case where every drive
answered, but never checked it.
- When the version merge found no version in quorum,
pickLatestQuorumFilesInfo marked every drive that answered as not
found, including drives that hold the object.
Both now require every drive to have answered: metadata, or a definite
file, version or volume not found, or corrupt metadata. Otherwise the
read keeps its read-quorum error. A key absent from every drive that
answered still gets 404, and so do inconsistent copies when every drive
answered. The dangling-object purge is unchanged.
Deletes, copies, metadata updates, decommission and rebalance that read
through such a view now fail and retry instead of treating the object
as absent. The change is local to the serving node: no storage RPC,
on-disk format or configuration changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestBatchExpirePurgeAuthorizesDriveRevisions writes a failed tag revision
to one drive of sixteen and needs the read quorum of the metadata to stay
below sixteen. Tests that run newTestConfig on a single-drive layer, such
as TestIsReqAuthenticated, leave a storage class of parity 0 behind, which
resetTestGlobals does not restore. With parity 0 the quorum history of the
key is unreadable, the purge correctly refuses, and the test failed in the
full ./cmd run while passing alone. Set the default parity for the test,
as other erasure tests do, and restore it afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A Walk that returned an error before any result cancelled the run like
an ordinary cancellation: the job was saved as neither complete nor
failed. The listing failure is now recorded first, and the run is saved
as failed with its cursor where it was. Start still returns the error,
so the job is left to a later run, which lists again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Key rotation authorized the version as the copies in all pools merge
it, but rewrote the copy that GetObjectInfo returned, routed by the
pools layer to the pool of the latest version of the key. Where the
copies differ, it rewrote metadata it had not authorized; where another
pool held another write of the same version ID, it replaced that
write's sealed key without re-encrypting its data, which no longer
decrypted. The rewrite also stored the metadata without the tags and
the expiry, which a read returns apart from it.
Rotation now authorizes both the copy it rewrites and the merged
version, and rewrites exactly that copy in its own pool. A version read
from a pool that a decommission or a rebalance is moving is left to a
later run, checked again right before the write, so that the source of
the move does not change under it. The tags and the expiry are kept.
A pool-state change during the commit itself affects every write of
the pools layer and is left to a fix there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions authorized the versions its
walk saw, and the purge then deleted the versions it read under its
lock. A version written after the walk, or tags changed on one since,
was deleted without being authorized for the submitter.
The purge takes an authorization that a batch purge must set and no
other delete may. Under the object lock, once it has approved the
history and checked every drive, and before its first delete, it
hands the writes it is about to delete to the job: every copy of every
approved version and every revision of one that a drive holds, one per
distinct version ID and tags, as they are then. The job evaluates the
expire actions on each for the submitter. A denial deletes nothing and
ends the job as refused; the time it takes counts against the purge's
budget. A locked version still refuses the whole key, counted as a
refused key. The checks on the walk's snapshot are removed.
Reviewed adversarially; the review's regressions are included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Submission and start with changed, expired or restricted credentials,
session policies with and without an authorization plugin, request
context in policy conditions, per-object denies, jobs without a
recorded submitter and describe output; expire, pull, push and
rotation over versions under retention or legal hold, on one and
sixteen drives; cursor and counter persistence; and the cases found in
review: destination versions and tags of pulls, encryption values,
S3 Tables statements, copies in several pools, claims, prefixes,
listing errors and retry accounting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch job ran with the server's object access, whoever submitted it,
and acted on versions that Object Lock protects.
Submitter: StartBatchJob binds the authenticated credential to the job
(type, access key, parent, identity revision and a keyed fingerprint).
Submission, every start and every object resolve the binding against
IAM and evaluate the action with the submitter's policies and session
policy. A credential that is gone, disabled, expired or changed fails
the job. A policy statement that would need request context the job
does not have refuses the action. With an authorization plugin, the
plugin and the session policy must both allow. A refusal is persisted
as the job's failure reason and ends the job. A job stored without a
binding fails and has to be submitted again; a definition holds one
job type.
Expire deletes each older version on its own, checking retention and
legal hold and authorizing it under the delete's lock; a locked version
is left and counted. Until the exact-key purge is integrated, a
retainVersions: 0 key is refused when a version the walk saw is locked.
Pull writes refuse to replace a locked destination version, use the
destination's versioning, validate copied retention and apply default
retention. Rotation leaves locked versions as they are.
The status reports versionsSkippedLocked, keysRefused and the failure
reason. Expire and rotation persist their cursor together with these
counts, so a resumed job counts each version once. Describe returns a
redacted copy of the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
IsAllowed returns the authorization plugin's decision when a plugin is
configured, and evaluates the local policies otherwise. Move the local
evaluation into isAllowedLocally so that a caller can combine it with
the plugin's decision. IsAllowed is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Add ObjectOptions.RefuseLockedOverwrite. With it set, PutObject (before
the rename) and CompleteMultipartUpload check, under the destination's
write lock, the stored version the write would replace: the version it
addresses, or the null version. DeleteObject checks the stored version
that a delete marker it writes names; in a multi-pool deployment that
version is resolved across pools. A version under retention or legal
hold is refused with ObjectLocked. The check reads the version's own
metadata, whatever the bucket configuration, and returns read errors.
Nothing sets the option yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The unreleased native-restore fixes after RELEASE.2026-10-26T00-00-00Z
let the lock decide who owns a restore. Since fa42a0982 a single-pool
worker holds the object lock for the whole tier download, and since
7f38bf628 a busy lock means "a restore runs". 7f38bf628 also keeps the
restore's data in a marked multipart upload. Two regressions followed:
- While an old version restored, every GET, HEAD and write of the same
key waited for the download. On a single pool this is new; with
several pools the released code already held the lock that long.
- An ordinary upload whose completion failed after staging its metadata
became InvalidUploadID for every caller, and could not be aborted for
an hour. The marker that hid restore uploads could not tell them apart
once a completion had rewritten the upload's xl.meta.
A restore now runs in two phases:
- The worker erasure-codes the tier copy into the tmp volume of the
version's set, without the object lock, as heal does. A drive that
fails any part, or misses a part when CheckParts runs before the
commit, takes no part in the commit.
- A restore lock per version tells whether a restore runs. POST takes it
before the object lock and hands it to the worker. It is a liveness
hint: only the checks under the object lock decide what is written.
- The commit takes the object lock briefly, on one pool or several. It
reads the version again and checks the attempt's ownership, the
version identity, the tier copy, the part layout and the erasure
layout. It then publishes with RenameData drive by drive, never
through erasure renameData's undo, and needs the write quorum. The
replaced data directory goes only when every published drive reports
the same one.
- A publication that reached some drives but not the write quorum
converges under the same lock hold, in bounded retries after it: back
to the unrestored version on every drive that holds it, or to a
committed restore when the copy turns out to be on a write quorum.
Each pass first checks that every drive holds what the version held
before the commit, the commit wrote, or a convergence wrote, and stops
on anything else. It deletes no directory.
Multipart goes back to main: erasure-multipart.go and
bucket-object-lock.go are identical to 69fcf50cd. The commit lock, the
marker and the restore exceptions of multipart completion and of the
null-version Object Lock check are gone, and single-pool completion
again takes the upload lock first and the object lock right before it
commits. With several pools, abandoned-data cleanup now takes the pools
lock that writes take, as heal does, so it cannot remove a directory
that a commit moved in place and has not yet referenced. The restore
capability becomes protocol 2, and a queued worker checks it again.
Recovery after a coordinator crash during publication, and storage RPCs
that complete after their caller returned, stay excluded. Main has the
same exposure, and they are tracked separately.
Tests:
- New regression tests cover both problems. They fail on f85d4a5fa,
and the multipart one also passes on main.
- Fault-injection tests cover a publication below the write quorum,
including seven of twelve drives on 8+4; lost answers; convergence
retries and stops; the old data directory; a stale inline marker; a
recreated tmp volume; a published revision read across its expiry; a
second POST; a target deleted during the download; a tier 404; and
pools-locked abandoned cleanup.
- Thirteen tests of the upload marker from 7f38bf628 are dropped, and
so is TestNativeRestoreConcurrentCompletion from fa42a0982, which
completed the restore's upload from a client. Four tests are reframed
for the new semantics, including the lock-domain restore test, which
now gates the commit.
- Three self-recursive WriteMetadata test doubles are fixed, and the
helpers of the dropped tests are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang