UpdateServiceAccount, SetUserStatus and UpdateUserSecretKey read the
identity from the serving server's cache, changed it and wrote it back
whole. On a server that missed a notification, an ordinary edit (a
description, a status, a secret) undid a rotation made through another
server and brought back a user or service account deleted meanwhile, with
its old secret.
Edits now go through iamIdentityEdit, inside the identity's cross-node
revision lock: the stored record is loaded and normalized as the loaders
do (access key from the path, claims from the token, a deprecated comment
for an empty description), the edit applies to it, and an absent or
deleted identity is refused and dropped from the cache of the server
that tried. The LDAP group refresh of a0de92fcb uses the same mechanism.
- The admin handler binds a service-account edit to the parent it
authorized: an account deleted and created again for another parent
meanwhile is not edited (parentUser in updateServiceAccountOpts;
replication does not set it).
- A replicated edit older than a deletion this site applied is answered
as stale, so site replication acknowledges it instead of retrying.
- SetUserStatus refuses a service or temporary credential as before, and
a refused edit drops only a cache entry of the addressed kind.
- A session-policy edit stores the claims of the token it signs; the
token is signed once.
- UpdateUserSecretKey addressed to a service-account key finds no user
instead of writing the account to the regular-user path.
Tests use a second IAM store over the same storage whose cache missed the
change: rotation, deletion, status and secret edits of users, parent
binding, legacy records, replicated edits around a deletion, and status
calls on a service account. Against the cache-based writers every case
fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The hourly LDAP group refresh, which every server runs, reads the STS and
service-account credentials from its own cache, queries the directory,
and wrote each credential whose groups changed back whole through
UpdateUserIdentity. A secret rotation or an edit made meanwhile, on this
server or one whose notification it missed, was undone, and a credential
deleted meanwhile came back enabled with its old secret.
RefreshUserGroups replaces that writer, whose only caller was the
refresh. Under the identity's cross-node revision lock it loads the
stored record, refuses one that is absent, deleted, expired or now has
another parent (whose groups the lookup did not compute), and otherwise
changes only the groups of the stored record. A record without its
access key gets it from the path, as the loaders do. A refused refresh
drops the stale entry from the cache, which loads what is stored on next
use; a written or unchanged one caches the stored record. An STS record
keeps the lease of its token's remaining lifetime.
Tests cover rotation and edit, deletion, a sibling server whose cache did
not hear of a rotation or a deletion, recreation under the same and under
another parent, unchanged groups, and a stored record without its access
key. Written back as the old snapshot write, every case fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestBatchExpirePurgeRefusesTieredKey marks a version transitioned through
DecomTieredObject, which takes its parity from the global storage class.
Run after tests that leave a parity the one-drive fixture cannot have,
it wrote on no drive and still returned nil, and the purge then removed
the key. The helper now sets the set's own parity for the test and reads
the transition back from every drive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The submitter binding compared the UpdatedAt of the identity record, so
any rewrite of the record ended every running job of the credential with
"changed since submission": a service account's description, name or
later expiry, and the hourly LDAP group refresh of every LDAP-backed
service account and STS credential whose groups changed.
Compare only what identifies the credential: its type, access key and
parent, the revocation boundary of the record, and the keyed binding of
its secret and session token. Status, expiry, groups and policies are
read again on every start and action. A deletion advances the
revocation boundary, so a user created again under the access key with
the same secret still fails the binding, and adding an expiry to a
service account turns it into a temporary credential, which still ends
the job. UpdatedAt stays in the binding for diagnosis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch job is authorized without a request, so a policy statement that
depends on request context the job does not have (source address,
transport, headers) was taken to refuse the action. Two parts of that
rule refused jobs that no request would refuse:
- The scan for unsupplied policy variables ran over the whole statement
before its resources were matched. A Deny on an unrelated bucket whose
condition value held a variable such as ${aws:UserAgent} refused every
job of the user. Resources are now matched first; a variable in a
resource still means the statement may apply.
- An Allow that needs absent context, such as an IP-restricted read,
refused the job even when another Allow granted the action. Such an
Allow now grants nothing to the job: it is dropped from the policy that
is evaluated, not merely exempted, since IfExists and Null conditions
hold for an absent key and would grant. Dropping an Allow can only
narrow the decision. A Deny that needs the context still refuses.
The narrowed policy is a new policy.Policy, because the statement index
of the original points into its own statements and the original may be a
cached document. For a session policy, whether one applies and its first
decision still come from the claims as before; only an allow is
re-evaluated on the narrowed policy.
Tests cover an unrelated Deny with a variable, a Deny with an unsupplied
variable in its resource, an IP-restricted Allow next to a plain one, a
Null Allow that plain evaluation without the address would grant, and
the same for session policies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
batchPolicySnapshot holds the IAM store, a context and a mutex. Without
an ignore directive, go generate emitted msgp encoders for it that do not
compile, so make check-gen (and the check-gen step of CI) failed on the
generated batch-authz files. The committed generated files are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The cache read lock is reached through the embedded IAMStorageAPI, so the
explicit selector is redundant for staticcheck (QF1008). Drop it and say
in a comment that rlock and runlock are the cache's locks, unlike
IAMStoreSys.lock, which also takes writeMu. No behavior change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Three safeguards of the lifecycle and tier cleanup paths had no test
that failed when they were removed:
- authoritativeHistory accepts a revision only on more than half of the
drives. TestPoolsLifecycleHalfDriveMarker holds a delete marker on
exactly half of sixteen drives, which is the read quorum of a marker,
and checks that the history is refused, that the expiry of the
version under it and of the key are refused and every drive keeps what
it has, and that one more drive decides. With the guard removed the
strict merge at the same majority still refuses the history; with the
merge relaxed as well the test fails, so it pins the refusal, not one
of its two layers.
- deleteAllApproved refuses a purge whose placement read differs from
the versions it approved. deleteAllBeforePlacement is a test-only
injection point between the two reads, nil in production.
TestPurgeAllVersionsRefusesChangedApproval writes a new version and
rewrites the latest one there, as a writer that bypasses the object
lock would, and checks the write quorum refusal, the
history_unavailable count, that nothing is deleted and that the next
purge removes the key. With the check disabled the purge deletes the
versions it never approved.
- readTierHistory refuses the ownership of a remote object while a
drive of any pool cannot be read, and xlMetaV2.DeleteVersion deletes
a free record only with the remote tuple it names.
TestTierOwnershipReadFailsClosed fails one drive of the pool that
holds nothing of the key as offline, not found, faulty and
unformatted, and checks that the expiry worker's and the
decommission's cleanup are refused with a read quorum error and
succeed once the drive reads again.
TestFreeRecordDeleteNeedsMatchingTuple deletes a free record with
another tier, remote object and remote version, in memory and on an
xlStorage drive, and checks the record stays until the tuple matches.
With either check removed the remote object or the record is deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
GET and HEAD recheck a local expiry decision against the versions of
the object in all pools before they hide the version or queue its
expiry. When that recheck failed, because the history of another pool
could not be read, or found that the rule no longer applies with the
versions in all pools, the read was served and nothing was queued,
without a trace or a count. During a degraded pool a version that is
due kept reading and never reached the expiry workers, and the operator
could learn of it only from the scanner's own counts.
recheckReadExpiry now reports such a read as the scanner reports an
expiry it did not complete: it counts expiry_incomplete_total by the
reason ilmExpiryNotDone gives the error, history_unavailable for a read
quorum failure, not_applicable when the version is no longer due, and
publishes an ILM trace with result=skipped and that reason. The reasons
are the existing label values; the response, the version read and the
decision not to queue are unchanged.
TestPoolsLifecycleReadHTTP now asserts the counter delta and the trace
of every case the recheck skips; before this change it counted nothing
for the unreadable-pool and the not-applicable cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A purge of all versions reads the history of the key in every pool
before it checks the drives. When half of the drives of a set could not
be read or decoded, objectVersions refused the history with a read
quorum error and the purge failed with InsufficientWriteQuorum before
deleteAllDrives, which alone queued a heal of each version when a drive
did not decode. With parity of half the drives, a key whose metadata
did not decode on exactly half of them still read and healed, but no
purge queued that heal: the refusal repeated until a scanner or drive
heal happened to repair the drives.
objectVersions now refuses such a history with undecodableHistory,
which wraps the read quorum error and carries the IDs of the versions
the decodable drives hold, the null version by its explicit ID. The
purge queues a heal of each of them before it fails, from the history
of the rule check as from the placement read, and still returns
InsufficientWriteQuorum. The threshold of unknown drives, the refusal
and every other caller of the history are unchanged; expiryApplies
matches the read quorum error through the wrapper.
TestPurgeAllVersionsHealsUndecodableHistory corrupts the metadata CRC
on four of eight drives; before this change the purge refused the key
without queuing any heal.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The native restore gate refused a POST restore and a queued restore
with one message, "unavailable during a rolling upgrade: not every
server of the cluster supports it yet", whether a server answered
another protocol or was simply not connected. A cluster with one server
down thus reported a rolling upgrade, and the operator looked for an
older binary instead of the server.
The gate now returns an error that wraps errNativeRestoreUnavailable
and says why: errNativeRestorePeerUnreachable names the server that is
not connected ("server host:port is not reachable, so it cannot be
shown to have stopped an older restore"), errNativeRestoreUnsupported
keeps the rolling-upgrade text for a server that reports another
protocol or reconnected while it was asked, and a check that its
deadline ended carries that error. The POST handler maps the error to
503 ServiceUnavailable with the message as before, the worker still
clears its marker and stops, and errors.Is(err,
errNativeRestoreUnavailable) holds for every refusal. The gate remains
closed in every case it was closed before.
TestNativeRestoreRefusalReasons posts a restore with a fake server that
is not connected and one that answers protocol 1; before this change
the first answered 503 with the rolling-upgrade text and without "not
reachable".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission walk removes a restored copy that is past its restore
deadline before it moves the version, as the scanner would. Since
0a546c7a7 a refused removal made filterLifecycle return the
DeleteRestored action: the walk counted the version as decommissioned
without moving it and without queuing an expiry, removeMovedVersions
then found a source copy with no retained copy and failed the key with
PreConditionFailed, the moved versions of the key stayed duplicated in
both pools, the bucket stayed queued and the decommission ended Failed;
the retry after a restart failed the same way while the refusal held.
The walk now moves the version like any other when the removal is
refused, as the release before 0a546c7a7 did: a copy past its deadline
reads as remote, so the version moves with its tier metadata and its
restore deadline through DecomTieredObject, the remote object stays,
the source copy retires with the key once the destination holds it, and
the scanner of the destination pool removes the restored copy by that
deadline. The rebalance walk never reaches this action for such a
version, which it skips as remote; its filterLifecycle now names the
same decision so neither walk counts a restored-copy expiry as a
deletion it performed.
TestDecommissionMovesRestoredCopyPastDeadline refuses the removal on
every drive of the set; before this change it failed with complete=false
failed=true, pool 0 holding the version and pool 1 nothing.
TestRebalanceLeavesRestoredCopyPastDeadline is the control for the
rebalance walk.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions purges each key as a whole,
and the purge refuses a key when one of its versions is transitioned to
a remote tier, as it refuses a key with a version Object Lock protects.
The job counted the Object Lock refusal as a refused key and completed,
but counted the tier refusal as failed versions and ended Failed, on
every run, with objectsFailed set to the versions the walk saw. ILM
expiry reports the same tier refusal as refused, not failed.
expireAllVersions now counts a key refused for a transitioned version as
refused, as it does a locked key: nothing was removed, the attempt is
not retried, and the job completes with keysRefused, which mcli reports.
A later run of the job checks the key again. A key whose history cannot
be read still fails the purge as before.
TestBatchExpirePurgeRefusesTieredKey runs a purge of a key with a
transitioned older version and requires a complete job with one refused
key and every version kept; TestPurgeAllVersionsRefusesTieredKeys now
requires refused keys rather than failed objects. Before this change
the job ended Failed with two failed objects.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Canceling a job ID this node does not own recorded a canceled owner so
that startup discovery, which may still admit the manifest of that ID,
would not run a job that was canceled before the node had loaded it.
The record was kept for any ID, at any time, and never removed: a caller
with admin:CancelBatchJob could grow the ownership table by one entry per
cancellation of an unknown ID, and the retry scheduler walks that table
every second.
Once discovery has finished, nothing admits a job this node does not own:
new submissions use new IDs, so the record has nothing left to fence.
canceler now records a canceled owner for an unknown ID only while
discovery is still running. Canceling an owned job, and the fencing of a
cancellation during discovery, are unchanged, and the cancel handler
still removes the manifest.
TestBatchCancelUnknownJobAfterDiscovery cancels an unknown ID during
discovery and requires its admission to be refused, then cancels a
thousand unknown IDs after discovery and requires the ownership table
not to grow. Before this change it held a thousand records.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Deleting a policy checks that no user or group mapping names it, but not
the STS mappings of OpenID parent users and LDAP user DNs. After such a
deletion the request path (MergePolicies) skips the confirmed-absent
policy and decides on the remaining mapped policies, while batch
authorization treated the absent document as a denial of the whole
action: jobs of such a submitter ended refused, although the same
principal's requests were allowed.
batchLocalPolicies now skips a policy document whose load reports that it
does not exist, as MergePolicies does, and still decides on the remaining
policies. Any other failure to read a document, and a load that returns
no document, still ends the attempt as unavailable, since it could hide a
Deny. With every mapped policy absent nothing allows the action, so the
job is refused as before.
TestBatchAuthorizeSkipsDeletedMappedPolicy maps an STS parent user to two
policies, deletes one, and requires the batch decision to agree with the
request decision, allowed with the other policy and denied once both are
deleted. Before this change the batch decision was a refusal while the
request was allowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Batch authorization evaluates the submitter's mapped policies from one IAM
cache revision. It took the cache write lock for the whole evaluation and
performed the lazy loads it needed under that lock: the mapping of a user
whose policies come from a group is never cached, so every authorization
of such a submitter (and of an LDAP or STS parent user) read storage twice
with the write lock held, and an evicted policy document was read under
it too. While a read lasted, every reader of the cache waited: request
authentication (GetUser, CheckKey) and PolicyDBGet on the whole node. An
expire job authorizes each version several times, so a large job with a
group-mapped submitter serialized node authentication behind IAM storage
latency, and a mapping load had no deadline at all.
batchPolicySnapshot now holds the store's writer mutex and the cache read
lock, and releases the read lock around each storage read, as
MergePolicies releases the cache lock around a document load. The writer
mutex still excludes cache writers, so the mapping and the documents of
one evaluation still come from one revision; the read lock is enough
because the batch path writes nothing to the cache except lazily loaded
mappings, which the request path stores under the read lock as well, and
taking the write lock back after each read would have made readers queue
behind it. Each read is bounded by the job's context and the same 5s
MergePolicies grants a document load. Concurrent reads of the group
fallback release the lock once and take it back after the last.
policyDBGet and policyDBGetGroups take the loads they need through the
iamMappingLoader interface, which *IAMStoreSys satisfies unchanged.
Unreadable policies still end the attempt as unavailable, and the error
classification is unchanged.
TestBatchAuthorizeSlowLoadKeepsCachedAuthorizationAvailable holds the
lazy load of a group-mapped user's mapping, and of an evicted document,
and requires that GetUser and IsAllowed of a cached user return within a
second and that a load ends with the job's context. Before this change
both cases timed out on the blocked cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestDynamicWebhookOutlivesConfigContext (0fa4a8c50) exercises the
goroutine lifecycle of webhook, audit and Kafka targets across a peer
config reload. It ran in CI only in the plain test-cmd job, outside
every race group. Add a race step for it next to the other lifecycle
groups; it takes about 35s locally under -race.
The dev build tag in make test, make install-race and several race
steps selects nothing: no file in this repository or its dependencies
carries a dev build constraint, and kqueue only changes the notify
backend on BSD and macOS. Say so in the Makefile and the workflow
instead of removing the tag, so that the documented command lines keep
working. No other job changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A RenameData that fails after its caller gave up removes its empty
object directory and the empty parents (ed05d982a, 73025824a). Only
the two renames of RenameData itself survived that cleanup, through
renameDataAll. Any other write of the same key that creates the object
directory failed on the drive when the cleanup removed the directory
between the directory creation and the open: WriteMetadata for a
delete marker on an absent key or for a first version, through
openFile, returned the raw open error, and writeAllDirect returned
errFileNotFound. The version was then missing on that drive, and
DeleteObject queues only offline drives for MRF. Before ed05d982a the
failure path never removed the directory, so this window is new.
Move the retry policy of renameDataAll into the shared helpers. A write
below the volume directory retries once per path level when the
directory creation or the use of the directory reports "not found"
(retriesAgainstCleanup): a cleanup removes each directory at most once
and each failed attempt needs one such removal. reliableRename applies
it in place of its blind single rename retry, which never recreated the
directory, and only while the source still exists, so renameDataAll is
replaced by renameAll. openFileAll applies it to the directory creation
and open shared by openFile and writeAllDirect. mkdirAllErr is split
out of mkdirAll, like renameAllErr, so that the retry decision uses the
OS error: on unix, mkdirAll maps a vanished parent to
errFileAccessDenied through isSysErrPathNotFound.
Invariants kept: the cleanup still removes only empty directories and
never a committed xl.meta or data directory; the error mapping of
mkdirAll, renameAll, openFile and writeAllDirect is unchanged, including
the Windows isSysErrPathNotFound branch; retries are bounded by the
path depth; a rename whose source is gone still fails at once.
UpdateMetadata renames over an existing xl.meta, so its directory is
never empty and the cleanup cannot remove it; it needs no change.
TestXLStorageMetadataWriteSurvivesLateCleanup injects, right after the
writer's os.Mkdir of the object directory, either a late RenameData
whose staged data is gone or a cleanup that walks up the whole path one
directory per step, for a delete marker, a first version and a large
xl.meta. Before the fix all six cases failed with "no such file or
directory" or "file not found".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of the delete-all reporting found that no
committed test checked the error a purge returns on a drive whose
metadata does not decode: restoring the old read quorum error passed.
TestPurgeAllVersionsRefusesCorruptDrive now sends the purge through
DeleteObject and checks errDeleteAllHistoryUnavailable and its
SlowDownWrite mapping; with the old error it fails.
The HELP text of expiry_incomplete_total now says that the
tier_unsupported_skipped reason counts the scanner's per-drive
observations, not expiry attempts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of this round's public fixes found one gap in
the code and four in the tests.
- A refused expiry was traced with its reason but without its error,
so the trace did not say what refused it. It now carries the error, as
a failed one does.
- The restore MRF test replaced the live MRF channel, which the heal
routines of the test servers read: a race, and a consumer could take
the entry under test. addPartialOp now offers an atomic hook that
receives the operation in place of the queue. Production never sets
it.
- Two fixtures restored the storage class with Update, which marks an
uninitialized configuration initialized and so changes the default
parity (-1) into an explicit 0 for later tests.
setTestStandardParity restores the whole saved value under
storageclass.ConfigLock.
- The v3 metric test built its own descriptor map, so dropping the
counter from the registered ILM collector still passed. It gathers
through the collector the handler registers.
- The audit test read one extra entry or waited 500 ms. It drains every
entry of the bucket up to the removal audited last, and checks one
failure audit, one removal audit and no refusal audit per object.
Each corrected test fails with the corresponding mutation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions of a key purges exactly that key, and the
purge can refuse it or stop early. Those outcomes were hard to see, and
the scanner reported a key it had queued as if it were already gone.
- Refusals and failures of a purge are counted in
expiry_incomplete_total with their own reasons: tier_unsupported (a
version is in a remote tier), budget (stopped after progress; the next
expiry continues), changed (the latest version a batch matched
changed) and history_unavailable. They are traced with the reason and
send no removal event. A partial purge keeps its audit record, now
with the reason. A tier refusal is also logged, once per key and hour,
in words.
- A drive that cannot be read or decoded during the purge's own drive
check now returns errDeleteAllHistoryUnavailable, which names the
cause and stays a write quorum error (503) for API callers.
- The scanner does not queue an expiry of all versions whose latest
version is hot when the pool holds a version in a remote tier: the
purge would refuse it each cycle. It counts each such observation as
tier_unsupported_skipped, once per drive scanner and cycle. A tiered
version only another pool holds is unknown here, so the purge still
decides; a transitioned latest version keeps the transitioned expiry.
The tier predicate is shared with the purge's own check, for FileInfo
and ObjectInfo, and excludes free versions.
- The scanner accounts every version of a key under an expiry of all
versions at its size, queued or not, instead of as already removed.
Usage, and so hard quota, drops after a fresh scan of the subtree once
the purge succeeded. The alerts on many or large versions count the
versions of a key that is not queued.
- A lifecycle expiry never deletes a prefix: DeleteObject refuses
DeletePrefix with Expiration.Expire as InvalidArgument. No caller
sends it since expiries of all versions became purges; a force delete
of a prefix is unchanged.
Not done from the plan: queuing an MRF heal per version when a drive's
xl.meta fails to decode. Object heal does not rewrite metadata that
fails its CRC check (it reports the drive as unknown and skips it), so
those heals would not repair the drive. Such a key stays refused until
its metadata on that drive is repaired.
Tests: TestPurgeAllVersionsRefusalReasons,
TestPurgeAllVersionsRefusesCorruptDrive,
TestScannerSkipsDeleteAllForTieredKey and
TestPoolsDeleteObjectRejectsExpirePrefix are new. Tests that expected
the scanner to queue a refused purge, the predictive accounting, or
the old prefix options are converted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission stopped at the first bucket it could not finish. A free
version whose tier is no longer configured is enough: its remote object
cannot be removed, so the free version stays, the bucket's check for
versions left behind fails, and every bucket after it stayed in the pool.
decommissionInBackground now keeps the first error, continues with the
next bucket, and returns that error after the last one. A canceled
context still returns at once. A bucket is retired from the resumable
queue only when it is empty, as before, so the decommission still fails
and a retry resumes with the buckets that are left.
removeFreeVersions now names the free version, its remote object and the
tier when the tier is not configured, and says to configure the tier
again under the same name. The free version is the only record of its
remote object, so it is kept; nothing is dropped on inference. Until the
tier is configured again, such a free version blocks the decommission
of its pool.
Tests: TestPoolsDecommissionContinuesPastStuckBucket and
TestRemoveFreeVersionsMissingTierError fail on the base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of the lock fixes found two gaps in the merge
test of lockObjects:
- Where the sets share one local table, it only tried an alias of the
clean name's own set, so dropping the shared-table normalization of
the lock domain still passed. It now tries an alias of another set in
every mode; with that mutation the shared case times out.
- In a split distributed domain it probed readers before writers. The
reader probe releases asynchronously and could hold off the writer
probe right after it, so a missing alias read lock passed some runs.
Writers are probed first now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore commit that reaches a write quorum but not every drive
queues an MRF heal of the version. It passed the raw version ID, which is
empty for the null version, while the convergence path passes the
normalized "null". The heal boundary already reads an empty ID as the
null version, so the old entry healed the right version; this makes the
two paths queue the same identity.
The header of restore-capability.go still described the multipart upload
of the first native-restore design. It now describes what the capability
gate protects in the two-phase design: restore ownership through the
restore lock and the attempt marker, the staged commit under the object
lock, and the attempt fence. It also says why an unreachable server
blocks native restore: it is not proven stopped.
Test: TestRestorePartialCommitQueuesNullVersionHeal fails one drive's
commit on a noncurrent null version, checks the queued entry names
"null", and heals it back onto that drive. It fails on the base at the
identity check only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An ILM expiry that is refused, because the rule no longer applies when
it is checked again under the object lock or Object Lock protects the
version, or that fails, for example for lack of quorum, left almost no
trace. The bulk noncurrent purge skipped such entries without audit,
trace, log or count; the scanner traced a refusal exactly like a
successful expiry; and nothing counted either. Operators could not see
why a version they expected gone was still there.
- Count every such expiry by reason in expiry_incomplete_total
{reason="not_applicable"|"locked"|"history_unavailable"|"error"}, in
metrics v2 and v3; the counting unit is one expiry attempt, and no
bucket or object is a label.
- Trace it with result=refused or result=failed and the reason. A
failure is also audited and logged once per object; a refusal changes
nothing and is not audited, since a split key is refused on every scan.
- Never announce it: no ObjectRemoved notification for a version that is
still there.
- Name a completed expiry by what its delete did. An expiry that
addressed no version and whose result is a delete marker created that
marker, also with the null version ID of the version it replaced; any
other removed the version it addressed, also a delete marker. The
scanner and the transitioned expiry took the identity from the queued
object instead: the expiry of a latest transitioned version in a
versioned bucket announced the removal of the version it kept, and the
removal of an expired delete marker was announced as a marker created.
Tests: classification of each error; a bulk purge with a refused, a
failed and a removed entry counts and traces the first two, audits only
the failure and announces only the removal; the expiry events name the
created marker for a latest version (versioned, suspended null,
transitioned) and the removed version for a noncurrent version and an
expired delete marker; both metrics versions export the counter by
reason. The #278 side-effect tests now tell removal traces from
refusal and failure traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Earlier tests can leave a storage class of parity 0 behind (newTestConfig
on a single drive), and a version written without parity cannot be
rebuilt on the drive the tests heal. Set the parity of the test pools'
sets for the duration of each drive heal test, as other tests that
depend on parity do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The implementation review of the lock fixes found test flaws, which are
corrected here; the production behavior is unchanged.
- The alias and merge tests replaced globalOperationTimeout while
background healing reads it, a data race under -race. They now bound
each request with its own context instead.
- The alias test now holds the cleaned name's write lock while the copy
runs, so it proves the names are checked before anything is locked;
it checks both objects before any later write, and covers a/./x as a
destination.
- The merge test picks aliases that hash to the clean name's set and,
where sets have their own lockers, ones that do not, and requires the
alias's own table to hold a read lock: readers admitted, writers
excluded.
- Release polling creates a fresh lock instance for every probe.
- The getNSLocks event hook and the bulk delete group limit are atomic,
so tests can set them while other goroutines read them.
Without the name check and the resource merge, both tests now fail in
every lock layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission or rebalance walk prechecks each due version of a key
with the versions of the key in all pools, to decide whether the version
moves or is left to its ILM expiry. Each precheck read and merged the
key's whole history from every pool, so a key with V due versions read
it V times in the walk alone; with long histories this grew roughly
quadratically.
Read the history once per key into a cache that the walk keeps for the
entry, and evaluate every version's precheck against it. The precheck
only chooses between moving a version and leaving it to the expiry; the
expiry reads the history again under the object lock before it deletes
anything, so a stale cache can only move a version or defer it to a
recheck. A read error is kept, so every due version of that key moves.
The expiry's own per-version reads are unchanged.
Test: decommission and rebalance of a key with 63 due noncurrent
versions read its history once, with the same outcome; without the
cache they read it 63 times.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
BatchJobExpire.Start checked the job's submitter before it looked at
the job's report, and refuse() clears Complete. A job that had completed
was therefore turned into a failed one on the next start: one that
completed before the upgrade records no submitter, and one completed
with a temporary credential that has expired since, a deleted user or a
changed password fails the binding. Such a job asked to be submitted
again, inviting an operator to run a destructive expiration twice.
Return before the submitter check when the report says the job is
complete, as replication and key rotation jobs already do.
Tests: a completed job without a submitter, and a completed job whose
temporary credential expired, keep their completed state and delete
nothing when started again; a job that did not complete is still
refused once its credential expired. The first two fail without the
change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A version can have copies in two pools, for example after an interrupted
decommission or rebalance. DeleteObject left ILM expiries out of the
reconciled delete of an addressed version, so a lifecycle version expiry
deleted only the copy that latest-key routing chose. The other copy stayed
readable; for a tiered version, expireTransitionedObject then deleted the
remote object that copy still referenced; and a migration of the pool
holding it left the expired version behind and failed its completion
check. With delete replication on, the replication completion's own
delete had reconciled the copies and hidden this.
Admit an ILM version expiry that the pools layer approved under the lock
with the versions in all pools (CheckedAllPools, and not a restored-copy
expiry) to deleteObjectReconciled. It removes the secondary copies before
the primary and stops at the first failure, so the remote object is only
deleted after every copy is gone. Each set still guards the protection
the pools approved. Marker creation, restored-copy expiry, free-version
cleanup and delete-all keep their own paths; the bulk noncurrent purge
already reconciled.
Tests: with copies in two pools, a hot and a tiered version expiry remove
both copies, and the tiered one deletes its remote object once; when the
secondary copy cannot be deleted the expiry fails, both copies and the
remote object stay; with replication configured the tiered expiry queues
nothing; decommissioning the pool that holds the second copy completes
and leaves no expired version. All fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The migration follow-up for #265 (PR #276) made lifecycle-created
deletes replication-aware: the scanner, the expiry of a transitioned
version and the bulk noncurrent purge evaluated the delete replication
decision and queued a delete marker or a version purge for the targets.
That is the content of the still-open PR #240. It changes behavior for
every bucket whose replication rule replicates deletes or delete
markers, the mc default: a target with longer retention loses versions
on the source's schedule, and a purge keeps its space as a pending
tombstone until the target confirms. SILO, like S3, does not replicate
lifecycle deletes; an opt-in can come later. The release did not
replicate them.
Remove exactly the replication parts: the decision callbacks and the
post-delete scheduling in applyExpiryOnNonTransitionedObjects and
expireTransitionedObject, the scheduling in deleteObjectVersions, and
ilmPurgeToDelete; the scanner builds noncurrent entries as before. The
rest of #276 stays: the hot-tier delete before the remote one, the
free-version retry record, the lifecycle-change fence and the migration
completion checks. API deletes still replicate, and the replication heal
still handles persisted pending markers and purges. As in the release,
the replication heal also resyncs an empty-state marker when
existing-object replication applies to it.
The delete-all variant of the scheduling, which queued a task with an
empty decision, goes with it.
Tests: the scanner's noncurrent purge, run with replication configured,
deletes the version at once and queues nothing; an ILM marker carries
no replication state and the heal leaves it unless existing-object
replication applies, in which case it is resynced as before; an API
version delete still queues a purge. The cross-pool batch, dispatch and
decommission tests now assert no replication, and a decommission whose
expiry used to wait for the target completes in one run. Adding back the
scanner's decision makes the first two fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The single-pool lock-domain change made a bulk delete lock each set's
names in that set and ran the groups one after another: a batch over S
sets cost about S lock round trips plus S set deletes, where the release
took one lock and then let erasureSets.DeleteObjects work on all sets at
once. Every distributed single pool with several sets paid for it,
including layouts where every set uses every host.
Run the groups concurrently, up to 16 at a time. Each group takes only
its own set's lock and waits for no other while it holds it, so the
groups cannot hold each other. A group checks for cancellation once it
is admitted, so a queued group of a canceled request does no storage
work. Each result is written only by the group of its object.
Tests: the first drive calls of two sets must overlap, which fails when
the groups run one after the other; with one worker, a group in flight
at a cancel keeps its own result, an earlier group keeps its success,
and a queued group makes no drive call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
getNSLocks takes the locks of a copy in one order and waited for a later
lock, such as the source, for up to two seconds while it held the earlier
ones, such as the destination; then it released them, paused up to 250 ms
and tried again until the operation timeout. When the source was write
locked for long (a multi-pool PUT for its whole upload, a heal, a
transition, another copy onto it), the copy held its destination write
lock most of the time: reads and writes of the destination waited for
seconds, their own backoff rarely hitting the gaps. In the release a copy
that waited for its source held nothing.
After a bounded wait for an atomic lock times out while earlier locks are
held, release them, wait for that lock alone with a fresh instance until
the deadline, release it once it is free, and start again from the first
lock in the same order. Holding nothing while waiting forms no cycle, and
the next attempt still takes every lock in one order with fresh
instances. A non-atomic request may hold some of its own names while it
waits, so it keeps the bounded retries.
A test hook reports the bounded timeout, the start of the idle wait and
its end, so the tests coordinate on events rather than timing: a lock
before the contended one is free for others for the whole wait, a cancel
or the deadline ends the idle wait and leaves nothing held, a non-atomic
request never waits alone, and HEAD of a copy's destination succeeds
while the copy waits for its source. All but the non-atomic control fail
without the idle wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
CopyObject locks its source and destination before it reads the source,
and lockObjects merged two requests only when their raw names were equal.
A lock is taken on the cleaned name, so an invalid source such as a//x or
a/./x and the destination a/x asked for a read lock and a write lock on
one resource: the copy waited on itself until the operation timeout and
answered 503, holding the key's lock in short windows meanwhile. Before
the lock-domain change the invalid source was refused at once, when the
source was read. This reached single nodes, multi-pool deployments and
single pools whose sets share hosts.
- Check the source with checkGetObjArgs before any lock, the first check
and the first error the release reported. Lock the destination only
when its name is valid. An invalid destination fails at the copy's own
destination check before it writes, after a missing source (404) or a
failed source precondition (412), as before.
- lockObjects merges requests by the resource the lockers see, the
cleaned bucket/object, within one lock table: the object's set where
the sets of a pool have their own lockers, else one table. This keeps
aliases on a shared local table from conflicting and never merges
requests of different distributed domains.
Tests: aliased sources and destinations are refused with 400 at once in
every lock layout and two pools; an invalid destination keeps the 404
and 412 precedence; lockObjects merges an alias with its clean name. The
alias and merge tests fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Drive healing evaluates lifecycle on the versions of one pool. For a
version it found due it enqueued the expiry and skipped the heal. The
expiry is decided again under the object lock, with the versions in all
pools and the Object Lock state it reads then, and can be refused: a
newer version in another pool, an unreadable history, a changed rule,
or a delete-all the exact-key purge refuses. Such a version was never
healed onto the replaced drive and kept reduced redundancy for as long
as the refusal lasted. The scanner heals only versions without a due
action, so it did not repair it either.
Keep the enqueue and heal the version. A version that the expiry then
removes costs one heal. Heal and expiry exclude each other through the
object lock in the writers' lock domain, which the single-pool lock
domain change provides; a version removed first reads as not found and
is skipped, and heal rebuilds only from read-quorum metadata, so it does
not bring back a version deleted on a write quorum.
Tests: a version refused across pools is healed onto the replaced drive
and survives the expiry; an unversioned object due is healed and then
removed from every drive; a versioned latest version due is healed and
hidden by a marker; a stale minority is not recreated. The first three
fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Three restore tests record the options of a POST's background restore
with a stub and restore with them later: TestRestoreAttemptGenerationFencing,
TestExpireRestoredUsesQuorumGeneration and TestRestoreHealNeedsRestoreQuorum.
The options carry the restore lock that POST hands to its worker. The
handler releases that lock only after the stub returns, while the test
proceeds as soon as the stub has signaled. When the test's restore
checked the lock before that release, it kept the lock, and the release
then canceled its context: every drive read failed with "context
canceled" and the restore reported insufficient read quorum.
Under -race this failed 11 times in 60 runs of the three tests on
69a4f938a, in the worker and old-reset modes of the fencing test and in
the multi-pool expiry test.
A new helper, restoreCaptureOptions, releases the lock before it records
the options, so a later restore takes it again, as a queued worker whose
lock ended does. TestRestoreSupersededAttempt already does this. With
the helper, the three tests passed 60 runs under -race.
Production code does not change: the handler's goroutine runs the
restore and releases the lock after it, and a lock lost during a
restore stops it as intended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang