The ready probe turns healthy before storage and IAM load, by design, so
that distributed nodes can reach each other during startup. Point
startup ordering such as Compose service_healthy at the cluster check,
and keep publishing not-ready addresses for StatefulSet peers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A failed or canceled native restore cleared its ongoing marker under a
fresh one-minute deadline, while the restore itself could still hold the
object lock that the cleanup waits for. The request then stalled for
about a minute (60 s in TestNativeRestoreWithoutOuterLock and
TestRestoreRecoversStrandedMarker).
Try the cleanup for 100 ms and leave the rest to the bounded background
retries, which already exist for the pools path and now also serve the
single-set path. Each retry still takes the object lock afresh and clears
only the marker of its own attempt. The CI restore race step drops from
about 260 s to 37 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The release workflow keeps its Object Lock race gate aligned with the
matching step in go.yml. Add the replicated-delete and directory bulk
delete tests there too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Record RELEASE.2026-11-09T00-00-00Z (package 20261109000000.0.0) in the
changelog: the fixes since 20261026, Console v2.7.0, mcli 20261109 and
silo-pkg v3.16.0, upgrade notes and known issues. Point the README quick
starts at the new image, replace the 20261026 release section with the
20261109 upgrade notice in both languages, refresh the contributor roll
to 2026-11-09, and add a second security contact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Console v2.7.0 was cut again at 4e91ae506e70 with the review fixes that
followed 26ce922424f1. Select that commit; no other module changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A directory object always has the null version. DeleteObjects without a
version ID chose whether to read the object before it turned the version
into null, so with Object Lock and no replication or tiering it checked
retention against empty metadata and removed a directory object under
legal hold or retention. Choose the null version before the lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A trusted replica DELETE skipped the destination's retention check for
every request, not only for the delete marker it is meant to create. A
caller allowed s3:ReplicateDelete (root, or the replication account)
could purge a version under legal hold or unexpired retention, or name
a stored version in a delete marker, which replaces that version.
Skip the check only for a delete-marker creation, whose version is not
yet stored here, and make that request refuse to replace a protected
version with RefuseLockedOverwrite, as batch replication pulls do; the
pools layer applies it to the version in any pool. A replicated purge
goes through the normal retention check. A source purge that bypassed
GOVERNANCE retention is now refused by a target whose copy still
carries it; the source records the purge as failed and retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The 20261026 release moved mcImage.tag but left image.tag, which the
templates deploy, at 20260916 (#296). Point the server and mcli images
and appVersion at RELEASE.2026-11-09T00-00-00Z and bump the chart to
7.0.4.
The Helm check now reads the chart version instead of a fixed file name,
and fails when image.tag differs from appVersion, so a release cannot
move one without the other again.
Upgrades that pass --reuse-values keep the old image.tag; set image.tag
explicitly when upgrading such a release.
Fixes#296
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The lifecycle expiry and native restore race step takes about 260 s on
an M-series Mac against its 5 minute limit, and the job holding it was
already near 9 of its 15 minutes before that step was added. GitHub's
hosted runners are slower, so allow the step 10 minutes and the job 25,
as the other jobs have.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Exercise concurrent overwrite and delete while initiation persists metadata, and completion after intervening object mutations. Check that failed conditions preserve uploaded parts for retry. Include conditional multipart regressions in the CI race set and record the multipart fix from Denis Babchuk in the contributor history.
Signed-off-by: Feng Ruohang <rh@vonng.com>
batchPolicySnapshot holds the IAM store, a context and a mutex. Without
an ignore directive, go generate emitted msgp encoders for it that do not
compile, so make check-gen (and the check-gen step of CI) failed on the
generated batch-authz files. The committed generated files are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit a6571a53e9)
Evaluate conditional multipart initiation against the logical current object across pools while retaining pool routing and holding the object lock through upload placement.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Copy LICENSE, NOTICE and CREDITS from the same verified mcli archive as
the bundled binary into /licenses/mcli; the server's CREDITS covers only
the server's own dependencies.
Correct the mcli archive pins to the RELEASE.2026-10-26T00-00-00Z
checksums. The release workflow overrides them from the attested
checksum file, but a plain local build failed with "Published checksum
drift".
Add a Dependabot docker update so the digest-pinned bases in
Dockerfile.goreleaser and Dockerfile.distroless keep receiving fixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The distroless variant inherited Debian's CA bundle, which still contains
the two roots (Comodo AAA Services root, emSign ECC Root CA - G3) that the
classic image blocklists per RHSB-2026-012. Build the same UBI bundle in a
throwaway stage and replace the only file under /etc/ssl/certs with it.
Pin the busybox and distroless bases by digest like Dockerfile.goreleaser,
and gate the shipped file in test-release: the bundle must parse and must
not contain either blocklisted subject.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
70914759e refused an edit whose stored record no longer matched the
caller's view, but left the stale copy in the serving server's cache in
two cases: a service-account edit bound to a parent whose account was
deleted and created again for another parent, and a replicated edit older
than what is stored, which site replication acknowledges. That server
kept authenticating the old credential until a notification or the
periodic load.
Both refusals now drop the cache entry of the addressed kind, as a
refusal for an absent or deleted identity does: the service-account
closure reports such a record as gone (callers still get
errNoSuchServiceAccount), and editStoredIdentity also drops the entry on
errIAMStaleUpdate. The LDAP group refresh again treats a stale update as
a refusal that writes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
UpdateServiceAccount, SetUserStatus and UpdateUserSecretKey read the
identity from the serving server's cache, changed it and wrote it back
whole. On a server that missed a notification, an ordinary edit (a
description, a status, a secret) undid a rotation made through another
server and brought back a user or service account deleted meanwhile, with
its old secret.
Edits now go through iamIdentityEdit, inside the identity's cross-node
revision lock: the stored record is loaded and normalized as the loaders
do (access key from the path, claims from the token, a deprecated comment
for an empty description), the edit applies to it, and an absent or
deleted identity is refused and dropped from the cache of the server
that tried. The LDAP group refresh of a0de92fcb uses the same mechanism.
- The admin handler binds a service-account edit to the parent it
authorized: an account deleted and created again for another parent
meanwhile is not edited (parentUser in updateServiceAccountOpts;
replication does not set it).
- A replicated edit older than a deletion this site applied is answered
as stale, so site replication acknowledges it instead of retrying.
- SetUserStatus refuses a service or temporary credential as before, and
a refused edit drops only a cache entry of the addressed kind.
- A session-policy edit stores the claims of the token it signs; the
token is signed once.
- UpdateUserSecretKey addressed to a service-account key finds no user
instead of writing the account to the regular-user path.
Tests use a second IAM store over the same storage whose cache missed the
change: rotation, deletion, status and secret edits of users, parent
binding, legacy records, replicated edits around a deletion, and status
calls on a service account. Against the cache-based writers every case
fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The hourly LDAP group refresh, which every server runs, reads the STS and
service-account credentials from its own cache, queries the directory,
and wrote each credential whose groups changed back whole through
UpdateUserIdentity. A secret rotation or an edit made meanwhile, on this
server or one whose notification it missed, was undone, and a credential
deleted meanwhile came back enabled with its old secret.
RefreshUserGroups replaces that writer, whose only caller was the
refresh. Under the identity's cross-node revision lock it loads the
stored record, refuses one that is absent, deleted, expired or now has
another parent (whose groups the lookup did not compute), and otherwise
changes only the groups of the stored record. A record without its
access key gets it from the path, as the loaders do. A refused refresh
drops the stale entry from the cache, which loads what is stored on next
use; a written or unchanged one caches the stored record. An STS record
keeps the lease of its token's remaining lifetime.
Tests cover rotation and edit, deletion, a sibling server whose cache did
not hear of a rotation or a deletion, recreation under the same and under
another parent, unchanged groups, and a stored record without its access
key. Written back as the old snapshot write, every case fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestBatchExpirePurgeRefusesTieredKey marks a version transitioned through
DecomTieredObject, which takes its parity from the global storage class.
Run after tests that leave a parity the one-drive fixture cannot have,
it wrote on no drive and still returned nil, and the purge then removed
the key. The helper now sets the set's own parity for the test and reads
the transition back from every drive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The submitter binding compared the UpdatedAt of the identity record, so
any rewrite of the record ended every running job of the credential with
"changed since submission": a service account's description, name or
later expiry, and the hourly LDAP group refresh of every LDAP-backed
service account and STS credential whose groups changed.
Compare only what identifies the credential: its type, access key and
parent, the revocation boundary of the record, and the keyed binding of
its secret and session token. Status, expiry, groups and policies are
read again on every start and action. A deletion advances the
revocation boundary, so a user created again under the access key with
the same secret still fails the binding, and adding an expiry to a
service account turns it into a temporary credential, which still ends
the job. UpdatedAt stays in the binding for diagnosis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch job is authorized without a request, so a policy statement that
depends on request context the job does not have (source address,
transport, headers) was taken to refuse the action. Two parts of that
rule refused jobs that no request would refuse:
- The scan for unsupplied policy variables ran over the whole statement
before its resources were matched. A Deny on an unrelated bucket whose
condition value held a variable such as ${aws:UserAgent} refused every
job of the user. Resources are now matched first; a variable in a
resource still means the statement may apply.
- An Allow that needs absent context, such as an IP-restricted read,
refused the job even when another Allow granted the action. Such an
Allow now grants nothing to the job: it is dropped from the policy that
is evaluated, not merely exempted, since IfExists and Null conditions
hold for an absent key and would grant. Dropping an Allow can only
narrow the decision. A Deny that needs the context still refuses.
The narrowed policy is a new policy.Policy, because the statement index
of the original points into its own statements and the original may be a
cached document. For a session policy, whether one applies and its first
decision still come from the claims as before; only an allow is
re-evaluated on the narrowed policy.
Tests cover an unrelated Deny with a variable, a Deny with an unsupplied
variable in its resource, an IP-restricted Allow next to a plain one, a
Null Allow that plain evaluation without the address would grant, and
the same for session policies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
batchPolicySnapshot holds the IAM store, a context and a mutex. Without
an ignore directive, go generate emitted msgp encoders for it that do not
compile, so make check-gen (and the check-gen step of CI) failed on the
generated batch-authz files. The committed generated files are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The cache read lock is reached through the embedded IAMStorageAPI, so the
explicit selector is redundant for staticcheck (QF1008). Drop it and say
in a comment that rlock and runlock are the cache's locks, unlike
IAMStoreSys.lock, which also takes writeMu. No behavior change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Three safeguards of the lifecycle and tier cleanup paths had no test
that failed when they were removed:
- authoritativeHistory accepts a revision only on more than half of the
drives. TestPoolsLifecycleHalfDriveMarker holds a delete marker on
exactly half of sixteen drives, which is the read quorum of a marker,
and checks that the history is refused, that the expiry of the
version under it and of the key are refused and every drive keeps what
it has, and that one more drive decides. With the guard removed the
strict merge at the same majority still refuses the history; with the
merge relaxed as well the test fails, so it pins the refusal, not one
of its two layers.
- deleteAllApproved refuses a purge whose placement read differs from
the versions it approved. deleteAllBeforePlacement is a test-only
injection point between the two reads, nil in production.
TestPurgeAllVersionsRefusesChangedApproval writes a new version and
rewrites the latest one there, as a writer that bypasses the object
lock would, and checks the write quorum refusal, the
history_unavailable count, that nothing is deleted and that the next
purge removes the key. With the check disabled the purge deletes the
versions it never approved.
- readTierHistory refuses the ownership of a remote object while a
drive of any pool cannot be read, and xlMetaV2.DeleteVersion deletes
a free record only with the remote tuple it names.
TestTierOwnershipReadFailsClosed fails one drive of the pool that
holds nothing of the key as offline, not found, faulty and
unformatted, and checks that the expiry worker's and the
decommission's cleanup are refused with a read quorum error and
succeed once the drive reads again.
TestFreeRecordDeleteNeedsMatchingTuple deletes a free record with
another tier, remote object and remote version, in memory and on an
xlStorage drive, and checks the record stays until the tuple matches.
With either check removed the remote object or the record is deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
GET and HEAD recheck a local expiry decision against the versions of
the object in all pools before they hide the version or queue its
expiry. When that recheck failed, because the history of another pool
could not be read, or found that the rule no longer applies with the
versions in all pools, the read was served and nothing was queued,
without a trace or a count. During a degraded pool a version that is
due kept reading and never reached the expiry workers, and the operator
could learn of it only from the scanner's own counts.
recheckReadExpiry now reports such a read as the scanner reports an
expiry it did not complete: it counts expiry_incomplete_total by the
reason ilmExpiryNotDone gives the error, history_unavailable for a read
quorum failure, not_applicable when the version is no longer due, and
publishes an ILM trace with result=skipped and that reason. The reasons
are the existing label values; the response, the version read and the
decision not to queue are unchanged.
TestPoolsLifecycleReadHTTP now asserts the counter delta and the trace
of every case the recheck skips; before this change it counted nothing
for the unreadable-pool and the not-applicable cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A purge of all versions reads the history of the key in every pool
before it checks the drives. When half of the drives of a set could not
be read or decoded, objectVersions refused the history with a read
quorum error and the purge failed with InsufficientWriteQuorum before
deleteAllDrives, which alone queued a heal of each version when a drive
did not decode. With parity of half the drives, a key whose metadata
did not decode on exactly half of them still read and healed, but no
purge queued that heal: the refusal repeated until a scanner or drive
heal happened to repair the drives.
objectVersions now refuses such a history with undecodableHistory,
which wraps the read quorum error and carries the IDs of the versions
the decodable drives hold, the null version by its explicit ID. The
purge queues a heal of each of them before it fails, from the history
of the rule check as from the placement read, and still returns
InsufficientWriteQuorum. The threshold of unknown drives, the refusal
and every other caller of the history are unchanged; expiryApplies
matches the read quorum error through the wrapper.
TestPurgeAllVersionsHealsUndecodableHistory corrupts the metadata CRC
on four of eight drives; before this change the purge refused the key
without queuing any heal.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The native restore gate refused a POST restore and a queued restore
with one message, "unavailable during a rolling upgrade: not every
server of the cluster supports it yet", whether a server answered
another protocol or was simply not connected. A cluster with one server
down thus reported a rolling upgrade, and the operator looked for an
older binary instead of the server.
The gate now returns an error that wraps errNativeRestoreUnavailable
and says why: errNativeRestorePeerUnreachable names the server that is
not connected ("server host:port is not reachable, so it cannot be
shown to have stopped an older restore"), errNativeRestoreUnsupported
keeps the rolling-upgrade text for a server that reports another
protocol or reconnected while it was asked, and a check that its
deadline ended carries that error. The POST handler maps the error to
503 ServiceUnavailable with the message as before, the worker still
clears its marker and stops, and errors.Is(err,
errNativeRestoreUnavailable) holds for every refusal. The gate remains
closed in every case it was closed before.
TestNativeRestoreRefusalReasons posts a restore with a fake server that
is not connected and one that answers protocol 1; before this change
the first answered 503 with the rolling-upgrade text and without "not
reachable".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission walk removes a restored copy that is past its restore
deadline before it moves the version, as the scanner would. Since
0a546c7a7 a refused removal made filterLifecycle return the
DeleteRestored action: the walk counted the version as decommissioned
without moving it and without queuing an expiry, removeMovedVersions
then found a source copy with no retained copy and failed the key with
PreConditionFailed, the moved versions of the key stayed duplicated in
both pools, the bucket stayed queued and the decommission ended Failed;
the retry after a restart failed the same way while the refusal held.
The walk now moves the version like any other when the removal is
refused, as the release before 0a546c7a7 did: a copy past its deadline
reads as remote, so the version moves with its tier metadata and its
restore deadline through DecomTieredObject, the remote object stays,
the source copy retires with the key once the destination holds it, and
the scanner of the destination pool removes the restored copy by that
deadline. The rebalance walk never reaches this action for such a
version, which it skips as remote; its filterLifecycle now names the
same decision so neither walk counts a restored-copy expiry as a
deletion it performed.
TestDecommissionMovesRestoredCopyPastDeadline refuses the removal on
every drive of the set; before this change it failed with complete=false
failed=true, pool 0 holding the version and pool 1 nothing.
TestRebalanceLeavesRestoredCopyPastDeadline is the control for the
rebalance walk.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions purges each key as a whole,
and the purge refuses a key when one of its versions is transitioned to
a remote tier, as it refuses a key with a version Object Lock protects.
The job counted the Object Lock refusal as a refused key and completed,
but counted the tier refusal as failed versions and ended Failed, on
every run, with objectsFailed set to the versions the walk saw. ILM
expiry reports the same tier refusal as refused, not failed.
expireAllVersions now counts a key refused for a transitioned version as
refused, as it does a locked key: nothing was removed, the attempt is
not retried, and the job completes with keysRefused, which mcli reports.
A later run of the job checks the key again. A key whose history cannot
be read still fails the purge as before.
TestBatchExpirePurgeRefusesTieredKey runs a purge of a key with a
transitioned older version and requires a complete job with one refused
key and every version kept; TestPurgeAllVersionsRefusesTieredKeys now
requires refused keys rather than failed objects. Before this change
the job ended Failed with two failed objects.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Canceling a job ID this node does not own recorded a canceled owner so
that startup discovery, which may still admit the manifest of that ID,
would not run a job that was canceled before the node had loaded it.
The record was kept for any ID, at any time, and never removed: a caller
with admin:CancelBatchJob could grow the ownership table by one entry per
cancellation of an unknown ID, and the retry scheduler walks that table
every second.
Once discovery has finished, nothing admits a job this node does not own:
new submissions use new IDs, so the record has nothing left to fence.
canceler now records a canceled owner for an unknown ID only while
discovery is still running. Canceling an owned job, and the fencing of a
cancellation during discovery, are unchanged, and the cancel handler
still removes the manifest.
TestBatchCancelUnknownJobAfterDiscovery cancels an unknown ID during
discovery and requires its admission to be refused, then cancels a
thousand unknown IDs after discovery and requires the ownership table
not to grow. Before this change it held a thousand records.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Deleting a policy checks that no user or group mapping names it, but not
the STS mappings of OpenID parent users and LDAP user DNs. After such a
deletion the request path (MergePolicies) skips the confirmed-absent
policy and decides on the remaining mapped policies, while batch
authorization treated the absent document as a denial of the whole
action: jobs of such a submitter ended refused, although the same
principal's requests were allowed.
batchLocalPolicies now skips a policy document whose load reports that it
does not exist, as MergePolicies does, and still decides on the remaining
policies. Any other failure to read a document, and a load that returns
no document, still ends the attempt as unavailable, since it could hide a
Deny. With every mapped policy absent nothing allows the action, so the
job is refused as before.
TestBatchAuthorizeSkipsDeletedMappedPolicy maps an STS parent user to two
policies, deletes one, and requires the batch decision to agree with the
request decision, allowed with the other policy and denied once both are
deleted. Before this change the batch decision was a refusal while the
request was allowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Batch authorization evaluates the submitter's mapped policies from one IAM
cache revision. It took the cache write lock for the whole evaluation and
performed the lazy loads it needed under that lock: the mapping of a user
whose policies come from a group is never cached, so every authorization
of such a submitter (and of an LDAP or STS parent user) read storage twice
with the write lock held, and an evicted policy document was read under
it too. While a read lasted, every reader of the cache waited: request
authentication (GetUser, CheckKey) and PolicyDBGet on the whole node. An
expire job authorizes each version several times, so a large job with a
group-mapped submitter serialized node authentication behind IAM storage
latency, and a mapping load had no deadline at all.
batchPolicySnapshot now holds the store's writer mutex and the cache read
lock, and releases the read lock around each storage read, as
MergePolicies releases the cache lock around a document load. The writer
mutex still excludes cache writers, so the mapping and the documents of
one evaluation still come from one revision; the read lock is enough
because the batch path writes nothing to the cache except lazily loaded
mappings, which the request path stores under the read lock as well, and
taking the write lock back after each read would have made readers queue
behind it. Each read is bounded by the job's context and the same 5s
MergePolicies grants a document load. Concurrent reads of the group
fallback release the lock once and take it back after the last.
policyDBGet and policyDBGetGroups take the loads they need through the
iamMappingLoader interface, which *IAMStoreSys satisfies unchanged.
Unreadable policies still end the attempt as unavailable, and the error
classification is unchanged.
TestBatchAuthorizeSlowLoadKeepsCachedAuthorizationAvailable holds the
lazy load of a group-mapped user's mapping, and of an evicted document,
and requires that GetUser and IsAllowed of a cached user return within a
second and that a load ends with the job's context. Before this change
both cases timed out on the blocked cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestDynamicWebhookOutlivesConfigContext (0fa4a8c50) exercises the
goroutine lifecycle of webhook, audit and Kafka targets across a peer
config reload. It ran in CI only in the plain test-cmd job, outside
every race group. Add a race step for it next to the other lifecycle
groups; it takes about 35s locally under -race.
The dev build tag in make test, make install-race and several race
steps selects nothing: no file in this repository or its dependencies
carries a dev build constraint, and kqueue only changes the notify
backend on BSD and macOS. Say so in the Makefile and the workflow
instead of removing the tag, so that the documented command lines keep
working. No other job changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A RenameData that fails after its caller gave up removes its empty
object directory and the empty parents (ed05d982a, 73025824a). Only
the two renames of RenameData itself survived that cleanup, through
renameDataAll. Any other write of the same key that creates the object
directory failed on the drive when the cleanup removed the directory
between the directory creation and the open: WriteMetadata for a
delete marker on an absent key or for a first version, through
openFile, returned the raw open error, and writeAllDirect returned
errFileNotFound. The version was then missing on that drive, and
DeleteObject queues only offline drives for MRF. Before ed05d982a the
failure path never removed the directory, so this window is new.
Move the retry policy of renameDataAll into the shared helpers. A write
below the volume directory retries once per path level when the
directory creation or the use of the directory reports "not found"
(retriesAgainstCleanup): a cleanup removes each directory at most once
and each failed attempt needs one such removal. reliableRename applies
it in place of its blind single rename retry, which never recreated the
directory, and only while the source still exists, so renameDataAll is
replaced by renameAll. openFileAll applies it to the directory creation
and open shared by openFile and writeAllDirect. mkdirAllErr is split
out of mkdirAll, like renameAllErr, so that the retry decision uses the
OS error: on unix, mkdirAll maps a vanished parent to
errFileAccessDenied through isSysErrPathNotFound.
Invariants kept: the cleanup still removes only empty directories and
never a committed xl.meta or data directory; the error mapping of
mkdirAll, renameAll, openFile and writeAllDirect is unchanged, including
the Windows isSysErrPathNotFound branch; retries are bounded by the
path depth; a rename whose source is gone still fails at once.
UpdateMetadata renames over an existing xl.meta, so its directory is
never empty and the cleanup cannot remove it; it needs no change.
TestXLStorageMetadataWriteSurvivesLateCleanup injects, right after the
writer's os.Mkdir of the object directory, either a late RenameData
whose staged data is gone or a cleanup that walks up the whole path one
directory per step, for a delete marker, a first version and a large
xl.meta. Before the fix all six cases failed with "no such file or
directory" or "file not found".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang