A version can have copies in two pools, for example after an interrupted
decommission or rebalance. DeleteObject left ILM expiries out of the
reconciled delete of an addressed version, so a lifecycle version expiry
deleted only the copy that latest-key routing chose. The other copy stayed
readable; for a tiered version, expireTransitionedObject then deleted the
remote object that copy still referenced; and a migration of the pool
holding it left the expired version behind and failed its completion
check. With delete replication on, the replication completion's own
delete had reconciled the copies and hidden this.
Admit an ILM version expiry that the pools layer approved under the lock
with the versions in all pools (CheckedAllPools, and not a restored-copy
expiry) to deleteObjectReconciled. It removes the secondary copies before
the primary and stops at the first failure, so the remote object is only
deleted after every copy is gone. Each set still guards the protection
the pools approved. Marker creation, restored-copy expiry, free-version
cleanup and delete-all keep their own paths; the bulk noncurrent purge
already reconciled.
Tests: with copies in two pools, a hot and a tiered version expiry remove
both copies, and the tiered one deletes its remote object once; when the
secondary copy cannot be deleted the expiry fails, both copies and the
remote object stay; with replication configured the tiered expiry queues
nothing; decommissioning the pool that holds the second copy completes
and leaves no expired version. All fail without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions authorized the versions its
walk saw, and the purge then deleted the versions it read under its
lock. A version written after the walk, or tags changed on one since,
was deleted without being authorized for the submitter.
The purge takes an authorization that a batch purge must set and no
other delete may. Under the object lock, once it has approved the
history and checked every drive, and before its first delete, it
hands the writes it is about to delete to the job: every copy of every
approved version and every revision of one that a drive holds, one per
distinct version ID and tags, as they are then. The job evaluates the
expire actions on each for the submitter. A denial deletes nothing and
ends the job as refused; the time it takes counts against the purge's
budget. A locked version still refuses the whole key, counted as a
refused key. The checks on the walk's snapshot are removed.
Reviewed adversarially; the review's regressions are included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Add ObjectOptions.RefuseLockedOverwrite. With it set, PutObject (before
the rename) and CompleteMultipartUpload check, under the destination's
write lock, the stored version the write would replace: the version it
addresses, or the null version. DeleteObject checks the stored version
that a delete marker it writes names; in a multi-pool deployment that
version is resolved across pools. A version under retention or legal
hold is refused with ObjectLocked. The check reads the version's own
metadata, whatever the bucket configuration, and returns read errors.
Nothing sets the option yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The unreleased native-restore fixes after RELEASE.2026-10-26T00-00-00Z
let the lock decide who owns a restore. Since fa42a0982 a single-pool
worker holds the object lock for the whole tier download, and since
7f38bf628 a busy lock means "a restore runs". 7f38bf628 also keeps the
restore's data in a marked multipart upload. Two regressions followed:
- While an old version restored, every GET, HEAD and write of the same
key waited for the download. On a single pool this is new; with
several pools the released code already held the lock that long.
- An ordinary upload whose completion failed after staging its metadata
became InvalidUploadID for every caller, and could not be aborted for
an hour. The marker that hid restore uploads could not tell them apart
once a completion had rewritten the upload's xl.meta.
A restore now runs in two phases:
- The worker erasure-codes the tier copy into the tmp volume of the
version's set, without the object lock, as heal does. A drive that
fails any part, or misses a part when CheckParts runs before the
commit, takes no part in the commit.
- A restore lock per version tells whether a restore runs. POST takes it
before the object lock and hands it to the worker. It is a liveness
hint: only the checks under the object lock decide what is written.
- The commit takes the object lock briefly, on one pool or several. It
reads the version again and checks the attempt's ownership, the
version identity, the tier copy, the part layout and the erasure
layout. It then publishes with RenameData drive by drive, never
through erasure renameData's undo, and needs the write quorum. The
replaced data directory goes only when every published drive reports
the same one.
- A publication that reached some drives but not the write quorum
converges under the same lock hold, in bounded retries after it: back
to the unrestored version on every drive that holds it, or to a
committed restore when the copy turns out to be on a write quorum.
Each pass first checks that every drive holds what the version held
before the commit, the commit wrote, or a convergence wrote, and stops
on anything else. It deletes no directory.
Multipart goes back to main: erasure-multipart.go and
bucket-object-lock.go are identical to 69fcf50cd. The commit lock, the
marker and the restore exceptions of multipart completion and of the
null-version Object Lock check are gone, and single-pool completion
again takes the upload lock first and the object lock right before it
commits. With several pools, abandoned-data cleanup now takes the pools
lock that writes take, as heal does, so it cannot remove a directory
that a commit moved in place and has not yet referenced. The restore
capability becomes protocol 2, and a queued worker checks it again.
Recovery after a coordinator crash during publication, and storage RPCs
that complete after their caller returned, stay excluded. Main has the
same exposure, and they are tracked separately.
Tests:
- New regression tests cover both problems. They fail on f85d4a5fa,
and the multipart one also passes on main.
- Fault-injection tests cover a publication below the write quorum,
including seven of twelve drives on 8+4; lost answers; convergence
retries and stops; the old data directory; a stale inline marker; a
recreated tmp volume; a published revision read across its expiry; a
second POST; a target deleted during the download; a tier 404; and
pools-locked abandoned cleanup.
- Thirteen tests of the upload marker from 7f38bf628 are dropped, and
so is TestNativeRestoreConcurrentCompletion from fa42a0982, which
completed the restore's upload from a client. Four tests are reframed
for the new semantics, including the lock-domain restore test, which
now gates the commit.
- Three self-recursive WriteMetadata test doubles are fixed, and the
helpers of the dropped tests are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
In a single pool, DELETE in every form, bulk delete, the CopyObject
destination, SelectObjectContent, batch key rotation and the upload-ID
guards of complete, abort, get-info and list-parts locked through the
first erasure set, while PUT, multipart commit, PutObjectPart, tag,
retention, legal-hold and metadata updates, restore, transition and heal
lock in the set the object hashes to. When the sets of a pool use
different hosts, as every automatic layout of 18 or more hosts does, the
two locks ask different lockers and do not exclude each other, and
concurrent read-modify-writes of xl.meta can lose acknowledged versions
and delete markers or leave an object on a minority of drives.
Route these locks through NewObjectNSLock, which returns the lock of
the object's own set in a single pool and the pools lock otherwise.
Upload guards choose the set by their object and keep the name
object/uploadID. True prefix deletion and the locks of names that are
not objects keep their domain.
Bulk delete groups its names by set and takes one set's lock at a time,
in ascending set index; a failed group fails alone. A single node, whose
sets share one local lock table, keeps its one bulk lock.
The COPY handler takes its source and destination locks before it opens
the source, sorted, and transactionally: bounded waits while holding a
lock, release in reverse order and retry with new lock instances, one
deadline. It reads the source under them, passes NoLock to every nested
call, and runs that work in the locks' context. This also removes a
deadlock of copies in opposite directions wherever both locks were in
one domain, such as a single node or several pools.
Lock names and RPCs are unchanged, but a pool is protected only when
every node runs this change: affected pools must be upgraded and rolled
back with a drained restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions (ExpiredObjectAllVersions, DelMarkerExpiration,
and batch expiration with the default purge) asked the pools layer for a
prefix delete. The pools layer rebuilt the options, so the request ran as
a recursive delete of the key's directory in every erasure set of every
pool. Object a/b is stored inside the directory of a, so every key under
a/ was removed with it, however new, with one audit entry and one
notification for a alone. The rebuilt options also skipped the lifecycle
recheck, so on a single pool a version written after the decision was
deleted too, and tiered versions lost their metadata without free
versions.
The expiry worker, the migration expiry and batch expiration now ask for
a purge with the in-memory ObjectOptions.DeleteAllVersions:
- DeleteObject hands it to the purge under the object lock, before the
all-pools check. ILM decides with the unchanged lifecycleEventsAllPools,
on a single pool too; batch needs the version its walk matched to still
be the latest. A second read of each pool's authoritative history
records every copy, older writes of the same ID included, and must
equal the approval. Anything else deletes nothing.
- Before deleting, the purge reads every drive of each set that holds a
copy. It refuses the key when a drive cannot be read, when any revision
of an approved version there is transitioned, or, with Object Lock,
when retention or a legal hold protects one. It then deletes on exactly
the drive handles it checked.
- Versions are deleted by ID with the drives' version delete, which only
removes the object directory once it is empty: oldest first, for each
version older writes before the copy reads choose, and the latest last,
only once every older version is gone from every pool. No marker,
replication state or free version is written. The purge stops at the
first error, repeats a quorum-successful delete on the drives that
missed it while the lock is held, and stops before the latest version
at a budget of 1000 versions or one minute, reads included.
- The ILM audit entry records how many versions were removed. A purge
that failed after it began to delete is audited as partial and sends
no removal event. Batch counts come from the purge; a key whose latest
version changed is not expired, and a failed delete marker fails the
job.
The option is rejected with conflicting options and by every layer below
the pools. Force delete and internal prefix deletes keep DeletePrefix.
Delete-all of a transitioned latest version keeps its current path, and
a key with tiered older versions is refused. On a single pool the purge
relies on the lock-domain fix for writer exclusion.
Reviewed adversarially in two design rounds and four patch rounds; the
last approved with nits, which are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The merged all-pool check from #279 decided on the versions that a read
quorum of each set merged non-strictly. A drive that missed one metadata
update could therefore supply the revision used for the decision, and a
partially unreadable set could drop a committed version from the history.
Both let an expiry delete data the rule did not cover, including a version
under legal hold.
- objectVersions accepts a revision only when max(n/2+1, the version's
metadata read quorum) drives hold it, identifies revisions by the header
plus a sha256 over a canonical, length-prefixed encoding of the
normalized metadata, filters non-accepted revisions before the strict
merge, and fails closed on omitted versions, too many unreadable drives,
or cancellation.
- Lock fields are read with the evaluator's key precedence. Cross-pool
copies that disagree on hold or retention must be strictly ordered by
valid timestamps; copies with different ModTime are separate writes and
an older protected write refuses the object. The set safeguard re-checks
protection and pending purges on its fresh read for checked version
deletes.
- Duplicate copies with semantically different replication or purge state
refuse the object.
- Batch expiry announces and audits only completed or already-absent
removals.
Reviewed adversarially over seven rounds; the final round approved with no
remaining findings in scope. All-versions expiry through deletePrefix and
tiered delete-all dispatch are tracked separately.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The POST handler marks a version ongoing before its background restore
takes the object lock, and only the restore clears that marker. Native
restore holds the lock through the tier download, so a restore of
another version of the same key waits behind it. Both restore entry
points gave up after one delete-operation timeout and returned before
any cleanup, leaving ongoing-request="true" with no owner. Every later
POST for that version was refused with 409 and nothing expired the
marker.
Retry the lock while the restore's context lives, with a dedicated
dynamic timeout so repeated waits do not lengthen delete timeouts.
Shutdown or a caller deadline still ends the wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Carry restore context through POST status reads, metadata copy routing, null-version Object Lock checks and multipart completion. A failed read must preserve the target and must not enqueue its remote object for garbage collection. Cover initial/status POST failures and metadata loss during single and multipart rehydration on real disks.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep the storage layer error precedence for missing buckets and uploads while retaining every real upload lock. Cover the alias with short deadlines and race-check the original complete-multipart suite in CI.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Preserve handler version context, resolve the requested version's pool, and
return on the initial read failure without writing metadata. Propagate owned
NoLock to single-part writes, multipart completion and failure cleanup in
the same patch. Preserve the original modification time on cleanup.
Route the restore status copy by version and allow internal multipart
rehydration only for a matching completed transition. Single-pool restores
own the metadata snapshot through download and cleanup. Keep public
completion in object-before-upload order, using the target set's object
lock for single-pool calls, so concurrent completion cannot invert locks.
Cover real signed POSTs, null/UUID and cross-pool identity, download failures,
multipart overwrite guards, Object Lock updates and concurrent completion.
This package depends on the restored-copy cleanup safety delivered in R2.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Wait for admitted data and metadata mutations, preserve overwrite backups during rollback, and reject rollback of a different generation. Report conflicting or unreadable pool metadata as read-quorum failures instead of exposing stale copies. Add barrier-driven local/RPC and multi-pool regressions for cancellation, self-copy, rollback and old-value reads.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Reuse deleteObjectReconciled for tiered bulk entries and unversioned single deletes. Its reference-aware cleanup avoids scheduling remote contents for garbage collection while another copy still needs them, including restored copies. Keep the normal local-data bulk path unchanged.
Extend existing tier GC regressions to bulk and unversioned deletes and include them in the multi-pool race CI step.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Resolve each addressed version under the shared write lock, keep bulk I/O parallel within cleanup and authoritative-deletion stages, and preserve replication response identity on errors. Treat reliable absence as successful physical cleanup on retries without accepting it for marker or replication-state writes.
Cover execution-time faults, partial cleanup recovery, mixed authoritative pools, response identity, null marker ordering, metadata uncertainty, cancellation, and concurrent PUT. Run the delete consistency regressions under race in CI.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Move solitary and null delete markers instead of dropping them, including
idempotent UUID marker retries. Check that a reader's source version still
exists before moving it, and preserve the established read winner for
conflicting timestamp ties.
Retire only the versions that were successfully moved while holding the
pooled object lock. Recheck the source and retained copies before cleanup,
so concurrent deletes and uncertain destination reads cannot revive data.
Count remaining markers when verifying decommission completion.
Add two/three-pool walker, history, purge, concurrent-delete, read-fault,
and timestamp-tie regressions.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Distinguish all-drive native absence from read uncertainty and confirm current pool targets before omitting a recovered name. Cover both existing single-pool writer lock domains, release before encoding or output, and preserve cache terminal errors without blocking the HTTP error response on remote publication.
Keep ordinary entries free of recovery I/O and locks. Bound the selective short lock-conflict wait, preserve existing retry intervals and quorum, and make retries cancelable with late-grant cleanup coverage. Restore the original raw listing callback and unchanged scanner/healing/migration callers now that recovery errors originate in the merger.
Add deterministic deletion, pool ordering, cache, lock-routing and cancellation regressions to race CI. Focused cmd/dsync/lsync race suites and verifiers pass; complete current-main integration and bare-binary performance validation are recorded in PR #239.
Refs: https://github.com/pgsty/silo/issues/218
Signed-off-by: Feng Ruohang <rh@vonng.com>
Continue PR #221 while retaining its original commits and authorship.
Route latest-only fallback reads through the namespace lock used by normal
writes, propagate terminal scan errors before cache completion, and retain
delete markers until the cross-pool merge has resolved them.
Recount narrowly eligible latest null-version metadata when mixed erasure
layouts prevent selection. Keep each EC group separate and preserve the
requested quorum, strict resolution, and version-listing behavior.
Add regression coverage for locking, terminal errors, cache completion,
delete markers, mixed EC layouts, and HTTP behavior; run focused race tests
in CI. Verified with make test, focused race tests, and local distributed
single- and multi-pool rolling-restart workloads with exact final oracles.
Refs: https://github.com/pgsty/silo/issues/218
Refs: https://github.com/pgsty/silo/pull/221
Signed-off-by: Feng Ruohang <rh@vonng.com>
Close SN-2026-015 by requiring PutObjectRetention with request-derived
condition keys before reading bucket or object state. Weakening unexpired
GOVERNANCE retention also requires the bypass header and permission. Treat
retention removal as zero remaining days and avoid duration saturation for
distant dates.
Close SN-2026-016 by requiring replication trust for caller-selected target
version IDs and checking persisted multipart targets and locked null
versions under the destination write lock, including across storage pools.
Preserve versioning transitions, trusted replication and batch Snowball
version IDs, and fail closed when lock state cannot be read.
Add authorization, concurrent lock-update, multipart transition, cross-pool,
configuration-failure and retention-boundary regression coverage. Validate
with make test, focused race tests and the original independent probes.
Developed with Codex (Astra).
Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit fe3935476b929e03193d7496a7448d760e635096)
Require write-quorum absence for missing purge retries, aggregate removed and already-absent votes only for physical purges, and resolve receiver purges by version across pools. Preserve marker metadata through creation and healing while retaining DELETE response semantics.
Add real-disk quorum, pool, callback, metadata, heal and outbound replication regressions. Synchronize capacity fixture installation and restoration with concurrent IAM readers.
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 22ba4d426677aa07470927fc349a43afd87e19b6)
Multi-pool PUT selected a destination by capacity and evaluated
If-Match/If-None-Match only against that destination's local object
state. An empty or stale destination could accept a stale ETag or
If-None-Match:* while another pool held the current object, replacing
it; a current ETag could instead be rejected with 412 or 404.
Under PUT's existing pools-layer object lock, resolve the comparison
object with objectPoolInfos (including draining pools), treat a latest
delete marker as absence, fail closed on unreadable pool metadata, and
clear an accepted callback before destination dispatch. Replica and
data-movement callbacks keep their addressed-version semantics and
metadata reconciliation.
Reproduced on 40220bd836 and RELEASE.2026-09-03T13-18-01Z with six
signed HTTP scenarios: four defect cases failed, two controls passed.
Refs #199
Signed-off-by: Feng Ruohang <rh@vonng.com>
Persist, send and reconcile empty tag states together with their revision
across COPY, PUT and multipart replication. Advance local tag mutations
under the existing locks and preserve current tags during replication ACK.
Cover signed HTTP, persistent single/multiple pool state, KMS, SSE-C key
rotation, ordering, retry and duplicate requests. Record real Opus plan
consensus, implementation review and local verification evidence.
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 115fe8b12329d147adbaf817faa1737392ecbf9b)
Signed-off-by: Feng Ruohang <rh@vonng.com>
Delete every copy of an explicitly addressed UUID, null version or delete
marker under the pool lock. Preserve retention and replication callbacks,
report unreadable pools and cleanup failures, and keep movement, incoming
replication, expiration and free-version cleanup on their existing paths.
Retain the separately developed general DELETE repair and replace its
access-mover-only coverage with a real interrupted rebalance copy followed
by HTTP deletion. Cover unqualified directory-marker DELETE and document
the existing pool-order-dependent 503 behavior that this makes consistent.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Reverse the first-parent diff of a3df317ae0,
including the feature branch compatibility and mover follow-up fixes.
Retain the independent multi-pool correctness fixes from #178 and migrate
their shared test fixture away from access-tier code.
Tolerate retired ILM keys and XML, read old v9 statistics while writing v8,
and document migration without moving objects or rewriting their metadata.
Include regression coverage using a historical scanner/writer v9 fixture.
Signed-off-by: Feng Ruohang <rh@vonng.com>
An independent adversarial review of the bucket metadata convergence
work found three defects it had introduced.
GetBucketInfo overwrote the physical creation probe with cached
metadata, which a bucket that never held a configuration legitimately
lacks. The new creation-time requirement then failed every policy, tag,
SSE, quota, versioning and Object Lock write on such a bucket, with no
operator recovery path, and initial synchronization skipped it silently.
Return the physical result unchanged when metadata is not requested, as
ListBuckets already does, recover the time during initial
synchronization, and pass it to MakeBucketHook so peers adopt the same
bucket generation.
Replication status compared parsed policies statement by statement while
heal compares the canonical key. An upgraded peer that stored an
equivalent statement order was therefore reported as mismatched forever,
and heal never had anything to write. Compare the key heal compares;
per-site presence counting is unchanged.
Heal diagnostics shared one log key across four conditions, so a real
peer RPC failure could be deduplicated away by an earlier message, and
they were logged at error level for the normal transient of a peer that
does not have the bucket yet. Give each reason its own key at warning
level, report only a field state that exists and still cannot be
ordered, and diagnose nothing when no site holds a state to propagate.
The recovery test now runs against the real ObjectLayer; the stub it
replaced returned the expected time and hid the defect. The policy
status test uses a statement order the canonical encoder reorders, and
adoption coverage is extended past a real field time.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang