Native restore chose its own single-pool lock in the object's erasure
set before the pools layer did. Now that every object write locks in
that domain through NewObjectNSLock, the restore lock is that lock.
Behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
In a single pool, DELETE in every form, bulk delete, the CopyObject
destination, SelectObjectContent, batch key rotation and the upload-ID
guards of complete, abort, get-info and list-parts locked through the
first erasure set, while PUT, multipart commit, PutObjectPart, tag,
retention, legal-hold and metadata updates, restore, transition and heal
lock in the set the object hashes to. When the sets of a pool use
different hosts, as every automatic layout of 18 or more hosts does, the
two locks ask different lockers and do not exclude each other, and
concurrent read-modify-writes of xl.meta can lose acknowledged versions
and delete markers or leave an object on a minority of drives.
Route these locks through NewObjectNSLock, which returns the lock of
the object's own set in a single pool and the pools lock otherwise.
Upload guards choose the set by their object and keep the name
object/uploadID. True prefix deletion and the locks of names that are
not objects keep their domain.
Bulk delete groups its names by set and takes one set's lock at a time,
in ascending set index; a failed group fails alone. A single node, whose
sets share one local lock table, keeps its one bulk lock.
The COPY handler takes its source and destination locks before it opens
the source, sorted, and transactionally: bounded waits while holding a
lock, release in reverse order and retry with new lock instances, one
deadline. It reads the source under them, passes NoLock to every nested
call, and runs that work in the locks' context. This also removes a
deadlock of copies in opposite directions wherever both locks were in
one domain, such as a single node or several pools.
Lock names and RPCs are unchanged, but a pool is protected only when
every node runs this change: affected pools must be upgraded and rolled
back with a drained restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The merged all-pool check from #279 decided on the versions that a read
quorum of each set merged non-strictly. A drive that missed one metadata
update could therefore supply the revision used for the decision, and a
partially unreadable set could drop a committed version from the history.
Both let an expiry delete data the rule did not cover, including a version
under legal hold.
- objectVersions accepts a revision only when max(n/2+1, the version's
metadata read quorum) drives hold it, identifies revisions by the header
plus a sha256 over a canonical, length-prefixed encoding of the
normalized metadata, filters non-accepted revisions before the strict
merge, and fails closed on omitted versions, too many unreadable drives,
or cancellation.
- Lock fields are read with the evaluator's key precedence. Cross-pool
copies that disagree on hold or retention must be strictly ordered by
valid timestamps; copies with different ModTime are separate writes and
an older protected write refuses the object. The set safeguard re-checks
protection and pending purges on its fresh read for checked version
deletes.
- Duplicate copies with semantically different replication or purge state
refuse the object.
- Batch expiry announces and audits only completed or already-absent
removals.
Reviewed adversarially over seven rounds; the final round approved with no
remaining findings in scope. All-versions expiry through deletePrefix and
tiered delete-all dispatch are tracked separately.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expiry worker now asks DeleteObjects to recheck ILM against all
pools. A rejected entry keeps its version and returns
errLifecycleNotApplicable, but the worker still wrote an ILMExpiry
audit record, an ILM trace and an s3:ObjectRemoved:Delete event for it.
Event delivery drops the error element, so consumers saw an ordinary
removal, repeated on every scan while the versions stay split across
pools.
Skip rejected entries; eligible entries of the same batch are still
reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The POST handler marks a version ongoing before its background restore
takes the object lock, and only the restore clears that marker. Native
restore holds the lock through the tier download, so a restore of
another version of the same key waits behind it. Both restore entry
points gave up after one delete-operation timeout and returned before
any cleanup, leaving ongoing-request="true" with no owner. Every later
POST for that version was refused with 409 and nothing expired the
marker.
Retry the lock while the restore's context lives, with a dedicated
dynamic timeout so repeated waits do not lengthen delete timeouts.
Shutdown or a caller deadline still ends the wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep the specialized scanner routing test aligned with the explicit lifecycle rejection result, and verify the addressed version and payload remain readable. This distinguishes refused expiry from an idempotent missing version.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Carry restore context through POST status reads, metadata copy routing, null-version Object Lock checks and multipart completion. A failed read must preserve the target and must not enqueue its remote object for garbage collection. Cover initial/status POST failures and metadata loss during single and multipart rehydration on real disks.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Merge lock and tag revisions for duplicate versions before all-pool
lifecycle evaluation using the existing metadata reconciliation rules.
Distinguish ordinary lifecycle refusal from object absence and clear
rejected batch replication state so a late policy withdrawal cannot
schedule a purge. Preserve restored-copy cleanup and real absence.
Cover single and batch protections, legitimate metadata removals, and
actual replication queue and HTTP target behavior.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Reject replacement markers and legacy objects before restored-copy cleanup
can mutate their type or data. Prevent restore lookups and failure cleanup
from purging dangling versions, preserve Transition on metadata rereads,
and return without writing when the initial restore read fails.
Add real-disk regressions for stale null tasks, minority metadata, initial
and cleanup read failures, and stored marker/legacy types. Keep ordinary
dangling cleanup and marker purge as positive controls.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep the storage layer error precedence for missing buckets and uploads while retaining every real upload lock. Cover the alias with short deadlines and race-check the original complete-multipart suite in CI.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Preserve handler version context, resolve the requested version's pool, and
return on the initial read failure without writing metadata. Propagate owned
NoLock to single-part writes, multipart completion and failure cleanup in
the same patch. Preserve the original modification time on cleanup.
Route the restore status copy by version and allow internal multipart
rehydration only for a matching completed transition. Single-pool restores
own the metadata snapshot through download and cleanup. Keep public
completion in object-before-upload order, using the target set's object
lock for single-pool calls, so concurrent completion cannot invert locks.
Cover real signed POSTs, null/UUID and cross-pool identity, download failures,
multipart overwrite guards, Object Lock updates and concurrent completion.
This package depends on the restored-copy cleanup safety delivered in R2.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Recheck only local ordinary-delete candidates that need an all-pool lifecycle decision. Match UUID/null identity to the version actually read and queue its global metadata and event; retain a successful read on recheck failure, absence, or refusal.
Keep GET reader cleanup, authorization, proxy reads, single-pool reads, restored-copy cleanup, and expiry response headers on their existing paths. Cover signed GET/HEAD, queue contents, metadata faults, missing matches, no-scan boundaries, and reader-lock release.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Merge maintainer follow-up for lifecycle-aware pool migration.
Preserve replication, tier cleanup, lifecycle-generation, and whole-object completion semantics on top of contributor fix#266.
Feng Ruohang