fix(pools): lock single-pool object writes in the object's own erasure set

In a single pool, DELETE in every form, bulk delete, the CopyObject
destination, SelectObjectContent, batch key rotation and the upload-ID
guards of complete, abort, get-info and list-parts locked through the
first erasure set, while PUT, multipart commit, PutObjectPart, tag,
retention, legal-hold and metadata updates, restore, transition and heal
lock in the set the object hashes to. When the sets of a pool use
different hosts, as every automatic layout of 18 or more hosts does, the
two locks ask different lockers and do not exclude each other, and
concurrent read-modify-writes of xl.meta can lose acknowledged versions
and delete markers or leave an object on a minority of drives.

Route these locks through NewObjectNSLock, which returns the lock of
the object's own set in a single pool and the pools lock otherwise.
Upload guards choose the set by their object and keep the name
object/uploadID. True prefix deletion and the locks of names that are
not objects keep their domain.

Bulk delete groups its names by set and takes one set's lock at a time,
in ascending set index; a failed group fails alone. A single node, whose
sets share one local lock table, keeps its one bulk lock.

The COPY handler takes its source and destination locks before it opens
the source, sorted, and transactionally: bounded waits while holding a
lock, release in reverse order and retry with new lock instances, one
deadline. It reads the source under them, passes NoLock to every nested
call, and runs that work in the locks' context. This also removes a
deadlock of copies in opposite directions wherever both locks were in
one domain, such as a single node or several pools.

Lock names and RPCs are unchanged, but a pool is protected only when
every node runs this change: affected pools must be upgraded and rolled
back with a drained restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
2026-11-05 03:31:03 +08:00
parent 7f38bf6282
commit 32ab506924
9 changed files with 2194 additions and 34 deletions
+6
View File
@@ -262,7 +262,13 @@ func (o *ObjectOptions) SetEvalRetentionBypassFn(f EvalRetentionBypassFn) {
// ObjectLayer implements primitives for object API layer.
type ObjectLayer interface {
// Locking operations on object.
// NewNSLock locks names in one stable domain. It is for names that are
// not objects, such as bucket and IAM metadata locks.
NewNSLock(bucket string, objects ...string) RWLocker
// NewObjectNSLock locks one object in the lock domain of its writes.
NewObjectNSLock(bucket, object string) RWLocker
// lockObjects locks several objects together, see erasureServerPools.
lockObjects(ctx context.Context, reqs ...objectLockRequest) (LockContext, func(), error)
// Storage operations.
Shutdown(context.Context) error