fix(pools): lock single-pool object writes in the object's own erasure set

In a single pool, DELETE in every form, bulk delete, the CopyObject
destination, SelectObjectContent, batch key rotation and the upload-ID
guards of complete, abort, get-info and list-parts locked through the
first erasure set, while PUT, multipart commit, PutObjectPart, tag,
retention, legal-hold and metadata updates, restore, transition and heal
lock in the set the object hashes to. When the sets of a pool use
different hosts, as every automatic layout of 18 or more hosts does, the
two locks ask different lockers and do not exclude each other, and
concurrent read-modify-writes of xl.meta can lose acknowledged versions
and delete markers or leave an object on a minority of drives.

Route these locks through NewObjectNSLock, which returns the lock of
the object's own set in a single pool and the pools lock otherwise.
Upload guards choose the set by their object and keep the name
object/uploadID. True prefix deletion and the locks of names that are
not objects keep their domain.

Bulk delete groups its names by set and takes one set's lock at a time,
in ascending set index; a failed group fails alone. A single node, whose
sets share one local lock table, keeps its one bulk lock.

The COPY handler takes its source and destination locks before it opens
the source, sorted, and transactionally: bounded waits while holding a
lock, release in reverse order and retry with new lock instances, one
deadline. It reads the source under them, passes NoLock to every nested
call, and runs that work in the locks' context. This also removes a
deadlock of copies in opposite directions wherever both locks were in
one domain, such as a single node or several pools.

Lock names and RPCs are unchanged, but a pool is protected only when
every node runs this change: affected pools must be upgraded and rolled
back with a drained restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
2026-11-05 03:31:03 +08:00
parent 7f38bf6282
commit 32ab506924
9 changed files with 2194 additions and 34 deletions
+6 -3
View File
@@ -265,9 +265,12 @@ func (z *erasureServerPools) listPathShortcutEOF(ctx context.Context, bucket str
}
// listingRLock covers the lock domains used by normal same-key mutations.
// Multi-pool writes use the pools lock. In a single pool, DELETE/COPY use the
// first set while PUT and metadata updates use the destination set, so recovery
// takes both in that order. No writer routing or wire protocol changes here.
// Multi-pool writes use the pools lock. In a single pool every object write,
// DELETE and COPY included, locks the object's own set (see NewObjectNSLock),
// which is the source set here. Releases before that routing locked
// single-pool DELETE and COPY in the first set, so recovery still takes the
// first set too, in that order, while nodes of those releases may run. It can
// drop that lock only behind a version boundary that keeps such nodes out.
func (z *erasureServerPools) listingRLock(ctx context.Context, bucket, object string, source *erasureObjects) (LockContext, func(), error) {
firstSet := z.serverPools[0].sets[0]
first := z.NewNSLock(bucket, object)