fix(restore): keep queued restores waiting for the object lock

The POST handler marks a version ongoing before its background restore
takes the object lock, and only the restore clears that marker. Native
restore holds the lock through the tier download, so a restore of
another version of the same key waits behind it. Both restore entry
points gave up after one delete-operation timeout and returned before
any cleanup, leaving ongoing-request="true" with no owner. Every later
POST for that version was refused with 409 and nothing expired the
marker.

Retry the lock while the restore's context lives, with a dedicated
dynamic timeout so repeated waits do not lengthen delete timeouts.
Shutdown or a caller deadline still ends the wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
2026-11-04 16:37:42 +08:00
parent 3ddf83182f
commit 82dd12faf8
4 changed files with 97 additions and 2 deletions
+14 -1
View File
@@ -2675,6 +2675,19 @@ func (er erasureObjects) updateRestoreMetadata(ctx context.Context, bucket, obje
return nil
}
// getRestoreLock waits for a restore's object lock for as long as ctx lives.
// The POST handler has already marked the version ongoing, and only the
// restore clears that marker. A restore queued behind another version's
// download must therefore keep waiting rather than give up after one timeout.
func getRestoreLock(ctx context.Context, lk RWLocker) (LockContext, error) {
for {
lkctx, err := lk.GetLock(ctx, globalRestoreLockTimeout)
if !errors.As(err, &OperationTimedOut{}) || ctx.Err() != nil {
return lkctx, err
}
}
}
// restoreTransitionedObject for multipart object chunks the file stream from remote tier into the same number of parts
// as in the xl.meta for this version and rehydrates the part.n into the fi.DataDir for this version as in the xl.meta
func (er erasureObjects) restoreTransitionedObject(ctx context.Context, bucket string, object string, opts ObjectOptions) error {
@@ -2682,7 +2695,7 @@ func (er erasureObjects) restoreTransitionedObject(ctx context.Context, bucket s
// Single-pool and direct callers must protect the metadata snapshot
// through download, rehydration and failure cleanup, just like pools.
lk := er.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalDeleteOperationTimeout)
lkctx, err := getRestoreLock(ctx, lk)
if err != nil {
return err
}