A RenameData that fails after its caller gave up removes its empty
object directory and the empty parents (ed05d982a, 73025824a). Only
the two renames of RenameData itself survived that cleanup, through
renameDataAll. Any other write of the same key that creates the object
directory failed on the drive when the cleanup removed the directory
between the directory creation and the open: WriteMetadata for a
delete marker on an absent key or for a first version, through
openFile, returned the raw open error, and writeAllDirect returned
errFileNotFound. The version was then missing on that drive, and
DeleteObject queues only offline drives for MRF. Before ed05d982a the
failure path never removed the directory, so this window is new.
Move the retry policy of renameDataAll into the shared helpers. A write
below the volume directory retries once per path level when the
directory creation or the use of the directory reports "not found"
(retriesAgainstCleanup): a cleanup removes each directory at most once
and each failed attempt needs one such removal. reliableRename applies
it in place of its blind single rename retry, which never recreated the
directory, and only while the source still exists, so renameDataAll is
replaced by renameAll. openFileAll applies it to the directory creation
and open shared by openFile and writeAllDirect. mkdirAllErr is split
out of mkdirAll, like renameAllErr, so that the retry decision uses the
OS error: on unix, mkdirAll maps a vanished parent to
errFileAccessDenied through isSysErrPathNotFound.
Invariants kept: the cleanup still removes only empty directories and
never a committed xl.meta or data directory; the error mapping of
mkdirAll, renameAll, openFile and writeAllDirect is unchanged, including
the Windows isSysErrPathNotFound branch; retries are bounded by the
path depth; a rename whose source is gone still fails at once.
UpdateMetadata renames over an existing xl.meta, so its directory is
never empty and the cleanup cannot remove it; it needs no change.
TestXLStorageMetadataWriteSurvivesLateCleanup injects, right after the
writer's os.Mkdir of the object directory, either a late RenameData
whose staged data is gone or a cleanup that walks up the whole path one
directory per step, for a delete marker, a first version and a large
xl.meta. Before the fix all six cases failed with "no such file or
directory" or "file not found".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The xl.meta rename at the end of RenameData creates the object directory
the same way the data rename does, and its failure branch left that
directory behind for writes without a data directory, such as inline
and zero-byte objects. Remove it and its empty parents there too, with
the same trailing-slash call that PR #286 adds to the data branch for
issue #285.
Making the cleanup safe against concurrent writes needs two more
changes:
- deleteFile now walks up with a trailing slash, so each parent step
removes only an empty directory. Between two steps of a walk, another
walk can remove the rest of the chain and a writer can commit an
object whose xl.meta sits at the path the first walk removes next,
for example the object "parent" under the key "parent/xl.meta/child".
The walk used to delete that file. Ordinary deletes could already do
this; the new cleanup made it reachable with failed writes.
- The two renames of RenameData retry when the directory creation or
the rename reports "not found" while the source is still in place,
up to once per path level. A late, failed RenameData can now remove
the directory another write of the same key has just created, and
that write used to fail on the drive although its data was intact.
The retry decides on the error of the attempt itself, because another
write can recreate the directories in between. renameAll's error
mapping moves unchanged into renameAllErr so that the retry can see
the raw error.
The stagedDataGoneDisks test helper now swaps the drives behind the
set's disk list under the lock that the list takes, instead of
replacing the getDisks callback that background tasks read unlocked.
Tests:
- TestXLStorageRenameDataCommitMetaMissingRemovesEmptyDirs: zero-byte
and inline commit failures leave no empty directories, a sibling and
a moved data directory stay.
- TestXLStorageWalkUpKeepsCommittedObject: a committed object at a
parent path survives the walk, through DeleteVersion and both
cleanup branches.
- TestRenameDataSurvivesConcurrentEmptyDirCleanup: PutObject (large
and inline) and CompleteMultipartUpload succeed when a late failed
RenameData of the same key removes their fresh directory on all 16
drives.
- TestXLStorageRenameDataRetriesWhileCleanupWalksUp: a write survives
a cleanup that removes each level of its path in turn, also when a
nested-key write recreates the directories in between.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
When the staged xl.meta disappears between the data rename and the
meta-file commit inside RenameData(), the affected drives return
errFileNotFound. An in-quorum errFileNotFound is reduced by
reduceWriteQuorumErrs() and previously surfaced as:
- IncompleteBody (400) from PutObject(), misattributing the failure
to a client disconnect, or
- ObjectNotFound (404 NoSuchKey) from CompleteMultipartUpload(),
telling clients the object is absent so they skip retries.
Map errFileNotFound at the commit step to a dedicated internal
errCommitMetaMissing, convert it to errErasureWriteQuorum in
renameData(), and map it to InsufficientWriteQuorum (HTTP 503) in
toObjectErr() so clients retry. Below-quorum outcomes already return
InsufficientWriteQuorum and are unchanged, as is the IncompleteBody
mapping for staging failures of client-disconnected uploads.
Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
- Implement a new xl.json 2.0.0 format to support,
this moves the entire marshaling logic to POSIX
layer, top layer always consumes a common FileInfo
construct which simplifies the metadata reads.
- Implement list object versions
- Migrate to siphash from crchash for new deployments
for object placements.
Fixes#2111
Feng Ruohang