Three safeguards of the lifecycle and tier cleanup paths had no test
that failed when they were removed:
- authoritativeHistory accepts a revision only on more than half of the
drives. TestPoolsLifecycleHalfDriveMarker holds a delete marker on
exactly half of sixteen drives, which is the read quorum of a marker,
and checks that the history is refused, that the expiry of the
version under it and of the key are refused and every drive keeps what
it has, and that one more drive decides. With the guard removed the
strict merge at the same majority still refuses the history; with the
merge relaxed as well the test fails, so it pins the refusal, not one
of its two layers.
- deleteAllApproved refuses a purge whose placement read differs from
the versions it approved. deleteAllBeforePlacement is a test-only
injection point between the two reads, nil in production.
TestPurgeAllVersionsRefusesChangedApproval writes a new version and
rewrites the latest one there, as a writer that bypasses the object
lock would, and checks the write quorum refusal, the
history_unavailable count, that nothing is deleted and that the next
purge removes the key. With the check disabled the purge deletes the
versions it never approved.
- readTierHistory refuses the ownership of a remote object while a
drive of any pool cannot be read, and xlMetaV2.DeleteVersion deletes
a free record only with the remote tuple it names.
TestTierOwnershipReadFailsClosed fails one drive of the pool that
holds nothing of the key as offline, not found, faulty and
unformatted, and checks that the expiry worker's and the
decommission's cleanup are refused with a read quorum error and
succeed once the drive reads again.
TestFreeRecordDeleteNeedsMatchingTuple deletes a free record with
another tier, remote object and remote version, in memory and on an
xlStorage drive, and checks the record stays until the tuple matches.
With either check removed the remote object or the record is deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions of a key purges exactly that key, and the
purge can refuse it or stop early. Those outcomes were hard to see, and
the scanner reported a key it had queued as if it were already gone.
- Refusals and failures of a purge are counted in
expiry_incomplete_total with their own reasons: tier_unsupported (a
version is in a remote tier), budget (stopped after progress; the next
expiry continues), changed (the latest version a batch matched
changed) and history_unavailable. They are traced with the reason and
send no removal event. A partial purge keeps its audit record, now
with the reason. A tier refusal is also logged, once per key and hour,
in words.
- A drive that cannot be read or decoded during the purge's own drive
check now returns errDeleteAllHistoryUnavailable, which names the
cause and stays a write quorum error (503) for API callers.
- The scanner does not queue an expiry of all versions whose latest
version is hot when the pool holds a version in a remote tier: the
purge would refuse it each cycle. It counts each such observation as
tier_unsupported_skipped, once per drive scanner and cycle. A tiered
version only another pool holds is unknown here, so the purge still
decides; a transitioned latest version keeps the transitioned expiry.
The tier predicate is shared with the purge's own check, for FileInfo
and ObjectInfo, and excludes free versions.
- The scanner accounts every version of a key under an expiry of all
versions at its size, queued or not, instead of as already removed.
Usage, and so hard quota, drops after a fresh scan of the subtree once
the purge succeeded. The alerts on many or large versions count the
versions of a key that is not queued.
- A lifecycle expiry never deletes a prefix: DeleteObject refuses
DeletePrefix with Expiration.Expire as InvalidArgument. No caller
sends it since expiries of all versions became purges; a force delete
of a prefix is unchanged.
Not done from the plan: queuing an MRF heal per version when a drive's
xl.meta fails to decode. Object heal does not rewrite metadata that
fails its CRC check (it reports the drive as unknown and skips it), so
those heals would not repair the drive. Such a key stays refused until
its metadata on that drive is repaired.
Tests: TestPurgeAllVersionsRefusalReasons,
TestPurgeAllVersionsRefusesCorruptDrive,
TestScannerSkipsDeleteAllForTieredKey and
TestPoolsDeleteObjectRejectsExpirePrefix are new. Tests that expected
the scanner to queue a refused purge, the predictive accounting, or
the old prefix options are converted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An ILM expiry that is refused, because the rule no longer applies when
it is checked again under the object lock or Object Lock protects the
version, or that fails, for example for lack of quorum, left almost no
trace. The bulk noncurrent purge skipped such entries without audit,
trace, log or count; the scanner traced a refusal exactly like a
successful expiry; and nothing counted either. Operators could not see
why a version they expected gone was still there.
- Count every such expiry by reason in expiry_incomplete_total
{reason="not_applicable"|"locked"|"history_unavailable"|"error"}, in
metrics v2 and v3; the counting unit is one expiry attempt, and no
bucket or object is a label.
- Trace it with result=refused or result=failed and the reason. A
failure is also audited and logged once per object; a refusal changes
nothing and is not audited, since a split key is refused on every scan.
- Never announce it: no ObjectRemoved notification for a version that is
still there.
- Name a completed expiry by what its delete did. An expiry that
addressed no version and whose result is a delete marker created that
marker, also with the null version ID of the version it replaced; any
other removed the version it addressed, also a delete marker. The
scanner and the transitioned expiry took the identity from the queued
object instead: the expiry of a latest transitioned version in a
versioned bucket announced the removal of the version it kept, and the
removal of an expired delete marker was announced as a marker created.
Tests: classification of each error; a bulk purge with a refused, a
failed and a removed entry counts and traces the first two, audits only
the failure and announces only the removal; the expiry events name the
created marker for a latest version (versioned, suspended null,
transitioned) and the removed version for a noncurrent version and an
expired delete marker; both metrics versions export the counter by
reason. The #278 side-effect tests now tell removal traces from
refusal and failure traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The merged all-pool check from #279 decided on the versions that a read
quorum of each set merged non-strictly. A drive that missed one metadata
update could therefore supply the revision used for the decision, and a
partially unreadable set could drop a committed version from the history.
Both let an expiry delete data the rule did not cover, including a version
under legal hold.
- objectVersions accepts a revision only when max(n/2+1, the version's
metadata read quorum) drives hold it, identifies revisions by the header
plus a sha256 over a canonical, length-prefixed encoding of the
normalized metadata, filters non-accepted revisions before the strict
merge, and fails closed on omitted versions, too many unreadable drives,
or cancellation.
- Lock fields are read with the evaluator's key precedence. Cross-pool
copies that disagree on hold or retention must be strictly ordered by
valid timestamps; copies with different ModTime are separate writes and
an older protected write refuses the object. The set safeguard re-checks
protection and pending purges on its fresh read for checked version
deletes.
- Duplicate copies with semantically different replication or purge state
refuse the object.
- Batch expiry announces and audits only completed or already-absent
removals.
Reviewed adversarially over seven rounds; the final round approved with no
remaining findings in scope. All-versions expiry through deletePrefix and
tiered delete-all dispatch are tracked separately.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Feng Ruohang