Three safeguards of the lifecycle and tier cleanup paths had no test
that failed when they were removed:
- authoritativeHistory accepts a revision only on more than half of the
drives. TestPoolsLifecycleHalfDriveMarker holds a delete marker on
exactly half of sixteen drives, which is the read quorum of a marker,
and checks that the history is refused, that the expiry of the
version under it and of the key are refused and every drive keeps what
it has, and that one more drive decides. With the guard removed the
strict merge at the same majority still refuses the history; with the
merge relaxed as well the test fails, so it pins the refusal, not one
of its two layers.
- deleteAllApproved refuses a purge whose placement read differs from
the versions it approved. deleteAllBeforePlacement is a test-only
injection point between the two reads, nil in production.
TestPurgeAllVersionsRefusesChangedApproval writes a new version and
rewrites the latest one there, as a writer that bypasses the object
lock would, and checks the write quorum refusal, the
history_unavailable count, that nothing is deleted and that the next
purge removes the key. With the check disabled the purge deletes the
versions it never approved.
- readTierHistory refuses the ownership of a remote object while a
drive of any pool cannot be read, and xlMetaV2.DeleteVersion deletes
a free record only with the remote tuple it names.
TestTierOwnershipReadFailsClosed fails one drive of the pool that
holds nothing of the key as offline, not found, faulty and
unformatted, and checks that the expiry worker's and the
decommission's cleanup are refused with a read quorum error and
succeed once the drive reads again.
TestFreeRecordDeleteNeedsMatchingTuple deletes a free record with
another tier, remote object and remote version, in memory and on an
xlStorage drive, and checks the record stays until the tuple matches.
With either check removed the remote object or the record is deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch expiration that keeps no versions purges each key as a whole,
and the purge refuses a key when one of its versions is transitioned to
a remote tier, as it refuses a key with a version Object Lock protects.
The job counted the Object Lock refusal as a refused key and completed,
but counted the tier refusal as failed versions and ended Failed, on
every run, with objectsFailed set to the versions the walk saw. ILM
expiry reports the same tier refusal as refused, not failed.
expireAllVersions now counts a key refused for a transitioned version as
refused, as it does a locked key: nothing was removed, the attempt is
not retried, and the job completes with keysRefused, which mcli reports.
A later run of the job checks the key again. A key whose history cannot
be read still fails the purge as before.
TestBatchExpirePurgeRefusesTieredKey runs a purge of a key with a
transitioned older version and requires a complete job with one refused
key and every version kept; TestPurgeAllVersionsRefusesTieredKeys now
requires refused keys rather than failed objects. Before this change
the job ended Failed with two failed objects.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions of a key purges exactly that key, and the
purge can refuse it or stop early. Those outcomes were hard to see, and
the scanner reported a key it had queued as if it were already gone.
- Refusals and failures of a purge are counted in
expiry_incomplete_total with their own reasons: tier_unsupported (a
version is in a remote tier), budget (stopped after progress; the next
expiry continues), changed (the latest version a batch matched
changed) and history_unavailable. They are traced with the reason and
send no removal event. A partial purge keeps its audit record, now
with the reason. A tier refusal is also logged, once per key and hour,
in words.
- A drive that cannot be read or decoded during the purge's own drive
check now returns errDeleteAllHistoryUnavailable, which names the
cause and stays a write quorum error (503) for API callers.
- The scanner does not queue an expiry of all versions whose latest
version is hot when the pool holds a version in a remote tier: the
purge would refuse it each cycle. It counts each such observation as
tier_unsupported_skipped, once per drive scanner and cycle. A tiered
version only another pool holds is unknown here, so the purge still
decides; a transitioned latest version keeps the transitioned expiry.
The tier predicate is shared with the purge's own check, for FileInfo
and ObjectInfo, and excludes free versions.
- The scanner accounts every version of a key under an expiry of all
versions at its size, queued or not, instead of as already removed.
Usage, and so hard quota, drops after a fresh scan of the subtree once
the purge succeeded. The alerts on many or large versions count the
versions of a key that is not queued.
- A lifecycle expiry never deletes a prefix: DeleteObject refuses
DeletePrefix with Expiration.Expire as InvalidArgument. No caller
sends it since expiries of all versions became purges; a force delete
of a prefix is unchanged.
Not done from the plan: queuing an MRF heal per version when a drive's
xl.meta fails to decode. Object heal does not rewrite metadata that
fails its CRC check (it reports the drive as unknown and skips it), so
those heals would not repair the drive. Such a key stays refused until
its metadata on that drive is repaired.
Tests: TestPurgeAllVersionsRefusalReasons,
TestPurgeAllVersionsRefusesCorruptDrive,
TestScannerSkipsDeleteAllForTieredKey and
TestPoolsDeleteObjectRejectsExpirePrefix are new. Tests that expected
the scanner to queue a refused purge, the predictive accounting, or
the old prefix options are converted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A batch job ran with the server's object access, whoever submitted it,
and acted on versions that Object Lock protects.
Submitter: StartBatchJob binds the authenticated credential to the job
(type, access key, parent, identity revision and a keyed fingerprint).
Submission, every start and every object resolve the binding against
IAM and evaluate the action with the submitter's policies and session
policy. A credential that is gone, disabled, expired or changed fails
the job. A policy statement that would need request context the job
does not have refuses the action. With an authorization plugin, the
plugin and the session policy must both allow. A refusal is persisted
as the job's failure reason and ends the job. A job stored without a
binding fails and has to be submitted again; a definition holds one
job type.
Expire deletes each older version on its own, checking retention and
legal hold and authorizing it under the delete's lock; a locked version
is left and counted. Until the exact-key purge is integrated, a
retainVersions: 0 key is refused when a version the walk saw is locked.
Pull writes refuse to replace a locked destination version, use the
destination's versioning, validate copied retention and apply default
retention. Rotation leaves locked versions as they are.
The status reports versionsSkippedLocked, keysRefused and the failure
reason. Expire and rotation persist their cursor together with these
counts, so a resumed job counts each version once. Describe returns a
redacted copy of the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
An expiry of all versions (ExpiredObjectAllVersions, DelMarkerExpiration,
and batch expiration with the default purge) asked the pools layer for a
prefix delete. The pools layer rebuilt the options, so the request ran as
a recursive delete of the key's directory in every erasure set of every
pool. Object a/b is stored inside the directory of a, so every key under
a/ was removed with it, however new, with one audit entry and one
notification for a alone. The rebuilt options also skipped the lifecycle
recheck, so on a single pool a version written after the decision was
deleted too, and tiered versions lost their metadata without free
versions.
The expiry worker, the migration expiry and batch expiration now ask for
a purge with the in-memory ObjectOptions.DeleteAllVersions:
- DeleteObject hands it to the purge under the object lock, before the
all-pools check. ILM decides with the unchanged lifecycleEventsAllPools,
on a single pool too; batch needs the version its walk matched to still
be the latest. A second read of each pool's authoritative history
records every copy, older writes of the same ID included, and must
equal the approval. Anything else deletes nothing.
- Before deleting, the purge reads every drive of each set that holds a
copy. It refuses the key when a drive cannot be read, when any revision
of an approved version there is transitioned, or, with Object Lock,
when retention or a legal hold protects one. It then deletes on exactly
the drive handles it checked.
- Versions are deleted by ID with the drives' version delete, which only
removes the object directory once it is empty: oldest first, for each
version older writes before the copy reads choose, and the latest last,
only once every older version is gone from every pool. No marker,
replication state or free version is written. The purge stops at the
first error, repeats a quorum-successful delete on the drives that
missed it while the lock is held, and stops before the latest version
at a budget of 1000 versions or one minute, reads included.
- The ILM audit entry records how many versions were removed. A purge
that failed after it began to delete is audited as partial and sends
no removal event. Batch counts come from the purge; a key whose latest
version changed is not expired, and a failed delete marker fails the
job.
The option is rejected with conflicting options and by every layer below
the pools. Force delete and internal prefix deletes keep DeletePrefix.
Delete-all of a transitioned latest version keeps its current path, and
a key with tiered older versions is refused. On a single pool the purge
relies on the lock-domain fix for writer exclusion.
Reviewed adversarially in two design rounds and four patch rounds; the
last approved with nits, which are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang