GET and HEAD recheck a local expiry decision against the versions of
the object in all pools before they hide the version or queue its
expiry. When that recheck failed, because the history of another pool
could not be read, or found that the rule no longer applies with the
versions in all pools, the read was served and nothing was queued,
without a trace or a count. During a degraded pool a version that is
due kept reading and never reached the expiry workers, and the operator
could learn of it only from the scanner's own counts.
recheckReadExpiry now reports such a read as the scanner reports an
expiry it did not complete: it counts expiry_incomplete_total by the
reason ilmExpiryNotDone gives the error, history_unavailable for a read
quorum failure, not_applicable when the version is no longer due, and
publishes an ILM trace with result=skipped and that reason. The reasons
are the existing label values; the response, the version read and the
decision not to queue are unchanged.
TestPoolsLifecycleReadHTTP now asserts the counter delta and the trace
of every case the recheck skips; before this change it counted nothing
for the unreadable-pool and the not-applicable cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A purge of all versions reads the history of the key in every pool
before it checks the drives. When half of the drives of a set could not
be read or decoded, objectVersions refused the history with a read
quorum error and the purge failed with InsufficientWriteQuorum before
deleteAllDrives, which alone queued a heal of each version when a drive
did not decode. With parity of half the drives, a key whose metadata
did not decode on exactly half of them still read and healed, but no
purge queued that heal: the refusal repeated until a scanner or drive
heal happened to repair the drives.
objectVersions now refuses such a history with undecodableHistory,
which wraps the read quorum error and carries the IDs of the versions
the decodable drives hold, the null version by its explicit ID. The
purge queues a heal of each of them before it fails, from the history
of the rule check as from the placement read, and still returns
InsufficientWriteQuorum. The threshold of unknown drives, the refusal
and every other caller of the history are unchanged; expiryApplies
matches the read quorum error through the wrapper.
TestPurgeAllVersionsHealsUndecodableHistory corrupts the metadata CRC
on four of eight drives; before this change the purge refused the key
without queuing any heal.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A decommission or rebalance walk prechecks each due version of a key
with the versions of the key in all pools, to decide whether the version
moves or is left to its ILM expiry. Each precheck read and merged the
key's whole history from every pool, so a key with V due versions read
it V times in the walk alone; with long histories this grew roughly
quadratically.
Read the history once per key into a cache that the walk keeps for the
entry, and evaluate every version's precheck against it. The precheck
only chooses between moving a version and leaving it to the expiry; the
expiry reads the history again under the object lock before it deletes
anything, so a stale cache can only move a version or defer it to a
recheck. A read error is kept, so every due version of that key moves.
The expiry's own per-version reads are unchanged.
Test: decommission and rebalance of a key with 63 due noncurrent
versions read its history once, with the same outcome; without the
cache they read it 63 times.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The merged all-pool check from #279 decided on the versions that a read
quorum of each set merged non-strictly. A drive that missed one metadata
update could therefore supply the revision used for the decision, and a
partially unreadable set could drop a committed version from the history.
Both let an expiry delete data the rule did not cover, including a version
under legal hold.
- objectVersions accepts a revision only when max(n/2+1, the version's
metadata read quorum) drives hold it, identifies revisions by the header
plus a sha256 over a canonical, length-prefixed encoding of the
normalized metadata, filters non-accepted revisions before the strict
merge, and fails closed on omitted versions, too many unreadable drives,
or cancellation.
- Lock fields are read with the evaluator's key precedence. Cross-pool
copies that disagree on hold or retention must be strictly ordered by
valid timestamps; copies with different ModTime are separate writes and
an older protected write refuses the object. The set safeguard re-checks
protection and pending purges on its fresh read for checked version
deletes.
- Duplicate copies with semantically different replication or purge state
refuse the object.
- Batch expiry announces and audits only completed or already-absent
removals.
Reviewed adversarially over seven rounds; the final round approved with no
remaining findings in scope. All-versions expiry through deletePrefix and
tiered delete-all dispatch are tracked separately.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge lock and tag revisions for duplicate versions before all-pool
lifecycle evaluation using the existing metadata reconciliation rules.
Distinguish ordinary lifecycle refusal from object absence and clear
rejected batch replication state so a late policy withdrawal cannot
schedule a purge. Preserve restored-copy cleanup and real absence.
Cover single and batch protections, legitimate metadata removals, and
actual replication queue and HTTP target behavior.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Recheck only local ordinary-delete candidates that need an all-pool lifecycle decision. Match UUID/null identity to the version actually read and queue its global metadata and event; retain a successful read on recheck failure, absence, or refusal.
Keep GET reader cleanup, authorization, proxy reads, single-pool reads, restored-copy cleanup, and expiry response headers on their existing paths. Cover signed GET/HEAD, queue contents, metadata faults, missing matches, no-scan boundaries, and reader-lock release.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang