fix(ilm): report expiries that remove nothing, and name what one removed

An ILM expiry that is refused, because the rule no longer applies when
it is checked again under the object lock or Object Lock protects the
version, or that fails, for example for lack of quorum, left almost no
trace. The bulk noncurrent purge skipped such entries without audit,
trace, log or count; the scanner traced a refusal exactly like a
successful expiry; and nothing counted either. Operators could not see
why a version they expected gone was still there.

- Count every such expiry by reason in expiry_incomplete_total
  {reason="not_applicable"|"locked"|"history_unavailable"|"error"}, in
  metrics v2 and v3; the counting unit is one expiry attempt, and no
  bucket or object is a label.
- Trace it with result=refused or result=failed and the reason. A
  failure is also audited and logged once per object; a refusal changes
  nothing and is not audited, since a split key is refused on every scan.
- Never announce it: no ObjectRemoved notification for a version that is
  still there.
- Name a completed expiry by what its delete did. An expiry that
  addressed no version and whose result is a delete marker created that
  marker, also with the null version ID of the version it replaced; any
  other removed the version it addressed, also a delete marker. The
  scanner and the transitioned expiry took the identity from the queued
  object instead: the expiry of a latest transitioned version in a
  versioned bucket announced the removal of the version it kept, and the
  removal of an expired delete marker was announced as a marker created.

Tests: classification of each error; a bulk purge with a refused, a
failed and a removed entry counts and traces the first two, audits only
the failure and announces only the removal; the expiry events name the
created marker for a latest version (versioned, suspended null,
transitioned) and the removed version for a noncurrent version and an
expired delete marker; both metrics versions export the counter by
reason. The #278 side-effect tests now tell removal traces from
refusal and failure traces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>

cmd/data-scanner.go

@@ -1210,10 +1210,7 @@ func applyTransitionRule(event lifecycle.Event, src lcEventSrc, obj ObjectInfo)
 func applyExpiryOnTransitionedObject(ctx context.Context, objLayer ObjectLayer, obj ObjectInfo, lcEvent lifecycle.Event, src lcEventSrc) (ok bool) {
 	timeILM := globalScannerMetrics.timeILM(lcEvent.Action)
 	if err := expireTransitionedObject(ctx, objLayer, &obj, lcEvent, src); err != nil {
-		if errors.Is(err, errLifecycleNotApplicable) || isErrObjectNotFound(err) || isErrVersionNotFound(err) {
-			return false
-		}
-		ilmLogIf(ctx, fmt.Errorf("expireTransitionedObject(%s, %s): %w", obj.Bucket, obj.Name, err))
+		reportILMExpiryNotDone(ctx, obj, lcEvent, src, err, "transition-expiry"+obj.Name, globalLifecycleSys.trace(obj))
 		return false
 	}
 	timeILM(1)
@@ -1272,14 +1269,10 @@ func applyExpiryOnNonTransitionedObjects(ctx context.Context, objLayer ObjectLay
 		return deleteAllNotCompleted(ctx, obj, lcEvent, src, purge, err, traceFn)
 	}
 	if err != nil {
-		if errors.Is(err, errLifecycleNotApplicable) || isErrObjectNotFound(err) || isErrVersionNotFound(err) {
+		// Assume it is still there, unless it is gone already.
+		if !reportILMExpiryNotDone(ctx, obj, lcEvent, src, err, "non-transition-expiry"+obj.Name, traceFn) {
 			traceFn(ILMExpiry, nil, nil)
-			return false
 		}
-		// Assume it is still there.
-		err := fmt.Errorf("DeleteObject(%s, %s): %w", obj.Bucket, obj.Name, err)
-		ilmLogOnceIf(ctx, err, "non-transition-expiry"+obj.Name)
-		traceFn(ILMExpiry, nil, err)
 		return false
 	}
 	if dobj.Name == "" {
@@ -1295,10 +1288,7 @@ func applyExpiryOnNonTransitionedObjects(ctx context.Context, objLayer ObjectLay
 	// Send audit for the lifecycle delete operation
 	auditLogLifecycle(ctx, dobj, ILMExpiry, tags, traceFn)
 
-	eventName := event.ObjectRemovedDelete
-	if obj.DeleteMarker {
-		eventName = event.ObjectRemovedDeleteMarkerCreated
-	}
+	eventName := ilmExpiryEventName(opts.VersionID, dobj)
 	switch lcEvent.Action {
 	case lifecycle.DeleteAllVersionsAction:
 		eventName = event.ObjectRemovedDeleteAllVersions