minio/docs/investigations/r5/verification.md

# R5 local verification

## Scope and source identity

This is a local repair of ordered tag deletion along selected replication requests. It does not authorize or establish a main merge, push, release, deployment, historical-state migration, or production multi-site acceptance.

- Research baseline: `9ebe81c1b3611f9cc73e676b5b741c2be62c467a`.
- Combined verification dependency: R4 `dbcf8dec589deb5d91e17d295cb70997635f5b55`; R5 does not modify `cmd/object-api-options.go`.
- Accepted plan: v2, SHA256 `5a782acf3f285b23d1ae43a73481c4eb772a9a6d917fc5a550ecfc7cbf7446ca`.
- Actual plan reviewer: `claude-opus-5`, explicit max effort; v1 requested changes, v2 approved with nonblocking notes and zero blockers. See `consensus.md`.
- Actual implementation reviewer: the same requested/observed model and effort, `GO_WITH_NONBLOCKING_NOTES`, zero blockers. Original review identity and hashes are in `opus-implementation-metadata.json`.
- `implementation-manifest.json` identifies the reviewed patch. `final-implementation-manifest.json` identifies the final source after a stronger multi-pool test assertion and gofumpt formatting. All seven production file hashes still match the review.

## Executed regression checks

All commands run from this worktree with `GOMAXPROCS=2` and `go test -p 1`, Go `go1.27.1 darwin/arm64`. Raw logs are in `/Users/vonng/tmp/silo-r5-20260915-77ad/`.

| Check | Observed result | Evidence |
|---|---|---|
| New R5 tests before implementation | Reproduced real signed-HTTP deletion resurrection, empty COPY loss, full-write persistence/skip, equal-value sender skip, local revision inversion and stale source ACK | `baseline*.log`, `discussion-baseline.log`; `reproduction.md` |
| Latest complete new R5 selection | 13 top-level tests passed, 9.161s | `fixed-targeted-latest.log` |
| Related selection, host capacity adapted | 134 passed; one POST fixture still failed the host minimum-free threshold, 56.007s | `related-suite-capacity-final.log`; exact 135 names in `related-test-names.txt` |
| Remaining POST test plus strengthened multi-pool replay test | Both passed, 3.232s; completes the 135-name selection across the two batches | `final-post-and-pools.log` |
| Existing `TestReplicationResync` in isolation | Passed on baseline (2.191s) and R5 (1.776s) | `baseline-resync.log`, `fixed-resync-isolated.log` |
| Final R5 plus tag-storage race selection | 16 top-level tests passed, 25.584s runtime, no race diagnostics | `targeted-race-final.log`; exact command/exit in `final-check-results.json` |
| Repository verifiers | Passed: lint 0 issues, generated files unchanged, branding/compatibility and entrypoint checks passed | `make-verifiers-serial.log`, `verifiers-result.json` |
| Repository build and binary invocation | `make build` passed; the resulting `silo --version` exited 0 | `make-build.log`, `build-result.json`, `silo-version.log` |

The selected regressions include existing replication trust/header poisoning, API preconditions, Object Lock, SSE-C retransmission, R4 KMS option/COPY tests, and pool metadata/cleanup/retry checks. The new R5 suite covers:

- Local PUT tags, repeated DELETE, empty PUT and ordinary empty COPY; tag revisions advance even without selected replication, while local tagging preserves object ModTime.
- Empty/nonempty and newer/stale/equal/missing revisions through signed COPY with both metadata directives, PUT and multipart; UUID/null and unqualified COPY; unrelated newer versions survive.
- Multipart deletion committed between initiation and completion, with the upload's saved revision checked at initiation and ordered again at completion.
- Exact SDK sender headers, nanosecond precision, legacy fallback only for nonempty tags, and rejection of malformed recorded times.
- Equal-value metadata resend, failed COPY reporting/retry, no incoming-replica requeue, and a stale queued source ACK preserving the current deletion.
- Uniform local tag revisions beyond every physical pool, deterministic inverted request/commit timestamps, normal-routing readback after replay and inspection of every retained pool copy.
- Destination KMS encryption/readback and signed SSE-C key rotation with decrypted GET. These are local handler/storage fixtures, not an encrypted-source-to-encrypted-destination two-site deployment.

## Baseline and environment failures retained

The first broad selection panics at `TestReplicationResync` before any R5 test executes. Replacing all seven R5 production files with the R4 baseline, and hiding the two new tests in a Go overlay, reproduces the same panic after the same preceding tests (`baseline-related-suite.log`). The test passes alone on both versions. The remaining 135-name selection therefore runs separately; this is not reported as an unfiltered full-package pass.

This host's used-space percentage makes existing allocation tests return `XMinioStorageFull`. The test-only overlays add the existing `tagTestCapacityDisk` via `r5Capacity` at the API/pool fixture boundaries and the final POST fixture. The adapter changes reported capacity only, delegates real I/O and propagates disk errors. The exact overlays, original/modified fixture hashes and diffs are retained as `capacity-fixture*` and `capacity-post*`. No fixture overlay or capacity-policy change enters production code.

The first adapted link and first verifier run also failed actual `ENOSPC` when the volume had about 200–500 MiB available (`related-suite-capacity.log`, `make-verifiers.log`). Regenerable Go cache data untouched for three days was reclaimed with an exact manifest (`cache-reclaim.json`); subsequent successful checks are distinguished from those failures. A subsequent verifier caught gofumpt formatting in the new helper; that formatting was corrected before final validation.

An earlier KMS multipart fixture used a single-PUT ETag with multipart data layout and failed decryption. Seeding a real multipart source fixed the fixture; the subsequent complete run passed plaintext readback. The failed log remains `fixed-targeted.log`, and this is not attributed to a production encryption change.

## Local delivery and remaining integration gates

Required scoped local checks are complete. `validation-results.json` records command results, and `evidence-manifest.json` identifies the raw files and binary by SHA256. The build compiled the working source identified by `final-implementation-manifest.json`; the Makefile stamped its pre-commit dependency ID `dbcf8dec5` into this local development binary. Final source identity is established by the file hashes, not by that pre-commit version label.

An unfiltered full `cmd` run and real multi-site deployment remain future integration gates before any separately authorized merge/release. Known scope limits are retained in plan v2 and `implementation-review-response.md`: tag-filter target eligibility, historical missing revisions, malformed stored source times, legacy peers dropping empty revisions, and arbitrary distributed clock skew.

The verifier uses the repository-pinned golangci-lint v2.13.1 through a local wrapper adding only `--allow-serial-runners`. This waits for the shared host lint lock instead of running another lint process concurrently. The first unscheduled attempt was rejected by that lock (`make-verifiers-success.log`; despite that filename, its recorded exit is 2). The final serialized run passed. The optional `typos` binary is unavailable and was skipped by the Makefile.

R4 has since merged as `af2b1794d38d9e70e1d2c3ee692426e4b6cab4bd` (PR #193). `dependency-handoff.json` verifies that its production options file and both test bodies match the dependency used above. The other differences are test license headers and R4 review/validation documents. The R5 delivery base is this exact merged dependency, with the old unsigned `dbcf8dec5` ancestor removed. The original recorded plan/review baseline remains intact as historical evidence. The final local commit, clean-worktree check and post-rebase file-hash comparison are recorded outside the commit in `/Users/vonng/tmp/silo-r5-20260915-77ad/final-delivery.json`.