minio/docs/investigations/r7/plan-v1.md

R7 plan v1: preserve normalized replica object metadata

Frozen scope and source

• Date: 2026-09-15. Worktree: /Users/vonng/.codex/worktrees/ad51/silo.
• Local branch: codex/r7-replication-content-encoding.
• Baseline: 9ebe81c1b3611f9cc73e676b5b741c2be62c467a, also returned by current gh api repos/pgsty/silo/commits/main and fetched origin/main.
• PR #187: OPEN, unmerged, no reviews/checks returned; head b8f2fdde41dff3dc3b8db669c1d42d30ca5c1d3d. GraphQL's PR baseRefOid is 89637554d60c27cfc51d2281d0a4fe15e415f06d; it is not the live main checked above. Snapshot and exact diff: /Users/vonng/tmp/silo-r7-20260915-ad51/pr187.{json,diff}.
• Introduction: 56fa63bfd155154157cd7e1fb6dc295a3b3104ed (2026-04-15), replication-header injection hardening. Keep its trust protections intact.
• Governing scope: PGSTY maintained stack, minimal compatible fix. No dependency, wire-format, credential, encryption algorithm or API changes.

Root cause and observable contract

extractMetadata calls the ordinary extractor, removes disallowed unencrypted-length/MD5 user metadata, and trims the exact aws-chunked transport token from content-encoding. The trusted-replica restoration currently calls the same broad extractor with allowReplication=true. That replays all supported headers and user metadata, reversing normalization and redaction.

Expected mappings are aws-chunked -> absent Content-Encoding, aws-chunked,gzip -> gzip, and gzip -> gzip. Object bytes are not transformed by this fix. AWS documents this behavior in SigV4 streaming. The helper reproduction is /Users/vonng/tmp/silo-r7-20260915-ad51/baseline_test.go with Go overlay and baseline.log; its failing expectations are evidence of the current defect, not implementation validation.

Persistence/read path: erasure-metadata.go reads fi.Metadata["content-encoding"] into ObjectInfo.ContentEncoding; api-headers.go exposes it on GET/HEAD. Outbound putReplicationOpts and metadata-only replication copy also use the object's content encoding. Preventing raw request metadata from being replayed at ingress is sufficient for this defect and avoids read-path masking.

Input and trust boundary audit

The helper does not authenticate; callers own authentication and authorization. evaluateReplicationTrust requires an authenticated principal, the exact single replication marker true, and s3:ReplicateObject; restoring replica-only metadata also requires REPLICA. Unauthorized declared replicas are rejected; marker-only/untrusted requests retain their existing sanitized behavior. Do not move restoration earlier or make headers themselves establish trust.

Actual caller	Metadata before restoration	Trust gate and intended result
PutObjectHandler	extractMetadataFromReq before trust evaluation	after successful signature verification, evaluate/apply trust; only replicaTrusted restores six fields
CopyObjectHandler via getCpObjMetadataFromHeader	REPLACE calls extractMetadataFromReq; COPY uses source metadata	authenticated source/destination checks; allowReplicationMetadata=replicaTrusted; REPLACE preserves normalization, COPY retains existing semantics
NewMultipartUploadHandler	extractMetadataFromReq after trust/sanitization	replicaTrusted restores six fields into initiation metadata; parts and completion reuse saved metadata
PutObjectExtractHandler, outer Snowball headers	only storage class and per-entry transform metadata, not generic extractMetadata	per-entry PutObject and ReplicateObject authorization; only replicaTrusted restores six fields. No-PAX entries must not inherit ordinary outer archive metadata
PutObjectExtractHandler, PAX entry metadata	extractMetadata on minio.metadata.* records	reuse per-entry trust; merge normalized entry metadata plus six allowed fields. Outer ordinary archive encoding must not leak even if the PAX map omits it

PutObjectPart, CopyObjectPart and CompleteMultipartUpload do not call this helper; no additional restoration is needed there. Validate completion persistence to catch assumptions at this boundary. POST form upload does not restore replication metadata. Metadata COPY does not normalize historical source metadata; that is deliberately outside this preventive fix.

Proposed production patch

Adopt the production change in PR #187, adjusted only if current-context application requires it:

1. Remove the extractMetadataFromMimeWithReplication boolean-mode helper.
2. Ordinary extractMetadataFromMime keeps header canonicalization and supported/user metadata extraction, always skips the replication-only mapping keys.
3. extractReplicationMetadataFromMime keeps nil-input error behavior and canonical header lookup; loops only over replicationToInternalHeaders and joins multi-values exactly as before.
4. Never re-read ordinary supported or user metadata in the restoration helper. Preserve keys already normalized, defaulted, redacted, or set by the caller.
5. Preserve all six mappings: sealed SSE-C key, seal algorithm, IV, encrypted-multipart marker (including its empty value), actual object size, and ReplicationSsecChecksumHeader (identity mapping). Preserve canonicalized/lowercase input header compatibility.
6. Clarify the comment to cover Snowball: ordinary metadata is owned by the caller; the common normalizing path runs before restoration, while outer archive metadata is not per-entry object metadata.

No normalizer/token grammar rewrite. The current exact-token trimming semantics, malformed duplicate-cased headers, and validation of SSE field payloads are outside this bug; retain existing behavior rather than expanding accepted formats or validation rules.

Verification matrix and acceptance

Use temporary overlay reproductions before consensus. Promote focused regressions only after recorded Opus agreement. Run targeted tests with bounded Go parallelism because sibling tasks share this host.

1. Helper pipeline: absent encoding, bare aws-chunked, aws-chunked,gzip, gzip, gzip,aws-chunked, multi-valued encoding; ordinary vs restoration; key absence for bare encoding; legitimate gzip retained; ordinary/user metadata sentinel values and redacted unencrypted metadata not restored. Exact expected six-field map, canonical/lowercase headers, empty multipart marker, nil input handling. Repeat restoration should not change ordinary metadata.
2. Real signed HTTP PUT -> persisted ObjectInfo -> GET and HEAD on the existing single-drive and 16-drive erasure fixtures. Use real streaming chunk signatures for transport cases and a valid gzip payload for gzip cases. Compare raw response bytes and content encoding; bare transport must have no header. Test authenticated ordinary, trusted replica, and marker without replication permission; declared replica without permission returns 403 and creates nothing.
3. Signed COPY REPLACE and multipart initiate/part/complete -> persisted ObjectInfo -> GET/HEAD for bare, mixed, and plain gzip. Include ordinary controls. Initiation carries object metadata; part/completion carry contrasting content encoding to prove they cannot replace it. COPY preserves existing source metadata semantics.
4. Snowball trusted entry tests with and without PAX, including PAX no Content-Encoding and PAX mixed encoding; outer aws-chunked never leaks. Existing per-entry trust test stays green.
5. Existing SSE-C single PUT and multipart replication round trips plus replication-header poisoning regressions. Helper matrix verifies all six field mappings; actual SSE-C tests verify readable ciphertext replicas, encryption metadata, checksum and multipart layout. Preserve bucket default encryption/compression behavior. Run relevant SSE-KMS/SSE-S3 option/replica tests if available without broadening R4 scope.
6. Targeted package tests, focused race run, build and repository verifiers. If environmental failures (e.g. disk free-space threshold) prevent existing tests from reaching the path, retain the original failure and use an explicitly documented temporary capacity adapter already used by the repository, keeping actual object I/O on test disks. Do not mislabel that as an unmodified pass.
7. Baseline regression must fail for raw/mixed trusted metadata and fixed code must pass identical expectations. Record exact commands, exit status, baseline/diff hashes and fixture limits. No full distributed sites/deployment acceptance claim from local handler tests.

Stored-object remediation proposal (separate, no execution)

Upgrade only prevents new pollution. Existing source/COPY metadata can remain wrong, and rollback reopens ingress pollution without undoing repairs. Do not rewrite production objects or private xl.meta files.

A separate operator-reviewed job must inventory bucket/key/version, original Content-Encoding and complete metadata, source/replica provenance, version/ETag/size/checksum and encryption/retention settings. Identify exact aws-chunked tokens and preserve other encodings/order. Verify source bytes/encoding before deciding; gzip must not be guessed or decompressed merely from the broken label. Keep an immutable manifest and metadata backup. Test a version-preserving supported metadata operation on a local replica of the relevant setup; ordinary S3 self-COPY can create a new version/change metadata timestamps and is not a universal version-preserving repair. Resolve object-lock, SSE-C keys, concurrent changes and replication ordering before approving the concrete write plan. Apply a small approved batch with concurrency guards, re-read exact versions, verify GET/HEAD and raw bytes/checksums, then reconcile replicas. Skips/conflicts need explicit reporting and a tested rollback. This task supplies the reviewable design only.

Effort, delivery and gate

Expected 0.5-1 engineer-day for patch, targeted tests and evidence on a familiar checkout; stored-object repair and release are separate work. Production patch is about 30 added/20 removed lines in one helper file; tests provide most of the new code.

Before implementation: actual Claude Code --model claude-opus-5 --effort max, read-only tools, same frozen plan hash + baseline + PR snapshot. Record raw review, actual assistant model(s), objections and dispositions. Any model mismatch/error/rate-limit is not consensus. Resolve substantive findings and get explicit approval of the same plan version before production changes. After consensus, implement and verify without another user permission request.

Deliver research, versioned plan, consensus/dispositions, minimal production diff, tests and verification summary. Local commit may package the reviewable result. No main merge, remote PR mutation, push, release, deployment, or existing-object rewrite is included.