Commit Graph

6607 Commits

Author SHA1 Message Date
Feng Ruohang f9f9fa6c90 test: reproduce cross-type bucket metadata lost updates
Pause one whole-record writer at the metadata PutObject boundary and let a different config writer commit from the same stale snapshot. Assert that policy+CORS and tagging+SSE both survive on disk and in the resident cache.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 07:20:53 +08:00
Feng Ruohang dee2c3a02b fix: preserve multi-delete authentication and audit context
Authenticate DeleteObjects before validating entry count, remove the obsolete per-version auth helper, and pin the marker-only request to the ordinary authorization path.\n\nRefs: #58

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 07:08:02 +08:00
Feng Ruohang 75a6734e49 fix: authorize explicit version deletes with DeleteObjectVersion
Select DeleteObject or DeleteObjectVersion from each request's effective version ID for single and multi-delete. Authenticate multi-delete once, bind version conditions to each XML entry, and keep the established DeleteObject plus ReplicateDelete target contract with explicit version denies honored only on trusted replication.\n\nRefs: #58

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 07:08:02 +08:00
Feng Ruohang f8b598f1d3 test: reproduce explicit-version delete authorization mismatch
Prove that DeleteObjectVersion-only principals cannot delete named UUID or null versions while DeleteObject-only principals can, contrary to the S3 action mapping in issue #58.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 07:08:02 +08:00
Feng Ruohang 04b097fd9f chore: refresh compatibility and lint baselines
Accept the new CORS test routes, resident getter, and replication header literals in the rebrand guard. Apply gofumpt, context-first helper ordering, and spelling fixes required by CI.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 02:31:20 +08:00
Feng Ruohang ab3ae99ca3 fix: preserve Snowball request defaults across workers
Snapshot per-entry requests after applying bucket encryption defaults but before streaming trailers are consumed. Keep authorization failures fatal while retaining Snowball ignore-errors behavior for object-lock failures.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 00:20:06 +08:00
Feng Ruohang 5db7be4ee4 fix: validate replication within the rule prefix
Place synthetic permission-check objects under each enabled rule's effective prefix, so least-privilege target policies are validated against the namespace they will actually replicate.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 00:03:33 +08:00
Feng Ruohang ff44527a3c fix: isolate Snowball replication trust per entry
Evaluate PutObject and ReplicateObject permissions with immutable per-entry request snapshots during concurrent Snowball extraction. Preserve the first API error without sharing mutable handler state, and cover prefix-scoped trust under the race detector.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-02 00:02:24 +08:00
Feng Ruohang c9ad746732 fix: verify replication permissions in validity probes
Evaluate ReplicateObject or ReplicateDelete before returning the no-op validation response, so underprivileged target credentials fail during replication setup instead of at runtime.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-01 23:16:09 +08:00
Feng Ruohang f3438b2602 fix: validate CORS state in replication status
Count only valid live CORS states in per-site summaries. Treat baselines and tombstones as absent, and diagnose malformed payloads or missing source timestamps.\n\nRefs: #77

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-01 23:16:04 +08:00
Feng Ruohang 938603458d fix: harden CORS and replication request trust
Keep pre-authentication CORS lookups resident-only so attacker-controlled path segments cannot trigger metadata I/O or grow the metadata cache. Preserve fail-closed behavior for startup, load failures, invalid metadata, and the internal namespace.

Centralize replication request trust after authentication, distinguish general replication from replica-only privileges, and gate SSE-C ciphertext handling, source metadata, object-lock bypasses, event suppression, delete semantics, and replica status on the appropriate permission. Add least-privilege, multipart, PostPolicy, CORS amplification, and compatibility regressions.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-01 20:50:08 +08:00
Feng Ruohang 74c97d005d fix: authorize SSE-C attribute reads by replication permission
GetObjectAttributes lets a replication peer read SSE-C attributes without
presenting the customer key. That carve-out was keyed on the
X-Minio-Source-Replication-Request header alone, which any client can set,
so a caller holding only s3:GetObject could read an SSE-C object's ETag,
plaintext size, part list, and checksums without the key -- the response was
byte-identical to one made with the correct key.

Gate the carve-out on s3:ReplicateObject for the target object, mirroring
CopyObjectHandler's existing replication check.

The pre-existing test asserted the carve-out with root credentials, which
hold every action and therefore cannot tell a gated check apart from an
ungated one. Add least-privilege cases that do: a reader without
s3:ReplicateObject is now refused, a caller that holds it keeps the
carve-out, and the ordinary key-bearing path is unchanged.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-30 15:50:24 +08:00
Feng Ruohang 0507c3d56a Merge remote-tracking branch 'origin/main' into codex/cors-hotpath 2026-08-29 20:02:51 +08:00
Feng Ruohang ca1f11b708 Merge remote-tracking branch 'origin/main' into codex/issue-77-status-accounting 2026-08-29 19:50:27 +08:00
Feng Ruohang f8106cd693 Merge remote-tracking branch 'origin/main' into codex/issue-78-preserve-bucket-config 2026-08-29 19:39:50 +08:00
Feng Ruohang b1ba685ac5 Merge pull request #89 from pgsty/codex/issue-76-object-lock-wire
fix: replicate object lock config in its own field
2026-08-29 19:39:22 +08:00
Feng Ruohang f377b5c95b Merge remote-tracking branch 'origin/main' into codex/issue-50-after-checksum-contract
# Conflicts:
#	cmd/erasure-multipart-fullobject_test.go
2026-08-29 19:28:06 +08:00
Feng Ruohang 20aaefa89c Merge remote-tracking branch 'origin/main' into codex/reject-unsupported-checksums 2026-08-29 19:14:51 +08:00
Feng Ruohang f2ba439022 Merge remote-tracking branch 'origin/main' into codex/get-object-attributes-ssec 2026-08-29 19:04:14 +08:00
Feng Ruohang 2ba1f38503 Merge pull request #87 from pgsty/codex/issue-82-after-83
fix: authenticate SSE-C keys on zero-byte reads
2026-08-29 19:03:57 +08:00
Feng Ruohang 8d58343263 Merge pull request #96 from pgsty/codex/crc64-completion-after-foundation
fix: reject composite CRC64NVME completion
2026-08-29 18:51:09 +08:00
Feng Ruohang 32b2aa49f1 fix: reject composite CRC64NVME completion
Remove the remaining type-only canonicalization at CompleteMultipartUpload while preserving legacy uploads stored as FULL_OBJECT.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 18:36:02 +08:00
Feng Ruohang 8448512a1f docs: clarify layered SSE-C key authentication
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 18:21:13 +08:00
Feng Ruohang 150e7b5f9e fix: preserve global CORS response semantics
Guard only the per-bucket metadata lookup, then retain the global handler Vary and originless preflight behavior for non-CORS traffic.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 18:19:57 +08:00
Feng Ruohang 21870fa2e7 fix: preserve replicated object attributes reads
Keep the existing trusted replication carve-out while authenticating ordinary SSE-C GetObjectAttributes requests.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 18:18:01 +08:00
Feng Ruohang d4c8da162b fix: reject composite CRC64NVME trailers
Apply the full-object-only rule to declared streaming checksum trailers and cover the HTTP mutation path.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 18:15:51 +08:00
Feng Ruohang fc7bf7b295 test: keep CORS hot-path coverage route-neutral
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:58:30 +08:00
Feng Ruohang d28885d0e5 fix: reject composite CRC64NVME checksums
Return InvalidArgument for CRC64NVME with COMPOSITE at multipart initiation and PutObject instead of silently canonicalizing the request to FULL_OBJECT.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:32:36 +08:00
Feng Ruohang c4fd97d0bf test: align zero-byte rotation key errors
After zero-byte reads authenticate SSE-C at the object layer, equal invalid rotation keys match non-empty reads and return AccessDenied before the rotation-specific distinction.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:32:25 +08:00
Feng Ruohang b73581b05d fix: authenticate SSE-C keys on zero-byte reads
Unseal supplied SSE-C and copy-source keys after range and request preconditions when a zero-byte read cannot build a decryptor. Preserve internal no-decryption, replication, restore, and absent-header reads.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:32:25 +08:00
Feng Ruohang 474cd5801e fix: authenticate SSE-C for object attributes
Unseal the supplied customer key after request preconditions before returning object size, ETag, checksum, or part attributes. Cover zero-byte and non-empty objects with correct, wrong, and missing keys.\n\nRefs #84.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:26:37 +08:00
Feng Ruohang fb406fdc94 fix: report site replication metadata per site
Count only each site own valid bucket metadata, populate quota totals, and keep malformed fields from suppressing unrelated bucket statistics. Emit bounded diagnostics for invalid payloads.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:25:11 +08:00
Feng Ruohang dd3bdb8086 fix: preserve bucket configs during site adoption
Keep existing Object Lock and enabled versioning documents and timestamps when adopting a same-name bucket. Bootstrap missing configs, enable suspended versioning, and retain custom excluded-prefix settings.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:24:27 +08:00
Feng Ruohang 7c103389f5 fix: reject unsupported checksum assertions
Reject unimplemented x-amz-checksum value and trailer names instead of accepting uploads without verification. Apply the same contract to PutObject, multipart initiation and parts, CopyObject, and UploadPartCopy while preserving the five supported algorithms.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 17:23:26 +08:00
Feng Ruohang 3861f33cba fix: replicate object lock config in its own field
Populate ObjectLockConfig for live, initial-sync, and heal events, while accepting the legacy Tags field during rolling upgrades. Exercise signed admin dispatch and remote-heal transport on both object-layer backends.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:54:00 +08:00
Feng Ruohang 07d92db523 fix: skip bucket CORS lookup without an origin
Bypass per-bucket metadata work for non-CORS traffic, including admin and Console routes. Keep operational metadata errors fail-closed and pin the existing global fallback for missing buckets.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:34:51 +08:00
Feng Ruohang 5732930102 test: cover null-version copy rewrite directions
Exercise the silent compression-on-copy path, compressed SSE-C re-encryption, and the equal-invalid-key error contract on both object-layer backends.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:28:42 +08:00
Feng Ruohang 2aea7fe9c4 fix: preserve named targets in config environment files
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:00:21 +08:00
Feng Ruohang ffb70eb373 fix: re-encrypt a key rotation the object layer has to rewrite
A key rotation rewraps the object key held in metadata; it never re-encrypts
the stored bytes. CopyObjectHandler took that shortcut whenever the request
looked like a same-object SSE-C rotation, on the assumption that the object
layer would then leave the stored bytes alone. That is the same assumption
copyRewritesObjectData() was added to stop making.

When the source is a null version on a bucket that gained versioning after the
object was written, the object layer cannot reference that version and falls
back to PutObject. The reader at that point holds plaintext decrypted with the
old key and no EncryptFn is set, so the destination ends up storing plaintext
under metadata that claims the object is SSE-C encrypted. A subsequent GET
failed with "sio: unsupported version".

Gate the rotation shortcut on the same prediction the compression metadata
already uses. When the object layer stores new object data the rotation falls
through to the regular re-encrypting copy, which decrypts with the old key and
re-encrypts with the new one. The source version selection moves next to the
gate because both decisions need it.

That fallback authenticates the source key through the source decryptor, which
GetObjectNInfo does not build for a zero byte object. Check the key explicitly
before the destination is written, so the gate cannot turn a rotation that the
shortcut rejected with AccessDenied into one that succeeds. The re-encrypting
copy regenerates the encrypted ETag, unlike an in-place rotation; the test
records that difference.

The other three object layer CopyObject callers that set metadataOnly -
PostRestoreObjectHandler, updateRestoreMetadata and batchKeyRotate - address
the same version on both sides and never set Versioned, so they only reach the
two in-place cases already covered by the copyRewritesObjectData table.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fk3PAD7DHCYzcyegYWqAmt
2026-08-29 16:00:21 +08:00
Feng Ruohang e73436c99d fix: decrypt CopyObject checksums with destination key
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:00:21 +08:00
Feng Ruohang 0b0ae2423a fix: keep copy metadata consistent with a rewritten null version
CopyObjectHandler recorded the source compression metadata whenever the copy
was metadata-only, on the assumption that the object layer would then leave
the stored bytes alone. That assumption does not hold. Both
erasureServerPools.CopyObject and erasureSets.CopyObject only skip a data
rewrite in three cases, and otherwise fall back to a full PutObject.

The reachable gap is a copy whose source is a null version on a bucket that
gained versioning after the object was written. Neither version ID is set, so
the self-referential version branch is skipped, the data is rewritten as
plaintext, and the preserved compression metadata then described bytes that
no longer exist. A subsequent GET failed with "s2: corrupt input".

Mirror the object layer's decision in copyRewritesObjectData and record the
compression metadata from it, so the metadata always describes whichever
bytes are finally stored. The source version selection that lets a versioned
metadata-only copy add a self-referential version moves next to the same
decision, since both depend on the effective metadata-only value.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 16:00:21 +08:00
Feng Ruohang 7e079ff05c fix: validate explicit multipart checksum type
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:00:21 +08:00
Feng Ruohang 229fe2b3c3 fix: authorize group status changes by target status
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:00:21 +08:00
Feng Ruohang 47cd7807d3 test: align federated client version
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 16:00:21 +08:00
Feng Ruohang 4bb8c813ac test: cover asymmetric CORS site counts
Exercise no-site, local-only, remote-only, and both-site CORS summary accounting through the real siteReplicationStatus seam.

Refs #75

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 10:03:01 +08:00
Feng Ruohang 0eebc928f7 fix: complete bucket CORS protocol validation
Integrate the strict B3 XML, validation, checksum, wildcard, MaxAge, and Origin-null response contract with the C-prime site-replication register from #75.

Preserve fail-closed metadata behavior and rejected-preflight cache variation while keeping legacy-invalid development metadata readable and repairable through a valid CORS PUT or DELETE.

Add combined parser, handler, browser-response, namespace, replication, restart, and legacy-repair regressions, and update the internal design contract.

Refs #75

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 09:10:58 +08:00
Feng Ruohang 724f8703d8 fix: make bucket CORS replication converge
Define a deterministic CORS replication register with durable tombstones, strict source timestamps, equal-time conflict ordering, full-state status, and heal convergence.

Serialize local and peer CORS transitions with a distributed namespace lock, validate canonical transport payloads, preserve initial-sync deletes, and fail closed on metadata errors.

Add adversarial, concurrent, restart, status, heal, signed admin-dispatch, protocol, and middleware coverage together with the reviewed site-replication design record.

Refs #75

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-29 01:33:24 +08:00
Feng Ruohang e4e3007da6 Merge pull request #71 from h5vx/feature/per-bucket-cors
feat: per-bucket CORS configuration with S3 ?cors enforcement\n\nRelease hardening and site-replication convergence follow-up: #75.
2026-08-28 10:34:52 +08:00
h5vx 13e6458d90 feat: replicate per-bucket CORS across sites and harden the protocol path
Site replication emitted SRBucketMetaTypeCorsConfig on PutBucketCors, but
the peer receive/apply, initial-sync, status, and heal paths did not carry
the CORS metadata. Replicated sites could therefore diverge on CORS config
even though the originating request succeeded.

Complete every site-replication path for CORS, mirroring the SSEConfig
pattern:
  - peer apply: PeerBucketCorsConfigHandler + item.Cors handling in
    PeerBucketMetadataUpdateHandler, with an updatedAt staleness guard
  - initial sync: push existing CorsConfigXML via BucketMetaHook
  - status: parse per-site CorsConfig, count/compare, surface
    CorsCfgMismatch/HasCorsCfgSet/ReplicatedCorsConfig, and include CORS in
    the bucket-stats aggregation filter
  - heal: healCORSMetadata, including nil -> delete propagation

Also harden the request/config path:
  - PutBucketCors validates the supplied Content-MD5/checksum via
    validateLengthAndChecksum
  - CORS validation rejects more than one wildcard per AllowedOrigin/
    AllowedHeader and enforces the 255-char rule ID limit
  - preflight responses Vary on Origin, Access-Control-Request-Method, and
    Access-Control-Request-Headers

Add focused tests for the CORS SR transport round-trip, the metadata
equality helper, and the new validation constraints.

Signed-off-by: h5vx <h5v@protonmail.com>
2026-08-27 12:46:47 +05:00
Feng Ruohang 5d152416de fix: align multipart completion checksum errors
Return AWS-compatible errors for CompleteMultipartUpload checksum failures without changing the global streaming checksum mapping. Compare explicit multipart checksum types symmetrically, distinguish missing composite part checksums, and preserve the CRC64NVME canonicalization pending a direct AWS probe.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-08-27 09:05:49 +08:00