Commit Graph

12955 Commits

Author SHA1 Message Date
Feng Ruohang 3ce8319251 fix(storage): preserve tags during pool reconciliation
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 19:26:58 +08:00
Feng Ruohang 9df0f4abaf Merge pull request #188 from pgsty/codex/remove-access-tiering-final
Remove access-frequency pool tiering and preserve independent multi-pool correctness fixes. Reconcile ordinary addressed-version DELETE across pools, retaining existing quorum and compatibility boundaries.

Verified delivery head: 4d0693cb8c. All 11 final CI checks and three qualified Linux upgrade runs passed. Introduction, retirement decisions and historical unresolved observations are documented.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 15:09:48 +08:00
Feng Ruohang 4d0693cb8c docs: record controlled retirement evidence and qualified upgrade runs
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 14:56:20 +08:00
Feng Ruohang bf59e3f222 docs: record retirement execution and blocked Linux acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 12:16:45 +08:00
Feng Ruohang 41aa846097 style(storage): satisfy callback selection lint rule
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:31:32 +08:00
Feng Ruohang 4093fa0d78 docs: record access tiering introduction and retirement decisions
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:02 +08:00
Feng Ruohang 13bf126ebd fix(storage): reuse resolved copies for version DELETE callbacks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:02 +08:00
Feng Ruohang 1cf529ce8a fix(storage): reconcile ordinary version DELETE across pools
Delete every copy of an explicitly addressed UUID, null version or delete
marker under the pool lock. Preserve retention and replication callbacks,
report unreadable pools and cleanup failures, and keep movement, incoming
replication, expiration and free-version cleanup on their existing paths.

Retain the separately developed general DELETE repair and replace its
access-mover-only coverage with a real interrupted rebalance copy followed
by HTTP deletion. Cover unqualified directory-marker DELETE and document
the existing pool-order-dependent 503 behavior that this makes consistent.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:01 +08:00
Feng Ruohang 9b76a21675 revert: remove access-frequency ILM tiering (#60)
Reverse the first-parent diff of a3df317ae0,
including the feature branch compatibility and mover follow-up fixes.
Retain the independent multi-pool correctness fixes from #178 and migrate
their shared test fixture away from access-tier code.

Tolerate retired ILM keys and XML, read old v9 statistics while writing v8,
and document migration without moving objects or rewriting their metadata.
Include regression coverage using a historical scanner/writer v9 fixture.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:01 +08:00
Feng Ruohang 89637554d6 Merge pull request #182 from pgsty/codex/docs-current-state-20260913
docs: align release notes and current component status
2026-09-13 10:50:22 +08:00
Feng Ruohang 2dd1e00da4 docs: align release notes and current component status
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 10:39:57 +08:00
Feng Ruohang 5d955b5b74 Merge pull request #181 from pgsty/codex/deps-release-20260913
Coordinate September dependency releases and replace vulnerable bundled curl
2026-09-13 10:14:57 +08:00
Feng Ruohang f7808a172c deps: pin the coordinated September 13 SILO stack
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:54:51 +08:00
Feng Ruohang acc9b514f5 Follow the curl builder rename in delivery verification
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:08:38 +08:00
Feng Ruohang 31ba01c5d5 Refresh Go dependencies and build current static curl for both architectures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:01:09 +08:00
Feng Ruohang 48ec10312f Merge pull request #180 from pgsty/codex/issue-77-metadata-convergence
fix(replication): preserve bucket metadata source times and converge deletions
2026-09-12 18:07:36 +08:00
Feng Ruohang 114dc10529 docs: archive issue 77 design, adversarial reviews and acceptance evidence
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:52:25 +08:00
Feng Ruohang 461e9a7210 test(replication): satisfy diagnostic regression style checks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:17:18 +08:00
Feng Ruohang fcbb93e895 fix(replication): diagnose unusable metadata without a heal source
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:09:34 +08:00
Feng Ruohang 62cf066ff5 fix(replication): recover physical creation time and align policy status
An independent adversarial review of the bucket metadata convergence
work found three defects it had introduced.

GetBucketInfo overwrote the physical creation probe with cached
metadata, which a bucket that never held a configuration legitimately
lacks. The new creation-time requirement then failed every policy, tag,
SSE, quota, versioning and Object Lock write on such a bucket, with no
operator recovery path, and initial synchronization skipped it silently.
Return the physical result unchanged when metadata is not requested, as
ListBuckets already does, recover the time during initial
synchronization, and pass it to MakeBucketHook so peers adopt the same
bucket generation.

Replication status compared parsed policies statement by statement while
heal compares the canonical key. An upgraded peer that stored an
equivalent statement order was therefore reported as mismatched forever,
and heal never had anything to write. Compare the key heal compares;
per-site presence counting is unchanged.

Heal diagnostics shared one log key across four conditions, so a real
peer RPC failure could be deduplicated away by an earlier message, and
they were logged at error level for the normal transient of a peer that
does not have the bucket yet. Give each reason its own key at warning
level, report only a field state that exists and still cannot be
ordered, and diagnose nothing when no site holds a state to propagate.

The recovery test now runs against the real ObjectLayer; the stub it
replaced returned the expected time and hid the defect. The policy
status test uses a statement order the canonical encoder reorders, and
adoption coverage is extended past a real field time.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 14:01:35 +08:00
Feng Ruohang 1ee64a8d89 fix(replication): gate metadata tombstone export during rolling upgrades
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang 01aaef2b50 fix(replication): converge bucket metadata using deterministic source states
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang bcc62afe3d fix(replication): apply bucket metadata source times under the metadata lock
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang 5c57658163 Merge pull request #179 from pgsty/codex/federation-copy-fixes
fix(federation): preserve committed copy times and reject raw SSE-C replicas
2026-09-12 02:00:57 +08:00
Feng Ruohang f175e98c34 fix(federation): bind copy timestamps to committed writes
Return the committed object or part time on federation write responses and
capture it for CopyObject and UploadPartCopy without a follow-up read.
Keep successful writes compatible with targets that do not supply a time.

Reject authenticated raw SSE-C replica CopyObject across deployments before
forwarding. Extend the existing SSE fixtures to verify empty objects,
stored checksums, KMS contexts and multipart sources.

Refs: #169, #168, #171
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 01:44:22 +08:00
Feng Ruohang 12f631b502 Merge pull request #178 from pgsty/codex/multipool-correctness-20260911
fix(storage): reconcile multi-pool writes and conditional deletes
2026-09-11 20:34:58 +08:00
Feng Ruohang ccb676e60c fix(storage): preserve shared tier references during pool cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:23:40 +08:00
Feng Ruohang 51d41345f7 test: record Linux restart and OIDC release acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:05:56 +08:00
Feng Ruohang e59a3d938e fix(storage): serialize and reconcile multi-pool object updates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:05:56 +08:00
Feng Ruohang b32f2d9dd0 Merge pull request #177 from pgsty/codex/release-consolidation-20260911
Fix signed payloads, Object Lock and federated copies; secure AMQP
2026-09-11 17:08:34 +08:00
Feng Ruohang d63c92e393 build(deps): secure AMQP frames and select merged Console
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:56:33 +08:00
Feng Ruohang 68127c5a63 build(deps): embed the consolidated Console source
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:40:42 +08:00
Feng Ruohang c4b5e1cb45 fix(auth): enforce header-only presigned payload checksums
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang 9a303f5096 fix(object): preserve retention and federated copy destination state
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang 87d8b5967f fix(auth): align signed request and policy condition semantics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang f760046c44 Merge remote-tracking branch 'origin/main' into codex/release-consolidation-20260911
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:13:42 +08:00
Feng Ruohang 93e7ef4bcc Merge pull request #175 from pgsty/codex/upstream-sdk-password-20260910
fix(iam)!: split self-service password permissions and update SDK stack
2026-09-10 17:51:08 +08:00
Feng Ruohang a4229b366f build(deps): align final coordinated SILO source pins
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 17:39:01 +08:00
Feng Ruohang 420340bc14 docs(iam): explain breaking password-policy semantics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 16:58:21 +08:00
Ayush Sharma e7e87402ed fix: forward the legal hold as an Object Lock header on federated CopyObject
A cross-deployment CopyObject that requests
`x-amz-object-lock-legal-hold: ON` answered 200 while the destination
carried no hold. The resolved value reached the remote as ordinary user
metadata, `X-Amz-Meta-X-Amz-Object-Lock-Legal-Hold`, so nothing applied it.
Retention requested on the same copy survived, which is what made the loss
easy to miss.

The federation branch passes the resolved metadata map straight to
`Core.PutObject` as `PutObjectOptions.UserMetadata`. minio-go's `Header()`
writes the typed lock fields first, then prefixes every UserMetadata key it
does not recognise with `x-amz-meta-`; `supportedHeaders` covers
`x-amz-object-lock-mode` and `x-amz-object-lock-retain-until-date` but not
`x-amz-object-lock-legal-hold`, and `isAmzHeader` does not match it either.
Retention therefore arrives as real headers and the hold does not. The
high-level `validate()` that would have rejected the key never runs, because
`Core.PutObject` goes straight to the low-level PUT.

Carry the hold on the typed `LegalHold` option and forward a cloned map with
the raw key removed. The clone matters twice: typed fields are written before
the UserMetadata loop, so a leftover raw key would add a bogus `x-amz-meta-`
entry beside the correct header, and the proxy's own response and event
metadata are rebuilt from the resolved values rather than the forwarding map,
which no longer carries the hold.

Retention stays in the map deliberately. It already passes through as a
standard header, and moving it to the typed `RetainUntilDate` field would
format with `time.RFC3339` and truncate a retain-until date to whole seconds.

The new test asserts the wire: the remote must receive
`X-Amz-Object-Lock-Legal-Hold` and never the `x-amz-meta-` spelling, and the
destination version must actually store the hold. It fails without the change
with "legal hold forwarded as user metadata [ON]".

Fixes #166

Signed-off-by: Ayush Sharma <72848455+Aeirx@users.noreply.github.com>
2026-09-10 13:15:10 +05:30
Feng Ruohang a164e1dda1 fix(iam): separate password changes and refresh coordinated SDK dependencies
Use ChangeMyPassword for the authenticated user and keep CreateUser for other users. Coordinate silo-pkg b3760f56ec23, mcli fa22b40b4eb7, Console 1b95b6cec652 and upstream minio-go 78bfa91607c2. Add legacy-policy and SDK streaming regressions plus upgrade guidance.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 15:16:21 +08:00
Feng Ruohang 2f61325d4a Merge pull request #174 from pgsty/codex/contributor-orenyomtov-20260910
docs: credit @orenyomtov for the SN-2026-011 report
2026-09-10 11:34:52 +08:00
Feng Ruohang a6145e1d2e docs: credit @orenyomtov for the SN-2026-011 report
Add Oren Yomtov (github.com/orenyomtov) to the contributor wall in
README.md, README_ZH.md, and CONTRIBUTORS.md, and to the Issue reports
table, for the private disclosure of the unsigned-header CopyObject
cross-object read fixed as SN-2026-011 (#173). Community contributor
count 40 -> 41.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 11:24:12 +08:00
Feng Ruohang 5232546690 Merge pull request #173 from pgsty/codex/unsigned-amz-header-copy-20260909
fix(auth): reject unsigned x-amz-* headers to close CopyObject confused-deputy (SN-2026-011)
2026-09-10 10:10:49 +08:00
Feng Ruohang 0c46cb641b docs(security): record SN-2026-011 (unsigned x-amz-* header CopyObject)
Ledger entry for the confused-deputy fix in 123325430: an unsigned
x-amz-copy-source header turned a presigned or signed PUT into a
server-side copy of any object the signing key can read. Reported by
Oren Yomtov; inherited from upstream minio/minio; CVE requested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 09:57:26 +08:00
Feng Ruohang 1233254309 fix(auth): reject unsigned x-amz-* headers to close CopyObject confused-deputy
A presigned or signed PUT authorized for a single object could be turned
into a server-side copy of any object the signing key can read by adding
an unsigned x-amz-copy-source header, executed as the signer. SigV4
verification only walked the signed-headers list, never the headers that
actually arrived; the meta-header check matched only X-Amz-Meta- and ran
only on the presigned path, so an unsigned x-amz-* header outside the
list was never seen while the router still dispatched the PUT to
CopyObjectHandler.

Reject any x-amz-* request header not covered by the signed headers, on
both the presigned (doesPresignedSignatureMatch) and Authorization-header
(doesSignatureMatch) paths, matching AWS S3. The check tests membership
in the signed set rather than value equality, so a header whose first
value is empty (e.g. {"", "/src/secret"}) cannot slip through.
X-Amz-Content-Sha256 is exempt (payload hash: read from the query for
presigned requests and bound into the string-to-sign for signed ones, so
it is self-protected) and X-Amz-Signature-Age is exempt (an internal
scratch header written after verification, so repeated verification of
the same request stays idempotent). The synthesized X-Amz-Tagging header
in PutObjectTagging is now injected after signature verification.

Tests that previously added x-amz-copy-source and friends after signing
(relying on the vulnerable behavior) now re-sign, mirroring real S3
clients. Adds checkUnsignedHeaders unit cases and TestPresignedVerifyIdempotent.

Reported by Oren Yomtov. Inherited unchanged from upstream minio/minio.
Tracked as SN-2026-011.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 09:57:01 +08:00
Feng Ruohang 8a2fe9b7a0 Merge pull request #164 from pgsty/codex/main-consolidation-20260909
fix: honor TLS defaults and accept valid bucket metadata reloads
2026-09-09 19:49:08 +08:00
Feng Ruohang f26ee6bf0a test: cover metadata reload with equal maximum timestamps
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:35:13 +08:00
Feng Ruohang d69c4ccfe4 Merge TLS default key exchange compatibility fixes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:31:35 +08:00
Feng Ruohang 086e619505 Merge accepted bucket metadata reload correction
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:31:10 +08:00