Compare commits

...

197 Commits

Author SHA1 Message Date
Feng Ruohang bf053386a4 docs: link compression and federation to maintained design records
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 18:26:03 +08:00
Feng Ruohang f99ed829b5 Merge pull request #213 from pgsty/codex/pr198-release-compat
fix: preserve multipart defaults and rollback compatibility
2026-09-16 16:42:56 +08:00
Feng Ruohang 956a1a8e33 Merge pull request #212 from pgsty/codex/docs-migration-cleanup
docs: consolidate repository entry points and retire migrated work records
2026-09-16 16:39:40 +08:00
Feng Ruohang 82f0a9828e fix: preserve multipart defaults and rollback compatibility
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:30:43 +08:00
Feng Ruohang 584b5a3a8f docs: link the canonical recovery runbooks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:19:56 +08:00
Feng Ruohang 39f49d548f Merge remote-tracking branch 'origin/main' into codex/docs-migration-cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>

# Conflicts:
#	CONTRIBUTORS.md
2026-09-16 16:13:31 +08:00
Feng Ruohang a168576adb Merge pull request #208 from pgsty/codex/conditional-put-release-notes
docs: describe conditional PUT behavior and tested recovery guidance
2026-09-16 16:09:22 +08:00
Feng Ruohang e85c4c8dfb docs: preserve reviewed contributor records and daily card references
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:09:00 +08:00
Feng Ruohang 0e3c43778e Merge pull request #211 from pgsty/codex/daily-repository-cards
ci: update README repository cards daily at 00:00 UTC
2026-09-16 15:40:09 +08:00
Feng Ruohang 9101fe78db ci: refresh README repository cards daily at 00:00 UTC
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 15:27:48 +08:00
Feng Ruohang 82948d6306 Merge pull request #198 from mrjavadseydi/fix/issue-79-list-multipart-uploads
Retain the original contributor commit and integrate the reviewed durable listing, cancellation confirmation and migration fixes. Capacity acceptance and delayed creation-write fencing remain open in issue #79.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:40:10 +08:00
Feng Ruohang dfb4b2a1d2 fix: return HTTP 503 when multipart scan capacity is exhausted
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:24:52 +08:00
Feng Ruohang 143f6970d8 fix: make multipart discovery and cancellation explicit and bounded
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:16:37 +08:00
Feng Ruohang ea5dac5a99 Merge current main into multipart listing contribution
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 12:51:20 +08:00
Feng Ruohang 3c26a8b0b5 Merge pull request #209 from pgsty/codex/credit-console-share-reporter
fix: integrate the bounded Console sharing proxy and credit its reporter
2026-09-16 12:19:29 +08:00
Feng Ruohang 2fabd436c0 fix: select the bounded Console sharing proxy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 12:05:00 +08:00
Feng Ruohang 07c68d6054 docs: credit Jiri Pejchal for the Console sharing report
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:48:38 +08:00
Feng Ruohang 15090dc4fd docs: retire migrated working material and clarify documentation ownership
Remove migrated investigation and security documents, repair their incoming links, and align the English/Chinese READMEs with current module and release boundaries. Track AGENTS.md as the shared repository guide and make CLAUDE.md import it; keep working artifacts outside the repository.

Publish pgsty/silo.pgsty.com commit c7682185e2832b981629e1c17aef74fc778c6ca1 before publishing this cleanup: the new documentation routes are not live yet.

Validation: make rebrand-guard; 92 Markdown files checked for deletion-induced broken relative paths; 199 source-to-site references and anchors resolve in the paired site build; git diff --check.
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:22:48 +08:00
Feng Ruohang e791640dac docs: describe merged conditional PUT behavior and recovery guidance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:21:16 +08:00
Feng Ruohang 9b4ae82a29 Merge pull request #207 from pgsty/codex/conditional-put-pools
fix(pools): evaluate cross-pool PUT conditions against the current object
2026-09-16 11:17:02 +08:00
Feng Ruohang 4620be394b test: synchronize conditional PUT disk fixtures
Replace unsynchronized getDisks swaps with backing disk-list updates under
erasureDisksMu, matching the existing GetDisks reader lock. Apply the same
helper to capacity and read-fault adapters while preserving nested restore
ordering.

Add a regression that overlaps fixture changes with the real IAM Walk
reader, and run the conditional PUT suite under the race detector in CI.
The regression reproduces the old fixture race; ten fixed race iterations
pass without warnings. Production conditional PUT behavior is unchanged.

Refs #199

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 10:53:24 +08:00
Feng Ruohang 5e7d603083 fix(pools): evaluate cross-pool PUT conditions against the current object
Multi-pool PUT selected a destination by capacity and evaluated
If-Match/If-None-Match only against that destination's local object
state. An empty or stale destination could accept a stale ETag or
If-None-Match:* while another pool held the current object, replacing
it; a current ETag could instead be rejected with 412 or 404.

Under PUT's existing pools-layer object lock, resolve the comparison
object with objectPoolInfos (including draining pools), treat a latest
delete marker as absence, fail closed on unreadable pool metadata, and
clear an accepted callback before destination dispatch. Replica and
data-movement callbacks keep their addressed-version semantics and
metadata reconciliation.

Reproduced on 40220bd836 and RELEASE.2026-09-03T13-18-01Z with six
signed HTTP scenarios: four defect cases failed, two controls passed.

Refs #199

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:21:21 +08:00
Feng Ruohang fb7c406ddc Merge pull request #206 from pgsty/codex/main-consolidation-20260916
test: restore valid credentials in integration fixtures
2026-09-16 08:18:56 +08:00
Feng Ruohang 416826f61f Merge pull request #205 from pgsty/codex/release-notes-followups
docs: complete September correctness and upgrade notes
2026-09-16 08:16:23 +08:00
Feng Ruohang a2e2f3ee82 test: restore valid credentials in integration fixtures
Port the shell-fixture changes from ebc9937d97b27871dcc4bb91d4b5771d3550b76a. Match the existing minimum secret length consistently across startup, aliases and helper commands.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:08:08 +08:00
Feng Ruohang 70c7ec4a9f docs: link reviewed runbook sources before website deployment
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:04:53 +08:00
Feng Ruohang 2b722b7e92 docs: complete September correctness and upgrade notes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 07:51:11 +08:00
mr javad seydi 4cbb074ccd fix: make multipart upload listing S3-compatible
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-15 23:17:55 +03:30
Feng Ruohang 40220bd836 Merge pull request #196 from pgsty/codex/merge-r4-r8
Complete R5, R6 and R8 on the main baseline containing R4 and R7.

Preserve the individual signed repairs, independent Opus 5 Max review, and integration validation evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 01:14:06 +08:00
Feng Ruohang df0dfa0a34 docs: record R4-R8 integration review and validation
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:58:31 +08:00
Feng Ruohang 80684fed59 chore: align integrated repair tests with contribution checks
Use the actual author and AGPL-3.0-or-later notices for new R6/R8 tests, preserving all test bodies and the Linux build tag. Record the existing replication ARN prefix used by the new R6 fixture in the compatibility inventory; no wire behavior changes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:44:34 +08:00
Feng Ruohang 055030ea53 fix(http): honor configured request header deadlines
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 0d48d32d7e038ae1ea5966f3d7e0cb86780a6311)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:56 +08:00
Feng Ruohang aea3882c95 docs: record R6 integration verification
(cherry picked from commit d38edb2c46182d3a8fa96e040604493d20a4b478)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:56 +08:00
Feng Ruohang 0c61128d23 fix(replication): retry marker purges through persisted MRF
(cherry picked from commit cf381a7151ef25fc95ace5fedcd767fa19410de2)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:55 +08:00
Feng Ruohang 680eac66e4 fix(replication): preserve ordered tag deletions
Persist, send and reconcile empty tag states together with their revision
across COPY, PUT and multipart replication. Advance local tag mutations
under the existing locks and preserve current tags during replication ACK.

Cover signed HTTP, persistent single/multiple pool state, KMS, SSE-C key
rotation, ordering, retry and duplicate requests. Record real Opus plan
consensus, implementation review and local verification evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 115fe8b12329d147adbaf817faa1737392ecbf9b)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:55 +08:00
Feng Ruohang 9f3037e941 Merge pull request #194 from pgsty/codex/r7-replication-content-encoding
fix(replication): preserve normalized replica metadata
2026-09-16 00:31:03 +08:00
Feng Ruohang 5f00f6762f Merge main after R4 validation into R7 candidate
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:21:21 +08:00
Feng Ruohang af2b1794d3 Merge pull request #193 from pgsty/codex/r4-kms-tag-timestamp
fix(replication): preserve SSE-KMS tag timestamps
2026-09-16 00:20:10 +08:00
Feng Ruohang d371f77dcc docs: record final Opus 5 Max R7 implementation review
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:10:31 +08:00
Feng Ruohang 022722a7a7 chore: align R4 contribution notices and record final review
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:08:09 +08:00
Feng Ruohang 03027727d1 fix(replication): preserve SSE-KMS tag timestamps
Carry the already-parsed source tagging timestamp through the KMS options
constructor so replica COPY can apply newer tag updates on explicitly or
automatically encrypted destinations.

Cover all option encryption modes and signed COPY persistence for newer,
stale, duplicate and timestamp-less updates, including bucket defaults.
Preserve the real Opus 5.0/max plan review, consensus and local validation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:03:52 +08:00
Feng Ruohang 4fcdf37ce6 fix(replication): preserve normalized replica metadata
Restore only the six replication-specific metadata fields after trust
validation, so streaming uploads retain their actual content encoding and
Snowball entries do not inherit ordinary metadata from the outer archive.

Include helper, authenticated PUT/COPY/multipart and Snowball regressions,
plus the R7 investigation, actual Opus 5 consensus and local verification.
The production change is based on PR #187 by Mikhail Khadarenka.

Co-authored-by: Mikhail Khadarenka <chodorenko@gmail.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:00:17 +08:00
Feng Ruohang 9ebe81c1b3 Merge pull request #192 from pgsty/codex/iam-revision-tombstones
fix(iam): retain revocation versions through replay and recovery
2026-09-15 23:14:22 +08:00
Feng Ruohang e5f5c9e7f6 Merge pull request #191 from pgsty/codex/iam-peer-delete-reload
fix(iam): reload committed state on peer deletion notifications
2026-09-15 23:10:38 +08:00
Feng Ruohang 7b4cacc392 chore: align IAM file notices with contribution policy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:59:04 +08:00
Feng Ruohang a0dd7dae9b fix(iam): resume site healing after leadership changes
Keep one healing loop per process across replication configuration reloads. Reacquire leadership after a lease is canceled and allow shutdown while waiting, so temporary quorum loss cannot permanently stop revocation propagation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:59:04 +08:00
Feng Ruohang 709d50a916 fix(iam): persist revocations across site replay and recovery
Retain source-ordered tombstones and parent grant boundaries across both IAM backends, cache reloads, and deliberate identity recreation. Reconcile deletions through a versioned, bounded replication protocol with restart-aware acknowledgements.

Cover inherited group grants, STS retention, same-key service recreation, absolute expiration, and failures after the durable commit. Document coordinated upgrades and the remaining consistency boundaries.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:59:04 +08:00
Feng Ruohang dff81f293b chore: align IAM test notice with contribution policy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:58:52 +08:00
Feng Ruohang f653a6ea03 fix(iam): reload committed state on peer deletion notifications
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:58:52 +08:00
Feng Ruohang 47d239f84f Merge pull request #190 from pgsty/codex/fix-pool-multipart-preconditions
fix(storage): evaluate multipart preconditions across pools
2026-09-15 21:57:26 +08:00
Feng Ruohang e069fe9d92 fix(storage): evaluate multipart preconditions across pools
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 21:13:37 +08:00
Feng Ruohang d848fb52b5 Merge pull request #189 from pgsty/codex/fix-pool-tag-reconciliation
fix(storage): preserve tags during pool reconciliation
2026-09-15 19:38:19 +08:00
Feng Ruohang 3ce8319251 fix(storage): preserve tags during pool reconciliation
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 19:26:58 +08:00
Feng Ruohang 9df0f4abaf Merge pull request #188 from pgsty/codex/remove-access-tiering-final
Remove access-frequency pool tiering and preserve independent multi-pool correctness fixes. Reconcile ordinary addressed-version DELETE across pools, retaining existing quorum and compatibility boundaries.

Verified delivery head: 4d0693cb8c. All 11 final CI checks and three qualified Linux upgrade runs passed. Introduction, retirement decisions and historical unresolved observations are documented.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 15:09:48 +08:00
Feng Ruohang 4d0693cb8c docs: record controlled retirement evidence and qualified upgrade runs
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 14:56:20 +08:00
Feng Ruohang bf59e3f222 docs: record retirement execution and blocked Linux acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 12:16:45 +08:00
Feng Ruohang 41aa846097 style(storage): satisfy callback selection lint rule
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:31:32 +08:00
Feng Ruohang 4093fa0d78 docs: record access tiering introduction and retirement decisions
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:02 +08:00
Feng Ruohang 13bf126ebd fix(storage): reuse resolved copies for version DELETE callbacks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:02 +08:00
Feng Ruohang 1cf529ce8a fix(storage): reconcile ordinary version DELETE across pools
Delete every copy of an explicitly addressed UUID, null version or delete
marker under the pool lock. Preserve retention and replication callbacks,
report unreadable pools and cleanup failures, and keep movement, incoming
replication, expiration and free-version cleanup on their existing paths.

Retain the separately developed general DELETE repair and replace its
access-mover-only coverage with a real interrupted rebalance copy followed
by HTTP deletion. Cover unqualified directory-marker DELETE and document
the existing pool-order-dependent 503 behavior that this makes consistent.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:01 +08:00
Feng Ruohang 9b76a21675 revert: remove access-frequency ILM tiering (#60)
Reverse the first-parent diff of a3df317ae0,
including the feature branch compatibility and mover follow-up fixes.
Retain the independent multi-pool correctness fixes from #178 and migrate
their shared test fixture away from access-tier code.

Tolerate retired ILM keys and XML, read old v9 statistics while writing v8,
and document migration without moving objects or rewriting their metadata.
Include regression coverage using a historical scanner/writer v9 fixture.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:01 +08:00
Feng Ruohang 89637554d6 Merge pull request #182 from pgsty/codex/docs-current-state-20260913
docs: align release notes and current component status
2026-09-13 10:50:22 +08:00
Feng Ruohang 2dd1e00da4 docs: align release notes and current component status
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 10:39:57 +08:00
Feng Ruohang 5d955b5b74 Merge pull request #181 from pgsty/codex/deps-release-20260913
Coordinate September dependency releases and replace vulnerable bundled curl
2026-09-13 10:14:57 +08:00
Feng Ruohang f7808a172c deps: pin the coordinated September 13 SILO stack
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:54:51 +08:00
Feng Ruohang acc9b514f5 Follow the curl builder rename in delivery verification
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:08:38 +08:00
Feng Ruohang 31ba01c5d5 Refresh Go dependencies and build current static curl for both architectures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:01:09 +08:00
Feng Ruohang 48ec10312f Merge pull request #180 from pgsty/codex/issue-77-metadata-convergence
fix(replication): preserve bucket metadata source times and converge deletions
2026-09-12 18:07:36 +08:00
Feng Ruohang 114dc10529 docs: archive issue 77 design, adversarial reviews and acceptance evidence
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:52:25 +08:00
Feng Ruohang 461e9a7210 test(replication): satisfy diagnostic regression style checks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:17:18 +08:00
Feng Ruohang fcbb93e895 fix(replication): diagnose unusable metadata without a heal source
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:09:34 +08:00
Feng Ruohang 62cf066ff5 fix(replication): recover physical creation time and align policy status
An independent adversarial review of the bucket metadata convergence
work found three defects it had introduced.

GetBucketInfo overwrote the physical creation probe with cached
metadata, which a bucket that never held a configuration legitimately
lacks. The new creation-time requirement then failed every policy, tag,
SSE, quota, versioning and Object Lock write on such a bucket, with no
operator recovery path, and initial synchronization skipped it silently.
Return the physical result unchanged when metadata is not requested, as
ListBuckets already does, recover the time during initial
synchronization, and pass it to MakeBucketHook so peers adopt the same
bucket generation.

Replication status compared parsed policies statement by statement while
heal compares the canonical key. An upgraded peer that stored an
equivalent statement order was therefore reported as mismatched forever,
and heal never had anything to write. Compare the key heal compares;
per-site presence counting is unchanged.

Heal diagnostics shared one log key across four conditions, so a real
peer RPC failure could be deduplicated away by an earlier message, and
they were logged at error level for the normal transient of a peer that
does not have the bucket yet. Give each reason its own key at warning
level, report only a field state that exists and still cannot be
ordered, and diagnose nothing when no site holds a state to propagate.

The recovery test now runs against the real ObjectLayer; the stub it
replaced returned the expected time and hid the defect. The policy
status test uses a statement order the canonical encoder reorders, and
adoption coverage is extended past a real field time.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 14:01:35 +08:00
Feng Ruohang 1ee64a8d89 fix(replication): gate metadata tombstone export during rolling upgrades
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang 01aaef2b50 fix(replication): converge bucket metadata using deterministic source states
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang bcc62afe3d fix(replication): apply bucket metadata source times under the metadata lock
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang 5c57658163 Merge pull request #179 from pgsty/codex/federation-copy-fixes
fix(federation): preserve committed copy times and reject raw SSE-C replicas
2026-09-12 02:00:57 +08:00
Feng Ruohang f175e98c34 fix(federation): bind copy timestamps to committed writes
Return the committed object or part time on federation write responses and
capture it for CopyObject and UploadPartCopy without a follow-up read.
Keep successful writes compatible with targets that do not supply a time.

Reject authenticated raw SSE-C replica CopyObject across deployments before
forwarding. Extend the existing SSE fixtures to verify empty objects,
stored checksums, KMS contexts and multipart sources.

Refs: #169, #168, #171
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 01:44:22 +08:00
Feng Ruohang 12f631b502 Merge pull request #178 from pgsty/codex/multipool-correctness-20260911
fix(storage): reconcile multi-pool writes and conditional deletes
2026-09-11 20:34:58 +08:00
Feng Ruohang ccb676e60c fix(storage): preserve shared tier references during pool cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:23:40 +08:00
Feng Ruohang 51d41345f7 test: record Linux restart and OIDC release acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:05:56 +08:00
Feng Ruohang e59a3d938e fix(storage): serialize and reconcile multi-pool object updates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:05:56 +08:00
Feng Ruohang b32f2d9dd0 Merge pull request #177 from pgsty/codex/release-consolidation-20260911
Fix signed payloads, Object Lock and federated copies; secure AMQP
2026-09-11 17:08:34 +08:00
Feng Ruohang d63c92e393 build(deps): secure AMQP frames and select merged Console
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:56:33 +08:00
Feng Ruohang 68127c5a63 build(deps): embed the consolidated Console source
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:40:42 +08:00
Feng Ruohang c4b5e1cb45 fix(auth): enforce header-only presigned payload checksums
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang 9a303f5096 fix(object): preserve retention and federated copy destination state
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang 87d8b5967f fix(auth): align signed request and policy condition semantics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang f760046c44 Merge remote-tracking branch 'origin/main' into codex/release-consolidation-20260911
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:13:42 +08:00
Feng Ruohang 93e7ef4bcc Merge pull request #175 from pgsty/codex/upstream-sdk-password-20260910
fix(iam)!: split self-service password permissions and update SDK stack
2026-09-10 17:51:08 +08:00
Feng Ruohang a4229b366f build(deps): align final coordinated SILO source pins
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 17:39:01 +08:00
Feng Ruohang 420340bc14 docs(iam): explain breaking password-policy semantics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 16:58:21 +08:00
Ayush Sharma e7e87402ed fix: forward the legal hold as an Object Lock header on federated CopyObject
A cross-deployment CopyObject that requests
`x-amz-object-lock-legal-hold: ON` answered 200 while the destination
carried no hold. The resolved value reached the remote as ordinary user
metadata, `X-Amz-Meta-X-Amz-Object-Lock-Legal-Hold`, so nothing applied it.
Retention requested on the same copy survived, which is what made the loss
easy to miss.

The federation branch passes the resolved metadata map straight to
`Core.PutObject` as `PutObjectOptions.UserMetadata`. minio-go's `Header()`
writes the typed lock fields first, then prefixes every UserMetadata key it
does not recognise with `x-amz-meta-`; `supportedHeaders` covers
`x-amz-object-lock-mode` and `x-amz-object-lock-retain-until-date` but not
`x-amz-object-lock-legal-hold`, and `isAmzHeader` does not match it either.
Retention therefore arrives as real headers and the hold does not. The
high-level `validate()` that would have rejected the key never runs, because
`Core.PutObject` goes straight to the low-level PUT.

Carry the hold on the typed `LegalHold` option and forward a cloned map with
the raw key removed. The clone matters twice: typed fields are written before
the UserMetadata loop, so a leftover raw key would add a bogus `x-amz-meta-`
entry beside the correct header, and the proxy's own response and event
metadata are rebuilt from the resolved values rather than the forwarding map,
which no longer carries the hold.

Retention stays in the map deliberately. It already passes through as a
standard header, and moving it to the typed `RetainUntilDate` field would
format with `time.RFC3339` and truncate a retain-until date to whole seconds.

The new test asserts the wire: the remote must receive
`X-Amz-Object-Lock-Legal-Hold` and never the `x-amz-meta-` spelling, and the
destination version must actually store the hold. It fails without the change
with "legal hold forwarded as user metadata [ON]".

Fixes #166

Signed-off-by: Ayush Sharma <72848455+Aeirx@users.noreply.github.com>
2026-09-10 13:15:10 +05:30
Feng Ruohang a164e1dda1 fix(iam): separate password changes and refresh coordinated SDK dependencies
Use ChangeMyPassword for the authenticated user and keep CreateUser for other users. Coordinate silo-pkg b3760f56ec23, mcli fa22b40b4eb7, Console 1b95b6cec652 and upstream minio-go 78bfa91607c2. Add legacy-policy and SDK streaming regressions plus upgrade guidance.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 15:16:21 +08:00
Feng Ruohang 2f61325d4a Merge pull request #174 from pgsty/codex/contributor-orenyomtov-20260910
docs: credit @orenyomtov for the SN-2026-011 report
2026-09-10 11:34:52 +08:00
Feng Ruohang a6145e1d2e docs: credit @orenyomtov for the SN-2026-011 report
Add Oren Yomtov (github.com/orenyomtov) to the contributor wall in
README.md, README_ZH.md, and CONTRIBUTORS.md, and to the Issue reports
table, for the private disclosure of the unsigned-header CopyObject
cross-object read fixed as SN-2026-011 (#173). Community contributor
count 40 -> 41.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 11:24:12 +08:00
Feng Ruohang 5232546690 Merge pull request #173 from pgsty/codex/unsigned-amz-header-copy-20260909
fix(auth): reject unsigned x-amz-* headers to close CopyObject confused-deputy (SN-2026-011)
2026-09-10 10:10:49 +08:00
Feng Ruohang 0c46cb641b docs(security): record SN-2026-011 (unsigned x-amz-* header CopyObject)
Ledger entry for the confused-deputy fix in 123325430: an unsigned
x-amz-copy-source header turned a presigned or signed PUT into a
server-side copy of any object the signing key can read. Reported by
Oren Yomtov; inherited from upstream minio/minio; CVE requested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 09:57:26 +08:00
Feng Ruohang 1233254309 fix(auth): reject unsigned x-amz-* headers to close CopyObject confused-deputy
A presigned or signed PUT authorized for a single object could be turned
into a server-side copy of any object the signing key can read by adding
an unsigned x-amz-copy-source header, executed as the signer. SigV4
verification only walked the signed-headers list, never the headers that
actually arrived; the meta-header check matched only X-Amz-Meta- and ran
only on the presigned path, so an unsigned x-amz-* header outside the
list was never seen while the router still dispatched the PUT to
CopyObjectHandler.

Reject any x-amz-* request header not covered by the signed headers, on
both the presigned (doesPresignedSignatureMatch) and Authorization-header
(doesSignatureMatch) paths, matching AWS S3. The check tests membership
in the signed set rather than value equality, so a header whose first
value is empty (e.g. {"", "/src/secret"}) cannot slip through.
X-Amz-Content-Sha256 is exempt (payload hash: read from the query for
presigned requests and bound into the string-to-sign for signed ones, so
it is self-protected) and X-Amz-Signature-Age is exempt (an internal
scratch header written after verification, so repeated verification of
the same request stays idempotent). The synthesized X-Amz-Tagging header
in PutObjectTagging is now injected after signature verification.

Tests that previously added x-amz-copy-source and friends after signing
(relying on the vulnerable behavior) now re-sign, mirroring real S3
clients. Adds checkUnsignedHeaders unit cases and TestPresignedVerifyIdempotent.

Reported by Oren Yomtov. Inherited unchanged from upstream minio/minio.
Tracked as SN-2026-011.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 09:57:01 +08:00
Feng Ruohang 8a2fe9b7a0 Merge pull request #164 from pgsty/codex/main-consolidation-20260909
fix: honor TLS defaults and accept valid bucket metadata reloads
2026-09-09 19:49:08 +08:00
Feng Ruohang f26ee6bf0a test: cover metadata reload with equal maximum timestamps
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:35:13 +08:00
Feng Ruohang d69c4ccfe4 Merge TLS default key exchange compatibility fixes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:31:35 +08:00
Feng Ruohang 086e619505 Merge accepted bucket metadata reload correction
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:31:10 +08:00
Feng Ruohang 48e1846525 fix(tls): honor Go key exchange defaults across transports
Remove the eight explicit curve overrides so Go 1.27 honors tlsmlkem=0
across Server listeners, node links and outbound transports. Remove the
unused shared curve option and add wire-level regression coverage.

Document CA trust and TLS upgrade behavior, retain the investigation
artifacts, and exclude their synthetic routes from the rebrand guard.
The product compatibility baseline remains unchanged.

Validation: focused race tests, HTTP tests, lint, compatibility guard
positive/negative controls, and a fresh Linux build with three isolated
OIDC integration scenarios all pass.

Adversarial review: Claude Code Fable 5.1, max effort.
Final verdict: APPROVE FOR COMMIT.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 18:50:12 +08:00
Feng Ruohang bcc8871b1d Merge pull request #163 from pgsty/fix/issue-158-federated-copy-sse
fix: forward plaintext on federated CopyObject of SSE objects (#158)
2026-09-09 18:02:45 +08:00
Feng Ruohang 25cb3511c9 test: cover multipart SSE sources on federated CopyObject
A multipart SSE-S3 source is encrypted per part, so its logical size is
the sum of the parts' decrypted sizes and the decrypting reader crosses
a part boundary. Copy such a source across the federation to a plain
and to an SSE-S3 destination and check the destination plaintext and
single-encryption size.

The test router registers routes in endpoint order and the plain
PutObject route has no query matcher, so the multipart endpoints are
listed first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 17:51:32 +08:00
Feng Ruohang cfefc049c1 fix: send the SSE-KMS context as a JSON object on federated copies
putOptsFromReq handed the parsed kms.Context straight to
encrypt.NewSSEKMS. kms.Context implements encoding.TextMarshaler, so the
SDK serialized it as a JSON string, and a request without a context
still produced one because the nil Context is a typed nil inside the
interface value and marshals to "{}". The receiving ParseHTTP rejects
both forms, so every federated CopyObject to an SSE-KMS destination
failed with InvalidArgument once the forwarded stream was correct.

Pass a plain map, or nothing when no context was requested, and cover
SSE-KMS destinations with and without an explicit context.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 17:51:32 +08:00
Feng Ruohang af56d17630 fix: forward plaintext on federated CopyObject of SSE objects (#158)
The legacy etcd bucket-federation branch of CopyObjectHandler reads its
source through getObjectNInfo, which yields the decrypted and
decompressed bytes, but it also ran the destination encryption locally
and then forwarded that stream to the remote PutObject with the source's
stored size and the destination SSE option. SSE to plain and plain to
SSE therefore failed on a Content-Length mismatch, while SSE to SSE
matched by coincidence: the remote encrypted the ciphertext a second
time and stored an unreadable object, and a destination GET returned
the inner ciphertext with HTTP 200.

The remote write owns the destination's storage transformations, so
hand it the logical bytes at their logical size and let it encrypt
exactly once. Compression was already excluded on this branch; apply
the same rule to encryption, size the forwarded reader by actualSize,
and declare that size on the forwarded PutObject.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 17:51:32 +08:00
Feng Ruohang d1105bbb3d Merge pull request #162 from pgsty/codex/replication-reliability-20260909
fix(replication): complete purges, expose MRF drops, and cancel resyncs reliably
2026-09-09 14:55:35 +08:00
Feng Ruohang 66fe61ff65 test: reuse replication compatibility fixtures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:44:02 +08:00
Feng Ruohang a1141a43f2 style: format replication regression fixture
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:24 +08:00
Feng Ruohang 702f113f51 fix(replication): scope resync cancellation and drain worker lifecycle
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:02 +08:00
Feng Ruohang 63aace4099 fix(replication): expose bounded MRF queue drops
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:02 +08:00
Feng Ruohang 9d7094b770 fix(replication): complete single-object delete marker purges
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:02 +08:00
Feng Ruohang 450dcb8484 Merge pull request #161 from pgsty/codex/server-dependency-refresh-20260909
build: refresh maintained SILO stack dependencies
2026-09-09 12:04:16 +08:00
Feng Ruohang 4074d00b96 build: refresh maintained SILO stack dependencies
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 11:49:13 +08:00
Feng Ruohang 75ba0ce402 fix: drop the broken bucket-metadata reload publication guard (issue #105 T3)
The #105 T3 change (PR #156) tried to keep the resident metadata cache
monotonic by guarding peer-reload publication on lastUpdate(). But
lastUpdate() is the max of per-config timestamps and cannot order whole
records: a node caching {policy@20, CORS@10} that receives a newer CORS@15
still has lastUpdate()==20, so the guard rejects the legitimately-newer
record and the periodic refresh (same comparator) cannot repair it. A
paused reload could also resurrect deleted resident state.

Per the maintainer decision, revert the reload publication to its original
unconditional (acceptable-until-refresh) behavior:
- remove setReloaded and restore the plain Set plus notification/target
  registry updates in LoadBucketMetadataHandler;
- restore refreshBucketsMetadataLoop's own lastUpdate() staleness check and
  globalEventNotifier.set / globalBucketTargetSys.set publication;
- restore the unconditional GetConfig cache-miss publication;
- document the known freshness limitation at the reload site (the periodic
  refresh is best-effort and cannot repair an equal-maximum-timestamp
  divergence).

The T1 lifecycle merge-under-lock (UpdateExpiryLCConfig) and both T2 fixes
(DeleteBucket takes metadata.lock before deleting; saveMetadata and
loadBucketMetadataParseUnderLock recheck physical bucket existence) are
kept fully intact.

Tests:
- drop the T3 reproductions (overlapping-reload resident-cache test and the
  peer-reload-preserves-current-targets publication test);
- add lockBucketMetadataAcquireHook, a nil-in-production atomic test hook in
  the shared metadata.lock path, so tests can deterministically observe a
  caller (notably DeleteBucket, whose lock is taken through its
  erasureServerPools receiver and is invisible to an injected object layer)
  reaching the lock;
- rewrite the T2 delete-race ghost test to hold metadata.lock MID-SAVE (past
  saveMetadata's existence recheck) and synchronize on the delete's actual
  lock attempt via the hook, so it isolates the lock-before-delete fix:
  removing only DeleteBucket's metadata.lock (recheck kept) now fails it;
- rewrite the cancellation test to observe the delete's actual lock attempt,
  then cancel and await its error while still holding the lock, so a
  scheduling-delayed delete stopped by the canceled context can no longer
  pass on a broken tree.

Refs #105. Follows #156.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 16:13:54 +08:00
Feng Ruohang da142327f2 Merge pull request #159 from pgsty/fix/issue-99-100-followups
fix: repair federated CopyObject checksum edge cases (empty body, inherited, multipart-suffix)
2026-09-08 15:58:22 +08:00
Feng Ruohang a3df317ae0 Merge pull request #60 from mrjavadseydi/feat/access-based-ilm
[ILM] Relocate hot objects across server pools by GET frequency
2026-09-08 15:42:15 +08:00
Feng Ruohang 2c50d11f72 fix: correct federated CopyObject checksum edge cases (#99 follow-ups)
Three residual checksum defects in the legacy etcd federation branch of
CopyObjectHandler, found by post-merge review of #157.

1. Empty-source 500 regression. A checksum-less object gains the S3 default
   CRC-64NVME (WantServerSideChecksumType is set), but minio-go streams no
   trailing checksum for a 0-byte body (contentLength == 0), so the remote
   computed none, federatedChecksumValue was empty, hash.NewChecksumWithType
   returned nil, and the handler returned 500 -- so every empty-object
   federated copy failed. For a 0-byte source, forward the empty-content digest
   as an ordinary checksum request header instead, so the remote validates,
   persists and returns it, matching the local path (e.g. CRC32 "AAAAAA==").

2. Inherited full-object checksum dropped. When the source already carries a
   full-object checksum, the local path sets dstOpts.WantChecksum, not
   WantServerSideChecksumType (only multipart-composite sources are promoted).
   The federated branch inspected only WantServerSideChecksumType, so a
   checksum-bearing source's checksum was silently discarded on a federated
   copy that requested no algorithm. Forward WantChecksum.Encoded (always a
   plain digest) as a checksum header so the remote validates and persists it,
   and bind the returned value, matching local persistence.

3. Multipart-suffixed remote value accepted. The bind accepted a value like
   "NSRBwg==-0": NewChecksumWithType parses the "-N" as ChecksumMultipart with
   WantParts 0 and the length-only validator passes, so the destination was
   returned as COMPOSITE. A single forwarded PutObject must yield a full-object
   digest, so reject a multipart-marked parsed value in addition to the
   existing nil (missing/malformed) rejection.

A forwarded checksum request header is stripped from objInfo.UserDefined so it
is not mistaken for object metadata.

Out of scope: the SSE federated-copy corruption (srcInfo.Reader/Size mismatch
for encrypted sources) predates this work and is filed separately.

New federated regressions cover empty source with requested and default
checksum (200 + correct value + persisted), an inherited full-object checksum
preserved without a requested algorithm, and a multipart-suffixed remote value
rejected. Red/green verified for each against the merged code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:38:11 +08:00
Feng Ruohang 5ac33e1583 test: make access move failure recovery deterministic
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:28:33 +08:00
Feng Ruohang d57c4e8407 Merge remote-tracking branch 'origin/main' into codex/access-tiering-ci-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:08:34 +08:00
Feng Ruohang 374de0fa32 fix: make access tier moves preserve versions and isolate writes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:08:34 +08:00
Feng Ruohang 80e23dc9f2 Merge pull request #156 from pgsty/codex/bucket-metadata-merge-20260908
fix: preserve bucket metadata across concurrent updates and reloads
2026-09-08 15:03:52 +08:00
Feng Ruohang 2cc0e3c6ed test: construct notification fixtures with the event ARN type
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:52:48 +08:00
Feng Ruohang f9da3b919d fix: order bucket deletion and metadata publication safely
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:43:15 +08:00
Feng Ruohang 1309853f57 Merge remote-tracking branch 'origin/main' into codex/access-tiering-ci-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:34:51 +08:00
Feng Ruohang 39b8e6c30a Merge remote-tracking branch 'origin/main' into codex/bucket-metadata-merge-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:24:22 +08:00
Feng Ruohang 9a6e1477f4 ci: record access tiering compatibility identifiers
Record the ten documented MINIO_ILM_ACCESS settings, the internal object
metadata stamp, and the tracker storage-path suffix introduced by this PR.
The guard places the /ilm/access string in its routes set, but the value
is a component of the tracker object prefix, not a public HTTP endpoint.

Keep all existing compatibility entries. The guard, delivery rebrand check,
and Docker entrypoint compatibility tests pass with the refreshed manifest.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 13:18:50 +08:00
Feng Ruohang 49375ed2d3 Merge pull request #157 from pgsty/codex/federated-copy-merge-20260908
fix: preserve metadata and checksums in federated object copies
2026-09-08 13:16:30 +08:00
Feng Ruohang f817b5261c Merge pull request #151 from nikitapogromsky/fix/idempotent-add-system-target
logger: make AddSystemTarget idempotent
2026-09-08 13:14:30 +08:00
Feng Ruohang 885bd2c20a fix: reject missing remote checksums on federated copies
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 13:05:45 +08:00
Feng Ruohang 89b75e7913 Merge branch 'main' into feat/access-based-ilm 2026-09-08 13:04:25 +08:00
Feng Ruohang 079ebb1926 fix: serialize logger initialization and publish the console target safely
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:58:21 +08:00
Feng Ruohang 04c29aac11 Merge branch 'audit/issue-105-bucketmeta-races' into codex/bucket-metadata-merge-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:55:51 +08:00
Feng Ruohang 95e7a190b3 Merge branch 'fix/issue-99-100-federated-copyobject' into codex/federated-copy-merge-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:55:51 +08:00
Feng Ruohang 8e2392e48f Merge remote-tracking branch 'origin/main' into codex/logger-idempotency-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:55:50 +08:00
Feng Ruohang 3abe0d95a5 Merge pull request #149 from pgsty/codex/embedded-console-compat
fix: restore embedded Console proxy and WebSocket configuration
2026-09-08 10:09:21 +08:00
Feng Ruohang 9c6c9805de fix: select the validated Console mainline for embedding
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 10:00:45 +08:00
nikitapogromsky 5cb900bfad logger: make AddSystemTarget idempotent
Subscribe re-registered the console target on every console-log subscription, producing duplicate minio_logger_webhook_* series on each /minio/metrics/v3 scrape. Fixes #150

Signed-off-by: nikitapogromsky <129324283+nikitapogromsky@users.noreply.github.com>
2026-09-07 12:03:39 +03:00
Feng Ruohang 0af5d22286 fix: restore embedded Console proxy and WebSocket configuration
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 13:36:24 +08:00
Feng Ruohang f1687f402b fix: scope rebrand checks to the retired repository
Match the exact retired repository while preserving links to distinct repositories and historical issue titles in contributor credits. Continue rejecting live links to the retired repository in those credits.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 11:49:23 +08:00
Feng Ruohang ce606df2c4 fix: align contribution tooling with SILO AGPL policy
Clarify SILO contribution ownership and preserve prior copyright notices. Consolidate issue templates and route the legacy credits command through the maintained generator.

Validation: make rebrand-guard; bash -n update-credits.sh; regenerated credits match CREDITS; template and link checks.
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 11:33:21 +08:00
Feng Ruohang fd44dc4e9b docs: include the latest merged community contribution
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 11:01:04 +08:00
Feng Ruohang 479745e764 docs: credit contributors across the SILO repositories
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:58:47 +08:00
Feng Ruohang cc1c54475f Merge pull request #146 from pgsty/fix/issue-108-console-loopback-tls
fix: keep embedded Console login working over loopback TLS (#108)
2026-09-07 10:53:45 +08:00
Feng Ruohang e7654d470c fix: close residual bucket-metadata races (issue #105 audit)
Audit of the three deferred #105 follow-ups. Each reproduces with a
deterministic red test in cmd/bucket-metadata-race_test.go, and each fix is
the minimal change that turns its test green while preserving the
<bucket>.lck -> metadata.lock -> .metadata.bin lock order established by #103.

1. Lifecycle expiry merge lost update (persistent). PeerBucketLCConfigHandler
   and healBucketILMExpiry read the current lifecycle with an unlocked
   GetConfigFromDisk, merged the replicated expiry rules with the local
   transition rules, then wrote the pre-computed blob via Update. Any lifecycle
   transition change committed between the merge read and the merge write was
   silently lost. New BucketMetadataSys.UpdateExpiryLCConfig performs the read,
   merge, and save under one metadata.lock; mergeExpiryWithLCConfig now takes
   the locked snapshot and validates object-lock retention from it instead of
   re-reading (avoids a re-entrant metadata load under the lock).

2. DeleteBucket ghost .metadata.bin (persistent). DeleteBucket took only
   <bucket>.lck while config writers take only metadata.lock, so a writer that
   was mid-save could re-create .metadata.bin after the prefix purge. The purge
   now runs under metadata.lock, with a best-effort unlocked fallback so a
   delete is never blocked from completing.

3. Overlapping peer reloads publishing a stale resident cache (freshness only;
   the persisted record stays correct). LoadBucketMetadataHandler and the
   GetConfig cache-miss path published with an unconditional Set, so a reload
   that read an older revision could overwrite a newer resident record until the
   next refresh. New BucketMetadataSys.setReloaded (and a matching GetConfig
   guard) refuses to regress a newer resident record, mirroring
   refreshBucketsMetadataLoop.

Verification: go build -tags kqueue,dev ./...; go vet ./cmd; gofmt clean;
rebrand-guard baseline unchanged; go test -tags kqueue,dev ./cmd (207s) green;
new tests plus the #103 metadata suite green under -race.

Refs #105. Parent #102. Foundation #103.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:33:11 +08:00
Feng Ruohang 4b25f7e819 fix: return and persist checksum on federated CopyObject (#99)
The legacy etcd federation branch of CopyObjectHandler forwards the copied
bytes with minio-go Core.PutObject but never asked the remote for a checksum
and discarded any it returned, so a cross-deployment whole-object copy that
requested a checksum returned 200 with an empty checksum, and a checksum-less
source did not gain the S3 default CRC-64NVME that the local path assigns. The
request was neither honored nor rejected. This is the whole-object counterpart
of #72, which repaired the same class of defect for federated UploadPartCopy.

When a server-side checksum is wanted -- explicitly requested, inherited from a
multipart source, or the CRC-64NVME default for a checksum-less object, all
already captured in dstOpts.WantServerSideChecksumType -- the forwarded
PutObject now streams a trailing checksum of that type, so the remote computes
and persists it and echoes it in the response. The value the remote reports for
that exact write is bound into objInfo.Checksum, matching how the local
CopyObject path carries checksums into the CopyObjectResult. Reading the value
from the same UploadInfo that produced the ETag keeps the pair bound to one
write.

Only the requested algorithm is returned; a malformed or absent remote value
leaves objInfo.Checksum unset, so an ordinary copy that wanted no checksum
still returns none. Two small mapping helpers convert between the server's
hash.ChecksumType and the minio-go request type and response field.

New end-to-end tests drive the real federation branch through
getRemoteInstanceClient and minio-go into a second in-process deployment and
assert that CRC32/CRC32C/SHA256/CRC64NVME and the no-algorithm default are all
returned in the CopyObjectResult and persisted on the destination, and that a
requested algorithm never leaks other algorithms into the response.

Fixes #99

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:32:59 +08:00
Feng Ruohang 711b092f86 fix: keep embedded Console login working over loopback TLS (#108)
The embedded Console reaches the S3/STS API at https://127.0.0.1:<port>
(minioConfigToConsoleFeatures), a loopback endpoint whose TLS certificate is
not expected to carry a 127.0.0.1 SAN. silo-console v2.3.x began verifying
every outbound TLS peer, so the Console's STS AssumeRole handshake to that
loopback endpoint now fails certificate validation and BOTH local and LDAP
logins fail with a generic "invalid login". The failure happens in the Console
HTTP client before any request reaches a server auth/STS/LDAP handler, so no
server-side auth error is logged, matching the report.

Restore the documented loopback bypass by opting the embedded Console into its
endpoint-scoped CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY switch whenever the server
falls back to the 127.0.0.1 endpoint under TLS. The exemption is scoped to that
single loopback origin inside Console; every other HTTPS peer (IdP, Prometheus,
webhooks) stays verified, preserving the v2.3.x hardening. An explicitly
configured endpoint is reached under its own verified name and is never
exempted. initConsoleServer unsets CONSOLE_* before re-deriving them, so the
switch cannot be supplied by the operator on the embedded path; the server must
assert it.

Fixes #108

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:32:59 +08:00
Feng Ruohang 2bc103b80c fix: strip all reserved metadata on federated CopyObject (#100)
The legacy etcd federation branch of CopyObjectHandler forwards the copied
source metadata to the remote deployment with minio-go Core.PutObject, after
removing only two reserved keys (compression and actual-size). Every small
object is stored inline, so its stored metadata also carries
X-Minio-Internal-inline-data; the remote's setRequestLimitMiddleware rejects
any request bearing a reserved-prefix header (containsReservedMetadata), so
the forwarded write failed with 400 InvalidArgument "Your metadata headers
are not supported." for the default COPY metadata directive.

A plain federated PutObject must not carry any internal storage metadata, so
strip the whole reserved-prefix class before forwarding instead of an
enumerated subset. Enumerating a third key would only defer the next leak:
besides inline-data, replication bookkeeping (replica/replication status and
timestamps) is added to the same map earlier in the handler and would be
rejected just the same. None of these keys is required by the remote for a
correct plain PutObject; they are internal storage details the remote sets
for itself. The stripping uses stringsHasPrefixFold, matching the remote's
own case-insensitive detection. Ordinary user metadata (x-amz-meta-*) is
untouched and still copied.

The pre-existing UUID ETag on the federated write (no Content-MD5 is sent) is
out of scope and left unchanged, as recorded in the issue.

A new end-to-end test drives the real federation branch through
getRemoteInstanceClient and minio-go into a second in-process deployment,
copying an inline source with the default COPY directive. It asserts the copy
now succeeds, that no reserved-prefix header reaches the remote on any
forwarded request, and that copied user metadata survives.

Fixes #100

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:32:59 +08:00
Feng Ruohang ad873c7357 Merge pull request #132 from mrjavadseydi/fix/issue-106-bucket-quota-metrics
fix: report effective bucket quotas in metrics
2026-09-07 09:54:01 +08:00
Feng Ruohang 0af0907eff Merge pull request #134 from pgsty/fix/issue-120-ssec-replica-retransmit
fix: retransmit and re-order Object Lock for SSE-C replicas (single erasure set)
2026-09-07 00:24:45 +08:00
Feng Ruohang e27ba2bc14 Merge pull request #131 from pgsty/fix/issue-117-lock-resend-compare
fix: stop re-replicating an object whose retention was removed
2026-09-07 00:22:21 +08:00
Feng Ruohang 236e163c0b fix: repair an undecodable SSE-C replica on retransmit
PutObjectHandler's precondition callback ran DecryptObjectInfo on the
stored object before checkPreconditionsPUT, so an authenticated raw
SSE-C replica overwrite was rejected when the stored version could not
decrypt. A replica a pre-fix destination (issue #109) left as
compress(ciphertext) or a re-encrypted body has an invalid decrypted
length, so DecryptObjectInfo returned errObjectTampered and the
retransmission that repairs it never ran -- the version stayed damaged
through resync. #134's raw-replica exemption only covered the
version/ETag duplicate check inside checkPreconditionsPUT, one step too
late.

Skip the stored object's decryption precondition only for a PURE raw
SSE-C replica overwrite (a trusted SSE-C replica write with no public
precondition), keyed on the incoming request's restored SSE-C metadata,
the same predicate checkPreconditionsPUT uses. Such a write fully
replaces the object, so requiring the damaged stored version to decrypt
is both wrong and unnecessary. A conditional request keeps the check:
DecryptObjectInfo also normalizes the stored sealed ETag to the
client-visible one, and If-Match/If-None-Match must compare against
that, not the sealed ETag -- skipping it for every replica inverted both
conditions. Ordinary writes and non-SSE-C replicas are unchanged.

Adds red/green regressions: a raw retransmit over a version staged as an
undecodable body returns 500 XMinioObjectTampered before this change and
200 with full customer-key recovery after; and a conditional replica PUT
(If-Match / If-None-Match) on the client-visible ETag is honoured rather
than inverted. Fixes the single-PUT compression-damage recovery gap in

Signed-off-by: Feng Ruohang <rh@vonng.com>
#120.
2026-09-07 00:11:04 +08:00
Feng Ruohang 7220210e8d test: reconcile #134 fixtures with #119 and refresh the rebrand baseline
Rebased onto current main. #119 made PutObjectPart derive an encrypted
part's plaintext length and reject a part that cannot be a valid sio
stream, so TestReplicaLockReconcileNullVersion's completeNullMPU fixture
(a 4-byte plaintext part under SSE-C metadata) no longer stores; build it
with sio.Encrypt like the other encrypted-part fixtures. Also regenerate
the rebrand-guard baseline for the replication SSE header the retransmit
path reintroduces (headers 84 -> 85). Mechanical integration only.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 00:11:04 +08:00
Feng Ruohang 34cbca97ea docs: point the multi-pool lock follow-up at pgsty/silo#133
Fill the tracked-issue number into the scope comments of the single
erasure set Object Lock reconcile added for SSE-C replica retransmit.
No behaviour change.

Refs pgsty/silo#120
Refs pgsty/silo#133

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 00:11:04 +08:00
Feng Ruohang 109d824e5f fix: retransmit and re-order Object Lock for SSE-C replicas (single erasure set)
Issue #120 routes an existing SSE-C replica through PutObjectHandler and
NewMultipartUploadHandler. On main those handlers assigned the incoming retention
and legal hold directly, without the source-timestamp ordering #111 added to
CopyObjectHandler and without persisting the ordering timestamps, so in
active-active replication a retransmit carrying an older value could overwrite a
destination version's newer lock state.

Share #111's ordering decision as applyReplicatedObjectLock in
cmd/bucket-object-lock.go and call it from CopyObject, PUT and multipart
initiation. A request that is not an actual trusted replica keeps ordinary write
semantics (a validated value is applied and stamped now); only a real replica
update is ordered against the stored version, so a marker-only peer write no
longer drops a validated hold or default retention. CopyObject keeps its SSE-C
key-rotation encMetadata reconciliation inline. putReplicationOpts now emits a
stored retention ordering timestamp even when the value keys are absent, so a
removal recorded on the retransmit PUT path still replicates onward. replicateAll
marks Failed and carries the error when putReplicationOpts fails.

The handler decision is made against the version as it stands then, which a
concurrent lock update can outrun before the write commits, and for multipart
across the whole initiation-to-completion span. Close that window under the
object write lock the receiving erasure set holds: a trusted SSE-C replica full
write sets ObjectOptions.ReplicaLockReconcile, and erasureObjects.PutObject and
CompleteMultipartUpload re-run the ordering (reconcileStoredObjectLock, which
orders retention and legal hold independently by their reserved timestamps)
against the destination version read on that set before committing. Persisted
upload metadata records the null version as an empty VersionID, so completion
looks that up as the null version rather than the latest. The reconcile runs only
against an existing version; a not-found destination keeps the write's own
accepted lock, including a pre-upgrade upload that persisted values without
ordering timestamps, and a non-not-found read error fails the write. Scoped to
the SSE-C paths this issue enables; CopyObject is left as #111 wrote it.

Scope: this orders Object Lock against the destination version under the write
lock and is correct for a single erasure set. A multi-pool deployment -- where a
version can have duplicate copies across pools, object ModTime ties do not track
per-field lock timestamps, and the object namespace lock is per-pool -- needs a
cross-pool lock-safe reconcile and is deliberately out of scope here, tracked in
pgsty/silo#TBD-multipool-lock.

Tests: TestAPISSECReplicaRetransmitObjectLockOrdering and its multipart sibling;
TestAPIReplicaMultipartNewerHoldSurvivesCompletion and
TestReplicaPutObjectLockReconcileUnderWriteLock (a hold or retention reaching the
version after the handler decision, or after multipart initiation, survives the
commit; a pre-upgrade upload on an absent version keeps its lock);
TestReplicaLockReconcileNullVersion (a null-version completion reconciles the null
version, not a coexisting UUID version, and an absent null version keeps its
accepted lock); TestAPIReplicaMarkerOnlyAppliesObjectLock; TestReplicaStoredLock;
the timestamp-only putReplicationOpts round trip; and the retransmit, exemption
and target-head tests. The #111 CopyObject replica suite and the existing #120
suite stay green, as do the PUT/multipart handler and object-layer regression
suites. Compatibility: the shared helper preserves #111's CopyObject behavior; a
non-replica PUT or multipart initiation that sets Object Lock now also stamps the
reserved ordering timestamp, matching CopyObject since #111; only trusted SSE-C
replica writes take the in-lock reconcile.

Refs pgsty/silo#120

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 00:11:04 +08:00
Feng Ruohang 87746913fc fix: retransmit existing SSE-C replicas instead of metadata-copying them
The replication sender's target HEAD carries no SSE-C customer key, so for
an SSE-C object the target answers 400 and replicateAll fell into a
metadata-only CopyObject that fails on any non-empty SSE-C object (the
undecryptable source checksum makes the target recompute one and rewrite
the data with a plaintext-sized reader). Once a non-empty SSE-C replica
existed, tag, retention and legal-hold changes never reached it, a heal
never retransmitted, and a resync neither repaired the replica nor
counted it correctly. Forcing a full retransmit alone was not enough:
checkPreconditionsPUT rejects a write whose PreserveETag and VersionID
match the stored version, only the single-part sealed ETag is truncated
before that comparison, so a multipart SSE-C retransmit answered 412,
which the sender turns into success. Inherited from upstream ad04afe38.

Select replicateAll when the SSE-C HEAD cannot answer (the two previous
assignments were dead: rAction still forced the metadata path), exempt an
authenticated replica write that carries an SSE-C seal from the duplicate
version and ETag rejection (the predicate is the incoming write's
restored SSE-C metadata, not what the destination holds), and send the
internal replication marker on the resync accounting HEAD for SSE-C
objects so a peer answers with the replica metadata instead of 400.

Tests: TestAPISSECReplicaRetransmitOverExistingVersion (multipart replica
initiation over the same version and ETag answered 412 on main, now 200
with parts sent and plaintext readback; single-part and zero-byte writes
unchanged), TestAPISSECReplicaWriteExemptionIsKeyedOnTheIncomingWrite
(plaintext replica over an SSE-C version still 412; SSE-C replica over a
plaintext version exempted and readable), and
TestAPISSECReplicationTargetHead (keyless HEAD 400, missing key 404,
marked HEAD 200 with metadata, metadata CopyObject ExcessData on a
non-empty object) on ErasureSD and Erasure. Compatibility: every update
of an SSE-C object now retransmits its bytes; a peer that rejects the
internal marker fails the accounting HEAD as before; the #109 destination
fix must be deployed first or a retransmitted replica is transformed
again.

Fixes pgsty/silo#120

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 00:11:04 +08:00
Feng Ruohang 7935c84f9a fix: recognize timestamp-only retention-removal tombstone in resend compare
retentionRemovedAtSource only recognized representation (1) of a removed
retention: the object lock key present with an empty value. But a removal
that arrived by replication persists representation (2): restoreRetention
(and the receiver's replica update path) writes only the retention ordering
timestamp when the mode is empty, leaving the mode and retain-until-date keys
absent. For that shape the helper returned false, so replicationActionForTarget
skipped the GetObjectRetention confirmation and let getReplicationAction's
replicateNone stand, silently dropping a needed removal when the destination
HEAD hides retention behind a permission-filtered credential.

Recognize representation (2) as well: a present retention ordering timestamp
with the mode value absent or empty is a removal. A present timestamp paired
with a non-empty mode is a retention that was set, not removed, and still
returns false.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 00:06:47 +08:00
Feng Ruohang c185635b43 fix: treat empty object lock values as absent when comparing
getReplicationAction builds its source map from oi1.UserDefined, where a
removed retention is a present key with an empty value, and its target map
from the destination's HEAD headers, which can never carry those keys because
setObjectHeaders skips empty lock values and FilterObjectLockMetadata drops
both keys when the mode is invalid. The comparison then always reports a
difference, the replicateNone fast path is dead for such versions, and an
otherwise matching version re-copies its metadata on every evaluation.

Skip an entry whose value is empty and whose key is x-amz-object-lock-mode or
x-amz-object-lock-retain-until-date, case-insensitively, in both comparison
loops, using the joined value on the target side. Normalizing only the source
would regress the case where both sides hold the empty pair.

HEAD also omits a real retention from a credential without
s3:GetObjectRetention, which the documented target policy does not grant, so
that normalization alone would read a destination hiding a retention as in
sync and drop the removal. replicationActionForTarget therefore confirms with
the destination before skipping the resend: only an explicit answer, no
retention on the version, clears it. Everything else keeps today's metadata
resend, including a denied or unreachable destination, a mode the SDK does not
recognize, and InvalidRequest, which names a bucket without Object Lock but is
also what a destination answers when its own read of that configuration fails.
The null version an existing object resync excludes is never reopened.

Tests: TestGetReplicationActionEmptyObjectLockValues (eight cases, red on 2
and 3 before this change), TestRetentionRemovedAtSource,
TestTargetRetentionConfirmedAbsent,
TestReplicationActionForTargetRetentionRemoval,
TestReplicationActionForTargetNullVersionResync and
TestEmptyRetentionValuesAreOmittedFromObjectResponseHeaders.
Compatibility: sender side only, no wire or storage change, so a fixed source
converges against any destination version.

Fixes pgsty/silo#117

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
2026-09-07 00:06:47 +08:00
Feng Ruohang c201148738 Merge pull request #145 from pgsty/fix/issue-10-conditional-delete
fix: support conditional DeleteObject (If-Match) with atomic precondition
2026-09-06 23:58:42 +08:00
Feng Ruohang 53adb21c52 Merge pull request #142 from pgsty/fix/issue-141-resync-dispatch-scope
fix: scope resync worker dispatch to the target being resynced
2026-09-06 23:57:17 +08:00
Feng Ruohang 58b0ee36ca Merge pull request #140 from pgsty/fix/issue-139-resync-classification
fix: count resync success by replication outcome, not target existence
2026-09-06 23:56:57 +08:00
Feng Ruohang 8a1f594add Merge pull request #138 from pgsty/fix/issue-136-resync-counter-flush
fix: persist an honest resync terminal status about object counts
2026-09-06 23:55:59 +08:00
Feng Ruohang 5b9959617e Merge pull request #130 from pgsty/docs/issue-116-startup-readiness
docs: describe the startup readiness window of the health probes
2026-09-06 23:55:37 +08:00
Feng Ruohang da19d91b64 Merge pull request #143 from pgsty/fix/issue-107-chunked-checksum
fix: honor a header-delivered checksum advertised as a chunked trailer
2026-09-06 23:55:33 +08:00
Feng Ruohang 40bee4b7ba fix(delete): honor If-Match precondition on DeleteObject (#10)
DeleteObject ignored the If-Match request header and always deleted the
object (204). AWS S3 conditional deletes require that when If-Match is
provided and does not match the object's current ETag, the delete is
refused with 412 Precondition Failed and the object is left intact.

The precondition is evaluated in erasureServerPools.DeleteObject, while
the server-pool delete lock is held, before the delete-marker short-circuit
and before any version is removed. It runs against the version that will
actually be deleted: pinfo.ObjInfo for a normal delete, or the specifically
addressed version (read under the held lock) for a version-scoped delete,
since getPoolInfoExistingWithOpts strips VersionID. The check is a pure
function (no ResponseWriter writes) and returns PreConditionFailed, which
toAPIError maps to 412; CheckPrecondFn is cleared before lower layers run
so the precondition is evaluated exactly once.

Semantics:
- If-Match mismatch on a live object -> 412, object preserved.
- If-Match "*" requires a live object; a delete-marker-latest -> 412, and
  an explicitly addressed delete-marker version -> 412 (getObjectInfo
  returns the marker with MethodNotAllowed; the marker is the precondition
  target, not a 405).
- SSE-C/SSE-KMS: compared against the public ETag derived without the
  customer key, so a satisfiable condition is never falsely rejected.
- Explicit versionId -> evaluated against that version; a missing addressed
  version -> NoSuchVersion whether or not the key exists; a missing object
  (no versionId) -> NoSuchKey; no If-Match -> unchanged (including the
  unconditional version-scoped delete's error behavior).

Scope: atomicity is guaranteed for a single erasure set (the default
deployment). Multi-pool conditional-delete atomicity (concurrent writers
across pools, cross-pool version selection) is tracked as a follow-up.

Tests: pure-helper unit test (delete marker, "*", SSE-C without key);
object-layer tests (unversioned match/mismatch/missing, versioned
delete-marker-latest and addressed delete-marker version, explicit-version
selection, missing version on present and absent keys, read-quorum loss);
handler tests (412/204/wildcard/404) across both backends, with red/green
demonstrated per guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 23:05:27 +08:00
Feng Ruohang 6a9b5d6763 fix: honor header checksum when x-amz-trailer is advertised on non-trailer chunked PUT (#107)
The AWS Java SDK v2, with chunked encoding enabled (its default), sends a
PutObject as a non-trailer signed aws-chunked stream
(x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD). When a checksum
algorithm is set it puts the precomputed value in the x-amz-checksum-crc32
header, yet still advertises the checksum in x-amz-trailer even though no
trailer chunk is ever sent.

GetContentChecksum treated any x-amz-trailer-advertised checksum as trailing
with an empty value, deferring it to a trailer. For the non-trailer auth type
the handler sets req.Trailer = nil, so at EOF the hash.Reader looked the value
up in a nil trailer, got "", and returned XAmzContentChecksumMismatch (HTTP
400) even though the correct value sat in the request header. Real S3 accepts
the request, and disabling chunked encoding removed the trailer advertisement,
matching the reported symptom.

Honor the header value directly when a trailer-advertised checksum is already
present in the request headers; fall back to trailing delivery only when the
header is absent. When the header carries the checksum but it does not parse,
reject the request with ErrInvalidChecksum instead of falling through to a
no-validation path, so a malformed client-supplied checksum is never silently
dropped. This also restores the checksum echo on the response and the stored
value, while keeping genuine trailer uploads and wrong-checksum rejection
intact.

Fixes #107.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 22:48:29 +08:00
Feng Ruohang 0720ed477e fix(replication): scope resync dispatch to the target being resynced
The resync worker pool runs for a single target (opts.arn), but the dispatch
loop admitted any object whose ExistingObjResync.mustResync() was true for ANY
target. On a bucket with per-target rules (A and B), a resync of A would pull
in objects that only qualify for B - even with a single active resync, since
qualification is any-target. After the outcome-based classification (previous
change) such a cross-target object leaves A absent from its per-object result
and is counted as an A failure - an object A was never responsible for.

Scope admission to the resync's own target: dispatch an object only if it must
resync for opts.arn specifically (mustResyncTarget), via a small pure helper
objectNeedsResyncForARN. Only opts.arn carries this resync's ResetID, and that
reset is already folded into its per-target decision, so the per-target check
both scopes dispatch and honors the reset. Each target has its own resyncBucket,
so no cross-target object is dropped - it is handled by that target's resync.
The classifier's absent-ARN failure path is now unreachable for normally
dispatched objects and remains only as defense-in-depth (e.g. a config/lock
error before replication is attempted).

Delete-marker/version-purge handling, the null-version exclusion, the
finalization ordering and the outcome-based classification are unchanged.

Adds a table-driven regression for the predicate: with A/B rules and a resync
of A, an object qualifying only for B is not admitted; any-target scoping
admits it and fails the test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 21:57:52 +08:00
Feng Ruohang 46e82eb54d fix(replication): count resync success by outcome, not target existence
The resync worker classified each object by whether the target version
merely existed (a tgt.StatObject HEAD), ignoring the outcome of the
replicateObject/replicateDelete call it had just made. A quota-rejected
update leaves the old version in place, so StatObject succeeded and the
resync recorded a false success - reported as Completed / N success /
0 failed and persisted across restart (issue #139). #134's SSE-C HEAD
marker made StatObject succeed for SSE-C too, exposing it there. The delete
path had the mirror flaw (a failed delete leaves the object, so the HEAD
succeeded), and FailedSize was never incremented (a failed 196,608-byte
object counted as 1 failed / 0 bytes).

replicateObject and replicateDelete already build the per-target
replicatedInfos (each replicatedTargetInfo carries Arn, ReplicationStatus
and Err) but discarded it. Return it (callers that only trigger replication
ignore the value - a Go call statement discards it, so the queue paths are
unchanged) and classify the resync from the target whose Arn == opts.arn via
a small pure helper:

- Completed without error -> replicated (+ that target's size, falling back
  to the object size).
- Failed or errored -> failed (+ the object size, fixing FailedSize).
- opts.arn absent from the result (not attempted) -> failed; a resync that
  cannot confirm the object reached the target is not a success.

The StatObject-existence block (including the delete-marker/MethodNotAllowed
special case, now subsumed by the delete outcome) is removed. #134's SSE-C
HEAD marker is left intact - it is needed for genuine SSE-C success.

Adds a table-driven regression for the classifier covering a completed
update, a failed update over an existing version, an errored-but-Completed
result, a delete failure, a delete-marker success (zero bytes) and an
un-attempted ARN. Classifying by existence makes the failed cases count
success and fails the test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 18:48:15 +08:00
Feng Ruohang f8ca4a8656 fix(replication): keep resync Completed status honest about object counts
resyncBucket could publish and persist a Completed resync status that did
not actually cover every object, in two ways:

1. It joined only the producer workers before the deferred markStatus ran,
   not the goroutine that folds each worker result into the status, so a
   Completed status could omit the last object (or a failed object) until the
   periodic ~1m flush (issue #136). The same finalization also closed the
   result channel on early-return paths while workers were still in flight,
   risking a send-on-closed-channel panic and a lost result.
2. markStatus persists under its own background context, so if the parent
   context was cancelled during the drain - workers then return without
   sending their computed result - or a worker dropped a result on the
   resync-cancel signal, a bare Completed was still recorded with counts that
   no longer matched the objects seen.

Fixes (count integrity only; the inherited cancellation deadlock, walker leak,
and single-token routing are tracked as separate follow-ups):

- Centralize shutdown in a resyncResults helper whose finish() stops the
  workers (closes inputs, waits for them to exit) before closing the result
  channel and waiting for the consumer to drain, then lets the deferred
  markStatus persist the final counts. finish() now runs on every exit path.
- Record a dropped result via sendResyncResult (a worker consuming the
  resync-cancel token returns without sending), and in the finalizer downgrade
  a Completed status to Failed via finalResyncStatus when the parent context
  was cancelled or a worker aborted - so a persisted Completed never
  misrepresents an incomplete resync.

Deterministic tests: an on-disk round-trip of the terminal status (complete
counts stay Completed; parent-cancel-during-drain and worker-abort each
downgrade to Failed), and testing/synctest drain/worker-order assertions that
fail deterministically if a finish() wait is removed. The inherited
cancellation structure (inline Walk, the dispatch send, the worker cancel
branches) is left unchanged for the follow-ups.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 15:28:55 +08:00
Feng Ruohang e7d0e62f16 Merge pull request #135 from pgsty/fix/attributes-encrypted-parts-test-reconcile
test: reconcile encrypted-parts attributes test with #119 write validation
2026-09-06 09:59:54 +08:00
Feng Ruohang aee290fc34 test: reconcile encrypted-parts attributes test with #119 write validation
TestAPIGetObjectAttributesEncryptedPartLengths (from #128) built its
fixtures by PutObjectPart-ing plaintext bodies under encrypted-object
metadata with per-part sizes 5245473 and 1. Since #119, PutObjectPart
always derives an encrypted part's plaintext length from the bytes
written and rejects a part that cannot be a valid sio stream, so those
fixtures can no longer be created through a normal write and both
variants failed at write time.

Such an on-disk shape now only exists as pre-#119 data or from an old
peer, which is exactly the state the GetObjectAttributes per-part
tamper check (#128) defends. Inject that ObjectInfo directly through a
stub object layer (the setObjectLayer pattern used by the #110 tamper
test) and exercise the handler, which is what this test pins. The
handler path, the crafted part sizes, and both assertions
(separately-encrypted-parts -> ErrObjectTampered; legacy-single-stream
-> stored fragment sizes) are unchanged. Test-only; reconciles two
already-merged correct changes (#119 and #128).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 09:51:09 +08:00
Feng Ruohang 62cce2b152 Merge pull request #121 from pgsty/fix/issue-110-tampered-status
fix: return 500 for unreadable objects instead of 206
2026-09-06 09:26:28 +08:00
Feng Ruohang 65d4806a7b Merge pull request #127 from pgsty/fix/issue-112-bucket-metadata-cors
fix: include per-bucket CORS in bucket metadata export and import
2026-09-06 09:26:21 +08:00
Feng Ruohang 765757473a Merge pull request #126 from pgsty/fix/issue-118-no-compressed-ssec
fix: exclude SSE-C objects from compression
2026-09-06 09:26:12 +08:00
Feng Ruohang 425bd7fff1 Merge pull request #124 from pgsty/fix/issue-119-ssec-part-actual-size
fix: record plaintext part sizes for replicated SSE-C multipart parts
2026-09-06 09:26:04 +08:00
Feng Ruohang e12e739a53 Merge pull request #128 from pgsty/fix/issue-114-115-object-attributes-parts
fix: report logical part sizes and end pagination correctly in GetObjectAttributes
2026-09-06 09:25:28 +08:00
Feng Ruohang 8b736dee34 Merge pull request #123 from pgsty/fix/issue-113-rotation-checksum-algorithm
fix: honor a requested checksum algorithm on SSE-C key rotation
2026-09-06 09:25:02 +08:00
Feng Ruohang 32e75c27bf Merge pull request #122 from pgsty/fix/issue-109-raw-ssec-replica
fix: store raw SSE-C replicas verbatim on the destination
2026-09-06 09:24:26 +08:00
Feng Ruohang 885ca604a1 Merge pull request #129 from pgsty/fix/issue-111-object-lock-replica-ordering
fix: order value-less replicated Object Lock updates by timestamp
2026-09-06 09:23:34 +08:00
Feng Ruohang d10382d0dc build: refresh rebrand compatibility baseline for the SSE-C replica helper
The raw SSE-C seal headers moved from an inline slice in
cmd/object-multipart-handlers.go to the shared isRawSSECReplica helper
in cmd/replication-trust.go (already recorded), so regenerate the
compat-baseline allowlist to drop the three stale object-multipart
entries. No behaviour change; make rebrand-guard is green.

Refs pgsty/silo#109

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-06 09:08:22 +08:00
mr javad seydi 0db4bf3b00 fix: report effective bucket quotas in metrics
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-05 21:23:46 +03:30
Feng Ruohang 87c621965d docs: describe the startup readiness window of the health probes
After a restart a node's remote erasure drives that could not be connected
during startup stay uninstalled until the next connectDisks pass, about 15
seconds later. During that window the liveness, readiness and both cluster
probes answer 200, admin info shows every drive online and mcli ready agrees,
because the cluster probes aggregate each peer's report of its own local
drives rather than the drives this node has installed. A PUT through that
node can still fail with 503 SlowDownWrite and a cross-node GET can answer
404 NoSuchKey until the window closes. The behaviour is inherited from
upstream and reproduced on the 0806 and 0903 releases alike.

Document the window and the bounded data-path check (PUT through each node,
read each object through every node, fixed deadline, re-read acknowledged
objects) that automation should use instead of the probes, and correct the
readiness probe description, which also fails on request-queue overload and
an unreachable KMS. No product change: the probes keep their documented
purpose, and changing them or the reconnect cadence was judged unproven
tuning in the agreed plan.

Refs pgsty/silo#116

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-05 17:17:05 +08:00
Feng Ruohang b2dca43fda fix: order value-less replicated Object Lock updates by timestamp
CopyObjectHandler rebuilt the destination metadata with the public Object Lock keys stripped (cmd/object-handlers.go:1708) and then restored a value only inside retentionMode.Valid() and legalHold.Status.Valid() (cmd/object-handlers.go:1715 and :1732), so a replica update that carried no retention or legal-hold value never reached the ordering comparison and silently erased whatever the destination held, however new it was; a retention removal that did win recorded no ordering timestamp either, so cmd/bucket-object-lock.go:370 later read an unparseable stored timestamp and let an older retained value back in. Each replica field is now decided on its source timestamp first and its incoming value second, and both restore helpers write the stored timestamp back before returning early on an empty stored value, which is the only way a removal timestamp survives the REPLACE metadata directive. Legal hold stays deliberately asymmetric: S3 has no legal-hold removal, an explicitly empty status is already rejected as invalid, and an absent status conveys no change even when an orphaned timestamp arrives with it, so only a valid ON or OFF can win.

Three inherited defects would have defeated that ordering, so they are fixed here too. The SSE-KMS branch of putOptsFromHeaders built its own ObjectOptions and dropped the parsed lock timestamps, leaving every replicated lock update unordered on a bucket with default KMS encryption; it now carries them. The in-place SSE-C key rotation snapshots the stored reserved metadata into encMetadata before the lock decision exists and merges it back afterwards to preserve the encryption headers, reinstating the ordering timestamp the decision had just replaced; the snapshot is now reconciled with the decision for a trusted replica. Finally, the value-less handling applies only to an actual replica: a trusted peer that sends the replication marker without REPLICA status keeps the previous behaviour, so a REPLACE copy carrying no lock headers still writes a version with no retention and no hold.

Tests: TestAPICopyObjectReplicaAbsentLockFieldsPreserveNewerState, TestAPICopyObjectReplicaRetentionRemovalKeepsOrderingTimestamp, TestAPICopyObjectReplicaObjectLockOrdering, TestAPICopyObjectReplicaRetentionRemovalUnderBucketKMS, TestAPICopyObjectReplicaLockTimestampSurvivesSSECKeyRotation and TestAPICopyObjectMarkerOnlyLeavesObjectLockUnchanged, all on ErasureSD and Erasure. Compatibility: no API, wire or stored-field change, and a field arriving with no source timestamp is unordered and now preserves destination state, so an un-upgraded 0806 peer keeps replicating safely while it still runs the old erasing receiver.

Fixes pgsty/silo#111

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-05 16:58:11 +08:00
Feng Ruohang 33a91d972f fix: carry per-bucket CORS through bucket metadata export/import
The admin bucket-metadata handlers enumerate every bucket config by name, and
per-bucket CORS was never added to that enumeration: export omitted cors.xml
(cmd/admin-bucket-handlers.go:414 cfgFiles) and import ignored the entry
outright, with no case in applyImportedBucketMetadata (:598) or SetStatus
(:629), so a CORS-only archive reported 0/0 buckets imported and a restored
bucket silently lost its configuration. Export now writes the stored document
verbatim and import validates it with the same parser and validator as
PutBucketCorsHandler, merging it under the existing bucket metadata lock and
announcing it through the dedicated SRBucketMetaTypeCorsConfig event; the local
CORS timestamp rule is extracted into localCORSUpdatedAt and reused so an
imported document always lands strictly above bucket creation, which matters
because the import stamps its fields before creating any missing bucket and a
CORS event below Created is dropped as an older bucket incarnation.

The import reads one byte past the declared entry size so archive/zip reaches
EOF and verifies the entry checksum, otherwise a corrupt or over-long entry
carrying well formed XML would overwrite the stored document; and the CORS
event is sent even when the shared bucket metadata hook failed, so an
unreachable peer cannot withhold an already committed CORS document from the
reachable ones.

Tests: TestAdminBucketMetadataCORSRoundTrip and
TestAdminBucketMetadataCORSImportReplicatesPastPeerFailure (new, ErasureSD and
Erasure).
Compatibility: the ZIP gains one entry, older archives stay importable and
leave CORS untouched; no mcli or madmin-go change is needed because
madmin.BucketStatus already carries Cors and mcli copies the export ZIP
verbatim.

Fixes pgsty/silo#112

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
2026-09-05 16:21:12 +08:00
Feng Ruohang 2cbd48a3c3 fix: end GetObjectAttributes part pagination correctly
GetObjectAttributes decided truncation by comparing the last returned
part number with the part count (cmd/object-handlers.go:720), which is
only a coincidence of contiguous numbering. Sparse parts 1/3 reported a
complete page as truncated with a marker that loops, and parts 1/3/5
with max-parts=1 stopped after part 3 and silently dropped part 5. Set
IsTruncated in the break that proves an eligible part was left
unreturned, and zero NextPartNumberMarker when the listing is complete,
as ListObjectParts already does. Also reject negative x-amz-max-parts
and x-amz-part-number-marker in getAndValidateAttributesOpts with the
same API errors ListObjectParts uses, instead of answering an invalid
request with an empty parts listing; an absent or zero max-parts still
means the default page size.

Tests: TestAPIGetObjectAttributesPartsPagination (sparse 1/3/5 and
contiguous 1/2 walks on ErasureSD and Erasure),
TestGetAndValidateAttributesOptsPartsRange, and the sparse variants of
TestAPIGetObjectAttributesMultipartLogicalPartSize. Compatibility: no
field is added or removed; IsTruncated and NextPartNumberMarker change
only where they were wrong, and negative pagination values that no SDK
sends now fail fast.

Fixes pgsty/silo#115

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
2026-09-05 16:17:39 +08:00
Feng Ruohang b5409ca112 fix: report logical part sizes in GetObjectAttributes
GetObjectAttributes filled ObjectPart.Size from the on-disk part length
(cmd/object-handlers.go:715), so every compressed or encrypted multipart
object reported transformed sizes that do not sum to the logical
ObjectSize the same response returns from objInfo.GetActualSize().
Report each part's uploaded plaintext length instead: a compressed part
uses its recorded ActualSize, and a separately encrypted part derives the
plaintext length with sio.DecryptedSize, because ActualSize is the
ciphertext length for a replicated SSE-C part and zero for parts written
before actualSize existed.

Only parts of an encrypted multipart object are streams of their own. A
legacy encrypted object carries no multipart marker and is one continuous
stream that the erasure writer split into storage fragments, so those
fragments keep their stored size. Where a part is a stream, one whose
length cannot be a valid encrypted stream has no logical length, and the
request now fails with XMinioObjectTampered rather than reporting the
ciphertext length; DecryptObjectInfo does not catch that case, because
ObjectInfo.isMultipart gives up on the first bad part and only the object
total is then validated.

Tests: TestAPIGetObjectAttributesMultipartLogicalPartSize (plain,
compressed, SSE-C and compressed+SSE-C, consecutive and sparse part
numbers), TestAPIGetObjectAttributesCompressedEmptyTrailingPart,
TestAPIGetObjectAttributesEncryptedPartLengths, and a part-size assertion
added to TestAPISSECMultipartReplicationTrust. Compatibility: the XML
shape is unchanged and nothing is written to disk, only the value of the
existing Size element is corrected.

Fixes pgsty/silo#114

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
2026-09-05 16:17:39 +08:00
Feng Ruohang 35bd75948a fix: exclude SSE-C objects from compression
With compression allow_encryption=on an SSE-C object is stored as
encrypt(s2(plaintext)), while replication reads it raw (NoDecryption at
cmd/erasure-object.go:257) and putReplicationOpts drops the internal
compression and actual-size headers (cmd/bucket-replication.go:786). The
replica keeps the source seal with no compression marker, so a GET with the
correct customer key returns HTTP 200 and the raw S2 stream instead of the
object, and the source records the transfer as COMPLETED.

Widen the one condition in excludeForCompression (cmd/object-api-utils.go:613)
so SSE-C is never compressed, whatever allow_encryption says. This covers all
four producers at once, PutObject, NewMultipartUpload, CopyObject and
PutObjectExtract, plus any future caller of isCompressible.
crypto.SSEC.IsRequested ignores copy-source headers, so a copy is judged on its
destination key only, and a raw SSE-C replica write is unaffected because it
carries no public SSE-C headers. allow_encryption keeps its meaning for SSE-S3
and SSE-KMS, where the server owns the key and decompresses before replicating.

Tests: TestAPISSECCompressionReplicaStaysReadable (single PUT and multipart),
TestAPISSECCompressionProducerMatrix, TestAPISSECCompressionSkippedOnCopyObject,
TestAPISSECCompressionSkippedOnSnowballExtract and the control
TestSSECBatchReplicationCannotRead in cmd/compression-ssec_test.go. Two existing
expectations pinned the removed shape and are updated:
TestAPICopyObjectSSECKeyRotationNullVersionCompressesRewrite is renamed
TestAPICopyObjectSSECKeyRotationNullVersionSkipsCompression and now expects an
uncompressed rewrite, keeping its body, checksum, version and ETag assertions;
the SSE-C compressed-encrypted variant of
TestAPICopyObjectServerSideChecksumEncryption becomes compressible-extension and
expects an uncompressed destination, its SSE-S3 sibling keeping the compressed
coverage.

Compatibility: a deliberate behaviour change. Deployments with
allow_encryption=on no longer store new or rewritten SSE-C data compressed, so
those writes cost more space; objects already stored compressed keep working on
the source and are the concern of pgsty/silo#109, which rejects them at
replication time. Multipart uploads initiated before this change keep
compressing their parts from the metadata saved at initiation. Upstream
468a9fae8 refused this combination at PUT time and a2cab0255 removed the guard;
upstream master is still unguarded, so this is a deliberate divergence.

Fixes pgsty/silo#118

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-05 15:51:24 +08:00
Feng Ruohang 0c8d74205b fix: record plaintext part sizes for replicated SSE-C multipart parts
Trusted SSE-C replication uploads parts as raw ciphertext with the
ciphertext length as Content-Length, and erasureObjects.PutObjectPart only
derived the plaintext length when the caller passed a negative size, so
each replicated part persisted the ciphertext length as ActualSize (the
field defined as the uploaded size without encryption bytes). On the
replica, partNumberToRangeSpec turned those lengths into a plaintext range,
so GET/HEAD ?partNumber=N returned the wrong bytes and shifted
Content-Range (2560, 2560 and 5120 bytes for 5 MiB, 5 MiB and 1 MiB
parts), and a later decommission or rebalance re-uploaded the parts with
the stale value and recomputed the object-level actual-size from their sum,
after which a whole-object GET advertised a Content-Length larger than the
body it wrote.

Derive the plaintext length of an encrypted, uncompressed part from the
bytes actually written (sio.DecryptedSize) in PutObjectPart, the single
place a part is persisted, rejecting a length that cannot be a valid
stream before the part is committed; and derive part lengths from
part.Size in partNumberToRangeSpec for encrypted, uncompressed objects,
returning an error instead of a nil range, so replicas already on disk
read correctly without a resync. Compressed parts keep ActualSize.

Tests: TestAPISSECReplicaPartNumberReads (three-part SSE-C replica,
?partNumber=N bytes, Content-Length and Content-Range equal the source)
and TestSSECReplicaPartActualSizeDataMovement (replay through the data
movement path leaves part and object sizes at plaintext values) fail on
main and pass with the fix on ErasureSD and Erasure;
TestAPIGetObjectWithPartNumberHandler, TestAPISSECMultipartReplicationTrust
and TestAPIListObjectPartsHandler stay green. Compatibility: no wire or
API change; objects an unfixed server already moved carry a poisoned
object-level actual-size and need a rewrite or resync.

Fixes pgsty/silo#119

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-05 15:33:23 +08:00
Feng Ruohang fcc4d77895 fix: honor a requested checksum algorithm on SSE-C key rotation
An in-place SSE-C key rotation takes the fast path at cmd/object-handlers.go:1523
that only rewraps the object key, while every line that turns
x-amz-checksum-algorithm into a stored checksum lives in the re-encrypting else
branch at 1571-1609, so a requested algorithm was silently dropped and the stale
source checksum was kept and reported. Extend the canRotateKeyInPlace guard so a
client request carrying the header falls through to the copy that recomputes,
stores and reports it. Replica-trusted requests keep the fast path: getOpts
leaves their source reader encrypted, so a rewrite would hash ciphertext, and a
replica has to keep the checksum its source assigned.

Tests: TestAPICopyObjectSSECKeyRotationChecksumAlgorithm (new, red before the
guard), TestAPICopyObjectSSECKeyRotationKeepsChecksumAbsence (new, pins the
accepted limitation that a headerless rotation preserves the stored checksum
state including absence, gaining no default CRC64NVME) and
TestAPICopyObjectSSECKeyRotationReplicaKeepsFastPath (new, pins the replica
carve-out on a non-empty and on a zero byte source).
Compatibility: no API or wire change; a rotation without the header and every
replica-trusted rotation are unchanged, while a client rotation carrying the
header now rewrites the object data, so the ETag changes, a multipart source
collapses to a single part object, the copy replicates as an object rather than
as metadata, and the rewritten bytes are compressed if compression is enabled for
that object, as AWS CopyObject documents. Upstream MinIO carries the same
defect from 2718d9a43 (minio/minio#21399); this is a deliberate divergence.

Fixes pgsty/silo#113

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
2026-09-05 15:25:08 +08:00
Feng Ruohang c52acc1a5d fix: store raw SSE-C replicas verbatim on the destination
A raw SSE-C replica write carries the source ciphertext and the source seal
in X-Minio-Replication-Server-Side-Encryption-* headers but no public SSE-C
request headers, so crypto.Requested() was false and PutObjectHandler and
NewMultipartUploadHandler applied the destination's default encryption and
compression to bytes that were already ciphertext (upstream 468a9fae8,
"Enable replication of SSE-C objects", never exempted the raw path). With
destination default SSE-S3 the replica's IV and seal were overwritten and
GET returned 400; with destination compression the replica stored
compress(ciphertext) and GET failed, while the source reported COMPLETED.

Recognize a validated raw SSE-C replica (replicaTrusted plus a seal header,
shared helper isRawSSECReplica) and skip bucket default encryption,
compression and the encryption branch on the single PUT path, and default
encryption plus compression on the multipart initiation path, which already
skipped key generation. On the sender, reject replication of an object that
is both compressed and SSE-C, since the wire carries no compression state
and the destination would otherwise store an undetectable S2 stream, and
make replicateObject/replicateAll report a putReplicationOpts failure as
Failed instead of Completed.

Tests: TestAPISSECReplicaSkipsDestinationTransforms (single PUT and
multipart under destination default SSE-S3, compression and an explicit SSE
header, plus an untrusted control), TestAPISSECMultipartReplicaRoundTripWith
Compression, and TestPutReplicationOptsRejectsCompressedSSEC fail on main
and pass with the fix on ErasureSD and Erasure; the replication-trust,
multipart and PutObject suites stay green. Compatibility: no wire, API or
metadata change; the destination change applies only to trusted replica
writes carrying a source seal; replicas already transformed must be
rewritten from an intact source (see pgsty/silo#120 for why a resync does
not do that yet).

Fixes pgsty/silo#109

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-05 15:07:22 +08:00
Feng Ruohang 5703426b3c fix: return 500 for unreadable objects instead of 206
ErrObjectTampered was mapped to http.StatusPartialContent since upstream
ca6b4773e (2017), so a GET or HEAD of an object the server cannot decode
(invalid encrypted size, malformed actual-size, bad multipart ETag shape)
answered with a success status and an XML error document that SDKs handed
back as object content; boto3 returned the XML as Body and a zero-length
HEAD as success. Every origin of errObjectTampered is a stored-state
defect, not caller input, so map the entry to 500 Internal Server Error
and keep the XMinioObjectTampered code and message. The comment records
the deliberate divergence from upstream.

Tests: TestObjectTamperedGETHEADStatus (signed GET and HEAD, ErasureSD and
Erasure) fails with 206 on main and passes with 500; TestAPIErrCode,
TestAPIErrCodeDefinition, TestAPIHeadObjectHandler,
TestAPIHeadObjectHandlerWithEncryption and TestAPIGetObjectHandler stay
green. Compatibility: only the status line of one MinIO-specific error
changes; clients now retry damaged-object reads per their 5xx policy.

Fixes pgsty/silo#110

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-05 14:50:00 +08:00
Feng Ruohang f0bd164b92 docs: refresh community contributors
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-04 19:21:26 +08:00
Feng Ruohang 9936a69d89 ci: recover container publication from verified component pins
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-04 15:10:43 +08:00
Feng Ruohang ce2326c946 build: pin the published mcli 20260903 archives
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-04 14:38:54 +08:00
Feng Ruohang 4c164907f5 build: align Helm defaults with the published release tag
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-04 14:36:12 +08:00
mr javad seydi 7a060cab1e feat(ilm): relocate hot objects across server pools by GET frequency
Keep NVMe/HDD pool pairs useful without remote tiering: promote objects that
are read often, demote previously moved objects once they go idle, and leave
the feature off until operators set a two-pool topology.

Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-08-15 13:48:22 +03:30
234 changed files with 32992 additions and 2988 deletions
-54
View File
@@ -1,54 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage
assignees: ''
---
## NOTE
Silo issues are handled by community maintainers on a best-effort basis. There
is no SLA, SLO, or emergency production-support channel. Follow the local
[Code of Conduct](../code_of_conduct.md) when participating. Report suspected
vulnerabilities through the private process in [SECURITY.md](../SECURITY.md),
not in a public issue.
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and include relevant Silo logs with secrets and credentials removed -->
1.
2.
3.
4.
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally include the Silo version, commit id, or PR that caused this regression. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`silo --version`):
* Server setup and configuration:
* Operating System and version (`uname -a`):
+8
View File
@@ -7,6 +7,14 @@ assignees: ''
---
Report bugs in the PGSTY SILO server (`pgsty/silo`) here. Community maintainers
handle reports on a best-effort basis. There is no SLA, SLO, or emergency
production-support channel. Follow the
[Code of Conduct](https://github.com/pgsty/silo/blob/main/code_of_conduct.md).
Report suspected vulnerabilities privately through
[SECURITY.md](https://github.com/pgsty/silo/blob/main/SECURITY.md).
For patches, see the [contribution guide](https://github.com/pgsty/silo/blob/main/CONTRIBUTING.md).
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
@@ -7,6 +7,9 @@ assignees: ''
---
Suggest improvements to the PGSTY SILO server (`pgsty/silo`) here.
For patches, see the [contribution guide](https://github.com/pgsty/silo/blob/main/CONTRIBUTING.md).
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+7 -3
View File
@@ -1,10 +1,14 @@
## Contribution Licensing (no CLA, inbound=outbound, DCO required)
This project does not use a CLA; contributions are accepted inbound=outbound.
This pull request contributes to PGSTY SILO (`pgsty/silo`). Code contributions
are accepted under AGPL-3.0-or-later, the same license as the server.
This project does not use a CLA or require a separate Apache-2.0 license grant.
By submitting this pull request I represent that I have the right to contribute
the changes, which are licensed under this repository's
the code changes under this repository's
[GNU Affero General Public License v3.0 or later](https://www.gnu.org/licenses/agpl-3.0.html)
and remain my copyright. Every commit must carry a DCO `Signed-off-by` trailer
and retain copyright in my original work. Existing copyright and license
notices remain intact; separately licensed material keeps its applicable terms.
Every commit must carry a DCO `Signed-off-by` trailer
(`git commit -s`) certifying the
[Developer Certificate of Origin](https://developercertificate.org/) — see
[CONTRIBUTING.md](https://github.com/pgsty/silo/blob/main/CONTRIBUTING.md).
+60 -3
View File
@@ -7,6 +7,11 @@ on:
description: "Published RELEASE.* tag to package as pgsty/silo"
required: true
type: string
recovery:
description: "Run the current main workflow against an already-published tag"
required: false
default: false
type: boolean
permissions:
contents: read
@@ -75,12 +80,18 @@ jobs:
fetch-depth: 0
- name: Verify workflow identity matches release source
env:
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
RECOVERY: ${{ inputs.recovery }}
run: |
set -euo pipefail
CHECKED_OUT_REVISION="$(git rev-parse HEAD)"
if [ "${CHECKED_OUT_REVISION}" != "${GITHUB_SHA}" ]; then
echo "Checked out ${CHECKED_OUT_REVISION}, but workflow identity is ${GITHUB_SHA}. Dispatch this workflow from ${RELEASE_TAG}." >&2
exit 1
if [ "${RECOVERY}" != "true" ] || [ "${GITHUB_REF}" != "refs/heads/${DEFAULT_BRANCH}" ]; then
echo "Checked out ${CHECKED_OUT_REVISION}, but workflow identity is ${GITHUB_SHA}. Dispatch from ${RELEASE_TAG}, or use recovery from ${DEFAULT_BRANCH}." >&2
exit 1
fi
echo "Recovery workflow ${GITHUB_SHA} is packaging published source ${CHECKED_OUT_REVISION}."
fi
- name: Prepare verified Docker contexts
@@ -129,10 +140,50 @@ jobs:
mkdir -p "${context}/dockerscripts"
tar -xzf "${archive}" -C "${context}" silo
cp Dockerfile.goreleaser Dockerfile.distroless LICENSE NOTICE CREDITS "${context}/"
cp dockerscripts/docker-entrypoint.sh dockerscripts/download-static-curl.sh \
cp dockerscripts/docker-entrypoint.sh dockerscripts/build-static-curl.sh \
"${context}/dockerscripts/"
done
# The classic image bundles mcli. Resolve its two archive digests
# from the immutable published release instead of trusting defaults
# copied into an older Server tag. This also gives a recovery run a
# narrow override when a tag selected the right mcli release but
# accidentally retained stale archive pins.
MC_REPO="$(awk -F= '/^ARG MC_REPO=/{print $2; exit}' Dockerfile.goreleaser)"
MC_VERSION="$(awk -F= '/^ARG MC_VERSION=/{print $2; exit}' Dockerfile.goreleaser)"
test -n "${MC_REPO}"
test -n "${MC_VERSION}"
MC_VERSION_HYPHEN="${MC_VERSION#RELEASE.}"
MC_PKG_VERSION="$(echo "${MC_VERSION_HYPHEN}" | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/\1\2\3\4\5\6.0.0/')"
if [ "${MC_PKG_VERSION}" = "${MC_VERSION_HYPHEN}" ]; then
echo "Invalid bundled mcli tag: ${MC_VERSION}" >&2
exit 1
fi
if [ "$(gh release view "${MC_VERSION}" --repo "${MC_REPO}" --json isDraft --jq .isDraft)" != false ] || \
[ "$(gh release view "${MC_VERSION}" --repo "${MC_REPO}" --json isPrerelease --jq .isPrerelease)" != false ] || \
[ "$(gh release view "${MC_VERSION}" --repo "${MC_REPO}" --json isImmutable --jq .isImmutable)" != true ]; then
echo "Bundled mcli ${MC_REPO}@${MC_VERSION} must be a published immutable release" >&2
exit 1
fi
mc_checksums="mcli_${MC_PKG_VERSION}_checksums.txt"
gh release download "${MC_VERSION}" --repo "${MC_REPO}" \
--dir "${assets_dir}" --pattern "${mc_checksums}"
gh attestation verify "${assets_dir}/${mc_checksums}" \
--repo "${MC_REPO}" \
--signer-workflow "${MC_REPO}/.github/workflows/release.yml" \
--source-ref "refs/tags/${MC_VERSION}" >/dev/null
MC_AMD64_SHA256="$(awk -v name="mcli_${MC_PKG_VERSION}_linux_amd64.tar.gz" '$2 == name {print $1}' "${assets_dir}/${mc_checksums}")"
MC_ARM64_SHA256="$(awk -v name="mcli_${MC_PKG_VERSION}_linux_arm64.tar.gz" '$2 == name {print $1}' "${assets_dir}/${mc_checksums}")"
[[ "${MC_AMD64_SHA256}" =~ ^[0-9a-f]{64}$ ]]
[[ "${MC_ARM64_SHA256}" =~ ^[0-9a-f]{64}$ ]]
{
echo "MC_AMD64_SHA256=${MC_AMD64_SHA256}"
echo "MC_ARM64_SHA256=${MC_ARM64_SHA256}"
} >> "${GITHUB_ENV}"
echo "RELEASE_REVISION=$(git rev-parse HEAD)" >> "${GITHUB_ENV}"
- name: Set up QEMU
@@ -162,6 +213,9 @@ jobs:
file: docker-release/amd64/Dockerfile.goreleaser
platforms: linux/amd64
push: true
build-args: |
MC_AMD64_SHA256=${{ env.MC_AMD64_SHA256 }}
MC_ARM64_SHA256=${{ env.MC_ARM64_SHA256 }}
tags: |
pgsty/silo:${{ env.RELEASE_TAG }}-amd64
pgsty/silo:latest-amd64
@@ -178,6 +232,9 @@ jobs:
file: docker-release/arm64/Dockerfile.goreleaser
platforms: linux/arm64
push: true
build-args: |
MC_AMD64_SHA256=${{ env.MC_AMD64_SHA256 }}
MC_ARM64_SHA256=${{ env.MC_ARM64_SHA256 }}
tags: |
pgsty/silo:${{ env.RELEASE_TAG }}-arm64
pgsty/silo:latest-arm64
+6
View File
@@ -90,6 +90,12 @@ jobs:
- name: Run S3 Select tests under race detector
run: go test -race ./internal/s3select/... -count=1
- name: Run conditional PUT tests under race detector
run: go test -race ./cmd -run '^Test(PoolsConditionalPut|SinglePoolConditionalPutHTTP)' -count=1 -timeout=5m
- name: Run multipart listing and cancellation tests under race detector
run: go test -race ./cmd -run '^Test(MultipartListing|MultipartAbort|PaginateMultipartUploads|ListMultipartUploads)' -count=1 -timeout=5m
crosscompile:
name: Cross Compile
runs-on: ubuntu-latest
+100
View File
@@ -0,0 +1,100 @@
name: Repository Cards
on:
schedule:
# 00:00 UTC = 08:00 Asia/Shanghai. GitHub may queue scheduled runs.
- cron: "0 0 * * *"
workflow_dispatch:
push:
branches: [main]
paths:
- .github/workflows/repository-cards.yml
- .github/silo.svg
- buildscripts/repository-cards/**
pull_request:
branches: [main]
paths:
- .github/workflows/repository-cards.yml
- .github/silo.svg
- buildscripts/repository-cards/**
permissions:
contents: read
concurrency:
group: repository-cards-${{ github.event.pull_request.number || 'publish' }}
cancel-in-progress: false
jobs:
check:
name: Validate repository cards
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.13"
- run: python -m pip install -r buildscripts/repository-cards/requirements.txt
- run: python -m unittest discover -s buildscripts/repository-cards -p 'test_*.py' -v
publish:
name: Update README images
needs: check
if: github.repository == 'pgsty/silo' && github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write
issues: read
pull-requests: read
env:
OUTPUT_BRANCH: codex/repository-cards
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.13"
- run: python -m pip install -r buildscripts/repository-cards/requirements.txt
- name: Load the generated-assets branch
shell: bash
run: |
set -euo pipefail
artifacts="$RUNNER_TEMP/repository-cards"
if git ls-remote --exit-code --heads origin "$OUTPUT_BRANCH"; then
git fetch --depth=1 origin "$OUTPUT_BRANCH"
git worktree add --detach "$artifacts" FETCH_HEAD
else
status=$?
# Exit 2 means no matching ref; transport/auth failures must stop.
if [ "$status" -ne 2 ]; then exit "$status"; fi
git worktree add --detach "$artifacts" HEAD
git -C "$artifacts" checkout --orphan "$OUTPUT_BRANCH"
git -C "$artifacts" rm -rf .
fi
- name: Refresh contributor and star cards
env:
GH_TOKEN: ${{ github.token }}
run: python buildscripts/repository-cards/update.py --output "$RUNNER_TEMP/repository-cards"
- name: Publish changed assets
shell: bash
run: |
set -euo pipefail
cd "$RUNNER_TEMP/repository-cards"
git add README.md history.json curated.json contributors.json \
contributors-light.svg contributors-dark.svg \
star-history-light.svg star-history-dark.svg
if git diff --cached --quiet; then
echo "Repository cards are already current."
exit 0
fi
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git commit -s -m "chore: update repository cards $(date -u +%F)"
# A normal push preserves history and refuses concurrent overwrites.
git push origin "HEAD:refs/heads/$OUTPUT_BRANCH"
+24 -2
View File
@@ -10,7 +10,7 @@ on:
- "Dockerfile.distroless"
- "cmd/healthcheck-main.go"
- "cmd/main.go"
- "dockerscripts/download-static-curl.sh"
- "dockerscripts/build-static-curl.sh"
- "dockerscripts/docker-entrypoint.sh"
- "dockerscripts/docker-entrypoint_test.sh"
- "silo.service"
@@ -41,6 +41,28 @@ permissions:
contents: read
jobs:
curl:
name: Static curl (${{ matrix.arch }})
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- arch: amd64
runner: ubuntu-latest
- arch: arm64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v7
- name: Build and exercise curl in an empty runtime
run: |
docker build --build-arg TARGETARCH=${{ matrix.arch }} \
--target curl-runtime -f Dockerfile.goreleaser -t silo-curl-test .
docker run --rm silo-curl-test --version | tee curl-version.txt
grep -F 'curl 8.22.0 ' curl-version.txt
grep -F 'HTTP2' curl-version.txt
docker run --rm silo-curl-test --fail --silent --show-error \
--connect-timeout 15 --max-time 60 https://curl.se/robots.txt
validate:
runs-on: ubuntu-latest
steps:
@@ -490,7 +512,7 @@ jobs:
bash -n buildscripts/package/lifecycle_test.sh
buildscripts/package/lifecycle_test.sh
bash -n dockerscripts/docker-entrypoint_test.sh
bash -n dockerscripts/download-static-curl.sh
bash -n dockerscripts/build-static-curl.sh
dockerscripts/docker-entrypoint_test.sh
go run ./buildscripts/rebrand-guard
buildscripts/verify-rebrand.sh
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
- name: Install govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@v1.7.0
go install golang.org/x/vuln/cmd/govulncheck@v1.8.0
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
- name: Run govulncheck
+2 -2
View File
@@ -62,10 +62,10 @@ dist/
.claude/
.codex/
AGENTS.md
CLAUDE.md
_bmad/
_bmad-output/
# Investigation and editorial working files belong outside this repository.
docs/investigations/
docs/security/
docs/rebranding.md
.release-sign/
+84
View File
@@ -0,0 +1,84 @@
# SILO Repository Guide
## Project map
- Server: `pgsty/silo` (this checkout, default branch `main`)
- Console: `pgsty/silo-console` (usual sibling checkout `../silo-console`)
- Client: `pgsty/mc`, shipped as `mcli` (usual sibling checkout `../mc`)
- Shared packages: `pgsty/silo-pkg` (usual sibling checkout `../silo-pkg`)
- Documentation: `pgsty/silo.pgsty.com` (usual sibling checkout `../silo.pgsty.com`),
published at <https://silo.pgsty.com/>
The server executable, package, systemd service, and container image use `silo`;
the public Docker image is `docker.io/pgsty/silo`. The server module remains
`github.com/minio/minio`. Consult the current `go.mod` and release configuration
for selected component versions and compatibility replacements.
## Supported stack and compatibility policy
The maintained, release-gating product graph is the coordinated PGSTY stack:
`silo` + `silo-console` + `mc` + `silo-pkg`.
Keep compatibility with upstream MinIO/MC on a best-effort basis. Preserve
inexpensive wire, configuration, CLI, migration, and import compatibility when
it helps users, and document known differences. Do not infer from source
lineage, MinIO-compatible protocols, retained `MINIO_*`/`MC_*` names, or an
inherited upstream test that unmodified upstream MinIO is a supported release
target.
Upstream-only compatibility checks may remain as advisory evidence, but they
must not force a downgrade of a maintained SILO component, a fork-only API
workaround, or a release block. Make upstream compatibility a hard gate only
when the user explicitly requests that scope.
## Dependency selection
- When PGSTY maintains a component, import and require it directly under its own
module path. In particular, prefer `github.com/pgsty/silo-pkg/v3` over
`github.com/minio/pkg/v3` in maintained SILO source.
- Do not use an upstream package merely to make unmodified upstream MinIO or MC
compile. Unavoidable transitive upstream modules should be documented and
kept separate from the maintained product dependency.
- `github.com/minio/minio-go/v7` is the explicit exception: use the verified
upstream module/commit while it contains the required fixes; do not recreate
a SILO fork without a concrete functional divergence.
- Check the current `go.mod` before changing versions. Coordinate breaking
import-path changes across package, client, Console, and server releases.
## Documentation and delivery
The companion site owns product, operations, migration, security-advisory,
design, and release documentation. Maintain English and Chinese content in
that repository and link to its canonical URLs from this one. The site's
`content/docs/` is the documentation entry point; detailed pages live under
`content/operations/`, `administration/`, `reference/`, `compatibility/`,
`about/`, and `blog/`, following the existing structure.
Keep repository entry points and contributor instructions here. Retained
upstream documents, examples, and test fixtures under `docs/` may be updated
when a code or tooling change requires it; do not build a second documentation
site in this tree.
Investigation plans, prompts, AI session transcripts, execution logs, temporary
reports, and private security material belong outside the checkout, for example
in a task-specific directory under `~/tmp/`. Do not recreate
`docs/investigations/`, `docs/security/`, or `docs/rebranding.md`, or move their
working material to another repository directory. Extract reusable, verified
knowledge into the companion site; keep private evidence outside public Git.
Compatibility pages may continue to describe similarities with upstream, but
must label that compatibility as best effort. The supported and tested server
for Console and mcli administration is `pgsty/silo`.
Before removing or moving documentation, check repository links and scripts,
the companion site's links and anchors, and references from this guide. Use
fixed commit URLs for historical evidence that should survive deletion from
the current tree. Run the site's `make check` for site changes and this
repository's `make rebrand-guard` for documentation cleanup that changes the
identifier inventory. When new public URLs are involved, publish the site
before publishing source changes that depend on those URLs.
Keep changes in the repository that owns the affected surface. Treat every
repository commit, tag, release, image, documentation update, and deployment as
a separate deliverable. Follow `CONTRIBUTING.md`, including DCO sign-off
(`git commit -s`). `CLAUDE.md` imports this guide so both agents use one policy.
+199
View File
@@ -0,0 +1,199 @@
# Changelog
## Unreleased
The entries below describe source changes on main since the latest published Server.
**The latest published Server remains 20260903.** These changes are not in its
binaries, packages or images. See the [component matrix](https://silo.pgsty.com/compatibility/versions/)
and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-09-03T13-18-01Z...main).
### Authorization and security
- Restrict embedded Console's anonymous sharing proxy to object-content GETs
at the configured S3 origin, and reject every redirect. Internal metrics,
system paths and non-download S3 operations cannot be reached through it.
Normal public, presigned and versioned downloads remain available without a
new setting; a full sharing-disable switch is not introduced. See
[Console #56](https://github.com/pgsty/silo-console/pull/56) and the
[design record](https://github.com/pgsty/silo-console/issues/52).
Thanks to Jiri Pejchal (@jiri-pejchal) for the report.
- Persist IAM deletion revisions and parent revocation boundaries so stale site
events cannot restore deleted identities, policies or their older grants
(#191, #192). Peer deletion notifications reload committed storage; deliberate
recreation requires a newer revision, and credentials issued before the
parent's revocation remain invalid.
**Coordinated upgrade required:** upgrade every participating node and site.
Mixed old/new nodes sharing an IAM backend and rolling downgrade are
unsupported. Back up complete IAM storage and encryption material; an admin
export of live records omits deletion history. Reissue credentials for
recreated parents and explicitly reconcile pre-upgrade revocations whose
history is already lost. Restoring an older backup can lose later revocations;
keep affected sites isolated until reconciliation/rekeying is complete. See
[the operator runbook](https://silo.pgsty.com/operations/replication/iam-upgrade/).
- Enforce an absolute HTTP/1 request-header deadline through the connection
wrapper (#196). Repeated small reads no longer extend that deadline, and
`--read-header-timeout` / `MINIO_READ_HEADER_TIMEOUT` now reaches the HTTP
server. HTTP/1 request bodies retain the rolling idle timeout; this does not
impose a total upload/download duration. A shorter setting also constrains
TLS handshake reads. The wrapper's strict header mode is not applied to HTTP/2.
- Reject unsigned `x-amz-*` request headers that could turn a signed PUT into a
copy of another object accessible to the signer (SN-2026-011). The latest
public Server is affected; the fix is on main. See [the advisory ledger](https://silo.pgsty.com/about/security-advisories/).
- Align signed request fields with policy conditions and enforce header-only
presigned payload checksums. See [the signed-header review](https://silo.pgsty.com/blog/design/signed-header-coverage/).
- **Breaking policy semantics:** separate self-service `admin:ChangeMyPassword`
from `admin:CreateUser`. Built-in read-only policies follow the split. Preserve
both denies if the previous combined restriction must survive upgrades or
rollback. Saved policies are not rewritten. Deploy with the matching Console
and pkg; see [the migration guide](docs/iam/password-permissions.md).
### Object storage and replication
- Make `ListMultipartUploads` discover quorum-valid uploads from durable state
across pools, erasure sets and drives, then apply S3 prefix, delimiter,
marker, ordering and 1,000-entry pagination semantics globally (#198). New uploads
store their canonical bucket and key as reserved fields in the existing
quorum-written `xl.meta`; completion removes those upload-only fields. Native
markers remain usable after their upload is completed or canceled. Strict
listing returns a diagnostic 503 for legacy uploads or uncertain coverage;
the default remains the released exact-key/cache-based `legacy` behavior.
Opt into strict mode only through `MINIO_API_MULTIPART_LISTING=strict`, after
upgrading every writer, draining old uploads, checking the read-only admin
`multipart-preflight` report and validating scan capacity. The process-only
setting is not persisted into shared API configuration. Historical
`multipart_listing` keys are ignored and can be removed with a targeted
`mcli admin config reset ALIAS api multipart_listing` before rollback. Per-process
admission, directory-entry, worker and time budgets bound scan scheduling;
each page still scans durable state. See [issue #79](https://github.com/pgsty/silo/issues/79)
and its [design record](https://silo.pgsty.com/blog/design/list-multipart-uploads/).
Thanks to mr javad seydi (@mrjavadseydi) for the original implementation.
- Retain released read-quorum and best-effort multipart cancellation in default
legacy mode. Strict mode requires majority deletion acknowledgements and
permits retries below read quorum. When most drives were already empty,
failed deletion of an observed remnant now returns 503 instead of being
masked by empty-drive successes. Wrong-key or wrong-bucket cancellation
preserves the valid upload's cache entry; successful cancellation notifies
peers with the request context after the distributed lock is released.
The HTTP response for an absent
upload remains 204; this is not proof of physical cleanup. **Known boundary:**
delayed creation writes can still restore an upload after cancellation;
this change does not add a durable creation fence.
- Preserve object tags during multi-pool metadata reconciliation by reading the
resolved tag field together with its revision (#189). Previously, reconciliation
could replace existing tags with an empty value.
- Preserve the tag revision on SSE-KMS metadata replication (#193), and advance
tag revisions monotonically on local PUT/DELETE tagging (#196). Empty tags
participate in reconciliation as an ordered deletion, preventing older
events from restoring removed tags. SSE-C key rotation also retains the tag
revision. Malformed historical revisions can fail and retry; their missing
history is not reconstructed by the upgrade.
- Complete delete-marker version purges and preserve their identity and retry
state through MRF recovery (#196). Recovery accepts a 405 marker response only
when its version, bucket, object name and modification time match the task.
Purge audit status is normalized from `COMPLETE` to `COMPLETED`.
Thanks to Julien Laurenceau (@julienlau) for the investigation and proposed
fix in #184 that helped shape this follow-up.
- Restore only the six replication-specific metadata fields after ordinary
request metadata extraction (#194). This prevents transport-only `aws-chunked`
from being stored as Content-Encoding while preserving the signed-header
protections. Trusted Snowball entries no longer inherit the outer archive's
ordinary metadata. Thanks to Mikhail Khadarenka (@chodorenko) for the fix in #187.
**Existing data:** these repairs prevent new errors; they do not scan or rewrite
historical object metadata, recover lost tags or prove that old purge work has
converged. Follow the [read-only audit procedure](https://silo.pgsty.com/operations/replication/replica-metadata-audit/)
before planning any repair of stored state.
- Evaluate conditional multipart completion against the logical current object
across all pools while holding the existing object lock. A stale `If-Match`
can no longer replace newer data in another pool, and the current ETag is no
longer rejected because the upload resides next to an older copy. Conditions
are evaluated once; a current delete marker counts as an absent object.
**Availability change:** if any pool's metadata cannot be read, conditional
completion fails even when another pool can still serve GET/HEAD. This also
applies when the unreadable pool may not hold the object: absence cannot be
verified. Retry after the pool recovers. Unconditional completion and the
single-pool path retain their existing behavior.
- Evaluate ordinary multi-pool conditional PUT against the logical current
object across all pools, including draining pools, under the existing object
lock (#207). A stale destination copy no longer accepts a stale ETag or rejects
the current one; a current delete marker is treated as absence.
**Availability change:** if any pool's object metadata cannot be verified,
the condition fails even when GET can use another pool; read-quorum failures
return 503. Restore readability or heal before retrying. Unconditional PUT,
single-pool conditions and internal replication retain their existing behavior.
A public condition with a destination `versionId` compares the current object
while preserving the requested write version. This change does not retire
stale copies in other pools, undo historical accepted overwrites or provide
a new global clock-ordering guarantee. The multipart-completion repair in #190
neither introduced nor repaired this separate PUT defect.
- Reconcile ordinary single-object version DELETE across all pools, including
null versions, delete markers and unqualified directory-marker DELETE. This
applies the deletion to every resolved pool copy under existing quorum
rules. Pending outbound delete replication retains versions until the
existing replication worker completes their purge; a successful response
does not imply immediate physical removal from every drive. Unreadable
pools now consistently return 503 instead of depending on pool traversal
order; insufficient read quorum returns `SlowDownRead`. This extends the
existing failure surface. Retry after recovery.
Cleanup failures also return an error. Batch deletion already fans out across
pools; replication and scanner cleanup keep their existing contracts. See
[scope and limitations](docs/bucket/lifecycle/access-tiering-removal.md#version-deletion-scope).
- Remove the opt-in GET-frequency pool-tiering feature from PR #60, including
its tracker, mover, scanner hooks, configuration, XML actions and metrics.
Accept and ignore retired configuration/XML and preserve ordinary statistics
when reading v9 caches. See [migration notes](docs/bucket/lifecycle/access-tiering-removal.md).
The [decision record](https://silo.pgsty.com/compatibility/access-tiering-removal/) preserves
the feature's introduction, subsequent fixes, rollback scope and review history.
- Preserve the independent multi-pool write, metadata, healing and conditional
deletion fixes from PR #178, including shared remote-tier reference protection.
- Enforce `If-Match` on DELETE, preserve retention and independently ordered
Object Lock/tag updates, and correctly retransmit encrypted replicas.
- Preserve plaintext part sizes and raw SSE-C replicas; prevent SSE-C
compression, honor key-rotation checksums, and complete attributes pagination.
- Repair federated CopyObject checksums, destination timestamps, reserved
metadata, encrypted-object forwarding, legal hold and KMS context.
- Make resync counters, target selection, cancellation and worker lifetimes
reflect actual work, and report bounded MRF drops.
- Converge bucket metadata with deterministic source state, deletion tombstones,
creation time recovery and diagnostics. The mixed-version export gate requires
coordinated upgrades before tombstones are exported. See [the #77 record](https://silo.pgsty.com/blog/design/bucket-metadata-convergence/).
- Include per-bucket CORS in metadata export/import, close metadata publication
and logger races, and report effective bucket quotas in metrics.
### Console, dependencies and delivery
- Restore embedded Console login over loopback TLS, trusted-proxy handling and
all four WebSocket connection limits. Preserve Go TLS defaults across transports.
- Directly require `github.com/pgsty/silo-pkg/v3` v3.14.0; select Console
`v0.0.0-20260916034812-56dfe455ac2f` and MC
`v0.0.0-20260913012246-4f609a4da3bb` with explicit PGSTY replacements.
- Pin upstream minio-go `v7.3.1-0.20260910142817-60bd07042d49`; refresh Go x/*
modules and security fixes including bounded AMQP frame handling. Keep Go
1.27.1 and go-systemd v22.6.0's NetBSD compatibility replacement.
- Refresh container base digests and build static curl 8.22.0 from verified
source for both Linux architectures. Pin the actual mcli 20260913 archives and
hashes. Helm's client image follows that release; its Server image still names
the latest published Server 20260903.
The dependency update passed the final candidate's Go, vulnerability and Test
Release workflows; native curl builds passed on both architectures. A local
ARM64 image passed startup, health, S3 transfer and embedded Console checks.
These checks do not publish a Server tag or production image and do not replace
cluster upgrade/rollback acceptance for the next release. Dated investigations
retain the exact source and runtime boundaries they tested.
## RELEASE.2026-09-03T13-18-01Z
Published source: `9b11dc9469e650815b775cb47b039610644f5da4`.
[Complete release notes](https://silo.pgsty.com/blog/release/silo-20260903/) ·
[GitHub release](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z)
This release ships Go 1.27.1, silo-pkg v3.13.2, upstream minio-go `0e78d3f18efe`,
mcli 20260903 and embedded Console source `464a59d73ada` (v2.3.0 version identity).
Installing the newer standalone mcli or Console does not replace components
inside this existing Server binary or image.
Earlier releases: [release archive](https://github.com/pgsty/silo/releases).
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
+24 -8
View File
@@ -77,11 +77,22 @@ test evidence, compatibility notes, and documentation impact. Public product
documentation is owned by the separate
[`pgsty/silo.pgsty.com`](https://github.com/pgsty/silo.pgsty.com) repository.
## Contributor recognition
Every human issue or pull-request author is recognized in [CONTRIBUTORS.md](CONTRIBUTORS.md),
including open issues, draft PRs, and PRs closed without merging. Merged fixes,
adopted proposals, and reports that lead to fixes receive greater prominence;
first participation guides the remaining order. Work incorporated through a
later PR retains credit without changing the original PR's recorded status.
Security disclosures are credited with the reporter's agreement. DCO sign-off
applies to code commits, not to opening an issue.
## Licensing of Contributions
Silo is licensed under the [GNU AGPL v3.0 or later](LICENSE). Its core is
Copyright (c) MinIO, Inc.; the combined work can never be relicensed, and this
fork does not try to.
Code contributions to PGSTY SILO (`pgsty/silo`) are accepted under the
[GNU AGPL v3.0 or later](LICENSE), the same license as the server. Submit issues
and pull requests to this repository's maintainers. No separate Apache-2.0
license grant to SILO or upstream MinIO maintainers is required.
* **No CLA.** We do not ask you to sign a Contributor License Agreement and we
do not take your copyright. Contributions are accepted inbound=outbound: you
@@ -112,15 +123,20 @@ fork does not try to.
`Signed-off-by` trailers) and add your own sign-off as the person passing it
along. Never import code from a proprietary distribution.
* **File headers.** Files derived from upstream keep the original MinIO
copyright header unchanged. New files added by this fork use the dual
header, followed by the standard AGPL boilerplate:
* **File headers.** Preserve existing copyright and license notices in inherited
and third-party files. New original files name their actual copyright holders
and use AGPL-3.0-or-later. Use a header such as the following, then append the
standard AGPL boilerplate:
```
// Copyright (c) 2015-2025 MinIO, Inc.
// Copyright (c) 2025-2026 PGSTY
// Copyright (c) 2026 Your Name
```
* **Separately licensed material.** Documentation contributions in `docs/`
follow its existing [CC BY 4.0 license](docs/LICENSE). Third-party components
and earlier Apache-2.0 contributions retain their original licenses and
attribution; this policy does not relicense earlier work.
* **Squash merges** must keep the `Signed-off-by:` trailers in the resulting
commit message.
+165 -64
View File
@@ -1,79 +1,180 @@
# Contributors
Silo is maintained by the Pigsty community. This file records the people who have contributed to
this fork since it was established in 2026 — merged code, proposed changes, and the bug reports and
compatibility findings that shaped the releases.
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
It is maintained by hand and updated at each release. GitHub's own contributor graph is unavailable
here because `pgsty/silo` is a fork, so this file — not that page — is the project's attribution
record. Contributors are listed by GitHub handle. Authorship of every merged commit is preserved in
the Git history and can be verified with `git log --format='%an <%ae>'`.
As of **2026-09-16**, **50 community contributors** are credited in SILO and related projects. The roll includes maintainers and every human issue / PR author, plus the previously acknowledged security disclosure.
Upstream MinIO authorship is recorded separately: this fork derives from
[`minio/minio`](https://github.com/minio/minio), [`NOTICE`](NOTICE) retains the upstream product
notice, and the Git history carries the full upstream commit record.
Opening an issue or PR counts, whether open, closed, draft, or unmerged. Each account appears once. Merged PR authors come first, followed by other PR authors and reporters. Within each group, significant fixes, adopted proposals, and reports that led to fixes take priority, followed by first participation. Gold rings highlight significant contributions.
## Code
When a proposal was incorporated through a later repair, its adoption is credited separately from the original PR status. A closed issue does not necessarily mean a fix, and merged source does not imply a published release. Automated accounts are excluded. Upstream authorship remains in Git history and copyright notices.
Contributors whose changes are merged into `main`.
<p align="center">
<a href="https://github.com/Vonng"><img src="https://silo.pgsty.com/images/contributors/Vonng.svg" width="60" height="60" alt="@Vonng" title="@Vonng — Maintains SILO, Console, mcli, shared packages, releases, and documentation"></a>
<a href="https://github.com/ZouhairCharef"><img src="https://silo.pgsty.com/images/contributors/ZouhairCharef.svg" width="60" height="60" alt="@ZouhairCharef" title="@ZouhairCharef — Patched CVE-2026-34986 in go-jose"></a>
<a href="https://github.com/mfredenhagen"><img src="https://silo.pgsty.com/images/contributors/mfredenhagen.svg" width="60" height="60" alt="@mfredenhagen" title="@mfredenhagen — Patched CVE-2026-39883 in OpenTelemetry"></a>
<a href="https://github.com/pinginfo"><img src="https://silo.pgsty.com/images/contributors/pinginfo.svg" width="60" height="60" alt="@pinginfo" title="@pinginfo — Repaired bucket notification streaming"></a>
<a href="https://github.com/ycjlin"><img src="https://silo.pgsty.com/images/contributors/ycjlin.svg" width="60" height="60" alt="@ycjlin" title="@ycjlin — Fixed missing-bucket ListObjects semantics"></a>
<a href="https://github.com/waterkip"><img src="https://silo.pgsty.com/images/contributors/waterkip.svg" width="60" height="60" alt="@waterkip" title="@waterkip — Repointed documentation links to the SILO portal"></a>
<a href="https://github.com/Dansyuqri"><img src="https://silo.pgsty.com/images/contributors/Dansyuqri.svg" width="60" height="60" alt="@Dansyuqri" title="@Dansyuqri — Added ChecksumType to multipart completion responses"></a>
<a href="https://github.com/mrjavadseydi"><img src="https://silo.pgsty.com/images/contributors/mrjavadseydi.svg" width="60" height="60" alt="@mrjavadseydi" title="@mrjavadseydi — Fixed bucket quota metrics; contributed access-frequency ILM and S3-compatible multipart listing proposals"></a>
<a href="https://github.com/h5vx"><img src="https://silo.pgsty.com/images/contributors/h5vx.svg" width="60" height="60" alt="@h5vx" title="@h5vx — Implemented per-bucket CORS configuration and enforcement"></a>
<a href="https://github.com/nikitapogromsky"><img src="https://silo.pgsty.com/images/contributors/nikitapogromsky.svg" width="60" height="60" alt="@nikitapogromsky" title="@nikitapogromsky — Fixed duplicate logger targets that broke Metrics V3 collection"></a>
<a href="https://github.com/Aeirx"><img src="https://silo.pgsty.com/images/contributors/Aeirx.svg" width="60" height="60" alt="@Aeirx" title="@Aeirx — Preserved Object Lock legal-hold headers in federated CopyObject"></a>
<a href="https://github.com/magicxor"><img src="https://silo.pgsty.com/images/contributors/magicxor.svg" width="60" height="60" alt="@magicxor" title="@magicxor — Reported and proposed conditional DELETE support for If-Match"></a>
<a href="https://github.com/davinkevin"><img src="https://silo.pgsty.com/images/contributors/davinkevin.svg" width="60" height="60" alt="@davinkevin" title="@davinkevin — Proposed the distroless container image and dependency automation"></a>
<a href="https://github.com/metaneutrons"><img src="https://silo.pgsty.com/images/contributors/metaneutrons.svg" width="60" height="60" alt="@metaneutrons" title="@metaneutrons — Reported and proposed explicit-version delete authorization"></a>
<a href="https://github.com/mikemikimike"><img src="https://silo.pgsty.com/images/contributors/mikemikimike.svg" width="60" height="60" alt="@mikemikimike" title="@mikemikimike — Contributed the replicated SSE-C plaintext part-size fix"></a>
<a href="https://github.com/julienlau"><img src="https://silo.pgsty.com/images/contributors/julienlau.svg" width="60" height="60" alt="@julienlau" title="@julienlau — Reported MRF queue and delete-marker retry defects; contributed repair proposals, tests, and documentation"></a>
<a href="https://github.com/chodorenko"><img src="https://silo.pgsty.com/images/contributors/chodorenko.svg" width="60" height="60" alt="@chodorenko" title="@chodorenko — Reported and contributed the fix for aws-chunked leaking into replica metadata"></a>
<a href="https://github.com/sulin37392"><img src="https://silo.pgsty.com/images/contributors/sulin37392.svg" width="48" height="48" alt="@sulin37392" title="@sulin37392 — Proposed dependency updates"></a>
<a href="https://github.com/lem21h"><img src="https://silo.pgsty.com/images/contributors/lem21h.svg" width="48" height="48" alt="@lem21h" title="@lem21h — Proposed robustness and goroutine improvements"></a>
<a href="https://github.com/vampywiz17"><img src="https://silo.pgsty.com/images/contributors/vampywiz17.svg" width="60" height="60" alt="@vampywiz17" title="@vampywiz17 — Reported LDAP TLS and Console login regressions"></a>
<a href="https://github.com/cbornet"><img src="https://silo.pgsty.com/images/contributors/cbornet.svg" width="60" height="60" alt="@cbornet" title="@cbornet — Reported multipart and streaming checksum defects and missing-bucket semantics"></a>
<a href="https://github.com/orenyomtov"><img src="https://silo.pgsty.com/images/contributors/orenyomtov.svg" width="60" height="60" alt="@orenyomtov" title="@orenyomtov — Reported the unsigned-header CopyObject cross-object read (SN-2026-011)"></a>
<a href="https://github.com/jiri-pejchal"><img src="https://silo.pgsty.com/images/contributors/jiri-pejchal.svg" width="60" height="60" alt="@jiri-pejchal" title="@jiri-pejchal — Reported the RabbitMQ client vulnerability and the Console share proxy&#x27;s exposure of internal metrics"></a>
<a href="https://github.com/AEGEGE"><img src="https://silo.pgsty.com/images/contributors/AEGEGE.svg" width="60" height="60" alt="@AEGEGE" title="@AEGEGE — Reported the slow-HTTP denial-of-service vulnerability"></a>
<a href="https://github.com/mosesdd"><img src="https://silo.pgsty.com/images/contributors/mosesdd.svg" width="48" height="48" alt="@mosesdd" title="@mosesdd — Requested a maintained Helm chart"></a>
<a href="https://github.com/Xavier-777"><img src="https://silo.pgsty.com/images/contributors/Xavier-777.svg" width="48" height="48" alt="@Xavier-777" title="@Xavier-777 — Reported Console lifecycle management and file preview gaps"></a>
<a href="https://github.com/jiadzh"><img src="https://silo.pgsty.com/images/contributors/jiadzh.svg" width="48" height="48" alt="@jiadzh" title="@jiadzh — Requested Windows build guidance"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://silo.pgsty.com/images/contributors/AntonOfTheWoods.svg" width="48" height="48" alt="@AntonOfTheWoods" title="@AntonOfTheWoods — Asked for clarity on Helm chart and operator options"></a>
<a href="https://github.com/TLINDEN"><img src="https://silo.pgsty.com/images/contributors/TLINDEN.svg" width="48" height="48" alt="@TLINDEN" title="@TLINDEN — Reported the missing client in release tarballs"></a>
<a href="https://github.com/zylpsrs"><img src="https://silo.pgsty.com/images/contributors/zylpsrs.svg" width="48" height="48" alt="@zylpsrs" title="@zylpsrs — Reported missing Console tiering and site replication"></a>
<a href="https://github.com/nsanitate"><img src="https://silo.pgsty.com/images/contributors/nsanitate.svg" width="48" height="48" alt="@nsanitate" title="@nsanitate — Proposed CNCF Sandbox governance"></a>
<a href="https://github.com/makinikm"><img src="https://silo.pgsty.com/images/contributors/makinikm.svg" width="48" height="48" alt="@makinikm" title="@makinikm — Reported the missing client in the container image"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://silo.pgsty.com/images/contributors/spaceg00se-r.svg" width="48" height="48" alt="@spaceg00se-r" title="@spaceg00se-r — Requested cpuv1 support and reported a workflow token failure"></a>
<a href="https://github.com/heroes1412"><img src="https://silo.pgsty.com/images/contributors/heroes1412.svg" width="48" height="48" alt="@heroes1412" title="@heroes1412 — Reported the unusable profiling option"></a>
<a href="https://github.com/chalukyaj"><img src="https://silo.pgsty.com/images/contributors/chalukyaj.svg" width="48" height="48" alt="@chalukyaj" title="@chalukyaj — Proposed making the SILO Operator easier to discover"></a>
<a href="https://github.com/jvasile"><img src="https://silo.pgsty.com/images/contributors/jvasile.svg" width="48" height="48" alt="@jvasile" title="@jvasile — Reported missing user, group, and defaults in Debian packages"></a>
<a href="https://github.com/Kesavaambati"><img src="https://silo.pgsty.com/images/contributors/Kesavaambati.svg" width="48" height="48" alt="@Kesavaambati" title="@Kesavaambati — Asked about community support and image maintenance"></a>
<a href="https://github.com/redfoxfox"><img src="https://silo.pgsty.com/images/contributors/redfoxfox.svg" width="48" height="48" alt="@redfoxfox" title="@redfoxfox — Reported Chinese documentation availability"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://silo.pgsty.com/images/contributors/kuldeep-link11.svg" width="48" height="48" alt="@kuldeep-link11" title="@kuldeep-link11 — Reported NATS JWT credentials and target reload issues"></a>
<a href="https://github.com/meesudzu"><img src="https://silo.pgsty.com/images/contributors/meesudzu.svg" width="48" height="48" alt="@meesudzu" title="@meesudzu — Requested the migration guide from upstream MinIO"></a>
<a href="https://github.com/pmezhuev"><img src="https://silo.pgsty.com/images/contributors/pmezhuev.svg" width="48" height="48" alt="@pmezhuev" title="@pmezhuev — Reported missing RPM package signatures"></a>
<a href="https://github.com/kh0mka"><img src="https://silo.pgsty.com/images/contributors/kh0mka.svg" width="48" height="48" alt="@kh0mka" title="@kh0mka — Reported inter-node I/O timeouts and OIDC configuration-fetch failures"></a>
<a href="https://github.com/bagutzu"><img src="https://silo.pgsty.com/images/contributors/bagutzu.svg" width="48" height="48" alt="@bagutzu" title="@bagutzu — Requested KES-compatible external KMS and OpenBao support"></a>
<a href="https://github.com/liuhaodongliu990-cmyk"><img src="https://silo.pgsty.com/images/contributors/liuhaodongliu990-cmyk.svg" width="48" height="48" alt="@liuhaodongliu990-cmyk" title="@liuhaodongliu990-cmyk — Reported indeterminate progress for prefix downloads"></a>
<a href="https://github.com/DestroyLee"><img src="https://silo.pgsty.com/images/contributors/DestroyLee.svg" width="48" height="48" alt="@DestroyLee" title="@DestroyLee — Reported the missing documentation navigation"></a>
<a href="https://github.com/sargarass"><img src="https://silo.pgsty.com/images/contributors/sargarass.svg" width="48" height="48" alt="@sargarass" title="@sargarass — Reported ListMultipartUploads prefix and pagination semantics"></a>
<a href="https://github.com/mumu-lab"><img src="https://silo.pgsty.com/images/contributors/mumu-lab.svg" width="48" height="48" alt="@mumu-lab" title="@mumu-lab — Reported bucket quota metrics reading a deprecated field"></a>
<a href="https://github.com/haiming236"><img src="https://silo.pgsty.com/images/contributors/haiming236.svg" width="48" height="48" alt="@haiming236" title="@haiming236 — Proposed a built-in image processing pipeline"></a>
<a href="https://github.com/tiredenzo"><img src="https://silo.pgsty.com/images/contributors/tiredenzo.svg" width="48" height="48" alt="@tiredenzo" title="@tiredenzo — Reported inconsistent documentation for two-drive EC:1 support"></a>
<a href="https://github.com/aschyolkin"><img src="https://silo.pgsty.com/images/contributors/aschyolkin.svg" width="48" height="48" alt="@aschyolkin" title="@aschyolkin — Reported a data race in CPU metrics collection"></a>
</p>
| Contributor | Change | Pull request | Commit |
| :-- | :-- | :-- | :-- |
| [@ZouhairCharef](https://github.com/ZouhairCharef) | Upgraded `go-jose` to v4.1.4 to patch CVE-2026-34986 | [#18](https://github.com/pgsty/silo/pull/18) | [`68e0ba9`](https://github.com/pgsty/silo/commit/68e0ba997) |
| [@mfredenhagen](https://github.com/mfredenhagen) | Bumped `go.opentelemetry.io` to address CVE-2026-39883 | [#19](https://github.com/pgsty/silo/pull/19) | [`1869bd3`](https://github.com/pgsty/silo/commit/1869bd30b) |
| [@pinginfo](https://github.com/pinginfo) | Implemented `Flush` on `trackingResponseWriter`, repairing bucket notification streaming | [#34](https://github.com/pgsty/silo/pull/34) | [`65795ee`](https://github.com/pgsty/silo/commit/65795ee1f) |
| [@waterkip](https://github.com/waterkip) | Repointed documentation links from the upstream domain to the Silo portal | [#41](https://github.com/pgsty/silo/pull/41) | [`d495d30`](https://github.com/pgsty/silo/commit/d495d30d5) |
| [@Dansyuqri](https://github.com/Dansyuqri) | Added `ChecksumType` to the `CompleteMultipartUpload` response | [#57](https://github.com/pgsty/silo/pull/57) | [`d014a12`](https://github.com/pgsty/silo/commit/d014a12cf) |
| [@ycjlin](https://github.com/ycjlin) | `ListObjects` returns `NoSuchBucket` for a prefix on a missing bucket | [#37](https://github.com/pgsty/silo/pull/37) | [`e9c5340`](https://github.com/pgsty/silo/commit/e9c5340be) |
| [@h5vx](https://github.com/h5vx) | Implemented per-bucket CORS: stored configuration, S3 handlers, and request enforcement | [#71](https://github.com/pgsty/silo/pull/71) | [`e4e3007`](https://github.com/pgsty/silo/commit/e4e3007da) |
## Merged pull requests
## Proposed changes
| Contributor | Contribution | Record |
| :-- | :-- | :-- |
| [@Vonng](https://github.com/Vonng) | Maintains SILO, Console, mcli, shared packages, releases, and documentation | [pgsty/silo: Merged PRs (77)](https://github.com/pgsty/silo/issues?q=author%3AVonng+is%3Apr+is%3Amerged)<br>PRs closed without merging: [pgsty/silo#155](https://github.com/pgsty/silo/pull/155), [pgsty/silo#195](https://github.com/pgsty/silo/pull/195)<br>[pgsty/silo: Closed issues (60)](https://github.com/pgsty/silo/issues?q=author%3AVonng+is%3Aissue+is%3Aclosed)<br>Open issues: [pgsty/silo#199](https://github.com/pgsty/silo/issues/199), [pgsty/silo#200](https://github.com/pgsty/silo/issues/200), [pgsty/silo#201](https://github.com/pgsty/silo/issues/201), [pgsty/silo#202](https://github.com/pgsty/silo/issues/202), [pgsty/silo#203](https://github.com/pgsty/silo/issues/203)<br>[pgsty/silo-console: Merged PRs (22)](https://github.com/pgsty/silo-console/issues?q=author%3AVonng+is%3Apr+is%3Amerged)<br>[pgsty/silo-console: Closed issues (32)](https://github.com/pgsty/silo-console/issues?q=author%3AVonng+is%3Aissue+is%3Aclosed)<br>Open issues: [pgsty/silo-console#32](https://github.com/pgsty/silo-console/issues/32), [pgsty/silo-console#34](https://github.com/pgsty/silo-console/issues/34)<br>[pgsty/mc: Merged PRs (24)](https://github.com/pgsty/mc/issues?q=author%3AVonng+is%3Apr+is%3Amerged)<br>[pgsty/mc: Closed issues (18)](https://github.com/pgsty/mc/issues?q=author%3AVonng+is%3Aissue+is%3Aclosed)<br>Merged PRs: [pgsty/silo-pkg#1](https://github.com/pgsty/silo-pkg/pull/1), [pgsty/silo-pkg#2](https://github.com/pgsty/silo-pkg/pull/2), [pgsty/silo-pkg#3](https://github.com/pgsty/silo-pkg/pull/3), [pgsty/silo-pkg#4](https://github.com/pgsty/silo-pkg/pull/4), [pgsty/silo-pkg#5](https://github.com/pgsty/silo-pkg/pull/5), [pgsty/silo-pkg#6](https://github.com/pgsty/silo-pkg/pull/6), [pgsty/silo-pkg#7](https://github.com/pgsty/silo-pkg/pull/7), [pgsty/silo-pkg#8](https://github.com/pgsty/silo-pkg/pull/8), [pgsty/silo-pkg#9](https://github.com/pgsty/silo-pkg/pull/9)<br>[pgsty/silo.pgsty.com: Merged PRs (24)](https://github.com/pgsty/silo.pgsty.com/issues?q=author%3AVonng+is%3Apr+is%3Amerged)<br>Open PRs: [pgsty/silo.pgsty.com#25](https://github.com/pgsty/silo.pgsty.com/pull/25) |
| [@ZouhairCharef](https://github.com/ZouhairCharef) | Patched CVE-2026-34986 in go-jose | Merged PRs: [pgsty/silo#18](https://github.com/pgsty/silo/pull/18) |
| [@mfredenhagen](https://github.com/mfredenhagen) | Patched CVE-2026-39883 in OpenTelemetry | Merged PRs: [pgsty/silo#19](https://github.com/pgsty/silo/pull/19) |
| [@pinginfo](https://github.com/pinginfo) | Repaired bucket notification streaming | Merged PRs: [pgsty/silo#34](https://github.com/pgsty/silo/pull/34) |
| [@ycjlin](https://github.com/ycjlin) | Fixed missing-bucket ListObjects semantics | Merged PRs: [pgsty/silo#37](https://github.com/pgsty/silo/pull/37) |
| [@waterkip](https://github.com/waterkip) | Repointed documentation links to the SILO portal | Merged PRs: [pgsty/silo#41](https://github.com/pgsty/silo/pull/41) |
| [@Dansyuqri](https://github.com/Dansyuqri) | Added ChecksumType to multipart completion responses | Merged PRs: [pgsty/silo#57](https://github.com/pgsty/silo/pull/57) |
| [@mrjavadseydi](https://github.com/mrjavadseydi) | Fixed bucket quota metrics; contributed access-frequency ILM and S3-compatible multipart listing proposals | Merged PRs: [pgsty/silo#60](https://github.com/pgsty/silo/pull/60), [pgsty/silo#132](https://github.com/pgsty/silo/pull/132), [pgsty/silo#198](https://github.com/pgsty/silo/pull/198)<br>The access-frequency feature in [#60](https://github.com/pgsty/silo/pull/60) was merged and later removed. The multipart-listing contribution in [#198](https://github.com/pgsty/silo/pull/198) was merged after follow-up durability and cancellation repairs; it remains unreleased. |
| [@h5vx](https://github.com/h5vx) | Implemented per-bucket CORS configuration and enforcement | Merged PRs: [pgsty/silo#71](https://github.com/pgsty/silo/pull/71) |
| [@nikitapogromsky](https://github.com/nikitapogromsky) | Fixed duplicate logger targets that broke Metrics V3 collection | Merged PRs: [pgsty/silo#151](https://github.com/pgsty/silo/pull/151)<br>Closed issues: [pgsty/silo#150](https://github.com/pgsty/silo/issues/150) |
| [@Aeirx](https://github.com/Aeirx) | Preserved Object Lock legal-hold headers in federated CopyObject | Merged PRs: [pgsty/silo#172](https://github.com/pgsty/silo/pull/172) |
Pull requests that are open for review, or that were closed after informing work that shipped
differently.
## Other pull-request authors
| Contributor | Change | Pull request | Status |
| :-- | :-- | :-- | :-- |
| [@magicxor](https://github.com/magicxor) | `DELETE` precondition checks for the `If-Match` header | [#12](https://github.com/pgsty/silo/pull/12) | Open, queued for review |
| [@davinkevin](https://github.com/davinkevin) | Distroless-based Docker image variant | [#21](https://github.com/pgsty/silo/pull/21) | Superseded by the distroless variant shipped in RELEASE.2026-08-06, which the PR anticipated by four months |
| [@lem21h](https://github.com/lem21h) | Assorted fixes and improvements | [#36](https://github.com/pgsty/silo/pull/36) | Closed |
| [@sulin37392](https://github.com/sulin37392) | Dependency updates against the fork | [#8](https://github.com/pgsty/silo/pull/8) | Closed |
| Contributor | Contribution | Record |
| :-- | :-- | :-- |
| [@magicxor](https://github.com/magicxor) | Reported and proposed conditional DELETE support for If-Match | PRs closed without merging: [pgsty/silo#12](https://github.com/pgsty/silo/pull/12)<br>Closed issues: [pgsty/silo#10](https://github.com/pgsty/silo/issues/10)<br>The original PR closed after conditional DELETE support landed in [#145](https://github.com/pgsty/silo/pull/145). |
| [@davinkevin](https://github.com/davinkevin) | Proposed the distroless container image and dependency automation | PRs closed without merging: [pgsty/silo#21](https://github.com/pgsty/silo/pull/21)<br>Open issues: [pgsty/silo#20](https://github.com/pgsty/silo/issues/20)<br>The distroless proposal was superseded by the image shipped in [RELEASE.2026-08-06](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-06T00-00-00Z). |
| [@metaneutrons](https://github.com/metaneutrons) | Reported and proposed explicit-version delete authorization | PRs closed without merging: [pgsty/silo#59](https://github.com/pgsty/silo/pull/59)<br>Closed issues: [pgsty/silo#58](https://github.com/pgsty/silo/issues/58)<br>The original PR was superseded; its authorization action split informed [#104](https://github.com/pgsty/silo/pull/104). |
| [@mikemikimike](https://github.com/mikemikimike) | Contributed the replicated SSE-C plaintext part-size fix | PRs closed without merging: [pgsty/silo#125](https://github.com/pgsty/silo/pull/125)<br>The original PR was superseded; its plaintext-size fix was incorporated through [#124](https://github.com/pgsty/silo/pull/124), with attribute handling in [#128](https://github.com/pgsty/silo/pull/128). |
| [@julienlau](https://github.com/julienlau) | Reported MRF queue and delete-marker retry defects; contributed repair proposals, tests, and documentation | PRs closed without merging: [pgsty/silo#184](https://github.com/pgsty/silo/pull/184)<br>Closed issues: [pgsty/silo#152](https://github.com/pgsty/silo/issues/152), [pgsty/silo#153](https://github.com/pgsty/silo/issues/153)<br>The original PR was not merged directly. Its purge-state and recovery analysis, proposed fix, tests, and documentation helped shape [#196](https://github.com/pgsty/silo/pull/196); see the [maintainer acknowledgement](https://github.com/pgsty/silo/pull/184#issuecomment-5689295222). |
| [@chodorenko](https://github.com/chodorenko) | Reported and contributed the fix for aws-chunked leaking into replica metadata | PRs closed without merging: [pgsty/silo#187](https://github.com/pgsty/silo/pull/187)<br>Closed issues: [pgsty/silo#185](https://github.com/pgsty/silo/issues/185)<br>The implementation was incorporated in [#194](https://github.com/pgsty/silo/pull/194), with [co-author credit](https://github.com/pgsty/silo/commit/4fcdf37ce656152b32ad0f615d47f5e3f9748c3a); the original PR was closed as superseded. |
| [@sulin37392](https://github.com/sulin37392) | Proposed dependency updates | PRs closed without merging: [pgsty/silo#8](https://github.com/pgsty/silo/pull/8) |
| [@lem21h](https://github.com/lem21h) | Proposed robustness and goroutine improvements | PRs closed without merging: [pgsty/silo#36](https://github.com/pgsty/silo/pull/36) |
## Reports
## Issue reports
Bug reports, compatibility findings, and proposals filed against this fork. Several shipped fixes
trace directly back to these: the bundled-client guarantee (#4, #9), the LDAP-over-TLS repair (#15),
the completed native package payload (#33), GPG-signed RPMs (#43), and the upstream migration guide
(#42).
| Contributor | Contribution | Record |
| :-- | :-- | :-- |
| [@vampywiz17](https://github.com/vampywiz17) | Reported LDAP TLS and Console login regressions | Closed issues: [pgsty/silo#15](https://github.com/pgsty/silo/issues/15), [pgsty/silo#108](https://github.com/pgsty/silo/issues/108) |
| [@cbornet](https://github.com/cbornet) | Reported multipart and streaming checksum defects and missing-bucket semantics | Closed issues: [pgsty/silo#31](https://github.com/pgsty/silo/issues/31), [pgsty/silo#32](https://github.com/pgsty/silo/issues/32), [pgsty/silo#107](https://github.com/pgsty/silo/issues/107) |
| [@orenyomtov](https://github.com/orenyomtov) | Reported the unsigned-header CopyObject cross-object read (SN-2026-011) | Security disclosure credited in [SN-2026-011](https://silo.pgsty.com/about/security-advisories/#sn-2026-011), with the source fix in [#173](https://github.com/pgsty/silo/pull/173). This credit is retained separately from public issue/PR authorship. |
| [@jiri-pejchal](https://github.com/jiri-pejchal) | Reported the RabbitMQ client vulnerability and the Console share proxy's exposure of internal metrics | Closed issues: [pgsty/silo#176](https://github.com/pgsty/silo/issues/176)<br>Closed issues: [pgsty/silo-console#52](https://github.com/pgsty/silo-console/issues/52) |
| [@AEGEGE](https://github.com/AEGEGE) | Reported the slow-HTTP denial-of-service vulnerability | Closed issues: [pgsty/silo#183](https://github.com/pgsty/silo/issues/183)<br>The reproduced slow-header defect was fixed on main through [#196](https://github.com/pgsty/silo/pull/196). |
| [@mosesdd](https://github.com/mosesdd) | Requested a maintained Helm chart | Closed issues: [pgsty/silo#1](https://github.com/pgsty/silo/issues/1) |
| [@Xavier-777](https://github.com/Xavier-777) | Reported Console lifecycle management and file preview gaps | Closed issues: [pgsty/silo#2](https://github.com/pgsty/silo/issues/2), [pgsty/silo#17](https://github.com/pgsty/silo/issues/17) |
| [@jiadzh](https://github.com/jiadzh) | Requested Windows build guidance | Closed issues: [pgsty/silo#3](https://github.com/pgsty/silo/issues/3) |
| [@AntonOfTheWoods](https://github.com/AntonOfTheWoods) | Asked for clarity on Helm chart and operator options | Closed issues: [pgsty/silo#5](https://github.com/pgsty/silo/issues/5) |
| [@TLINDEN](https://github.com/TLINDEN) | Reported the missing client in release tarballs | Closed issues: [pgsty/silo#4](https://github.com/pgsty/silo/issues/4) |
| [@zylpsrs](https://github.com/zylpsrs) | Reported missing Console tiering and site replication | Closed issues: [pgsty/silo#6](https://github.com/pgsty/silo/issues/6) |
| [@nsanitate](https://github.com/nsanitate) | Proposed CNCF Sandbox governance | Closed issues: [pgsty/silo#7](https://github.com/pgsty/silo/issues/7) |
| [@makinikm](https://github.com/makinikm) | Reported the missing client in the container image | Closed issues: [pgsty/silo#9](https://github.com/pgsty/silo/issues/9) |
| [@spaceg00se-r](https://github.com/spaceg00se-r) | Requested cpuv1 support and reported a workflow token failure | Closed issues: [pgsty/silo#11](https://github.com/pgsty/silo/issues/11), [pgsty/silo#14](https://github.com/pgsty/silo/issues/14) |
| [@heroes1412](https://github.com/heroes1412) | Reported the unusable profiling option | Closed issues: [pgsty/silo#13](https://github.com/pgsty/silo/issues/13) |
| [@chalukyaj](https://github.com/chalukyaj) | Proposed making the SILO Operator easier to discover | Open issues: [pgsty/silo#30](https://github.com/pgsty/silo/issues/30) |
| [@jvasile](https://github.com/jvasile) | Reported missing user, group, and defaults in Debian packages | Closed issues: [pgsty/silo#33](https://github.com/pgsty/silo/issues/33) |
| [@Kesavaambati](https://github.com/Kesavaambati) | Asked about community support and image maintenance | Closed issues: [pgsty/silo#35](https://github.com/pgsty/silo/issues/35) |
| [@redfoxfox](https://github.com/redfoxfox) | Reported Chinese documentation availability | Closed issues: [pgsty/silo#38](https://github.com/pgsty/silo/issues/38) |
| [@kuldeep-link11](https://github.com/kuldeep-link11) | Reported NATS JWT credentials and target reload issues | Closed issues: [pgsty/silo#39](https://github.com/pgsty/silo/issues/39)<br>Open issues: [pgsty/silo#40](https://github.com/pgsty/silo/issues/40) |
| [@meesudzu](https://github.com/meesudzu) | Requested the migration guide from upstream MinIO | Closed issues: [pgsty/silo#42](https://github.com/pgsty/silo/issues/42) |
| [@pmezhuev](https://github.com/pmezhuev) | Reported missing RPM package signatures | Closed issues: [pgsty/silo#43](https://github.com/pgsty/silo/issues/43) |
| [@kh0mka](https://github.com/kh0mka) | Reported inter-node I/O timeouts and OIDC configuration-fetch failures | Closed issues: [pgsty/silo#51](https://github.com/pgsty/silo/issues/51), [pgsty/silo#154](https://github.com/pgsty/silo/issues/154) |
| [@bagutzu](https://github.com/bagutzu) | Requested KES-compatible external KMS and OpenBao support | Open issues: [pgsty/silo#61](https://github.com/pgsty/silo/issues/61) |
| [@liuhaodongliu990-cmyk](https://github.com/liuhaodongliu990-cmyk) | Reported indeterminate progress for prefix downloads | Closed issues: [pgsty/silo#62](https://github.com/pgsty/silo/issues/62) |
| [@DestroyLee](https://github.com/DestroyLee) | Reported the missing documentation navigation | Closed issues: [pgsty/silo.pgsty.com#4](https://github.com/pgsty/silo.pgsty.com/issues/4) |
| [@sargarass](https://github.com/sargarass) | Reported ListMultipartUploads prefix and pagination semantics | Open issues: [pgsty/silo#79](https://github.com/pgsty/silo/issues/79) |
| [@mumu-lab](https://github.com/mumu-lab) | Reported bucket quota metrics reading a deprecated field | Closed issues: [pgsty/silo#106](https://github.com/pgsty/silo/issues/106) |
| [@haiming236](https://github.com/haiming236) | Proposed a built-in image processing pipeline | Open issues: [pgsty/silo#160](https://github.com/pgsty/silo/issues/160) |
| [@tiredenzo](https://github.com/tiredenzo) | Reported inconsistent documentation for two-drive EC:1 support | Open issues: [pgsty/silo#197](https://github.com/pgsty/silo/issues/197) |
| [@aschyolkin](https://github.com/aschyolkin) | Reported a data race in CPU metrics collection | Open issues: [pgsty/silo#210](https://github.com/pgsty/silo/issues/210) |
| Contributor | Reported |
| :-- | :-- |
| [@mosesdd](https://github.com/mosesdd) | [#1](https://github.com/pgsty/silo/issues/1) Helm chart availability |
| [@Xavier-777](https://github.com/Xavier-777) | [#2](https://github.com/pgsty/silo/issues/2) Console bucket lifecycle management · [#17](https://github.com/pgsty/silo/issues/17) Log and XML file preview |
| [@jiadzh](https://github.com/jiadzh) | [#3](https://github.com/pgsty/silo/issues/3) Windows build guidance |
| [@TLINDEN](https://github.com/TLINDEN) | [#4](https://github.com/pgsty/silo/issues/4) `mc` missing from released tarballs |
| [@AntonOfTheWoods](https://github.com/AntonOfTheWoods) | [#5](https://github.com/pgsty/silo/issues/5) Upstream Helm chart and operator options |
| [@zylpsrs](https://github.com/zylpsrs) | [#6](https://github.com/pgsty/silo/issues/6) Console missing Tiering and Site Replication |
| [@nsanitate](https://github.com/nsanitate) | [#7](https://github.com/pgsty/silo/issues/7) CNCF Sandbox governance proposal |
| [@makinikm](https://github.com/makinikm) | [#9](https://github.com/pgsty/silo/issues/9) `mc` missing from the Docker image |
| [@magicxor](https://github.com/magicxor) | [#10](https://github.com/pgsty/silo/issues/10) `DeleteObject` ignores the `If-Match` header |
| [@spaceg00se-r](https://github.com/spaceg00se-r) | [#11](https://github.com/pgsty/silo/issues/11) `cpuv1` support · [#14](https://github.com/pgsty/silo/issues/14) Project workflow token failure |
| [@heroes1412](https://github.com/heroes1412) | [#13](https://github.com/pgsty/silo/issues/13) Profile option unusable |
| [@vampywiz17](https://github.com/vampywiz17) | [#15](https://github.com/pgsty/silo/issues/15) LDAP TLS regression breaking Console login on Kubernetes |
| [@davinkevin](https://github.com/davinkevin) | [#20](https://github.com/pgsty/silo/issues/20) Renovate for automated dependency updates |
| [@chalukyaj](https://github.com/chalukyaj) | [#30](https://github.com/pgsty/silo/issues/30) Silo Operator discoverability |
| [@cbornet](https://github.com/cbornet) | [#31](https://github.com/pgsty/silo/issues/31) Multipart uploads with `FULL_OBJECT` CRC32 · [#32](https://github.com/pgsty/silo/issues/32) `ListObjects` bucket-existence semantics |
| [@jvasile](https://github.com/jvasile) | [#33](https://github.com/pgsty/silo/issues/33) `.deb` missing user, group, and default files |
| [@Kesavaambati](https://github.com/Kesavaambati) | [#35](https://github.com/pgsty/silo/issues/35) Community support for the Docker images |
| [@redfoxfox](https://github.com/redfoxfox) | [#38](https://github.com/pgsty/silo/issues/38) Chinese documentation site unreachable |
| [@kuldeep-link11](https://github.com/kuldeep-link11) | [#39](https://github.com/pgsty/silo/issues/39) `notify_nats` rejects JWT credentials files · [#40](https://github.com/pgsty/silo/issues/40) `notify_nats` target changes require a restart |
| [@meesudzu](https://github.com/meesudzu) | [#42](https://github.com/pgsty/silo/issues/42) Migration guide from upstream MinIO |
| [@pmezhuev](https://github.com/pmezhuev) | [#43](https://github.com/pgsty/silo/issues/43) RPM package missing its GPG signature |
| [@kh0mka](https://github.com/kh0mka) | [#51](https://github.com/pgsty/silo/issues/51) Inter-node I/O timeout in `ReadFileStreamHandler` |
## Audit scope
## Adding yourself
Counts below come from every page of the GitHub issue / PR records, across all states, including automated accounts. There are 49 distinct human public issue / PR authors, plus 1 credited security reporter. Excluded automated accounts: `Copilot`, `dependabot[bot]`.
Contributions are accepted inbound=outbound under AGPL-3.0-or-later with no CLA; see
[`CONTRIBUTING.md`](CONTRIBUTING.md). Merged pull requests are added here at the next release. If a
contribution is missing or recorded incorrectly, open an issue or say so on the pull request and it
will be fixed.
| Repository | Issues | Pull requests |
| :-- | --: | --: |
| [pgsty/silo](https://github.com/pgsty/silo) | 108 | 99 |
| [pgsty/silo-console](https://github.com/pgsty/silo-console) | 35 | 22 |
| [pgsty/mc](https://github.com/pgsty/mc) | 18 | 25 |
| [pgsty/silo-pkg](https://github.com/pgsty/silo-pkg) | 0 | 9 |
| [pgsty/silo-go](https://github.com/pgsty/silo-go) | 0 | 0 |
| [pgsty/kes](https://github.com/pgsty/kes) | 0 | 0 |
| [pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com) | 1 | 26 |
| [pgsty/minio-docs](https://github.com/pgsty/minio-docs) | 0 | 0 |
| **Total** | **162** | **181** |
## Maintainer record
<details>
<summary>@Vonng — complete issue / PR record</summary>
Merged PRs: [pgsty/silo#44](https://github.com/pgsty/silo/pull/44), [pgsty/silo#45](https://github.com/pgsty/silo/pull/45), [pgsty/silo#56](https://github.com/pgsty/silo/pull/56), [pgsty/silo#66](https://github.com/pgsty/silo/pull/66), [pgsty/silo#69](https://github.com/pgsty/silo/pull/69), [pgsty/silo#70](https://github.com/pgsty/silo/pull/70), [pgsty/silo#72](https://github.com/pgsty/silo/pull/72), [pgsty/silo#73](https://github.com/pgsty/silo/pull/73), [pgsty/silo#74](https://github.com/pgsty/silo/pull/74), [pgsty/silo#80](https://github.com/pgsty/silo/pull/80), [pgsty/silo#81](https://github.com/pgsty/silo/pull/81), [pgsty/silo#85](https://github.com/pgsty/silo/pull/85), [pgsty/silo#86](https://github.com/pgsty/silo/pull/86), [pgsty/silo#87](https://github.com/pgsty/silo/pull/87), [pgsty/silo#88](https://github.com/pgsty/silo/pull/88), [pgsty/silo#89](https://github.com/pgsty/silo/pull/89), [pgsty/silo#90](https://github.com/pgsty/silo/pull/90), [pgsty/silo#91](https://github.com/pgsty/silo/pull/91), [pgsty/silo#92](https://github.com/pgsty/silo/pull/92), [pgsty/silo#93](https://github.com/pgsty/silo/pull/93), [pgsty/silo#94](https://github.com/pgsty/silo/pull/94), [pgsty/silo#95](https://github.com/pgsty/silo/pull/95), [pgsty/silo#96](https://github.com/pgsty/silo/pull/96), [pgsty/silo#97](https://github.com/pgsty/silo/pull/97), [pgsty/silo#98](https://github.com/pgsty/silo/pull/98), [pgsty/silo#101](https://github.com/pgsty/silo/pull/101), [pgsty/silo#103](https://github.com/pgsty/silo/pull/103), [pgsty/silo#104](https://github.com/pgsty/silo/pull/104), [pgsty/silo#121](https://github.com/pgsty/silo/pull/121), [pgsty/silo#122](https://github.com/pgsty/silo/pull/122), [pgsty/silo#123](https://github.com/pgsty/silo/pull/123), [pgsty/silo#124](https://github.com/pgsty/silo/pull/124), [pgsty/silo#126](https://github.com/pgsty/silo/pull/126), [pgsty/silo#127](https://github.com/pgsty/silo/pull/127), [pgsty/silo#128](https://github.com/pgsty/silo/pull/128), [pgsty/silo#129](https://github.com/pgsty/silo/pull/129), [pgsty/silo#130](https://github.com/pgsty/silo/pull/130), [pgsty/silo#131](https://github.com/pgsty/silo/pull/131), [pgsty/silo#134](https://github.com/pgsty/silo/pull/134), [pgsty/silo#135](https://github.com/pgsty/silo/pull/135), [pgsty/silo#138](https://github.com/pgsty/silo/pull/138), [pgsty/silo#140](https://github.com/pgsty/silo/pull/140), [pgsty/silo#142](https://github.com/pgsty/silo/pull/142), [pgsty/silo#143](https://github.com/pgsty/silo/pull/143), [pgsty/silo#145](https://github.com/pgsty/silo/pull/145), [pgsty/silo#146](https://github.com/pgsty/silo/pull/146), [pgsty/silo#149](https://github.com/pgsty/silo/pull/149), [pgsty/silo#156](https://github.com/pgsty/silo/pull/156), [pgsty/silo#157](https://github.com/pgsty/silo/pull/157), [pgsty/silo#159](https://github.com/pgsty/silo/pull/159), [pgsty/silo#161](https://github.com/pgsty/silo/pull/161), [pgsty/silo#162](https://github.com/pgsty/silo/pull/162), [pgsty/silo#163](https://github.com/pgsty/silo/pull/163), [pgsty/silo#164](https://github.com/pgsty/silo/pull/164), [pgsty/silo#173](https://github.com/pgsty/silo/pull/173), [pgsty/silo#174](https://github.com/pgsty/silo/pull/174), [pgsty/silo#175](https://github.com/pgsty/silo/pull/175), [pgsty/silo#177](https://github.com/pgsty/silo/pull/177), [pgsty/silo#178](https://github.com/pgsty/silo/pull/178), [pgsty/silo#179](https://github.com/pgsty/silo/pull/179), [pgsty/silo#180](https://github.com/pgsty/silo/pull/180), [pgsty/silo#181](https://github.com/pgsty/silo/pull/181), [pgsty/silo#182](https://github.com/pgsty/silo/pull/182), [pgsty/silo#188](https://github.com/pgsty/silo/pull/188), [pgsty/silo#189](https://github.com/pgsty/silo/pull/189), [pgsty/silo#190](https://github.com/pgsty/silo/pull/190), [pgsty/silo#191](https://github.com/pgsty/silo/pull/191), [pgsty/silo#192](https://github.com/pgsty/silo/pull/192), [pgsty/silo#193](https://github.com/pgsty/silo/pull/193), [pgsty/silo#194](https://github.com/pgsty/silo/pull/194), [pgsty/silo#196](https://github.com/pgsty/silo/pull/196), [pgsty/silo#205](https://github.com/pgsty/silo/pull/205), [pgsty/silo#206](https://github.com/pgsty/silo/pull/206), [pgsty/silo#207](https://github.com/pgsty/silo/pull/207), [pgsty/silo#208](https://github.com/pgsty/silo/pull/208), [pgsty/silo#209](https://github.com/pgsty/silo/pull/209), [pgsty/silo#211](https://github.com/pgsty/silo/pull/211)
PRs closed without merging: [pgsty/silo#155](https://github.com/pgsty/silo/pull/155), [pgsty/silo#195](https://github.com/pgsty/silo/pull/195)
Closed issues: [pgsty/silo#22](https://github.com/pgsty/silo/issues/22), [pgsty/silo#23](https://github.com/pgsty/silo/issues/23), [pgsty/silo#24](https://github.com/pgsty/silo/issues/24), [pgsty/silo#25](https://github.com/pgsty/silo/issues/25), [pgsty/silo#26](https://github.com/pgsty/silo/issues/26), [pgsty/silo#27](https://github.com/pgsty/silo/issues/27), [pgsty/silo#28](https://github.com/pgsty/silo/issues/28), [pgsty/silo#46](https://github.com/pgsty/silo/issues/46), [pgsty/silo#47](https://github.com/pgsty/silo/issues/47), [pgsty/silo#48](https://github.com/pgsty/silo/issues/48), [pgsty/silo#49](https://github.com/pgsty/silo/issues/49), [pgsty/silo#50](https://github.com/pgsty/silo/issues/50), [pgsty/silo#52](https://github.com/pgsty/silo/issues/52), [pgsty/silo#53](https://github.com/pgsty/silo/issues/53), [pgsty/silo#55](https://github.com/pgsty/silo/issues/55), [pgsty/silo#63](https://github.com/pgsty/silo/issues/63), [pgsty/silo#64](https://github.com/pgsty/silo/issues/64), [pgsty/silo#65](https://github.com/pgsty/silo/issues/65), [pgsty/silo#67](https://github.com/pgsty/silo/issues/67), [pgsty/silo#68](https://github.com/pgsty/silo/issues/68), [pgsty/silo#75](https://github.com/pgsty/silo/issues/75), [pgsty/silo#76](https://github.com/pgsty/silo/issues/76), [pgsty/silo#77](https://github.com/pgsty/silo/issues/77), [pgsty/silo#78](https://github.com/pgsty/silo/issues/78), [pgsty/silo#82](https://github.com/pgsty/silo/issues/82), [pgsty/silo#83](https://github.com/pgsty/silo/issues/83), [pgsty/silo#84](https://github.com/pgsty/silo/issues/84), [pgsty/silo#99](https://github.com/pgsty/silo/issues/99), [pgsty/silo#100](https://github.com/pgsty/silo/issues/100), [pgsty/silo#102](https://github.com/pgsty/silo/issues/102), [pgsty/silo#105](https://github.com/pgsty/silo/issues/105), [pgsty/silo#109](https://github.com/pgsty/silo/issues/109), [pgsty/silo#110](https://github.com/pgsty/silo/issues/110), [pgsty/silo#111](https://github.com/pgsty/silo/issues/111), [pgsty/silo#112](https://github.com/pgsty/silo/issues/112), [pgsty/silo#113](https://github.com/pgsty/silo/issues/113), [pgsty/silo#114](https://github.com/pgsty/silo/issues/114), [pgsty/silo#115](https://github.com/pgsty/silo/issues/115), [pgsty/silo#116](https://github.com/pgsty/silo/issues/116), [pgsty/silo#117](https://github.com/pgsty/silo/issues/117), [pgsty/silo#118](https://github.com/pgsty/silo/issues/118), [pgsty/silo#119](https://github.com/pgsty/silo/issues/119), [pgsty/silo#120](https://github.com/pgsty/silo/issues/120), [pgsty/silo#133](https://github.com/pgsty/silo/issues/133), [pgsty/silo#136](https://github.com/pgsty/silo/issues/136), [pgsty/silo#137](https://github.com/pgsty/silo/issues/137), [pgsty/silo#139](https://github.com/pgsty/silo/issues/139), [pgsty/silo#141](https://github.com/pgsty/silo/issues/141), [pgsty/silo#144](https://github.com/pgsty/silo/issues/144), [pgsty/silo#147](https://github.com/pgsty/silo/issues/147), [pgsty/silo#148](https://github.com/pgsty/silo/issues/148), [pgsty/silo#158](https://github.com/pgsty/silo/issues/158), [pgsty/silo#165](https://github.com/pgsty/silo/issues/165), [pgsty/silo#166](https://github.com/pgsty/silo/issues/166), [pgsty/silo#167](https://github.com/pgsty/silo/issues/167), [pgsty/silo#168](https://github.com/pgsty/silo/issues/168), [pgsty/silo#169](https://github.com/pgsty/silo/issues/169), [pgsty/silo#170](https://github.com/pgsty/silo/issues/170), [pgsty/silo#171](https://github.com/pgsty/silo/issues/171), [pgsty/silo#204](https://github.com/pgsty/silo/issues/204)
Open issues: [pgsty/silo#199](https://github.com/pgsty/silo/issues/199), [pgsty/silo#200](https://github.com/pgsty/silo/issues/200), [pgsty/silo#201](https://github.com/pgsty/silo/issues/201), [pgsty/silo#202](https://github.com/pgsty/silo/issues/202), [pgsty/silo#203](https://github.com/pgsty/silo/issues/203)
Merged PRs: [pgsty/silo-console#9](https://github.com/pgsty/silo-console/pull/9), [pgsty/silo-console#10](https://github.com/pgsty/silo-console/pull/10), [pgsty/silo-console#11](https://github.com/pgsty/silo-console/pull/11), [pgsty/silo-console#38](https://github.com/pgsty/silo-console/pull/38), [pgsty/silo-console#39](https://github.com/pgsty/silo-console/pull/39), [pgsty/silo-console#40](https://github.com/pgsty/silo-console/pull/40), [pgsty/silo-console#41](https://github.com/pgsty/silo-console/pull/41), [pgsty/silo-console#42](https://github.com/pgsty/silo-console/pull/42), [pgsty/silo-console#43](https://github.com/pgsty/silo-console/pull/43), [pgsty/silo-console#44](https://github.com/pgsty/silo-console/pull/44), [pgsty/silo-console#45](https://github.com/pgsty/silo-console/pull/45), [pgsty/silo-console#46](https://github.com/pgsty/silo-console/pull/46), [pgsty/silo-console#47](https://github.com/pgsty/silo-console/pull/47), [pgsty/silo-console#48](https://github.com/pgsty/silo-console/pull/48), [pgsty/silo-console#49](https://github.com/pgsty/silo-console/pull/49), [pgsty/silo-console#50](https://github.com/pgsty/silo-console/pull/50), [pgsty/silo-console#51](https://github.com/pgsty/silo-console/pull/51), [pgsty/silo-console#53](https://github.com/pgsty/silo-console/pull/53), [pgsty/silo-console#54](https://github.com/pgsty/silo-console/pull/54), [pgsty/silo-console#55](https://github.com/pgsty/silo-console/pull/55), [pgsty/silo-console#56](https://github.com/pgsty/silo-console/pull/56), [pgsty/silo-console#57](https://github.com/pgsty/silo-console/pull/57)
Closed issues: [pgsty/silo-console#1](https://github.com/pgsty/silo-console/issues/1), [pgsty/silo-console#2](https://github.com/pgsty/silo-console/issues/2), [pgsty/silo-console#3](https://github.com/pgsty/silo-console/issues/3), [pgsty/silo-console#4](https://github.com/pgsty/silo-console/issues/4), [pgsty/silo-console#5](https://github.com/pgsty/silo-console/issues/5), [pgsty/silo-console#6](https://github.com/pgsty/silo-console/issues/6), [pgsty/silo-console#7](https://github.com/pgsty/silo-console/issues/7), [pgsty/silo-console#8](https://github.com/pgsty/silo-console/issues/8), [pgsty/silo-console#12](https://github.com/pgsty/silo-console/issues/12), [pgsty/silo-console#13](https://github.com/pgsty/silo-console/issues/13), [pgsty/silo-console#14](https://github.com/pgsty/silo-console/issues/14), [pgsty/silo-console#15](https://github.com/pgsty/silo-console/issues/15), [pgsty/silo-console#16](https://github.com/pgsty/silo-console/issues/16), [pgsty/silo-console#17](https://github.com/pgsty/silo-console/issues/17), [pgsty/silo-console#18](https://github.com/pgsty/silo-console/issues/18), [pgsty/silo-console#19](https://github.com/pgsty/silo-console/issues/19), [pgsty/silo-console#20](https://github.com/pgsty/silo-console/issues/20), [pgsty/silo-console#21](https://github.com/pgsty/silo-console/issues/21), [pgsty/silo-console#22](https://github.com/pgsty/silo-console/issues/22), [pgsty/silo-console#23](https://github.com/pgsty/silo-console/issues/23), [pgsty/silo-console#24](https://github.com/pgsty/silo-console/issues/24), [pgsty/silo-console#25](https://github.com/pgsty/silo-console/issues/25), [pgsty/silo-console#26](https://github.com/pgsty/silo-console/issues/26), [pgsty/silo-console#27](https://github.com/pgsty/silo-console/issues/27), [pgsty/silo-console#28](https://github.com/pgsty/silo-console/issues/28), [pgsty/silo-console#29](https://github.com/pgsty/silo-console/issues/29), [pgsty/silo-console#30](https://github.com/pgsty/silo-console/issues/30), [pgsty/silo-console#31](https://github.com/pgsty/silo-console/issues/31), [pgsty/silo-console#33](https://github.com/pgsty/silo-console/issues/33), [pgsty/silo-console#35](https://github.com/pgsty/silo-console/issues/35), [pgsty/silo-console#36](https://github.com/pgsty/silo-console/issues/36), [pgsty/silo-console#37](https://github.com/pgsty/silo-console/issues/37)
Open issues: [pgsty/silo-console#32](https://github.com/pgsty/silo-console/issues/32), [pgsty/silo-console#34](https://github.com/pgsty/silo-console/issues/34)
Merged PRs: [pgsty/mc#1](https://github.com/pgsty/mc/pull/1), [pgsty/mc#2](https://github.com/pgsty/mc/pull/2), [pgsty/mc#3](https://github.com/pgsty/mc/pull/3), [pgsty/mc#4](https://github.com/pgsty/mc/pull/4), [pgsty/mc#8](https://github.com/pgsty/mc/pull/8), [pgsty/mc#9](https://github.com/pgsty/mc/pull/9), [pgsty/mc#10](https://github.com/pgsty/mc/pull/10), [pgsty/mc#11](https://github.com/pgsty/mc/pull/11), [pgsty/mc#13](https://github.com/pgsty/mc/pull/13), [pgsty/mc#22](https://github.com/pgsty/mc/pull/22), [pgsty/mc#24](https://github.com/pgsty/mc/pull/24), [pgsty/mc#27](https://github.com/pgsty/mc/pull/27), [pgsty/mc#32](https://github.com/pgsty/mc/pull/32), [pgsty/mc#33](https://github.com/pgsty/mc/pull/33), [pgsty/mc#34](https://github.com/pgsty/mc/pull/34), [pgsty/mc#35](https://github.com/pgsty/mc/pull/35), [pgsty/mc#36](https://github.com/pgsty/mc/pull/36), [pgsty/mc#37](https://github.com/pgsty/mc/pull/37), [pgsty/mc#38](https://github.com/pgsty/mc/pull/38), [pgsty/mc#39](https://github.com/pgsty/mc/pull/39), [pgsty/mc#40](https://github.com/pgsty/mc/pull/40), [pgsty/mc#41](https://github.com/pgsty/mc/pull/41), [pgsty/mc#42](https://github.com/pgsty/mc/pull/42), [pgsty/mc#43](https://github.com/pgsty/mc/pull/43)
Closed issues: [pgsty/mc#5](https://github.com/pgsty/mc/issues/5), [pgsty/mc#6](https://github.com/pgsty/mc/issues/6), [pgsty/mc#7](https://github.com/pgsty/mc/issues/7), [pgsty/mc#12](https://github.com/pgsty/mc/issues/12), [pgsty/mc#14](https://github.com/pgsty/mc/issues/14), [pgsty/mc#15](https://github.com/pgsty/mc/issues/15), [pgsty/mc#16](https://github.com/pgsty/mc/issues/16), [pgsty/mc#17](https://github.com/pgsty/mc/issues/17), [pgsty/mc#18](https://github.com/pgsty/mc/issues/18), [pgsty/mc#19](https://github.com/pgsty/mc/issues/19), [pgsty/mc#20](https://github.com/pgsty/mc/issues/20), [pgsty/mc#21](https://github.com/pgsty/mc/issues/21), [pgsty/mc#23](https://github.com/pgsty/mc/issues/23), [pgsty/mc#25](https://github.com/pgsty/mc/issues/25), [pgsty/mc#28](https://github.com/pgsty/mc/issues/28), [pgsty/mc#29](https://github.com/pgsty/mc/issues/29), [pgsty/mc#30](https://github.com/pgsty/mc/issues/30), [pgsty/mc#31](https://github.com/pgsty/mc/issues/31)
Merged PRs: [pgsty/silo-pkg#1](https://github.com/pgsty/silo-pkg/pull/1), [pgsty/silo-pkg#2](https://github.com/pgsty/silo-pkg/pull/2), [pgsty/silo-pkg#3](https://github.com/pgsty/silo-pkg/pull/3), [pgsty/silo-pkg#4](https://github.com/pgsty/silo-pkg/pull/4), [pgsty/silo-pkg#5](https://github.com/pgsty/silo-pkg/pull/5), [pgsty/silo-pkg#6](https://github.com/pgsty/silo-pkg/pull/6), [pgsty/silo-pkg#7](https://github.com/pgsty/silo-pkg/pull/7), [pgsty/silo-pkg#8](https://github.com/pgsty/silo-pkg/pull/8), [pgsty/silo-pkg#9](https://github.com/pgsty/silo-pkg/pull/9)
Merged PRs: [pgsty/silo.pgsty.com#2](https://github.com/pgsty/silo.pgsty.com/pull/2), [pgsty/silo.pgsty.com#3](https://github.com/pgsty/silo.pgsty.com/pull/3), [pgsty/silo.pgsty.com#5](https://github.com/pgsty/silo.pgsty.com/pull/5), [pgsty/silo.pgsty.com#6](https://github.com/pgsty/silo.pgsty.com/pull/6), [pgsty/silo.pgsty.com#7](https://github.com/pgsty/silo.pgsty.com/pull/7), [pgsty/silo.pgsty.com#8](https://github.com/pgsty/silo.pgsty.com/pull/8), [pgsty/silo.pgsty.com#9](https://github.com/pgsty/silo.pgsty.com/pull/9), [pgsty/silo.pgsty.com#10](https://github.com/pgsty/silo.pgsty.com/pull/10), [pgsty/silo.pgsty.com#11](https://github.com/pgsty/silo.pgsty.com/pull/11), [pgsty/silo.pgsty.com#13](https://github.com/pgsty/silo.pgsty.com/pull/13), [pgsty/silo.pgsty.com#14](https://github.com/pgsty/silo.pgsty.com/pull/14), [pgsty/silo.pgsty.com#15](https://github.com/pgsty/silo.pgsty.com/pull/15), [pgsty/silo.pgsty.com#16](https://github.com/pgsty/silo.pgsty.com/pull/16), [pgsty/silo.pgsty.com#17](https://github.com/pgsty/silo.pgsty.com/pull/17), [pgsty/silo.pgsty.com#19](https://github.com/pgsty/silo.pgsty.com/pull/19), [pgsty/silo.pgsty.com#20](https://github.com/pgsty/silo.pgsty.com/pull/20), [pgsty/silo.pgsty.com#21](https://github.com/pgsty/silo.pgsty.com/pull/21), [pgsty/silo.pgsty.com#22](https://github.com/pgsty/silo.pgsty.com/pull/22), [pgsty/silo.pgsty.com#23](https://github.com/pgsty/silo.pgsty.com/pull/23), [pgsty/silo.pgsty.com#24](https://github.com/pgsty/silo.pgsty.com/pull/24), [pgsty/silo.pgsty.com#26](https://github.com/pgsty/silo.pgsty.com/pull/26), [pgsty/silo.pgsty.com#27](https://github.com/pgsty/silo.pgsty.com/pull/27), [pgsty/silo.pgsty.com#28](https://github.com/pgsty/silo.pgsty.com/pull/28), [pgsty/silo.pgsty.com#29](https://github.com/pgsty/silo.pgsty.com/pull/29)
Open PRs: [pgsty/silo.pgsty.com#25](https://github.com/pgsty/silo.pgsty.com/pull/25)
</details>
## Keeping this record current
The reviewed [website contributor data](https://github.com/pgsty/silo.pgsty.com/blob/main/data/home/contributors.yaml) is shared by this record, the component records, README avatar walls, and both website languages. After auditing all issue / PR pages and reviewing adoption evidence, update that data and run `python3 bin/contributors.py` and `python3 bin/contributor_avatars.py` in the site checkout. Validate the generated records with `python3 bin/contributors.py --check`.
Code contributions follow the no-CLA, DCO policy in [CONTRIBUTING.md](CONTRIBUTING.md).
+19 -11
View File
@@ -1,4 +1,15 @@
FROM golang:1.27.1-alpine AS build
FROM golang:1.27.1-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS curl-build
ARG TARGETARCH
COPY dockerscripts/build-static-curl.sh /build/build-static-curl
RUN /bin/sh /build/build-static-curl
# Exercise the exact shipped curl without a dynamic loader or shared libraries.
FROM scratch AS curl-runtime
COPY --from=curl-build /go/bin/curl /curl
COPY --from=curl-build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/curl"]
FROM golang:1.27.1-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS build
ARG TARGETARCH
@@ -6,9 +17,9 @@ ENV GOPATH=/go
ENV CGO_ENABLED=0
ARG MC_REPO=pgsty/mc
ARG MC_VERSION=RELEASE.2026-09-03T07-13-05Z
ARG MC_AMD64_SHA256=6387cbeebb17c4bd52b447ee332ba22777e129034befa6598308c3aa04f02d09
ARG MC_ARM64_SHA256=5fc434c7e416bb4787e92306a8165d55284aac639a29744160b2a198fdd7573a
ARG MC_VERSION=RELEASE.2026-09-13T00-00-00Z
ARG MC_AMD64_SHA256=9d2a92de9c7b887d9b944fe9ddce68d23f1b6df3415092e737594e56593f5e2b
ARG MC_ARM64_SHA256=3d82e9ea6c601c4cb44fe5dd5f2ad1b7d7d64369378110f9ada9c524688a452a
RUN apk add -U --no-cache \
ca-certificates \
@@ -56,18 +67,14 @@ RUN apk add -U --no-cache \
chmod +x /go/bin/mcli && \
ln -sf mcli /go/bin/mc
COPY dockerscripts/download-static-curl.sh /build/download-static-curl
RUN chmod +x /build/download-static-curl && \
/build/download-static-curl
FROM registry.access.redhat.com/ubi9/ubi:latest AS certs
FROM registry.access.redhat.com/ubi9/ubi:latest@sha256:206b65b8ee0f04b992818c9a51b29081b14974630d4850bc358097d0c44ea156 AS certs
RUN dnf -y install ca-certificates && \
update-ca-trust && \
cp /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /tmp/ca-certificates.crt && \
dnf clean all && \
rm -rf /var/cache/dnf
FROM registry.access.redhat.com/ubi9/ubi-micro:latest
FROM registry.access.redhat.com/ubi9/ubi-micro:latest@sha256:f332c99eb8f798a8486821c91937f10ad64ee83d7e739303be2df051040918f6
LABEL org.opencontainers.image.title="Silo" \
org.opencontainers.image.description="S3-Interface Libre Object Storage" \
@@ -88,7 +95,8 @@ ENV MINIO_ACCESS_KEY_FILE=access_key \
COPY --from=certs /tmp/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY silo /usr/bin/silo
COPY --from=build /go/bin/mcli /usr/bin/mcli
COPY --from=build /go/bin/curl* /usr/bin/
COPY --from=curl-build /go/bin/curl /usr/bin/curl
COPY --from=curl-build /go/share/curl /licenses/curl
COPY dockerscripts/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
COPY LICENSE /licenses/LICENSE
COPY NOTICE /licenses/NOTICE
+1 -1
View File
@@ -216,7 +216,7 @@ docker: checks build-debugging ## builds the local Linux Silo container image
--ldflags "$(LDFLAGS)" -o "$$context/silo"; \
mkdir -p "$$context/dockerscripts"; \
cp Dockerfile.goreleaser LICENSE NOTICE CREDITS "$$context/"; \
cp dockerscripts/docker-entrypoint.sh dockerscripts/download-static-curl.sh \
cp dockerscripts/docker-entrypoint.sh dockerscripts/build-static-curl.sh \
"$$context/dockerscripts/"; \
docker build -q --no-cache --platform linux/$(GOARCH) -t $(TAG) --build-arg TARGETARCH=$(GOARCH) \
-f "$$context/Dockerfile.goreleaser" "$$context"
+59 -47
View File
@@ -35,6 +35,14 @@
> [!NOTE]
> Renamed from `pgsty/minio` to `pgsty/silo`, default branch `master` → `main`, on 2026-08-06. Artifacts under the original MinIO identity stay published on the archived [`minio`](https://github.com/pgsty/silo/tree/minio) branch and in releases up to [`RELEASE.2026-08-04T00-00-00Z`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z).
## Current release and main branch
The latest published Server is [20260903](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z).
As of 2026-09-16, the main branch has newer security, storage, Console and
shared-package changes that have not shipped in a Server release. See
[CHANGELOG.md](CHANGELOG.md) and the [component version matrix](https://silo.pgsty.com/compatibility/versions/)
for the exact release/source boundary, including SN-2026-011 and password-policy migration.
## Overview
PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
@@ -85,63 +93,67 @@ Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub
## Compatibility
The S3 API, `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, the `github.com/minio/*` import paths, and the on-disk format (including `.minio.sys`) are preserved and held in place by a CI compatibility check. Only Silo-owned delivery surfaces change: the `silo` executable, package, service, Helm chart, and container image — no `minio` binary alias is installed.
Silo preserves S3 and storage-format compatibility, including existing `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, and `.minio.sys` data. CI guards selected compatibility identifiers; release notes document intentional security and behavior changes. Silo-owned delivery surfaces use the `silo` executable, package, service, Helm chart, and container image; no `minio` server binary alias is installed.
The supported release stack is `pgsty/silo` + `pgsty/silo-console` + `pgsty/mc` + `pgsty/silo-pkg`; compatibility with unmodified upstream MinIO/MC is best effort. The Server, Console, and client retain their historical module paths where needed, while maintained code imports `github.com/pgsty/silo-pkg/v3` directly. The SDK `github.com/minio/minio-go/v7` is an explicit upstream dependency. See the current [go.mod](go.mod) for versions and replacements.
Every divergence from upstream is listed in the code-verified [compatibility audit](https://silo.pgsty.com/compatibility/server/). Treat each release as a downstream upgrade: pin versions, read the [release notes](https://silo.pgsty.com/tags/silo/), and keep a rollback path.
### TLS and Go upgrades
The following TLS repair is on main and is not included in Server 20260903.
With that repair, TLS key exchange follows Go's defaults across the S3 listener, node links,
replication, identity providers, etcd, and external HTTP services. If an endpoint
cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges
for the process; certificate verification remains enabled. This option does not
disable ML-DSA signatures or resolve every TLS reset. Prefer updating the
incompatible endpoint before removing the temporary setting.
If only the new SecP hybrids cause problems, `GODEBUG=tlssecpmlkem=0` disables
those groups while retaining X25519MLKEM768.
For builds targeting Go 1.27, setting either `SSL_CERT_FILE` or `SSL_CERT_DIR`
on macOS replaces Keychain trust with on-disk roots and Go's verifier. Stale or
incomplete CA paths can break previously trusted connections; unset inherited
values to restore Keychain trust. Explicit certificates in the configured `CAs`
directory remain additive to the selected root pool.
Go 1.27 binaries require macOS 13 or later. See the
[Go release notes](https://go.dev/doc/go1.27) and the
[Go 1.27 TLS and OIDC discovery guide](https://silo.pgsty.com/blog/design/go127-tls-oidc-discovery/).
## Documentation ownership
User documentation is maintained at [silo.pgsty.com](https://silo.pgsty.com/docs/),
with source in [pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com).
The remaining `docs/` tree contains inherited references, examples, and tooling
fixtures. Investigation logs, AI work records, and temporary reports are kept
outside this repository; reusable findings belong in the companion site.
See [AGENTS.md](AGENTS.md) for repository ownership and maintenance rules.
## Security & Contributing
Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); every fix ships with a public [advisory](https://silo.pgsty.com/blog/security/). Contributions are accepted inbound=outbound under AGPL-3.0-or-later with no CLA — only DCO sign-off (`git commit -s`) is required; see [`CONTRIBUTING.md`](CONTRIBUTING.md).
## Contributors
<table>
<tr>
<td align="center" width="150">
<a href="https://github.com/ZouhairCharef"><img src="https://github.com/ZouhairCharef.png?size=100" width="72" alt="ZouhairCharef"><br><sub><b>@ZouhairCharef</b></sub></a><br><sub>CVE-2026-34986</sub>
</td>
<td align="center" width="150">
<a href="https://github.com/mfredenhagen"><img src="https://github.com/mfredenhagen.png?size=100" width="72" alt="mfredenhagen"><br><sub><b>@mfredenhagen</b></sub></a><br><sub>CVE-2026-39883</sub>
</td>
<td align="center" width="150">
<a href="https://github.com/pinginfo"><img src="https://github.com/pinginfo.png?size=100" width="72" alt="pinginfo"><br><sub><b>@pinginfo</b></sub></a><br><sub>Notification streaming</sub>
</td>
<td align="center" width="150">
<a href="https://github.com/waterkip"><img src="https://github.com/waterkip.png?size=100" width="72" alt="waterkip"><br><sub><b>@waterkip</b></sub></a><br><sub>Documentation links</sub>
</td>
</tr>
</table>
<p>
<a href="https://github.com/magicxor"><img src="https://github.com/magicxor.png?size=64" width="44" alt="magicxor" title="@magicxor"></a>
<a href="https://github.com/ycjlin"><img src="https://github.com/ycjlin.png?size=64" width="44" alt="ycjlin" title="@ycjlin"></a>
<a href="https://github.com/h5vx"><img src="https://github.com/h5vx.png?size=64" width="44" alt="h5vx" title="@h5vx"></a>
<a href="https://github.com/Dansyuqri"><img src="https://github.com/Dansyuqri.png?size=64" width="44" alt="Dansyuqri" title="@Dansyuqri"></a>
<a href="https://github.com/davinkevin"><img src="https://github.com/davinkevin.png?size=64" width="44" alt="davinkevin" title="@davinkevin"></a>
<a href="https://github.com/lem21h"><img src="https://github.com/lem21h.png?size=64" width="44" alt="lem21h" title="@lem21h"></a>
<a href="https://github.com/sulin37392"><img src="https://github.com/sulin37392.png?size=64" width="44" alt="sulin37392" title="@sulin37392"></a>
<a href="https://github.com/mosesdd"><img src="https://github.com/mosesdd.png?size=64" width="44" alt="mosesdd" title="@mosesdd"></a>
<a href="https://github.com/Xavier-777"><img src="https://github.com/Xavier-777.png?size=64" width="44" alt="Xavier-777" title="@Xavier-777"></a>
<a href="https://github.com/jiadzh"><img src="https://github.com/jiadzh.png?size=64" width="44" alt="jiadzh" title="@jiadzh"></a>
<a href="https://github.com/TLINDEN"><img src="https://github.com/TLINDEN.png?size=64" width="44" alt="TLINDEN" title="@TLINDEN"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://github.com/AntonOfTheWoods.png?size=64" width="44" alt="AntonOfTheWoods" title="@AntonOfTheWoods"></a>
<a href="https://github.com/zylpsrs"><img src="https://github.com/zylpsrs.png?size=64" width="44" alt="zylpsrs" title="@zylpsrs"></a>
<a href="https://github.com/nsanitate"><img src="https://github.com/nsanitate.png?size=64" width="44" alt="nsanitate" title="@nsanitate"></a>
<a href="https://github.com/makinikm"><img src="https://github.com/makinikm.png?size=64" width="44" alt="makinikm" title="@makinikm"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://github.com/spaceg00se-r.png?size=64" width="44" alt="spaceg00se-r" title="@spaceg00se-r"></a>
<a href="https://github.com/heroes1412"><img src="https://github.com/heroes1412.png?size=64" width="44" alt="heroes1412" title="@heroes1412"></a>
<a href="https://github.com/vampywiz17"><img src="https://github.com/vampywiz17.png?size=64" width="44" alt="vampywiz17" title="@vampywiz17"></a>
<a href="https://github.com/chalukyaj"><img src="https://github.com/chalukyaj.png?size=64" width="44" alt="chalukyaj" title="@chalukyaj"></a>
<a href="https://github.com/cbornet"><img src="https://github.com/cbornet.png?size=64" width="44" alt="cbornet" title="@cbornet"></a>
<a href="https://github.com/jvasile"><img src="https://github.com/jvasile.png?size=64" width="44" alt="jvasile" title="@jvasile"></a>
<a href="https://github.com/Kesavaambati"><img src="https://github.com/Kesavaambati.png?size=64" width="44" alt="Kesavaambati" title="@Kesavaambati"></a>
<a href="https://github.com/redfoxfox"><img src="https://github.com/redfoxfox.png?size=64" width="44" alt="redfoxfox" title="@redfoxfox"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://github.com/kuldeep-link11.png?size=64" width="44" alt="kuldeep-link11" title="@kuldeep-link11"></a>
<a href="https://github.com/meesudzu"><img src="https://github.com/meesudzu.png?size=64" width="44" alt="meesudzu" title="@meesudzu"></a>
<a href="https://github.com/pmezhuev"><img src="https://github.com/pmezhuev.png?size=64" width="44" alt="pmezhuev" title="@pmezhuev"></a>
<a href="https://github.com/kh0mka"><img src="https://github.com/kh0mka.png?size=64" width="44" alt="kh0mka" title="@kh0mka"></a>
</p>
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
GitHub does not generate a contributor graph for forks, so [`CONTRIBUTORS.md`](CONTRIBUTORS.md) — not the Insights page — is this project's attribution record. It names everyone alongside the change or report they contributed.
Every human issue or pull-request author is part of the SILO community, including open and unmerged work. Merged fixes, adopted proposals, and actionable reports receive priority, with first participation guiding the remaining order. Gold rings highlight reviewed significant contributions.
<a href="CONTRIBUTORS.md">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-light.svg" alt="SILO community contributors">
</picture>
</a>
[View contribution notes and actual PR status](CONTRIBUTORS.md).
## Star History
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-light.svg" alt="SILO GitHub star history">
</picture>
## Background
+41 -47
View File
@@ -35,6 +35,13 @@
> [!NOTE]
> 2026-08-06,本仓库由 `pgsty/minio` 更名为 `pgsty/silo`,默认分支由 `master` 更名为 `main`。以原 MinIO 形态维持的归档构件仍位于归档的 [`minio`](https://github.com/pgsty/silo/tree/minio) 分支,以及截止 [`RELEASE.2026-08-04T00-00-00Z`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z) 的历次发布中。
## 当前发行版与主分支
最新已发布的 Server 仍为 [20260903](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z)。
截至 2026-09-16,主分支已合入更新的安全、存储、Console 与共享包改动,但尚未发布新 Server。
准确的已发布/源码边界见 [CHANGELOG.md](CHANGELOG.md) 与[组件版本矩阵](https://silo.pgsty.com/zh/compatibility/versions/),
其中包括 SN-2026-011 修复状态与密码权限迁移要求。
## 概述
上游停止社区发行后,Silo 为开源 MinIO 服务端维护一条持续可用的版本线:构建、软件包、多架构镜像、安全修复与完整 Web 控制台。Pigsty 在生产环境中用它承载 PostgreSQL 备份存储。
@@ -85,63 +92,50 @@ docker exec silo mcli mb local/demo && docker exec silo mcli ls local
## 兼容性
S3 API、`MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*` 路由、`github.com/minio/*` 导入路径与磁盘格式(含 `.minio.sys`)原样保留,并由 CI 兼容性门禁冻结。只有 Silo 自有交付面改名:`silo` 可执行文件、软件包、服务、Helm Chart 与容器镜像 —— 原生交付物不会安装 `minio` 二进制别名。
Silo 保留 S3 与存储格式兼容性,包括既有 `MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*` 路由与 `.minio.sys` 数据。CI 守卫检查选定的兼容性标识,有意的安全与行为变化在发布说明中记录。Silo 自有交付面使用 `silo` 可执行文件、软件包、服务、Helm Chart 与容器镜像;原生交付物不会安装 `minio` 服务端二进制别名。
正式支持和发布验收的组合为 `pgsty/silo` + `pgsty/silo-console` + `pgsty/mc` + `pgsty/silo-pkg`,对未修改的上游 MinIO/MC 尽最大努力保持兼容。Server、Console 与客户端按需保留历史模块路径,维护源码直接导入 `github.com/pgsty/silo-pkg/v3`;SDK `github.com/minio/minio-go/v7` 是明确保留的上游依赖。具体版本与 replace 以当前 [go.mod](go.mod) 为准。
与上游的全部分歧,以逐项核验代码的[兼容性审计](https://silo.pgsty.com/zh/compatibility/server/)形式维护。每个版本仍应视为下游升级:锁定版本,阅读[版本说明](https://silo.pgsty.com/zh/tags/silo/),并保留回滚路径。
### TLS 与 Go 升级
Server 恢复 Go 默认密钥交换策略的修复已在 main,尚未包含在 Server 20260903。
`GODEBUG=tlsmlkem=0`、`tlssecpmlkem=0` 的适用范围、macOS 根证书来源变化,
以及 OIDC discovery 的诊断方法见 [Go 1.27 TLS 与 OIDC 指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
## 文档归属
用户文档统一维护在 [silo.pgsty.com](https://silo.pgsty.com/zh/docs/),源码位于
[pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com)。本仓库保留的 `docs/`
主要是继承的参考材料、示例和工具测试夹具。调查日志、AI 工作记录与临时报告放在仓库外;
可复用的结论应整理进伴生文档站。仓库职责与维护规则见 [AGENTS.md](AGENTS.md)。
## 安全与贡献
请按照 [`SECURITY.md`](SECURITY.md) 私密报告漏洞;每项修复都会发布公开[安全公告](https://silo.pgsty.com/zh/blog/security/)。本项目不要求签署 CLA:贡献按 AGPL-3.0-or-later(inbound=outbound)接收,只需 DCO 签署(`git commit -s`),详见 [`CONTRIBUTING.md`](CONTRIBUTING.md)。
## 贡献者
<table>
<tr>
<td align="center" width="150">
<a href="https://github.com/ZouhairCharef"><img src="https://github.com/ZouhairCharef.png?size=100" width="72" alt="ZouhairCharef"><br><sub><b>@ZouhairCharef</b></sub></a><br><sub>CVE-2026-34986</sub>
</td>
<td align="center" width="150">
<a href="https://github.com/mfredenhagen"><img src="https://github.com/mfredenhagen.png?size=100" width="72" alt="mfredenhagen"><br><sub><b>@mfredenhagen</b></sub></a><br><sub>CVE-2026-39883</sub>
</td>
<td align="center" width="150">
<a href="https://github.com/pinginfo"><img src="https://github.com/pinginfo.png?size=100" width="72" alt="pinginfo"><br><sub><b>@pinginfo</b></sub></a><br><sub>桶通知流式输出</sub>
</td>
<td align="center" width="150">
<a href="https://github.com/waterkip"><img src="https://github.com/waterkip.png?size=100" width="72" alt="waterkip"><br><sub><b>@waterkip</b></sub></a><br><sub>文档链接修正</sub>
</td>
</tr>
</table>
<p>
<a href="https://github.com/magicxor"><img src="https://github.com/magicxor.png?size=64" width="44" alt="magicxor" title="@magicxor"></a>
<a href="https://github.com/ycjlin"><img src="https://github.com/ycjlin.png?size=64" width="44" alt="ycjlin" title="@ycjlin"></a>
<a href="https://github.com/h5vx"><img src="https://github.com/h5vx.png?size=64" width="44" alt="h5vx" title="@h5vx"></a>
<a href="https://github.com/Dansyuqri"><img src="https://github.com/Dansyuqri.png?size=64" width="44" alt="Dansyuqri" title="@Dansyuqri"></a>
<a href="https://github.com/davinkevin"><img src="https://github.com/davinkevin.png?size=64" width="44" alt="davinkevin" title="@davinkevin"></a>
<a href="https://github.com/lem21h"><img src="https://github.com/lem21h.png?size=64" width="44" alt="lem21h" title="@lem21h"></a>
<a href="https://github.com/sulin37392"><img src="https://github.com/sulin37392.png?size=64" width="44" alt="sulin37392" title="@sulin37392"></a>
<a href="https://github.com/mosesdd"><img src="https://github.com/mosesdd.png?size=64" width="44" alt="mosesdd" title="@mosesdd"></a>
<a href="https://github.com/Xavier-777"><img src="https://github.com/Xavier-777.png?size=64" width="44" alt="Xavier-777" title="@Xavier-777"></a>
<a href="https://github.com/jiadzh"><img src="https://github.com/jiadzh.png?size=64" width="44" alt="jiadzh" title="@jiadzh"></a>
<a href="https://github.com/TLINDEN"><img src="https://github.com/TLINDEN.png?size=64" width="44" alt="TLINDEN" title="@TLINDEN"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://github.com/AntonOfTheWoods.png?size=64" width="44" alt="AntonOfTheWoods" title="@AntonOfTheWoods"></a>
<a href="https://github.com/zylpsrs"><img src="https://github.com/zylpsrs.png?size=64" width="44" alt="zylpsrs" title="@zylpsrs"></a>
<a href="https://github.com/nsanitate"><img src="https://github.com/nsanitate.png?size=64" width="44" alt="nsanitate" title="@nsanitate"></a>
<a href="https://github.com/makinikm"><img src="https://github.com/makinikm.png?size=64" width="44" alt="makinikm" title="@makinikm"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://github.com/spaceg00se-r.png?size=64" width="44" alt="spaceg00se-r" title="@spaceg00se-r"></a>
<a href="https://github.com/heroes1412"><img src="https://github.com/heroes1412.png?size=64" width="44" alt="heroes1412" title="@heroes1412"></a>
<a href="https://github.com/vampywiz17"><img src="https://github.com/vampywiz17.png?size=64" width="44" alt="vampywiz17" title="@vampywiz17"></a>
<a href="https://github.com/chalukyaj"><img src="https://github.com/chalukyaj.png?size=64" width="44" alt="chalukyaj" title="@chalukyaj"></a>
<a href="https://github.com/cbornet"><img src="https://github.com/cbornet.png?size=64" width="44" alt="cbornet" title="@cbornet"></a>
<a href="https://github.com/jvasile"><img src="https://github.com/jvasile.png?size=64" width="44" alt="jvasile" title="@jvasile"></a>
<a href="https://github.com/Kesavaambati"><img src="https://github.com/Kesavaambati.png?size=64" width="44" alt="Kesavaambati" title="@Kesavaambati"></a>
<a href="https://github.com/redfoxfox"><img src="https://github.com/redfoxfox.png?size=64" width="44" alt="redfoxfox" title="@redfoxfox"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://github.com/kuldeep-link11.png?size=64" width="44" alt="kuldeep-link11" title="@kuldeep-link11"></a>
<a href="https://github.com/meesudzu"><img src="https://github.com/meesudzu.png?size=64" width="44" alt="meesudzu" title="@meesudzu"></a>
<a href="https://github.com/pmezhuev"><img src="https://github.com/pmezhuev.png?size=64" width="44" alt="pmezhuev" title="@pmezhuev"></a>
<a href="https://github.com/kh0mka"><img src="https://github.com/kh0mka.png?size=64" width="44" alt="kh0mka" title="@kh0mka"></a>
</p>
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
GitHub 不为 fork 仓库生成贡献者图表,因此 [`CONTRIBUTORS.md`](CONTRIBUTORS.md)(而非 Insights 页面)才是本项目的署名记录,其中逐一记录了每个人对应的改动或报告。
每位 issue 或 PR 的作者都是 SILO 社区的一员,包括尚未合并的工作。已合并的修复、被采纳的方案和有效报告优先展示,其余参考首次参与时间;金色圆环突出经过审核的显著贡献。
<a href="CONTRIBUTORS.md">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-light.svg" alt="SILO 社区贡献者">
</picture>
</a>
[查看贡献记录与实际 PR 状态](CONTRIBUTORS.md)。
## Star History
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-light.svg" alt="SILO GitHub 星标历史">
</picture>
## 背景
+3 -3
View File
@@ -7,7 +7,7 @@ Silo-specific fixes or release notes.
## Supported Versions
Security fixes are tracked on the active development branch and summarized in
[docs/security/advisories.md](docs/security/advisories.md). Only the current
[the security advisory ledger](https://silo.pgsty.com/about/security-advisories/). Only the current
Silo release line is supported unless an advisory says otherwise.
## Inherited Fix Evidence
@@ -26,7 +26,7 @@ separately even when the fork preserves the original commit object and SHA.
The inherited [service-account](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/admin-handlers-users_test.go#L211-L212)
and [STS](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/sts-handlers_test.go#L45-L46)
regression groups remain part of `go test ./cmd`; see the
[canonical ledger](docs/security/advisories.md#inherited-upstream-advisory-baseline)
[canonical ledger](https://silo.pgsty.com/about/security-advisories/#inherited)
for the operator-facing record.
## Reporting a Vulnerability
@@ -42,4 +42,4 @@ For vulnerabilities in this fork:
## Disclosure Process
Fork-specific fixes and user-visible upgrade notes are published in [docs/security/advisories.md](docs/security/advisories.md). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
Fork-specific fixes and user-visible upgrade notes are published in [the security advisory ledger](https://silo.pgsty.com/about/security-advisories/). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
+1 -1
View File
@@ -34,4 +34,4 @@ Based on the report, the Silo maintainers investigate:
If the vulnerability exists in this fork itself, the maintainers will, when
feasible, fix the issue or implement reasonable countermeasures such that the
vulnerability can no longer be exploited. Fork-specific upgrade notes and
security advisories are published in `docs/security/advisories.md`.
security advisories are published in [the security advisory ledger](https://silo.pgsty.com/about/security-advisories/).
+1 -1
View File
@@ -40,7 +40,7 @@ if [ -n "${MCLI_BIN:-}" ]; then
exit 0
fi
release=${MCLI_RELEASE:-RELEASE.2026-09-03T07-13-05Z}
release=${MCLI_RELEASE:-RELEASE.2026-09-13T00-00-00Z}
version_hyphen=${release#RELEASE.}
package_version=$(printf '%s\n' "${version_hyphen}" | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/\1\2\3\4\5\6.0.0/')
if [ "${package_version}" = "${version_hyphen}" ]; then
@@ -132,8 +132,8 @@
"MINIO_API_DELETE_CLEANUP_INTERVAL",
"MINIO_API_DISABLE_ODIRECT",
"MINIO_API_GZIP_OBJECTS",
"MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH",
"MINIO_API_LIST_QUORUM",
"MINIO_API_MULTIPART_LISTING",
"MINIO_API_OBJECT_MAX_VERSIONS",
"MINIO_API_ODIRECT",
"MINIO_API_REMOTE_TRANSPORT_DEADLINE",
@@ -502,6 +502,7 @@
"MINIO_SITE_COMMENT",
"MINIO_SITE_NAME",
"MINIO_SITE_REGION",
"MINIO_SITE_REPLICATION_METADATA_TOMBSTONES",
"MINIO_STORAGE_CLASS_COMMENT",
"MINIO_STORAGE_CLASS_INLINE_BLOCK",
"MINIO_STORAGE_CLASS_OPTIMIZE",
@@ -594,6 +595,7 @@
"x-minio-internal-encrypted-multipart",
"x-minio-internal-encryptedmultipart",
"x-minio-internal-erasure-upgraded",
"x-minio-internal-ilm-atier",
"x-minio-internal-inline-data",
"x-minio-internal-objectlock-legalhold-timestamp",
"x-minio-internal-replica-status",
@@ -616,6 +618,7 @@
"x-minio-internal-transitioned-object",
"x-minio-internal-xyz",
"x-minio-key",
"x-minio-last-modified",
"x-minio-lifecycleconfig-updatedat",
"x-minio-meta",
"x-minio-meta-appid",
@@ -624,13 +627,13 @@
"x-minio-origin-endpoint",
"x-minio-prefixes-total",
"x-minio-read-quorum",
"x-minio-replication",
"x-minio-replication-actual-object-size",
"x-minio-replication-delete-status",
"x-minio-replication-deletemarker-status",
"x-minio-replication-encrypted-multipart",
"x-minio-replication-ready",
"x-minio-replication-reset-status",
"x-minio-replication-server-side-encryption",
"x-minio-replication-server-side-encryption-iv",
"x-minio-replication-server-side-encryption-seal-algorithm",
"x-minio-replication-server-side-encryption-sealed-key",
@@ -762,6 +765,7 @@
"/metrics/v3",
"/minio/grid/",
"/minio/grid/lock/",
"/multipart-preflight",
"/netperf",
"/notification",
"/oauth2/callback",
@@ -825,6 +829,7 @@
"/site-replication/peer/bucket-ops",
"/site-replication/peer/edit",
"/site-replication/peer/iam-item",
"/site-replication/peer/iam-revisions",
"/site-replication/peer/idp-settings",
"/site-replication/peer/join",
"/site-replication/peer/remove",
@@ -873,6 +878,7 @@
"/v2/metrics/cluster",
"/v2/metrics/node",
"/v2/metrics/resource",
"/v3/site-replication/peer/iam-revisions",
"/var/vcap/bosh",
"/verifybinary",
"/version",
@@ -900,6 +906,7 @@
".minio.sys/config/config.json",
".minio.sys/config/hello.txt",
".minio.sys/config/iam/${username}/identity.json",
".minio.sys/config/ilm/access",
".minio.sys/format.json",
".minio.sys/multipart",
".minio.sys/multipart/bucket/object/uploads.json",
@@ -927,6 +934,7 @@
"arn:minio:kms:::this-is-disregarded",
"arn:minio:kms:::xyz-test-key",
"arn:minio:replication:",
"arn:minio:replication::",
"arn:minio:replication::8320b6d18f9032b4700f1f03b50d8d1853de8f22cab86931ee794e12f190852c:destinationbucket",
"arn:minio:replication:::",
"arn:minio:replication:::dest-bucket",
@@ -1058,12 +1066,10 @@
"cmd/metrics.go=\"Version of current MinIO server instance\"",
"cmd/metrics.go=\"minio\"",
"cmd/object-api-utils.go=\".minio.sys\"",
"cmd/object-handlers-common.go=\"X-Minio-Last-Modified\"",
"cmd/object-handlers.go=\"minio-federated\"",
"cmd/object-handlers.go=\"minio.metadata.\"",
"cmd/object-handlers.go=\"minio.versionId\"",
"cmd/object-multipart-handlers.go=\"X-Minio-Replication-Server-Side-Encryption-Iv\"",
"cmd/object-multipart-handlers.go=\"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm\"",
"cmd/object-multipart-handlers.go=\"X-Minio-Replication-Server-Side-Encryption-Sealed-Key\"",
"cmd/replication-trust.go=\"X-Minio-Replication-Encrypted-Multipart\"",
"cmd/replication-trust.go=\"X-Minio-Replication-Server-Side-Encryption-Iv\"",
"cmd/replication-trust.go=\"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm\"",
+1
View File
@@ -115,6 +115,7 @@ func collect(repo string) (manifest, error) {
fset := token.NewFileSet()
for _, rel := range files {
// Migration notes and guard fixtures contain archived identifiers.
if rel == "SILO_REBRANDING_MIGRATION.md" ||
strings.HasPrefix(rel, "buildscripts/rebrand-guard/") ||
strings.HasPrefix(rel, "buildscripts/helm-migration-guard/") {
+1
View File
@@ -0,0 +1 @@
__pycache__/
+163
View File
@@ -0,0 +1,163 @@
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Pure, self-contained SVG rendering for SILO's README cards."""
from datetime import date, timedelta
from html import escape
import math
import xml.etree.ElementTree as ET
themes = {
'light': dict(bg='#ffffff', wash='#f2f7fc', edge='#d9e3ee', ink='#16222e',
muted='#62758a', blue='#1d588c', copper='#b4762e', grid='#e5edf5',
line='#2b6ca3', ring='#dce5ef', field='#f7f9fc', label='#3d4e61'),
'dark': dict(bg='#101923', wash='#152738', edge='#2b3c50', ink='#e8eef6',
muted='#93a3b8', blue='#7fb8e8', copper='#e0a35c', grid='#263749',
line='#5da2dd', ring='#3a4e63', field='#0b1119', label='#b6c2d2'),
}
def read_emblem(path):
emblem = ET.parse(path).getroot()
body = ''.join(ET.tostring(child, encoding='unicode') for child in emblem
if child.tag.rsplit('}', 1)[-1] in ('defs', 'g'))
return '\n'.join(line.rstrip() for line in body.splitlines()).strip()
def txt(x, y, value, size=14, color=None, weight=400, anchor='start', mono=False, spacing=None):
family = 'Menlo,Consolas,monospace' if mono else 'Arial,Helvetica,sans-serif'
extra = f' letter-spacing="{spacing}"' if spacing is not None else ''
return (f'<text x="{x}" y="{y}" font-family="{family}" font-size="{size}" '
f'font-weight="{weight}" fill="{color}" text-anchor="{anchor}"{extra}>'
f'{escape(str(value))}</text>')
def start(height, theme, title, description, emblem_body):
t = themes[theme]
return [f'<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="{height}" '
f'viewBox="0 0 1000 {height}" role="img" aria-labelledby="title desc">',
f'<title id="title">{escape(title)}</title><desc id="desc">{escape(description)}</desc>',
'<defs><linearGradient id="surface" x1="0" y1="1" x2="1" y2="0">'
f'<stop offset="0" stop-color="{t["bg"]}"/>'
f'<stop offset="1" stop-color="{t["wash"]}"/></linearGradient>'
'<linearGradient id="accent" x1="0" y1="0" x2="1" y2="0">'
f'<stop offset="0" stop-color="{t["blue"]}"/>'
f'<stop offset="1" stop-color="{t["copper"]}"/></linearGradient>'
'<linearGradient id="area" x1="0" y1="0" x2="0" y2="1">'
f'<stop offset="0" stop-color="{t["line"]}" stop-opacity=".22"/>'
f'<stop offset="1" stop-color="{t["line"]}" stop-opacity=".015"/>'
'</linearGradient></defs>',
f'<rect x=".75" y=".75" width="998.5" height="{height-1.5}" rx="22" '
f'fill="url(#surface)" stroke="{t["edge"]}" stroke-width="1.5"/>',
f'<svg x="40" y="25" width="25" height="25" viewBox="230 213 570 570">{emblem_body}</svg>']
def heading(parts, t, eyebrow, title, subtitle, value, value_label):
parts.extend([
txt(76, 43, eyebrow, 11, t['muted'], 600, mono=True, spacing=1.7),
txt(40, 94, title, 32, t['ink'], 700),
txt(41, 123, subtitle, 14, t['muted']),
txt(958, 89, f'{value:,}', 45, t['ink'], 700, anchor='end'),
txt(957, 114, value_label, 10, t['muted'], 600, anchor='end', mono=True, spacing=1.5),
f'<path d="M40 146 H960" stroke="{t["edge"]}"/>',
])
def contributors(theme, people, snapshot, emblem_body):
height = 206 + 76 * math.ceil(len(people) / 10)
t = themes[theme]
parts = start(height, theme, f'SILO community — {len(people)} contributors',
f'The existing SILO community roll, including code, proposals and reports across related projects. '
f'Gold rings retain the existing significant-contribution designation. Snapshot {snapshot}.', emblem_body)
heading(parts, t, 'SILO / COMMUNITY', 'Contributors',
'Code, proposals & reports across SILO and related projects', len(people), 'COMMUNITY CONTRIBUTORS')
for row in range(math.ceil(len(people) / 10)):
group = people[row * 10:(row + 1) * 10]
row_width = len(group) * 91
for col, person in enumerate(group):
x = (1000 - row_width) / 2 + col * 91 + 45.5
y = 199 + row * 76
identifier = f'avatar-{row}-{col}'
featured = bool(person.get('featured'))
parts.append(f'<g><title>@{escape(person["handle"])} — {escape(person["what"])}</title>')
parts.append(f'<defs><clipPath id="{identifier}"><circle cx="{x}" cy="{y}" r="29"/></clipPath></defs>')
if featured:
parts.append(f'<circle cx="{x}" cy="{y}" r="34" fill="{t["copper"]}" opacity=".09"/>')
if person.get('avatarDataUrl'):
parts.append(f'<image x="{x-29}" y="{y-29}" width="58" height="58" '
f'clip-path="url(#{identifier})" href="{escape(person["avatarDataUrl"])}"/>')
else:
parts.append(f'<circle cx="{x}" cy="{y}" r="29" fill="{t["ring"]}"/>')
parts.append(txt(x, y + 9, person['handle'][0].upper(), 26, t['ink'], 700, 'middle'))
parts.append(f'<circle cx="{x}" cy="{y}" r="30.5" fill="none" '
f'stroke="{t["copper"] if featured else t["ring"]}" stroke-width="{2 if featured else 1.25}"/></g>')
parts.extend([
f'<path d="M40 {height-42} H960" stroke="{t["edge"]}"/>',
f'<circle cx="47" cy="{height-21}" r="4" fill="none" stroke="{t["copper"]}" stroke-width="1.5"/>',
txt(61, height-17, 'Gold rings mark significant contributions', 12, t['muted']),
txt(959, height-17, f'AS OF {snapshot}', 10, t['muted'], 500, 'end', mono=True, spacing=.6),
'</svg>',
])
return ''.join(parts)
def stars(theme, history, snapshot, emblem_body):
points = history['points']
star_count = points[-1]['stars']
t = themes[theme]
provenance = ('Initial history reconstructed · Daily totals since ' + history['bootstrap']['through'] + ' · UTC'
if history['bootstrap']['reconstructed'] else 'Observed daily star totals · UTC')
parts = start(558, theme, f'SILO star history — {star_count:,} stars',
f'GitHub repository pgsty/silo. {star_count:,} stars as of {snapshot}. ' +
provenance, emblem_body)
heading(parts, t, 'SILO / GITHUB', 'Star History', 'pgsty/silo', star_count, 'GITHUB STARS')
left, right, top, bottom = 76, 958, 177, 440
begin = date.fromisoformat(points[0]['date'])
end = date.fromisoformat(points[-1]['date'])
days = max(1, (end - begin).days)
maximum = max(500, math.ceil(max(p['stars'] for p in points) / 500) * 500)
tick_step = 10 ** max(0, int(math.log10(maximum)))
xy = lambda day, n: (left + (right-left)*(date.fromisoformat(day)-begin).days / days,
bottom-(bottom-top)*n/maximum)
for value in range(0, maximum+1, tick_step):
y = xy(points[0]['date'], value)[1]
parts.append(f'<path d="M{left} {y:.2f} H{right}" stroke="{t["grid"]}" stroke-dasharray="4 6"/>')
parts.append(txt(left-16, round(y+4, 2), f'{value / 1000:g}k' if value >= 1000 else str(value), 12, t['muted'], anchor='end'))
dates = sorted({begin + timedelta(days=round((end-begin).days*i/5)) for i in range(6)})
ticks = [(d.isoformat(), d.strftime('%b %Y') if days > 90 else d.strftime('%b %d')) for d in dates]
for day, label in ticks:
x = xy(day, 0)[0]
parts.append(f'<path d="M{x:.2f} {top} V{bottom}" stroke="{t["grid"]}" stroke-opacity=".65"/>')
anchor = 'start' if day == points[0]['date'] else 'end' if day == snapshot else 'middle'
parts.append(txt(round(x, 2), 466, label, 12, t['muted'], anchor=anchor))
coords = [xy(p['date'], p['stars']) for p in points]
line = 'M' + ' L'.join(f'{x:.2f} {y:.2f}' for x, y in coords)
area = line + f' L{coords[-1][0]:.2f} {bottom} L{left} {bottom} Z'
parts.extend([
f'<path d="{area}" fill="url(#area)"/>',
f'<path d="{line}" fill="none" stroke="url(#accent)" stroke-width="3" '
'stroke-linecap="round" stroke-linejoin="round"/>',
f'<path d="M{left} {bottom} H{right}" stroke="{t["edge"]}"/>',
])
x, y = coords[-1]
parts.extend([
f'<circle cx="{x}" cy="{y}" r="10" fill="{t["copper"]}" opacity=".12"/>',
f'<circle cx="{x}" cy="{y}" r="5" fill="{t["copper"]}" stroke="{t["bg"]}" stroke-width="2"/>',
f'<path d="M40 491 H960" stroke="{t["edge"]}"/>',
txt(40, 516, f'{begin:%b %Y} — {end:%b %Y}'.upper(), 10, t['muted'], 500, mono=True, spacing=.7),
txt(959, 516, f'SNAPSHOT {snapshot}', 10, t['muted'], 500, 'end', mono=True, spacing=.6),
txt(40, 539, provenance, 11, t['muted']),
'</svg>',
])
return ''.join(parts)
@@ -0,0 +1 @@
PyYAML==6.0.3
+178
View File
@@ -0,0 +1,178 @@
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Regression checks for historical accuracy, contributor scope and SVG safety."""
import base64
import json
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
from urllib.error import URLError
import xml.etree.ElementTree as ET
import render
import update
NS = {'s': 'http://www.w3.org/2000/svg'}
PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+a7mgAAAAASUVORK5CYII=')
def person(handle='Alice', group='reports', featured=False):
return {'handle': handle, 'group': group, 'featured': featured,
'what': 'A reviewed contribution', 'firstContribution': '2026-09-01'}
def history():
return {'repository': 'pgsty/silo',
'bootstrap': {'through': '2026-09-15', 'reconstructed': True},
'points': [{'date': '2026-09-14', 'stars': 100}, {'date': '2026-09-15', 'stars': 105}]}
class HistoryTests(unittest.TestCase):
def test_new_day_preserves_old_counts_and_unstars(self):
before = history()
after = update.update_history(before, '2026-09-16', 103)
self.assertEqual(after['points'][:-1], before['points'])
self.assertEqual(after['points'][-1], {'date': '2026-09-16', 'stars': 103})
self.assertEqual(before, history())
def test_same_day_rerun_replaces_instead_of_appending(self):
first = update.update_history(history(), '2026-09-15', 107)
self.assertEqual(len(first['points']), 2)
self.assertEqual(first, update.update_history(first, '2026-09-15', 107))
def test_missing_days_are_not_invented(self):
result = update.update_history(history(), '2026-09-18', 106)
self.assertEqual([p['date'] for p in result['points']], ['2026-09-14', '2026-09-15', '2026-09-18'])
def test_rejects_wrong_repository_and_corrupt_history(self):
cases = []
wrong = history(); wrong['repository'] = 'someone/else'; cases.append(wrong)
duplicate = history(); duplicate['points'].append(duplicate['points'][-1]); cases.append(duplicate)
unordered = history(); unordered['points'].reverse(); cases.append(unordered)
negative = history(); negative['points'][0]['stars'] = -1; cases.append(negative)
future = history(); future['points'][-1]['date'] = '2026-09-20'; cases.append(future)
for case in cases:
with self.subTest(case=case), self.assertRaises(ValueError):
update.update_history(case, '2026-09-16', 100)
def test_first_run_has_no_fabricated_history(self):
result = update.update_history(None, '2026-09-16', 10)
self.assertFalse(result['bootstrap']['reconstructed'])
self.assertEqual(result['points'], [{'date': '2026-09-16', 'stars': 10}])
class ContributorTests(unittest.TestCase):
def test_retries_truncated_json_before_using_it(self):
with patch('update.request', side_effect=[b'{"partial":', b'{"ok":true}']), patch('update.time.sleep'):
self.assertEqual(update.GitHub('').get('repos/pgsty/silo'), {'ok': True})
def test_paginates_past_one_full_page(self):
class API(update.GitHub):
def __init__(self): self.calls = []
def get(self, path):
self.calls.append(path)
return list(range(100)) if 'page=1&' in path else [100]
api = API()
self.assertEqual(len(list(api.issues('pgsty/silo'))), 101)
self.assertIn('state=all', api.calls[0])
self.assertIn('page=2&', api.calls[1])
def test_bots_deduplication_unmerged_work_and_reviewed_credit(self):
def issue(login, kind='issue', user_type='User'):
item = {'user': {'login': login, 'type': user_type, 'avatar_url': ''}, 'created_at': '2026-09-02T00:00:00Z'}
if kind != 'issue': item['pull_request'] = {'merged_at': None if kind == 'open' else '2026-09-03T00:00:00Z'}
return item
class API:
def issues(self, _repo):
return [issue('alice'), issue('Bob', 'open'), issue('Bob', 'merged'),
issue('Carol', 'open'), issue('Copilot'), issue('robot', user_type='Bot')]
curated = {'repositories': ['pgsty/silo', 'pgsty/mc'], 'bots': ['Copilot'],
'people': [person('Alice', featured=True), person('Reporter'), person('Copilot')]}
result = update.collect_people(API(), curated)
self.assertEqual({p['handle'] for p in result}, {'Alice', 'Bob', 'Carol', 'Reporter'})
self.assertEqual(result[0]['handle'], 'Bob')
self.assertEqual(result[1]['handle'], 'Carol')
self.assertTrue(next(p for p in result if p['handle'] == 'Alice')['featured'])
self.assertEqual(next(p for p in result if p['handle'] == 'Bob')['group'], 'code')
self.assertFalse(next(p for p in result if p['handle'] == 'Carol')['featured'])
def test_newer_reviewed_preview_survives_until_site_catches_up(self):
remote = {'updated': '2026-09-16T03:00:00+00:00'}
cached = {'updated': '2026-09-16T04:00:00+00:00'}
self.assertIs(update.select_curated(remote, cached), cached)
newer = {'updated': '2026-09-17T03:00:00+00:00'}
self.assertIs(update.select_curated(newer, cached), newer)
def test_avatar_failure_reuses_raster_cache(self):
previous = {**person(), 'avatarDataUrl': update.raster_data_url(PNG)}
with patch('update.request', side_effect=URLError('unavailable')):
result = update.add_avatars(None, [{**person(), 'avatarUrl': 'https://avatars.githubusercontent.com/u/1'}], [previous])
self.assertEqual(result[0]['avatarDataUrl'], previous['avatarDataUrl'])
with self.assertRaises(ValueError): update.raster_data_url(b'<svg onload="bad()"/>')
with self.assertRaises(ValueError): update.cached_avatar({'avatarDataUrl': 'data:image/svg+xml;base64,PHN2Zy8+'})
def test_fetch_failure_leaves_published_assets_untouched(self):
class API:
def get(self, path):
if path == 'repos/pgsty/silo': return {'full_name': 'pgsty/silo', 'stargazers_count': 106}
raise URLError('roster unavailable')
with tempfile.TemporaryDirectory() as directory:
out = Path(directory)
original = json.dumps(history())
(out / 'history.json').write_text(original)
(out / 'contributors-light.svg').write_text('previous image')
with self.assertRaises(URLError): update.refresh(out, API(), Path('.'))
self.assertEqual((out / 'history.json').read_text(), original)
self.assertEqual((out / 'contributors-light.svg').read_text(), 'previous image')
class RenderTests(unittest.TestCase):
def test_real_emblem_generates_clean_xml(self):
emblem = render.read_emblem(Path(__file__).resolve().parents[2] / '.github/silo.svg')
svg = render.contributors('light', [person()], '2026-09-16', emblem)
ET.fromstring(svg)
self.assertTrue(all(line == line.rstrip() for line in svg.splitlines()))
def test_all_avatars_fit_when_the_roster_grows(self):
people = [{**person(f'person-{i}'), 'avatarDataUrl': update.raster_data_url(PNG)} for i in range(151)]
for theme in ('light', 'dark'):
root = ET.fromstring(render.contributors(theme, people, '2026-09-16', ''))
images = root.findall('.//s:image', NS)
self.assertEqual(len(images), 151)
footer = float(root.attrib['height']) - 42
self.assertTrue(all(float(i.attrib['y']) + float(i.attrib['height']) < footer for i in images))
self.assertTrue(all(i.attrib['href'].startswith('data:image/png;base64,') for i in images))
def test_untrusted_text_is_escaped(self):
data = [{**person(), 'what': '<script>alert("x")</script> & contributions'}]
svg = render.contributors('light', data, '2026-09-16', '')
root = ET.fromstring(svg)
self.assertEqual(root.findall('.//s:script', NS), [])
self.assertIn('&lt;script&gt;', svg)
def test_single_point_and_decreasing_star_history_render(self):
for data in (update.update_history(None, '2026-09-16', 0), update.update_history(history(), '2026-09-16', 90)):
for theme in ('light', 'dark'):
svg = render.stars(theme, data, '2026-09-16', '')
ET.fromstring(svg)
self.assertNotIn('nan', svg.lower())
self.assertNotIn('inf', svg.lower())
if __name__ == '__main__':
unittest.main()
+295
View File
@@ -0,0 +1,295 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Refresh the generated-asset checkout; publishing is handled by the workflow."""
import argparse
import base64
from concurrent.futures import ThreadPoolExecutor
from datetime import date, datetime, timezone
import json
from http.client import IncompleteRead
import os
from pathlib import Path
import re
import sys
import time
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse
from urllib.request import Request, urlopen
import xml.etree.ElementTree as ET
import yaml
import render
REPOSITORY = 'pgsty/silo'
SOURCE = 'repos/pgsty/silo.pgsty.com/contents/data/home/contributors.yaml?ref=main'
GROUPS = ('code', 'proposed', 'reports')
HANDLE = re.compile(r'[A-Za-z0-9][A-Za-z0-9-]{0,38}\Z')
def request(url, token='', limit=8 * 1024 * 1024):
headers = {'User-Agent': 'silo-repository-cards', 'Accept': 'application/vnd.github+json'}
if urlparse(url).netloc == 'api.github.com':
headers['X-GitHub-Api-Version'] = '2022-11-28'
if token:
headers['Authorization'] = f'Bearer {token}'
for attempt in range(3):
try:
with urlopen(Request(url, headers=headers), timeout=25) as response:
data = response.read(limit + 1)
if len(data) > limit:
raise ValueError('Response exceeds the size limit')
expected = response.headers.get('Content-Length')
if expected is not None and len(data) != int(expected):
raise URLError('Incomplete response body')
return data
except HTTPError as exc:
if exc.code < 500 or attempt == 2:
raise
except (URLError, TimeoutError, IncompleteRead):
if attempt == 2:
raise
time.sleep(attempt + 1)
class GitHub:
def __init__(self, token):
self.token = token
def get(self, path):
for attempt in range(3):
try:
return json.loads(request('https://api.github.com/' + path, self.token))
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
if attempt == 2:
raise ValueError(f'Incomplete or invalid GitHub JSON: {path}') from exc
time.sleep(attempt + 1)
def issues(self, repository):
page = 1
while True:
batch = self.get(f'repos/{repository}/issues?state=all&per_page=100&page={page}&sort=created&direction=asc')
if not isinstance(batch, list):
raise ValueError(f'Invalid issues response for {repository}')
yield from batch
if len(batch) < 100:
return
page += 1
def curated_snapshot(data, revision):
updated = str(data['updated'])
datetime.fromisoformat(updated)
repositories = [item['repo'] for item in data['repositories']]
if not repositories or any(not re.fullmatch(r'pgsty/[A-Za-z0-9_.-]+', repo) for repo in repositories):
raise ValueError('Invalid contributor repository scope')
people = []
for group in GROUPS:
for entry in data[group]:
if not HANDLE.fullmatch(entry['handle']):
raise ValueError('Invalid GitHub contributor handle')
people.append({
'handle': entry['handle'], 'group': group,
'featured': bool(entry.get('featured')), 'what': entry['what'],
'firstContribution': str(entry.get('firstContribution', '9999-12-31')),
})
if not people or len({p['handle'].lower() for p in people}) != len(people):
raise ValueError('Empty or duplicate contributor roster')
return {'updated': updated, 'revision': revision, 'repositories': repositories,
'bots': data.get('bots', ['Copilot', 'dependabot[bot]']), 'people': people}
def select_curated(remote, cached):
# The initial, approved preview can contain reviewed credit not published by
# the companion site yet. Keep that newer snapshot until the site catches up.
if cached and datetime.fromisoformat(cached['updated']) > datetime.fromisoformat(remote['updated']):
return cached
return remote
def collect_people(api, curated):
bots = {name.lower() for name in curated['bots']}
people = {p['handle'].lower(): dict(p) for p in curated['people']
if p['handle'].lower() not in bots and not p['handle'].lower().endswith('[bot]')}
order = {p['handle'].lower(): index for index, p in enumerate(curated['people'])}
for repository in curated['repositories']:
print(f'Reading issue and PR authors: {repository}', flush=True)
for issue in api.issues(repository):
user = issue.get('user') or {}
handle = user.get('login', '')
key = handle.lower()
if user.get('type') != 'User' or key in bots or key.endswith('[bot]'):
continue
if not HANDLE.fullmatch(handle):
raise ValueError('Invalid issue author')
pr = issue.get('pull_request')
group = 'code' if pr and pr.get('merged_at') else 'proposed' if pr else 'reports'
first = issue['created_at'][:10]
date.fromisoformat(first)
person = people.setdefault(key, {
'handle': handle, 'group': group, 'featured': False,
'what': 'Contributed an issue or pull request to SILO and related projects',
'firstContribution': first,
})
person['avatarUrl'] = user.get('avatar_url', '')
person['firstContribution'] = min(person['firstContribution'], first)
if GROUPS.index(group) < GROUPS.index(person['group']):
person['group'] = group
if not people:
raise ValueError('No human contributors were collected')
return sorted(people.values(), key=lambda p: (
GROUPS.index(p['group']), not p['featured'],
order.get(p['handle'].lower(), len(order)), p['firstContribution'], p['handle'].lower()))
def raster_data_url(data):
if data.startswith(b'\x89PNG\r\n\x1a\n'):
mime = 'image/png'
elif data.startswith(b'\xff\xd8\xff'):
mime = 'image/jpeg'
elif data.startswith((b'GIF87a', b'GIF89a')):
mime = 'image/gif'
elif data[:4] == b'RIFF' and data[8:12] == b'WEBP':
mime = 'image/webp'
else:
raise ValueError('Avatar is not a raster image')
return f'data:{mime};base64,' + base64.b64encode(data).decode('ascii')
def cached_avatar(person):
value = person.get('avatarDataUrl', '')
if not value:
return ''
prefix, encoded = value.split(',', 1)
if prefix not in ('data:image/png;base64', 'data:image/jpeg;base64', 'data:image/gif;base64', 'data:image/webp;base64'):
raise ValueError('Invalid cached avatar format')
raw = base64.b64decode(encoded, validate=True)
if len(raw) > 512 * 1024 or raster_data_url(raw) != value:
raise ValueError('Invalid cached avatar')
return value
def add_avatars(api, people, previous):
cached = {p['handle'].lower(): cached_avatar(p) for p in previous}
def update(person):
person = dict(person)
try:
url = person.pop('avatarUrl', '') or api.get('users/' + person['handle'])['avatar_url']
parsed = urlparse(url)
if parsed.scheme != 'https' or parsed.netloc != 'avatars.githubusercontent.com':
raise ValueError('Unexpected avatar host')
data = request(url + ('&' if '?' in url else '?') + 's=96', limit=512 * 1024)
person['avatarDataUrl'] = raster_data_url(data)
except (HTTPError, URLError, TimeoutError, IncompleteRead, ValueError, KeyError) as exc:
person.pop('avatarUrl', None)
person['avatarDataUrl'] = cached.get(person['handle'].lower(), '')
print(f'Avatar fallback for @{person["handle"]}: {type(exc).__name__}', file=sys.stderr)
return person
with ThreadPoolExecutor(max_workers=6) as pool:
return list(pool.map(update, people))
def update_history(history, day, stars):
date.fromisoformat(day)
if type(stars) is not int or stars < 0:
raise ValueError('Invalid repository star count')
if history is None:
history = {'repository': REPOSITORY, 'bootstrap': {'through': day, 'reconstructed': False}, 'points': []}
if history['repository'] != REPOSITORY:
raise ValueError('Star history belongs to a different repository')
date.fromisoformat(history['bootstrap']['through'])
dates = []
for point in history['points']:
date.fromisoformat(point['date'])
if type(point['stars']) is not int or point['stars'] < 0:
raise ValueError('Invalid historical star count')
dates.append(point['date'])
if dates != sorted(set(dates)) or any(d > day for d in dates):
raise ValueError('History contains duplicate, unordered, or future dates')
# Replace today's observation, preserve previous days, and allow unstars.
points = [dict(p) for p in history['points'] if p['date'] != day]
points.append({'date': day, 'stars': stars})
return {**history, 'points': points}
def read_json(path, default=None):
return json.loads(path.read_text()) if path.exists() else default
def refresh(output, api, source_root):
day = datetime.now(timezone.utc).date().isoformat()
metadata = api.get('repos/' + REPOSITORY)
if metadata['full_name'].lower() != REPOSITORY:
raise ValueError('Unexpected repository metadata')
history = update_history(read_json(output / 'history.json'), day, metadata['stargazers_count'])
source = api.get(SOURCE)
reviewed = yaml.safe_load(base64.b64decode(source['content'], validate=False))
curated = select_curated(curated_snapshot(reviewed, source['sha']), read_json(output / 'curated.json'))
people = collect_people(api, curated)
previous = read_json(output / 'contributors.json', {}).get('people', [])
people = add_avatars(api, people, previous)
emblem = render.read_emblem(source_root / '.github/silo.svg')
payloads = {}
for theme in ('light', 'dark'):
payloads[f'contributors-{theme}.svg'] = render.contributors(theme, people, day, emblem) + '\n'
payloads[f'star-history-{theme}.svg'] = render.stars(theme, history, day, emblem) + '\n'
for svg in payloads.values():
ET.fromstring(svg)
for name, data in {
'history.json': history,
'curated.json': curated,
'contributors.json': {'repository': REPOSITORY, 'updated': day, 'people': people},
}.items():
payloads[name] = json.dumps(data, indent=2, ensure_ascii=False) + '\n'
payloads['README.md'] = f'''# SILO repository cards
Generated by [Repository Cards](https://github.com/pgsty/silo/actions/workflows/repository-cards.yml)
at 00:00 UTC daily (08:00 Asia/Shanghai). GitHub may queue scheduled runs.
Snapshot: {day}. {metadata['stargazers_count']:,} stars; {len(people)} community contributors.
- `contributors-light.svg` / `contributors-dark.svg`: human issue and PR authors across the SILO project scope, plus reviewed acknowledgements. Bots are excluded. Gold rings follow the reviewed companion-site roster; new authors are collected automatically.
- `star-history-light.svg` / `star-history-dark.svg`: initial history reconstructed from the then-current stargazers; later points are daily observed totals, including decreases. Missing days are not fabricated.
- `curated.json`: a cache of reviewed contributor credit from `pgsty/silo.pgsty.com/data/home/contributors.yaml`. The approved initial preview may be newer than the published site; a newer reviewed snapshot is retained until the site catches up.
- `contributors.json`: generated contributor data and embedded raster avatars. Failed avatar refreshes use the previous image, or an initial when no image is available.
- `history.json`: persistent daily totals. Keep this file when regenerating images.
The SVGs are self-contained. Source and instructions live on the default branch;
this branch contains generated assets only. Do not merge it into `main`.
'''
# Collect and validate everything before touching the publication checkout.
output.mkdir(parents=True, exist_ok=True)
for filename, text in payloads.items():
(output / filename).write_text(text)
print(f'{day}: {len(people)} contributors; {metadata["stargazers_count"]:,} stars; {len(history["points"])} history points')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output', type=Path, required=True)
args = parser.parse_args()
configured = os.environ.get('GITHUB_REPOSITORY', REPOSITORY)
if configured.lower() != REPOSITORY:
raise SystemExit('This workflow is scoped to pgsty/silo')
refresh(args.output, GitHub(os.environ.get('GH_TOKEN', '')), Path(__file__).resolve().parents[2])
if __name__ == '__main__':
main()
+11 -9
View File
@@ -36,7 +36,7 @@ for file in \
buildscripts/verify-helm-migration.sh \
Dockerfile.goreleaser \
Dockerfile.distroless \
dockerscripts/download-static-curl.sh \
dockerscripts/build-static-curl.sh \
dockerscripts/docker-entrypoint.sh \
helm/silo/Chart.yaml \
helm/silo/values.yaml \
@@ -101,7 +101,7 @@ require_text Dockerfile.goreleaser "Published checksum drift"
require_text Dockerfile.distroless 'COPY --chmod=0755 silo /usr/bin/silo'
require_text Dockerfile.distroless 'ENTRYPOINT ["/usr/bin/silo"]'
require_text Dockerfile.distroless '"/usr/bin/silo", "healthcheck", "ready"'
require_text dockerscripts/download-static-curl.sh "sha256sum -c"
require_text dockerscripts/build-static-curl.sh "sha256sum -c"
require_text helm/silo/Chart.yaml "name: silo"
require_text helm/silo/values.yaml "repository: pgsty/silo"
require_text helm/silo/templates/deployment.yaml "/usr/bin/docker-entrypoint.sh silo server"
@@ -159,11 +159,13 @@ fi
# The repository and its default branch are pgsty/silo and main. The invariant
# is that the old name is never a live target, not that it is never spoken: the
# READMEs have to name it to explain the rename and to point at the archived
# artifacts, which is the opposite of stranding a reader on it.
# artifacts, which is the opposite of stranding a reader on it. CONTRIBUTORS.md
# also quotes historical issue titles.
#
# So two rules. First, no live URL may resolve to the old repository anywhere,
# READMEs included.
stale_repo_url="$(rg -n -e 'github\.com/pgsty/minio' -e 'hub\.docker\.com/r/pgsty/minio' \
# READMEs and CONTRIBUTORS.md included.
old_repo_pattern='pgsty/minio(\.git)?([^[:alnum:]_.-]|$)'
stale_repo_url="$(rg -n -e "github\.com/${old_repo_pattern}" -e "hub\.docker\.com/r/${old_repo_pattern}" \
--glob '!.git/**' --glob '!dist/**' \
--glob '!SILO_REBRANDING_MIGRATION.md' \
--glob '!buildscripts/rebrand-guard/compat-baseline.json' . |
@@ -175,10 +177,10 @@ fi
# Second, the bare name may only appear where it is deliberate: the pinned
# pre-rebrand image digest in the upgrade test, the two guards that refuse a
# legacy image, and the two READMEs that document the rename and the archived
# minio branch.
repo_guard_allowlist='^(buildscripts/minio-upgrade\.sh|buildscripts/verify-rebrand\.sh|buildscripts/helm-migration-guard/main\.go|README\.md|README_ZH\.md):'
stale_repo="$(rg -n 'pgsty/minio' --glob '!.git/**' --glob '!dist/**' \
# legacy image, the two READMEs that document the rename and the archived
# minio branch, and historical issue titles in CONTRIBUTORS.md.
repo_guard_allowlist='^(buildscripts/minio-upgrade\.sh|buildscripts/verify-rebrand\.sh|buildscripts/helm-migration-guard/main\.go|README\.md|README_ZH\.md|CONTRIBUTORS\.md):'
stale_repo="$(rg -n "${old_repo_pattern}" --glob '!.git/**' --glob '!dist/**' \
--glob '!SILO_REBRANDING_MIGRATION.md' \
--glob '!buildscripts/rebrand-guard/compat-baseline.json' . |
sed 's#^\./##' | grep -Ev "${repo_guard_allowlist}" || true)"
+348
View File
@@ -0,0 +1,348 @@
package cmd
import (
"archive/zip"
"bytes"
"encoding/base64"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
"github.com/minio/mux"
)
func corsAdminRequest(t *testing.T, cred auth.Credentials, method, path string, body []byte) *httptest.ResponseRecorder {
t.Helper()
router := mux.NewRouter()
registerAdminRouter(router, true)
req, err := newTestSignedRequestV4(method, adminPathPrefix+adminAPIVersionPrefix+path,
int64(len(body)), bytes.NewReader(body), cred.AccessKey, cred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("admin %s: %d: %s", path, rec.Code, rec.Body.String())
}
return rec
}
func corsImportReport(t *testing.T, rec *httptest.ResponseRecorder) madmin.BucketMetaImportErrs {
t.Helper()
var rpt madmin.BucketMetaImportErrs
if err := json.Unmarshal(rec.Body.Bytes(), &rpt); err != nil {
t.Fatalf("import report %q: %v", rec.Body.String(), err)
}
return rpt
}
func corsZip(t *testing.T, entries map[string][]byte) []byte {
t.Helper()
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
for name, data := range entries {
w, err := zw.Create(name)
if err != nil {
t.Fatal(err)
}
if _, err = w.Write(data); err != nil {
t.Fatal(err)
}
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
// corsCorruptedZip builds an archive holding a stored (uncompressed) cors.xml
// whose payload is altered after the checksum is computed, plus the given
// companion entries. The altered document stays well formed, so only the zip
// checksum tells the two apart.
func corsCorruptedZip(t *testing.T, name string, doc []byte, others map[string][]byte) []byte {
t.Helper()
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
w, err := zw.CreateHeader(&zip.FileHeader{Name: name, Method: zip.Store})
if err != nil {
t.Fatal(err)
}
if _, err = w.Write(doc); err != nil {
t.Fatal(err)
}
for other, data := range others {
ow, err := zw.Create(other)
if err != nil {
t.Fatal(err)
}
if _, err = ow.Write(data); err != nil {
t.Fatal(err)
}
}
if err = zw.Close(); err != nil {
t.Fatal(err)
}
raw := buf.Bytes()
at := bytes.Index(raw, []byte("app.example.com"))
if at < 0 {
t.Fatalf("stored CORS payload not found in archive")
}
raw[at] = 'A'
return raw
}
// TestAdminBucketMetadataCORSRoundTrip covers the export/import round trip for
// per-bucket CORS, per-file error reporting for an invalid document, and that
// an archive without cors.xml leaves an existing configuration alone.
func TestAdminBucketMetadataCORSRoundTrip(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instanceType, bucket string, _ http.Handler, cred auth.Credentials, t *testing.T) {
corsXML := []byte(testSiteReplicationCORSDoc)
if _, err := updateLocalBucketCORSMetadata(t.Context(), obj, bucket, corsXML); err != nil {
t.Fatal(err)
}
// Export must carry the stored document verbatim.
rec := corsAdminRequest(t, cred, http.MethodGet, "/export-bucket-metadata?bucket="+bucket, nil)
archive := rec.Body.Bytes()
zr, err := zip.NewReader(bytes.NewReader(archive), int64(len(archive)))
if err != nil {
t.Fatal(err)
}
var exported []byte
for _, f := range zr.File {
if f.Name != bucket+"/"+bucketCorsConfig {
continue
}
r, err := f.Open()
if err != nil {
t.Fatal(err)
}
exported, err = io.ReadAll(r)
r.Close()
if err != nil {
t.Fatal(err)
}
}
if !bytes.Equal(exported, corsXML) {
t.Fatalf("%s: exported CORS = %q, want %q", instanceType, exported, corsXML)
}
// Drop the configuration: the archive must then omit the entry.
if _, err = updateLocalBucketCORSMetadata(t.Context(), obj, bucket, nil); err != nil {
t.Fatal(err)
}
if _, _, err = globalBucketMetadataSys.GetCorsConfigXML(bucket); err == nil {
t.Fatalf("%s: CORS still present before restore", instanceType)
}
rec = corsAdminRequest(t, cred, http.MethodGet, "/export-bucket-metadata?bucket="+bucket, nil)
empty := rec.Body.Bytes()
zr, err = zip.NewReader(bytes.NewReader(empty), int64(len(empty)))
if err != nil {
t.Fatal(err)
}
for _, f := range zr.File {
if f.Name == bucket+"/"+bucketCorsConfig {
t.Fatalf("%s: export emitted %s for a bucket without CORS", instanceType, f.Name)
}
}
rec = corsAdminRequest(t, cred, http.MethodPut, "/import-bucket-metadata", archive)
if st := corsImportReport(t, rec).Buckets[bucket]; !st.Cors.IsSet || st.Cors.Err != "" {
t.Fatalf("%s: import report cors = %+v", instanceType, st.Cors)
}
stored, storedAt, err := globalBucketMetadataSys.GetCorsConfigXML(bucket)
if err != nil || !bytes.Equal(stored, corsXML) {
t.Fatalf("%s: restored CORS = %q, err = %v", instanceType, stored, err)
}
created, err := globalBucketMetadataSys.CreatedAt(bucket)
if err != nil {
t.Fatal(err)
}
if !storedAt.After(created) {
t.Fatalf("%s: restored CORS timestamp %v is not after bucket creation %v", instanceType, storedAt, created)
}
// An archive without cors.xml must not remove the configuration.
corsAdminRequest(t, cred, http.MethodPut, "/import-bucket-metadata",
corsZip(t, map[string][]byte{bucket + "/quota.json": []byte(`{"quota":0}`)}))
if stored, _, err = globalBucketMetadataSys.GetCorsConfigXML(bucket); err != nil || !bytes.Equal(stored, corsXML) {
t.Fatalf("%s: import without cors.xml changed CORS: %q, err = %v", instanceType, stored, err)
}
// A bucket the import itself creates must still land above its own
// creation time, otherwise CORS replication would drop the restore.
fresh := "cors-import-created-bucket"
rec = corsAdminRequest(t, cred, http.MethodPut, "/import-bucket-metadata",
corsZip(t, map[string][]byte{fresh + "/" + bucketCorsConfig: corsXML}))
if st := corsImportReport(t, rec).Buckets[fresh]; !st.Cors.IsSet || st.Cors.Err != "" {
t.Fatalf("%s: fresh bucket import report cors = %+v", instanceType, st.Cors)
}
freshStored, freshAt, err := globalBucketMetadataSys.GetCorsConfigXML(fresh)
if err != nil || !bytes.Equal(freshStored, corsXML) {
t.Fatalf("%s: fresh bucket CORS = %q, err = %v", instanceType, freshStored, err)
}
freshCreated, err := globalBucketMetadataSys.CreatedAt(fresh)
if err != nil {
t.Fatal(err)
}
if !freshAt.After(freshCreated) {
t.Fatalf("%s: fresh bucket CORS timestamp %v is not after creation %v", instanceType, freshAt, freshCreated)
}
// An invalid document must fail loudly for that bucket and change nothing.
rec = corsAdminRequest(t, cred, http.MethodPut, "/import-bucket-metadata",
corsZip(t, map[string][]byte{bucket + "/" + bucketCorsConfig: []byte("<CORSConfiguration><CORSRule>")}))
if st := corsImportReport(t, rec).Buckets[bucket]; st.Cors.Err == "" {
t.Fatalf("%s: invalid CORS import reported no error: %+v", instanceType, st)
}
if stored, _, err = globalBucketMetadataSys.GetCorsConfigXML(bucket); err != nil || !bytes.Equal(stored, corsXML) {
t.Fatalf("%s: invalid CORS import changed stored config: %q, err = %v", instanceType, stored, err)
}
// A well formed document carried by a corrupt zip entry must be
// rejected too, leaving the stored document and its timestamp alone
// while the other configs in the same archive still apply.
_, corsAt, err := globalBucketMetadataSys.GetCorsConfigXML(bucket)
if err != nil {
t.Fatal(err)
}
rec = corsAdminRequest(t, cred, http.MethodPut, "/import-bucket-metadata",
corsCorruptedZip(t, bucket+"/"+bucketCorsConfig, corsXML,
map[string][]byte{bucket + "/quota.json": []byte(`{"quota":4096,"quotatype":"hard"}`)}))
st := corsImportReport(t, rec).Buckets[bucket]
if st.Cors.Err == "" {
t.Fatalf("%s: corrupt CORS entry reported no error: %+v", instanceType, st)
}
if !st.Quota.IsSet || st.Quota.Err != "" {
t.Fatalf("%s: corrupt CORS entry blocked the neighboring quota: %+v", instanceType, st.Quota)
}
stored, storedAt, err = globalBucketMetadataSys.GetCorsConfigXML(bucket)
if err != nil || !bytes.Equal(stored, corsXML) || !storedAt.Equal(corsAt) {
t.Fatalf("%s: corrupt CORS entry changed stored config: %q at %v (was %v), err = %v", instanceType, stored, storedAt, corsAt, err)
}
quota, _, err := globalBucketMetadataSys.GetQuotaConfig(t.Context(), bucket)
if err != nil || quota == nil || quota.Quota != 4096 {
t.Fatalf("%s: neighboring quota not applied: %+v, err = %v", instanceType, quota, err)
}
}})
}
// corsPeerStub is a stand-in site-replication peer. It records every
// SRBucketMeta it is asked to apply and answers with status.
func corsPeerStub(t *testing.T, applied chan<- madmin.SRBucketMeta, status int) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPut && applied != nil {
var item madmin.SRBucketMeta
if err := json.NewDecoder(r.Body).Decode(&item); err != nil {
t.Errorf("decode peer apply: %v", err)
w.WriteHeader(http.StatusBadRequest)
return
}
applied <- item
}
w.WriteHeader(status)
}))
}
// TestAdminBucketMetadataCORSImportReplicatesPastPeerFailure pins that an
// imported CORS document reaches the reachable peers even when the shared
// bucket metadata hook failed against an unreachable one, and that both
// failures are still reported for the bucket.
func TestAdminBucketMetadataCORSImportReplicatesPastPeerFailure(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instanceType, bucket string, _ http.Handler, cred auth.Credentials, t *testing.T) {
ctx := t.Context()
corsXML := []byte(testSiteReplicationCORSDoc)
healthyApplies := make(chan madmin.SRBucketMeta, 4)
healthy := corsPeerStub(t, healthyApplies, http.StatusOK)
defer healthy.Close()
broken := corsPeerStub(t, nil, http.StatusBadRequest)
defer broken.Close()
// With site replication on, admin requests resolve their token signing
// key through the site replicator account, so it has to exist.
serviceCred, err := auth.CreateCredentials(siteReplicatorSvcAcc, "cors-import-service-secret")
if err != nil {
t.Fatal(err)
}
serviceCred.ParentUser = cred.AccessKey
if _, err = globalIAMSys.store.AddServiceAccount(ctx, serviceCred); err != nil {
t.Fatal(err)
}
defer globalIAMSys.DeleteServiceAccount(ctx, serviceCred.AccessKey, false)
globalSiteReplicatorCred.Set(serviceCred.SecretKey)
defer globalSiteReplicatorCred.Set("")
globalSiteReplicationSys.Lock()
oldEnabled, oldState := globalSiteReplicationSys.enabled, globalSiteReplicationSys.state
globalSiteReplicationSys.enabled = true
globalSiteReplicationSys.state = srState{
Name: "cors-import-test",
ServiceAccountAccessKey: serviceCred.AccessKey,
Peers: map[string]madmin.PeerInfo{
globalDeploymentID(): {Name: "local", DeploymentID: globalDeploymentID()},
"peer-healthy": {Name: "healthy", DeploymentID: "peer-healthy", Endpoint: healthy.URL},
"peer-broken": {Name: "broken", DeploymentID: "peer-broken", Endpoint: broken.URL},
},
}
globalSiteReplicationSys.Unlock()
defer func() {
globalSiteReplicationSys.Lock()
globalSiteReplicationSys.enabled, globalSiteReplicationSys.state = oldEnabled, oldState
globalSiteReplicationSys.Unlock()
}()
rec := corsAdminRequest(t, cred, http.MethodPut, "/import-bucket-metadata",
corsZip(t, map[string][]byte{
bucket + "/" + bucketCorsConfig: corsXML,
bucket + "/quota.json": []byte(`{"quota":8192,"quotatype":"hard"}`),
}))
st := corsImportReport(t, rec).Buckets[bucket]
if !st.Cors.IsSet || st.Cors.Err != "" {
t.Fatalf("%s: import report cors = %+v", instanceType, st.Cors)
}
stored, storedAt, err := globalBucketMetadataSys.GetCorsConfigXML(bucket)
if err != nil || !bytes.Equal(stored, corsXML) {
t.Fatalf("%s: stored CORS = %q, err = %v", instanceType, stored, err)
}
// The reachable peer must have been told about the CORS document,
// carrying exactly the timestamp that was saved locally.
var corsSeen, sharedSeen bool
for range 2 {
select {
case item := <-healthyApplies:
if item.Type != madmin.SRBucketMetaTypeCorsConfig {
sharedSeen = item.Bucket == bucket && item.Quota != nil
continue
}
if item.Bucket != bucket || item.Cors == nil || !item.UpdatedAt.Equal(storedAt) {
t.Fatalf("%s: peer CORS event = %#v, want %s at %v", instanceType, item, bucket, storedAt)
}
payload, decErr := base64.StdEncoding.Strict().DecodeString(*item.Cors)
if decErr != nil || !bytes.Equal(payload, corsXML) {
t.Fatalf("%s: peer CORS payload = %q, err = %v", instanceType, payload, decErr)
}
corsSeen = true
case <-time.After(10 * time.Second):
t.Fatalf("%s: healthy peer received no further events (shared=%v cors=%v)", instanceType, sharedSeen, corsSeen)
}
}
if !sharedSeen || !corsSeen {
t.Fatalf("%s: healthy peer events shared=%v cors=%v, want both", instanceType, sharedSeen, corsSeen)
}
// Both hook failures against the unreachable peer stay reported.
if got := strings.Count(st.Err, "->broken:"); got != 2 {
t.Fatalf("%s: bucket error mentions the broken peer %d times, want 2: %q", instanceType, got, st.Err)
}
}})
}
+93 -10
View File
@@ -76,7 +76,7 @@ func (a adminAPIHandlers) PutBucketQuotaConfigHandler(w http.ResponseWriter, r *
return
}
quotaConfig, err := parseBucketQuota(bucket, data)
_, err = parseBucketQuota(bucket, data)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -94,9 +94,6 @@ func (a adminAPIHandlers) PutBucketQuotaConfigHandler(w http.ResponseWriter, r *
Quota: data,
UpdatedAt: updatedAt,
}
if quotaConfig.Size == 0 && quotaConfig.Quota == 0 {
bucketMeta.Quota = nil
}
// Call site replication hook.
replLogIf(ctx, globalSiteReplicationSys.BucketMetaHook(ctx, bucketMeta))
@@ -417,6 +414,7 @@ func (a adminAPIHandlers) ExportBucketMetadataHandler(w http.ResponseWriter, r *
bucketLifecycleConfig,
bucketSSEConfig,
bucketTaggingConfig,
bucketCorsConfig,
bucketQuotaConfigFile,
objectLockConfig,
bucketVersioningConfig,
@@ -437,7 +435,7 @@ func (a adminAPIHandlers) ExportBucketMetadataHandler(w http.ResponseWriter, r *
writeErrorResponse(ctx, w, exportError(ctx, err, cfgFile, bucket), r.URL)
return
}
configData, err := json.Marshal(config)
configData, err := canonicalBucketPolicy(config)
if err != nil {
writeErrorResponse(ctx, w, exportError(ctx, err, cfgFile, bucket), r.URL)
return
@@ -517,6 +515,19 @@ func (a adminAPIHandlers) ExportBucketMetadataHandler(w http.ResponseWriter, r *
return
}
rawDataFn(bytes.NewReader(configData), cfgPath, len(configData))
case bucketCorsConfig:
// Export the stored document verbatim: GetBucketCors returns
// the bytes exactly as they were PUT, so the archive must
// round-trip them unchanged.
configData, _, err := globalBucketMetadataSys.GetCorsConfigXML(bucket)
if err != nil {
if errors.Is(err, errConfigNotFound) {
continue
}
writeErrorResponse(ctx, w, exportError(ctx, err, cfgFile, bucket), r.URL)
return
}
rawDataFn(bytes.NewReader(configData), cfgPath, len(configData))
case objectLockConfig:
config, _, err := globalBucketMetadataSys.GetObjectLockConfig(bucket)
if err != nil {
@@ -616,6 +627,13 @@ func applyImportedBucketMetadata(dst *BucketMetadata, src BucketMetadata, fields
case bucketQuotaConfigFile:
dst.QuotaConfigJSON = bytes.Clone(src.QuotaConfigJSON)
dst.QuotaConfigUpdatedAt = src.QuotaConfigUpdatedAt
case bucketCorsConfig:
// The import stamps its fields before creating any missing bucket,
// and a CORS event stamped before bucket creation is discarded as
// belonging to an older incarnation, so the imported document takes
// the same monotonic timestamp a local PutBucketCors would assign.
dst.CorsConfigUpdatedAt = localCORSUpdatedAt(*dst, src.CorsConfigUpdatedAt)
dst.CorsConfigXML = bytes.Clone(src.CorsConfigXML)
case objectLockConfig:
dst.ObjectLockConfigXML = bytes.Clone(src.ObjectLockConfigXML)
dst.ObjectLockConfigUpdatedAt = src.ObjectLockConfigUpdatedAt
@@ -645,6 +663,8 @@ func (i *importMetaReport) SetStatus(bucket, fname string, err error) {
st.Tagging = madmin.MetaStatus{IsSet: true, Err: errMsg}
case bucketQuotaConfigFile:
st.Quota = madmin.MetaStatus{IsSet: true, Err: errMsg}
case bucketCorsConfig:
st.Cors = madmin.MetaStatus{IsSet: true, Err: errMsg}
case objectLockConfig:
st.ObjectLock = madmin.MetaStatus{IsSet: true, Err: errMsg}
case bucketVersioningConfig:
@@ -899,7 +919,7 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
continue
}
configData, err := json.Marshal(bucketPolicy)
configData, err := canonicalBucketPolicy(bucketPolicy)
if err != nil {
rpt.SetStatus(bucket, fileName, err)
continue
@@ -1015,6 +1035,32 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
bucketMap[bucket].QuotaConfigUpdatedAt = updatedAt
markImported(bucket, fileName)
rpt.SetStatus(bucket, fileName, nil)
case bucketCorsConfig:
if sz > maxBucketCorsSize {
rpt.SetStatus(bucket, fileName, errors.New(ErrEntityTooLarge.String()))
continue
}
// Read one byte past the declared size: stopping exactly at sz
// leaves archive/zip short of EOF, so it never verifies the entry
// checksum and a corrupt entry carrying well formed XML would be
// stored as a valid document. The extra byte also lets the reader
// reject an entry longer than it declares.
corsData, err := io.ReadAll(io.LimitReader(reader, sz+1))
if err != nil {
rpt.SetStatus(bucket, fileName, err)
continue
}
if err = validateCORSReplicationPayload(corsData); err != nil {
rpt.SetStatus(bucket, fileName, fmt.Errorf("%s (%s)", errorCodes[ErrMalformedXML].Description, err))
continue
}
bucketMap[bucket].CorsConfigXML = corsData
bucketMap[bucket].CorsConfigUpdatedAt = updatedAt
markImported(bucket, fileName)
rpt.SetStatus(bucket, fileName, nil)
}
}
@@ -1032,18 +1078,39 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
continue
}
var merged BucketMetadata
var commitAt time.Time
err := func() error {
lockCtx, unlock, err := lockBucketMetadata(ctx, objectAPI, bucket)
if err != nil {
return err
}
defer unlock()
merged, err = loadBucketMetadataParse(lockCtx, objectAPI, bucket, true)
merged, err = loadBucketMetadataParse(lockCtx, objectAPI, bucket, false)
if err != nil {
return err
}
if err := ensureBucketMetadataCreated(lockCtx, objectAPI, &merged); err != nil {
return err
}
commitAt = UTCNow()
for _, file := range replicatedBucketConfigs {
if _, ok := fields[file]; ok {
commitAt = localBucketConfigUpdatedAt(merged, file, commitAt)
}
}
applyImportedBucketMetadata(&merged, *meta, fields)
return globalBucketMetadataSys.saveMetadata(lockCtx, objectAPI, merged)
for _, file := range replicatedBucketConfigs {
if _, ok := fields[file]; !ok {
continue
}
data, at := replicatedBucketConfig(&merged, file)
payload, _, err := bucketConfigPayload(bucket, file, *data, len(merged.ObjectLockConfigXML) != 0)
if err != nil {
return err
}
*data, *at = payload, commitAt
}
return globalBucketMetadataSys.saveMetadata(lockCtx, objectAPI, &merged)
}()
if err != nil {
rpt.SetStatus(bucket, "", err)
@@ -1051,7 +1118,7 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
}
*meta = merged
globalNotificationSys.LoadBucketMetadata(bgContext(ctx), bucket)
hook := madmin.SRBucketMeta{Bucket: bucket, UpdatedAt: updatedAt}
hook := madmin.SRBucketMeta{Bucket: bucket, UpdatedAt: commitAt}
var hookNeeded bool
if _, ok := fields[bucketQuotaConfigFile]; ok {
hook.Quota = meta.QuotaConfigJSON
@@ -1059,7 +1126,7 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
}
if _, ok := fields[bucketPolicyConfig]; ok {
hook.Policy = meta.PolicyConfigJSON
hookNeeded = true
hookNeeded = hookNeeded || len(hook.Policy) != 0
}
if _, ok := fields[bucketVersioningConfig]; ok {
hook.Versioning = enc(meta.VersioningConfigXML)
@@ -1080,6 +1147,22 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
if hookNeeded {
err = globalSiteReplicationSys.BucketMetaHook(ctx, hook)
}
if _, ok := fields[bucketPolicyConfig]; ok && len(meta.PolicyConfigJSON) == 0 {
// An omitted bulk Policy cannot express deletion.
err = errors.Join(err, globalSiteReplicationSys.BucketMetaHook(ctx, madmin.SRBucketMeta{
Type: madmin.SRBucketMetaTypePolicy, Bucket: bucket, UpdatedAt: commitAt,
}))
}
if _, ok := fields[bucketCorsConfig]; ok {
// CORS carries its own timestamp, so it replicates through the
// dedicated event rather than the shared bucket metadata hook. It
// is announced even when the shared hook failed: the document is
// already committed locally, and a peer that is unreachable for
// one config must not withhold CORS from the reachable ones.
if corsEvent, live := newBucketCORSReplicationEvent(bucket, *meta); live {
err = errors.Join(err, globalSiteReplicationSys.BucketMetaHook(ctx, corsEvent))
}
}
if err != nil {
rpt.SetStatus(bucket, "", err)
continue
+5 -1
View File
@@ -502,11 +502,15 @@ func (a adminAPIHandlers) AddUser(w http.ResponseWriter, r *http.Request) {
}
checkDenyOnly := accessKey == cred.AccessKey
action := policy.Action(policy.CreateUserAdminAction)
if checkDenyOnly {
action = policy.ChangeMyPasswordAdminAction
}
if !globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.CreateUserAdminAction,
Action: action,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
+102
View File
@@ -243,6 +243,7 @@ func TestIAMInternalIDPServerSuite(t *testing.T) {
suite.SetUpSuite(c)
suite.TestUserCreate(c)
suite.TestUserPasswordActionAuthorization(c)
suite.TestUserStatusActionAuthorization(c)
suite.TestGroupStatusActionAuthorization(c)
suite.TestUserPolicyEscalationBug(c)
@@ -356,6 +357,106 @@ func (s *TestSuiteIAM) TestUserCreate(c *check) {
}
}
func (s *TestSuiteIAM) TestUserPasswordActionAuthorization(c *check) {
for _, tt := range []struct {
name string
statements string
self bool
other bool
}{
{"readonly", "", true, false},
{"consolereadonly", "", true, false},
{"password grant", `{"Effect":"Allow","Action":"admin:ChangeMyPassword"}`, true, false},
{"legacy CreateUser deny", `{"Effect":"Deny","Action":"admin:CreateUser","Resource":"arn:aws:s3:::*"}`, true, false},
{"password deny", `{"Effect":"Deny","Action":"admin:ChangeMyPassword"}`, false, false},
{"user admin", `{"Effect":"Allow","Action":"admin:CreateUser"}`, true, true},
{"user admin with password deny", `{"Effect":"Allow","Action":"admin:CreateUser"},{"Effect":"Deny","Action":"admin:ChangeMyPassword"}`, false, true},
{"password deny overrides grant", `{"Effect":"Allow","Action":"admin:ChangeMyPassword"},{"Effect":"Deny","Action":"admin:ChangeMyPassword"}`, false, false},
{"wildcard deny", `{"Effect":"Deny","Action":"admin:*"}`, false, false},
} {
c.Run(tt.name, func(t *testing.T) {
c := &check{t, s.serverType}
ctx, cancel := context.WithTimeout(context.Background(), testDefaultTimeout)
defer cancel()
var users []string
policyName := tt.name
defer func() {
for _, user := range users {
if err := s.adm.RemoveUser(ctx, user); err != nil {
c.Errorf("remove test user: %v", err)
}
}
if tt.statements != "" {
if err := s.adm.RemoveCannedPolicy(ctx, policyName); err != nil {
c.Errorf("remove test policy: %v", err)
}
}
}()
createUser := func() (string, string) {
accessKey, secretKey := mustGenerateCredentials(c)
if err := s.adm.SetUser(ctx, accessKey, secretKey, madmin.AccountEnabled); err != nil {
c.Fatalf("create test user: %v", err)
}
users = append(users, accessKey)
return accessKey, secretKey
}
client := func(accessKey, secretKey string) *madmin.AdminClient {
adm, err := madmin.New(s.endpoint, accessKey, secretKey, s.secure)
if err != nil {
c.Fatal(err)
}
adm.SetCustomTransport(s.TestSuiteCommon.client.Transport)
return adm
}
if tt.statements != "" {
policyName = getRandomBucketName()
doc := []byte(`{"Version":"2012-10-17","Statement":[` + tt.statements + `]}`)
if err := s.adm.AddCannedPolicy(ctx, policyName, doc); err != nil {
c.Fatalf("save test policy: %v", err)
}
}
accessKey, secretKey := createUser()
if _, err := s.adm.AttachPolicy(ctx, madmin.PolicyAssociationReq{
User: accessKey, Policies: []string{policyName},
}); err != nil {
c.Fatalf("attach test policy: %v", err)
}
adm := client(accessKey, secretKey)
_, newSecretKey := mustGenerateCredentials(c)
err := adm.SetUser(ctx, accessKey, newSecretKey, madmin.AccountEnabled)
if tt.self {
if err != nil {
c.Fatalf("change own password: %v", err)
}
if _, err = adm.AccountInfo(ctx, madmin.AccountOpts{}); err == nil {
c.Fatal("old password still authenticates")
}
adm = client(accessKey, newSecretKey)
} else if err == nil || madmin.ToErrorResponse(err).Code != "AccessDenied" {
c.Fatalf("self password change: expected AccessDenied, got %v", err)
}
if _, err := adm.AccountInfo(ctx, madmin.AccountOpts{}); err != nil {
c.Fatalf("current password no longer authenticates: %v", err)
}
target, _ := createUser()
newUser, newUserSecret := mustGenerateCredentials(c)
for _, key := range []string{target, newUser} {
err := adm.SetUser(ctx, key, newUserSecret, madmin.AccountEnabled)
if tt.other {
if err != nil {
c.Fatalf("create or update another user: %v", err)
}
if key == newUser {
users = append(users, newUser)
}
} else if err == nil || madmin.ToErrorResponse(err).Code != "AccessDenied" {
c.Fatalf("create or update another user: expected AccessDenied, got %v", err)
}
}
})
}
}
func (s *TestSuiteIAM) TestUserStatusActionAuthorization(c *check) {
ctx, cancel := context.WithTimeout(context.Background(), testDefaultTimeout)
defer cancel()
@@ -946,6 +1047,7 @@ func (s *TestSuiteIAM) TestCannedPolicies(c *check) {
defaultPolicies := []string{
"readwrite",
"readonly",
"consolereadonly",
"writeonly",
"diagnostics",
"consoleAdmin",
+2
View File
@@ -163,6 +163,7 @@ func registerAdminRouter(router *mux.Router, enableConfigOps bool) {
// StorageInfo operations
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/storageinfo").HandlerFunc(adminMiddleware(adminAPI.StorageInfoHandler, traceAllFlag))
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/multipart-preflight").HandlerFunc(adminMiddleware(adminAPI.MultipartPreflightHandler, traceAllFlag))
// DataUsageInfo operations
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/datausageinfo").HandlerFunc(adminMiddleware(adminAPI.DataUsageInfoHandler, traceAllFlag))
// Metrics operation
@@ -388,6 +389,7 @@ func registerAdminRouter(router *mux.Router, enableConfigOps bool) {
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/join").HandlerFunc(adminMiddleware(adminAPI.SRPeerJoin))
adminRouter.Methods(http.MethodPut).Path(adminVersion+"/site-replication/peer/bucket-ops").HandlerFunc(adminMiddleware(adminAPI.SRPeerBucketOps)).Queries("bucket", "{bucket:.*}").Queries("operation", "{operation:.*}")
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/iam-item").HandlerFunc(adminMiddleware(adminAPI.SRPeerReplicateIAMItem))
adminRouter.Methods(http.MethodGet, http.MethodPut).Path(adminVersion + "/site-replication/peer/iam-revisions").HandlerFunc(adminMiddleware(adminAPI.SRPeerIAMRevisions))
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/bucket-meta").HandlerFunc(adminMiddleware(adminAPI.SRPeerReplicateBucketItem))
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/site-replication/peer/idp-settings").HandlerFunc(adminMiddleware(adminAPI.SRPeerGetIDPSettings))
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/edit").HandlerFunc(adminMiddleware(adminAPI.SiteReplicationEdit))
+29 -1
View File
@@ -450,6 +450,9 @@ const (
ErrAdminNoSecretKey
ErrIAMNotInitialized
ErrMultipartListingLegacy
ErrMultipartListingIdentity
ErrSlowDown
apiErrCodeEnd // This is used only for the testing code
)
@@ -1336,6 +1339,21 @@ var errorCodes = errorCodeMap{
Description: "IAM sub-system not initialized yet, please try again.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMultipartListingLegacy: {
Code: "MultipartListingNotReady",
Description: "Legacy multipart uploads prevent a complete listing. Upgrade all writers, drain old uploads and run the multipart preflight check.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMultipartListingIdentity: {
Code: "MultipartListingMetadataInvalid",
Description: "Multipart upload metadata is inconsistent. Run the multipart preflight check to locate the affected storage set.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrSlowDown: {
Code: "SlowDown",
Description: "Please reduce your request rate",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrBucketMetadataNotInitialized: {
Code: "XMinioBucketMetadataNotInitialized",
Description: "Bucket metadata not initialized yet, please try again.",
@@ -1523,10 +1541,14 @@ var errorCodes = errorCodeMap{
Description: "Your Host header is malformed.",
HTTPStatusCode: http.StatusBadRequest,
},
// The stored object cannot be served: a server-side data condition, not a
// successful partial read. Upstream maps it to http.StatusPartialContent
// (since ca6b4773e, 2017), which lets SDKs accept the XML error document
// as object content; SILO deliberately diverges and returns 500.
ErrObjectTampered: {
Code: "XMinioObjectTampered",
Description: errObjectTampered.Error(),
HTTPStatusCode: http.StatusPartialContent,
HTTPStatusCode: http.StatusInternalServerError,
},
ErrSiteReplicationInvalidRequest: {
@@ -2169,6 +2191,10 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
err = unwrapAll(err)
switch err {
case errMultipartListingLegacy:
apiErr = ErrMultipartListingLegacy
case errMultipartListingIdentity:
apiErr = ErrMultipartListingIdentity
case errCompleteMultipartChecksumMismatch, errCompleteMultipartChecksumTypeMismatch:
apiErr = ErrBadDigest
case errMissingPartChecksum:
@@ -2299,6 +2325,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
}
switch err.(type) {
case SlowDown:
apiErr = ErrSlowDown
case StorageFull:
apiErr = ErrStorageFull
case hash.BadDigest:
+4 -1
View File
@@ -212,7 +212,10 @@ func setObjectHeaders(ctx context.Context, w http.ResponseWriter, objInfo Object
}
if rs == nil && opts.PartNumber > 0 {
rs = partNumberToRangeSpec(objInfo, opts.PartNumber)
rs, err = partNumberToRangeSpec(objInfo, opts.PartNumber)
if err != nil {
return err
}
}
// For providing ranged content
+1 -1
View File
@@ -41,7 +41,7 @@ import (
const (
maxObjectList = 1000 // Limit number of objects in a listObjectsResponse/listObjectsVersionsResponse.
maxDeleteList = 1000 // Limit number of objects deleted in a delete call.
maxUploadsList = 10000 // Limit number of uploads in a listUploadsResponse.
maxUploadsList = 1000 // Limit number of uploads in a listUploadsResponse.
maxPartsList = 10000 // Limit number of parts in a listPartsResponse.
)
File diff suppressed because one or more lines are too long
+4 -11
View File
@@ -632,18 +632,11 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
return ErrInvalidDigest
}
// Extract either 'X-Amz-Content-Sha256' header or 'X-Amz-Content-Sha256' query parameter (if V4 presigned)
// Do not verify 'X-Amz-Content-Sha256' if skipSHA256.
// Honor the selected header/query checksum, including the header fallback
// for a presigned request. STS separately hashes its body for its signature.
var contentSHA256 []byte
if skipSHA256 := skipContentSha256Cksum(r); !skipSHA256 && isRequestPresignedSignatureV4(r) {
if sha256Sum, ok := r.Form[xhttp.AmzContentSha256]; ok && len(sha256Sum) > 0 {
contentSHA256, err = hex.DecodeString(sha256Sum[0])
if err != nil {
return ErrContentSHA256Mismatch
}
}
} else if _, ok := r.Header[xhttp.AmzContentSha256]; !skipSHA256 && ok {
contentSHA256, err = hex.DecodeString(r.Header.Get(xhttp.AmzContentSha256))
if !skipContentSha256Cksum(r) {
contentSHA256, err = hex.DecodeString(getContentSha256Cksum(r, serviceS3))
if err != nil || len(contentSHA256) == 0 {
return ErrContentSHA256Mismatch
}
-19
View File
@@ -467,25 +467,6 @@ func testAdversarialHealCorsPropagatesNewerEqualValueTimestamp(obj ObjectLayer,
}
}
func TestAdversarialBucketMetadataComparisonIsBase64CaseSensitive(t *testing.T) {
upper := "QQ=="
lower := "qQ=="
upperBytes, err := base64.StdEncoding.Strict().DecodeString(upper)
if err != nil {
t.Fatal(err)
}
lowerBytes, err := base64.StdEncoding.Strict().DecodeString(lower)
if err != nil {
t.Fatal(err)
}
if string(upperBytes) == string(lowerBytes) {
t.Fatal("test inputs unexpectedly decode to the same bytes")
}
if isBucketMetadataEqual(&upper, &lower) {
t.Fatal("different decoded payloads were treated as equal")
}
}
func TestSiteReplicationStatusDetectsCorsTimestampMismatch(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
-8
View File
@@ -277,14 +277,6 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
return
}
if keyMarker != "" {
// Marker not common with prefix is not implemented.
if !HasPrefix(keyMarker, prefix) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrNotImplemented), r.URL)
return
}
}
listMultipartsInfo, err := objectAPI.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
+3 -3
View File
@@ -425,7 +425,7 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
shouldPass: true,
},
// Test case - 4.
// Setting Invalid prefix and marker combination.
// A key marker outside the prefix is valid and produces an empty page.
{
bucket: bucketName,
prefix: "asia",
@@ -435,8 +435,8 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
maxUploads: "0",
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
expectedRespStatus: http.StatusNotImplemented,
shouldPass: false,
expectedRespStatus: http.StatusOK,
shouldPass: true,
},
// Test case - 5.
// Invalid upload id and marker combination.
+111
View File
@@ -0,0 +1,111 @@
// Copyright 2026 PGSTY contributors.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"errors"
"fmt"
"net/http"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/auth"
)
func TestDeleteBucketMetadataLockCancellation(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testDeleteBucketMetadataLockCancellation})
}
func testDeleteBucketMetadataLockCancellation(obj ObjectLayer, instanceType, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
_, unlock, err := lockBucketMetadata(t.Context(), obj, bucket)
if err != nil {
t.Fatal(err)
}
release := sync.OnceFunc(unlock)
defer release()
// Observe DeleteBucket's ACTUAL metadata.lock attempt. Set the hook after
// our own acquisition above so it only trips on the delete.
delAtLock := make(chan struct{})
var once sync.Once
hook := func(b string) {
if b == bucket {
once.Do(func() { close(delAtLock) })
}
}
lockBucketMetadataAcquireHook.Store(&hook)
defer lockBucketMetadataAcquireHook.Store(nil)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() { done <- obj.DeleteBucket(ctx, bucket, DeleteBucketOptions{Force: true, NoLock: true}) }()
select {
case <-delAtLock:
// Fixed tree: the delete reached metadata.lock and is blocking on the
// lock we hold. Cancel it and confirm it fails WITHOUT deleting, while we
// still hold the lock (release stays deferred until after the checks).
cancel()
if err := <-done; err == nil {
t.Errorf("%s: canceled deletion succeeded", instanceType)
}
if _, err := obj.GetBucketInfo(t.Context(), bucket, BucketOptions{}); err != nil {
t.Errorf("%s: bucket disappeared while metadata.lock was held: %v", instanceType, err)
}
if _, err := readBucketMetadata(t.Context(), obj, bucket); err != nil {
t.Errorf("%s: canceled deletion removed metadata: %v", instanceType, err)
}
case err := <-done:
// Broken tree: the delete finished without ever taking metadata.lock,
// i.e. it did not serialize the destructive operation behind the lock.
t.Errorf("%s: delete bypassed metadata.lock (err=%v)", instanceType, err)
}
}
func TestQueuedMetadataUpdateAfterDelete(t *testing.T) {
defer DetectTestLeak(t)()
for _, expiry := range []bool{false, true} {
t.Run(fmt.Sprintf("expiry=%v", expiry), func(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instanceType, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
previous := newObjectLayerFn()
barrier := &lcMergeBarrier{ObjectLayer: obj, bucket: bucket, mAtLock: make(chan struct{}), mProceed: make(chan struct{})}
setObjectLayer(barrier)
defer setObjectLayer(previous)
release := sync.OnceFunc(func() { close(barrier.mProceed) })
defer release()
ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
defer cancel()
ctx = context.WithValue(ctx, lcMergeWriterKey{}, "M")
done := make(chan error, 1)
go func() {
if expiry {
done <- globalBucketMetadataSys.UpdateExpiryLCConfig(ctx, bucket, nil, UTCNow())
return
}
_, err := globalBucketMetadataSys.Update(ctx, bucket, bucketTaggingConfig, []byte(`<Tagging><TagSet/></Tagging>`))
done <- err
}()
select {
case <-barrier.mAtLock:
case <-ctx.Done():
t.Fatal("writer did not reach metadata.lock")
}
if err := obj.DeleteBucket(t.Context(), bucket, DeleteBucketOptions{Force: true}); err != nil {
t.Fatal(err)
}
release()
if err := <-done; !isErrBucketNotFound(err) {
t.Errorf("%s: queued update should reject a deleted bucket, got %v", instanceType, err)
}
if _, err := readBucketMetadata(t.Context(), obj, bucket); !errors.Is(err, errConfigNotFound) && !isErrBucketNotFound(err) {
t.Errorf("%s: queued update recreated metadata: %v", instanceType, err)
}
}})
})
}
}
+278
View File
@@ -0,0 +1,278 @@
// Copyright (c) 2015-2026 MinIO, Inc.
// Copyright (c) 2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
package cmd
import (
"context"
"encoding/base64"
"errors"
"net/http"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/auth"
)
// ---------------------------------------------------------------------------
// Target 1 (issue #105): a higher-level lifecycle XML merge racing another
// bucket-metadata transition outside BucketMetadataSys.Delete.
//
// Before the fix, PeerBucketLCConfigHandler / healBucketILMExpiry read the
// current lifecycle document without metadata.lock, merged the replicated expiry
// rules with the local transition rules, and then persisted the merged blob with
// BucketMetadataSys.Update. Update re-read the record under metadata.lock but
// overwrote LifecycleConfigXML wholesale with the pre-computed blob, so any
// lifecycle transition change committed between the merge read and the merge
// write was silently lost on disk. BucketMetadataSys.UpdateExpiryLCConfig now
// performs the read, merge, and save under a single metadata.lock. This test
// drives PeerBucketLCConfigHandler and asserts the concurrent change survives.
// ---------------------------------------------------------------------------
type lcMergeWriterKey struct{}
// lcMergeBarrier pauses the merge writer (context value "M") exactly when it
// tries to take metadata.lock for its persisting Update. By that point the
// merge has already read the stale lifecycle document, so the test can commit a
// concurrent transition change before releasing the merge write.
type lcMergeBarrier struct {
ObjectLayer
bucket string
mAtLock chan struct{}
mProceed chan struct{}
mOnce sync.Once
}
func (o *lcMergeBarrier) metadataLock() string {
return pathJoin(bucketMetaPrefix, o.bucket, "metadata.lock")
}
func (o *lcMergeBarrier) NewNSLock(bucket string, objects ...string) RWLocker {
lock := o.ObjectLayer.NewNSLock(bucket, objects...)
if bucket != minioMetaBucket || len(objects) != 1 || objects[0] != o.metadataLock() {
return lock
}
return metadataObservedRWLocker{RWLocker: lock, onLock: func(ctx context.Context) {
if ctx.Value(lcMergeWriterKey{}) != "M" {
return
}
o.mOnce.Do(func() { close(o.mAtLock) })
select {
case <-o.mProceed:
case <-ctx.Done():
}
}}
}
func TestLifecycleExpiryMergeRaceLosesConcurrentTransition(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testLifecycleExpiryMergeRaceLosesConcurrentTransition,
})
}
func testLifecycleExpiryMergeRaceLosesConcurrentTransition(obj ObjectLayer, instanceType, bucket string,
_ http.Handler, _ auth.Credentials, t *testing.T,
) {
// Revision N: a single transition-only rule.
baseXML := []byte(`<LifecycleConfiguration><Rule><ID>keep</ID><Filter><Prefix>data/</Prefix></Filter><Status>Enabled</Status><Transition><Days>30</Days><StorageClass>WARM</StorageClass></Transition></Rule></LifecycleConfiguration>`)
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketLifecycleConfig, baseXML); err != nil {
t.Fatalf("%s: seed lifecycle: %v", instanceType, err)
}
// A replicated expiry-only rule arriving from a peer site.
expXML := `<LifecycleConfiguration><Rule><ID>expire</ID><Filter><Prefix>tmp/</Prefix></Filter><Status>Enabled</Status><Expiration><Days>7</Days></Expiration></Rule></LifecycleConfiguration>`
expLCConfig := base64.StdEncoding.EncodeToString([]byte(expXML))
previousObjectAPI := newObjectLayerFn()
barrier := &lcMergeBarrier{
ObjectLayer: obj,
bucket: bucket,
mAtLock: make(chan struct{}),
mProceed: make(chan struct{}),
}
setObjectLayer(barrier)
defer setObjectLayer(previousObjectAPI)
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
mCtx := context.WithValue(ctx, lcMergeWriterKey{}, "M")
mDone := make(chan error, 1)
go func() {
mDone <- globalSiteReplicationSys.PeerBucketLCConfigHandler(mCtx, bucket, &expLCConfig, UTCNow())
}()
// Wait until the merge writer has read revision N and is about to persist.
select {
case <-barrier.mAtLock:
case err := <-mDone:
t.Fatalf("%s: merge writer finished before persisting: %v", instanceType, err)
case <-ctx.Done():
t.Fatalf("%s: merge writer never reached metadata.lock: %v", instanceType, ctx.Err())
}
// Concurrent local lifecycle transition change commits revision N+1.
concurrentXML := []byte(`<LifecycleConfiguration><Rule><ID>keep</ID><Filter><Prefix>data/</Prefix></Filter><Status>Enabled</Status><Transition><Days>10</Days><StorageClass>COLD</StorageClass></Transition></Rule></LifecycleConfiguration>`)
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketLifecycleConfig, concurrentXML); err != nil {
t.Fatalf("%s: concurrent transition update: %v", instanceType, err)
}
// Release the merge write so it lands after the concurrent commit.
close(barrier.mProceed)
if err := <-mDone; err != nil {
t.Fatalf("%s: merge writer failed: %v", instanceType, err)
}
// The persisted lifecycle must contain both the replicated expiry rule and
// the concurrent transition change.
cfg, _, err := globalBucketMetadataSys.GetLifecycleConfig(bucket)
if err != nil {
t.Fatalf("%s: read merged lifecycle: %v", instanceType, err)
}
var keep, expire bool
for i := range cfg.Rules {
switch cfg.Rules[i].ID {
case "keep":
keep = true
if cfg.Rules[i].Transition.Days != 10 || cfg.Rules[i].Transition.StorageClass != "COLD" {
t.Fatalf("%s: lifecycle merge overwrote the concurrent transition change: got Days=%d StorageClass=%q, want Days=10 StorageClass=COLD",
instanceType, cfg.Rules[i].Transition.Days, cfg.Rules[i].Transition.StorageClass)
}
case "expire":
expire = true
}
}
if !expire {
t.Fatalf("%s: merged lifecycle dropped the replicated expiry rule: %+v", instanceType, cfg.Rules)
}
if !keep {
t.Fatalf("%s: merged lifecycle dropped the transition rule entirely: %+v", instanceType, cfg.Rules)
}
}
// ---------------------------------------------------------------------------
// Target 2 (issue #105): DeleteBucket racing an in-flight metadata writer must
// not resurrect a ghost .metadata.bin record.
//
// Before the fix, erasureServerPools.DeleteBucket took only <bucket>.lck and
// purged the metadata prefix while config writers (updateAndParse) took only
// metadata.lock, so a writer already MID-SAVE (holding metadata.lock, past
// saveMetadata's existence recheck) could persist .metadata.bin after the purge.
// DeleteBucket now takes metadata.lock before deleting, so it waits for that
// writer and then purges whatever the writer wrote.
//
// This test isolates the DeleteBucket-lock fix specifically: the writer holds
// metadata.lock and is paused at the .metadata.bin PutObject, so the earlier
// saveMetadata existence recheck cannot save it — only serializing the delete
// behind the writer can. Removing just DeleteBucket's metadata.lock (keeping the
// recheck) therefore makes this test fail. The delete's ACTUAL metadata.lock
// attempt is observed with lockBucketMetadataAcquireHook (its lock is taken
// through the erasureServerPools receiver, invisible to the object-layer
// barrier), so the handshake is deterministic with no timing assumption.
// ---------------------------------------------------------------------------
func TestDeleteBucketResurrectsGhostMetadata(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testDeleteBucketResurrectsGhostMetadata,
})
}
func testDeleteBucketResurrectsGhostMetadata(obj ObjectLayer, instanceType, bucket string,
_ http.Handler, _ auth.Credentials, t *testing.T,
) {
previousObjectAPI := newObjectLayerFn()
// Writer A holds metadata.lock and pauses at the .metadata.bin PutObject,
// i.e. already past saveMetadata's existence recheck and mid-save.
barrier := &metadataRMWBarrierObjectLayer{
ObjectLayer: obj,
bucket: bucket,
aReady: make(chan struct{}),
aRelease: make(chan struct{}),
bLockAttempt: make(chan struct{}),
}
setObjectLayer(barrier)
defer setObjectLayer(previousObjectAPI)
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
aCtx := context.WithValue(ctx, metadataRMWWriterKey{}, "A")
policyJSON := []byte(`{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::` + bucket + `/*"}]}`)
aReleased := sync.OnceFunc(func() { close(barrier.aRelease) })
defer aReleased()
aDone := make(chan error, 1)
go func() {
_, err := globalBucketMetadataSys.Update(aCtx, bucket, bucketPolicyConfig, policyJSON)
aDone <- err
}()
select {
case <-barrier.aReady:
case err := <-aDone:
t.Fatalf("%s: writer A finished before persisting: %v", instanceType, err)
case <-ctx.Done():
t.Fatalf("%s: writer A never reached metadata save: %v", instanceType, ctx.Err())
}
// A now holds metadata.lock mid-save. Observe DeleteBucket's ACTUAL
// metadata.lock attempt via the acquire hook, set only now so A's earlier
// acquisition does not trip it.
delAtLock := make(chan struct{})
var once sync.Once
hook := func(b string) {
if b == bucket {
once.Do(func() { close(delAtLock) })
}
}
lockBucketMetadataAcquireHook.Store(&hook)
defer lockBucketMetadataAcquireHook.Store(nil)
delDone := make(chan error, 1)
go func() {
delDone <- obj.DeleteBucket(ctx, bucket, DeleteBucketOptions{Force: true})
}()
select {
case <-delAtLock:
// Fixed tree: DeleteBucket reached metadata.lock and blocks on A. Release
// A so it finishes its save and unlocks; the delete then acquires the
// lock and purges the record A wrote.
aReleased()
if err := <-aDone; err != nil {
t.Fatalf("%s: writer A save failed while holding metadata.lock: %v", instanceType, err)
}
if err := <-delDone; err != nil {
t.Fatalf("%s: delete bucket: %v", instanceType, err)
}
case err := <-delDone:
// Broken tree: DeleteBucket purged without taking metadata.lock. Release
// A so its mid-save PutObject recreates .metadata.bin (the ghost).
if err != nil {
t.Fatalf("%s: delete bucket: %v", instanceType, err)
}
aReleased()
if err := <-aDone; err != nil {
t.Fatalf("%s: writer A save failed: %v", instanceType, err)
}
}
// After DeleteBucket, no .metadata.bin record may remain on disk.
if meta, err := readBucketMetadata(ctx, obj, bucket); err == nil {
t.Fatalf("%s: ghost .metadata.bin resurrected after DeleteBucket: name=%q created=%s policyLen=%d",
instanceType, meta.Name, meta.Created, len(meta.PolicyConfigJSON))
} else if !errors.Is(err, errConfigNotFound) && !isErrBucketNotFound(err) && !errors.Is(err, errVolumeNotFound) {
t.Fatalf("%s: unexpected error reading deleted bucket metadata: %v", instanceType, err)
}
}
+54
View File
@@ -0,0 +1,54 @@
// Copyright 2026 PGSTY contributors.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"net/http"
"testing"
"time"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/grid"
)
func TestPeerMetadataReloadWithEqualMaximumTimestamp(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testPeerMetadataReloadWithEqualMaximumTimestamp})
}
func testPeerMetadataReloadWithEqualMaximumTimestamp(obj ObjectLayer, instanceType, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
disk, err := loadBucketMetadata(t.Context(), obj, bucket)
if err != nil {
t.Fatal(err)
}
disk.TaggingConfigXML = []byte(`<Tagging><TagSet><Tag><Key>revision</Key><Value>new</Value></Tag></TagSet></Tagging>`)
disk.TaggingConfigUpdatedAt = UTCNow()
// An unrelated configuration has the greatest timestamp in both records.
disk.PolicyConfigUpdatedAt = disk.TaggingConfigUpdatedAt.Add(time.Hour)
if err := disk.Save(t.Context(), obj); err != nil {
t.Fatal(err)
}
resident := disk
resident.TaggingConfigXML = []byte(`<Tagging><TagSet><Tag><Key>revision</Key><Value>old</Value></Tag></TagSet></Tagging>`)
resident.TaggingConfigUpdatedAt = disk.TaggingConfigUpdatedAt.Add(-time.Minute)
if err := resident.parseAllConfigs(t.Context(), obj); err != nil {
t.Fatal(err)
}
if !resident.lastUpdate().Equal(disk.lastUpdate()) {
t.Fatal("fixture must have equal maximum timestamps")
}
globalBucketMetadataSys.Set(bucket, resident)
args := grid.MSS{peerRESTBucket: bucket}
if _, err := (&peerRESTServer{}).LoadBucketMetadataHandler(&args); err != nil {
t.Fatal(err)
}
tagging, _, err := globalBucketMetadataSys.GetTaggingConfig(bucket)
if err != nil {
t.Fatal(err)
}
if got := tagging.ToMap()["revision"]; got != "new" {
t.Errorf("%s: peer reload retained tag %q despite a newer tagging configuration", instanceType, got)
}
}
+330
View File
@@ -0,0 +1,330 @@
// Copyright (c) 2015-2026 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"encoding/json"
"errors"
"sort"
"time"
"github.com/minio/minio-go/v7/pkg/tags"
bucketsse "github.com/minio/minio/internal/bucket/encryption"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
"github.com/minio/minio/internal/bucket/versioning"
"github.com/pgsty/silo-pkg/v3/policy"
)
// Only these fields share the site-replication source-time ordering contract.
// Bulk apply/import process Object Lock before Versioning, whose effective
// document depends on it. Periodic heal retains its existing type order.
var replicatedBucketConfigs = [...]string{
objectLockConfig, bucketVersioningConfig, bucketPolicyConfig,
bucketTaggingConfig, bucketSSEConfig, bucketQuotaConfigFile,
}
func replicatedBucketConfig(meta *BucketMetadata, file string) (*[]byte, *time.Time) {
switch file {
case bucketPolicyConfig:
return &meta.PolicyConfigJSON, &meta.PolicyConfigUpdatedAt
case bucketTaggingConfig:
return &meta.TaggingConfigXML, &meta.TaggingConfigUpdatedAt
case bucketSSEConfig:
return &meta.EncryptionConfigXML, &meta.EncryptionConfigUpdatedAt
case bucketQuotaConfigFile:
return &meta.QuotaConfigJSON, &meta.QuotaConfigUpdatedAt
case bucketVersioningConfig:
return &meta.VersioningConfigXML, &meta.VersioningConfigUpdatedAt
case objectLockConfig:
return &meta.ObjectLockConfigXML, &meta.ObjectLockConfigUpdatedAt
}
return nil, nil
}
// Callers that only need to know whether a file is under the contract must not
// probe replicatedBucketConfig with a throwaway BucketMetadata.
func isReplicatedBucketConfig(file string) bool {
for _, replicated := range replicatedBucketConfigs {
if replicated == file {
return true
}
}
return false
}
func bucketConfigUpdateOnly(file string) bool {
return file == bucketVersioningConfig || file == objectLockConfig
}
// Reuse the persistence rule before comparison, so an accepted Versioning
// event and the document Save actually writes have the same comparison key.
func effectiveBucketVersioning(data []byte, lockEnabled bool) []byte {
if lockEnabled {
config, err := versioning.ParseConfig(bytes.NewReader(data))
if err != nil || !config.Enabled() || config.PrefixesExcluded() {
return enabledBucketVersioningConfig
}
}
return data
}
// A parsed policy still contains map-backed sets with nondeterministic Marshal
// order. Sort every set array recursively, including statements and conditions.
// RawMessage keeps integer values intact; decoding through float64 would not.
func canonicalBucketPolicyJSON(data json.RawMessage) (json.RawMessage, error) {
data = bytes.TrimSpace(data)
if len(data) == 0 {
return nil, nil
}
switch data[0] {
case '{':
var obj map[string]json.RawMessage
if err := json.Unmarshal(data, &obj); err != nil {
return nil, err
}
for key, value := range obj {
var err error
obj[key], err = canonicalBucketPolicyJSON(value)
if err != nil {
return nil, err
}
}
return json.Marshal(obj)
case '[':
var arr []json.RawMessage
if err := json.Unmarshal(data, &arr); err != nil {
return nil, err
}
for i := range arr {
var err error
arr[i], err = canonicalBucketPolicyJSON(arr[i])
if err != nil {
return nil, err
}
}
sort.Slice(arr, func(i, j int) bool { return bytes.Compare(arr[i], arr[j]) < 0 })
return json.Marshal(arr)
default:
var compact bytes.Buffer
if err := json.Compact(&compact, data); err != nil {
return nil, err
}
return compact.Bytes(), nil
}
}
// Encode the validated policy fields explicitly: BPStatement's required
// Action/Resource tags otherwise try to marshal empty sets for the supported
// NotAction/NotResource alternatives. This stays within the existing schema.
func canonicalBucketPolicy(cfg *policy.BucketPolicy) ([]byte, error) {
if cfg.IsEmpty() {
return nil, nil
}
doc := map[string]any{"Version": cfg.Version}
if cfg.ID != "" {
doc["ID"] = cfg.ID
}
statements := make([]map[string]any, 0, len(cfg.Statements))
for _, st := range cfg.Statements {
statement := map[string]any{"Effect": st.Effect, "Principal": st.Principal}
if st.SID != "" {
statement["Sid"] = st.SID
}
if len(st.Actions) != 0 {
statement["Action"] = st.Actions
}
if len(st.NotActions) != 0 {
statement["NotAction"] = st.NotActions
}
if len(st.Resources) != 0 {
statement["Resource"] = st.Resources
}
if len(st.NotResources) != 0 {
statement["NotResource"] = st.NotResources
}
if len(st.Conditions) != 0 {
statement["Condition"] = st.Conditions
}
statements = append(statements, statement)
}
doc["Statement"] = statements
data, err := json.Marshal(doc)
if err != nil {
return nil, err
}
return canonicalBucketPolicyJSON(data)
}
// Validate with the same parsers as Save. Policy's established empty-policy
// semantics are deletion; a parsed zero quota is still a live document.
func bucketConfigPayload(bucket, file string, data []byte, lockEnabled bool) ([]byte, []byte, error) {
if len(data) == 0 {
return nil, nil, nil
}
var err error
key := data
switch file {
case bucketPolicyConfig:
var cfg *policy.BucketPolicy
cfg, err = policy.ParseBucketPolicyConfig(bytes.NewReader(data), bucket)
if err == nil {
if cfg.IsEmpty() {
return nil, nil, nil
}
key, err = canonicalBucketPolicy(cfg)
}
case bucketQuotaConfigFile:
cfg, parseErr := parseBucketQuota(bucket, data)
err = parseErr
if err == nil {
key, err = json.Marshal(cfg)
}
case bucketTaggingConfig:
_, err = tags.ParseBucketXML(bytes.NewReader(data))
case bucketSSEConfig:
_, err = bucketsse.ParseBucketSSEConfig(bytes.NewReader(data))
case objectLockConfig:
_, err = objectlock.ParseObjectLockConfig(bytes.NewReader(data))
case bucketVersioningConfig:
data = effectiveBucketVersioning(data, lockEnabled)
key = data
_, err = versioning.ParseConfig(bytes.NewReader(data))
}
return data, key, err
}
type bucketConfigState struct {
data, key []byte
at time.Time
real, valid bool
}
func newBucketConfigState(bucket, file string, data []byte, at, created time.Time, lockEnabled bool) (bucketConfigState, error) {
data, key, err := bucketConfigPayload(bucket, file, data, lockEnabled)
if err != nil {
return bucketConfigState{}, err
}
if at.IsZero() {
at = created
}
valid := !created.IsZero() && !at.Before(created)
modified := valid && at.After(created)
if bucketConfigUpdateOnly(file) && len(data) == 0 {
modified = false
}
return bucketConfigState{data: data, key: key, at: at, real: modified, valid: valid}, nil
}
func (s bucketConfigState) candidate() bool {
return s.valid && (s.real || len(s.data) != 0)
}
func compareBucketConfigStates(a, b bucketConfigState) int {
if a.valid != b.valid {
if a.valid {
return 1
}
return -1
}
if a.real != b.real {
if a.real {
return 1
}
return -1
}
if a.real {
if n := a.at.Compare(b.at); n != 0 {
return n
}
// At equal source time a real deletion wins, preventing resurrection.
if (len(a.data) == 0) != (len(b.data) == 0) {
if len(a.data) == 0 {
return 1
}
return -1
}
}
return bytes.Compare(a.key, b.key)
}
func localBucketConfigUpdatedAt(meta BucketMetadata, file string, now time.Time) time.Time {
_, at := replicatedBucketConfig(&meta, file)
for _, lower := range []time.Time{meta.Created, *at} {
if !now.After(lower) {
now = lower.Add(time.Nanosecond)
}
}
return now.UTC()
}
func ensureBucketMetadataCreated(ctx context.Context, obj ObjectLayer, meta *BucketMetadata) error {
if !meta.Created.IsZero() {
return nil
}
info, err := obj.GetBucketInfo(ctx, meta.Name, BucketOptions{NoMetadata: true})
if err != nil {
return err
}
if info.Created.IsZero() {
return errors.New("bucket metadata creation time is unknown")
}
meta.Created = info.Created.UTC()
return nil
}
// applyBucketConfig runs under metadata.lock, on freshly loaded metadata. It
// changes only the selected field; the caller persists once after all checks.
func applyBucketConfig(meta *BucketMetadata, file string, data []byte, at time.Time) (bool, error) {
if bucketConfigUpdateOnly(file) && len(data) == 0 {
return false, nil
}
current, currentAt := replicatedBucketConfig(meta, file)
lockEnabled := len(meta.ObjectLockConfigXML) != 0
incoming, err := newBucketConfigState(meta.Name, file, data, at, meta.Created, lockEnabled)
if err != nil {
return false, err
}
if !incoming.candidate() {
return false, nil
}
local, err := newBucketConfigState(meta.Name, file, *current, *currentAt, meta.Created, lockEnabled)
if err != nil {
return false, err
}
if compareBucketConfigStates(incoming, local) <= 0 {
return false, nil
}
*current, *currentAt = bytes.Clone(incoming.data), incoming.at.UTC()
return true, nil
}
func rebaseBucketConfigDefaults(meta *BucketMetadata, oldCreated time.Time) {
if meta.Created.Equal(oldCreated) {
return
}
// These six fields alone use Created to distinguish a baseline from a
// tombstone. Preserve actual source times when adopting an existing bucket.
for _, file := range replicatedBucketConfigs {
_, at := replicatedBucketConfig(meta, file)
if at.IsZero() || at.Equal(oldCreated) {
*at = meta.Created
}
}
}
+160 -51
View File
@@ -24,6 +24,7 @@ import (
"fmt"
"math/rand"
"sync"
"sync/atomic"
"time"
"github.com/minio/madmin-go/v3"
@@ -125,19 +126,42 @@ func (sys *BucketMetadataSys) Set(bucket string, meta BucketMetadata) {
}
}
func (sys *BucketMetadataSys) updateAndParse(ctx context.Context, bucket string, configFile string, configData []byte, parse, lifecycleDelete bool) (updatedAt time.Time, err error) {
// bucketMetadataUpdate returns the committed snapshot to the caller. meta and
// updatedAt hold the saved state only when changed is true. Local writes always
// change state, because localBucketConfigUpdatedAt is strictly greater than the
// current field time, so their handlers can broadcast meta without rechecking.
type bucketMetadataUpdate struct {
meta BucketMetadata
updatedAt time.Time
changed bool
}
func (sys *BucketMetadataSys) updateAndParse(ctx context.Context, bucket, configFile string, configData []byte, parse, lifecycleDelete bool) (time.Time, error) {
result, err := sys.updateAndParseMetadata(ctx, bucket, configFile, configData, parse, lifecycleDelete, nil)
return result.updatedAt, err
}
func (sys *BucketMetadataSys) updateAndParseMetadata(ctx context.Context, bucket string, configFile string, configData []byte, parse, lifecycleDelete bool, sourceTime *time.Time) (result bucketMetadataUpdate, err error) {
objAPI := newObjectLayerFn()
if objAPI == nil {
return updatedAt, errServerNotInitialized
return result, errServerNotInitialized
}
if isMinioMetaBucketName(bucket) {
return updatedAt, errInvalidArgument
return result, errInvalidArgument
}
// Load deletions without parsed caches (notably quota), and compare the
// six replicated fields against the raw document under the same lock.
if isReplicatedBucketConfig(configFile) {
parse = false
if bucketConfigUpdateOnly(configFile) && len(configData) == 0 {
return result, nil
}
}
notifyCtx := ctx
ctx, unlock, err := lockBucketMetadata(ctx, objAPI, bucket)
if err != nil {
return updatedAt, err
return result, err
}
err = func() error {
@@ -157,55 +181,73 @@ func (sys *BucketMetadataSys) updateAndParse(ctx context.Context, bucket string,
return err
}
}
updatedAt = UTCNow()
switch configFile {
case bucketPolicyConfig:
meta.PolicyConfigJSON = configData
meta.PolicyConfigUpdatedAt = updatedAt
case bucketNotificationConfig:
meta.NotificationConfigXML = configData
meta.NotificationConfigUpdatedAt = updatedAt
case bucketLifecycleConfig:
meta.LifecycleConfigXML = configData
meta.LifecycleConfigUpdatedAt = updatedAt
case bucketSSEConfig:
meta.EncryptionConfigXML = configData
meta.EncryptionConfigUpdatedAt = updatedAt
case bucketTaggingConfig:
meta.TaggingConfigXML = configData
meta.TaggingConfigUpdatedAt = updatedAt
case bucketQuotaConfigFile:
meta.QuotaConfigJSON = configData
meta.QuotaConfigUpdatedAt = updatedAt
case objectLockConfig:
meta.ObjectLockConfigXML = configData
meta.ObjectLockConfigUpdatedAt = updatedAt
case bucketVersioningConfig:
meta.VersioningConfigXML = configData
meta.VersioningConfigUpdatedAt = updatedAt
case bucketReplicationConfig:
meta.ReplicationConfigXML = configData
meta.ReplicationConfigUpdatedAt = updatedAt
case bucketTargetsFile:
meta.BucketTargetsConfigJSON, meta.BucketTargetsConfigMetaJSON, err = encryptBucketMetadata(ctx, meta.Name, configData, kms.Context{
bucket: meta.Name,
bucketTargetsFile: bucketTargetsFile,
})
if err != nil {
return fmt.Errorf("Error encrypting bucket target metadata %w", err)
updatedAt := UTCNow()
if isReplicatedBucketConfig(configFile) {
if err := ensureBucketMetadataCreated(ctx, objAPI, &meta); err != nil {
var at time.Time
if sourceTime != nil {
at = *sourceTime
}
logBucketConfigReplication(ctx, bucket, configFile, "indeterminate", at, meta.Created, err.Error())
return err
}
if sourceTime == nil || sourceTime.IsZero() {
updatedAt = localBucketConfigUpdatedAt(meta, configFile, updatedAt)
if sourceTime != nil {
logBucketConfigReplication(ctx, bucket, configFile, "legacy-zero", *sourceTime, meta.Created, "assigned local source time")
}
} else {
updatedAt = sourceTime.UTC()
}
if updatedAt.Before(meta.Created) {
logBucketConfigReplication(ctx, bucket, configFile, "before-created", updatedAt, meta.Created, "peer event")
return nil
}
changed, err := applyBucketConfig(&meta, configFile, configData, updatedAt)
if err != nil {
return err
}
if !changed {
return nil
}
} else {
switch configFile {
case bucketNotificationConfig:
meta.NotificationConfigXML = configData
meta.NotificationConfigUpdatedAt = updatedAt
case bucketLifecycleConfig:
meta.LifecycleConfigXML = configData
meta.LifecycleConfigUpdatedAt = updatedAt
case bucketReplicationConfig:
meta.ReplicationConfigXML = configData
meta.ReplicationConfigUpdatedAt = updatedAt
case bucketTargetsFile:
meta.BucketTargetsConfigJSON, meta.BucketTargetsConfigMetaJSON, err = encryptBucketMetadata(ctx, meta.Name, configData, kms.Context{
bucket: meta.Name,
bucketTargetsFile: bucketTargetsFile,
})
if err != nil {
return fmt.Errorf("Error encrypting bucket target metadata %w", err)
}
meta.BucketTargetsConfigUpdatedAt = updatedAt
meta.BucketTargetsConfigMetaUpdatedAt = updatedAt
default:
return fmt.Errorf("Unknown bucket %s metadata update requested %s", bucket, configFile)
}
meta.BucketTargetsConfigUpdatedAt = updatedAt
meta.BucketTargetsConfigMetaUpdatedAt = updatedAt
default:
return fmt.Errorf("Unknown bucket %s metadata update requested %s", bucket, configFile)
}
return sys.saveMetadata(ctx, objAPI, meta)
if err := sys.saveMetadata(ctx, objAPI, &meta); err != nil {
return err
}
result = bucketMetadataUpdate{meta: meta, updatedAt: updatedAt, changed: true}
return nil
}()
if err != nil {
return updatedAt, err
return result, err
}
globalNotificationSys.LoadBucketMetadata(bgContext(notifyCtx), bucket) // Do not use caller context here
return updatedAt, nil
if result.changed {
globalNotificationSys.LoadBucketMetadata(bgContext(notifyCtx), bucket)
}
return result, nil
}
func (sys *BucketMetadataSys) save(ctx context.Context, meta BucketMetadata) error {
@@ -218,7 +260,7 @@ func (sys *BucketMetadataSys) save(ctx context.Context, meta BucketMetadata) err
return errInvalidArgument
}
if err := sys.saveMetadata(ctx, objAPI, meta); err != nil {
if err := sys.saveMetadata(ctx, objAPI, &meta); err != nil {
return err
}
@@ -228,11 +270,16 @@ func (sys *BucketMetadataSys) save(ctx context.Context, meta BucketMetadata) err
// saveMetadata persists and publishes metadata locally. Callers performing a
// read-modify-write must hold metadata.lock and release it before peer fan-out.
func (sys *BucketMetadataSys) saveMetadata(ctx context.Context, objAPI ObjectLayer, meta BucketMetadata) error {
func (sys *BucketMetadataSys) saveMetadata(ctx context.Context, objAPI ObjectLayer, meta *BucketMetadata) error {
// A writer may have queued for metadata.lock before DeleteBucket completed.
// Recheck the physical bucket under that lock, before recreating metadata.
if _, err := objAPI.GetBucketInfo(ctx, meta.Name, BucketOptions{NoMetadata: true}); err != nil {
return err
}
if err := meta.Save(ctx, objAPI); err != nil {
return err
}
sys.Set(meta.Name, meta)
sys.Set(meta.Name, *meta)
return nil
}
@@ -240,8 +287,19 @@ func lockBucketMetadata(ctx context.Context, objectAPI ObjectLayer, bucket strin
return lockBucketMetadataWithTimeout(ctx, objectAPI, bucket, globalOperationTimeout)
}
// lockBucketMetadataAcquireHook, when set, is invoked at the start of every
// metadata.lock acquisition, immediately before the blocking Lock() call. It is
// nil in production (a single atomic load, no behavior change) and exists only
// so tests can deterministically observe a caller reaching the metadata lock —
// notably DeleteBucket, whose lock is taken through its erasureServerPools
// receiver and is therefore invisible to an injected object layer.
var lockBucketMetadataAcquireHook atomic.Pointer[func(bucket string)]
func lockBucketMetadataWithTimeout(ctx context.Context, objectAPI ObjectLayer, bucket string, timeout *dynamicTimeout) (context.Context, func(), error) {
lock := objectAPI.NewNSLock(minioMetaBucket, pathJoin(bucketMetaPrefix, bucket, "metadata.lock"))
if hook := lockBucketMetadataAcquireHook.Load(); hook != nil {
(*hook)(bucket)
}
lkctx, err := lock.GetLock(ctx, timeout)
if err != nil {
return nil, nil, err
@@ -292,6 +350,57 @@ func (sys *BucketMetadataSys) Update(ctx context.Context, bucket string, configF
return sys.updateAndParse(ctx, bucket, configFile, configData, true, false)
}
// UpdateExpiryLCConfig merges a replicated ILM expiry configuration with the
// bucket's current lifecycle document and persists the merged result while
// holding metadata.lock across the read, merge, and save. The site-replication
// expiry heal and peer-apply paths must use this instead of computing the merge
// from an unlocked GetConfigFromDisk read and then writing it with Update: that
// two-step sequence drops any lifecycle transition change committed in between
// (issue #105). Lock order stays <bucket>.lck -> metadata.lock -> .metadata.bin;
// the merge and save run under metadata.lock and the peer fan-out runs after it
// is released.
func (sys *BucketMetadataSys) UpdateExpiryLCConfig(ctx context.Context, bucket string, expLCConfig *string, updatedAt time.Time) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
if isMinioMetaBucketName(bucket) {
return errInvalidArgument
}
notifyCtx := ctx
ctx, unlock, err := lockBucketMetadata(ctx, objAPI, bucket)
if err != nil {
return err
}
err = func() error {
defer unlock()
meta, err := loadBucketMetadataParse(ctx, objAPI, bucket, true)
if err != nil {
if !globalIsErasure && !globalIsDistErasure && errors.Is(err, errVolumeNotFound) {
// Only single drive mode needs this fallback.
meta = newBucketMetadata(bucket)
} else {
return err
}
}
configData, err := mergeExpiryWithLCConfig(bucket, meta, expLCConfig, updatedAt)
if err != nil {
return err
}
meta.LifecycleConfigXML = configData
meta.LifecycleConfigUpdatedAt = UTCNow()
return sys.saveMetadata(ctx, objAPI, &meta)
}()
if err != nil {
return err
}
globalNotificationSys.LoadBucketMetadata(bgContext(notifyCtx), bucket) // Do not use caller context here
return nil
}
// Get metadata for a bucket.
// If no metadata exists errConfigNotFound is returned and a new metadata is returned.
// Only a shallow copy is returned, so referenced data should not be modified,
+4 -9
View File
@@ -303,6 +303,9 @@ func loadBucketMetadataParseUnderLock(ctx context.Context, objectAPI ObjectLayer
return newBucketMetadata(bucket), fmt.Errorf("%w: %v", errBucketMetadataMigrationLockUnavailable, err)
}
defer unlock()
if _, err := objectAPI.GetBucketInfo(ctx, bucket, BucketOptions{NoMetadata: true}); err != nil {
return newBucketMetadata(bucket), err
}
return loadBucketMetadataParse(ctx, objectAPI, bucket, parse)
}
@@ -385,15 +388,7 @@ func (b *BucketMetadata) parseAllConfigs(ctx context.Context, objectAPI ObjectLa
} else {
b.objectLockConfig = nil
}
if b.objectLockConfig != nil {
// Object Lock requires every object to be versioned. Whatever the lock
// document contains, a suspended or prefix-excluded versioning document
// is replaced by plain Enabled versioning; Save persists the result.
config, versioningErr := versioning.ParseConfig(bytes.NewReader(b.VersioningConfigXML))
if versioningErr != nil || !config.Enabled() || config.PrefixesExcluded() {
b.VersioningConfigXML = enabledBucketVersioningConfig
}
}
b.VersioningConfigXML = effectiveBucketVersioning(b.VersioningConfigXML, b.objectLockConfig != nil)
if len(b.VersioningConfigXML) != 0 {
b.versioningConfig, err = versioning.ParseConfig(bytes.NewReader(b.VersioningConfigXML))
+25
View File
@@ -39,3 +39,28 @@ func TestBucketMetadataCorsRoundTrip(t *testing.T) {
t.Fatalf("CorsConfigUpdatedAt not preserved")
}
}
// A persisted retired extension must not prevent the whole bucket's metadata
// from loading, including unrelated versioning and ordinary lifecycle rules.
func TestBucketMetadataRetiredAccessTiering(t *testing.T) {
meta := newBucketMetadata("retired-access")
meta.LifecycleConfigXML = []byte(`<LifecycleConfiguration><AccessTierQuota>500GiB</AccessTierQuota><Rule><ID>access</ID><Status>Enabled</Status><Filter><Prefix>logs/</Prefix></Filter><AccessTransition><Window>10m</Window><PromoteAfterAccesses>10</PromoteAfterAccesses></AccessTransition></Rule><Rule><ID>ordinary</ID><Status>Enabled</Status><Filter><Prefix>expired/</Prefix></Filter><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>`)
meta.VersioningConfigXML = []byte(`<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Enabled</Status></VersioningConfiguration>`)
data, err := meta.MarshalMsg(nil)
if err != nil {
t.Fatal(err)
}
got := newBucketMetadata(meta.Name)
if _, err := got.UnmarshalMsg(data); err != nil {
t.Fatal(err)
}
if err := got.parseAllConfigs(t.Context(), nil); err != nil {
t.Fatalf("bucket metadata failed to load: %v", err)
}
if got.lifecycleConfig == nil || got.lifecycleConfig.HasActiveRules("logs/") || !got.lifecycleConfig.HasActiveRules("expired/") {
t.Fatal("unexpected lifecycle behavior")
}
if got.versioningConfig == nil || !got.versioningConfig.Enabled() {
t.Fatal("unrelated versioning lost")
}
}
+125 -10
View File
@@ -25,6 +25,7 @@ import (
"strings"
"time"
"github.com/minio/minio/internal/amztime"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
xhttp "github.com/minio/minio/internal/http"
@@ -333,11 +334,10 @@ func checkPutObjectLockAllowed(ctx context.Context, rq *http.Request, bucket, ob
return mode, retainDate, legalHold, ErrObjectLocked
}
if !legalHoldRequested && retentionCfg.LockEnabled {
// inherit retention from bucket configuration
return retentionCfg.Mode, objectlock.RetentionDate{Time: t.Add(retentionCfg.Validity)}, legalHold, ErrNone
}
return "", objectlock.RetentionDate{}, legalHold, ErrNone
// Inherit retention from the bucket configuration. A legal-hold header
// on the same request, ON or OFF, is independent of retention and must
// not suppress the default (#165).
return retentionCfg.Mode, objectlock.RetentionDate{Time: t.Add(retentionCfg.Validity)}, legalHold, ErrNone
}
return mode, retainDate, legalHold, ErrNone
}
@@ -386,22 +386,137 @@ func (s objectLockState) legalHoldIsOlderThan(src time.Time) bool {
// restoreRetention and restoreLegalHold put the stored state back into
// metadata that was rebuilt from a request whose update was not applied.
func (s objectLockState) restoreRetention(metadata map[string]string) {
// The stored timestamp orders the next update and must survive even when
// the stored value is empty, which is how a removal is recorded.
if s.retentionTimestamp != "" {
metadata[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp] = s.retentionTimestamp
}
if s.mode == "" {
return
}
metadata[strings.ToLower(xhttp.AmzObjectLockMode)] = s.mode
metadata[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = s.retainUntil
if s.retentionTimestamp != "" {
metadata[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp] = s.retentionTimestamp
}
}
func (s objectLockState) restoreLegalHold(metadata map[string]string) {
if s.legalHoldTimestamp != "" {
metadata[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp] = s.legalHoldTimestamp
}
if s.legalHold == "" {
return
}
metadata[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = s.legalHold
if s.legalHoldTimestamp != "" {
metadata[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp] = s.legalHoldTimestamp
}
// replicaStoredLock reads the Object Lock state stored on the addressed version
// so a trusted replica write can order its update against it. A missing object
// or version yields an empty state, which is correct for the first write of a
// version; any other read error is returned so the caller fails the write rather
// than ordering an incoming update against lock state it merely failed to read
// (an older incoming value must not win over a newer stored one just because the
// read timed out).
func replicaStoredLock(ctx context.Context, getObjectInfo GetObjectInfoFn, bucket, object, versionID string) (objectLockState, error) {
oi, err := getObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: versionID})
switch {
case err == nil:
return storedObjectLockState(oi.UserDefined), nil
case isErrObjectNotFound(err) || isErrVersionNotFound(err):
return objectLockState{}, nil
default:
return objectLockState{}, err
}
}
// applyReplicatedObjectLock writes the retention and legal-hold decision into
// metadata for a PUT, CopyObject, or multipart-initiation request. A request
// that is not an actual trusted replica -- a normal user write, or a trusted
// peer that carried the replication marker without REPLICA status -- takes
// ordinary write semantics: a validated value is applied and stamped now, and a
// missing value is left as is. Only an actual replica update is ordered against
// the state already stored on the addressed version, so a stale value cannot
// overwrite a newer one and a full retransmit cannot roll a destination back.
// The stored argument is meaningful only for a replica; callers pass an empty
// state otherwise. Only the two Object Lock keys and their reserved ordering
// timestamps are touched; any encryption-metadata reconciliation stays with the
// caller.
func applyReplicatedObjectLock(metadata map[string]string, stored objectLockState,
replicaTrusted bool,
retentionMode objectlock.RetMode, retentionDate objectlock.RetentionDate,
legalHold objectlock.ObjectLegalHold, srcRetentionTimestamp, srcLegalholdTimestamp time.Time,
) {
switch {
case !replicaTrusted:
// Ordinary write semantics: apply a validated retention and stamp it now;
// a missing value carries no instruction, so leave the metadata as it is.
if retentionMode.Valid() {
metadata[strings.ToLower(xhttp.AmzObjectLockMode)] = string(retentionMode)
metadata[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(retentionDate.UTC())
metadata[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp] = UTCNow().Format(time.RFC3339Nano)
}
case !stored.retentionIsOlderThan(srcRetentionTimestamp):
// The stored update is at least as new as this replica's, or the replica
// carries no ordering timestamp: keep what is stored. This is also how a
// stale retransmit is rejected.
stored.restoreRetention(metadata)
default:
// The replica update wins. A removal carries no value but still records
// the source timestamp that orders it.
if retentionMode.Valid() {
metadata[strings.ToLower(xhttp.AmzObjectLockMode)] = string(retentionMode)
metadata[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(retentionDate.UTC())
}
metadata[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp] = srcRetentionTimestamp.UTC().Format(time.RFC3339Nano)
}
// Legal hold has no removal in S3: an explicitly empty header is already
// rejected as an invalid status, so the only value-less shape that gets here
// is an absent one, which conveys no legal-hold change. Only a valid status
// can win.
switch {
case !replicaTrusted:
if legalHold.Status.Valid() {
metadata[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = string(legalHold.Status)
metadata[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp] = UTCNow().Format(time.RFC3339Nano)
}
case legalHold.Status.Valid() && stored.legalHoldIsOlderThan(srcLegalholdTimestamp):
metadata[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = string(legalHold.Status)
metadata[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp] = srcLegalholdTimestamp.UTC().Format(time.RFC3339Nano)
default:
stored.restoreLegalHold(metadata)
}
}
// reconcileStoredObjectLock re-orders the Object Lock already written into
// metadata against the state currently stored on the destination version, both
// compared by their reserved ordering timestamps. It runs inside the object
// layer under the namespace write lock that guards the version replacement,
// after the destination version is read and before the new one is committed, so
// a replica update whose ordering was decided at handler time (or, for multipart,
// at initiation) cannot overwrite a newer lock update that reached the version in
// between. metadata already carries the incoming update with its source
// timestamps; a stored value that is not older than the incoming one is put back,
// which for a stored removal means clearing the incoming value and keeping only
// the removal's timestamp. Only the two lock keys and their reserved timestamps
// move; a non-replica write never sets the flag that invokes this.
func reconcileStoredObjectLock(metadata map[string]string, stored objectLockState) {
incoming := storedObjectLockState(metadata)
incomingRetentionTS, _ := time.Parse(time.RFC3339Nano, incoming.retentionTimestamp)
if !stored.retentionIsOlderThan(incomingRetentionTS) {
// The stored retention is at least as new as the incoming one (or the
// incoming update is unordered): drop the incoming value and put the stored
// state back, which may itself be a removal (value keys absent, timestamp
// present).
delete(metadata, strings.ToLower(xhttp.AmzObjectLockMode))
delete(metadata, strings.ToLower(xhttp.AmzObjectLockRetainUntilDate))
delete(metadata, ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp)
stored.restoreRetention(metadata)
}
incomingLegalHoldTS, _ := time.Parse(time.RFC3339Nano, incoming.legalHoldTimestamp)
if incoming.legalHold == "" || !stored.legalHoldIsOlderThan(incomingLegalHoldTS) {
delete(metadata, strings.ToLower(xhttp.AmzObjectLockLegalHold))
delete(metadata, ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp)
stored.restoreLegalHold(metadata)
}
}
+5 -6
View File
@@ -19,7 +19,6 @@ package cmd
import (
"bytes"
"encoding/json"
"io"
"net/http"
@@ -100,13 +99,13 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
return
}
configData, err := json.Marshal(bucketPolicy)
configData, err := canonicalBucketPolicy(bucketPolicy)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
updatedAt, err := globalBucketMetadataSys.Update(ctx, bucket, bucketPolicyConfig, configData)
result, err := globalBucketMetadataSys.updateAndParseMetadata(ctx, bucket, bucketPolicyConfig, configData, false, false, nil)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -116,8 +115,8 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
replLogIf(ctx, globalSiteReplicationSys.BucketMetaHook(ctx, madmin.SRBucketMeta{
Type: madmin.SRBucketMetaTypePolicy,
Bucket: bucket,
Policy: bucketPolicyBytes,
UpdatedAt: updatedAt,
Policy: result.meta.PolicyConfigJSON,
UpdatedAt: result.updatedAt,
}))
// Success.
@@ -200,7 +199,7 @@ func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *ht
return
}
configData, err := json.Marshal(config)
configData, err := canonicalBucketPolicy(config)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
+26 -7
View File
@@ -255,13 +255,32 @@ func getConditionValuesWithTags(r *http.Request, lc string, cred auth.Credential
}
cloneHeader := r.Header.Clone()
signatureAge := cloneHeader.Get("x-amz-signature-age")
cloneHeader.Del("x-amz-signature-age")
// The presigned V4 verifier overwrites this internal scratch header after
// validating the signature. Ignore a value supplied on every other request
// type, where it would otherwise synthesize s3:signatureAge.
if authType == authTypePresigned && signatureAge != "" {
args["signatureAge"] = []string{signatureAge}
// s3:signatureAge is derived from the presigned X-Amz-Date rather than from
// anything the verifier writes back: PutObject and UploadPart authorize
// before they verify the signature, so a post-verification value is not yet
// available on the first evaluation. The date is bound by the signature
// (doesPresignedSignatureMatch rebuilds and compares it), so a forged date
// only changes the authorization outcome of a request that then fails
// verification. A date that does not parse leaves the key absent; the
// verifier rejects the request as ErrMalformedPresignedDate.
if authType == authTypePresigned {
if signedDate, err := time.Parse(iso8601Format, r.Form.Get(xhttp.AmzDate)); err == nil {
args["signatureAge"] = []string{strconv.FormatInt(currTime.Sub(signedDate).Milliseconds(), 10)}
}
}
// s3:x-amz-content-sha256 must name the payload hash the request is actually
// verified and enforced against, and only one such value. Presence of the
// header controls whether the key exists at all (AWS documents that the
// query-string form does not populate it), but the value comes from the same
// selection getContentSha256Cksum makes for verification: the presigned query
// value takes precedence over the header, and a repeated header contributes
// only its first value. Exposing every raw header value instead let a
// request satisfy a policy with a value the verifier never checked.
if _, ok := cloneHeader[xhttp.AmzContentSha256]; ok {
args[xhttp.AmzContentSha256] = []string{getContentSha256Cksum(r, serviceS3)}
cloneHeader.Del(xhttp.AmzContentSha256)
}
userTags := cloneHeader.Get(xhttp.AmzObjectTagging)
+34 -6
View File
@@ -23,8 +23,10 @@ import (
"net/url"
"os"
"slices"
"strconv"
"strings"
"testing"
"time"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/handlers"
@@ -579,8 +581,20 @@ func TestGetConditionValuesRejectsAbsentInternalKeys(t *testing.T) {
}
}
// s3:signatureAge is derived from the presigned X-Amz-Date, which the signature
// binds. A client header under the former scratch name must never supply it on
// any auth type, and a presign whose date is missing or malformed leaves the
// key absent (the verifier then rejects the request).
func TestGetConditionValuesOnlyAcceptsPresignedSignatureAge(t *testing.T) {
const signatureAgeHeader = "x-amz-signature-age"
signedDate := UTCNow().Add(-90 * time.Second)
presignQuery := func(date string) string {
q := url.Values{xhttp.AmzCredential: {"access/20260803/us-east-1/s3/aws4_request"}}
if date != "" {
q.Set(xhttp.AmzDate, date)
}
return "http://minio.local/bkt/obj?" + q.Encode()
}
for _, tc := range []struct {
name string
@@ -602,19 +616,33 @@ func TestGetConditionValuesOnlyAcceptsPresignedSignatureAge(t *testing.T) {
},
},
{
name: "presigned verifier value",
target: "http://minio.local/bkt/obj?" + url.Values{
xhttp.AmzCredential: {"access/20260803/us-east-1/s3/aws4_request"},
}.Encode(),
name: "presigned client header without date",
target: presignQuery(""),
headers: map[string]string{signatureAgeHeader: "250"},
},
{
name: "presigned malformed date",
target: presignQuery("yesterday"),
},
{
name: "presigned signed date",
target: presignQuery(signedDate.Format(iso8601Format)),
headers: map[string]string{signatureAgeHeader: "250"},
want: true,
},
} {
t.Run(tc.name, func(t *testing.T) {
got := condValuesForRequest(t, tc.target, tc.headers)
_, ok := got["signatureAge"]
v, ok := got["signatureAge"]
if ok != tc.want {
t.Fatalf("signatureAge presence: expected %v, got %v", tc.want, got["signatureAge"])
t.Fatalf("signatureAge presence: expected %v, got %v", tc.want, v)
}
if !tc.want {
return
}
age, err := strconv.ParseInt(strings.Join(v, ""), 10, 64)
if err != nil || age < (90*time.Second).Milliseconds() || age > (2*time.Minute).Milliseconds() {
t.Fatalf("signatureAge = %v, want about 90s derived from X-Amz-Date rather than the client header", v)
}
})
}
+12 -8
View File
@@ -43,6 +43,17 @@ func NewBucketQuotaSys() *BucketQuotaSys {
return &BucketQuotaSys{}
}
// getBucketQuotaSize returns the effective enforced hard-quota size.
func getBucketQuotaSize(quota *madmin.BucketQuota) uint64 {
if quota == nil || quota.Type != madmin.HardQuota {
return 0
}
if quota.Size > 0 {
return quota.Size
}
return quota.Quota
}
var bucketStorageCache = cachevalue.New[DataUsageInfo]()
// Init initialize bucket quota.
@@ -110,14 +121,7 @@ func (sys *BucketQuotaSys) enforceQuotaHard(ctx context.Context, bucket string,
return err
}
var quotaSize uint64
if q != nil && q.Type == madmin.HardQuota {
if q.Size > 0 {
quotaSize = q.Size
} else if q.Quota > 0 {
quotaSize = q.Quota
}
}
quotaSize := getBucketQuotaSize(q)
if quotaSize > 0 {
if uint64(size) >= quotaSize { // check if file size already exceeds the quota
return BucketQuotaExceeded{Bucket: bucket}
+77
View File
@@ -0,0 +1,77 @@
// Copyright (c) 2015-2025 MinIO, Inc.
// Copyright (c) 2025-2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"testing"
"github.com/minio/madmin-go/v3"
)
func TestGetBucketQuotaSize(t *testing.T) {
tests := []struct {
name string
quota *madmin.BucketQuota
want uint64
}{
{name: "nil"},
{name: "empty", quota: &madmin.BucketQuota{}},
{name: "current size", quota: &madmin.BucketQuota{Type: madmin.HardQuota, Size: 1024}, want: 1024},
{name: "legacy quota", quota: &madmin.BucketQuota{Type: madmin.HardQuota, Quota: 2048}, want: 2048},
{name: "size takes precedence", quota: &madmin.BucketQuota{Type: madmin.HardQuota, Size: 1024, Quota: 2048}, want: 1024},
{name: "missing type", quota: &madmin.BucketQuota{Size: 1024}},
{name: "unsupported type", quota: &madmin.BucketQuota{Type: "fifo", Size: 1024}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := getBucketQuotaSize(tt.quota); got != tt.want {
t.Fatalf("getBucketQuotaSize() = %d, want %d", got, tt.want)
}
})
}
}
func TestIsBktQuotaCfgReplicated(t *testing.T) {
hardQuota := func(size, legacy uint64) *madmin.BucketQuota {
return &madmin.BucketQuota{Type: madmin.HardQuota, Size: size, Quota: legacy}
}
tests := []struct {
name string
quotas []*madmin.BucketQuota
want bool
}{
{name: "none configured", quotas: []*madmin.BucketQuota{nil, nil}, want: true},
{name: "missing from one site", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), nil}},
{name: "matching size", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), hardQuota(1024, 0)}, want: true},
{name: "different size", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), hardQuota(2048, 0)}},
{name: "equivalent representations", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), hardQuota(0, 1024)}, want: true},
{name: "different typeless size", quotas: []*madmin.BucketQuota{{Size: 1024}, {Size: 2048}}},
{name: "different type", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), {Type: "fifo", Size: 1024}}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isBktQuotaCfgReplicated(len(tt.quotas), tt.quotas); got != tt.want {
t.Fatalf("isBktQuotaCfgReplicated() = %v, want %v", got, tt.want)
}
})
}
}
+7 -4
View File
@@ -418,6 +418,9 @@ func getReplicationState(rinfos replicatedInfos, prevState ReplicationState, vID
for _, rinfo := range rinfos.Targets {
if rinfo.ResyncTimestamp != "" {
if rs.ResetStatusesMap == nil {
rs.ResetStatusesMap = make(map[string]string)
}
rs.ResetStatusesMap[targetResetHeader(rinfo.Arn)] = rinfo.ResyncTimestamp
}
}
@@ -640,10 +643,10 @@ type VersionPurgeStatusType = replication.VersionPurgeStatusType
type replicationResyncer struct {
// map of bucket to their resync status
statusMap map[string]BucketReplicationResyncStatus
workerSize int
resyncCancelCh chan struct{}
workerCh chan struct{}
statusMap map[string]BucketReplicationResyncStatus
workerSize int
cancelResyncs map[resyncOpts]context.CancelCauseFunc
workerCh chan struct{}
sync.RWMutex
}
+565 -176
View File
File diff suppressed because it is too large Load Diff
+788
View File
@@ -18,13 +18,21 @@
package cmd
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"path"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
)
@@ -307,3 +315,783 @@ func TestReplicationValidationObjectUsesRulePrefix(t *testing.T) {
})
}
}
// The resync-finalization tests below exercise the real result sink, the
// finish() shutdown ordering, and the sendResyncResult / finalResyncStatus
// helpers, plus (for the persistence cases) markStatus with on-disk
// round-tripping. resyncBucket cannot be driven end to end in a unit test
// because its workers call a live remote target (StatObject), so the helpers it
// uses are exercised directly. The blocking-order assertions run under
// testing/synctest so a removed wait fails deterministically, with no timing
// windows.
func newTestResyncer(bucket, arn string) (*replicationResyncer, resyncOpts) {
s := &replicationResyncer{
statusMap: map[string]BucketReplicationResyncStatus{},
}
brs := newBucketResyncStatus(bucket)
brs.TargetsMap[arn] = TargetReplicationResyncStatus{ResyncStatus: ResyncStarted, ResyncID: "reset-" + bucket}
s.statusMap[bucket] = brs
return s, resyncOpts{bucket: bucket, arn: arn, resyncID: "reset-" + bucket}
}
// TestResyncBucketFinalize round-trips the terminal status through a real
// ObjectLayer: a clean run persists Completed with every result, while a run
// whose parent context was canceled during the drain is downgraded to Failed;
// a user-canceled run persists Canceled. Completed never misrepresents an
// incomplete resync.
func TestResyncBucketFinalize(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
objAPI, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatalf("prepare erasure backend: %v", err)
}
defer removeRoots(fsDirs)
// persistTerminal applies resyncBucket's finalizer logic (finalResyncStatus
// then markStatus, which persists) and reads the status back the way the
// resync status API does.
persistTerminal := func(t *testing.T, s *replicationResyncer, opts resyncOpts, status ResyncStatusType, cause error) TargetReplicationResyncStatus {
t.Helper()
s.markStatus(finalResyncStatus(status, cause), opts, objAPI)
brs, err := loadBucketResyncMetadata(ctx, opts.bucket, objAPI)
if err != nil {
t.Fatalf("load persisted resync metadata: %v", err)
}
return brs.TargetsMap[opts.arn]
}
// 1. Clean completion: every result - including the failed object - is folded
// into the persisted status, which stays Completed.
t.Run("persists complete counts", func(t *testing.T) {
s, opts := newTestResyncer("finalize-counts", "arn1")
results := s.newResyncResults(opts)
results.ch <- TargetReplicationResyncStatus{Object: "ok-1", ReplicatedCount: 1, ReplicatedSize: 100}
results.ch <- TargetReplicationResyncStatus{Object: "ok-2", ReplicatedCount: 1, ReplicatedSize: 200}
results.ch <- TargetReplicationResyncStatus{Object: "bad", FailedCount: 1, FailedSize: 300}
var wg sync.WaitGroup // no producer workers for this case
results.finish(nil, &wg)
st := persistTerminal(t, s, opts, ResyncCompleted, nil)
if st.ResyncStatus != ResyncCompleted {
t.Fatalf("persisted status = %s, want Completed", st.ResyncStatus)
}
if st.ReplicatedCount != 2 || st.ReplicatedSize != 300 || st.FailedCount != 1 || st.FailedSize != 300 {
t.Fatalf("persisted counts = {replicated:%d/%d failed:%d/%d}, want {2/300 1/300}",
st.ReplicatedCount, st.ReplicatedSize, st.FailedCount, st.FailedSize)
}
})
// 2. Parent context canceled during the drain -> Completed downgraded to
// Failed (markStatus persists under its own context, so nothing else stops
// a bare Completed from being recorded).
t.Run("parent cancel during drain downgrades to failed", func(t *testing.T) {
s, opts := newTestResyncer("finalize-parent-cancel", "arn1")
results := s.newResyncResults(opts)
results.ch <- TargetReplicationResyncStatus{Object: "ok-1", ReplicatedCount: 1, ReplicatedSize: 100}
var wg sync.WaitGroup
results.finish(nil, &wg)
cctx, ccancel := context.WithCancel(context.Background())
ccancel()
st := persistTerminal(t, s, opts, ResyncCompleted, context.Cause(cctx))
if st.ResyncStatus != ResyncFailed {
t.Fatalf("persisted status = %s, want Failed (parent canceled during drain)", st.ResyncStatus)
}
})
// 3. A user-canceled worker cannot report a completed resync.
t.Run("user cancel persists canceled", func(t *testing.T) {
s, opts := newTestResyncer("finalize-worker-abort", "arn1")
ctx, cancel := context.WithCancelCause(context.Background())
cancel(errResyncCanceled)
ch := make(chan TargetReplicationResyncStatus)
if s.sendResyncResult(ctx, ch, TargetReplicationResyncStatus{Object: "dropped", ReplicatedCount: 1}) {
t.Fatal("sendResyncResult reported success after cancellation")
}
st := persistTerminal(t, s, opts, ResyncCompleted, context.Cause(ctx))
if st.ResyncStatus != ResyncCanceled {
t.Fatalf("persisted status = %s, want Canceled", st.ResyncStatus)
}
})
}
// TestResyncFinishDrainsResults asserts finish() does not return until the
// consumer has applied the final result (the #136 defect). A gated apply holds
// the last result unapplied; under synctest finish() must stay durably blocked
// until it is released - if rr.wg.Wait() is removed, finish() returns early and
// the test fails deterministically.
func TestResyncFinishDrainsResults(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
s, opts := newTestResyncer("drain", "arn1")
reachedFinal := make(chan struct{})
release := make(chan struct{})
results := startResyncResults(func(r TargetReplicationResyncStatus) {
if r.Object == "final" {
close(reachedFinal)
<-release
}
s.incStats(r, opts)
})
results.ch <- TargetReplicationResyncStatus{Object: "ok-1", ReplicatedCount: 1, ReplicatedSize: 100}
results.ch <- TargetReplicationResyncStatus{Object: "final", FailedCount: 1, FailedSize: 200}
<-reachedFinal // consumer received "final" but is gated before incStats(final)
var wg sync.WaitGroup
finishDone := make(chan struct{})
go func() {
results.finish(nil, &wg)
close(finishDone)
}()
synctest.Wait()
select {
case <-finishDone:
close(release)
synctest.Wait()
t.Fatal("finish() returned before the final result was drained (drain wait missing)")
default:
// finish() is durably blocked in rr.wg.Wait() - correct.
}
close(release)
synctest.Wait()
<-finishDone
st := s.statusMap[opts.bucket].TargetsMap[opts.arn]
if st.ReplicatedCount != 1 || st.FailedCount != 1 || st.FailedSize != 200 {
t.Fatalf("status after finish = {replicated:%d failed:%d/%d}, want {1 1/200}",
st.ReplicatedCount, st.FailedCount, st.FailedSize)
}
})
}
// TestResyncFinishWaitsForInflightWorker asserts finish() stops the producer
// workers before it closes the result channel, so an in-flight worker (as on an
// early-return path) never sends on a closed channel and its result is not lost.
// A gated worker stays in flight past the shutdown request; under synctest
// finish() must stay durably blocked until the worker is released - if
// workerWg.Wait() is removed, finish() returns early and the test fails
// deterministically.
func TestResyncFinishWaitsForInflightWorker(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
s, opts := newTestResyncer("workers", "arn1")
results := startResyncResults(func(r TargetReplicationResyncStatus) { s.incStats(r, opts) })
workers := []chan ReplicateObjectInfo{make(chan ReplicateObjectInfo, 1)}
var wg sync.WaitGroup
gotRoi := make(chan struct{})
release := make(chan struct{})
wg.Add(1)
go func() {
defer wg.Done()
for roi := range workers[0] {
close(gotRoi)
<-release
// Mirror the real worker's send; recover so that if finish()
// wrongly closed the result channel first, the test fails via the
// assertion below instead of crashing on send-on-closed.
func() {
defer func() { _ = recover() }()
results.ch <- TargetReplicationResyncStatus{Object: roi.Name, ReplicatedCount: 1, ReplicatedSize: 500}
}()
}
}()
workers[0] <- ReplicateObjectInfo{Name: "inflight"}
<-gotRoi // worker holds a result in flight, not yet delivered
finishDone := make(chan struct{})
go func() {
results.finish(workers, &wg)
close(finishDone)
}()
synctest.Wait()
select {
case <-finishDone:
close(release)
synctest.Wait()
t.Fatal("finish() closed the result channel before the in-flight worker finished (worker wait missing)")
default:
// finish() is durably blocked in workerWg.Wait() - correct.
}
close(release)
synctest.Wait()
<-finishDone
st := s.statusMap[opts.bucket].TargetsMap[opts.arn]
if st.ReplicatedCount != 1 || st.ReplicatedSize != 500 {
t.Fatalf("status after finish = {replicated:%d/%d}, want {1/500}", st.ReplicatedCount, st.ReplicatedSize)
}
})
}
// TestResyncResultFor asserts the resync worker classifies a target from the
// actual replication outcome, not from whether the target version merely exists.
// The key regression is the "failed update over an existing version" case: a
// quota-rejected update leaves the old version in place, and counting existence
// (the previous behavior) would score it a success. It also checks a genuine
// success, an errored-but-Completed result, a delete failure, a delete-marker
// success (zero bytes), and an ARN that was never attempted.
func TestResyncResultFor(t *testing.T) {
const arn = "arn:minio:replication::id:bucket"
obj := ReplicateObjectInfo{Name: "obj", Bucket: "bucket", Size: 196608}
deleteMarker := ReplicateObjectInfo{Name: "dm", Bucket: "bucket", Size: 0, DeleteMarker: true}
tests := []struct {
name string
roi ReplicateObjectInfo
rinfos replicatedInfos
wantRepl, wantReplSize, wantFail, wantFailSize int64
}{
{
name: "completed update",
roi: obj,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, ReplicationStatus: replication.Completed, Size: 196608},
}},
wantRepl: 1, wantReplSize: 196608,
},
{
name: "failed update over existing version",
roi: obj,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, ReplicationStatus: replication.Failed, Err: fmt.Errorf("quota exceeded"), Size: 196608},
}},
wantFail: 1, wantFailSize: 196608,
},
{
name: "completed but errored is a failure",
roi: obj,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, ReplicationStatus: replication.Completed, Err: fmt.Errorf("boom"), Size: 196608},
}},
wantFail: 1, wantFailSize: 196608,
},
{
name: "delete failed",
roi: deleteMarker,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, ReplicationStatus: replication.Failed},
}},
wantFail: 1, wantFailSize: 0,
},
{
name: "delete marker replicated counts zero bytes",
roi: deleteMarker,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, ReplicationStatus: replication.Completed},
}},
wantRepl: 1, wantReplSize: 0,
},
{
name: "arn not attempted is a failure",
roi: obj,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: "arn:minio:replication::id2:bucket", ReplicationStatus: replication.Completed, Size: 196608},
}},
wantFail: 1, wantFailSize: 196608,
},
{
name: "completed with zero size falls back to object size",
roi: obj,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, ReplicationStatus: replication.Completed, Size: 0},
}},
wantRepl: 1, wantReplSize: 196608,
},
{
name: "version purge complete is a success",
roi: ReplicateObjectInfo{Name: "purge", Bucket: "bucket", Size: 196608, VersionPurgeStatus: replication.VersionPurgePending},
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
// a successful purge sets only VersionPurgeStatus; ReplicationStatus stays empty.
{Arn: arn, VersionPurgeStatus: replication.VersionPurgeComplete},
}},
wantRepl: 1, wantReplSize: 196608,
},
{
name: "version purge failed is a failure",
roi: ReplicateObjectInfo{Name: "purge", Bucket: "bucket", Size: 196608, VersionPurgeStatus: replication.VersionPurgePending},
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
{Arn: arn, VersionPurgeStatus: replication.VersionPurgeFailed, Err: fmt.Errorf("quota exceeded")},
}},
wantFail: 1, wantFailSize: 196608,
},
{
name: "benign duplicate 412 is a success",
roi: obj,
rinfos: replicatedInfos{Targets: []replicatedTargetInfo{
// the destination answers PreconditionFailed for an exact duplicate;
// replicateAll keeps Completed but retains the error.
{Arn: arn, ReplicationStatus: replication.Completed, Err: minio.ErrorResponse{Code: "PreconditionFailed"}, Size: 196608},
}},
wantRepl: 1, wantReplSize: 196608,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
st := resyncResultFor(tc.rinfos, arn, tc.roi)
if st.Object != tc.roi.Name || st.Bucket != tc.roi.Bucket {
t.Fatalf("object/bucket = %s/%s, want %s/%s", st.Object, st.Bucket, tc.roi.Name, tc.roi.Bucket)
}
if st.ReplicatedCount != tc.wantRepl || st.ReplicatedSize != tc.wantReplSize ||
st.FailedCount != tc.wantFail || st.FailedSize != tc.wantFailSize {
t.Fatalf("resyncResultFor = {replicated:%d/%d failed:%d/%d}, want {%d/%d %d/%d}",
st.ReplicatedCount, st.ReplicatedSize, st.FailedCount, st.FailedSize,
tc.wantRepl, tc.wantReplSize, tc.wantFail, tc.wantFailSize)
}
})
}
}
// TestObjectNeedsResyncForARN asserts the resync dispatch is scoped to the
// target being resynced. The worker pool runs for a single target (opts.arn),
// so an object that only qualifies for a different target must be skipped: with
// A/B rules and a resync of A, an object that needs replication only for B must
// not be admitted to A's worker. Otherwise (after outcome-based classification)
// A would be absent from that object's result and miscounted as an A failure.
func TestObjectNeedsResyncForARN(t *testing.T) {
const (
arnA = "arn:minio:replication::id:bucket"
arnB = "arn:minio:replication::id2:bucket"
)
tests := []struct {
name string
decision ResyncDecision
arn string
want bool
}{
{
name: "target must resync",
decision: ResyncDecision{targets: map[string]ResyncTargetDecision{arnA: {Replicate: true}}},
arn: arnA,
want: true,
},
{
name: "object qualifies for B only, resyncing A",
decision: ResyncDecision{targets: map[string]ResyncTargetDecision{arnB: {Replicate: true}}},
arn: arnA,
want: false,
},
{
name: "A present but not replicating, B replicating, resyncing A",
decision: ResyncDecision{targets: map[string]ResyncTargetDecision{
arnA: {Replicate: false},
arnB: {Replicate: true},
}},
arn: arnA,
want: false,
},
{
name: "object qualifies for both, resyncing A",
decision: ResyncDecision{targets: map[string]ResyncTargetDecision{
arnA: {Replicate: true},
arnB: {Replicate: true},
}},
arn: arnA,
want: true,
},
{
name: "no resync decision",
decision: ResyncDecision{},
arn: arnA,
want: false,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
roi := ReplicateObjectInfo{Name: "obj", Bucket: "bucket", ExistingObjResync: tc.decision}
if got := objectNeedsResyncForARN(roi, tc.arn); got != tc.want {
t.Fatalf("objectNeedsResyncForARN(arn=%s) = %v, want %v", tc.arn, got, tc.want)
}
})
}
}
// newMatchingReplicationPair returns a source/target pair that getReplicationAction must
// classify as replicateNone: same ETag, version id, size, modification time and content
// type. Any action other than replicateNone is therefore attributable to the object lock
// entries a caller adds on top.
func newMatchingReplicationPair() (ObjectInfo, minio.ObjectInfo) {
mtime := time.Date(2026, 9, 5, 10, 0, 0, 0, time.UTC)
size := int64(7)
src := ObjectInfo{
Bucket: "bucket",
Name: "object",
ETag: "d41d8cd98f00b204e9800998ecf8427e",
VersionID: "b0ff1d6e-0000-4000-8000-000000000001",
Size: size,
ActualSize: &size,
ModTime: mtime,
ContentType: "application/octet-stream",
UserDefined: map[string]string{"content-type": "application/octet-stream"},
}
tgt := minio.ObjectInfo{
ETag: src.ETag,
VersionID: src.VersionID,
Size: size,
LastModified: mtime,
ContentType: src.ContentType,
Metadata: http.Header{},
}
return src, tgt
}
// TestGetReplicationActionEmptyObjectLockValues covers the comparison of object lock entries
// whose value is empty. Removing retention from a version stores the mode and retain-until-date
// keys with empty values, while the target's HEAD response omits them entirely, so the two must
// compare equal or the version can never be reported as in sync. Cases 3 and 4 are synthetic
// comparison inputs, since a SILO target cannot return empty lock headers; cases 7 and 8 guard
// against over-normalizing.
func TestGetReplicationActionEmptyObjectLockValues(t *testing.T) {
var (
modeKey = strings.ToLower(xhttp.AmzObjectLockMode)
dateKey = strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)
until = "2026-10-05T10:00:00.000Z"
)
emptyRetention := map[string]string{modeKey: "", dateKey: ""}
realRetention := map[string]string{modeKey: "GOVERNANCE", dateKey: until}
tests := []struct {
name string
srcMeta map[string]string
tgtHdr map[string]string
want replicationAction
}{
{"1-both-clean-never-had-retention", nil, nil, replicateNone},
{"2-source-present-empty-target-absent", emptyRetention, nil, replicateNone},
{"3-source-absent-target-present-empty", nil, emptyRetention, replicateNone},
{"4-both-present-empty", emptyRetention, emptyRetention, replicateNone},
{"5-both-governance-equal", realRetention, realRetention, replicateNone},
{"6-source-governance-target-absent", realRetention, nil, replicateMetadata},
{"7-source-empty-target-real-retention", emptyRetention, realRetention, replicateMetadata},
{"8-empty-user-metadata-is-not-normalized", map[string]string{"x-amz-meta-foo": ""}, nil, replicateMetadata},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
src, tgt := newMatchingReplicationPair()
for k, v := range test.srcMeta {
src.UserDefined[k] = v
}
for k, v := range test.tgtHdr {
tgt.Metadata.Set(k, v)
}
if got := getReplicationAction(src, tgt, replication.HealReplicationType); got != test.want {
t.Fatalf("getReplicationAction() = %q, want %q (source %v, target %v)", got, test.want, src.UserDefined, tgt.Metadata)
}
})
}
}
// TestEmptyRetentionValuesAreOmittedFromObjectResponseHeaders records why the target half of the
// comparison in getReplicationAction can never report an empty object lock entry:
// FilterObjectLockMetadata drops both keys because an empty mode is not a valid retention mode,
// and setObjectHeaders skips them when writing response headers. Neither filter reaches the
// replication wire: the empty entries are still carried by getCopyObjMetadata and sent by the
// metadata CopyObject, which is why the sender's comparison is what has to tolerate them.
// FilterObjectLockMetadata is also applied by CopyObject (cmd/object-handlers.go:1708), where it
// strips the source's lock metadata before the destination re-derives it from the request.
func TestEmptyRetentionValuesAreOmittedFromObjectResponseHeaders(t *testing.T) {
modeKey := strings.ToLower(xhttp.AmzObjectLockMode)
dateKey := strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)
meta := map[string]string{
modeKey: "",
dateKey: "",
"content-type": "application/octet-stream",
}
filtered := objectlock.FilterObjectLockMetadata(meta, false, false)
if _, ok := filtered[modeKey]; ok {
t.Errorf("FilterObjectLockMetadata() kept the empty lock mode key: %v", filtered)
}
if _, ok := filtered[dateKey]; ok {
t.Errorf("FilterObjectLockMetadata() kept the empty retain-until-date key: %v", filtered)
}
rec := httptest.NewRecorder()
if err := setObjectHeaders(t.Context(), rec, ObjectInfo{UserDefined: meta, ModTime: time.Now(), Size: 7}, nil, ObjectOptions{}); err != nil {
t.Fatalf("setObjectHeaders() = %v", err)
}
if v, ok := rec.Header()[http.CanonicalHeaderKey(xhttp.AmzObjectLockMode)]; ok {
t.Errorf("setObjectHeaders() emitted an empty lock mode header: %v", v)
}
if v, ok := rec.Header()[http.CanonicalHeaderKey(xhttp.AmzObjectLockRetainUntilDate)]; ok {
t.Errorf("setObjectHeaders() emitted an empty retain-until-date header: %v", v)
}
}
// fakeRetentionGetter answers GetObjectRetention with a fixed result and counts its calls.
type fakeRetentionGetter struct {
mode *minio.RetentionMode
err error
calls int
}
func (f *fakeRetentionGetter) GetObjectRetention(_ context.Context, _, _, _ string) (*minio.RetentionMode, *time.Time, error) {
f.calls++
return f.mode, nil, f.err
}
func TestRetentionRemovedAtSource(t *testing.T) {
modeKey := strings.ToLower(xhttp.AmzObjectLockMode)
dateKey := strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)
tsKey := ReservedMetadataPrefixLower + ObjectLockRetentionTimestamp
stamp := "2026-09-06T01:00:00Z"
tests := []struct {
name string
meta map[string]string
want bool
}{
{"no lock keys", map[string]string{"content-type": "text/plain"}, false},
{"empty pair", map[string]string{modeKey: "", dateKey: ""}, true},
{"empty mode only", map[string]string{modeKey: ""}, true},
{"empty date only", map[string]string{dateKey: ""}, true},
{"real retention", map[string]string{modeKey: "GOVERNANCE", dateKey: "2026-10-05T10:00:00.000Z"}, false},
{"canonical case", map[string]string{xhttp.AmzObjectLockMode: ""}, true},
{"empty user metadata", map[string]string{"x-amz-meta-foo": ""}, false},
// Representation (2): a replicated removal persists the ordering timestamp alone,
// with the mode and retain-until-date keys absent (restoreRetention).
{"timestamp only, mode absent", map[string]string{tsKey: stamp}, true},
{"timestamp with empty mode", map[string]string{tsKey: stamp, modeKey: ""}, true},
{"timestamp with real retention is a set, not a removal", map[string]string{tsKey: stamp, modeKey: "GOVERNANCE", dateKey: "2026-10-05T10:00:00.000Z"}, false},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if got := retentionRemovedAtSource(ObjectInfo{UserDefined: test.meta}); got != test.want {
t.Fatalf("retentionRemovedAtSource() = %v, want %v", got, test.want)
}
})
}
}
// TestTargetRetentionConfirmedAbsent pins the rule that only an explicit answer from the
// destination clears a removed retention. A denied or unreachable destination must read as still
// holding retention, because HEAD hides a real retention from a credential without
// s3:GetObjectRetention exactly as it hides one that does not exist.
func TestTargetRetentionConfirmedAbsent(t *testing.T) {
governance := minio.Governance
var emptyMode minio.RetentionMode
unknownMode := minio.RetentionMode("ARCHIVE")
tests := []struct {
name string
mode *minio.RetentionMode
err error
want bool
}{
{"version holds governance retention", &governance, nil, false},
{"no retention on the version", nil, minio.ErrorResponse{Code: "NoSuchObjectLockConfiguration"}, true},
{
// The destination also answers this when its own read of the bucket's Object Lock
// configuration fails, so it does not establish that Object Lock is disabled.
"invalid request naming a missing object lock configuration",
nil,
minio.ErrorResponse{Code: "InvalidRequest", Message: "Bucket is missing ObjectLockConfiguration"},
false,
},
{
"unrelated invalid request",
nil,
minio.ErrorResponse{Code: "InvalidRequest", Message: "Object is WORM protected and cannot be overwritten"},
false,
},
{"retention read denied", nil, minio.ErrorResponse{Code: "AccessDenied"}, false},
{"destination unreachable", nil, errors.New("dial tcp: connection refused"), false},
{"empty mode returned", &emptyMode, nil, true},
{"unknown non-empty mode returned", &unknownMode, nil, false},
{"nil mode returned", nil, nil, true},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
tgt := &fakeRetentionGetter{mode: test.mode, err: test.err}
if got := targetRetentionConfirmedAbsent(t.Context(), tgt, "bucket", "object", "v1"); got != test.want {
t.Fatalf("targetRetentionConfirmedAbsent() = %v, want %v", got, test.want)
}
if tgt.calls != 1 {
t.Fatalf("GetObjectRetention called %d times, want 1", tgt.calls)
}
})
}
}
// TestReplicationActionForTargetRetentionRemoval covers the decision the replication worker makes
// for a version whose retention was removed. The destination's HEAD never reports the empty keys,
// so the comparison alone reads every one of these as in sync; only the confirmation separates a
// destination that really dropped the retention from one that is hiding it.
func TestReplicationActionForTargetRetentionRemoval(t *testing.T) {
modeKey := strings.ToLower(xhttp.AmzObjectLockMode)
dateKey := strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)
governance := minio.Governance
tests := []struct {
name string
srcMeta map[string]string
mode *minio.RetentionMode
err error
want replicationAction
wantCalls int
}{
{
name: "removal confirmed by destination",
srcMeta: map[string]string{modeKey: "", dateKey: ""},
err: minio.ErrorResponse{Code: "NoSuchObjectLockConfiguration"},
want: replicateNone,
wantCalls: 1,
},
{
name: "destination still holds the retention hidden from HEAD",
srcMeta: map[string]string{modeKey: "", dateKey: ""},
mode: &governance,
want: replicateMetadata,
wantCalls: 1,
},
{
name: "retention hidden from HEAD by permissions",
srcMeta: map[string]string{modeKey: "", dateKey: ""},
err: minio.ErrorResponse{Code: "AccessDenied"},
want: replicateMetadata,
wantCalls: 1,
},
{
// A destination that names a missing Object Lock configuration answers the same way
// when its own read of that configuration failed, so it confirms nothing.
name: "destination reports no object lock configuration",
srcMeta: map[string]string{modeKey: "", dateKey: ""},
err: minio.ErrorResponse{Code: "InvalidRequest", Message: "Bucket is missing ObjectLockConfiguration"},
want: replicateMetadata,
wantCalls: 1,
},
{
name: "version never had retention is not confirmed",
srcMeta: nil,
want: replicateNone,
wantCalls: 0,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
src, tgtInfo := newMatchingReplicationPair()
for k, v := range test.srcMeta {
src.UserDefined[k] = v
}
tgt := &fakeRetentionGetter{mode: test.mode, err: test.err}
got := replicationActionForTarget(t.Context(), src, tgtInfo, replication.HealReplicationType, tgt, "bucket", "object")
if got != test.want {
t.Fatalf("replicationActionForTarget() = %q, want %q", got, test.want)
}
if tgt.calls != test.wantCalls {
t.Fatalf("GetObjectRetention called %d times, want %d", tgt.calls, test.wantCalls)
}
})
}
}
// TestReplicationActionForTargetNullVersionResync pins that the confirmation does not reopen the
// null-version exclusion at the head of getReplicationAction. An existing object resync returns
// replicateNone for a null version whose source modification time is later than the target's,
// before comparing anything, and that must stand even when the source carries a removed retention
// and the destination would report retention or refuse to answer.
func TestReplicationActionForTargetNullVersionResync(t *testing.T) {
modeKey := strings.ToLower(xhttp.AmzObjectLockMode)
dateKey := strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)
governance := minio.Governance
tests := []struct {
name string
mode *minio.RetentionMode
err error
}{
{"destination holds retention", &governance, nil},
{"retention read denied", nil, minio.ErrorResponse{Code: "AccessDenied"}},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
src, tgtInfo := newMatchingReplicationPair()
// A null version whose source modification time is later, and whose content differs,
// so only the exclusion can hold the action at replicateNone.
src.VersionID = nullVersionID
src.ModTime = tgtInfo.LastModified.Add(time.Hour)
src.ETag = "5d41402abc4b2a76b9719d911017c592"
src.UserDefined[modeKey] = ""
src.UserDefined[dateKey] = ""
tgtInfo.VersionID = nullVersionID
tgt := &fakeRetentionGetter{mode: test.mode, err: test.err}
got := replicationActionForTarget(t.Context(), src, tgtInfo, replication.ExistingObjectReplicationType, tgt, "bucket", "object")
if got != replicateNone {
t.Fatalf("replicationActionForTarget() = %q, want %q", got, replicateNone)
}
if tgt.calls != 0 {
t.Fatalf("GetObjectRetention called %d times, want 0", tgt.calls)
}
})
}
}
// TestReplicationActionForTargetTimestampOnlyRemoval covers representation (2) of a removed
// retention. A removal that arrived by replication persists only the retention ordering timestamp,
// with the mode and retain-until-date keys absent, because restoreRetention writes the timestamp
// alone when the mode is empty (cmd/bucket-object-lock.go). The comparison in getReplicationAction
// reads such a source as in sync with a matching destination, so only the GetObjectRetention
// confirmation separates a destination that dropped the retention from one hiding it behind a
// permission-filtered HEAD. The source is built through the real restoreRetention path so the
// fixture is the metadata a replicated removal actually leaves on disk, not a hand-rolled map.
func TestReplicationActionForTargetTimestampOnlyRemoval(t *testing.T) {
governance := minio.Governance
stamp := time.Date(2026, 9, 6, 1, 0, 0, 0, time.UTC).Format(time.RFC3339Nano)
tests := []struct {
name string
mode *minio.RetentionMode
err error
want replicationAction
wantCalls int
}{
{
// The reference case: a destination that denies the retention read is
// indistinguishable from one still holding it, so the removal is resent.
name: "retention hidden from HEAD by permissions",
err: minio.ErrorResponse{Code: "AccessDenied"},
want: replicateMetadata,
wantCalls: 1,
},
{
name: "destination still holds the retention",
mode: &governance,
want: replicateMetadata,
wantCalls: 1,
},
{
name: "removal confirmed by destination",
err: minio.ErrorResponse{Code: "NoSuchObjectLockConfiguration"},
want: replicateNone,
wantCalls: 1,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
src, tgtInfo := newMatchingReplicationPair()
// Persist the timestamp-only tombstone the same way an applied replica removal does.
objectLockState{retentionTimestamp: stamp}.restoreRetention(src.UserDefined)
if !retentionRemovedAtSource(src) {
t.Fatalf("restoreRetention fixture not recognized as a removal: %v", src.UserDefined)
}
if _, ok := src.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)]; ok {
t.Fatalf("restoreRetention fixture wrote a mode key, fixture is not timestamp-only: %v", src.UserDefined)
}
tgt := &fakeRetentionGetter{mode: test.mode, err: test.err}
got := replicationActionForTarget(t.Context(), src, tgtInfo, replication.HealReplicationType, tgt, "bucket", "object")
if got != test.want {
t.Fatalf("replicationActionForTarget() = %q, want %q", got, test.want)
}
if tgt.calls != test.wantCalls {
t.Fatalf("GetObjectRetention called %d times, want %d", tgt.calls, test.wantCalls)
}
})
}
}
+6 -2
View File
@@ -319,7 +319,9 @@ func (r *ReplicationStats) getNodeQueueStats(bucket string) (qs ReplQNodeStats)
qs.QStats = r.qCache.getBucketStats(bucket)
qs.TgtXferStats = make(map[string]map[RMetricName]XferStats)
qs.MRFStats = ReplicationMRFStats{
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
TotalDroppedCount: atomic.LoadUint64(&r.mrfStats.TotalDroppedCount),
TotalDroppedBytes: atomic.LoadUint64(&r.mrfStats.TotalDroppedBytes),
}
r.RLock()
@@ -410,7 +412,9 @@ func (r *ReplicationStats) getNodeQueueStatsSummary() (qs ReplQNodeStats) {
qs.XferStats = make(map[RMetricName]XferStats)
qs.QStats = r.qCache.getSiteStats()
qs.MRFStats = ReplicationMRFStats{
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
TotalDroppedCount: atomic.LoadUint64(&r.mrfStats.TotalDroppedCount),
TotalDroppedBytes: atomic.LoadUint64(&r.mrfStats.TotalDroppedBytes),
}
r.RLock()
defer r.RUnlock()
+3 -3
View File
@@ -97,7 +97,7 @@ func (api objectAPIHandlers) PutBucketVersioningHandler(w http.ResponseWriter, r
return
}
updatedAt, err := globalBucketMetadataSys.Update(ctx, bucket, bucketVersioningConfig, configData)
result, err := globalBucketMetadataSys.updateAndParseMetadata(ctx, bucket, bucketVersioningConfig, configData, false, false, nil)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -107,12 +107,12 @@ func (api objectAPIHandlers) PutBucketVersioningHandler(w http.ResponseWriter, r
//
// We encode the xml bytes as base64 to ensure there are no encoding
// errors.
cfgStr := base64.StdEncoding.EncodeToString(configData)
cfgStr := base64.StdEncoding.EncodeToString(result.meta.VersioningConfigXML)
replLogIf(ctx, globalSiteReplicationSys.BucketMetaHook(ctx, madmin.SRBucketMeta{
Type: madmin.SRBucketMetaTypeVersionConfig,
Bucket: bucket,
Versioning: &cfgStr,
UpdatedAt: updatedAt,
UpdatedAt: result.updatedAt,
}))
writeSuccessResponseHeadersOnly(w)
+54 -7
View File
@@ -120,15 +120,40 @@ func init() {
const consolePrefix = "CONSOLE_"
// consoleMinIOServerEnv derives the CONSOLE_MINIO_SERVER value the embedded
// Console uses to reach the S3/STS API, and whether TLS verification of that
// endpoint must be skipped. With no explicit endpoint configured the Console
// reaches the API over the loopback address, whose TLS certificate is not
// expected to carry a 127.0.0.1 SAN; because Console verifies outbound TLS by
// default, the loopback origin has to be exempted or embedded login (local and
// LDAP alike) fails at the STS handshake. The exemption is endpoint-scoped in
// Console, so every other HTTPS peer stays verified. An explicitly configured
// endpoint is always reached under its own verified name and is never exempted.
func consoleMinIOServerEnv(endpoint string, isTLS bool, port string) (server string, skipVerify bool) {
if endpoint != "" {
return endpoint, false
}
return fmt.Sprintf("%s://127.0.0.1:%s", getURLScheme(isTLS), port), isTLS
}
func minioConfigToConsoleFeatures() {
os.Setenv("CONSOLE_PBKDF_SALT", globalDeploymentID())
os.Setenv("CONSOLE_PBKDF_PASSPHRASE", globalDeploymentID())
if globalMinioEndpoint != "" {
os.Setenv("CONSOLE_MINIO_SERVER", globalMinioEndpoint)
consoleServer, skipVerify := consoleMinIOServerEnv(globalMinioEndpoint, globalIsTLS, globalMinioPort)
os.Setenv("CONSOLE_MINIO_SERVER", consoleServer)
if skipVerify {
// The embedded Console reaches the loopback S3/STS endpoint above, whose
// certificate is not expected to carry a 127.0.0.1 SAN. Console verifies
// outbound TLS by default (silo-console v2.3.x), so opt into the
// endpoint-scoped compatibility switch to preserve the documented loopback
// bypass; every other HTTPS peer (IdP, Prometheus, webhooks, ...) stays
// verified. initConsoleServer unsets CONSOLE_* before calling this, so the
// switch cannot be supplied by the operator on the embedded path.
os.Setenv("CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY", "on")
} else {
// Explicitly set 127.0.0.1 so Console will automatically bypass TLS verification to the local S3 API.
// This will save users from providing a certificate with IP or FQDN SAN that points to the local host.
os.Setenv("CONSOLE_MINIO_SERVER", fmt.Sprintf("%s://127.0.0.1:%s", getURLScheme(globalIsTLS), globalMinioPort))
// An explicitly configured endpoint is reached under its own verified name;
// never let a loopback exemption apply to it.
os.Unsetenv("CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY")
}
if value := env.Get(config.EnvMinIOLogQueryURL, ""); value != "" {
os.Setenv("CONSOLE_LOG_QUERY_URL", value)
@@ -232,11 +257,31 @@ func buildOpenIDConsoleConfig() consoleoauth2.OpenIDPCfg {
return m
}
func initConsoleServer() (*consoleapi.Server, error) {
// unset all console_ environment variables.
// resetConsoleEnvironment preserves the embedded Console's supported resource
// settings verbatim. Server derives all other Console settings itself.
func resetConsoleEnvironment() {
for _, cenv := range env.List(consolePrefix) {
switch cenv {
case consoleapi.ConsoleWSMaxConnections,
consoleapi.ConsoleWSMaxConnectionsPerClient,
consoleapi.ConsoleWSMaxAnonymousConnections,
consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient:
continue
}
os.Unsetenv(cenv)
}
}
func initConsoleServer() (*consoleapi.Server, error) {
resetConsoleEnvironment()
// Validate explicitly: ConfigureAPI logs errors, but embedded Console logs
// are normally silenced. Return configuration failures to Server startup.
if err := consoleapi.ConfigureEmbeddedSourceIPTrust(); err != nil {
return nil, err
}
if err := consoleapi.ConfigureWebSocketLimits(); err != nil {
return nil, err
}
// enable all console environment variables
minioConfigToConsoleFeatures()
@@ -400,6 +445,7 @@ func buildServerCtxt(ctx *cli.Context, ctxt *serverCtxt) (err error) {
ctxt.SendBufSize = ctx.Int("send-buf-size")
ctxt.RecvBufSize = ctx.Int("recv-buf-size")
ctxt.IdleTimeout = ctx.Duration("idle-timeout")
ctxt.ReadHeaderTimeout = ctx.Duration("read-header-timeout")
ctxt.UserTimeout = ctx.Duration("conn-user-timeout")
if conf := ctx.String("config"); len(conf) > 0 {
@@ -855,6 +901,7 @@ func serverHandleEnvVars() {
}
globalEnableSyncBoot = env.Get("MINIO_SYNC_BOOT", config.EnableOff) == config.EnableOn
globalSiteReplicationMetadataTombstones = env.Get("MINIO_SITE_REPLICATION_METADATA_TOMBSTONES", config.EnableOff) == config.EnableOn
}
func loadRootCredentials() auth.Credentials {
+136
View File
@@ -26,6 +26,7 @@ import (
"strings"
"testing"
consoleapi "github.com/minio/console/api"
"github.com/minio/minio/internal/config"
)
@@ -372,3 +373,138 @@ func TestConfigEnvFileNamedTargetDiscovery(t *testing.T) {
t.Fatalf("named target %q not discovered from %s: %v", "my-hook", key, targets)
}
}
// TestConsoleMinIOServerEnv locks in the loopback TLS exemption that keeps
// embedded Console login working (issue #108) while ensuring an explicitly
// configured endpoint is never silently exempted from TLS verification.
func TestConsoleMinIOServerEnv(t *testing.T) {
tests := []struct {
name string
endpoint string
isTLS bool
port string
wantServer string
wantSkipVerify bool
}{
{
name: "loopback TLS is exempted so embedded login works",
isTLS: true,
port: "9000",
wantServer: "https://127.0.0.1:9000",
wantSkipVerify: true,
},
{
name: "loopback plain HTTP needs no exemption",
isTLS: false,
port: "9000",
wantServer: "http://127.0.0.1:9000",
},
{
name: "explicit https endpoint stays verified",
endpoint: "https://silo.example:9000",
isTLS: true,
port: "9000",
wantServer: "https://silo.example:9000",
},
{
name: "explicit http endpoint stays verified",
endpoint: "http://silo.example:9000",
isTLS: false,
port: "9000",
wantServer: "http://silo.example:9000",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
server, skipVerify := consoleMinIOServerEnv(tt.endpoint, tt.isTLS, tt.port)
if server != tt.wantServer {
t.Fatalf("server = %q, want %q", server, tt.wantServer)
}
if skipVerify != tt.wantSkipVerify {
t.Fatalf("skipVerify = %v, want %v", skipVerify, tt.wantSkipVerify)
}
})
}
}
// The startup path clears process environment, so preserve all existing Console
// variables, including ones unrelated to this test, before exercising it.
func preserveConsoleEnvironment(t *testing.T) {
t.Helper()
for _, entry := range os.Environ() {
if strings.HasPrefix(entry, consolePrefix) {
name, value, _ := strings.Cut(entry, "=")
t.Setenv(name, value)
}
}
}
func TestResetConsoleEnvironment(t *testing.T) {
preserveConsoleEnvironment(t)
settings := map[string]string{
consoleapi.ConsoleWSMaxConnections: "2048",
consoleapi.ConsoleWSMaxConnectionsPerClient: "512",
consoleapi.ConsoleWSMaxAnonymousConnections: "128",
consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient: "16",
}
for key, value := range settings {
t.Setenv(key, value)
}
decoys := []string{"CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY", "CONSOLE_MINIO_SERVER", "CONSOLE_PBKDF_SALT", "CONSOLE_TRUSTED_PROXIES", "CONSOLE_WS_MAX_UNKNOWN"}
for _, key := range decoys {
t.Setenv(key, "operator-value")
}
resetConsoleEnvironment()
for key, want := range settings {
if got, present := os.LookupEnv(key); !present || got != want {
t.Errorf("%s = %q, present = %v; want %q", key, got, present, want)
}
}
for _, key := range decoys {
if _, present := os.LookupEnv(key); present {
t.Errorf("unsupported override %s survived", key)
}
}
for _, raw := range []string{"", " 16 ", "env://missing-limit"} {
t.Setenv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient, raw)
resetConsoleEnvironment()
if got, present := os.LookupEnv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient); !present || got != raw {
t.Fatalf("raw value %q was changed to %q (present = %v)", raw, got, present)
}
}
os.Unsetenv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient)
resetConsoleEnvironment()
if _, present := os.LookupEnv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient); present {
t.Fatal("unset setting became present")
}
}
func TestInitConsoleServerConfigurationErrors(t *testing.T) {
for _, tt := range []struct {
name, proxy, limit, want string
}{
{"proxy error precedes limit error", "proxy.internal", "bad", "MINIO_API_TRUSTED_PROXIES"},
{"blank limit", "", "", "CONSOLE_WS_MAX_ANONYMOUS_CONNECTIONS_PER_CLIENT"},
{"non-integer limit", "", "bad", "CONSOLE_WS_MAX_ANONYMOUS_CONNECTIONS_PER_CLIENT"},
{"out-of-range limit", "", "0", "CONSOLE_WS_MAX_ANONYMOUS_CONNECTIONS_PER_CLIENT"},
{"inconsistent limits", "", "256", "must be less than"},
} {
t.Run(tt.name, func(t *testing.T) {
// Restore the process-wide library configuration after environment cleanup.
t.Cleanup(func() {
_ = consoleapi.ConfigureEmbeddedSourceIPTrust()
_ = consoleapi.ConfigureWebSocketLimits()
})
preserveConsoleEnvironment(t)
t.Setenv(consoleapi.EnvMinIOTrustedProxies, tt.proxy)
t.Setenv(consoleapi.ConsoleWSMaxConnections, "1024")
t.Setenv(consoleapi.ConsoleWSMaxConnectionsPerClient, "256")
t.Setenv(consoleapi.ConsoleWSMaxAnonymousConnections, "64")
t.Setenv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient, tt.limit)
server, err := initConsoleServer()
if err == nil || !strings.Contains(err.Error(), tt.want) || server != nil {
t.Fatalf("initConsoleServer() = %v, %v; want nil server and %q error", server, err, tt.want)
}
})
}
}
+738
View File
@@ -0,0 +1,738 @@
// Copyright (c) 2015-2026 MinIO, Inc.
// Copyright (c) 2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"archive/tar"
"bytes"
"crypto/md5"
"encoding/base64"
"encoding/xml"
"io"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"
"github.com/klauspost/compress/s2"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/crypto"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/kms"
)
// disableCompression turns the global compression config off and returns a
// restore func. It is the replication destination that applies no transform of
// its own; setCopyChecksumCompression covers the enabled cases.
func disableCompression() func() {
globalCompressConfigMu.Lock()
previous := globalCompressConfig
globalCompressConfig.Enabled = false
globalCompressConfigMu.Unlock()
return func() {
globalCompressConfigMu.Lock()
globalCompressConfig = previous
globalCompressConfigMu.Unlock()
}
}
// ssecTestHeaders builds the customer key headers for a key made of the given
// repeated byte.
func ssecTestHeaders(b byte) map[string]string {
key := bytes.Repeat([]byte{b}, 32)
keyMD5 := md5.Sum(key)
return map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(keyMD5[:]),
}
}
// assertStoredSSECUncompressed requires the stored object to be SSE-C sealed
// and to carry no compression marker, so that "not compressed" is never
// reported for an object that is not encrypted either.
func assertStoredSSECUncompressed(t *testing.T, obj ObjectLayer, bucketName, object string) ObjectInfo {
t.Helper()
info, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if _, sealed := info.UserDefined[crypto.MetaSealedKeySSEC]; !sealed {
t.Fatalf("%s is not SSE-C sealed, the fixture proves nothing (userDefined=%v)", object, info.UserDefined)
}
if marker, compressed := info.UserDefined[ReservedMetadataPrefix+"compression"]; compressed {
t.Errorf("%s was stored as a compressed SSE-C object (compression=%q); such an object cannot be replicated",
object, marker)
}
return info
}
// assertSSECPlaintext GETs an SSE-C object with its customer key and requires
// the body to equal the plaintext.
func assertSSECPlaintext(t *testing.T, apiRouter http.Handler, credentials auth.Credentials,
bucketName, object string, sseHeaders map[string]string, want []byte,
) {
t.Helper()
req, err := newTestSignedRequestV4(http.MethodGet, getGetObjectURL("", bucketName, object),
0, nil, credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("GET %s status %d: %s", object, rec.Code, rec.Body.String())
}
if bytes.Equal(rec.Body.Bytes(), want) {
return
}
body := rec.Body.Bytes()
head := body
if len(head) > 16 {
head = head[:16]
}
t.Errorf("GET %s returned %d bytes, want the %d byte plaintext; first bytes % x",
object, len(body), len(want), head)
// Name the failure mode: a body that s2-decodes to the plaintext is the raw
// S2 stream of a compressed source shipped without its compression marker.
if decoded, derr := io.ReadAll(s2.NewReader(bytes.NewReader(body))); derr == nil && bytes.Equal(decoded, want) {
t.Errorf("the returned body is the raw S2 stream: s2-decoding it yields the %d byte plaintext", len(want))
}
}
// TestAPISSECCompressionReplicaStaysReadable replicates an SSE-C object written
// with compression enabled and allow_encryption=on, and requires the replica to
// read back as the source plaintext.
//
// The source object is read the way the replication worker reads it
// (ReplicationRequest, hence NoDecryption), its wire headers come from the
// production option builder putReplicationOpts, and the replica is written the
// way a destination that applies no transform of its own stores it: compression
// off and no default encryption.
//
// Before the SSE-C compression exclusion the source was stored as
// encrypt(s2(plaintext)) while putReplicationOpts dropped
// X-Minio-Internal-compression, so the replica decrypted to an S2 stream and a
// correct-key GET returned HTTP 200 with the wrong body.
func TestAPISSECCompressionReplicaStaysReadable(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPISSECCompressionReplicaStaysReadable,
})
}
func testAPISSECCompressionReplicaStaysReadable(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
replicator := newObjectAttributesAuthzUser(t, instanceType, bucketName, `"s3:PutObject","s3:GetObject","s3:ReplicateObject"`)
sseHeaders := ssecTestHeaders(0x42)
// Highly compressible and comfortably above minCompressibleSize (4096).
data := bytes.Repeat([]byte("silo compressed ssec replication payload "), 8192)
t.Run(instanceType+"/single-put", func(t *testing.T) {
object := "replication/ssec-single.txt"
// --- Source side: compression ON with allow_encryption ON. ---
restore := setCopyChecksumCompression(true)
srcReq, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, object),
int64(len(data)), bytes.NewReader(data), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
restore()
t.Fatal(err)
}
srcRec := httptest.NewRecorder()
apiRouter.ServeHTTP(srcRec, srcReq)
if srcRec.Code != http.StatusOK {
restore()
t.Fatalf("source PUT status %d: %s", srcRec.Code, srcRec.Body.String())
}
sourceInfo := assertStoredSSECUncompressed(t, obj, bucketName, object)
t.Logf("source: stored size=%d compression=%q plaintext=%d", sourceInfo.Size,
sourceInfo.UserDefined[ReservedMetadataPrefix+"compression"], len(data))
// The replication worker's read: raw stored bytes, no decryption.
gr, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{},
ObjectOptions{ReplicationRequest: true})
if err != nil {
restore()
t.Fatal(err)
}
sourceInfo = gr.ObjInfo
raw, err := io.ReadAll(gr)
gr.Close()
if err != nil {
restore()
t.Fatal(err)
}
replicationOpts, isMP, err := putReplicationOpts(t.Context(), "", sourceInfo)
if err != nil {
restore()
t.Fatalf("putReplicationOpts rejected the source: %v", err)
}
if isMP {
restore()
t.Fatal("single PUT source classified as multipart")
}
headers := map[string]string{}
for name, values := range replicationOpts.Header() {
if len(values) > 0 {
headers[name] = values[0]
}
}
restore()
// --- Destination side: NO compression, NO default encryption. ---
restoreDst := disableCompression()
defer restoreDst()
replReq, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, object),
int64(len(raw)), bytes.NewReader(raw), replicator.AccessKey, replicator.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
replRec := httptest.NewRecorder()
apiRouter.ServeHTTP(replRec, replReq)
if replRec.Code != http.StatusOK {
t.Fatalf("replica PUT status %d: %s", replRec.Code, replRec.Body.String())
}
info, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
t.Logf("replica: stored size=%d compression=%q actual-size=%q", info.Size,
info.UserDefined[ReservedMetadataPrefix+"compression"],
info.UserDefined[ReservedMetadataPrefix+"actual-size"])
assertSSECPlaintext(t, apiRouter, credentials, bucketName, object, sseHeaders, data)
})
t.Run(instanceType+"/multipart", func(t *testing.T) {
object := "replication/ssec-mpu.txt"
restore := setCopyChecksumCompression(true)
newReq, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
restore()
t.Fatal(err)
}
newRec := httptest.NewRecorder()
apiRouter.ServeHTTP(newRec, newReq)
if newRec.Code != http.StatusOK {
restore()
t.Fatalf("source NewMultipart status %d: %s", newRec.Code, newRec.Body.String())
}
var sourceInit InitiateMultipartUploadResponse
if err = xmlDecoder(newRec.Body, &sourceInit, int64(newRec.Body.Len())); err != nil {
restore()
t.Fatal(err)
}
partReq, err := newTestSignedRequestV4(http.MethodPut,
getPutObjectPartURL("", bucketName, object, sourceInit.UploadID, "1"),
int64(len(data)), bytes.NewReader(data), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
restore()
t.Fatal(err)
}
partRec := httptest.NewRecorder()
apiRouter.ServeHTTP(partRec, partReq)
if partRec.Code != http.StatusOK {
restore()
t.Fatalf("source PutPart status %d: %s", partRec.Code, partRec.Body.String())
}
completeBody, err := xml.Marshal(CompleteMultipartUpload{Parts: []CompletePart{
{PartNumber: 1, ETag: canonicalizeETag(partRec.Header()[xhttp.ETag][0])},
}})
if err != nil {
restore()
t.Fatal(err)
}
completeReq, err := newTestSignedRequestV4(http.MethodPost,
getCompleteMultipartUploadURL("", bucketName, object, sourceInit.UploadID),
int64(len(completeBody)), bytes.NewReader(completeBody), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
restore()
t.Fatal(err)
}
completeRec := httptest.NewRecorder()
apiRouter.ServeHTTP(completeRec, completeReq)
if completeRec.Code != http.StatusOK {
restore()
t.Fatalf("source Complete status %d: %s", completeRec.Code, completeRec.Body.String())
}
assertStoredSSECUncompressed(t, obj, bucketName, object)
gr, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{},
ObjectOptions{ReplicationRequest: true})
if err != nil {
restore()
t.Fatal(err)
}
sourceInfo := gr.ObjInfo
rawPart, err := io.ReadAll(gr)
gr.Close()
if err != nil {
restore()
t.Fatal(err)
}
actualSize, err := sourceInfo.GetActualSize()
if err != nil {
restore()
t.Fatal(err)
}
t.Logf("source mpu: stored size=%d actual-size=%d rawRead=%d plaintext=%d compression=%q",
sourceInfo.Size, actualSize, len(rawPart), len(data),
sourceInfo.UserDefined[ReservedMetadataPrefix+"compression"])
replicationOpts, isMP, err := putReplicationOpts(t.Context(), "", sourceInfo)
if err != nil {
restore()
t.Fatalf("putReplicationOpts rejected the source: %v", err)
}
if !isMP {
restore()
t.Fatal("SSE-C multipart source not recognized as multipart")
}
replicationOpts.Internal.SourceMTime = time.Time{}
headers := map[string]string{}
for name, values := range replicationOpts.Header() {
if len(values) > 0 {
headers[name] = values[0]
}
}
restore()
// --- Destination: no compression, no default encryption. ---
restoreDst := disableCompression()
defer restoreDst()
replNewReq, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, replicator.AccessKey, replicator.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
replNewRec := httptest.NewRecorder()
apiRouter.ServeHTTP(replNewRec, replNewReq)
if replNewRec.Code != http.StatusOK {
t.Fatalf("replica NewMultipart status %d: %s", replNewRec.Code, replNewRec.Body.String())
}
var replicaInit InitiateMultipartUploadResponse
if err = xmlDecoder(replNewRec.Body, &replicaInit, int64(replNewRec.Body.Len())); err != nil {
t.Fatal(err)
}
replPartReq, err := newTestSignedRequestV4(http.MethodPut,
getPutObjectPartURL("", bucketName, object, replicaInit.UploadID, "1"),
int64(len(rawPart)), bytes.NewReader(rawPart), replicator.AccessKey, replicator.SecretKey,
map[string]string{xhttp.MinIOSourceReplicationRequest: "true"})
if err != nil {
t.Fatal(err)
}
replPartRec := httptest.NewRecorder()
apiRouter.ServeHTTP(replPartRec, replPartReq)
if replPartRec.Code != http.StatusOK {
t.Fatalf("replica PutPart status %d: %s", replPartRec.Code, replPartRec.Body.String())
}
replCompleteBody, err := xml.Marshal(CompleteMultipartUpload{Parts: []CompletePart{
{PartNumber: 1, ETag: canonicalizeETag(replPartRec.Header()[xhttp.ETag][0])},
}})
if err != nil {
t.Fatal(err)
}
replCompleteReq, err := newTestSignedRequestV4(http.MethodPost,
getCompleteMultipartUploadURL("", bucketName, object, replicaInit.UploadID),
int64(len(replCompleteBody)), bytes.NewReader(replCompleteBody), replicator.AccessKey, replicator.SecretKey,
map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.MinIOSourceMTime: sourceInfo.ModTime.Format(time.RFC3339Nano),
xhttp.MinIOSourceETag: sourceInfo.ETag,
xhttp.MinIOReplicationActualObjectSize: strconv.FormatInt(actualSize, 10),
})
if err != nil {
t.Fatal(err)
}
replCompleteRec := httptest.NewRecorder()
apiRouter.ServeHTTP(replCompleteRec, replCompleteReq)
if replCompleteRec.Code != http.StatusOK {
t.Fatalf("replica Complete status %d: %s", replCompleteRec.Code, replCompleteRec.Body.String())
}
info, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
reportedActual, aerr := info.GetActualSize()
t.Logf("replica mpu: stored size=%d compression=%q actual-size=%q GetActualSize=%d(err=%v)",
info.Size, info.UserDefined[ReservedMetadataPrefix+"compression"],
info.UserDefined[ReservedMetadataPrefix+"actual-size"], reportedActual, aerr)
assertSSECPlaintext(t, apiRouter, credentials, bucketName, object, sseHeaders, data)
})
}
// TestAPISSECCompressionProducerMatrix pins the scope of the exclusion across
// the PutObject and NewMultipartUpload producers: SSE-C is never compressed,
// while plaintext, SSE-S3 and SSE-KMS keep following allow_encryption.
func TestAPISSECCompressionProducerMatrix(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPISSECCompressionProducerMatrix,
})
}
func testAPISSECCompressionProducerMatrix(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
ssecHeaders := ssecTestHeaders(0x5a)
sseS3Headers := map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionAES}
sseKMSHeaders := map[string]string{
xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionKMS,
xhttp.AmzServerSideEncryptionKmsID: "compressed-ssec-producer-matrix",
}
previousKMS := GlobalKMS
GlobalKMS = kms.NewStub("compressed-ssec-producer-matrix")
defer func() { GlobalKMS = previousKMS }()
big := bytes.Repeat([]byte("silo producer matrix payload "), 8192)
small := bytes.Repeat([]byte("s"), 1024) // below minCompressibleSize
for _, tc := range []struct {
name string
allowEncrypted bool
headers map[string]string
body []byte
wantCompressed bool
}{
// SSE-C is excluded from compression in both configurations, because the
// replication wire cannot carry the compression state.
{"ssec+allow_encryption-on+large", true, ssecHeaders, big, false},
{"ssec+allow_encryption-on+small", true, ssecHeaders, small, false},
{"ssec+allow_encryption-off+large", false, ssecHeaders, big, false},
// Plaintext still compresses in both configurations.
{"plain+allow_encryption-on+large", true, nil, big, true},
{"plain+allow_encryption-off+large", false, nil, big, true},
// SSE-S3 and SSE-KMS are the reason allow_encryption exists: the server
// owns the key, so the source decompresses before replicating.
{"sse-s3+allow_encryption-on+large", true, sseS3Headers, big, true},
{"sse-s3+allow_encryption-off+large", false, sseS3Headers, big, false},
{"sse-kms+allow_encryption-on+large", true, sseKMSHeaders, big, true},
{"sse-kms+allow_encryption-off+large", false, sseKMSHeaders, big, false},
} {
t.Run(instanceType+"/put/"+tc.name, func(t *testing.T) {
restore := setCopyChecksumCompression(tc.allowEncrypted)
defer restore()
object := "producer/" + tc.name + ".txt"
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, object),
int64(len(tc.body)), bytes.NewReader(tc.body), credentials.AccessKey, credentials.SecretKey, tc.headers)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("PUT status %d, want %d: %s", rec.Code, http.StatusOK, rec.Body.String())
}
info, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
_, compressed := info.UserDefined[ReservedMetadataPrefix+"compression"]
if compressed != tc.wantCompressed {
t.Errorf("compressed=%v, want %v (userDefined=%v)", compressed, tc.wantCompressed, info.UserDefined)
}
})
}
// NewMultipartUpload has no size gate, so the exclusion turns on the SSE-C
// and allow_encryption combination alone.
for _, tc := range []struct {
name string
allowEncrypted bool
headers map[string]string
wantCompressed bool
}{
{"ssec+allow_encryption-on", true, ssecHeaders, false},
{"ssec+allow_encryption-off", false, ssecHeaders, false},
{"plain+allow_encryption-on", true, nil, true},
} {
t.Run(instanceType+"/mpu/"+tc.name, func(t *testing.T) {
restore := setCopyChecksumCompression(tc.allowEncrypted)
defer restore()
object := "producer/mpu-" + tc.name + ".txt"
req, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, credentials.AccessKey, credentials.SecretKey, tc.headers)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("NewMultipartUpload status %d, want %d: %s", rec.Code, http.StatusOK, rec.Body.String())
}
var init InitiateMultipartUploadResponse
if err = xmlDecoder(rec.Body, &init, int64(rec.Body.Len())); err != nil {
t.Fatal(err)
}
mi, err := obj.GetMultipartInfo(t.Context(), bucketName, object, init.UploadID, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
_, compressed := mi.UserDefined[ReservedMetadataPrefix+"compression"]
if compressed != tc.wantCompressed {
t.Errorf("upload compressed=%v, want %v", compressed, tc.wantCompressed)
}
})
}
}
// TestAPISSECCompressionSkippedOnCopyObject covers the third producer:
// CopyObjectHandler decides compression before it encrypts, so a copy with a
// destination customer key and allow_encryption=on used to store a compressed
// SSE-C object from a plaintext source. It also covers the reverse direction,
// where only copy-source customer headers are present and compression must
// still apply.
func TestAPISSECCompressionSkippedOnCopyObject(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPISSECCompressionSkippedOnCopyObject,
endpoints: []string{"CopyObject", "PutObject", "GetObject"},
})
}
func testAPISSECCompressionSkippedOnCopyObject(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
ssecHeaders := ssecTestHeaders(0x7c)
data := bytes.Repeat([]byte("copy object compressed ssec payload "), 8192)
restore := setCopyChecksumCompression(true)
defer restore()
// An unencrypted source, stored compressed because compression is on. Only
// the copy adds encryption, so only the copy can change the decision.
src := "copysrc/plain.txt"
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, src),
int64(len(data)), bytes.NewReader(data), credentials.AccessKey, credentials.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("source PUT status %d: %s", rec.Code, rec.Body.String())
}
dst := "copydst/ssec.txt"
copyHeaders := map[string]string{"X-Amz-Copy-Source": SlashSeparator + bucketName + SlashSeparator + src}
for k, v := range ssecHeaders {
copyHeaders[k] = v
}
copyReq, err := newTestSignedRequestV4(http.MethodPut, getCopyObjectURL("", bucketName, dst),
0, nil, credentials.AccessKey, credentials.SecretKey, copyHeaders)
if err != nil {
t.Fatal(err)
}
copyRec := httptest.NewRecorder()
apiRouter.ServeHTTP(copyRec, copyReq)
if copyRec.Code != http.StatusOK {
t.Fatalf("CopyObject status %d: %s", copyRec.Code, copyRec.Body.String())
}
assertStoredSSECUncompressed(t, obj, bucketName, dst)
assertSSECPlaintext(t, apiRouter, credentials, bucketName, dst, ssecHeaders, data)
// The plaintext source is untouched by the copy and stays compressed.
srcInfo, err := obj.GetObjectInfo(t.Context(), bucketName, src, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if _, compressed := srcInfo.UserDefined[ReservedMetadataPrefix+"compression"]; !compressed {
t.Errorf("the plaintext copy source lost compression (userDefined=%v)", srcInfo.UserDefined)
}
// The reverse direction: a copy-source customer key is not a destination
// key, so copying the SSE-C object on to a plaintext destination still
// compresses. crypto.SSEC.IsRequested ignores the copy-source headers.
plain := "copydst/decrypted.txt"
decryptHeaders := map[string]string{
"X-Amz-Copy-Source": SlashSeparator + bucketName + SlashSeparator + dst,
xhttp.AmzServerSideEncryptionCopyCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCopyCustomerKey: ssecHeaders[xhttp.AmzServerSideEncryptionCustomerKey],
xhttp.AmzServerSideEncryptionCopyCustomerKeyMD5: ssecHeaders[xhttp.AmzServerSideEncryptionCustomerKeyMD5],
}
decryptReq, err := newTestSignedRequestV4(http.MethodPut, getCopyObjectURL("", bucketName, plain),
0, nil, credentials.AccessKey, credentials.SecretKey, decryptHeaders)
if err != nil {
t.Fatal(err)
}
decryptRec := httptest.NewRecorder()
apiRouter.ServeHTTP(decryptRec, decryptReq)
if decryptRec.Code != http.StatusOK {
t.Fatalf("CopyObject to a plaintext destination status %d: %s", decryptRec.Code, decryptRec.Body.String())
}
plainInfo, err := obj.GetObjectInfo(t.Context(), bucketName, plain, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if _, sealed := plainInfo.UserDefined[crypto.MetaSealedKeySSEC]; sealed {
t.Fatalf("%s is still SSE-C sealed, the fixture proves nothing", plain)
}
if _, compressed := plainInfo.UserDefined[ReservedMetadataPrefix+"compression"]; !compressed {
t.Errorf("a copy carrying only copy-source SSE-C headers was not compressed (userDefined=%v)", plainInfo.UserDefined)
}
}
// TestAPISSECCompressionSkippedOnSnowballExtract covers the fourth producer:
// PutObjectExtractHandler decides compression per entry before it encrypts, so
// a tar extract carrying customer key headers used to store every entry as a
// compressed SSE-C object.
func TestAPISSECCompressionSkippedOnSnowballExtract(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPISSECCompressionSkippedOnSnowballExtract,
})
}
func testAPISSECCompressionSkippedOnSnowballExtract(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
entry := "extracted/entry.txt"
payload := bytes.Repeat([]byte("snowball compressed ssec entry "), 4096)
var body bytes.Buffer
tw := tar.NewWriter(&body)
if err := tw.WriteHeader(&tar.Header{Name: entry, Mode: 0o600, Size: int64(len(payload))}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(payload); err != nil {
t.Fatal(err)
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
restore := setCopyChecksumCompression(true)
defer restore()
ssecHeaders := ssecTestHeaders(0x2d)
headers := map[string]string{xhttp.AmzSnowballExtract: "true"}
for k, v := range ssecHeaders {
headers[k] = v
}
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, "snowball.tar"),
int64(body.Len()), bytes.NewReader(body.Bytes()), credentials.AccessKey, credentials.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("snowball extract status %d: %s", rec.Code, rec.Body.String())
}
assertStoredSSECUncompressed(t, obj, bucketName, entry)
assertSSECPlaintext(t, apiRouter, credentials, bucketName, entry, ssecHeaders, payload)
}
// TestSSECBatchReplicationCannotRead is the control for the corruption path:
// batch replication reads without ReplicationRequest, so NoDecryption is never
// set and a non-empty SSE-C source fails at read time. Batch replication
// therefore cannot reach the replica shape in
// TestAPISSECCompressionReplicaStaysReadable; it cannot replicate a non-empty
// SSE-C object at all, compressed or not. A zero-byte object takes the reader
// shortcut, whose key check passes without a customer key.
func TestSSECBatchReplicationCannotRead(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testSSECBatchReplicationCannotRead,
})
}
func testSSECBatchReplicationCannotRead(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
ssecHeaders := ssecTestHeaders(0x6b)
data := bytes.Repeat([]byte("batch ssec payload "), 8192)
object := "batch/ssec-plain.txt"
restore := disableCompression()
defer restore()
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, object),
int64(len(data)), bytes.NewReader(data), credentials.AccessKey, credentials.SecretKey, ssecHeaders)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("source PUT status %d: %s", rec.Code, rec.Body.String())
}
// The read shape used by BatchJobReplicateV1.ReplicateToTarget and
// writeAsArchive: no ReplicationRequest, so NoDecryption is never set.
gr, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{}, ObjectOptions{})
if err == nil {
gr.Close()
t.Fatal("batch-shaped read of an SSE-C object unexpectedly succeeded")
}
t.Logf("batch-shaped read of an SSE-C object fails as expected: %v", err)
// Control: the replication worker's read shape succeeds and yields ciphertext.
gr2, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{},
ObjectOptions{ReplicationRequest: true})
if err != nil {
t.Fatalf("replication-shaped read failed: %v", err)
}
raw, err := io.ReadAll(gr2)
gr2.Close()
if err != nil {
t.Fatal(err)
}
if bytes.Equal(raw, data) {
t.Fatal("replication-shaped read returned plaintext")
}
t.Logf("replication-shaped read returns %d bytes of ciphertext (plaintext %d)", len(raw), len(data))
}
+89
View File
@@ -0,0 +1,89 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"encoding/json"
"os"
"reflect"
"testing"
)
// Read a real pre-removal scanner cache with nonzero hot-tier bytes. A v8
// payload with a relabeled header would not exercise the ignored hts field.
func TestDataUsageCacheReadV9(t *testing.T) {
raw, err := os.ReadFile("testdata/data-usage-v9/data-usage-v9.bin")
if err != nil {
t.Fatal(err)
}
if len(raw) == 0 || raw[0] != 9 {
t.Fatal("fixture is not v9")
}
expected, err := os.ReadFile("testdata/data-usage-v9/data-usage-v9.json")
if err != nil {
t.Fatal(err)
}
var want, got dataUsageCache
if err := json.Unmarshal(expected, &want); err != nil {
t.Fatal(err)
}
if err := got.deserialize(bytes.NewReader(raw)); err != nil {
t.Fatal(err)
}
if !got.Info.LastUpdate.Equal(want.Info.LastUpdate) {
t.Fatal("cache timestamp changed")
}
// msgp restores local time while JSON preserves the UTC representation.
want.Info.LastUpdate = got.Info.LastUpdate
if !reflect.DeepEqual(got, want) {
t.Fatalf("v9 ordinary cache fields changed:\ngot: %#v\nwant: %#v", got, want)
}
flat := got.flatten(*got.root())
if flat.Size != 74962 || flat.Objects != 3 || flat.Versions != 5 || flat.DeleteMarkers != 2 {
t.Fatalf("statistics changed: %+v", flat)
}
if flat.AllTierStats == nil || flat.AllTierStats.Tiers["COLD"] != (tierStats{TotalSize: 65536, NumVersions: 1, NumObjects: 1}) {
t.Fatalf("remote tier statistics changed: %+v", flat.AllTierStats)
}
buckets := []BucketInfo{{Name: "v9-bucket"}}
if gotInfo, wantInfo := got.dui(dataUsageRoot, buckets), want.dui(dataUsageRoot, buckets); !reflect.DeepEqual(gotInfo, wantInfo) {
t.Fatalf("bucket usage aggregation changed: %+v != %+v", gotInfo, wantInfo)
}
var buf bytes.Buffer
if err := got.serializeTo(&buf); err != nil {
t.Fatal(err)
}
if buf.Bytes()[0] != 8 {
t.Fatalf("wrote cache version %d, want 8", buf.Bytes()[0])
}
var roundtrip dataUsageCache
if err := roundtrip.deserialize(&buf); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, roundtrip) {
t.Fatal("v8 round trip lost ordinary fields")
}
if err := roundtrip.deserialize(bytes.NewReader(raw[:len(raw)/2])); err == nil {
t.Fatal("truncated v9 cache accepted")
}
raw[0] = 10
if err := roundtrip.deserialize(bytes.NewReader(raw)); err == nil {
t.Fatal("unknown cache version accepted")
}
}
+2 -1
View File
@@ -981,6 +981,7 @@ func (d *dataUsageCache) save(ctx context.Context, store objectIO, name string)
// and write new data with the new version.
const (
dataUsageCacheVerCurrent = 8
dataUsageCacheVerV9 = 9 // Retired access-tier cache; only adds the ignored "hts" entry key.
dataUsageCacheVerV7 = 7
dataUsageCacheVerV6 = 6
dataUsageCacheVerV5 = 5
@@ -1182,7 +1183,7 @@ func (d *dataUsageCache) deserialize(r io.Reader) error {
}
return nil
case dataUsageCacheVerCurrent:
case dataUsageCacheVerCurrent, dataUsageCacheVerV9:
// Zstd compressed.
dec, err := zstd.NewReader(r, zstd.WithDecoderConcurrency(2))
if err != nil {
+5
View File
@@ -1068,6 +1068,11 @@ func (er erasureObjects) HealObject(ctx context.Context, bucket, object, version
newReqInfo = logger.NewReqInfo("", "", globalDeploymentID(), "", "Heal", bucket, object)
}
healCtx := logger.SetReqInfo(GlobalContext, newReqInfo)
if opts.NoLock {
// The caller owns the namespace lock. Stop if that lock's context is
// canceled instead of continuing metadata writes after losing it.
healCtx = logger.SetReqInfo(ctx, newReqInfo)
}
// Healing directories handle it separately.
if HasSuffix(object, SlashSeparator) {
+385
View File
@@ -0,0 +1,385 @@
// Copyright (c) 2026 mr javad seydi and Ruohang Feng
//
// This file is part of Silo Object Storage stack.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/pgsty/silo-pkg/v3/policy"
)
const multipartScanEntryLimit = 100_000
var (
multipartScanSlots = make(chan struct{}, 2)
errMultipartListingLegacy = errors.New("legacy multipart uploads require a coordinated upgrade and drain")
errMultipartListingIdentity = errors.New("multipart upload identity is invalid")
)
// One budget and admission slot cover the entire request, including all pools.
// A slot is released only after the scan workers have actually stopped.
type multipartScan struct {
ctx context.Context
cancel context.CancelFunc
remaining atomic.Int64
metadataSlots chan struct{}
preflight bool
sets []multipartScanSet
}
type multipartScanSet struct {
Pool int `json:"pool"`
Set int `json:"set"`
Drives int `json:"drives"`
ScannedDrives int `json:"scannedDrives"`
UncoveredDrives []int `json:"uncoveredDrives,omitempty"`
Candidates int `json:"candidates"`
LegacyUploads int `json:"legacyUploads"`
OldestLegacy time.Time `json:"oldestLegacy,omitempty"`
Error string `json:"error,omitempty"`
}
type multipartPreflightReport struct {
Ready bool `json:"ready"`
Complete bool `json:"complete"`
Mode string `json:"mode"`
ScannedEntries int64 `json:"scannedEntries"`
LegacyUploads int `json:"legacyUploads"`
Sets []multipartScanSet `json:"sets"`
}
func startMultipartScan(ctx context.Context, preflight bool) (*multipartScan, error) {
select {
case multipartScanSlots <- struct{}{}:
case <-ctx.Done():
return nil, ctx.Err()
default:
return nil, SlowDown{}
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
s := &multipartScan{ctx: ctx, cancel: cancel, preflight: preflight, metadataSlots: make(chan struct{}, multipartMetadataScanConcurrency)}
s.remaining.Store(multipartScanEntryLimit)
return s, nil
}
func (s *multipartScan) close() {
s.cancel()
<-multipartScanSlots
}
func (s *multipartScan) listDir(disk StorageAPI, bucket, dir string) ([]string, error) {
if err := s.ctx.Err(); err != nil {
return nil, SlowDown{}
}
remaining := s.remaining.Load()
if remaining <= 0 {
return nil, SlowDown{}
}
// Request one extra entry to detect overflow, never a silently partial page.
entries, err := disk.ListDir(s.ctx, bucket, minioMetaMultipartBucket, dir, int(remaining)+1)
if errors.Is(err, errFileNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
if s.remaining.Add(-int64(len(entries))) < 0 {
return nil, SlowDown{}
}
return entries, nil
}
func (s *multipartScan) listUploadDirs(disk StorageAPI, bucket string) ([]string, error) {
hashDirs, err := s.listDir(disk, bucket, "")
if err != nil {
return nil, err
}
var candidates []string
for _, hashDir := range hashDirs {
if !strings.HasSuffix(hashDir, SlashSeparator) {
continue
}
hashDir = strings.TrimSuffix(hashDir, SlashSeparator)
uploadDirs, err := s.listDir(disk, bucket, hashDir)
if err != nil {
return nil, err
}
for _, uploadDir := range uploadDirs {
if strings.HasSuffix(uploadDir, SlashSeparator) {
candidates = append(candidates, pathJoin(hashDir, strings.TrimSuffix(uploadDir, SlashSeparator)))
}
}
}
return candidates, nil
}
// Identity is immutable at hash/uploadUUID. Check the hash before using even
// a single source drive to exclude another bucket. Missing/bad identities must
// fall back to the full metadata read; they cannot prove absence.
func (er erasureObjects) multipartIdentity(fi FileInfo, shaDir string) (string, string, bool) {
bucket, object := fi.Metadata[multipartMetaBucket], fi.Metadata[multipartMetaObject]
return bucket, object, bucket != "" && object != "" && IsValidBucketName(bucket) &&
IsValidObjectPrefix(object) && er.getMultipartSHADir(bucket, object) == shaDir
}
func (er erasureObjects) readMultipartUploadCandidate(s *multipartScan, bucket, candidate string, source StorageAPI) (MultipartInfo, bool, bool, error) {
if s.ctx.Err() != nil {
return MultipartInfo{}, false, false, SlowDown{}
}
shaDir, uploadUUID, ok := strings.Cut(candidate, SlashSeparator)
if !ok || shaDir == "" || uploadUUID == "" || strings.Contains(uploadUUID, SlashSeparator) {
return MultipartInfo{}, false, false, errMultipartListingIdentity
}
if !s.preflight && bucket != "" && source != nil {
fi, err := source.ReadVersion(s.ctx, bucket, minioMetaMultipartBucket, candidate, "", ReadOptions{})
if err == nil {
storedBucket, _, valid := er.multipartIdentity(fi, shaDir)
if valid && storedBucket != bucket {
return MultipartInfo{}, false, false, nil
}
}
}
if s.ctx.Err() != nil {
return MultipartInfo{}, false, false, SlowDown{}
}
select {
case s.metadataSlots <- struct{}{}:
defer func() { <-s.metadataSlots }()
case <-s.ctx.Done():
return MultipartInfo{}, false, false, SlowDown{}
}
disks := er.getDisks()
metadata, errs := readAllFileInfo(s.ctx, disks, bucket, minioMetaMultipartBucket, candidate, "", false, false)
if s.preflight {
// Readiness is stronger than listing liveness: even one readable legacy
// copy must be drained, and an unreadable copy cannot certify readiness.
var oldest MultipartInfo
legacy := false
_, nativeID := multipartUploadTime(uploadUUID)
for i, err := range errs {
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
continue
}
if err != nil {
return MultipartInfo{}, false, false, err
}
fi := metadata[i]
storedBucket, storedObject, valid := er.multipartIdentity(fi, shaDir)
if storedBucket != "" && storedObject != "" && !valid {
return MultipartInfo{}, false, false, errMultipartListingIdentity
}
if !valid || !nativeID {
info := multipartUploadInfo(storedBucket, storedObject, uploadUUID, fi.ModTime)
if !legacy || info.Initiated.Before(oldest.Initiated) {
oldest = info
}
legacy = true
}
}
if legacy {
return oldest, false, true, nil
}
}
readQuorum, _, err := objectQuorumFromMeta(s.ctx, metadata, errs, er.defaultParityCount)
if err != nil {
return MultipartInfo{}, false, false, err
}
_, modTime, etag := listOnlineDisks(disks, metadata, errs, readQuorum)
if err := reduceReadQuorumErrs(s.ctx, errs, objectOpIgnoredErrs, readQuorum); err != nil {
return MultipartInfo{}, false, false, err
}
fi, err := pickValidFileInfo(s.ctx, metadata, modTime, etag, readQuorum)
if err != nil {
return MultipartInfo{}, false, false, err
}
storedBucket, storedObject, valid := er.multipartIdentity(fi, shaDir)
info := multipartUploadInfo(storedBucket, storedObject, uploadUUID, fi.ModTime)
if storedBucket == "" || storedObject == "" {
return info, false, true, nil
}
if !valid {
return MultipartInfo{}, false, false, fmt.Errorf("%w: %s", errMultipartListingIdentity, candidate)
}
if _, ok := multipartUploadTime(uploadUUID); !ok {
return info, false, true, nil
}
if bucket != "" && bucket != storedBucket {
return MultipartInfo{}, false, false, nil
}
return info, true, false, nil
}
func (er erasureObjects) scanMultipartUploads(s *multipartScan, bucket string, poolIdx, setIdx int) ([]MultipartInfo, bool, error) {
disks := er.getDisks()
report := multipartScanSet{Pool: poolIdx, Set: setIdx, Drives: er.setDriveCount}
var resultErr error
defer func() {
if resultErr != nil {
report.Error = resultErr.Error()
}
s.sets = append(s.sets, report)
}()
var candidateMu sync.Mutex
candidates := make(map[string]StorageAPI)
errs := make([]error, len(disks))
var wg sync.WaitGroup
for i, disk := range disks {
wg.Add(1)
go func() {
defer wg.Done()
if disk == nil || !disk.IsOnline() {
errs[i] = errDiskNotFound
return
}
paths, err := s.listUploadDirs(disk, bucket)
errs[i] = err
if err != nil {
return
}
candidateMu.Lock()
defer candidateMu.Unlock()
for _, p := range paths {
candidates[p] = disk
}
}()
}
wg.Wait()
for i, err := range errs {
if err == nil {
report.ScannedDrives++
} else {
report.UncoveredDrives = append(report.UncoveredDrives, i)
if _, limited := err.(SlowDown); limited {
resultErr = err
}
}
}
report.Candidates = len(candidates)
// A successful scan must intersect every metadata read quorum, including
// records left with R copies by a partially failed cancellation.
if report.ScannedDrives < er.setDriveCount/2+1 && resultErr == nil {
resultErr = toObjectErr(errErasureReadQuorum, bucket)
}
if resultErr != nil {
return nil, false, resultErr
}
type candidate struct {
path string
source StorageAPI
}
jobs := make(chan candidate)
var mu sync.Mutex
var uploads []MultipartInfo
for range min(16, len(candidates)) {
wg.Add(1)
go func() {
defer wg.Done()
for c := range jobs {
mu.Lock()
stopped := resultErr != nil
mu.Unlock()
if stopped || s.ctx.Err() != nil {
continue
}
upload, found, legacy, err := er.readMultipartUploadCandidate(s, bucket, c.path, c.source)
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
continue
}
mu.Lock()
switch {
case err != nil && resultErr == nil:
resultErr = err
case legacy:
report.LegacyUploads++
if report.OldestLegacy.IsZero() || upload.Initiated.Before(report.OldestLegacy) {
report.OldestLegacy = upload.Initiated
}
case found && !s.preflight:
uploads = append(uploads, upload)
}
mu.Unlock()
}
}()
}
dispatch:
for p, source := range candidates {
select {
case jobs <- candidate{p, source}:
case <-s.ctx.Done():
break dispatch
}
}
close(jobs)
wg.Wait()
if s.ctx.Err() != nil {
resultErr = SlowDown{}
}
return uploads, report.LegacyUploads != 0, resultErr
}
// multipartPreflight scans all pools, sets and drives, independent of caches.
// A majority suffices for normal listing; upgrade readiness requires every
// drive to have been inspected. The operator must also upgrade all writers.
func (z *erasureServerPools) multipartPreflight(ctx context.Context) (multipartPreflightReport, error) {
s, err := startMultipartScan(ctx, true)
if err != nil {
return multipartPreflightReport{}, err
}
defer s.close()
report := multipartPreflightReport{Complete: true, Mode: "strict"}
if globalAPIConfig.getMultipartListingLegacy() {
report.Mode = "legacy"
}
for p, pool := range z.serverPools {
for i, set := range pool.sets {
_, _, _ = set.scanMultipartUploads(s, "", p, i)
}
}
report.Sets = s.sets
for _, set := range s.sets {
report.LegacyUploads += set.LegacyUploads
if set.Error != "" || set.ScannedDrives != set.Drives {
report.Complete = false
}
}
report.ScannedEntries = multipartScanEntryLimit - s.remaining.Load()
report.Ready = report.Complete && report.LegacyUploads == 0
return report, nil
}
// MultipartPreflightHandler is a read-only storage-admin diagnostic. It never
// accepts an arbitrary deletion path or changes upload lifetime settings.
func (a adminAPIHandlers) MultipartPreflightHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
obj, _ := validateAdminReq(ctx, w, r, policy.StorageInfoAdminAction)
if obj == nil {
return
}
z, ok := obj.(*erasureServerPools)
if !ok {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrNotImplemented), r.URL)
return
}
report, err := z.multipartPreflight(ctx)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
b, err := json.Marshal(report)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
writeSuccessResponseJSON(w, b)
}
+819
View File
@@ -0,0 +1,819 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/minio/minio/internal/config/storageclass"
)
// Check the real handler and storage path, including a successful abort between
// pages. Choose IDs increasing in both initiation time and lexical order so the
// failure does not depend on differing interpretations of S3 marker ordering.
func TestMultipartListingAbortBetweenHTTPPages(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads", "AbortMultipart"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
var firstID, secondID string
for attempt := 0; attempt < 32; attempt++ {
one, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
two, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if one.UploadID < two.UploadID {
firstID, secondID = one.UploadID, two.UploadID
break
}
for _, id := range []string{one.UploadID, two.UploadID} {
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", id, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
}
if firstID == "" {
t.Fatal("could not construct increasing upload IDs")
}
if _, err := z.NewMultipartUpload(t.Context(), bucket, "b", ObjectOptions{}); err != nil {
t.Fatal(err)
}
request := func(method, u string) *httptest.ResponseRecorder {
t.Helper()
req, err := newTestSignedRequestV4(method, u, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
return rec
}
list := func(keyMarker, uploadMarker string, limit int) ListMultipartUploadsResponse {
t.Helper()
rec := request(http.MethodGet, getListMultipartUploadsURLWithParams("", bucket, "", keyMarker, uploadMarker, "", strconv.Itoa(limit)))
if rec.Code != http.StatusOK {
t.Fatalf("list HTTP %d: %s", rec.Code, rec.Body.String())
}
var result ListMultipartUploadsResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &result); err != nil {
t.Fatal(err)
}
return result
}
first := list("", "", 1)
if len(first.Uploads) != 1 || first.Uploads[0].UploadID != firstID || !first.IsTruncated {
t.Fatalf("unexpected first page: %+v", first)
}
rec := request(http.MethodDelete, getAbortMultipartUploadURL("", bucket, "a", firstID))
if rec.Code != http.StatusNoContent {
t.Fatalf("abort HTTP %d: %s", rec.Code, rec.Body.String())
}
rest := list(first.NextKeyMarker, first.NextUploadIDMarker, 10)
var keys []string
for _, upload := range rest.Uploads {
keys = append(keys, upload.Key)
}
t.Logf("GET first page=200; DELETE marker=204; GET next page=200 keys=%v truncated=%v", keys, rest.IsTruncated)
if _, err := z.GetMultipartInfo(t.Context(), bucket, "a", secondID, ObjectOptions{}); err != nil {
t.Fatalf("remaining upload is not valid: %v", err)
}
if len(rest.Uploads) != 2 || rest.Uploads[0].UploadID != secondID {
t.Fatalf("valid remaining upload for key a is missing from continuation: %+v", rest.Uploads)
}
}
type multipartListingFaultDisk struct {
StorageAPI
read func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error)
delete func(context.Context, string, string, DeleteOptions) error
}
func (d multipartListingFaultDisk) ReadVersion(ctx context.Context, original, volume, object, version string, opts ReadOptions) (FileInfo, error) {
if d.read != nil {
return d.read(ctx, original, volume, object, version, opts)
}
return d.StorageAPI.ReadVersion(ctx, original, volume, object, version, opts)
}
func (d multipartListingFaultDisk) Delete(ctx context.Context, volume, object string, opts DeleteOptions) error {
if d.delete != nil {
return d.delete(ctx, volume, object, opts)
}
return d.StorageAPI.Delete(ctx, volume, object, opts)
}
func TestMultipartListingLegacyPreflight(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
const other = "multipart-legacy-other"
if err := z.MakeBucket(t.Context(), other, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
old, err := z.NewMultipartUpload(t.Context(), other, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
fi, metadata, err := set.checkUploadIDExists(t.Context(), other, "old", old.UploadID, true)
if err != nil {
t.Fatal(err)
}
for i := range metadata {
delete(metadata[i].Metadata, multipartMetaBucket)
delete(metadata[i].Metadata, multipartMetaObject)
}
if _, err = writeAllMetadata(t.Context(), set.getDisks(), other, minioMetaMultipartBucket,
set.getUploadIDDir(other, "old", old.UploadID), metadata, fi.WriteQuorum(set.defaultWQuorum())); err != nil {
t.Fatal(err)
}
if _, err = z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
z.mpCache.Clear()
t.Run("legacy-upgrade", func(t *testing.T) {
setMultipartListingTestMode(t, true)
exact, err := z.ListMultipartUploads(t.Context(), other, "old", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, exact, "old")
const empty = "multipart-legacy-empty"
if err := z.MakeBucket(t.Context(), empty, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
got, err := z.ListMultipartUploads(t.Context(), empty, "", "", "", "", 10)
if err != nil || len(got.Uploads) != 0 {
t.Fatalf("old upload in another bucket broke legacy listing: %+v %v", got, err)
}
})
_, err = z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if !errors.Is(err, errMultipartListingLegacy) {
t.Fatalf("old upload in another bucket: %v", err)
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || !report.Complete || report.LegacyUploads != 1 {
t.Fatalf("legacy preflight: %+v %v", report, err)
}
if err = z.AbortMultipartUpload(t.Context(), other, "old", old.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
report, err = z.multipartPreflight(t.Context())
if err != nil || !report.Ready || report.LegacyUploads != 0 {
t.Fatalf("drained preflight: %+v %v", report, err)
}
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
}
func TestMultipartListingIdentityFallback(t *testing.T) {
for _, kind := range []string{"old", "corrupt", "read-failure", "wrong-bucket"} {
t.Run(kind, func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
if _, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
var first atomic.Bool
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(ctx context.Context, b, v, p, version string, opts ReadOptions) (FileInfo, error) {
fi, err := d.ReadVersion(ctx, b, v, p, version, opts)
if v == minioMetaMultipartBucket && first.CompareAndSwap(false, true) {
if kind == "read-failure" {
return FileInfo{}, errDiskNotFound
}
if kind == "corrupt" {
return FileInfo{}, errFileCorrupt
}
fi.Metadata = cloneMSS(fi.Metadata)
if kind == "old" {
delete(fi.Metadata, multipartMetaBucket)
} else {
fi.Metadata[multipartMetaBucket] = "another-bucket"
}
}
return fi, err
}}
}
return disks
}
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
})
}
}
func TestMultipartAbortPoolsAndRetry(t *testing.T) {
z, bucket := consistencyPools(t)
setMultipartListingTestMode(t, false)
mp, err := z.serverPools[1].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
emptySet := z.serverPools[0].getHashedSet("a")
original := emptySet.getDisks
t.Cleanup(func() { emptySet.getDisks = original })
emptySet.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error) {
return FileInfo{}, errDiskNotFound
}}
}
return disks
}
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("unknown pool must not acknowledge cancellation: %v", err)
}
emptySet.getDisks = original
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if _, ok := err.(InvalidUploadID); !ok {
t.Fatalf("retry after all pools confirm absence: %v", err)
}
mp, err = z.serverPools[1].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if _, err = z.serverPools[1].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err == nil {
t.Fatal("empty first pool hid the actual upload")
}
}
func TestMultipartAbortRetryBelowReadQuorum(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
disks[3] = nil
d := disks[2]
disks[2] = multipartListingFaultDisk{StorageAPI: d, delete: func(ctx context.Context, v, p string, opts DeleteOptions) error {
if err := d.Delete(ctx, v, p, opts); err != nil {
return err
}
return context.DeadlineExceeded // operation finished, acknowledgement lost
}}
return disks
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err == nil {
t.Fatal("lost acknowledgement must fail")
}
set.getDisks = original
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("one remaining metadata copy prevented retry: %v", err)
}
}
func TestMultipartListingMarkerHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
for _, tc := range []struct {
key, marker string
status int
}{
{"", "not-base64=", 200},
{"a", "not-base64=", 404},
{"a", base64.RawURLEncoding.EncodeToString([]byte("not-native")), 400},
{"a", multipartListingTestID(time.Unix(100, 0), 1), 200},
} {
u := getListMultipartUploadsURLWithParams("", bucket, "", tc.key, tc.marker, "", "10")
req, err := newTestSignedRequestV4(http.MethodGet, u, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != tc.status {
t.Fatalf("key=%q marker=%q: %d %s", tc.key, tc.marker, rec.Code, rec.Body.String())
}
}
}
func TestMultipartPreflightAdminHTTP(t *testing.T) {
bed, err := prepareAdminErasureTestBed(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { bed.done(); bed.objLayer.Shutdown(context.Background()); removeRoots(bed.erasureDirs) })
const target = "/minio/admin/v3/multipart-preflight"
rec := httptest.NewRecorder()
bed.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, target, nil))
if rec.Code != 403 {
t.Fatalf("anonymous preflight: %d", rec.Code)
}
req, err := newTestSignedRequestV4(http.MethodGet, target, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec = httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != 200 {
t.Fatalf("admin preflight: %d %s", rec.Code, rec.Body.String())
}
var report multipartPreflightReport
if err = json.Unmarshal(rec.Body.Bytes(), &report); err != nil || !report.Ready || !report.Complete {
t.Fatalf("preflight: %+v %v", report, err)
}
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
rec = httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("busy admin preflight: %d %s", rec.Code, rec.Body.String())
}
}
type multipartLateCreateDisk struct {
StorageAPI
late chan<- func() error
}
func (d multipartLateCreateDisk) WriteMetadata(_ context.Context, original, volume, object string, fi FileInfo) error {
if volume == minioMetaMultipartBucket {
d.late <- func() error { return d.StorageAPI.WriteMetadata(context.Background(), original, volume, object, fi) }
return context.DeadlineExceeded
}
return d.StorageAPI.WriteMetadata(context.Background(), original, volume, object, fi)
}
// This characterization is deliberately NOT an assertion of terminal abort
// correctness. It preserves an executable example of the separately scoped
// late-creation-write limitation. Change the expectation when fencing is added.
func TestMultipartAbortLateCreateBoundary(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), 16)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
setMultipartListingTestMode(t, false)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
saved := globalStorageClass
globalStorageClass.Update(storageclass.Config{Standard: storageclass.StorageClass{Parity: 8}})
t.Cleanup(func() { globalStorageClass.Update(saved) })
const bucket = "multipart-late-create"
if err = z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
set := z.serverPools[0].getHashedSet("a")
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
late := make(chan func() error, 16)
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 9; i < 16; i++ {
disks[i] = multipartLateCreateDisk{disks[i], late}
}
return disks
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if len(late) != 7 {
t.Fatalf("expected seven timed-out creation writes, got %d", len(late))
}
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 2; i < 9; i++ {
disks[i] = nil
}
return disks
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
for range 7 {
if err = (<-late)(); err != nil {
t.Fatal(err)
}
}
set.getDisks = original
_, metadata, err := set.checkUploadIDExists(t.Context(), bucket, "a", mp.UploadID, true)
if err != nil {
t.Fatalf("late creation boundary changed; review and remove the documented limitation: %v", err)
}
count := 0
for _, fi := range metadata {
if fi.IsValid() {
count++
}
}
if count != 14 {
t.Fatalf("expected fourteen resurrected copies, got %d", count)
}
t.Log("KNOWN UNRESOLVED BOUNDARY: seven delayed creation writes plus seven offline copies restore a writable upload after acknowledged cancellation")
}
type multipartListingCountingDisk struct {
StorageAPI
directoryCalls *atomic.Int64
metadataCalls *atomic.Int64
}
func (d multipartListingCountingDisk) ListDir(ctx context.Context, original, volume, dir string, count int) ([]string, error) {
if volume == minioMetaMultipartBucket {
d.directoryCalls.Add(1)
}
return d.StorageAPI.ListDir(ctx, original, volume, dir, count)
}
func (d multipartListingCountingDisk) ReadVersion(ctx context.Context, original, volume, object, version string, opts ReadOptions) (FileInfo, error) {
if volume == minioMetaMultipartBucket {
d.metadataCalls.Add(1)
}
return d.StorageAPI.ReadVersion(ctx, original, volume, object, version, opts)
}
// Counts storage API calls; this is not a deployment throughput benchmark.
func TestMultipartListingScanCosts(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), 4)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
setMultipartListingTestMode(t, false)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket, otherBucket = "r9-scan-target", "r9-scan-unrelated"
for _, name := range []string{bucket, otherBucket} {
if err := z.MakeBucket(t.Context(), name, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
}
if _, err := z.NewMultipartUpload(t.Context(), bucket, "dir/target", ObjectOptions{}); err != nil {
t.Fatal(err)
}
var directoryCalls, metadataCalls atomic.Int64
for _, pool := range z.serverPools {
for _, set := range pool.sets {
original := set.getDisks
set.getDisks = func() []StorageAPI {
disks := original()
wrapped := make([]StorageAPI, len(disks))
for i, disk := range disks {
if disk != nil {
wrapped[i] = multipartListingCountingDisk{disk, &directoryCalls, &metadataCalls}
}
}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
}
}
measure := func(label string) (int64, int64) {
t.Helper()
directoryCalls.Store(0)
metadataCalls.Store(0)
result, err := z.ListMultipartUploads(t.Context(), bucket, "dir/", "", "", "", 1)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, result, "dir/target")
d, m := directoryCalls.Load(), metadataCalls.Load()
t.Logf("%s: max-uploads=1 returned=%d ListDir=%d ReadVersion=%d", label, len(result.Uploads), d, m)
return d, m
}
_, baseline := measure("no unrelated uploads")
for n := 0; n < 32; n++ {
if _, err := z.NewMultipartUpload(t.Context(), otherBucket, fmt.Sprintf("other/%03d", n), ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
_, loaded := measure("32 uploads in another bucket")
_, repeated := measure("same query repeated")
if loaded != baseline+32 || repeated != loaded {
t.Fatalf("unexpected scan accounting: baseline=%d loaded=%d repeated=%d", baseline, loaded, repeated)
}
}
func multipartListingTestID(created time.Time, n int) string {
return base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("00000000-0000-4000-8000-000000000000.00000000-0000-4000-8000-%012xx%d", n, created.UnixNano())))
}
func multipartListingFixture(t *testing.T) (*erasureServerPools, *erasureObjects, string) {
t.Helper()
obj, dirs, err := prepareErasure(t.Context(), 4)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket = "multipart-listing-test"
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
return z, z.serverPools[0].getHashedSet("a"), bucket
}
func TestMultipartListingMissingMarkers(t *testing.T) {
base := time.Unix(100, 0)
sameTime := []string{multipartListingTestID(base.Add(time.Second), 1), multipartListingTestID(base.Add(time.Second), 2), multipartListingTestID(base.Add(time.Second), 3)}
slices.Sort(sameTime)
uploads := []MultipartInfo{
{Bucket: "bucket", Object: "a", UploadID: multipartListingTestID(base, 1), Initiated: base},
{Bucket: "bucket", Object: "a", UploadID: sameTime[1], Initiated: base.Add(time.Second)},
{Bucket: "bucket", Object: "b", UploadID: multipartListingTestID(base, 4), Initiated: base},
}
for _, tc := range []struct {
name string
created time.Time
id int
want []string
}{
{"before", base.Add(-time.Second), 0, []string{"a", "a", "b"}},
{"between", base.Add(time.Second / 2), 2, []string{"a", "b"}},
{"same-time-before", base.Add(time.Second), 2, []string{"a", "b"}},
{"same-time-after", base.Add(time.Second), 4, []string{"b"}},
{"after", base.Add(2 * time.Second), 5, []string{"b"}},
} {
t.Run(tc.name, func(t *testing.T) {
marker := multipartListingTestID(tc.created, tc.id)
if tc.name == "same-time-before" {
marker = sameTime[0]
}
if tc.name == "same-time-after" {
marker = sameTime[2]
}
if err := checkListMultipartArgs(t.Context(), "bucket", "", "a", marker, ""); err != nil {
t.Fatal(err)
}
got := paginateMultipartUploads(uploads, "", "a", marker, "", 10)
if !slices.Equal(multipartUploadKeys(got.Uploads), tc.want) {
t.Fatalf("got %v want %v", multipartUploadKeys(got.Uploads), tc.want)
}
})
}
}
func TestMultipartListingAbortRecovery(t *testing.T) {
for _, offline := range []int{1, 2} {
t.Run(fmt.Sprint(offline), func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 0; i < offline; i++ {
disks[i] = nil
}
return disks
}
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if offline == 1 && err != nil {
t.Fatal(err)
}
if offline == 2 && (err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503) {
t.Fatalf("two offline drives must not acknowledge cancellation: %v", err)
}
set.getDisks = original
if offline == 2 {
// Retry a partially completed deletion after the original disks return.
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 100)
if err != nil || len(got.Uploads) != 0 {
t.Fatalf("canceled upload reappeared: %+v %v", got, err)
}
})
}
}
func TestMultipartListingCoverage(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
disks := original()
// Leave a readable two-copy record, simulating a partially failed abort.
for _, d := range disks[:2] {
if err = d.Delete(t.Context(), minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
set.getDisks = func() []StorageAPI { return []StorageAPI{disks[0], disks[1], nil, nil} }
_, err = z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("incomplete discovery returned success: %v", err)
}
set.getDisks = func() []StorageAPI { return []StorageAPI{nil, disks[1], disks[2], disks[3]} }
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || report.Complete || len(report.Sets[0].UncoveredDrives) != 1 {
t.Fatalf("offline upgrade preflight: %+v %v", report, err)
}
}
func TestMultipartListingBudgetAndAdmission(t *testing.T) {
_, set, bucket := multipartListingFixture(t)
if _, err := set.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
scan.remaining.Store(1)
_, _, err = set.scanMultipartUploads(scan, bucket, 0, 0)
var limited SlowDown
if !errors.As(err, &limited) {
t.Fatalf("budget: %v", err)
}
if apiErr := toAPIError(t.Context(), err); apiErr.HTTPStatusCode != http.StatusServiceUnavailable || apiErr.Code != "SlowDown" {
t.Fatalf("budget error mapping: %+v", apiErr)
}
second, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer second.close()
if third, err := startMultipartScan(t.Context(), false); err == nil {
third.close()
t.Fatal("third scan admitted")
}
}
func TestMultipartListingAdmissionHTTP(t *testing.T) {
z, _, _ := multipartListingFixture(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
req, err := newTestSignedRequestV4(http.MethodGet,
getListMultipartUploadsURLWithParams("", bucket, "", "", "", "", "1"),
0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
var response APIErrorResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &response); err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusServiceUnavailable || response.Code != "SlowDown" {
t.Fatalf("admission returned %d %s", rec.Code, rec.Body.String())
}
}
func TestMultipartListingPreflightMinorityLegacy(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
path := set.getUploadIDDir(bucket, "old", mp.UploadID)
disks := set.getDisks()
fi, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, path, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
if err = disks[0].WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, path, fi); err != nil {
t.Fatal(err)
}
for _, d := range disks[1:] {
if err = d.Delete(t.Context(), minioMetaMultipartBucket, path, DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || !report.Complete || report.LegacyUploads != 1 {
t.Fatalf("minority legacy copy must prevent readiness: %+v %v", report, err)
}
}
func TestMultipartListingCancellationRetainsAdmission(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
for i := range 40 {
if _, err := z.NewMultipartUpload(t.Context(), bucket, fmt.Sprintf("key-%02d", i), ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
entered := make(chan struct{}, 40)
release := make(chan struct{})
var once sync.Once
defer once.Do(func() { close(release) })
var reads atomic.Int32
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(ctx context.Context, b, v, p, version string, opts ReadOptions) (FileInfo, error) {
if v != minioMetaMultipartBucket {
return d.ReadVersion(ctx, b, v, p, version, opts)
}
reads.Add(1)
entered <- struct{}{}
// Model an RPC which does not return immediately on cancellation.
<-release
return FileInfo{}, ctx.Err()
}}
}
return disks
}
second, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer second.close()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() {
_, err := z.ListMultipartUploads(ctx, bucket, "", "", "", "", 10)
done <- err
}()
for range 16 {
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("identity workers did not start")
}
}
cancel()
if extra, err := startMultipartScan(t.Context(), false); err == nil {
extra.close()
t.Fatal("canceled request released admission while RPCs were still running")
}
start := time.Now()
once.Do(func() { close(release) })
select {
case err := <-done:
if err == nil {
t.Fatal("canceled scan returned a successful partial list")
}
case <-time.After(2 * time.Second):
t.Fatal("scan did not stop after blocked RPCs returned")
}
if got := reads.Load(); got != 16 {
t.Fatalf("scheduled more identity/metadata reads after cancellation: %d", got)
}
t.Logf("16 identity RPCs bounded; admission held until return; cancellation settled in %s", time.Since(start))
}
+331
View File
@@ -0,0 +1,331 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/base64"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
"github.com/minio/minio/internal/config/api"
"github.com/minio/minio/internal/config/storageclass"
"github.com/minio/minio/internal/dsync"
"github.com/minio/minio/internal/grid"
xnet "github.com/pgsty/silo-pkg/v3/net"
)
func setMultipartListingTestMode(t *testing.T, legacy bool) {
t.Helper()
globalAPIConfig.mu.Lock()
previous := globalAPIConfig.multipartListingStrict
globalAPIConfig.multipartListingStrict = !legacy
globalAPIConfig.mu.Unlock()
t.Cleanup(func() {
globalAPIConfig.mu.Lock()
globalAPIConfig.multipartListingStrict = previous
globalAPIConfig.mu.Unlock()
})
}
func TestMultipartListingDefaultMode(t *testing.T) {
var uninitialized apiConfig
if !uninitialized.getMultipartListingLegacy() {
t.Fatal("runtime zero value enabled strict mode before config initialization")
}
for _, mode := range []string{"", "legacy", "strict"} {
var local apiConfig
local.init(api.Config{RequestsMax: 1, MultipartListing: mode}, []int{4}, false)
if local.getMultipartListingLegacy() != (mode != "strict") {
t.Fatalf("unexpected mode for %q", mode)
}
}
}
func TestMultipartAbortLegacyAvailability(t *testing.T) {
for _, drives := range []int{4, 16} {
for _, offline := range []int{0, 1, drives / 2} {
t.Run(fmt.Sprintf("drives=%d/offline=%d", drives, offline), func(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), drives)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
setMultipartListingTestMode(t, true)
previous := globalStorageClass
globalStorageClass.Update(storageclass.Config{Standard: storageclass.StorageClass{Parity: drives / 2}})
t.Cleanup(func() { globalStorageClass.Update(previous) })
const bucket, key = "multipart-legacy-availability", "keep-available"
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
set := z.serverPools[0].getHashedSet(key)
original := set.getDisks
disks := original()
set.getDisks = func() []StorageAPI {
visible := append([]StorageAPI(nil), disks...)
clear(visible[:offline])
return visible
}
t.Cleanup(func() { set.getDisks = original })
if err := z.AbortMultipartUpload(t.Context(), bucket, key, mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("released read-quorum availability regressed: %v", err)
}
for _, disk := range disks[offline:] {
_, err := disk.ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, key, mp.UploadID), "", ReadOptions{})
if !errors.Is(err, errFileNotFound) {
t.Fatalf("online replica not cleaned: %v", err)
}
}
})
}
}
}
func TestMultipartAbortLegacyPoolOrder(t *testing.T) {
for _, owner := range []int{0, 1} {
t.Run(fmt.Sprintf("owner=%d", owner), func(t *testing.T) {
z, bucket := consistencyPools(t)
setMultipartListingTestMode(t, true)
mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
emptySet := z.serverPools[1-owner].getHashedSet("a")
original := emptySet.getDisks
t.Cleanup(func() { emptySet.getDisks = original })
emptySet.getDisks = func() []StorageAPI { return make([]StorageAPI, emptySet.setDriveCount) }
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if owner == 0 && err != nil {
t.Fatalf("unrelated later pool blocked legacy cancellation: %v", err)
}
if owner == 1 {
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("unknown earlier pool must retain released error behavior: %v", err)
}
if _, err := z.serverPools[owner].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("later pool visited despite earlier error: %v", err)
}
emptySet.getDisks = original
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
})
}
}
func TestMultipartAbortMinorityLegacyCleanup(t *testing.T) {
for _, failDelete := range []bool{false, true} {
t.Run(fmt.Sprintf("delete-fails=%v", failDelete), func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
path := set.getUploadIDDir(bucket, "old", mp.UploadID)
original := set.getDisks
disks := original()
fi, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, path, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
if err := disks[0].WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, path, fi); err != nil {
t.Fatal(err)
}
for _, d := range disks[1:] {
if err := d.Delete(t.Context(), minioMetaMultipartBucket, path, DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || report.LegacyUploads != 1 {
t.Fatalf("invalid legacy fixture: %+v %v", report, err)
}
if failDelete {
set.getDisks = func() []StorageAPI {
wrapped := append([]StorageAPI(nil), disks...)
wrapped[0] = multipartListingFaultDisk{StorageAPI: disks[0], delete: func(context.Context, string, string, DeleteOptions) error { return errFaultyDisk }}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
err := z.AbortMultipartUpload(t.Context(), bucket, "old", mp.UploadID, ObjectOptions{})
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("empty drives masked failed deletion of the observed replica: %v", err)
}
if _, present := z.mpCache.Load(mp.UploadID); !present {
t.Fatal("failed cancellation evicted upload from cache")
}
set.getDisks = original
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "old", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
report, err = z.multipartPreflight(t.Context())
if err != nil || !report.Ready || report.LegacyUploads != 0 {
t.Fatalf("acknowledged cleanup left online legacy remnants: %+v %v", report, err)
}
if _, present := z.mpCache.Load(mp.UploadID); present {
t.Fatal("successful cancellation retained cache entry")
}
})
}
}
func TestMultipartAbortWrongTargetKeepsCache(t *testing.T) {
for _, legacy := range []bool{true, false} {
t.Run(fmt.Sprintf("legacy=%v", legacy), func(t *testing.T) {
z, _, bucket := multipartListingFixture(t)
setMultipartListingTestMode(t, legacy)
const other = "multipart-other-bucket"
if err := z.MakeBucket(t.Context(), other, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, "valid-key", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, target := range [][2]string{{bucket, "wrong-key"}, {other, "valid-key"}} {
err := z.AbortMultipartUpload(t.Context(), target[0], target[1], mp.UploadID, ObjectOptions{})
var invalid InvalidUploadID
if !errors.As(err, &invalid) {
t.Fatalf("wrong target result: %v", err)
}
if _, present := z.mpCache.Load(mp.UploadID); !present {
t.Fatal("wrong-target abort evicted another upload")
}
if _, err := z.GetMultipartInfo(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("wrong-target abort harmed valid upload: %v", err)
}
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("repeat abort: %v", err)
}
})
}
}
func TestMultipartAbortRejectsUnsafeID(t *testing.T) {
for _, legacy := range []bool{true, false} {
t.Run(fmt.Sprintf("legacy=%v", legacy), func(t *testing.T) {
z, _, bucket := multipartListingFixture(t)
setMultipartListingTestMode(t, legacy)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "keep", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, suffix := range []string{".", "..", "../other", "a/b", "a\\b", ""} {
id := base64.RawURLEncoding.EncodeToString([]byte("deployment." + suffix))
err := z.AbortMultipartUpload(t.Context(), bucket, "keep", id, ObjectOptions{})
var invalid InvalidUploadID
if !errors.As(err, &invalid) {
t.Fatalf("unsafe ID accepted: suffix=%q err=%v", suffix, err)
}
if _, err := z.GetMultipartInfo(t.Context(), bucket, "keep", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("valid upload harmed by rejected ID: %v", err)
}
}
})
}
}
func TestMultipartAbortPeerNotification(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
tg, err := grid.SetupTestGrid(2)
if err != nil {
t.Fatal(err)
}
t.Cleanup(tg.Cleanup)
var notifications atomic.Int32
if err := cleanupUploadIDCacheMetaRPC.Register(tg.Managers[1], func(*grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
notifications.Add(1)
return grid.NoPayload{}, nil
}); err != nil {
t.Fatal(err)
}
host, err := xnet.ParseHost(strings.TrimPrefix(tg.Hosts[1], "http://"))
if err != nil {
t.Fatal(err)
}
previous := globalNotificationSys
globalNotificationSys = &NotificationSys{peerClients: []*peerRESTClient{{
host: host,
gridConn: func() *grid.Connection { return tg.Managers[0].Connection(tg.Hosts[1]) },
}}}
t.Cleanup(func() { globalNotificationSys = previous })
// Use the real distributed lock implementation: Unlock cancels its derived
// context. Local locks do not, and would hide a broken notification context.
oldMutex, oldLockers := set.nsMutex, set.getLockers
lockers := []dsync.NetLocker{newLocker(), newLocker()}
set.nsMutex = newNSLock(true)
set.getLockers = func() ([]dsync.NetLocker, string) { return lockers, "multipart-test" }
t.Cleanup(func() { set.nsMutex, set.getLockers = oldMutex, oldLockers })
for _, legacy := range []bool{true, false} {
t.Run(fmt.Sprintf("legacy=%v", legacy), func(t *testing.T) {
setMultipartListingTestMode(t, legacy)
for range 4 {
mp, err := z.NewMultipartUpload(t.Context(), bucket, "valid", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
before := notifications.Load()
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "wrong", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("wrong target: %v", err)
}
if notifications.Load() != before {
t.Fatal("wrong-target abort sent a destructive peer notification")
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if notifications.Load() != before+1 {
t.Fatal("successful abort lost peer notification after unlocking")
}
}
})
}
}
func TestMultipartAbortStrictMajority(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
disks := original()
set.getDisks = func() []StorageAPI {
wrapped := append([]StorageAPI(nil), disks...)
wrapped[0] = multipartListingFaultDisk{StorageAPI: disks[0], delete: func(context.Context, string, string, DeleteOptions) error { return errFaultyDisk }}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("ordinary majority cancellation was tightened: %v", err)
}
// One failed deletion is tolerated in the ordinary majority path. Retrying
// after recovery must also clean the now-minority remnant.
if _, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), "", ReadOptions{}); err != nil {
t.Fatalf("fault fixture did not leave a remnant: %v", err)
}
set.getDisks = original
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
+790
View File
@@ -0,0 +1,790 @@
// Copyright (c) 2015-2026 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"crypto/md5"
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/crypto"
"github.com/minio/minio/internal/hash"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/sio"
)
// TestAPISSECReplicaPartNumberReads replicates a three-part SSE-C multipart
// object through the trusted-replication write path and compares what
// GET ?partNumber=N returns before and after the replica overwrite.
func TestAPISSECReplicaPartNumberReads(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPISSECReplicaPartNumberReads,
})
}
func testAPISSECReplicaPartNumberReads(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
replicator := newObjectAttributesAuthzUser(t, instanceType, bucketName, `"s3:PutObject","s3:GetObject","s3:ReplicateObject"`)
key := bytes.Repeat([]byte{0x42}, 32)
keyMD5 := md5.Sum(key)
sseHeaders := map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(keyMD5[:]),
}
const mib = 1024 * 1024
partLens := []int{5 * mib, 5 * mib, 1 * mib}
plaintext := make([]byte, 0, 11*mib)
partData := make([][]byte, len(partLens))
for i, n := range partLens {
b := make([]byte, n)
for j := range b {
// Distinct, position-dependent bytes so an off-by-N shift is visible.
b[j] = byte(i*7 + j%251)
}
partData[i] = b
plaintext = append(plaintext, b...)
}
object := "ssec-mp-3part"
// ---- 1. Build the source: a real three-part SSE-C multipart object. ----
newRec := httptest.NewRecorder()
newReq, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
apiRouter.ServeHTTP(newRec, newReq)
if newRec.Code != http.StatusOK {
t.Fatalf("source NewMultipart status %d: %s", newRec.Code, newRec.Body.String())
}
var srcInit InitiateMultipartUploadResponse
if err = xmlDecoder(newRec.Body, &srcInit, int64(newRec.Body.Len())); err != nil {
t.Fatal(err)
}
srcParts := make([]CompletePart, len(partLens))
for i, b := range partData {
pn := strconv.Itoa(i + 1)
partReq, err := newTestSignedRequestV4(http.MethodPut,
getPutObjectPartURL("", bucketName, object, srcInit.UploadID, pn),
int64(len(b)), bytes.NewReader(b), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, partReq)
if rec.Code != http.StatusOK {
t.Fatalf("source PutPart %s status %d: %s", pn, rec.Code, rec.Body.String())
}
srcParts[i] = CompletePart{PartNumber: i + 1, ETag: canonicalizeETag(rec.Header()[xhttp.ETag][0])}
}
srcCompleteBody, err := xml.Marshal(CompleteMultipartUpload{Parts: srcParts})
if err != nil {
t.Fatal(err)
}
completeReq, err := newTestSignedRequestV4(http.MethodPost,
getCompleteMultipartUploadURL("", bucketName, object, srcInit.UploadID), int64(len(srcCompleteBody)),
bytes.NewReader(srcCompleteBody), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
completeRec := httptest.NewRecorder()
apiRouter.ServeHTTP(completeRec, completeReq)
if completeRec.Code != http.StatusOK {
t.Fatalf("source Complete status %d: %s", completeRec.Code, completeRec.Body.String())
}
// ---- 2. Record the source's per-part metadata and per-part GET answers. ----
srcOI, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
t.Logf("[%s] SOURCE parts:", instanceType)
for _, p := range srcOI.Parts {
t.Logf(" part %d Size=%d ActualSize=%d", p.Number, p.Size, p.ActualSize)
}
srcActual, err := srcOI.GetActualSize()
if err != nil {
t.Fatal(err)
}
t.Logf("[%s] SOURCE object Size=%d GetActualSize=%d actual-size-meta=%q",
instanceType, srcOI.Size, srcActual, srcOI.UserDefined[ReservedMetadataPrefix+"actual-size"])
type getResult struct {
status int
clen string
crange string
body []byte
}
doGet := func(query string, extra map[string]string) getResult {
hdrs := map[string]string{}
for k, v := range sseHeaders {
hdrs[k] = v
}
for k, v := range extra {
hdrs[k] = v
}
u := getGetObjectURL("", bucketName, object) + query
req, err := newTestSignedRequestV4(http.MethodGet, u, 0, nil, credentials.AccessKey, credentials.SecretKey, hdrs)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
return getResult{
status: rec.Code,
clen: rec.Header().Get(xhttp.ContentLength),
crange: rec.Header().Get(xhttp.ContentRange),
body: append([]byte(nil), rec.Body.Bytes()...),
}
}
doHead := func(query string) getResult {
u := getGetObjectURL("", bucketName, object) + query
req, err := newTestSignedRequestV4(http.MethodHead, u, 0, nil, credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
return getResult{
status: rec.Code,
clen: rec.Header().Get(xhttp.ContentLength),
crange: rec.Header().Get(xhttp.ContentRange),
}
}
queries := []string{"?partNumber=1", "?partNumber=2", "?partNumber=3"}
srcGets := make([]getResult, len(queries))
for i, q := range queries {
srcGets[i] = doGet(q, nil)
t.Logf("[%s] SOURCE GET %s -> status=%d Content-Length=%s Content-Range=%s len(body)=%d",
instanceType, q, srcGets[i].status, srcGets[i].clen, srcGets[i].crange, len(srcGets[i].body))
}
// Range GET crossing the part1/part2 boundary.
boundaryRange := fmt.Sprintf("bytes=%d-%d", 5*mib-16, 5*mib+15)
srcRange := doGet("", map[string]string{"Range": boundaryRange})
t.Logf("[%s] SOURCE GET Range %s -> status=%d Content-Length=%s len(body)=%d",
instanceType, boundaryRange, srcRange.status, srcRange.clen, len(srcRange.body))
// Sanity: the source must return exactly the part bytes.
for i := range partData {
if !bytes.Equal(srcGets[i].body, partData[i]) {
t.Fatalf("[%s] SOURCE partNumber=%d returned wrong bytes (len %d want %d)",
instanceType, i+1, len(srcGets[i].body), len(partData[i]))
}
}
// ---- 3. Read the raw ciphertext the replication worker would ship. ----
gr, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{}, ObjectOptions{ReplicationRequest: true})
if err != nil {
t.Fatal(err)
}
sourceInfo := gr.ObjInfo
rawAll, err := io.ReadAll(gr)
gr.Close()
if err != nil {
t.Fatal(err)
}
if bytes.Equal(rawAll, plaintext) {
t.Fatal("source replication read did not return encrypted bytes")
}
rawParts := make([][]byte, len(sourceInfo.Parts))
off := int64(0)
for i, p := range sourceInfo.Parts {
rawParts[i] = rawAll[off : off+p.Size]
off += p.Size
}
if off != int64(len(rawAll)) {
t.Fatalf("raw ciphertext length %d != sum of part sizes %d", len(rawAll), off)
}
// ---- 4. Replicate onto the same key through the trusted write path. ----
replicationOpts, isMP, err := putReplicationOpts(t.Context(), "", sourceInfo)
if err != nil {
t.Fatal(err)
}
if !isMP {
t.Fatal("SSE-C multipart source was not recognized as multipart")
}
replicationOpts.Internal.SourceMTime = time.Time{}
replicationHeaders := make(map[string]string)
for name, values := range replicationOpts.Header() {
if len(values) > 0 {
replicationHeaders[name] = values[0]
}
}
replNewReq, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, replicator.AccessKey, replicator.SecretKey, replicationHeaders)
if err != nil {
t.Fatal(err)
}
replNewRec := httptest.NewRecorder()
apiRouter.ServeHTTP(replNewRec, replNewReq)
if replNewRec.Code != http.StatusOK {
t.Fatalf("replica NewMultipart status %d: %s", replNewRec.Code, replNewRec.Body.String())
}
var replInit InitiateMultipartUploadResponse
if err = xmlDecoder(replNewRec.Body, &replInit, int64(replNewRec.Body.Len())); err != nil {
t.Fatal(err)
}
replParts := make([]CompletePart, len(rawParts))
for i, raw := range rawParts {
pn := strconv.Itoa(i + 1)
req, err := newTestSignedRequestV4(http.MethodPut,
getPutObjectPartURL("", bucketName, object, replInit.UploadID, pn),
int64(len(raw)), bytes.NewReader(raw), replicator.AccessKey, replicator.SecretKey,
map[string]string{xhttp.MinIOSourceReplicationRequest: "true"})
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("replica PutPart %s status %d: %s", pn, rec.Code, rec.Body.String())
}
replParts[i] = CompletePart{PartNumber: i + 1, ETag: canonicalizeETag(rec.Header()[xhttp.ETag][0])}
}
replCompleteBody, err := xml.Marshal(CompleteMultipartUpload{Parts: replParts})
if err != nil {
t.Fatal(err)
}
srcActualSize, err := sourceInfo.GetActualSize()
if err != nil {
t.Fatal(err)
}
replCompleteHeaders := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.MinIOSourceMTime: sourceInfo.ModTime.Format(time.RFC3339Nano),
xhttp.MinIOSourceETag: sourceInfo.ETag,
xhttp.MinIOReplicationActualObjectSize: strconv.FormatInt(srcActualSize, 10),
}
replCompleteReq, err := newTestSignedRequestV4(http.MethodPost,
getCompleteMultipartUploadURL("", bucketName, object, replInit.UploadID), int64(len(replCompleteBody)),
bytes.NewReader(replCompleteBody), replicator.AccessKey, replicator.SecretKey, replCompleteHeaders)
if err != nil {
t.Fatal(err)
}
replCompleteRec := httptest.NewRecorder()
apiRouter.ServeHTTP(replCompleteRec, replCompleteReq)
if replCompleteRec.Code != http.StatusOK {
t.Fatalf("replica Complete status %d: %s", replCompleteRec.Code, replCompleteRec.Body.String())
}
// ---- 5. Same reads against the replica. ----
repOI, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
t.Logf("[%s] REPLICA parts:", instanceType)
for _, p := range repOI.Parts {
t.Logf(" part %d Size=%d ActualSize=%d", p.Number, p.Size, p.ActualSize)
}
repActual, err := repOI.GetActualSize()
if err != nil {
t.Fatal(err)
}
t.Logf("[%s] REPLICA object Size=%d GetActualSize=%d actual-size-meta=%q",
instanceType, repOI.Size, repActual, repOI.UserDefined[ReservedMetadataPrefix+"actual-size"])
// Whole-object GET must still be byte-identical.
whole := doGet("", nil)
if whole.status != http.StatusOK || !bytes.Equal(whole.body, plaintext) {
t.Errorf("[%s] REPLICA whole-object GET: status=%d len=%d want %d, equal=%v",
instanceType, whole.status, len(whole.body), len(plaintext), bytes.Equal(whole.body, plaintext))
} else {
t.Logf("[%s] REPLICA whole-object GET: OK, %d bytes identical", instanceType, len(whole.body))
}
// Fresh replica parts must record the plaintext lengths, not the
// ciphertext lengths the sender shipped.
if len(repOI.Parts) != len(partLens) {
t.Fatalf("[%s] REPLICA has %d parts, want %d", instanceType, len(repOI.Parts), len(partLens))
}
for i, p := range repOI.Parts {
if p.ActualSize != int64(partLens[i]) {
t.Errorf("[%s] REPLICA part %d ActualSize=%d, want the uploaded length %d",
instanceType, p.Number, p.ActualSize, partLens[i])
}
}
// Expected framing from independent prefix sums of the uploaded lengths.
total := len(plaintext)
start := 0
for i, q := range queries {
wantLen := partLens[i]
wantRange := fmt.Sprintf("bytes %d-%d/%d", start, start+wantLen-1, total)
start += wantLen
got := doGet(q, nil)
want := srcGets[i]
if got.status != want.status || got.status != http.StatusPartialContent {
t.Errorf("[%s] REPLICA GET %s status=%d, source=%d, want 206", instanceType, q, got.status, want.status)
}
if got.clen != strconv.Itoa(wantLen) || got.crange != wantRange {
t.Errorf("[%s] REPLICA GET %s Content-Length=%s Content-Range=%s, want %d and %q",
instanceType, q, got.clen, got.crange, wantLen, wantRange)
}
if want.clen != strconv.Itoa(wantLen) || want.crange != wantRange {
t.Errorf("[%s] SOURCE GET %s Content-Length=%s Content-Range=%s, want %d and %q",
instanceType, q, want.clen, want.crange, wantLen, wantRange)
}
if len(got.body) != wantLen {
t.Errorf("[%s] REPLICA GET %s body is %d bytes, want %d", instanceType, q, len(got.body), wantLen)
}
if !bytes.Equal(got.body, want.body) {
firstDiff := -1
for k := 0; k < len(got.body) && k < len(want.body); k++ {
if got.body[k] != want.body[k] {
firstDiff = k
break
}
}
t.Errorf("[%s] REPLICA partNumber=%d returned DIFFERENT bytes than the source: got %d bytes (Content-Range %q), want %d bytes (Content-Range %q), first differing byte at %d",
instanceType, i+1, len(got.body), got.crange, len(want.body), want.crange, firstDiff)
}
head := doHead(q)
if head.status != http.StatusPartialContent || head.clen != strconv.Itoa(wantLen) || head.crange != wantRange {
t.Errorf("[%s] REPLICA HEAD %s status=%d Content-Length=%s Content-Range=%s, want 206, %d and %q",
instanceType, q, head.status, head.clen, head.crange, wantLen, wantRange)
}
}
repRange := doGet("", map[string]string{"Range": boundaryRange})
sameRange := bytes.Equal(repRange.body, srcRange.body)
t.Logf("[%s] REPLICA GET Range %s -> status=%d Content-Length=%s len(body)=%d | source len=%d | bytes-equal=%v",
instanceType, boundaryRange, repRange.status, repRange.clen, len(repRange.body), len(srcRange.body), sameRange)
if !sameRange {
t.Errorf("[%s] REPLICA boundary Range GET returned different bytes", instanceType)
}
}
// TestSSECReplicaPartActualSizeDataMovement reproduces what a decommission or
// rebalance does to a replica whose parts already carry the ciphertext length in
// ActualSize: it replays the object through the object layer exactly the way
// decommissionObject does (cmd/erasure-server-pool-decom.go:605-667), passing the
// stale ActualSize to PutObjectPart and completing without ReplicationRequest, so
// CompleteMultipartUpload recomputes the object-level actual-size from the sum of
// part ActualSizes (cmd/erasure-multipart.go:1365,1440).
func TestSSECReplicaPartActualSizeDataMovement(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testSSECReplicaPartActualSizeDataMovement,
})
}
func testSSECReplicaPartActualSizeDataMovement(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
key := bytes.Repeat([]byte{0x37}, 32)
keyMD5 := md5.Sum(key)
sseHeaders := map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(keyMD5[:]),
}
const mib = 1024 * 1024
partLens := []int{5 * mib, 5 * mib, 1 * mib}
plaintext := make([]byte, 0, 11*mib)
partData := make([][]byte, len(partLens))
for i, n := range partLens {
b := make([]byte, n)
for j := range b {
b[j] = byte(i*13 + j%241)
}
partData[i] = b
plaintext = append(plaintext, b...)
}
object := "ssec-mp-datamovement"
newRec := httptest.NewRecorder()
newReq, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
apiRouter.ServeHTTP(newRec, newReq)
if newRec.Code != http.StatusOK {
t.Fatalf("NewMultipart status %d: %s", newRec.Code, newRec.Body.String())
}
var init InitiateMultipartUploadResponse
if err = xmlDecoder(newRec.Body, &init, int64(newRec.Body.Len())); err != nil {
t.Fatal(err)
}
srcParts := make([]CompletePart, len(partLens))
for i, b := range partData {
req, err := newTestSignedRequestV4(http.MethodPut,
getPutObjectPartURL("", bucketName, object, init.UploadID, strconv.Itoa(i+1)),
int64(len(b)), bytes.NewReader(b), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("PutPart %d status %d: %s", i+1, rec.Code, rec.Body.String())
}
srcParts[i] = CompletePart{PartNumber: i + 1, ETag: canonicalizeETag(rec.Header()[xhttp.ETag][0])}
}
body, err := xml.Marshal(CompleteMultipartUpload{Parts: srcParts})
if err != nil {
t.Fatal(err)
}
cReq, err := newTestSignedRequestV4(http.MethodPost,
getCompleteMultipartUploadURL("", bucketName, object, init.UploadID), int64(len(body)),
bytes.NewReader(body), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
cRec := httptest.NewRecorder()
apiRouter.ServeHTTP(cRec, cReq)
if cRec.Code != http.StatusOK {
t.Fatalf("Complete status %d: %s", cRec.Code, cRec.Body.String())
}
// Replay it the way decommissionObject does, but hand PutObjectPart the STALE
// ActualSize an already-written SSE-C replica carries: the ciphertext length.
gr, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{},
ObjectOptions{NoDecryption: true, NoLock: true, NoAuditLog: true})
if err != nil {
t.Fatal(err)
}
oi := gr.ObjInfo
res, err := obj.NewMultipartUpload(t.Context(), bucketName, object, ObjectOptions{
UserDefined: oi.UserDefined,
DataMovement: true,
NoAuditLog: true,
})
if err != nil {
gr.Close()
t.Fatal(err)
}
moved := make([]CompletePart, len(oi.Parts))
for i, part := range oi.Parts {
staleActual := part.Size // what a bad replica records
hr, herr := hash.NewReader(t.Context(), io.LimitReader(gr, part.Size), part.Size, "", "", staleActual)
if herr != nil {
gr.Close()
t.Fatal(herr)
}
pi, perr := obj.PutObjectPart(t.Context(), bucketName, object, res.UploadID, part.Number,
NewPutObjReader(hr), ObjectOptions{
PreserveETag: part.ETag,
IndexCB: func() []byte { return part.Index },
NoAuditLog: true,
})
if perr != nil {
gr.Close()
t.Fatalf("data-movement PutObjectPart part %d: %v", part.Number, perr)
}
moved[i] = CompletePart{ETag: pi.ETag, PartNumber: pi.PartNumber}
}
gr.Close()
// decommissionObject/rebalanceObject complete WITHOUT ReplicationRequest, so
// the object-level actual-size is recomputed from the part ActualSizes.
if _, err = obj.CompleteMultipartUpload(t.Context(), bucketName, object, res.UploadID, moved,
ObjectOptions{DataMovement: true, MTime: oi.ModTime, NoAuditLog: true}); err != nil {
t.Fatalf("data-movement CompleteMultipartUpload: %v", err)
}
after, err := obj.GetObjectInfo(t.Context(), bucketName, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
t.Logf("[%s] AFTER DATA MOVEMENT parts:", instanceType)
for _, p := range after.Parts {
t.Logf(" part %d Size=%d ActualSize=%d", p.Number, p.Size, p.ActualSize)
}
gotActual, err := after.GetActualSize()
if err != nil {
t.Fatalf("GetActualSize after data movement: %v", err)
}
t.Logf("[%s] AFTER DATA MOVEMENT object Size=%d GetActualSize=%d actual-size-meta=%q",
instanceType, after.Size, gotActual, after.UserDefined[ReservedMetadataPrefix+"actual-size"])
wantActual := int64(len(plaintext))
if gotActual != wantActual {
t.Errorf("[%s] object-level actual size after data movement = %d, want %d",
instanceType, gotActual, wantActual)
}
for i, p := range after.Parts {
if p.ActualSize != int64(partLens[i]) {
t.Errorf("[%s] part %d ActualSize after data movement = %d, want %d",
instanceType, p.Number, p.ActualSize, partLens[i])
}
}
getReq, err := newTestSignedRequestV4(http.MethodGet, getGetObjectURL("", bucketName, object),
0, nil, credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
getRec := httptest.NewRecorder()
apiRouter.ServeHTTP(getRec, getReq)
if getRec.Code != http.StatusOK || !bytes.Equal(getRec.Body.Bytes(), plaintext) {
t.Errorf("[%s] whole-object GET after data movement: status=%d len=%d want %d equal=%v",
instanceType, getRec.Code, getRec.Body.Len(), len(plaintext), bytes.Equal(getRec.Body.Bytes(), plaintext))
}
// The advertised length must be the body length: a poisoned object-level
// actual-size shows up here as a Content-Length larger than the body.
if clen := getRec.Header().Get(xhttp.ContentLength); clen != strconv.Itoa(len(plaintext)) || clen != strconv.Itoa(getRec.Body.Len()) {
t.Errorf("[%s] whole-object GET after data movement advertises Content-Length=%s for a %d-byte body (plaintext %d)",
instanceType, clen, getRec.Body.Len(), len(plaintext))
}
}
// TestPartNumberToRangeSpecEncryptedParts pins the read-side repair: for an
// encrypted, uncompressed object the part range is derived from the stored
// ciphertext length, so a replica whose parts still record the ciphertext
// length in ActualSize reads correctly, while plaintext and compressed objects
// keep using ActualSize, and a part whose length cannot be a valid encrypted
// stream is reported as tampered by both callers. See pgsty/silo#119.
func TestPartNumberToRangeSpecEncryptedParts(t *testing.T) {
const mib = 1024 * 1024
plain := []int64{5 * mib, 5 * mib, 1024}
cipher := make([]int64, len(plain))
for i, n := range plain {
c, err := sio.EncryptedSize(uint64(n))
if err != nil {
t.Fatal(err)
}
cipher[i] = int64(c)
}
sum := func(v []int64) (s int64) {
for _, n := range v {
s += n
}
return s
}
encMeta := map[string]string{
crypto.MetaSealedKeySSEC: "sealed-key",
crypto.MetaIV: "iv",
crypto.MetaAlgorithm: crypto.InsecureSealAlgorithm,
}
compressedEncMeta := map[string]string{ReservedMetadataPrefix + "compression": compressionAlgorithmV2}
for k, v := range encMeta {
compressedEncMeta[k] = v
}
mkParts := func(sizes, actual []int64) []ObjectPartInfo {
parts := make([]ObjectPartInfo, len(sizes))
for i := range sizes {
parts[i] = ObjectPartInfo{Number: i + 1, Size: sizes[i], ActualSize: actual[i]}
}
return parts
}
// A compressed part's ActualSize is the uploaded length before compression,
// which bears no relation to the ciphertext length: use lengths whose
// decrypted size differs from ActualSize so that dropping the compression
// exclusion is detectable.
uploaded := []int64{2 * plain[0], 2 * plain[1], 2 * plain[2]}
wantRangeOf := func(lens []int64, pn int) (start, end int64) {
for i := 0; i < pn-1; i++ {
start += lens[i]
}
return start, start + lens[pn-1] - 1
}
for _, tc := range []struct {
name string
oi ObjectInfo
lens []int64
}{
{"encrypted, stale ciphertext ActualSize", ObjectInfo{Size: sum(cipher), UserDefined: encMeta, Parts: mkParts(cipher, cipher)}, plain},
{"encrypted, correct ActualSize", ObjectInfo{Size: sum(cipher), UserDefined: encMeta, Parts: mkParts(cipher, plain)}, plain},
{"plaintext", ObjectInfo{Size: sum(plain), UserDefined: map[string]string{}, Parts: mkParts(plain, plain)}, plain},
{"compressed and encrypted keeps ActualSize", ObjectInfo{Size: sum(cipher), UserDefined: compressedEncMeta, Parts: mkParts(cipher, uploaded)}, uploaded},
} {
for pn := 1; pn <= len(plain); pn++ {
rs, err := partNumberToRangeSpec(tc.oi, pn)
if err != nil {
t.Fatalf("%s: partNumber=%d: %v", tc.name, pn, err)
}
start, end := wantRangeOf(tc.lens, pn)
if rs == nil || rs.Start != start || rs.End != end {
t.Errorf("%s: partNumber=%d range %+v, want %d-%d", tc.name, pn, rs, start, end)
}
}
}
// A 31-byte part cannot be a sio stream: both callers report it as tampered.
bad := ObjectInfo{
Size: 31 + cipher[1] + cipher[2],
UserDefined: encMeta,
Parts: mkParts([]int64{31, cipher[1], cipher[2]}, []int64{31, plain[1], plain[2]}),
}
for pn := 1; pn <= 2; pn++ {
if _, err := partNumberToRangeSpec(bad, pn); err != errObjectTampered {
t.Errorf("malformed part: partNumber=%d err=%v, want errObjectTampered", pn, err)
}
}
if _, _, _, err := NewGetObjectReader(nil, bad, ObjectOptions{PartNumber: 1}, http.Header{}); err != errObjectTampered {
t.Errorf("NewGetObjectReader on a malformed part: err=%v, want errObjectTampered", err)
}
if err := setObjectHeaders(t.Context(), httptest.NewRecorder(), bad, nil, ObjectOptions{PartNumber: 1}); err != errObjectTampered {
t.Errorf("setObjectHeaders on a malformed part: err=%v, want errObjectTampered", err)
}
// A zero-length trailing part is a valid (empty) stream and stays accepted.
zero := ObjectInfo{Size: cipher[0], UserDefined: encMeta, Parts: mkParts([]int64{cipher[0], 0}, []int64{cipher[0], 0})}
rs, err := partNumberToRangeSpec(zero, 2)
if err != nil || rs == nil || rs.Start != plain[0] {
t.Errorf("zero-length trailing part: range %+v err=%v, want start %d", rs, err, plain[0])
}
}
// TestAPISSECReplicaMalformedPartIsRejected asserts that a trusted SSE-C replica
// part whose ciphertext length cannot be a valid encrypted stream is rejected
// as tampered before the part is committed. See pgsty/silo#119.
func TestAPISSECReplicaMalformedPartIsRejected(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testAPISSECReplicaMalformedPartIsRejected})
}
func testAPISSECReplicaMalformedPartIsRejected(obj ObjectLayer, instanceType, bucketName string,
apiRouter http.Handler, credentials auth.Credentials, t *testing.T,
) {
previousTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = previousTLS }()
replicator := newObjectAttributesAuthzUser(t, instanceType, bucketName, `"s3:PutObject","s3:GetObject","s3:ReplicateObject"`)
key := bytes.Repeat([]byte{0x45}, 32)
keyMD5 := md5.Sum(key)
sseHeaders := map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(keyMD5[:]),
}
object := "ssec-replica-malformed"
data := bytes.Repeat([]byte("malformed-part-"), 1024)
putReq, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucketName, object), int64(len(data)),
bytes.NewReader(data), credentials.AccessKey, credentials.SecretKey, sseHeaders)
if err != nil {
t.Fatal(err)
}
putRec := httptest.NewRecorder()
apiRouter.ServeHTTP(putRec, putReq)
if putRec.Code != http.StatusOK {
t.Fatalf("%s: source PUT %d: %s", instanceType, putRec.Code, putRec.Body.String())
}
gr, err := obj.GetObjectNInfo(t.Context(), bucketName, object, nil, http.Header{}, ObjectOptions{ReplicationRequest: true})
if err != nil {
t.Fatal(err)
}
sourceInfo := gr.ObjInfo
raw, err := io.ReadAll(gr)
gr.Close()
if err != nil {
t.Fatal(err)
}
replicationOpts, _, err := putReplicationOpts(t.Context(), "", sourceInfo)
if err != nil {
t.Fatal(err)
}
replicationOpts.Internal.SourceMTime = time.Time{}
replicationHeaders := make(map[string]string)
for name, values := range replicationOpts.Header() {
if len(values) > 0 {
replicationHeaders[name] = values[0]
}
}
newReq, err := newTestSignedRequestV4(http.MethodPost, getNewMultipartURL("", bucketName, object),
0, nil, replicator.AccessKey, replicator.SecretKey, replicationHeaders)
if err != nil {
t.Fatal(err)
}
newRec := httptest.NewRecorder()
apiRouter.ServeHTTP(newRec, newReq)
if newRec.Code != http.StatusOK {
t.Fatalf("%s: replica NewMultipart %d: %s", instanceType, newRec.Code, newRec.Body.String())
}
var init InitiateMultipartUploadResponse
if err = xmlDecoder(newRec.Body, &init, int64(newRec.Body.Len())); err != nil {
t.Fatal(err)
}
partHeaders := map[string]string{xhttp.MinIOSourceReplicationRequest: "true"}
// 31 bytes cannot be a sio stream (the package header plus its
// authentication tag occupy 32 bytes).
badReq, err := newTestSignedRequestV4(http.MethodPut, getPutObjectPartURL("", bucketName, object, init.UploadID, "1"),
31, bytes.NewReader(raw[:31]), replicator.AccessKey, replicator.SecretKey, partHeaders)
if err != nil {
t.Fatal(err)
}
badRec := httptest.NewRecorder()
apiRouter.ServeHTTP(badRec, badReq)
if badRec.Code == http.StatusOK || !strings.Contains(badRec.Body.String(), "XMinioObjectTampered") {
t.Fatalf("%s: malformed replica part answered %d: %s", instanceType, badRec.Code, badRec.Body.String())
}
lpi, err := obj.ListObjectParts(t.Context(), bucketName, object, init.UploadID, 0, 10, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if len(lpi.Parts) != 0 {
t.Fatalf("%s: malformed replica part was committed: %+v", instanceType, lpi.Parts)
}
// Control: the real ciphertext still uploads on the same upload.
goodReq, err := newTestSignedRequestV4(http.MethodPut, getPutObjectPartURL("", bucketName, object, init.UploadID, "1"),
int64(len(raw)), bytes.NewReader(raw), replicator.AccessKey, replicator.SecretKey, partHeaders)
if err != nil {
t.Fatal(err)
}
goodRec := httptest.NewRecorder()
apiRouter.ServeHTTP(goodRec, goodReq)
if goodRec.Code != http.StatusOK {
t.Fatalf("%s: valid replica part answered %d: %s", instanceType, goodRec.Code, goodRec.Body.String())
}
lpi, err = obj.ListObjectParts(t.Context(), bucketName, object, init.UploadID, 0, 10, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if len(lpi.Parts) != 1 || lpi.Parts[0].ActualSize != int64(len(data)) {
t.Fatalf("%s: valid replica part recorded %+v, want one part with ActualSize %d", instanceType, lpi.Parts, len(data))
}
}
@@ -157,6 +157,13 @@ func copyPartWithoutChecksumHTTP(t *testing.T, apiRouter http.Handler, creds aut
if sourceRange != "" {
req.Header.Set(xhttp.AmzCopySourceRange, sourceRange)
}
// Re-sign so the copy-source x-amz-* headers are covered by the signature,
// as real S3 clients send them; the verifier rejects unsigned x-amz-*.
if creds.AccessKey != "" && creds.SecretKey != "" {
if err := signRequestV4(req, creds.AccessKey, creds.SecretKey); err != nil {
t.Fatalf("failed to re-sign UploadPartCopy request: %v", err)
}
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
+343 -61
View File
@@ -31,6 +31,7 @@ import (
"sync"
"time"
"github.com/google/uuid"
"github.com/klauspost/readahead"
"github.com/minio/minio-go/v7/pkg/set"
"github.com/minio/minio/internal/config/storageclass"
@@ -44,6 +45,15 @@ import (
"github.com/pgsty/silo-pkg/v3/sync/errgroup"
)
const (
multipartMetaBucket = ReservedMetadataPrefixLower + "multipart-v1-bucket"
multipartMetaObject = ReservedMetadataPrefixLower + "multipart-v1-object"
// ponytail: keep scan concurrency fixed until the multipart-list benchmark
// establishes a better adaptive limit.
multipartMetadataScanConcurrency = 4
)
func (er erasureObjects) getUploadIDDir(bucket, object, uploadID string) string {
uploadUUID := uploadID
uploadBytes, err := base64.RawURLEncoding.DecodeString(uploadID)
@@ -251,14 +261,152 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
})
}
// ListMultipartUploads - lists all the pending multipart
// uploads for a particular object in a bucket.
//
// Implements minimal S3 compatible ListMultipartUploads API. We do
// not support prefix based listing, this is a deliberate attempt
// towards simplification of multipart APIs.
// The resulting ListMultipartsInfo structure is unmarshalled directly as XML.
func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
func multipartUploadInfo(bucket, object, uploadUUID string, fallback time.Time) MultipartInfo {
initiated := fallback
if parsed, ok := multipartUploadTime(uploadUUID); ok {
initiated = parsed
}
return MultipartInfo{
Bucket: bucket,
Object: object,
UploadID: base64.RawURLEncoding.EncodeToString(fmt.Appendf(nil, "%s.%s", globalDeploymentID(), uploadUUID)),
Initiated: initiated,
}
}
// multipartUploadTime is shared by stored records and continuation markers.
// The time is part of the immutable upload ID, so a removed marker still
// identifies the same ordering boundary.
func multipartUploadTime(uploadUUID string) (time.Time, bool) {
if len(uploadUUID) < 38 || uploadUUID[36] != 'x' {
return time.Time{}, false
}
if _, err := uuid.Parse(uploadUUID[:36]); err != nil {
return time.Time{}, false
}
ns, err := strconv.ParseInt(uploadUUID[37:], 10, 64)
if err != nil || ns <= 0 || strconv.FormatInt(ns, 10) != uploadUUID[37:] {
return time.Time{}, false
}
return time.Unix(0, ns), true
}
func multipartMarkerTime(uploadID string) (time.Time, bool) {
b, err := base64.RawURLEncoding.DecodeString(uploadID)
if err != nil {
return time.Time{}, false
}
_, uploadUUID, ok := strings.Cut(string(b), ".")
if !ok {
return time.Time{}, false
}
return multipartUploadTime(uploadUUID)
}
type multipartListEntry struct {
upload *MultipartInfo
commonPrefix string
}
// paginateMultipartUploads applies the S3 ordering, prefix, delimiter, marker,
// and page rules exactly once after all pools and sets have been merged.
func paginateMultipartUploads(uploads []MultipartInfo, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) ListMultipartsInfo {
if maxUploads > maxUploadsList {
maxUploads = maxUploadsList
}
result := ListMultipartsInfo{
MaxUploads: maxUploads,
KeyMarker: keyMarker,
UploadIDMarker: uploadIDMarker,
Prefix: prefix,
Delimiter: delimiter,
}
deduplicated := make([]MultipartInfo, 0, len(uploads))
seenUploads := make(map[string]struct{}, len(uploads))
for _, upload := range uploads {
identity := upload.Bucket + "\x00" + upload.Object + "\x00" + upload.UploadID
if _, ok := seenUploads[identity]; ok {
continue
}
seenUploads[identity] = struct{}{}
deduplicated = append(deduplicated, upload)
}
sort.Slice(deduplicated, func(i, j int) bool {
if deduplicated[i].Object != deduplicated[j].Object {
return deduplicated[i].Object < deduplicated[j].Object
}
if !deduplicated[i].Initiated.Equal(deduplicated[j].Initiated) {
return deduplicated[i].Initiated.Before(deduplicated[j].Initiated)
}
return deduplicated[i].UploadID < deduplicated[j].UploadID
})
markerTime, _ := multipartMarkerTime(uploadIDMarker)
seenPrefixes := make(map[string]struct{})
entries := make([]multipartListEntry, 0, len(deduplicated))
for i := range deduplicated {
upload := &deduplicated[i]
if !strings.HasPrefix(upload.Object, prefix) {
continue
}
if keyMarker != "" {
switch strings.Compare(upload.Object, keyMarker) {
case -1:
continue
case 0:
if uploadIDMarker == "" || upload.Initiated.Before(markerTime) ||
(upload.Initiated.Equal(markerTime) && upload.UploadID <= uploadIDMarker) {
continue
}
}
}
if delimiter != "" {
remainder := strings.TrimPrefix(upload.Object, prefix)
if i := strings.Index(remainder, delimiter); i >= 0 {
commonPrefix := prefix + remainder[:i+len(delimiter)]
if keyMarker != "" && commonPrefix <= keyMarker {
continue
}
if _, ok := seenPrefixes[commonPrefix]; ok {
continue
}
seenPrefixes[commonPrefix] = struct{}{}
entries = append(entries, multipartListEntry{commonPrefix: commonPrefix})
continue
}
}
entries = append(entries, multipartListEntry{upload: upload})
}
if maxUploads <= 0 {
return result
}
pageSize := min(maxUploads, len(entries))
for _, entry := range entries[:pageSize] {
if entry.upload != nil {
result.Uploads = append(result.Uploads, *entry.upload)
continue
}
result.CommonPrefixes = append(result.CommonPrefixes, entry.commonPrefix)
}
result.IsTruncated = pageSize < len(entries)
if result.IsTruncated && pageSize > 0 {
last := entries[pageSize-1]
if last.upload != nil {
result.NextKeyMarker = last.upload.Object
result.NextUploadIDMarker = last.upload.UploadID
} else {
result.NextKeyMarker = last.commonPrefix
}
}
return result
}
// listMultipartUploadsExact preserves the hashed exact-object lookup used by
// multipart write placement and by rolling-upgrade legacy mode.
func (er erasureObjects) listMultipartUploadsExact(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
auditObjectErasureSet(ctx, "ListMultipartUploads", object, &er)
result.MaxUploads = maxUploads
@@ -311,20 +459,16 @@ func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, objec
if populatedUploadIDs.Contains(uploadID) {
continue
}
// If present, use time stored in ID.
startTime := time.Now()
if split := strings.Split(uploadID, "x"); len(split) == 2 {
t, err := strconv.ParseInt(split[1], 10, 64)
if err == nil {
startTime = time.Unix(0, t)
var fallback time.Time
if _, ok := multipartUploadTime(uploadID); !ok {
fi, err := disk.ReadVersion(ctx, bucket, minioMetaMultipartBucket,
pathJoin(er.getMultipartSHADir(bucket, object), uploadID), "", ReadOptions{})
if err != nil {
return result, toObjectErr(err, bucket, object)
}
fallback = fi.ModTime
}
uploads = append(uploads, MultipartInfo{
Bucket: bucket,
Object: object,
UploadID: base64.RawURLEncoding.EncodeToString(fmt.Appendf(nil, "%s.%s", globalDeploymentID(), uploadID)),
Initiated: startTime,
})
uploads = append(uploads, multipartUploadInfo(bucket, object, uploadID, fallback))
populatedUploadIDs.Add(uploadID)
}
@@ -365,6 +509,25 @@ func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, objec
return result, nil
}
func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
uploads, legacy, err := er.scanMultipartUploads(scan, bucket, 0, 0)
if err != nil {
return ListMultipartsInfo{}, err
}
if legacy {
return ListMultipartsInfo{}, errMultipartListingLegacy
}
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
// newMultipartUpload - wrapper for initializing a new multipart
// request; returns a unique upload id.
//
@@ -402,6 +565,8 @@ func (er erasureObjects) newMultipartUpload(ctx context.Context, bucket string,
}
userDefined := cloneMSS(opts.UserDefined)
userDefined[multipartMetaBucket] = bucket
userDefined[multipartMetaObject] = object
if opts.PreserveETag != "" {
userDefined["etag"] = opts.PreserveETag
}
@@ -710,20 +875,26 @@ func (er erasureObjects) PutObjectPart(ctx context.Context, bucket, object, uplo
}
actualSize := data.ActualSize()
if actualSize < 0 {
_, encrypted := crypto.IsEncrypted(fi.Metadata)
compressed := fi.IsCompressed()
switch {
case compressed:
// ... nothing changes for compressed stream.
// if actualSize is -1 we have no known way to
// determine what is the actualSize.
case encrypted:
decSize, err := sio.DecryptedSize(uint64(n))
if err == nil {
actualSize = int64(decSize)
}
default:
_, encrypted := crypto.IsEncrypted(fi.Metadata)
compressed := fi.IsCompressed()
switch {
case compressed:
// ... nothing changes for compressed stream.
// if actualSize is -1 we have no known way to
// determine what is the actualSize.
case encrypted:
// The uploaded length of an encrypted part is always derivable from the
// bytes just written, and the caller's value cannot be trusted: trusted
// SSE-C replication and the data movement paths hand over the ciphertext
// length. Derive it with the arithmetic the read path applies to
// part.Size, so the stored value matches how the part is read back.
decSize, err := sio.DecryptedSize(uint64(n))
if err != nil {
return pi, toObjectErr(errObjectTampered, bucket, object, uploadID)
}
actualSize = int64(decSize)
default:
if actualSize < 0 {
actualSize = n
}
}
@@ -1108,7 +1279,7 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
auditObjectErasureSet(ctx, "CompleteMultipartUpload", object, &er)
}
if opts.CheckPrecondFn != nil {
if opts.CheckPrecondFn != nil || opts.ReplicaLockReconcile {
if !opts.NoLock {
ns := er.NewNSLock(bucket, object)
lkctx, err := ns.GetLock(ctx, globalOperationTimeout)
@@ -1120,18 +1291,24 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
opts.NoLock = true
}
obj, err := er.getObjectInfo(ctx, bucket, object, opts)
if err == nil && opts.CheckPrecondFn(obj) {
return ObjectInfo{}, PreConditionFailed{}
}
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
return ObjectInfo{}, err
}
// The Object Lock reconcile below needs the version being committed, read
// after checkUploadIDExists, so only the precondition read happens here;
// both run under this same write lock, held until the version is renamed
// into place.
if opts.CheckPrecondFn != nil {
obj, err := er.getObjectInfo(ctx, bucket, object, opts)
if err == nil && opts.CheckPrecondFn(obj) {
return ObjectInfo{}, PreConditionFailed{}
}
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
return ObjectInfo{}, err
}
// if object doesn't exist return error for If-Match conditional requests
// If-None-Match should be allowed to proceed for non-existent objects
if err != nil && opts.HasIfMatch && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
return ObjectInfo{}, err
// if object doesn't exist return error for If-Match conditional requests
// If-None-Match should be allowed to proceed for non-existent objects
if err != nil && opts.HasIfMatch && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
return ObjectInfo{}, err
}
}
}
@@ -1143,6 +1320,40 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
return oi, toObjectErr(err, bucket, object, uploadID)
}
// Reconcile against the upload's persisted version, under the object lock.
// Multi-pool callers supply a resolver spanning all pools, including those
// draining their contents; the upload itself stays in its original pool.
if opts.ReplicaLockReconcile {
// A persisted upload records the null version as an empty VersionID; look
// it up as the null version so the reconcile reads the addressed version's
// stored lock, not the latest version's.
lookupVersionID := fi.VersionID
if lookupVersionID == "" {
lookupVersionID = nullVersionID
}
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
curr, gerr := getObjectInfo(ctx, bucket, object, ObjectOptions{
VersionID: lookupVersionID,
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoLock: true,
})
switch {
case gerr == nil:
reconcileStoredObjectLock(fi.Metadata, storedObjectLockState(curr.UserDefined))
reconcileStoredObjectTags(fi.Metadata, curr.UserTags, curr.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
case isErrVersionNotFound(gerr) || isErrObjectNotFound(gerr):
// No existing version to order against: keep the upload's own accepted
// lock, including a pre-upgrade upload that persisted values without
// their ordering timestamps.
default:
return oi, toObjectErr(gerr, bucket, object)
}
}
uploadIDPath := er.getUploadIDDir(bucket, object, uploadID)
onlineDisks := er.getDisks()
writeQuorum := fi.WriteQuorum(er.defaultWQuorum())
@@ -1413,6 +1624,8 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
// Remove superfluous internal headers.
delete(fi.Metadata, hash.MinIOMultipartChecksum)
delete(fi.Metadata, hash.MinIOMultipartChecksumType)
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
// Save the final object size and modtime.
fi.Size = objectSize
@@ -1540,22 +1753,91 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
return fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
}
// AbortMultipartUpload - aborts an ongoing multipart operation
// signified by the input uploadID. This is an atomic operation
// doesn't require clients to initiate multiple such requests.
//
// All parts are purged from all disks and reference to the uploadID
// would be removed from the system, rollback is not possible on this
// operation.
func (er erasureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (err error) {
// abortMultipartUpload retains read-quorum validation and best-effort cleanup
// in legacy mode. Strict mode requires majority deletion acknowledgements and
// permits retrying remnants below read quorum. Neither mode fences creation
// writes still executing after a storage timeout.
func (er erasureObjects) abortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions, legacy bool) (bool, error) {
if !opts.NoAuditLog {
auditObjectErasureSet(ctx, "AbortMultipartUpload", object, &er)
}
// Cleanup all uploaded parts.
defer er.deleteAll(ctx, minioMetaMultipartBucket, er.getUploadIDDir(bucket, object, uploadID))
// Validates if upload ID exists.
_, _, err = er.checkUploadIDExists(ctx, bucket, object, uploadID, false)
return toObjectErr(err, bucket, object, uploadID)
b, err := base64.RawURLEncoding.DecodeString(uploadID)
if err != nil {
return false, MalformedUploadID{UploadID: uploadID}
}
_, internalID, ok := strings.Cut(string(b), ".")
if !ok || internalID == "" || internalID == "." || internalID == ".." || strings.ContainsAny(internalID, "/\\") {
return false, InvalidUploadID{Bucket: bucket, Object: object, UploadID: uploadID}
}
if legacy {
// Keep the released read-quorum and best-effort cleanup behavior.
// The upload ID safety check above applies to both modes.
defer er.deleteAll(ctx, minioMetaMultipartBucket, er.getUploadIDDir(bucket, object, uploadID))
_, _, err := er.checkUploadIDExists(ctx, bucket, object, uploadID, false)
err = toObjectErr(err, bucket, object, uploadID)
if _, absent := err.(InvalidUploadID); absent {
return false, nil
}
return err == nil, err
}
disks := er.getDisks()
uploadPath := er.getUploadIDDir(bucket, object, uploadID)
_, errs := readAllFileInfo(ctx, disks, bucket, minioMetaMultipartBucket, uploadPath, "", false, false)
quorum := er.setDriveCount/2 + 1
found, absent := false, 0
observed := make([]bool, len(disks))
for i, err := range errs {
switch {
case err == nil, errors.Is(err, errFileCorrupt):
found = true
observed[i] = true
case errors.Is(err, errFileNotFound), errors.Is(err, errFileVersionNotFound):
absent++
}
}
if absent >= quorum && !found {
return false, nil
}
if !found {
return false, toObjectErr(errErasureReadQuorum, bucket, object, uploadID)
}
g := errgroup.WithNErrs(len(disks))
for i, disk := range disks {
g.Go(func() error {
if disk == nil {
return errDiskNotFound
}
err := disk.Delete(ctx, minioMetaMultipartBucket, uploadPath, DeleteOptions{Recursive: true, Immediate: false})
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
return nil
}
return err
}, i)
}
deleteErrs := g.Wait()
if err := reduceWriteQuorumErrs(ctx, deleteErrs, nil, quorum); err != nil {
return true, toObjectErr(err, bucket, object, uploadID)
}
if absent >= quorum {
// Missing disks must not mask a failed cleanup of known remnants.
for i, found := range observed {
if found && deleteErrs[i] != nil {
return true, toObjectErr(errErasureWriteQuorum, bucket, object, uploadID)
}
}
}
return true, nil
}
// AbortMultipartUpload cancels an upload using the configured mode. Offline
// part data may still need stale-upload cleanup after its drives return.
func (er erasureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (err error) {
found, err := er.abortMultipartUpload(ctx, bucket, object, uploadID, opts, globalAPIConfig.getMultipartListingLegacy())
if err != nil {
return err
}
if !found {
return InvalidUploadID{Bucket: bucket, Object: object, UploadID: uploadID}
}
return nil
}
@@ -0,0 +1,295 @@
// Copyright (c) 2015-2025 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"net/http"
"testing"
)
// TestDeleteObjectConditional verifies that a conditional DeleteObject
// (If-Match, wired through opts.CheckPrecondFn) is evaluated atomically at the
// object layer: a non-matching ETag must fail with PreConditionFailed and leave
// the object intact, a matching ETag must delete it, and an If-Match against a
// missing object must return a not-found error rather than silently succeeding.
func TestDeleteObjectConditional(t *testing.T) {
ctx := context.Background()
obj, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatal(err)
}
defer obj.Shutdown(context.Background())
defer removeRoots(fsDirs)
bucket := "test-bucket"
object := "test-object"
if err = obj.MakeBucket(ctx, bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
if _, err = obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("test-value")),
int64(len("test-value")), "", ""), ObjectOptions{}); err != nil {
t.Fatal(err)
}
objInfo, err := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
existingETag := objInfo.ETag
// If-Match with a wrong ETag must fail and preserve the object.
t.Run("wrong-etag-precondition-failed", func(t *testing.T) {
opts := ObjectOptions{
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
return !isETagEqual(oi.ETag, "wrong-etag")
},
}
if _, err := obj.DeleteObject(ctx, bucket, object, opts); !isErrPreconditionFailed(err) {
t.Errorf("expected PreConditionFailed, got: %v", err)
}
if _, err := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{}); err != nil {
t.Errorf("object must still exist after a failed conditional delete, got: %v", err)
}
})
// If-Match against a missing object must return a not-found error.
t.Run("missing-object-not-found", func(t *testing.T) {
opts := ObjectOptions{
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
return !isETagEqual(oi.ETag, existingETag)
},
}
_, err := obj.DeleteObject(ctx, bucket, "does-not-exist", opts)
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
t.Errorf("expected ObjectNotFound/VersionNotFound, got: %v", err)
}
})
// If-Match with the correct ETag must delete the object (run last).
t.Run("correct-etag-succeeds", func(t *testing.T) {
opts := ObjectOptions{
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
return !isETagEqual(oi.ETag, existingETag)
},
}
if _, err := obj.DeleteObject(ctx, bucket, object, opts); err != nil {
t.Errorf("expected a successful delete with matching ETag, got: %v", err)
}
if _, err := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{}); !isErrObjectNotFound(err) {
t.Errorf("object must be removed after a matching conditional delete, got: %v", err)
}
})
}
// TestDeleteObjectConditionalWithReadQuorumFailure verifies that a conditional
// (If-Match) DeleteObject does NOT proceed when the object's current state
// cannot be read due to read-quorum loss: without a verified ETag the delete
// must fail rather than remove the object blindly.
func TestDeleteObjectConditionalWithReadQuorumFailure(t *testing.T) {
ctx := context.Background()
obj, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatal(err)
}
defer obj.Shutdown(context.Background())
defer removeRoots(fsDirs)
z := obj.(*erasureServerPools)
xl := z.serverPools[0].sets[0]
bucket := "test-bucket"
object := "test-object"
if err = obj.MakeBucket(ctx, bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
if _, err = obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("test-value")),
int64(len("test-value")), "", ""), ObjectOptions{}); err != nil {
t.Fatal(err)
}
objInfo, err := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
existingETag := objInfo.ETag
// Simulate read-quorum loss by taking 8 of 16 disks offline (EC 8+8).
erasureDisks := xl.getDisks()
z.serverPools[0].erasureDisksMu.Lock()
xl.getDisks = func() []StorageAPI {
for i := range erasureDisks[:8] {
erasureDisks[i] = nil
}
return erasureDisks
}
z.serverPools[0].erasureDisksMu.Unlock()
// Even with the correct ETag we must not delete: the current state (hence the
// ETag) cannot be verified under read-quorum loss.
opts := ObjectOptions{
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
return !isETagEqual(oi.ETag, existingETag)
},
}
if _, err := obj.DeleteObject(ctx, bucket, object, opts); err == nil {
t.Error("expected an error for a conditional delete under read-quorum loss, got nil (object may have been deleted without ETag verification)")
}
}
// TestDeleteObjectConditionalVersioned verifies conditional DeleteObject on a
// versioned bucket, where the precondition is evaluated at the server-pool layer
// against the version that will actually be removed:
// - If-Match "*" when the latest version is a delete marker must fail (412),
// because there is no live object to match.
// - An explicit versionId If-Match is evaluated against the addressed version,
// not the latest one (match deletes it, mismatch is refused).
// - An If-Match against a missing version returns VersionNotFound, which the
// handler maps to NoSuchVersion.
func TestDeleteObjectConditionalVersioned(t *testing.T) {
ctx := context.Background()
obj, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatal(err)
}
defer obj.Shutdown(context.Background())
defer removeRoots(fsDirs)
bucket := "test-bucket"
if err = obj.MakeBucket(ctx, bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
t.Fatal(err)
}
versioned := globalBucketVersioningSys.PrefixEnabled(bucket, "any")
if !versioned {
t.Fatalf("expected versioning to be enabled on %q", bucket)
}
put := func(object, content string) ObjectInfo {
oi, perr := obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte(content)), int64(len(content)), "", ""),
ObjectOptions{Versioned: versioned})
if perr != nil {
t.Fatalf("put %q: %v", object, perr)
}
return oi
}
ifMatch := func(value string) CheckPreconditionFn {
return func(oi ObjectInfo) bool {
return deleteIfMatchPreconditionFailed(http.Header{}, value, oi)
}
}
// If-Match "*" against a delete-marker-latest must fail with 412.
t.Run("wildcard-on-delete-marker-latest", func(t *testing.T) {
object := "dm-object"
put(object, "v1")
// Create a delete marker (unconditional), making the latest a delete marker.
if _, derr := obj.DeleteObject(ctx, bucket, object, ObjectOptions{Versioned: versioned}); derr != nil {
t.Fatalf("create delete marker: %v", derr)
}
opts := ObjectOptions{Versioned: versioned, HasIfMatch: true, CheckPrecondFn: ifMatch("*")}
if _, derr := obj.DeleteObject(ctx, bucket, object, opts); !isErrPreconditionFailed(derr) {
t.Errorf("expected PreConditionFailed for If-Match:* on a delete-marker-latest, got: %v", derr)
}
})
// Explicit versionId is evaluated against the addressed (older) version.
t.Run("explicit-version-selection", func(t *testing.T) {
object := "ver-object"
v1 := put(object, "first")
v2 := put(object, "second-longer") // v2 is now the latest with a different ETag
if v1.ETag == v2.ETag {
t.Fatalf("test setup: versions must have distinct ETags")
}
// Mismatch: delete v2 with v1's ETag must be refused, v2 preserved.
mismatch := ObjectOptions{Versioned: versioned, VersionID: v2.VersionID, HasIfMatch: true, CheckPrecondFn: ifMatch(v1.ETag)}
if _, derr := obj.DeleteObject(ctx, bucket, object, mismatch); !isErrPreconditionFailed(derr) {
t.Errorf("expected PreConditionFailed deleting v2 with v1 ETag, got: %v", derr)
}
if _, gerr := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: v2.VersionID}); gerr != nil {
t.Errorf("v2 must still exist after a refused conditional delete, got: %v", gerr)
}
// Match: delete v1 with v1's ETag must succeed even though v1 is not latest.
match := ObjectOptions{Versioned: versioned, VersionID: v1.VersionID, HasIfMatch: true, CheckPrecondFn: ifMatch(v1.ETag)}
if _, derr := obj.DeleteObject(ctx, bucket, object, match); derr != nil {
t.Errorf("expected the addressed version to be deleted, got: %v", derr)
}
if _, gerr := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: v1.VersionID}); !isErrVersionNotFound(gerr) {
t.Errorf("v1 must be gone after a matching conditional delete, got: %v", gerr)
}
if _, gerr := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: v2.VersionID}); gerr != nil {
t.Errorf("v2 must remain after deleting v1, got: %v", gerr)
}
})
// If-Match against a missing version on an EXISTING key returns VersionNotFound.
t.Run("missing-version", func(t *testing.T) {
object := "missing-version-object"
put(object, "only")
opts := ObjectOptions{Versioned: versioned, VersionID: mustGetUUID(), HasIfMatch: true, CheckPrecondFn: ifMatch("anything")}
if _, derr := obj.DeleteObject(ctx, bucket, object, opts); !isErrVersionNotFound(derr) {
t.Errorf("expected VersionNotFound for If-Match on a missing version, got: %v", derr)
}
})
// If-Match against a missing version on an ABSENT key must also return
// VersionNotFound (NoSuchVersion), not NoSuchKey: the request addresses a
// specific version, which does not exist regardless of the key.
t.Run("missing-version-absent-key", func(t *testing.T) {
opts := ObjectOptions{Versioned: versioned, VersionID: mustGetUUID(), HasIfMatch: true, CheckPrecondFn: ifMatch("anything")}
if _, derr := obj.DeleteObject(ctx, bucket, "never-existed", opts); !isErrVersionNotFound(derr) {
t.Errorf("expected VersionNotFound for If-Match on a version of an absent key, got: %v", derr)
}
})
// If-Match "*" addressing a delete-marker VERSION by id must fail with 412,
// not 405: a delete marker has no entity-tag to match. getObjectInfo returns
// the marker alongside MethodNotAllowed; the precondition runs on the marker.
t.Run("wildcard-on-explicit-delete-marker-version", func(t *testing.T) {
object := "explicit-dm-object"
put(object, "live")
dm, derr := obj.DeleteObject(ctx, bucket, object, ObjectOptions{Versioned: versioned})
if derr != nil {
t.Fatalf("create delete marker: %v", derr)
}
if !dm.DeleteMarker || dm.VersionID == "" {
t.Fatalf("expected a delete-marker version, got DeleteMarker=%v VersionID=%q", dm.DeleteMarker, dm.VersionID)
}
opts := ObjectOptions{Versioned: versioned, VersionID: dm.VersionID, HasIfMatch: true, CheckPrecondFn: ifMatch("*")}
if _, derr := obj.DeleteObject(ctx, bucket, object, opts); !isErrPreconditionFailed(derr) {
t.Errorf("expected PreConditionFailed for If-Match:* on an addressed delete-marker version, got: %v", derr)
}
})
}
+32 -8
View File
@@ -133,6 +133,11 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
return fi.ToObjectInfo(srcBucket, srcObject, srcOpts.Versioned || srcOpts.VersionSuspended), toObjectErr(errMethodNotAllowed, srcBucket, srcObject)
}
if dstOpts.ReplicaLockReconcile {
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(fi.Metadata))
reconcileStoredObjectTags(srcInfo.UserDefined, fi.Metadata[xhttp.AmzObjectTagging], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
}
filterOnlineDisksInplace(fi, metaArr, onlineDisks)
versionID := srcInfo.VersionID
@@ -1268,7 +1273,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
data := r.Reader
if opts.CheckPrecondFn != nil {
if opts.CheckPrecondFn != nil || opts.ReplicaLockReconcile {
if !opts.NoLock {
ns := er.NewNSLock(bucket, object)
lkctx, err := ns.GetLock(ctx, globalOperationTimeout)
@@ -1280,18 +1285,33 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
opts.NoLock = true
}
obj, err := er.getObjectInfo(ctx, bucket, object, opts)
if err == nil && opts.CheckPrecondFn(obj) {
return objInfo, PreConditionFailed{}
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
obj, err := getObjectInfo(ctx, bucket, object, opts)
// A destination read that fails for a reason other than not-found must not
// be taken as a passed precondition or as absent lock state.
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
return objInfo, err
}
if opts.CheckPrecondFn != nil {
if err == nil && opts.CheckPrecondFn(obj) {
return objInfo, PreConditionFailed{}
}
// if object doesn't exist return error for If-Match conditional requests
// If-None-Match should be allowed to proceed for non-existent objects
if err != nil && opts.HasIfMatch && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
return objInfo, err
}
}
// if object doesn't exist return error for If-Match conditional requests
// If-None-Match should be allowed to proceed for non-existent objects
if err != nil && opts.HasIfMatch && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
return objInfo, err
// Re-read under the write lock, using the pools-layer resolver when
// present. A missing version keeps the incoming accepted state; an
// existing version contributes independently ordered lock and tags.
if opts.ReplicaLockReconcile && err == nil {
reconcileStoredObjectLock(opts.UserDefined, storedObjectLockState(obj.UserDefined))
reconcileStoredObjectTags(opts.UserDefined, obj.UserTags, obj.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
}
}
@@ -2307,7 +2327,11 @@ func (er erasureObjects) PutObjectTags(ctx context.Context, bucket, object strin
fi.Metadata[xhttp.AmzObjectTagging] = tags
fi.ReplicationState = opts.PutReplicationState()
stamp := monotonicTaggingTimestamp(opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
maps.Copy(fi.Metadata, opts.UserDefined)
if stamp != "" {
fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
}
if err = er.updateObjectMeta(ctx, bucket, object, fi, onlineDisks); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
+392
View File
@@ -0,0 +1,392 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"maps"
"sort"
"strings"
"time"
madmin "github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/bucket/lifecycle"
xhttp "github.com/minio/minio/internal/http"
)
// objectPoolInfos reads the addressed version in every pool, including pools
// draining their contents. The caller must hold the pools-layer object lock.
// An unreadable pool may contain a newer version or metadata; it is not absence.
func (z *erasureServerPools) objectPoolInfos(ctx context.Context, bucket, object string, opts ObjectOptions) ([]PoolObjInfo, error) {
opts.NoLock = true
opts.CheckPrecondFn = nil
var copies []PoolObjInfo
for i, pool := range z.serverPools {
oi, err := pool.GetObjectInfo(ctx, bucket, object, opts)
if err == nil || (oi.DeleteMarker && (isErrObjectNotFound(err) || isErrMethodNotAllowed(err))) {
copies = append(copies, PoolObjInfo{Index: i, ObjInfo: oi})
continue
}
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return nil, err
}
}
sort.Slice(copies, func(i, j int) bool {
a, b := copies[i], copies[j]
if a.ObjInfo.ModTime.Equal(b.ObjInfo.ModTime) {
return a.Index < b.Index
}
return a.ObjInfo.ModTime.After(b.ObjInfo.ModTime)
})
if len(copies) == 0 {
if opts.VersionID != "" {
return nil, VersionNotFound{Bucket: bucket, Object: decodeDirObject(object), VersionID: opts.VersionID}
}
return nil, ObjectNotFound{Bucket: bucket, Object: decodeDirObject(object)}
}
return copies, nil
}
// Healing also commits object metadata. Both queued healing and drive healing
// must use the same namespace as pooled writes, rather than racing them under
// a destination set's independent namespace.
func (z *erasureServerPools) healObjectInPool(ctx context.Context, er *erasureObjects, bucket, object, versionID string, opts madmin.HealOpts) (madmin.HealResultItem, error) {
if !z.SinglePool() && !opts.NoLock {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
return madmin.HealResultItem{}, err
}
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
opts.NoLock = true
}
return er.HealObject(ctx, bucket, object, versionID, opts)
}
// mergePoolLockState resolves each independently ordered field. An ordered
// removal is a value in its own right; an absent unordered field is not one.
func mergePoolLockState(copies []PoolObjInfo) objectLockState {
state := storedObjectLockState(copies[0].ObjInfo.UserDefined)
for _, copy := range copies[1:] {
next := storedObjectLockState(copy.ObjInfo.UserDefined)
retentionTime, _ := time.Parse(time.RFC3339Nano, next.retentionTimestamp)
if state.retentionIsOlderThan(retentionTime) || (state.retentionTimestamp == "" && state.mode == "" && next.mode != "") {
state.mode, state.retainUntil, state.retentionTimestamp = next.mode, next.retainUntil, next.retentionTimestamp
}
holdTime, _ := time.Parse(time.RFC3339Nano, next.legalHoldTimestamp)
if state.legalHoldIsOlderThan(holdTime) || (state.legalHoldTimestamp == "" && state.legalHold == "" && next.legalHold != "") {
state.legalHold, state.legalHoldTimestamp = next.legalHold, next.legalHoldTimestamp
}
}
return state
}
func replaceObjectLockMetadata(metadata map[string]string, state objectLockState) {
for _, key := range []string{
strings.ToLower(xhttp.AmzObjectLockMode), strings.ToLower(xhttp.AmzObjectLockRetainUntilDate),
strings.ToLower(xhttp.AmzObjectLockLegalHold),
ReservedMetadataPrefixLower + ObjectLockRetentionTimestamp,
ReservedMetadataPrefixLower + ObjectLockLegalHoldTimestamp,
} {
// Metadata updates use maps.Copy at the set layer. Empty values also
// clear a previous value there, including timestamp-only removals.
if _, exists := metadata[key]; exists {
metadata[key] = ""
}
}
state.restoreRetention(metadata)
state.restoreLegalHold(metadata)
}
func mergedPoolObjectInfo(copies []PoolObjInfo) ObjectInfo {
oi := copies[0].ObjInfo
oi.UserDefined = maps.Clone(oi.UserDefined)
if oi.UserDefined == nil {
oi.UserDefined = make(map[string]string)
}
replaceObjectLockMetadata(oi.UserDefined, mergePoolLockState(copies))
for _, copy := range copies[1:] {
ts := copy.ObjInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
stamp, _ := time.Parse(time.RFC3339Nano, ts)
if olderThan(oi.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], stamp) {
oi.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = ts
oi.UserTags = copy.ObjInfo.UserTags
}
}
return oi
}
func (z *erasureServerPools) replicaObjectInfo(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
if copies[0].ObjInfo.DeleteMarker {
return copies[0].ObjInfo, MethodNotAllowed{Bucket: bucket, Object: decodeDirObject(object), VersionID: opts.VersionID}
}
return mergedPoolObjectInfo(copies), nil
}
// metadataPoolInfos resolves an unqualified metadata request to one logical
// version before collecting its copies, rather than mixing different versions.
func (z *erasureServerPools) metadataPoolInfos(ctx context.Context, bucket, object string, opts ObjectOptions) ([]PoolObjInfo, error) {
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
if err != nil {
return nil, err
}
if copies[0].ObjInfo.DeleteMarker {
return nil, MethodNotAllowed{Bucket: bucket, Object: decodeDirObject(object), VersionID: opts.VersionID}
}
if opts.VersionID == "" {
opts.VersionID = copies[0].ObjInfo.VersionID
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
return z.objectPoolInfos(ctx, bucket, object, opts)
}
return copies, nil
}
func (z *erasureServerPools) updatePoolMetadata(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
copies, err := z.metadataPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
updated := mergedPoolObjectInfo(copies)
before := maps.Clone(updated.UserDefined)
if opts.EvalMetadataFn != nil {
if _, err := opts.EvalMetadataFn(&updated, nil); err != nil {
return ObjectInfo{}, err
}
}
changes := make(map[string]string)
for key, value := range updated.UserDefined {
if prior, ok := before[key]; !ok || prior != value {
changes[key] = value
}
}
for key := range before {
if _, ok := updated.UserDefined[key]; !ok {
changes[key] = ""
}
}
// Metadata callbacks may explicitly replace tags in the raw write map.
// Otherwise retain the merged value from the ObjectInfo read model.
tags, ok := updated.UserDefined[xhttp.AmzObjectTagging]
if !ok {
tags = updated.UserTags
}
state := storedObjectLockState(updated.UserDefined)
opts.VersionID = updated.VersionID
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
opts.NoLock = true
opts.EvalMetadataFn = func(oi *ObjectInfo, _ error) (ReplicateDecision, error) {
maps.Copy(oi.UserDefined, changes)
replaceObjectLockMetadata(oi.UserDefined, state)
// Reassemble the tag value and its ordering timestamp for storage.
if tags != "" || oi.UserTags != "" {
oi.UserDefined[xhttp.AmzObjectTagging] = tags
}
key := ReservedMetadataPrefixLower + TaggingTimestamp
if value, exists := updated.UserDefined[key]; exists || oi.UserDefined[key] != "" {
oi.UserDefined[key] = value
}
return ReplicateDecision{}, nil
}
var primary ObjectInfo
for _, copy := range copies {
oi, err := z.serverPools[copy.Index].PutObjectMetadata(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
if copy.Index == copies[0].Index {
primary = oi
}
}
return primary, nil
}
// Pass the stored tag value explicitly: ObjectInfo.UserDefined excludes it,
// whereas FileInfo.Metadata retains the raw storage key.
func reconcileStoredObjectTags(metadata map[string]string, storedTags, storedTimestamp string) {
key := ReservedMetadataPrefixLower + TaggingTimestamp
stamp, err := time.Parse(time.RFC3339Nano, storedTimestamp)
if err != nil {
return
}
incoming, err := time.Parse(time.RFC3339Nano, metadata[key])
if err != nil || !stamp.Before(incoming) {
metadata[key] = storedTimestamp
metadata[xhttp.AmzObjectTagging] = storedTags
}
}
// Local tagging mutations must advance the revision they overwrite, even when
// a request's clock or lock acquisition order is behind the stored revision.
// Replica writes use reconcileStoredObjectTags instead of minting a revision.
func monotonicTaggingTimestamp(incoming, stored string) string {
requested, err := time.Parse(time.RFC3339Nano, incoming)
if err != nil {
return incoming
}
current, err := time.Parse(time.RFC3339Nano, stored)
if err != nil || requested.After(current) {
return incoming
}
return current.Add(time.Nanosecond).UTC().Format(time.RFC3339Nano)
}
// A restored version still owns its tier reference even while IsRemote is
// false. Only the last copy of a reference may schedule its contents for GC.
func sharesTierObject(oi ObjectInfo, copies []PoolObjInfo) bool {
ref := oi.TransitionedObject
if ref.Status != lifecycle.TransitionComplete {
return false
}
for _, copy := range copies {
other := copy.ObjInfo.TransitionedObject
if other.Status == lifecycle.TransitionComplete && ref.Tier == other.Tier && ref.Name == other.Name && ref.VersionID == other.VersionID {
return true
}
}
return false
}
// retireReplicaCopies runs only after committing a replacement. Failures are
// returned to the caller, so a stale copy cannot be hidden behind a successful
// response. Data movement owns its source cleanup and does not use this helper.
func (z *erasureServerPools) retireReplicaCopies(ctx context.Context, bucket, object string, keep int, oi ObjectInfo) error {
versionID := oi.VersionID
if versionID == "" {
versionID = nullVersionID
}
copies, err := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{VersionID: versionID})
if err != nil {
return err
}
var retained []PoolObjInfo
for _, copy := range copies {
if copy.Index == keep {
retained = []PoolObjInfo{copy}
break
}
}
if len(retained) == 0 {
return VersionNotFound{Bucket: bucket, Object: decodeDirObject(object), VersionID: versionID}
}
for i, copy := range copies {
if copy.Index == keep {
continue
}
_, err := z.serverPools[copy.Index].DeleteObject(ctx, bucket, object,
ObjectOptions{
VersionID: versionID, NoLock: true, NoAuditLog: true,
SkipFreeVersion: sharesTierObject(copy.ObjInfo, retained) || sharesTierObject(copy.ObjInfo, copies[i+1:]),
})
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return err
}
}
return nil
}
// deleteObjectReconciled evaluates any condition once against the logical
// version, then removes all its copies under the same lock as pooled writers.
func (z *erasureServerPools) deleteObjectReconciled(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
primary := copies[0]
if opts.CheckPrecondFn != nil && opts.CheckPrecondFn(primary.ObjInfo) {
return ObjectInfo{}, PreConditionFailed{}
}
opts.CheckPrecondFn = nil
opts.NoLock = true
if opts.EvalRetentionBypassFn != nil || opts.EvalMetadataFn != nil {
logical := primary.ObjInfo
var gerr error
switch {
case logical.DeleteMarker:
// Markers can be deleted by version ID. Match the set layer's
// callback inputs instead of rejecting them as metadata updates.
gerr = toObjectErr(errMethodNotAllowed, bucket, object)
if opts.VersionID == "" || opts.DeleteMarker {
gerr = toObjectErr(errFileNotFound, bucket, object)
}
case opts.VersionID != "":
// An addressed version already resolved every copy above.
logical = mergedPoolObjectInfo(copies)
default:
versions, err := z.metadataPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
logical = mergedPoolObjectInfo(versions)
}
// Keep the retention gate first. These callbacks independently evaluate
// the logical version and run once before any deletion; the handler only
// sweeps metadata's transition state after a successful delete.
if opts.EvalRetentionBypassFn != nil {
if err := opts.EvalRetentionBypassFn(logical, gerr); err != nil {
return ObjectInfo{}, err
}
opts.EvalRetentionBypassFn = nil
}
if opts.EvalMetadataFn != nil {
decision, err := opts.EvalMetadataFn(&logical, gerr)
if err != nil {
return ObjectInfo{}, err
}
if decision.ReplicateAny() {
opts.SetDeleteReplicationState(decision, opts.VersionID)
}
opts.EvalMetadataFn = nil
}
}
if opts.VersionID == "" && (opts.Versioned || opts.VersionSuspended) {
// A single new delete marker hides the current version. Older versions
// remain history and must not be removed by an unqualified DELETE.
oi, err := z.serverPools[primary.Index].DeleteObject(ctx, bucket, object, opts)
oi.Name = decodeDirObject(object)
oi.replicationDecision = opts.DeleteReplication.ReplicateDecisionStr
return oi, err
}
// Retire non-authoritative copies first. If cleanup fails, retain the
// authoritative version and report the error instead of acknowledging a
// deletion that would expose an older copy.
for i := 1; i < len(copies); i++ {
candidate := copies[i]
deleteOpts := opts
deleteOpts.SkipFreeVersion = opts.SkipFreeVersion || sharesTierObject(candidate.ObjInfo, copies[:1]) || sharesTierObject(candidate.ObjInfo, copies[i+1:])
_, err := z.serverPools[candidate.Index].DeleteObject(ctx, bucket, object, deleteOpts)
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return ObjectInfo{}, err
}
}
oi, err := z.serverPools[primary.Index].DeleteObject(ctx, bucket, object, opts)
oi.Name = decodeDirObject(object)
oi.replicationDecision = opts.DeleteReplication.ReplicateDecisionStr
return oi, err
}
File diff suppressed because it is too large Load Diff
+5 -1
View File
@@ -23,6 +23,10 @@ import (
)
func prepareErasurePools() (ObjectLayer, []string, error) {
return prepareErasurePoolsWithContext(context.Background())
}
func prepareErasurePoolsWithContext(ctx context.Context) (ObjectLayer, []string, error) {
nDisks := 32
fsDirs, err := getRandomDisks(nDisks)
if err != nil {
@@ -32,7 +36,7 @@ func prepareErasurePools() (ObjectLayer, []string, error) {
pools := mustGetPoolEndpoints(0, fsDirs[:16]...)
pools = append(pools, mustGetPoolEndpoints(1, fsDirs[16:]...)...)
objLayer, _, err := initObjectLayer(context.Background(), pools)
objLayer, _, err := initObjectLayer(ctx, pools)
if err != nil {
removeRoots(fsDirs)
return nil, nil, err
@@ -0,0 +1,172 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"testing"
"time"
)
// A multipart completion's If-Match must be evaluated against the logical
// latest object across pools, not against the copy local to a pool.
//
// Multi-pool write placement is not sticky (getPoolIdx picks by available
// space even for existing objects), so an upload and a newer overwrite of
// the same name routinely end up in different pools. Uploads are pinned to
// their pools directly: routing through z.NewMultipartUpload would make the
// placement depend on the space-weighted random choice.
func TestPoolsMultipartConditionalUsesLogicalLatest(t *testing.T) {
z, bucket := consistencyPools(t)
ctx := t.Context()
ifMatch := func(etag string) (opts ObjectOptions) {
return ObjectOptions{
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
return oi.ETag != etag
},
}
}
uploadPart := func(t *testing.T, bucket, object, uploadID string) []CompletePart {
t.Helper()
pi, err := z.PutObjectPart(ctx, bucket, object, uploadID, 1,
mustGetPutObjReader(t, bytes.NewBufferString("part"), 4, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
return []CompletePart{{PartNumber: 1, ETag: pi.ETag}}
}
// Scenario A: the uploaded If-Match carries the stale ETag of the pool-0
// copy while the logical latest object lives in pool 1. The completion
// must fail with 412 instead of shadowing the newer logical state.
objectA := "cond-mp-stale-etag"
base := time.Now()
oldA := putConsistencyObject(t, z, bucket, objectA, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
mpA, err := z.serverPools[0].NewMultipartUpload(ctx, bucket, objectA, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
newerA := putConsistencyObject(t, z, bucket, objectA, 1, "new", ObjectOptions{
MTime: base.Add(-time.Minute),
})
latest, _, err := z.getLatestObjectInfoWithIdx(ctx, bucket, objectA, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if latest.ETag != newerA.ETag {
t.Fatalf("logical latest should be the pool-1 copy: got %s want %s", latest.ETag, newerA.ETag)
}
if _, err = z.CompleteMultipartUpload(ctx, bucket, objectA, mpA.UploadID,
uploadPart(t, bucket, objectA, mpA.UploadID), ifMatch(oldA.ETag)); err == nil {
t.Fatal("If-Match with the stale pool-0 ETag must not complete over the newer pool-1 object")
} else if _, ok := err.(PreConditionFailed); !ok {
t.Fatalf("expected PreconditionFailed, got %v", err)
}
if latest, _, err = z.getLatestObjectInfoWithIdx(ctx, bucket, objectA, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if latest.ETag != newerA.ETag {
t.Fatalf("the newer pool-1 object must remain the logical latest, got %s", latest.ETag)
}
// Scenario B: the uploaded If-Match carries the logical latest ETag (the
// pool-1 copy) while the upload sits next to the stale pool-0 copy. The
// precondition is satisfied, so completion must succeed and its result
// must become the logical latest.
objectB := "cond-mp-latest-etag"
putConsistencyObject(t, z, bucket, objectB, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
mpB, err := z.serverPools[0].NewMultipartUpload(ctx, bucket, objectB, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
newerB := putConsistencyObject(t, z, bucket, objectB, 1, "new", ObjectOptions{
MTime: base.Add(-time.Minute),
})
oiB, err := z.CompleteMultipartUpload(ctx, bucket, objectB, mpB.UploadID,
uploadPart(t, bucket, objectB, mpB.UploadID), ifMatch(newerB.ETag))
if err != nil {
t.Fatalf("If-Match with the logical latest ETag must complete, got %v", err)
}
if oiB.ETag == "" {
t.Fatal("completion returned an empty ETag")
}
if latest, _, err = z.getLatestObjectInfoWithIdx(ctx, bucket, objectB, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if latest.ETag != oiB.ETag {
t.Fatalf("the completed object must be the logical latest: got %s want %s", latest.ETag, oiB.ETag)
}
// Scenario C: the upload lives in pool 1 with the newer copy while pool 0
// holds the stale one. A set-local evaluation order would let pool 0's
// stale copy fail the request before pool 1 is reached; the logical
// latest ETag must complete.
objectC := "cond-mp-upload-other-pool"
putConsistencyObject(t, z, bucket, objectC, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
mpC, err := z.serverPools[1].NewMultipartUpload(ctx, bucket, objectC, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
newerC := putConsistencyObject(t, z, bucket, objectC, 1, "new", ObjectOptions{
MTime: base.Add(-time.Minute),
})
if _, err = z.CompleteMultipartUpload(ctx, bucket, objectC, mpC.UploadID,
uploadPart(t, bucket, objectC, mpC.UploadID), ifMatch(newerC.ETag)); err != nil {
t.Fatalf("If-Match with the logical latest ETag must complete regardless of upload pool, got %v", err)
}
// Scenario D: pool 1 holds the newer copy but cannot be read. An
// unreadable pool may contain the newest state, so the unverifiable
// condition must fail the request rather than pass it against pool 0's
// stale ETag.
objectD := "cond-mp-unreadable-pool"
oldD := putConsistencyObject(t, z, bucket, objectD, 0, "old", ObjectOptions{MTime: base.Add(-2 * time.Minute)})
mpD, err := z.serverPools[0].NewMultipartUpload(ctx, bucket, objectD, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
newerD := putConsistencyObject(t, z, bucket, objectD, 1, "new", ObjectOptions{
MTime: base.Add(-time.Minute),
})
if latest, _, err = z.getLatestObjectInfoWithIdx(ctx, bucket, objectD, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if latest.ETag != newerD.ETag {
t.Fatalf("logical latest before faulting pool 1 should be its copy: got %s want %s", latest.ETag, newerD.ETag)
}
set := z.serverPools[1].getHashedSet(objectD)
getDisks := set.getDisks
faulty := append([]StorageAPI(nil), getDisks()...)
for i := range faulty {
faulty[i] = consistencyReadFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: objectD}
}
set.getDisks = func() []StorageAPI { return faulty }
defer func() { set.getDisks = getDisks }()
_, err = z.CompleteMultipartUpload(ctx, bucket, objectD, mpD.UploadID,
uploadPart(t, bucket, objectD, mpD.UploadID), ifMatch(oldD.ETag))
if !isErrReadQuorum(err) {
t.Fatalf("expected an insufficient read quorum error, got %v", err)
}
}
@@ -0,0 +1,341 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
xhttp "github.com/minio/minio/internal/http"
)
func multipartConditionRequest(t *testing.T, router http.Handler, method, target, body string, headers map[string]string) *httptest.ResponseRecorder {
t.Helper()
req, err := newTestSignedRequestV4(method, target, int64(len(body)), strings.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
return rec
}
// The server writes the wire spelling ETag directly into Header; unlike a
// network response, httptest's map has not canonicalized it to Etag.
func multipartConditionResponseETag(rec *httptest.ResponseRecorder) string {
for key, values := range rec.Header() {
if strings.EqualFold(key, xhttp.ETag) && len(values) > 0 {
return strings.Trim(values[0], "\"")
}
}
return ""
}
func multipartConditionUpload(t *testing.T, z *erasureServerPools, bucket, object string, owner int, opts ObjectOptions) (string, []CompletePart) {
t.Helper()
mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, object, opts)
if err != nil {
t.Fatal(err)
}
part, err := z.serverPools[owner].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
return mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}
}
func multipartConditionCompleteBody(parts []CompletePart) string {
data, err := xml.Marshal(CompleteMultipartUpload{Parts: parts})
if err != nil {
panic(err)
}
return string(data)
}
func multipartConditionError(t *testing.T, rec *httptest.ResponseRecorder, code string) {
t.Helper()
decoder := xml.NewDecoder(strings.NewReader(rec.Body.String()))
var response APIErrorResponse
if err := decoder.Decode(&response); err != nil || response.Code != code {
t.Fatalf("expected %s error, got %q: %v", code, rec.Body.String(), err)
}
if err := decoder.Decode(&response); err != io.EOF {
t.Fatalf("expected exactly one error response, got %q: %v", rec.Body.String(), err)
}
}
// State is deliberately placed per pool; the final request uses signed HTTP
// and the real handler, precondition callback, erasure metadata and rename.
func TestPoolsMultipartConditionalHTTPMatrix(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
for owner := range 2 {
for _, localOld := range []bool{false, true} {
for _, condition := range []string{"match-old", "match-current", "none-match"} {
t.Run(fmt.Sprintf("owner=%d/old=%t/%s", owner, localOld, condition), func(t *testing.T) {
object := fmt.Sprintf("http-%d-%t-%s", owner, localOld, condition)
oldETag := "old-does-not-exist"
if localOld {
oldETag = putConsistencyObject(t, z, bucket, object, owner, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)}).ETag
}
id, parts := multipartConditionUpload(t, z, bucket, object, owner, ObjectOptions{})
current := putConsistencyObject(t, z, bucket, object, 1-owner, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
head := multipartConditionRequest(t, router, http.MethodHead, getGetObjectURL("", bucket, object), "", nil)
if head.Code != 200 || multipartConditionResponseETag(head) != current.ETag {
t.Fatalf("bad HEAD: %d %v", head.Code, head.Header())
}
h := map[string]string{xhttp.IfMatch: "\"" + oldETag + "\""}
want := 412
if condition == "match-current" {
h[xhttp.IfMatch] = "\"" + current.ETag + "\""
want = 200
}
if condition == "none-match" {
h = map[string]string{xhttp.IfNoneMatch: "*"}
}
rec := multipartConditionRequest(t, router, http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, id), multipartConditionCompleteBody(parts), h)
t.Logf("HEAD current=%s, requested=%v, complete HTTP=%d", current.ETag, h, rec.Code)
if rec.Code != want {
t.Errorf("want HTTP %d, got %d: %s", want, rec.Code, rec.Body.String())
}
if want == 412 {
multipartConditionError(t, rec, "PreconditionFailed")
got := multipartConditionRequest(t, router, http.MethodGet, getGetObjectURL("", bucket, object), "", nil)
if got.Code != 200 || got.Body.String() != "current" {
t.Errorf("rejected request must preserve current data: %d %q", got.Code, got.Body.String())
}
if _, err := z.serverPools[owner].ListObjectParts(t.Context(), bucket, object, id, 0, 10, ObjectOptions{}); err != nil {
t.Errorf("rejected request consumed upload: %v", err)
}
}
})
}
}
}
}
func TestPoolsMultipartConditionalHTTPAbsentObject(t *testing.T) {
for _, deleted := range []bool{false, true} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("delete-marker=%t/if-match=%t", deleted, match), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: deleted})
if err != nil {
t.Fatal(err)
}
object := "http-absent"
if deleted {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
_, err = z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)})
if err != nil {
t.Fatal(err)
}
}
id, parts := multipartConditionUpload(t, z, bucket, object, 0, ObjectOptions{Versioned: deleted})
headers := map[string]string{xhttp.IfNoneMatch: "*"}
want := http.StatusOK
if match {
headers = map[string]string{xhttp.IfMatch: "\"missing\""}
want = http.StatusNotFound
}
rec := multipartConditionRequest(t, router, http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, id), multipartConditionCompleteBody(parts), headers)
if rec.Code != want {
t.Fatalf("expected HTTP %d, got %d: %s", want, rec.Code, rec.Body.String())
}
if match {
multipartConditionError(t, rec, "NoSuchKey")
if _, err := z.serverPools[0].ListObjectParts(t.Context(), bucket, object, id, 0, 10, ObjectOptions{}); err != nil {
t.Errorf("rejected request consumed upload: %v", err)
}
}
})
}
}
}
// All writes below use ordinary signed S3 requests. The result must be correct
// for every placement; the matrix above deterministically covers split pools.
func TestPoolsMultipartConditionalHTTPNormalRouting(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
object := "normal-routing"
url := getPutObjectURL("", bucket, object)
old := multipartConditionRequest(t, router, http.MethodPut, url, "old-data", nil)
if old.Code != http.StatusOK {
t.Fatalf("initial PUT %d: %s", old.Code, old.Body.String())
}
init := multipartConditionRequest(t, router, http.MethodPost, url+"?uploads", "", nil)
if init.Code != http.StatusOK {
t.Fatalf("init %d: %s", init.Code, init.Body.String())
}
var mp InitiateMultipartUploadResponse
if err := xml.Unmarshal(init.Body.Bytes(), &mp); err != nil {
t.Fatal(err)
}
part := multipartConditionRequest(t, router, http.MethodPut, getPutObjectPartURL("", bucket, object, mp.UploadID, "1"), "replacement", nil)
if part.Code != http.StatusOK {
t.Fatalf("part %d: %s", part.Code, part.Body.String())
}
parts := []CompletePart{{PartNumber: 1, ETag: multipartConditionResponseETag(part)}}
newer := multipartConditionRequest(t, router, http.MethodPut, url, "newer-data", nil)
if newer.Code != http.StatusOK {
t.Fatalf("new PUT %d: %s", newer.Code, newer.Body.String())
}
head := multipartConditionRequest(t, router, http.MethodHead, url, "", nil)
if head.Code != http.StatusOK || multipartConditionResponseETag(head) != multipartConditionResponseETag(newer) {
t.Fatalf("HEAD did not pick new object: %d %v", head.Code, head.Header())
}
rec := multipartConditionRequest(t, router, http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, mp.UploadID), multipartConditionCompleteBody(parts), map[string]string{xhttp.IfMatch: "\"" + multipartConditionResponseETag(old) + "\""})
if rec.Code != http.StatusPreconditionFailed {
t.Errorf("stale If-Match should be HTTP 412, got %d: %s", rec.Code, rec.Body.String())
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "newer-data" {
t.Errorf("conditional completion changed newer data: %d %q", get.Code, get.Body.String())
}
}
func TestPoolsMultipartConditionalUnreadablePool(t *testing.T) {
z, bucket := consistencyPools(t)
object := "quorum-with-readable-copy"
old := putConsistencyObject(t, z, bucket, object, 0, "readable", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, 1, "hidden-newer", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
id, parts := multipartConditionUpload(t, z, bucket, object, 0, ObjectOptions{})
set := z.serverPools[1].getHashedSet(object)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
for i := range disks {
disks[i] = consistencyReadFaultDisk{StorageAPI: disks[i], bucket: bucket, object: object}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
read, readErr := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
t.Logf("ordinary GET lookup: etag=%s err=%v", read.ETag, readErr)
called := 0
_, err := z.CompleteMultipartUpload(t.Context(), bucket, object, id, parts, ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { called++; return oi.ETag != old.ETag }})
if !isErrReadQuorum(err) {
t.Errorf("conditional write must fail on unreadable pool, got %v", err)
}
if called != 0 {
t.Errorf("callback evaluated without complete state: %d calls", called)
}
}
func TestPoolsMultipartConditionalLatestVersionAndCallbackOnce(t *testing.T) {
for _, explicit := range []bool{false, true} {
t.Run(fmt.Sprintf("explicit=%t", explicit), func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "tie-version"
opts := ObjectOptions{MTime: UTCNow().Add(-time.Hour), Versioned: explicit}
if explicit {
opts.VersionID = mustGetUUID()
}
current := putConsistencyObject(t, z, bucket, object, 0, "first", opts)
putConsistencyObject(t, z, bucket, object, 1, "second", opts)
if explicit {
current = putConsistencyObject(t, z, bucket, object, 1, "latest-other-version", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
}
id, parts := multipartConditionUpload(t, z, bucket, object, 1, opts)
called := 0
opts.CheckPrecondFn = func(oi ObjectInfo) bool { called++; return oi.ETag != current.ETag }
opts.MTime = time.Time{}
opts.HasIfMatch = true
_, err := z.CompleteMultipartUpload(t.Context(), bucket, object, id, parts, opts)
if err != nil {
t.Errorf("logical current object should match: %v", err)
}
if called != 1 {
t.Errorf("condition evaluated %d times; want exactly once", called)
}
})
}
}
func TestPoolsMultipartConditionalConcurrentCompletes(t *testing.T) {
z, bucket := consistencyPools(t)
object := "concurrent-completes"
old := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, 1, "old", ObjectOptions{MTime: old.ModTime})
ids := make([]string, 2)
parts := make([][]CompletePart, 2)
for i := range 2 {
ids[i], parts[i] = multipartConditionUpload(t, z, bucket, object, i, ObjectOptions{})
}
entered := make(chan struct{})
release := make(chan struct{})
var gate, releaseOnce sync.Once
defer releaseOnce.Do(func() { close(release) })
errs := make([]error, 2)
var wg sync.WaitGroup
// Let pool 1's completion hold the object lock before pool 0's starts.
// Once pool 1 commits, a set-local read in pool 0 would still see the
// old ETag and incorrectly accept the second completion.
wg.Add(1)
go func() {
defer wg.Done()
_, errs[1] = z.CompleteMultipartUpload(t.Context(), bucket, object, ids[1], parts[1], ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool {
gate.Do(func() { close(entered); <-release })
return oi.ETag != old.ETag
}})
}()
select {
case <-entered:
case <-time.After(10 * time.Second):
t.Fatal("first completion did not enter its condition callback")
}
started := make(chan struct{})
wg.Add(1)
go func() {
defer wg.Done()
close(started)
_, errs[0] = z.CompleteMultipartUpload(t.Context(), bucket, object, ids[0], parts[0], ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { return oi.ETag != old.ETag }})
}()
<-started
releaseOnce.Do(func() { close(release) })
wg.Wait()
success, failed := 0, 0
for _, err := range errs {
var p PreConditionFailed
switch {
case err == nil:
success++
case errors.As(err, &p):
failed++
default:
t.Errorf("unexpected completion error %v", err)
}
}
if success != 1 || failed != 1 {
t.Errorf("CAS writers: success=%d conditional failures=%d errors=%v", success, failed, errs)
}
}
@@ -0,0 +1,135 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"errors"
"fmt"
"testing"
"time"
)
func TestPoolsMultipartConditionMatrix(t *testing.T) {
for owner := range 2 {
for _, withOld := range []bool{false, true} {
for _, condition := range []string{"match-current", "match-old", "none-match-any"} {
t.Run(fmt.Sprintf("upload-pool=%d/old-copy=%t/%s", owner, withOld, condition), func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "conditional-multipart"
oldETag := "arbitrary-old"
if withOld {
old := putConsistencyObject(t, z, bucket, object, owner, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
oldETag = old.ETag
}
mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
part, err := z.serverPools[owner].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
current := putConsistencyObject(t, z, bucket, object, 1-owner, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
visible, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
if err != nil || visible.ETag != current.ETag {
t.Fatalf("invalid current state: %v", err)
}
opts := ObjectOptions{HasIfMatch: condition != "none-match-any", CheckPrecondFn: func(oi ObjectInfo) bool {
switch condition {
case "match-current":
return oi.ETag != current.ETag
case "match-old":
return oi.ETag != oldETag
default:
return true
}
}}
_, err = z.CompleteMultipartUpload(t.Context(), bucket, object, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, opts)
if condition == "match-current" {
if err != nil {
t.Errorf("correct logical If-Match rejected: %v", err)
}
} else {
var expected PreConditionFailed
if !errors.As(err, &expected) {
t.Errorf("logical condition must fail, got %v", err)
}
}
})
}
}
}
}
func TestPoolsMultipartConditionBoundaries(t *testing.T) {
for _, state := range []string{"missing", "latest-delete-marker", "unreadable-other-pool"} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/if-match=%t", state, match), func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "conditional-boundary"
versioned := state == "latest-delete-marker"
if versioned {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
mp, err := z.serverPools[0].NewMultipartUpload(t.Context(), bucket, object, ObjectOptions{Versioned: versioned})
if err != nil {
t.Fatal(err)
}
part, err := z.serverPools[0].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("new"), 3, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if state == "unreadable-other-pool" {
set := z.serverPools[1].getHashedSet(object)
getDisks := set.getDisks
faulty := append([]StorageAPI(nil), getDisks()...)
for i := range faulty {
faulty[i] = consistencyReadFaultDisk{StorageAPI: faulty[i], bucket: bucket, object: object}
}
set.getDisks = func() []StorageAPI { return faulty }
defer func() { set.getDisks = getDisks }()
}
opts := ObjectOptions{Versioned: versioned, HasIfMatch: match, CheckPrecondFn: func(ObjectInfo) bool { return true }}
_, err = z.CompleteMultipartUpload(t.Context(), bucket, object, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, opts)
switch {
case state == "unreadable-other-pool":
if !isErrReadQuorum(err) {
t.Errorf("unreadable pool must not mean absence: %v", err)
}
case match:
if !isErrObjectNotFound(err) {
t.Errorf("If-Match against logical absence should report absence: %v", err)
}
default:
if err != nil {
t.Errorf("If-None-Match against logical absence must succeed: %v", err)
}
}
if err != nil {
if _, lerr := z.serverPools[0].ListObjectParts(t.Context(), bucket, object, mp.UploadID, 0, 10, ObjectOptions{}); lerr != nil {
t.Errorf("failed condition consumed upload: %v", lerr)
}
}
})
}
}
}
@@ -0,0 +1,721 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"fmt"
"maps"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/kms"
)
// Change allocation capacity only; metadata and data use real fixture disks.
// Restoring the adapters also lets encrypted fixtures move the write target
// between requests without changing the production allocation policy.
type conditionalPutCapacityDisk struct {
StorageAPI
full bool
}
func (d conditionalPutCapacityDisk) DiskInfo(ctx context.Context, opts DiskInfoOptions) (DiskInfo, error) {
info, err := d.StorageAPI.DiskInfo(ctx, opts)
info.Total, info.Used = 1<<40, 0
if d.full {
info.Used = info.Total - (1 << 20)
}
info.Free = info.Total - info.Used
return info, err
}
// Keep getDisks immutable while background IAM and storage readers use it.
// GetDisks takes this same mutex when it copies the backing disk list.
func conditionalPutSwapDisks(pool *erasureSets, object string, wrap func(StorageAPI) StorageAPI) func() {
setIndex := pool.getHashedSet(object).setIndex
pool.erasureDisksMu.Lock()
previous := pool.erasureDisks[setIndex]
disks := append([]StorageAPI(nil), previous...)
for i, disk := range disks {
disks[i] = wrap(disk)
}
pool.erasureDisks[setIndex] = disks
pool.erasureDisksMu.Unlock()
return func() {
pool.erasureDisksMu.Lock()
pool.erasureDisks[setIndex] = previous
pool.erasureDisksMu.Unlock()
}
}
func conditionalPutPool(t *testing.T, z *erasureServerPools, object string, target int) func() {
t.Helper()
var restore []func()
for i, pool := range z.serverPools {
restore = append(restore, conditionalPutSwapDisks(pool, object, func(disk StorageAPI) StorageAPI {
return conditionalPutCapacityDisk{StorageAPI: disk, full: i != target}
}))
}
return func() {
for _, fn := range restore {
fn()
}
}
}
func conditionalPutBucket(t *testing.T, z *erasureServerPools, mode string) (string, http.Handler) {
t.Helper()
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: mode != "unversioned"})
if err != nil {
t.Fatal(err)
}
if mode == "suspended" {
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig,
[]byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
}
return bucket, router
}
func TestPoolsConditionalPutHTTP(t *testing.T) {
for _, mode := range []string{"unversioned", "versioned", "suspended"} {
t.Run(mode, func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, mode)
for target := range 2 {
for _, tc := range []struct {
name string
oldCopy, localCurrent bool
condition string
status int
}{
{"stale-etag-accepted", true, false, "old", http.StatusPreconditionFailed},
{"current-etag-rejected", true, false, "current", http.StatusOK},
{"create-only-overwrites-other-pool", false, false, "none", http.StatusPreconditionFailed},
{"current-etag-missing-in-write-pool", false, false, "current", http.StatusOK},
{"control-current-in-write-pool", true, true, "current", http.StatusOK},
{"control-stale-etag-rejected", true, true, "old", http.StatusPreconditionFailed},
{"none-match-current-etag", true, false, "none-current", http.StatusPreconditionFailed},
{"none-match-old-etag", true, false, "none-old", http.StatusOK},
} {
t.Run(fmt.Sprintf("target=%d/%s", target, tc.name), func(t *testing.T) {
object := fmt.Sprintf("%d-%s", target, tc.name)
currentPool := 1 - target
if tc.localCurrent {
currentPool = target
}
opts := ObjectOptions{Versioned: mode == "versioned", VersionSuspended: mode == "suspended", MTime: UTCNow().Add(-time.Hour)}
oldETag := "absent-old"
if tc.oldCopy {
oldETag = putConsistencyObject(t, z, bucket, object, 1-currentPool, "old", opts).ETag
}
opts.MTime = UTCNow().Add(-time.Minute)
current := putConsistencyObject(t, z, bucket, object, currentPool, "current", opts)
defer conditionalPutPool(t, z, object, target)()
idx, err := z.getWritePoolIdx(t.Context(), bucket, object, 11, false)
if err != nil || idx != target {
t.Fatalf("allocation target=%d: idx=%d err=%v", target, idx, err)
}
headers := map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", current.ETag)}
if tc.condition == "old" {
headers[xhttp.IfMatch] = fmt.Sprintf("%q", oldETag)
}
if tc.condition == "none" {
headers = map[string]string{xhttp.IfNoneMatch: "*"}
}
if tc.condition == "none-current" {
headers = map[string]string{xhttp.IfNoneMatch: fmt.Sprintf("%q", current.ETag)}
}
if tc.condition == "none-old" {
headers = map[string]string{xhttp.IfNoneMatch: fmt.Sprintf("%q", oldETag)}
}
url := getPutObjectURL("", bucket, object)
before := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if before.Code != http.StatusOK || before.Body.String() != "current" || multipartConditionResponseETag(before) != current.ETag {
t.Fatalf("invalid current object: %d %q %v", before.Code, before.Body.String(), before.Header())
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != tc.status {
t.Errorf("PUT status=%d want=%d: %s", put.Code, tc.status, put.Body.String())
}
if put.Code == http.StatusPreconditionFailed {
multipartConditionError(t, put, "PreconditionFailed")
if multipartConditionResponseETag(put) != current.ETag || put.Header().Get(xhttp.LastModified) != before.Header().Get(xhttp.LastModified) {
t.Errorf("412 headers do not describe current object: %v", put.Header())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
wantBody, wantETag := "current", current.ETag
if tc.status == http.StatusOK {
wantBody, wantETag = "replacement", fmt.Sprintf("%x", md5.Sum([]byte("replacement")))
}
t.Logf("PUT %d; GET %d bytes=%q ETag=%s", put.Code, get.Code, get.Body.String(), multipartConditionResponseETag(get))
if get.Code != http.StatusOK || get.Body.String() != wantBody || multipartConditionResponseETag(get) != wantETag {
t.Errorf("GET=%d bytes=%q ETag=%s; want %q %s", get.Code, get.Body.String(), multipartConditionResponseETag(get), wantBody, wantETag)
}
})
}
}
})
}
}
func TestPoolsConditionalPutHTTPAbsence(t *testing.T) {
for _, state := range []string{"missing", "uuid-marker", "null-marker"} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/match=%t", state, match), func(t *testing.T) {
z, _ := consistencyPools(t)
mode := "versioned"
if state == "null-marker" {
mode = "suspended"
}
bucket, router := conditionalPutBucket(t, z, mode)
object := "absent-key"
if state != "missing" {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
vid := mustGetUUID()
if state == "null-marker" {
vid = nullVersionID
}
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: vid, DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
defer conditionalPutPool(t, z, object, 0)()
headers := map[string]string{xhttp.IfNoneMatch: "*"}
want := http.StatusOK
if match {
headers = map[string]string{xhttp.IfMatch: "*"}
want = http.StatusNotFound
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url, "new", headers)
if put.Code != want {
t.Fatalf("PUT %d want %d: %s", put.Code, want, put.Body.String())
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if match {
multipartConditionError(t, put, "NoSuchKey")
if get.Code != http.StatusNotFound {
t.Fatalf("failed PUT exposed data: %d %s", get.Code, get.Body.String())
}
} else if get.Code != http.StatusOK || get.Body.String() != "new" || multipartConditionResponseETag(get) != multipartConditionResponseETag(put) {
t.Fatalf("successful create GET: %d %q %v", get.Code, get.Body.String(), get.Header())
}
})
}
}
}
func TestPoolsConditionalPutUnreadable(t *testing.T) {
for faultPool := range 2 {
for _, present := range []bool{false, true} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("fault-pool=%d/present=%t/match=%t", faultPool, present, match), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
object := "unreadable-key"
if present {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
}
defer conditionalPutPool(t, z, object, 0)()
restoreFault := conditionalPutSwapDisks(z.serverPools[faultPool], object, func(disk StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
})
defer restoreFault()
headers := map[string]string{xhttp.IfNoneMatch: "*"}
if match {
headers = map[string]string{xhttp.IfMatch: "*"}
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != http.StatusServiceUnavailable {
t.Errorf("unverified PUT must fail: %d %s", put.Code, put.Body.String())
}
called := 0
_, err := z.PutObject(t.Context(), bucket, object, mustGetPutObjReader(t, strings.NewReader("replacement"), 11, "", ""), ObjectOptions{HasIfMatch: match, CheckPrecondFn: func(ObjectInfo) bool { called++; return false }})
if !isErrReadQuorum(err) || called != 0 {
t.Errorf("lookup error=%v callback calls=%d", err, called)
}
restoreFault()
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if present {
if get.Code != http.StatusOK || get.Body.String() != "current" || multipartConditionResponseETag(get) != fmt.Sprintf("%x", md5.Sum([]byte("current"))) {
t.Fatalf("failed PUT changed object: %d %q", get.Code, get.Body.String())
}
} else if get.Code != http.StatusNotFound {
t.Fatalf("failed PUT created object: %d %q", get.Code, get.Body.String())
}
})
}
}
}
}
func TestPoolsConditionalPutEncryptedETag(t *testing.T) {
for _, kind := range []string{"SSE-C", "SSE-S3", "SSE-KMS"} {
t.Run(kind, func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
oldKMS, oldTLS := GlobalKMS, globalIsTLS
GlobalKMS, globalIsTLS = kms.NewStub("conditional-put-key"), true
defer func() { GlobalKMS, globalIsTLS = oldKMS, oldTLS }()
headers := map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionAES}
readHeaders := map[string]string{}
if kind == "SSE-C" {
key := bytes.Repeat([]byte{0x42}, 32)
digest := md5.Sum(key)
headers = map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(digest[:]),
}
readHeaders = maps.Clone(headers)
}
if kind == "SSE-KMS" {
headers[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionKMS
headers[xhttp.AmzServerSideEncryptionKmsID] = "conditional-put-key"
}
object := "encrypted-key"
url := getPutObjectURL("", bucket, object)
restore := conditionalPutPool(t, z, object, 0)
old := multipartConditionRequest(t, router, http.MethodPut, url, "old", headers)
restore()
restore = conditionalPutPool(t, z, object, 1)
current := multipartConditionRequest(t, router, http.MethodPut, url, "current", headers)
restore()
if old.Code != http.StatusOK || current.Code != http.StatusOK {
t.Fatalf("encrypted setup: %d %s / %d %s", old.Code, old.Body.String(), current.Code, current.Body.String())
}
defer conditionalPutPool(t, z, object, 0)()
for _, stale := range []bool{true, false} {
h := maps.Clone(headers)
h[xhttp.IfMatch] = fmt.Sprintf("%q", multipartConditionResponseETag(current))
wantStatus, wantBody, wantETag := http.StatusOK, "replacement", ""
if stale {
h[xhttp.IfMatch] = fmt.Sprintf("%q", multipartConditionResponseETag(old))
wantStatus, wantBody, wantETag = http.StatusPreconditionFailed, "current", multipartConditionResponseETag(current)
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", h)
if put.Code != wantStatus {
t.Fatalf("encrypted condition: %d want %d: %s", put.Code, wantStatus, put.Body.String())
}
if !stale {
wantETag = multipartConditionResponseETag(put)
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", readHeaders)
if get.Code != http.StatusOK || get.Body.String() != wantBody || multipartConditionResponseETag(get) != wantETag {
t.Fatalf("encrypted GET: %d %q %v", get.Code, get.Body.String(), get.Header())
}
}
})
}
}
func TestPoolsConditionalPutVersionSelection(t *testing.T) {
for _, kind := range []string{"public-version", "replica", "replica-preserve-etag", "movement", "no-lock", "tie", "draining"} {
t.Run(kind, func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "version-selection"
addressed := putConsistencyObject(t, z, bucket, object, 1, "addressed", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
opts := ObjectOptions{Versioned: true, VersionID: addressed.VersionID, HasIfMatch: true}
want := current
switch kind {
case "replica":
opts.ReplicaLockReconcile, opts.ReplicationRequest = true, true
want = addressed
case "replica-preserve-etag":
opts.PreserveETag = addressed.ETag
opts.ReplicaLockReconcile, opts.ReplicationRequest = true, true
want = addressed
case "movement":
opts.DataMovement, opts.SrcPoolIdx = true, 1
want = addressed
case "tie":
want = putConsistencyObject(t, z, bucket, object, 0, "tie-winner", ObjectOptions{Versioned: true, MTime: current.ModTime})
case "draining":
z.poolMetaMutex.Lock()
z.poolMeta.Pools[1].Decommission = &PoolDecommissionInfo{}
z.poolMetaMutex.Unlock()
}
defer conditionalPutPool(t, z, object, 0)()
ctx := t.Context()
if kind == "no-lock" {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
t.Fatal(err)
}
defer lk.Unlock(lkctx)
ctx, opts.NoLock = lkctx.Context(), true
}
called := 0
opts.UserDefined = make(map[string]string)
opts.CheckPrecondFn = func(oi ObjectInfo) bool {
called++
if oi.ETag != want.ETag || oi.VersionID != want.VersionID {
t.Errorf("comparison ETag/version=%s/%s want %s/%s", oi.ETag, oi.VersionID, want.ETag, want.VersionID)
}
return oi.ETag != want.ETag
}
oi, err := z.PutObject(ctx, bucket, object, mustGetPutObjReader(t, strings.NewReader("replacement"), 11, "", ""), opts)
if err != nil || called != 1 {
t.Fatalf("PUT err=%v callback calls=%d", err, called)
}
if oi.VersionID != addressed.VersionID {
t.Fatalf("destination version changed: %s", oi.VersionID)
}
if opts.PreserveETag != "" && oi.ETag != opts.PreserveETag {
t.Fatalf("PreserveETag changed: %s", oi.ETag)
}
})
}
}
func TestPoolsConditionalPutReplicaDuplicateHTTP(t *testing.T) {
for _, null := range []bool{false, true} {
t.Run(fmt.Sprintf("null=%t", null), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "replica-duplicate"
vid := mustGetUUID()
if null {
vid = nullVersionID
}
addressed := putConsistencyObject(t, z, bucket, object, 1, "addressed", ObjectOptions{Versioned: true, VersionID: vid, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
headers := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA",
xhttp.MinIOSourceETag: addressed.ETag,
xhttp.MinIOSourceMTime: addressed.ModTime.Format(time.RFC3339Nano),
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url+"?versionId="+vid, "addressed", headers)
if put.Code != http.StatusPreconditionFailed {
t.Fatalf("replica duplicate: %d %s", put.Code, put.Body.String())
}
multipartConditionError(t, put, "PreconditionFailed")
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "current" || multipartConditionResponseETag(get) != current.ETag {
t.Fatalf("duplicate changed current object: %d %q", get.Code, get.Body.String())
}
get = multipartConditionRequest(t, router, http.MethodGet, url+"?versionId="+vid, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "addressed" || multipartConditionResponseETag(get) != addressed.ETag {
t.Fatalf("duplicate changed addressed version: %d %q", get.Code, get.Body.String())
}
})
}
}
func TestPoolsConditionalPutConcurrentHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
for _, match := range []bool{false, true} {
for iteration := range 5 {
t.Run(fmt.Sprintf("match=%t/iteration=%d", match, iteration), func(t *testing.T) {
object := fmt.Sprintf("concurrent-%t-%d", match, iteration)
headers := map[string]string{xhttp.IfNoneMatch: "*"}
if match {
oi := putConsistencyObject(t, z, bucket, object, 1, "old", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
headers = map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", oi.ETag)}
}
defer conditionalPutPool(t, z, object, 0)()
start := make(chan struct{})
results := make(chan *httptest.ResponseRecorder, 2)
url := getPutObjectURL("", bucket, object)
for i := range 2 {
body := fmt.Sprintf("writer-%d", i)
req, err := newTestSignedRequestV4(http.MethodPut, url, int64(len(body)), strings.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
go func() {
<-start
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
results <- rec
}()
}
close(start)
success, failed, winnerETag := 0, 0, ""
for range 2 {
result := <-results
switch result.Code {
case http.StatusOK:
success++
winnerETag = multipartConditionResponseETag(result)
case http.StatusPreconditionFailed:
failed++
multipartConditionError(t, result, "PreconditionFailed")
default:
t.Errorf("unexpected PUT %d: %s", result.Code, result.Body.String())
}
}
if success != 1 || failed != 1 {
t.Fatalf("success=%d precondition failures=%d", success, failed)
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || multipartConditionResponseETag(get) != winnerETag || fmt.Sprintf("%x", md5.Sum(get.Body.Bytes())) != winnerETag {
t.Fatalf("winner lost: %d %q %v", get.Code, get.Body.String(), get.Header())
}
})
}
}
}
func TestPoolsConditionalPutSerializesMutation(t *testing.T) {
for _, deletion := range []bool{false, true} {
t.Run(fmt.Sprintf("delete=%t", deletion), func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "conditional-mutation"
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
defer cancel()
gate := &consistencyGateReader{Reader: strings.NewReader("replacement"), entered: make(chan struct{}), resume: make(chan struct{})}
release := func() { gate.release.Do(func() { close(gate.resume) }) }
defer release()
reader := mustGetPutObjReader(t, gate, 11, "", "")
written := make(chan error, 1)
go func() {
_, err := z.PutObject(ctx, bucket, object, reader, ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { return oi.ETag != current.ETag }})
written <- err
}()
select {
case <-gate.entered:
case err := <-written:
t.Fatalf("PUT failed before body read: %v", err)
case <-ctx.Done():
t.Fatal(ctx.Err())
}
mutated := make(chan error, 1)
go func() {
var err error
if deletion {
_, err = z.DeleteObject(ctx, bucket, object, ObjectOptions{})
} else {
_, err = z.PutObjectMetadata(ctx, bucket, object, ObjectOptions{EvalMetadataFn: func(oi *ObjectInfo, _ error) (ReplicateDecision, error) {
oi.UserDefined["x-amz-meta-after-put"] = "present"
return ReplicateDecision{}, nil
}})
}
mutated <- err
}()
select {
case err := <-mutated:
t.Fatalf("mutation escaped PUT lock: %v", err)
case <-time.After(100 * time.Millisecond):
}
release()
if err := <-written; err != nil {
t.Fatal(err)
}
if err := <-mutated; err != nil {
t.Fatal(err)
}
oi, err := z.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if deletion {
if !isErrObjectNotFound(err) {
t.Fatalf("delete lost: %v", err)
}
} else if err != nil || oi.UserDefined["x-amz-meta-after-put"] != "present" || oi.ETag != fmt.Sprintf("%x", md5.Sum([]byte("replacement"))) {
t.Fatalf("metadata/PUT lost: %+v %v", oi, err)
}
})
}
}
func TestSinglePoolConditionalPutHTTP(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure16(ctx)
if err != nil {
cancel()
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { cancel(); z.Shutdown(context.Background()); removeRoots(dirs) })
if !z.SinglePool() {
t.Fatal("fixture is not a single pool")
}
bucket, router := conditionalPutBucket(t, z, "unversioned")
object := "single-pool-condition"
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object)
for _, tc := range []struct {
body, match, none string
status int
}{
{"missing", "*", "", http.StatusNotFound},
{"first", "", "*", http.StatusOK},
{"blocked", "", "*", http.StatusPreconditionFailed},
{"blocked", "stale", "", http.StatusPreconditionFailed},
{"second", fmt.Sprintf("%x", md5.Sum([]byte("first"))), "", http.StatusOK},
} {
rec := multipartConditionRequest(t, router, http.MethodPut, url, tc.body, map[string]string{xhttp.IfMatch: tc.match, xhttp.IfNoneMatch: tc.none})
if rec.Code != tc.status {
t.Fatalf("PUT %d want %d: %s", rec.Code, tc.status, rec.Body.String())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "second" {
t.Fatalf("single-pool GET: %d %q", get.Code, get.Body.String())
}
}
// An internal replica callback without an addressed version is not a public
// condition. Preserve its availability when another pool is unreadable. An
// addressed replica already requires all pools for lock/tag reconciliation.
func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
for _, addressed := range []bool{false, true} {
t.Run(fmt.Sprintf("addressed=%t", addressed), func(t *testing.T) {
z, _ := consistencyPools(t)
mode := "unversioned"
if addressed {
mode = "versioned"
}
bucket, router := conditionalPutBucket(t, z, mode)
object := "replica-availability"
oi := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: addressed, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
restoreFault := conditionalPutSwapDisks(z.serverPools[1], object, func(disk StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
})
defer restoreFault()
headers := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA",
xhttp.MinIOSourceETag: fmt.Sprintf("%x", md5.Sum([]byte("replacement"))),
}
url := getPutObjectURL("", bucket, object)
wantStatus, wantBody := http.StatusOK, "replacement"
if addressed {
url += "?versionId=" + oi.VersionID
wantStatus, wantBody = http.StatusServiceUnavailable, "old"
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != wantStatus {
t.Fatalf("replica PUT: %d want %d: %s", put.Code, wantStatus, put.Body.String())
}
restoreFault()
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != wantBody {
t.Fatalf("replica GET: %d %q", get.Code, get.Body.String())
}
})
}
}
func TestPoolsConditionalPutDestinationVersionHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "client-version"
old := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object) + "?versionId=" + old.VersionID
for _, stale := range []bool{true, false} {
tag, status := current.ETag, http.StatusOK
if stale {
tag, status = old.ETag, http.StatusPreconditionFailed
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", tag)})
if put.Code != status {
t.Fatalf("version-addressed public PUT: %d want %d: %s", put.Code, status, put.Body.String())
}
if got := put.Header()[xhttp.AmzVersionID]; !stale && (len(got) != 1 || got[0] != old.VersionID) {
t.Fatalf("write version changed: %v", put.Header())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "replacement" {
t.Fatalf("addressed GET: %d %q", get.Code, get.Body.String())
}
}
func TestPoolsConditionalPutDeleteMarkerTie(t *testing.T) {
for markerPool := range 2 {
t.Run(fmt.Sprintf("marker-pool=%d", markerPool), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "marker-tie"
mtime := UTCNow().Add(-time.Minute)
putConsistencyObject(t, z, bucket, object, 1-markerPool, "live", ObjectOptions{Versioned: true, MTime: mtime})
if _, err := z.serverPools[markerPool].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: mtime}); err != nil {
t.Fatal(err)
}
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object)
before := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
want := http.StatusPreconditionFailed
if markerPool == 0 {
want = http.StatusOK
if before.Code != http.StatusNotFound {
t.Fatalf("GET tie: %d", before.Code)
}
} else if before.Code != http.StatusOK {
t.Fatalf("GET tie: %d", before.Code)
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfNoneMatch: "*"})
if put.Code != want {
t.Fatalf("PUT tie: %d want %d: %s", put.Code, want, put.Body.String())
}
})
}
}
// Overlap fixture changes with the real IAM Walk reader instead of relying on
// its periodic refresh timer to expose an unsynchronized disk-adapter swap.
func TestPoolsConditionalPutFixtureConcurrentIAM(t *testing.T) {
z, _ := consistencyPools(t)
conditionalPutBucket(t, z, "unversioned")
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
started, finished := make(chan struct{}), make(chan error, 1)
iam := globalIAMSys
go func() {
close(started)
for range 20 {
if err := iam.Load(ctx, false); err != nil {
finished <- err
return
}
}
finished <- nil
}()
<-started
for i := range 5000 {
restore := conditionalPutPool(t, z, "fixture-concurrent-iam", i%2)
restore()
}
if err := <-finished; err != nil {
t.Fatal(err)
}
}
+459
View File
@@ -0,0 +1,459 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"fmt"
"maps"
"strings"
"testing"
xhttp "github.com/minio/minio/internal/http"
)
// The fixture places copies directly in real erasure pools. It models a
// duplicated version; it does not claim to exercise a rebalance workflow.
func TestPoolsMetadataUpdatePreservesTags(t *testing.T) {
z, bucket := consistencyPools(t)
const (
old = "2026-09-09T09:00:00Z"
recent = "2026-09-09T10:00:00Z"
timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
)
for _, test := range []struct {
name string
tags, stamps [2]string
winner int
single bool
}{
{name: "newer-secondary", tags: [2]string{"key=old", "key=new"}, stamps: [2]string{old, recent}, winner: 1},
{name: "newer-primary", tags: [2]string{"key=new", "key=old"}, stamps: [2]string{recent, old}},
{name: "empty-secondary", tags: [2]string{"key=old", ""}, stamps: [2]string{old, recent}, winner: 1},
{name: "empty-primary", tags: [2]string{"", "key=old"}, stamps: [2]string{recent, old}},
{name: "single-copy-primary", tags: [2]string{"key=only", ""}, stamps: [2]string{recent, ""}, single: true},
{name: "single-copy-secondary", tags: [2]string{"", "key=only"}, stamps: [2]string{"", recent}, winner: 1, single: true},
{name: "legacy-single-copy", tags: [2]string{"key=legacy", ""}, single: true},
{name: "legacy-duplicates", tags: [2]string{"key=legacy", "key=legacy"}},
} {
t.Run(test.name, func(t *testing.T) {
object := test.name
var original ObjectInfo
for pool := range 2 {
if test.single && pool != test.winner {
continue
}
metadata := map[string]string{
xhttp.AmzObjectTagging: test.tags[pool],
"copy-local": fmt.Sprint(pool),
}
if test.stamps[pool] != "" {
metadata[timestamp] = test.stamps[pool]
}
original = putConsistencyObject(t, z, bucket, object, pool, "data", ObjectOptions{
Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, UserDefined: metadata,
})
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: original.VersionID})
if err != nil || got.UserTags != test.tags[pool] || got.UserDefined[timestamp] != test.stamps[pool] {
t.Fatalf("pool %d fixture: tags=%q timestamp=%q err=%v", pool, got.UserTags, got.UserDefined[timestamp], err)
}
if _, exists := got.UserDefined[xhttp.AmzObjectTagging]; exists {
t.Fatal("fixture must use the cleaned ObjectInfo representation")
}
}
wantTags, wantStamp := test.tags[test.winner], test.stamps[test.winner]
called := 0
got, err := z.PutObjectMetadata(t.Context(), bucket, object, ObjectOptions{
VersionID: original.VersionID, MTime: original.ModTime,
EvalMetadataFn: func(current *ObjectInfo, _ error) (ReplicateDecision, error) {
called++
if current.UserTags != wantTags || current.UserDefined[timestamp] != wantStamp {
t.Errorf("callback tags=%q timestamp=%q; want %q %q", current.UserTags, current.UserDefined[timestamp], wantTags, wantStamp)
}
current.UserDefined["unrelated-update"] = "preserved"
return ReplicateDecision{}, nil
},
})
if err != nil || called != 1 {
t.Fatalf("metadata update: %v, callbacks=%d", err, called)
}
if got.UserTags != wantTags || got.UserDefined[timestamp] != wantStamp {
t.Errorf("response tags=%q timestamp=%q; want %q %q", got.UserTags, got.UserDefined[timestamp], wantTags, wantStamp)
}
for pool := range 2 {
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: original.VersionID})
if test.single && pool != test.winner {
if !isErrVersionNotFound(err) {
t.Errorf("metadata update created another copy: %v", err)
}
continue
}
if err != nil {
t.Fatal(err)
}
t.Logf("pool %d persisted tags=%q timestamp=%q", pool, got.UserTags, got.UserDefined[timestamp])
if got.UserTags != wantTags || got.UserDefined[timestamp] != wantStamp {
t.Errorf("pool %d persisted tags=%q timestamp=%q; want %q %q", pool, got.UserTags, got.UserDefined[timestamp], wantTags, wantStamp)
}
if got.UserDefined["unrelated-update"] != "preserved" || got.UserDefined["copy-local"] != fmt.Sprint(pool) {
t.Errorf("pool %d lost unrelated metadata: %v", pool, got.UserDefined)
}
}
})
}
}
func TestReplicaWritesPreserveTagOrdering(t *testing.T) {
z, bucket := consistencyPools(t)
// Pool allocation checks the host's used-space percentage. Present only
// its free space as fixture capacity; all reads and writes still use the
// real disks. This keeps unrelated host disk usage out of the tag test.
for _, pool := range z.serverPools {
for _, set := range pool.sets {
getDisks := set.getDisks
disks := append([]StorageAPI(nil), getDisks()...)
for i := range disks {
disks[i] = tagTestCapacityDisk{StorageAPI: disks[i]}
}
set.getDisks = func() []StorageAPI { return disks }
t.Cleanup(func() { set.getDisks = getDisks })
}
}
const (
old = "2026-09-09T09:00:00Z"
recent = "2026-09-09T10:00:00Z"
timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
)
for _, path := range []struct {
name, operation string
direct bool
winner int
}{
{name: "set-put", operation: "put", direct: true},
{name: "set-multipart", operation: "multipart", direct: true},
{name: "set-copy", operation: "copy", direct: true},
{name: "pools-put", operation: "put", winner: 1},
{name: "pools-multipart", operation: "multipart", winner: 1},
{name: "pools-copy-primary", operation: "copy"},
{name: "pools-copy-secondary", operation: "copy", winner: 1},
} {
for _, test := range []struct {
name, storedTags, incomingTags, storedStamp, incomingStamp, wantTags string
}{
{"stored-newer", "key=stored", "key=incoming", recent, old, "key=stored"},
{"stored-deleted", "", "key=incoming", recent, old, ""},
{"incoming-newer", "key=stored", "key=incoming", old, recent, "key=incoming"},
{"incoming-deleted", "key=stored", "", old, recent, ""},
} {
t.Run(path.name+"/"+test.name, func(t *testing.T) {
object := path.name + "-" + test.name
var original ObjectInfo
for pool := range 2 {
if path.direct && pool != 0 {
continue
}
metadata := map[string]string{
xhttp.AmzObjectTagging: "key=older-copy",
timestamp: "2026-09-09T08:00:00Z",
}
if pool == path.winner {
metadata[xhttp.AmzObjectTagging] = test.storedTags
metadata[timestamp] = test.storedStamp
}
original = putConsistencyObject(t, z, bucket, object, pool, "data", ObjectOptions{
Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, UserDefined: metadata,
})
}
opts := ObjectOptions{
Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, ReplicaLockReconcile: true,
UserDefined: map[string]string{
xhttp.AmzObjectTagging: test.incomingTags,
timestamp: test.incomingStamp,
},
}
var got ObjectInfo
var err error
switch path.operation {
case "put":
put := z.PutObject
if path.direct {
put = z.serverPools[0].PutObject
}
got, err = put(t.Context(), bucket, object, mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), opts)
case "multipart":
// Persist the incoming tags with the upload, before completion
// reconciles the destination version through its real resolver.
mp, err := z.serverPools[0].NewMultipartUpload(t.Context(), bucket, object, opts)
if err != nil {
t.Fatal(err)
}
part, err := z.serverPools[0].PutObjectPart(t.Context(), bucket, object, mp.UploadID, 1,
mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
complete := z.CompleteMultipartUpload
if path.direct {
complete = z.serverPools[0].CompleteMultipartUpload
}
got, err = complete(t.Context(), bucket, object, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{
Versioned: true, MTime: original.ModTime, ReplicaLockReconcile: true,
})
if err != nil {
t.Fatal(err)
}
case "copy":
src := original
src.metadataOnly = true
src.UserDefined = maps.Clone(opts.UserDefined)
copyObject := z.CopyObject
if path.direct {
copyObject = z.serverPools[0].CopyObject
}
got, err = copyObject(t.Context(), bucket, object, bucket, object, src, ObjectOptions{VersionID: original.VersionID}, opts)
}
if err != nil {
t.Fatal(err)
}
if got.UserTags != test.wantTags || got.UserDefined[timestamp] != recent {
t.Errorf("response tags=%q timestamp=%q; want %q %q", got.UserTags, got.UserDefined[timestamp], test.wantTags, recent)
}
copies := 0
for pool := range 2 {
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: original.VersionID})
if isErrVersionNotFound(err) {
continue
}
if err != nil {
t.Fatal(err)
}
copies++
if got.UserTags != test.wantTags || got.UserDefined[timestamp] != recent {
t.Errorf("pool %d persisted tags=%q timestamp=%q; want %q %q", pool, got.UserTags, got.UserDefined[timestamp], test.wantTags, recent)
}
}
if copies != 1 {
t.Errorf("replacement left %d copies; want 1", copies)
}
})
}
}
}
type tagTestCapacityDisk struct{ StorageAPI }
func (d tagTestCapacityDisk) DiskInfo(ctx context.Context, opts DiskInfoOptions) (DiskInfo, error) {
info, err := d.StorageAPI.DiskInfo(ctx, opts)
info.Total, info.Used = info.Free, 0
return info, err
}
func TestMergedPoolObjectInfoTagOrdering(t *testing.T) {
const (
old = "2026-09-09T09:00:00Z"
recent = "2026-09-09T10:00:00Z"
timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
)
for _, test := range []struct {
name, firstStamp, secondStamp, secondTags string
winner int
}{
{"newer", old, recent, "key=second", 1},
{"newer-removal", old, recent, "", 1},
{"equal", recent, recent, "key=second", 0},
{"unordered", "", "", "key=second", 0},
{"missing-first", "", recent, "key=second", 1},
{"missing-second", recent, "", "key=second", 0},
{"invalid-first", "invalid", recent, "key=second", 1},
{"invalid-second", recent, "invalid", "key=second", 0},
} {
t.Run(test.name, func(t *testing.T) {
tagValues := []string{"key=first", test.secondTags}
stamps := []string{test.firstStamp, test.secondStamp}
copies := make([]PoolObjInfo, 2)
before := make([]map[string]string, 2)
for i := range copies {
fi := FileInfo{Metadata: map[string]string{xhttp.AmzObjectTagging: tagValues[i]}}
if stamps[i] != "" {
fi.Metadata[timestamp] = stamps[i]
}
copies[i] = PoolObjInfo{Index: i, ObjInfo: fi.ToObjectInfo("bucket", "object", true)}
before[i] = maps.Clone(copies[i].ObjInfo.UserDefined)
}
got := mergedPoolObjectInfo(copies)
if got.UserTags != tagValues[test.winner] || got.UserDefined[timestamp] != stamps[test.winner] {
t.Errorf("merged tags=%q timestamp=%q; want %q %q", got.UserTags, got.UserDefined[timestamp], tagValues[test.winner], stamps[test.winner])
}
if _, exists := got.UserDefined[xhttp.AmzObjectTagging]; exists {
t.Error("merged ObjectInfo leaked the raw tagging key into UserDefined")
}
for i := range copies {
if !maps.Equal(copies[i].ObjInfo.UserDefined, before[i]) || copies[i].ObjInfo.UserTags != tagValues[i] {
t.Errorf("merge mutated input copy %d", i)
}
}
})
}
}
func TestPoolsMetadataCallbackReplacesTags(t *testing.T) {
z, bucket := consistencyPools(t)
const timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
for _, test := range []struct{ name, tags string }{
{"replace", "key=callback"},
{"remove", ""},
} {
t.Run(test.name, func(t *testing.T) {
var original ObjectInfo
for pool := range 2 {
original = putConsistencyObject(t, z, bucket, test.name, pool, "data", ObjectOptions{
Versioned: true, VersionID: original.VersionID, MTime: original.ModTime,
UserDefined: map[string]string{
xhttp.AmzObjectTagging: []string{"key=old", "key=new"}[pool],
timestamp: []string{"2026-09-09T09:00:00Z", "2026-09-09T10:00:00Z"}[pool],
},
})
}
const updatedStamp = "2026-09-09T11:00:00Z"
got, err := z.PutObjectMetadata(t.Context(), bucket, test.name, ObjectOptions{
VersionID: original.VersionID, MTime: original.ModTime,
EvalMetadataFn: func(current *ObjectInfo, _ error) (ReplicateDecision, error) {
if current.UserTags != "key=new" {
t.Errorf("callback read tags=%q; want key=new", current.UserTags)
}
if _, exists := current.UserDefined[xhttp.AmzObjectTagging]; exists {
t.Error("callback received the raw tagging key")
}
current.UserDefined[xhttp.AmzObjectTagging] = test.tags
current.UserDefined[timestamp] = updatedStamp
return ReplicateDecision{}, nil
},
})
if err != nil {
t.Fatal(err)
}
if got.UserTags != test.tags || got.UserDefined[timestamp] != updatedStamp {
t.Errorf("callback update response tags=%q timestamp=%q", got.UserTags, got.UserDefined[timestamp])
}
for pool := range 2 {
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, test.name, ObjectOptions{VersionID: original.VersionID})
if err != nil || got.UserTags != test.tags || got.UserDefined[timestamp] != updatedStamp {
t.Errorf("pool %d did not persist callback tags: tags=%q timestamp=%q err=%v", pool, got.UserTags, got.UserDefined[timestamp], err)
}
}
})
}
}
func TestReconcileStoredObjectTagOrdering(t *testing.T) {
const (
old = "2026-09-09T09:00:00Z"
recent = "2026-09-09T10:00:00Z"
timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
)
for _, test := range []struct {
name, storedStamp, incomingStamp, storedTags string
wantStored bool
}{
{"stored-newer", recent, old, "key=stored", true},
{"incoming-newer", old, recent, "key=stored", false},
{"equal", recent, recent, "key=stored", true},
{"equal-removal", recent, recent, "", true},
{"missing-stored", "", recent, "key=stored", false},
{"missing-incoming", recent, "", "key=stored", true},
{"invalid-stored", "invalid", recent, "key=stored", false},
{"invalid-incoming", recent, "invalid", "key=stored", true},
} {
t.Run(test.name, func(t *testing.T) {
metadata := map[string]string{
xhttp.AmzObjectTagging: "key=incoming",
timestamp: test.incomingStamp,
"unrelated": "preserved",
}
reconcileStoredObjectTags(metadata, test.storedTags, test.storedStamp)
wantTags, wantStamp := "key=incoming", test.incomingStamp
if test.wantStored {
wantTags, wantStamp = test.storedTags, test.storedStamp
}
if metadata[xhttp.AmzObjectTagging] != wantTags || metadata[timestamp] != wantStamp || metadata["unrelated"] != "preserved" {
t.Errorf("reconciled metadata=%v; want tags=%q timestamp=%q and unrelated field preserved", metadata, wantTags, wantStamp)
}
})
}
}
func TestPoolsMetadataUpdatePreservesAbsentTags(t *testing.T) {
z, bucket := consistencyPools(t)
const (
old = "2026-09-09T09:00:00Z"
recent = "2026-09-09T10:00:00Z"
timestamp = ReservedMetadataPrefixLower + TaggingTimestamp
)
for _, removal := range []bool{false, true} {
t.Run(fmt.Sprintf("timestamp-only-removal=%t", removal), func(t *testing.T) {
object := fmt.Sprintf("absent-tags-%t", removal)
var original ObjectInfo
checkStored := func(pool int, wantKey bool, wantTags, wantStamp string) {
t.Helper()
infos, errs := readAllFileInfo(t.Context(), z.serverPools[pool].getHashedSet(object).getDisks(), "", bucket, object, original.VersionID, false, false)
for disk, info := range infos {
if errs[disk] != nil {
t.Fatal(errs[disk])
}
tags, exists := info.Metadata[xhttp.AmzObjectTagging]
if exists != wantKey || tags != wantTags || info.Metadata[timestamp] != wantStamp {
t.Errorf("pool %d disk %d raw tagging key=%t value=%q stamp=%q; want %t %q %q", pool, disk, exists, tags, info.Metadata[timestamp], wantKey, wantTags, wantStamp)
}
}
}
for pool := range 2 {
metadata := map[string]string{}
if removal {
metadata[timestamp] = recent
if pool == 1 {
metadata[xhttp.AmzObjectTagging] = "key=old"
metadata[timestamp] = old
}
}
original = putConsistencyObject(t, z, bucket, object, pool, "data", ObjectOptions{
Versioned: true, VersionID: original.VersionID, MTime: original.ModTime, UserDefined: metadata,
})
checkStored(pool, removal && pool == 1, metadata[xhttp.AmzObjectTagging], metadata[timestamp])
}
_, err := z.PutObjectMetadata(t.Context(), bucket, object, ObjectOptions{
VersionID: original.VersionID, MTime: original.ModTime,
EvalMetadataFn: func(current *ObjectInfo, _ error) (ReplicateDecision, error) {
current.UserDefined["unrelated-update"] = "preserved"
return ReplicateDecision{}, nil
},
})
if err != nil {
t.Fatal(err)
}
wantStamp := ""
if removal {
wantStamp = recent
}
for pool := range 2 {
// A previously non-empty key needs an explicit empty value to
// propagate deletion; an absent key should remain absent.
checkStored(pool, removal && pool == 1, "", wantStamp)
}
})
}
}
+321 -68
View File
@@ -635,7 +635,14 @@ func (z *erasureServerPools) getPoolIdxNoLock(ctx context.Context, bucket, objec
// if none are found falls back to most available space pool, this function is
// designed to be only used by PutObject, CopyObject (newObject creation) and NewMultipartUpload.
func (z *erasureServerPools) getPoolIdx(ctx context.Context, bucket, object string, size int64) (idx int, err error) {
return z.getWritePoolIdx(ctx, bucket, object, size, false)
}
// getWritePoolIdx keeps the write-allocation policy when the caller already
// holds the pools-layer object lock and must not reacquire a set read lock.
func (z *erasureServerPools) getWritePoolIdx(ctx context.Context, bucket, object string, size int64, noLock bool) (idx int, err error) {
pinfo, _, err := z.getPoolInfoExistingWithOpts(ctx, bucket, object, ObjectOptions{
NoLock: noLock,
SkipDecommissioned: true,
SkipRebalancing: true,
})
@@ -1131,7 +1138,52 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
return z.serverPools[0].PutObject(ctx, bucket, object, data, opts)
}
idx, err := z.getPoolIdx(ctx, bucket, object, data.Size())
if !opts.NoLock {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
return ObjectInfo{}, err
}
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
}
opts.NoLock = true
// Public write conditions compare the logical current object while the
// pools-layer write lock is held. The destination selected by capacity may
// be empty or stale, and draining pools can still hold the current object.
// Replica callbacks retain their existing addressed-version semantics and
// metadata reconciliation at the set layer.
if opts.CheckPrecondFn != nil && !opts.ReplicationRequest &&
!opts.ReplicaLockReconcile && !opts.DataMovement {
copies, lerr := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{
VersionID: "", // Compare the current object, not the write's version.
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoAuditLog: true,
})
var latest ObjectInfo
if lerr == nil {
latest = copies[0].ObjInfo
if latest.DeleteMarker {
lerr = toObjectErr(errFileNotFound, bucket, object)
}
}
// An unreadable pool may hold the newest object; it is not absence.
if lerr != nil && !isErrObjectNotFound(lerr) && !isErrVersionNotFound(lerr) {
return ObjectInfo{}, lerr
}
if lerr == nil && opts.CheckPrecondFn(latest) {
return ObjectInfo{}, PreConditionFailed{}
}
if lerr != nil && opts.HasIfMatch {
return ObjectInfo{}, lerr
}
// Do not repeat an accepted condition against the destination's copy.
opts.CheckPrecondFn = nil
}
idx, err := z.getWritePoolIdx(ctx, bucket, object, data.Size(), true)
if err != nil {
return ObjectInfo{}, err
}
@@ -1145,7 +1197,15 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
}
}
return z.serverPools[idx].PutObject(ctx, bucket, object, data, opts)
if opts.ReplicaLockReconcile || opts.DataMovement {
opts.ReplicaLockReconcile = true
opts.replicaObjectInfo = z.replicaObjectInfo
}
oi, err := z.serverPools[idx].PutObject(ctx, bucket, object, data, opts)
if err == nil && opts.ReplicaLockReconcile && !opts.DataMovement {
err = z.retireReplicaCopies(ctx, bucket, object, idx, oi)
}
return oi, err
}
func (z *erasureServerPools) deletePrefix(ctx context.Context, bucket string, prefix string) error {
@@ -1166,7 +1226,7 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
object = encodeDirObject(object)
}
// Acquire a write lock before deleting the object.
// Serialize logical deletion with pooled writes and metadata updates.
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalDeleteOperationTimeout)
if err != nil {
@@ -1179,6 +1239,13 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
return ObjectInfo{}, z.deletePrefix(ctx, bucket, object)
}
// Reconcile ordinary addressed-version deletes independently of pool movement.
reconcileVersion := opts.VersionID != "" && !opts.DataMovement &&
!opts.ReplicationRequest && !opts.Expiration.Expire && !opts.InclFreeVersions
if !z.SinglePool() && (opts.CheckPrecondFn != nil || reconcileVersion) {
return z.deleteObjectReconciled(ctx, bucket, object, opts)
}
gopts := opts
gopts.NoLock = true
@@ -1187,9 +1254,55 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
if _, ok := err.(InsufficientReadQuorum); ok {
return objInfo, InsufficientWriteQuorum{}
}
// A conditional (If-Match) delete addressing a specific version treats an
// absent key as an absent version. getPoolInfoExistingWithOpts strips
// VersionID, so a missing key surfaces ObjectNotFound here even for a
// version-scoped delete; normalize it to VersionNotFound (NoSuchVersion),
// matching this function's tail. The unconditional path is unchanged.
if opts.CheckPrecondFn != nil && opts.VersionID != "" && isErrObjectNotFound(err) {
return objInfo, VersionNotFound{Bucket: bucket, Object: object, VersionID: opts.VersionID}
}
return objInfo, err
}
// Evaluate the conditional (If-Match) precondition while the write lock
// acquired above is held, before the delete-marker short-circuit and before
// any version is removed, so the object cannot change between the check and
// the delete. This is scoped to a single erasure set (see the note at the
// lock above): only there do the delete lock and the write path share the
// same lock namespace, making the check-then-delete atomic.
if opts.CheckPrecondFn != nil {
// pinfo.ObjInfo is the current latest version. getPoolInfoExistingWithOpts
// intentionally strips VersionID, so for a version-scoped delete read the
// specifically addressed version and evaluate the precondition against it.
checkInfo := pinfo.ObjInfo
if opts.VersionID != "" {
vopts := opts
vopts.NoLock = true // delete lock already held above
vopts.CheckPrecondFn = nil
vi, verr := z.serverPools[pinfo.Index].GetObjectInfo(ctx, bucket, object, vopts)
if verr != nil && (!isErrMethodNotAllowed(verr) || !vi.DeleteMarker) {
// Genuine read failure for the addressed version: a missing
// version -> VersionNotFound (NoSuchVersion), read-quorum loss, etc.
return objInfo, verr
}
// verr is nil for a live version, or MethodNotAllowed with a populated
// delete-marker ObjectInfo when the addressed version is a delete
// marker. In the latter case evaluate the precondition against the
// marker, which fails any If-Match (-> 412), rather than surfacing 405.
checkInfo = vi
} else if checkInfo.Name == "" {
// The current state could not be read (e.g. read-quorum loss); refuse
// the conditional delete rather than act on an unverified precondition.
return objInfo, InsufficientReadQuorum{}
}
if opts.CheckPrecondFn(checkInfo) {
return objInfo, PreConditionFailed{}
}
// Precondition satisfied; lower layers must not re-evaluate it.
opts.CheckPrecondFn = nil
}
// Delete marker already present we are not going to create new delete markers.
if pinfo.ObjInfo.DeleteMarker && opts.VersionID == "" {
pinfo.ObjInfo.Name = decodeDirObject(object)
@@ -1268,8 +1381,12 @@ func (z *erasureServerPools) deleteObjectFromAllPools(ctx context.Context, bucke
// the delete call tries to clean up other pools during DeleteObject call.
objInfo = dobjects[0]
objInfo.Name = decodeDirObject(object)
err = derrs[0]
return objInfo, err
for _, derr := range derrs {
if derr != nil && !isErrObjectNotFound(derr) && !isErrVersionNotFound(derr) {
return objInfo, derr
}
}
return objInfo, derrs[0]
}
func (z *erasureServerPools) DeleteObjects(ctx context.Context, bucket string, objects []ObjectToDelete, opts ObjectOptions) ([]DeletedObject, []error) {
@@ -1357,11 +1474,38 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
dstOpts.NoLock = true
}
if !z.SinglePool() && cpSrcDstSame && srcInfo.metadataOnly && dstOpts.ReplicaLockReconcile && srcOpts.VersionID == dstOpts.VersionID {
copies, err := z.metadataPoolInfos(ctx, dstBucket, dstObject, dstOpts)
if err != nil {
return ObjectInfo{}, err
}
stored := mergedPoolObjectInfo(copies)
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(stored.UserDefined))
reconcileStoredObjectTags(srcInfo.UserDefined, stored.UserTags, stored.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
idx := copies[0].Index
oi, err := z.serverPools[idx].CopyObject(ctx, srcBucket, srcObject, dstBucket, dstObject, srcInfo, srcOpts, dstOpts)
if err == nil {
err = z.retireReplicaCopies(ctx, dstBucket, dstObject, idx, oi)
}
return oi, err
}
poolIdx, err := z.getPoolIdxNoLock(ctx, dstBucket, dstObject, srcInfo.Size)
if err != nil {
return objInfo, err
}
if !z.SinglePool() && dstOpts.ReplicaLockReconcile {
dstOpts.replicaObjectInfo = z.replicaObjectInfo
if !dstOpts.DataMovement {
defer func() {
if err == nil {
err = z.retireReplicaCopies(ctx, dstBucket, dstObject, poolIdx, objInfo)
}
}()
}
}
// CopyObjectHandler predicts the outcome of this decision in
// copyRewritesObjectData(); keep the two in sync.
if cpSrcDstSame && srcInfo.metadataOnly {
@@ -1396,6 +1540,8 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
EncryptFn: dstOpts.EncryptFn,
WantChecksum: dstOpts.WantChecksum,
WantServerSideChecksumType: dstOpts.WantServerSideChecksumType,
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
replicaObjectInfo: dstOpts.replicaObjectInfo,
}
return z.serverPools[poolIdx].PutObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, putOpts)
@@ -1745,7 +1891,41 @@ func (z *erasureServerPools) ListMultipartUploads(ctx context.Context, bucket, p
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
return ListMultipartsInfo{}, toObjectErr(err, bucket)
}
if globalAPIConfig.getMultipartListingLegacy() {
return z.listMultipartUploadsLegacy(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
var uploads []MultipartInfo
var keyless bool
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
poolUploads, poolKeyless, err := pool.scanMultipartUploads(scan, bucket, idx)
if err != nil {
return ListMultipartsInfo{}, err
}
uploads = append(uploads, poolUploads...)
keyless = keyless || poolKeyless
}
// The old format cannot be enumerated authoritatively. Migration mode is
// explicit: another bucket must never silently change this API's semantics.
if keyless {
return ListMultipartsInfo{}, errMultipartListingLegacy
}
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
func (z *erasureServerPools) listMultipartUploadsLegacy(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
poolResult := ListMultipartsInfo{}
poolResult.MaxUploads = maxUploads
poolResult.KeyMarker = keyMarker
@@ -1771,15 +1951,14 @@ func (z *erasureServerPools) ListMultipartUploads(ctx context.Context, bucket, p
}
if z.SinglePool() {
return z.serverPools[0].ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
return z.serverPools[0].getHashedSet(prefix).listMultipartUploadsExact(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
}
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
result, err := pool.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker,
delimiter, maxUploads)
result, err := pool.getHashedSet(prefix).listMultipartUploadsExact(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
return result, err
}
@@ -1815,7 +1994,7 @@ func (z *erasureServerPools) NewMultipartUpload(ctx context.Context, bucket, obj
continue
}
result, err := pool.ListMultipartUploads(ctx, bucket, object, "", "", "", maxUploadsList)
result, err := pool.listMultipartUploadsExact(ctx, bucket, object)
if err != nil {
return nil, err
}
@@ -1977,13 +2156,18 @@ func (z *erasureServerPools) AbortMultipartUpload(ctx context.Context, bucket, o
if err := checkAbortMultipartArgs(ctx, bucket, object, uploadID); err != nil {
return err
}
if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
return toObjectErr(err, bucket)
}
defer func() {
// Unlock cancels the derived lock context before this notification runs.
// Keep the request context so successful cancellation reaches peer caches.
defer func(ctx context.Context) {
if err == nil {
z.mpCache.Delete(uploadID)
globalNotificationSys.DeleteUploadID(ctx, uploadID)
}
}()
}(ctx)
lk := z.NewNSLock(bucket, pathJoin(object, uploadID))
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
@@ -1993,23 +2177,28 @@ func (z *erasureServerPools) AbortMultipartUpload(ctx context.Context, bucket, o
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
if z.SinglePool() {
return z.serverPools[0].AbortMultipartUpload(ctx, bucket, object, uploadID, opts)
}
legacy := globalAPIConfig.getMultipartListingLegacy()
found := false
var firstErr error
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
err := pool.AbortMultipartUpload(ctx, bucket, object, uploadID, opts)
if err == nil {
return nil
poolFound, err := pool.getHashedSet(object).abortMultipartUpload(ctx, bucket, object, uploadID, opts, legacy)
if legacy && (poolFound || err != nil) {
// Match the released first-matching-pool behavior.
return err
}
if _, ok := err.(InvalidUploadID); ok {
// upload id not found move to next pool
continue
found = found || poolFound
if err != nil && firstErr == nil {
firstErr = err
}
return err
}
if firstErr != nil {
return firstErr
}
if found {
return nil
}
return InvalidUploadID{
Bucket: bucket,
@@ -2031,6 +2220,60 @@ func (z *erasureServerPools) CompleteMultipartUpload(ctx context.Context, bucket
}
}()
if !z.SinglePool() {
if !opts.NoLock {
lk := z.NewNSLock(bucket, encodeDirObject(object))
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
return objInfo, err
}
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
}
opts.NoLock = true
// A conditional completion must be evaluated against the logical
// latest object across pools, under the object write lock held for
// this operation. The pool hosting the upload may only hold a stale
// duplicate, so its set-local check would both accept an outdated
// ETag and reject the current one. An unreadable pool is not
// absence: it may hold the newest copy, so a read that cannot be
// verified fails the request instead of passing the condition.
// Once satisfied, the callback is cleared so the set layer does not
// re-evaluate it against its local copy.
if opts.CheckPrecondFn != nil {
copies, lerr := z.objectPoolInfos(ctx, bucket, encodeDirObject(object), ObjectOptions{
// Conditions always compare the logical current object,
// independently of the completion's destination version.
VersionID: "",
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoAuditLog: true,
})
var latest ObjectInfo
if lerr == nil {
latest = copies[0].ObjInfo
if latest.DeleteMarker {
// A delete-marker latest reads as an absent key, matching
// the set layer's getObjectInfo.
lerr = toObjectErr(errFileNotFound, bucket, object)
}
}
if lerr == nil && opts.CheckPrecondFn(latest) {
return ObjectInfo{}, PreConditionFailed{}
}
if lerr != nil && !isErrVersionNotFound(lerr) && !isErrObjectNotFound(lerr) {
return ObjectInfo{}, lerr
}
// if object doesn't exist return error for If-Match conditional requests
// If-None-Match should be allowed to proceed for non-existent objects
if lerr != nil && opts.HasIfMatch && (isErrObjectNotFound(lerr) || isErrVersionNotFound(lerr)) {
return ObjectInfo{}, lerr
}
opts.CheckPrecondFn = nil
}
}
// Hold write locks to verify uploaded parts, also disallows any
// parallel PutObjectPart() requests.
uploadIDLock := z.NewNSLock(bucket, pathJoin(object, uploadID))
@@ -2045,13 +2288,21 @@ func (z *erasureServerPools) CompleteMultipartUpload(ctx context.Context, bucket
return z.serverPools[0].CompleteMultipartUpload(ctx, bucket, object, uploadID, uploadedParts, opts)
}
if opts.ReplicaLockReconcile || opts.DataMovement {
opts.ReplicaLockReconcile = true
opts.replicaObjectInfo = z.replicaObjectInfo
}
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
objInfo, err = pool.CompleteMultipartUpload(ctx, bucket, object, uploadID, uploadedParts, opts)
if err == nil {
return objInfo, nil
if opts.ReplicaLockReconcile && !opts.DataMovement {
err = z.retireReplicaCopies(ctx, bucket, encodeDirObject(object), idx, objInfo)
}
return objInfo, err
}
if _, ok := err.(InvalidUploadID); ok {
// upload id not found move to next pool
@@ -2073,6 +2324,11 @@ func (z *erasureServerPools) GetBucketInfo(ctx context.Context, bucket string, o
if err != nil {
return bucketInfo, toObjectErr(err, bucket)
}
// Physical existence/creation probes must not be overwritten by cached
// metadata, which can legitimately lack Created on an unmigrated bucket.
if opts.NoMetadata {
return bucketInfo, nil
}
meta, err := globalBucketMetadataSys.Get(bucket)
if err == nil {
@@ -2141,7 +2397,15 @@ func (z *erasureServerPools) DeleteBucket(ctx context.Context, bucket string, op
opts.Force = true
}
err := z.s3Peer.DeleteBucket(ctx, bucket, opts)
// Take the metadata writer lock before deleting anything. Failure or
// cancellation must leave both the bucket and its metadata intact.
ctx, unlock, err := lockBucketMetadata(ctx, z, bucket)
if err != nil {
return toObjectErr(err, bucket)
}
defer unlock()
err = z.s3Peer.DeleteBucket(ctx, bucket, opts)
if err == nil || isErrBucketNotFound(err) {
// If site replication is configured, hold on to deleted bucket state until sites sync
if opts.SRDeleteOp == MarkDelete {
@@ -2150,8 +2414,9 @@ func (z *erasureServerPools) DeleteBucket(ctx context.Context, bucket string, op
}
if err == nil {
// Purge the entire bucket metadata entirely.
z.deleteAll(context.Background(), minioMetaBucket, pathJoin(bucketMetaPrefix, bucket))
// Finish cleanup after a committed delete even if the client disconnects.
// Both the bucket-name and metadata locks remain held until return.
z.deleteAll(context.WithoutCancel(ctx), minioMetaBucket, pathJoin(bucketMetaPrefix, bucket))
}
return toObjectErr(err, bucket)
@@ -2569,7 +2834,7 @@ func (z *erasureServerPools) HealObject(ctx context.Context, bucket, object, ver
wg.Add(1)
go func(idx int, pool *erasureSets) {
defer wg.Done()
result, err := pool.HealObject(ctx, bucket, object, versionID, opts)
result, err := z.healObjectInPool(ctx, pool.getHashedSet(object), bucket, object, versionID, opts)
result.Object = decodeDirObject(result.Object)
errs[idx] = err
results[idx] = result
@@ -2838,15 +3103,7 @@ func (z *erasureServerPools) PutObjectMetadata(ctx context.Context, bucket, obje
defer lk.Unlock(lkctx)
}
opts.MetadataChg = true
opts.NoLock = true
// We don't know the size here set 1GiB at least.
idx, err := z.getPoolIdxExistingWithOpts(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
return z.serverPools[idx].PutObjectMetadata(ctx, bucket, object, opts)
return z.updatePoolMetadata(ctx, bucket, object, opts)
}
// PutObjectTags - replace or add tags to an existing object
@@ -2867,44 +3124,40 @@ func (z *erasureServerPools) PutObjectTags(ctx context.Context, bucket, object s
defer lk.Unlock(lkctx)
}
opts.MetadataChg = true
opts.NoLock = true
// We don't know the size here set 1GiB at least.
idx, err := z.getPoolIdxExistingWithOpts(ctx, bucket, object, opts)
copies, err := z.metadataPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
return z.serverPools[idx].PutObjectTags(ctx, bucket, object, tags, opts)
// Ordinary reads and replication can return any owning pool. Persist one
// revision beyond all copies, so the returned value and every pool agree.
if stamp := opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]; stamp != "" {
for _, copy := range copies {
stamp = monotonicTaggingTimestamp(stamp, copy.ObjInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
}
opts.UserDefined = cloneMSS(opts.UserDefined)
opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
}
opts.NoLock = true
opts.VersionID = copies[0].ObjInfo.VersionID
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
var primary ObjectInfo
for _, copy := range copies {
oi, err := z.serverPools[copy.Index].PutObjectTags(ctx, bucket, object, tags, opts)
if err != nil {
return ObjectInfo{}, err
}
if copy.Index == copies[0].Index {
primary = oi
}
}
return primary, nil
}
// DeleteObjectTags - delete object tags from an existing object
func (z *erasureServerPools) DeleteObjectTags(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
object = encodeDirObject(object)
if z.SinglePool() {
return z.serverPools[0].DeleteObjectTags(ctx, bucket, object, opts)
}
if !opts.NoLock {
// Lock the object before deleting tags.
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
return ObjectInfo{}, err
}
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
}
opts.MetadataChg = true
opts.NoLock = true
idx, err := z.getPoolIdxExistingWithOpts(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
return z.serverPools[idx].DeleteObjectTags(ctx, bucket, object, opts)
return z.PutObjectTags(ctx, bucket, object, "", opts)
}
// GetObjectTags - get object tags from an existing object
@@ -2914,7 +3167,7 @@ func (z *erasureServerPools) GetObjectTags(ctx context.Context, bucket, object s
return z.serverPools[0].GetObjectTags(ctx, bucket, object, opts)
}
oi, _, err := z.getLatestObjectInfoWithIdx(ctx, bucket, object, opts)
oi, err := z.GetObjectInfo(ctx, bucket, object, opts)
if err != nil {
return nil, err
}
+34 -4
View File
@@ -880,10 +880,40 @@ func (s *erasureSets) CopyObject(ctx context.Context, srcBucket, srcObject, dstB
}
func (s *erasureSets) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
// In list multipart uploads we are going to treat input prefix as the object,
// this means that we are not supporting directory navigation.
set := s.getHashedSet(prefix)
return set.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
uploads, legacy, err := s.scanMultipartUploads(scan, bucket, 0)
if err != nil {
return ListMultipartsInfo{}, err
}
if legacy {
return ListMultipartsInfo{}, errMultipartListingLegacy
}
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
func (s *erasureSets) scanMultipartUploads(scan *multipartScan, bucket string, poolIdx int) ([]MultipartInfo, bool, error) {
var uploads []MultipartInfo
var keyless bool
for i, set := range s.sets {
setUploads, setKeyless, err := set.scanMultipartUploads(scan, bucket, poolIdx, i)
if err != nil {
return nil, false, err
}
uploads = append(uploads, setUploads...)
keyless = keyless || setKeyless
}
return uploads, keyless, nil
}
func (s *erasureSets) listMultipartUploadsExact(ctx context.Context, bucket, object string) (ListMultipartsInfo, error) {
return s.getHashedSet(object).listMultipartUploadsExact(ctx, bucket, object, "", "", "", maxUploadsList)
}
// Initiate a new multipart upload on a hashedSet based on object name.
+8 -2
View File
@@ -166,6 +166,12 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
if objAPI == nil {
return errServerNotInitialized
}
healInPool := er.HealObject
if z, ok := objAPI.(*erasureServerPools); ok && !z.SinglePool() {
healInPool = func(ctx context.Context, bucket, object, versionID string, opts madmin.HealOpts) (madmin.HealResultItem, error) {
return z.healObjectInPool(ctx, er, bucket, object, versionID, opts)
}
}
started := tracker.Started
if started.IsZero() || started.Equal(timeSentinel) {
@@ -419,7 +425,7 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
var result healEntryResult
fivs, err := entry.fileInfoVersions(bucket)
if err != nil {
res, err := er.HealObject(ctx, bucket, encodedEntryName, "",
res, err := healInPool(ctx, bucket, encodedEntryName, "",
madmin.HealOpts{
ScanMode: scanMode,
Remove: healDeleteDangling,
@@ -455,7 +461,7 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
continue
}
res, err := er.HealObject(ctx, bucket, encodedEntryName,
res, err := healInPool(ctx, bucket, encodedEntryName,
version.VersionID, madmin.HealOpts{
ScanMode: scanMode,
Remove: healDeleteDangling,
+4 -6
View File
@@ -51,9 +51,8 @@ func initGlobalGrid(ctx context.Context, eps EndpointServerPools) error {
grid.ContextDialer(xhttp.DialContextWithLookupHost(lookupHost, xhttp.NewInternodeDialContext(rest.DefaultTimeout, globalTCPOptions.ForWebsocket()))),
newCachedAuthToken(),
&tls.Config{
RootCAs: globalRootCAs,
CipherSuites: crypto.TLSCiphers(),
CurvePreferences: crypto.TLSCurveIDs(),
RootCAs: globalRootCAs,
CipherSuites: crypto.TLSCiphers(),
}),
Local: local,
Hosts: hosts,
@@ -84,9 +83,8 @@ func initGlobalLockGrid(ctx context.Context, eps EndpointServerPools) error {
grid.ContextDialer(xhttp.DialContextWithLookupHost(lookupHost, xhttp.NewInternodeDialContext(rest.DefaultTimeout, globalTCPOptions.ForWebsocket()))),
newCachedAuthToken(),
&tls.Config{
RootCAs: globalRootCAs,
CipherSuites: crypto.TLSCiphers(),
CurvePreferences: crypto.TLSCurveIDs(),
RootCAs: globalRootCAs,
CipherSuites: crypto.TLSCiphers(),
}, grid.RouteLockPath),
Local: local,
Hosts: hosts,
+8
View File
@@ -50,6 +50,7 @@ type apiConfig struct {
transitionWorkers int
staleUploadsExpiry time.Duration
multipartListingStrict bool
staleUploadsCleanupInterval time.Duration
deleteCleanupInterval time.Duration
enableODirect bool
@@ -181,6 +182,7 @@ func (t *apiConfig) init(cfg api.Config, setDriveCounts []int, legacy bool) {
t.transitionWorkers = cfg.TransitionWorkers
t.staleUploadsExpiry = cfg.StaleUploadsExpiry
t.multipartListingStrict = cfg.MultipartListing == "strict"
t.deleteCleanupInterval = cfg.DeleteCleanupInterval
t.enableODirect = cfg.EnableODirect
t.gzipObjects = cfg.GzipObjects
@@ -206,6 +208,12 @@ func (t *apiConfig) odirectEnabled() bool {
return t.enableODirect
}
func (t *apiConfig) getMultipartListingLegacy() bool {
t.mu.RLock()
defer t.mu.RUnlock()
return !t.multipartListingStrict
}
func (t *apiConfig) shouldGzipObjects() bool {
t.mu.RLock()
defer t.mu.RUnlock()
+33 -23
View File
@@ -246,16 +246,6 @@ func extractMetadata(ctx context.Context, mimesHeader ...textproto.MIMEHeader) (
// extractMetadata extracts metadata from map values.
func extractMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
return extractMetadataFromMimeWithReplication(ctx, v, m, false)
}
// extractReplicationMetadataFromMime restores replication-only metadata after the
// caller has validated that the request is a trusted replication write.
func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
return extractMetadataFromMimeWithReplication(ctx, v, m, true)
}
func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIMEHeader, m map[string]string, allowReplication bool) error {
if v == nil {
bugLogIf(ctx, errInvalidArgument)
return errInvalidArgument
@@ -267,18 +257,14 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
nv[http.CanonicalHeaderKey(k)] = kv
}
// Save all supported headers.
// Save ordinary object metadata. Replication-only headers are restored only
// after the request has been validated as a trusted replication write.
for _, supportedHeader := range supportedHeaders {
value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]
if ok {
if v, ok := replicationToInternalHeaders[supportedHeader]; ok {
if !allowReplication {
continue
}
m[v] = strings.Join(value, ",")
} else {
m[supportedHeader] = strings.Join(value, ",")
}
if _, ok := replicationToInternalHeaders[supportedHeader]; ok {
continue
}
if value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]; ok {
m[supportedHeader] = strings.Join(value, ",")
}
}
@@ -287,8 +273,7 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
if !stringsHasPrefixFold(key, prefix) {
continue
}
value, ok := nv[http.CanonicalHeaderKey(key)]
if ok {
if value, ok := nv[http.CanonicalHeaderKey(key)]; ok {
m[key] = strings.Join(value, ",")
break
}
@@ -297,6 +282,31 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
return nil
}
// extractReplicationMetadataFromMime restores replication-only metadata after the
// caller has validated that the request is a trusted replication write.
func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
if v == nil {
bugLogIf(ctx, errInvalidArgument)
return errInvalidArgument
}
nv := make(textproto.MIMEHeader, len(v))
for k, kv := range v {
// Canonicalize all headers, to remove any duplicates.
nv[http.CanonicalHeaderKey(k)] = kv
}
// Ordinary object metadata belongs to the caller. Re-extracting it would
// undo normalization (such as removing aws-chunked) or copy an outer
// Snowball archive's metadata onto its individual entries.
for header, internalHeader := range replicationToInternalHeaders {
if value, ok := nv[http.CanonicalHeaderKey(header)]; ok {
m[internalHeader] = strings.Join(value, ",")
}
}
return nil
}
// Returns access credentials in the request Authorization header.
func getReqAccessCred(r *http.Request, region string) (cred auth.Credentials) {
cred, _, _ = getReqAccessKeyV4(r, region, serviceS3)
+12 -1
View File
@@ -254,6 +254,9 @@ func TestExtractMetadataFromRequestKeepsQueryCompatibility(t *testing.T) {
func TestExtractReplicationMetadataHeaders(t *testing.T) {
header := http.Header{
"Content-Type": []string{"application/wasm"},
"Content-Encoding": []string{"aws-chunked"},
"X-Amz-Meta-Source": []string{"client"},
"X-Minio-Replication-Server-Side-Encryption-Sealed-Key": []string{"sealed-key"},
"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
"X-Minio-Replication-Server-Side-Encryption-Iv": []string{"iv"},
@@ -262,12 +265,17 @@ func TestExtractReplicationMetadataHeaders(t *testing.T) {
ReplicationSsecChecksumHeader: []string{"checksum"},
}
metadata := make(map[string]string)
metadata := map[string]string{
"content-type": "application/wasm",
"x-amz-meta-source": "client",
}
if err := extractReplicationMetadataFromMime(t.Context(), textproto.MIMEHeader(header), metadata); err != nil {
t.Fatalf("failed to extract replication metadata: %v", err)
}
expected := map[string]string{
"content-type": "application/wasm",
"x-amz-meta-source": "client",
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key": "sealed-key",
"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "DAREv2-HMAC-SHA256",
"X-Minio-Internal-Server-Side-Encryption-Iv": "iv",
@@ -279,6 +287,9 @@ func TestExtractReplicationMetadataHeaders(t *testing.T) {
if !reflect.DeepEqual(metadata, expected) {
t.Fatalf("unexpected replication metadata: expected %#v, got %#v", expected, metadata)
}
if _, ok := metadata["content-encoding"]; ok {
t.Fatalf("replication metadata restored transport content-encoding: %#v", metadata)
}
}
func TestGetCopyObjectMetadataFromHeaderReplication(t *testing.T) {
+111
View File
@@ -0,0 +1,111 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"errors"
"testing"
"time"
"github.com/minio/minio/internal/auth"
)
func TestIAMCredentialRetention(t *testing.T) {
for _, backend := range []string{"object", "etcd"} {
t.Run(backend, func(t *testing.T) {
ctx, sys, _ := prepareIAMRevisionFixture(t, backend)
secret, err := getTokenSigningKey()
mustIAM(t, err)
parent := "external-idp-parent"
credential := func(exp time.Time) auth.Credentials {
cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": exp.Unix(), parentClaim: parent}, secret)
mustIAM(t, err)
cred.ParentUser = parent
return cred
}
// Disablement of an external identity must include cached STS,
// which are kept separately from regular and service accounts.
cred := credential(UTCNow().Add(time.Hour))
_, err = sys.SetTempUser(ctx, cred.AccessKey, cred, "")
mustIAM(t, err)
mustIAM(t, sys.store.DeleteUsers(ctx, []string{parent}))
r, err := loadIAMRevision(ctx, sys.store, getUserIdentityPath(cred.AccessKey, stsUser))
mustIAM(t, err)
if !r.Deleted || !r.ExpiresAt.Equal(cred.Expiration.Add(globalMaxSkewTime)) || r.Credentials.SessionToken != "" || r.Credentials.SecretKey != "" {
t.Fatal("early STS revocation lost its retention boundary or retained a secret")
}
if _, ok := sys.store.GetUser(cred.AccessKey); ok {
t.Fatal("external disablement left the STS cache live")
}
_, err = sys.SetTempUser(withIAMReplicationTime(ctx, UTCNow().Add(time.Minute)), cred.AccessKey, cred, "")
if !errors.Is(err, errIAMStaleUpdate) {
t.Fatalf("same revoked token was reissued by replay: %v", err)
}
var mp MappedPolicy
err = sys.store.loadIAMConfig(ctx, &mp, getMappedPolicyPath(cred.AccessKey, stsUser, false))
if !errors.Is(err, errConfigNotFound) {
t.Fatalf("random STS key produced a permanent mapping: %v", err)
}
// Seed genuinely expired immutable tokens, as an ordinary startup
// loader sees them. Natural expiry leaves no permanent tombstone.
expired := credential(UTCNow().Add(-time.Hour))
path := getUserIdentityPath(expired.AccessKey, stsUser)
mustIAM(t, sys.store.saveIAMConfig(ctx, &UserIdentity{Version: 1, Credentials: expired, UpdatedAt: UTCNow().Add(-2 * time.Hour)}, path))
_ = sys.store.loadUser(ctx, expired.AccessKey, stsUser, make(map[string]UserIdentity))
var u UserIdentity
if err := sys.store.loadIAMConfig(ctx, &u, path); !errors.Is(err, errConfigNotFound) {
t.Fatalf("natural expiration retained a random key: %v", err)
}
// A retained early-revocation record is collectable only after the
// immutable token's expiration plus the skew allowance.
tomb := UserIdentity{Version: 1, Deleted: true, UpdatedAt: UTCNow().Add(-2 * time.Hour), ExpiresAt: expired.Expiration.Add(globalMaxSkewTime)}
mustIAM(t, sys.store.saveIAMConfig(ctx, &tomb, path))
_ = sys.store.loadUser(ctx, expired.AccessKey, stsUser, make(map[string]UserIdentity))
if err := sys.store.loadIAMConfig(ctx, &u, path); !errors.Is(err, errConfigNotFound) {
t.Fatalf("expired STS revocation not collected: %v", err)
}
if _, ok := sys.store.revisionIndex().snapshot()[path]; ok {
t.Fatal("expired STS retained an index entry")
}
})
}
}
func TestIAMPolicyDeletionRemainsExplicit(t *testing.T) {
for _, backend := range []string{"object", "etcd"} {
t.Run(backend, func(t *testing.T) {
ctx, sys, _ := prepareIAMRevisionFixture(t, backend)
mustIAM(t, sys.DeletePolicy(ctx, "misspelled-policy", true))
r, err := loadIAMRevision(ctx, sys.store, getPolicyDocPath("misspelled-policy"))
mustIAM(t, err)
if r.Deleted {
t.Fatal("local nonexistent policy created a tombstone")
}
p, err := sys.store.GetPolicy("readwrite")
mustIAM(t, err)
if err := sys.DeletePolicy(ctx, "readwrite", true); err == nil {
t.Fatal("local pristine builtin policy became deletable")
}
_, err = sys.SetPolicy(ctx, "readwrite", p)
mustIAM(t, err)
mustIAM(t, sys.DeletePolicy(ctx, "readwrite", true))
mustIAM(t, sys.store.LoadIAMCache(ctx, false))
if _, err := sys.store.GetPolicy("readwrite"); !errors.Is(err, errNoSuchPolicy) {
t.Fatalf("reload restored an explicitly deleted override: %v", err)
}
_, err = sys.SetPolicy(ctx, "readwrite", p)
mustIAM(t, err)
if _, err := sys.store.GetPolicy("readwrite"); err != nil {
t.Fatal("explicit policy recreation failed", err)
}
mustIAM(t, globalSiteReplicationSys.PeerAddPolicyHandler(ctx, "remote-unknown-policy", nil, UTCNow()))
r, err = loadIAMRevision(ctx, sys.store, getPolicyDocPath("remote-unknown-policy"))
mustIAM(t, err)
if !r.Deleted {
t.Fatal("replicated unknown deletion lost its version")
}
})
}
}
+65 -71
View File
@@ -26,7 +26,6 @@ import (
"sync"
"time"
jsoniter "github.com/json-iterator/go"
"github.com/minio/minio-go/v7/pkg/set"
"github.com/minio/minio/internal/config"
"github.com/minio/minio/internal/kms"
@@ -62,6 +61,7 @@ type IAMEtcdStore struct {
sync.RWMutex
*iamCache
index iamRevisionIndex
usersSysType UsersSysType
@@ -69,13 +69,17 @@ type IAMEtcdStore struct {
}
func newIAMEtcdStore(client *etcd.Client, usersSysType UsersSysType) *IAMEtcdStore {
return &IAMEtcdStore{
store := &IAMEtcdStore{
iamCache: newIamCache(),
client: client,
usersSysType: usersSysType,
}
store.revisions = &store.index
return store
}
func (ies *IAMEtcdStore) revisionIndex() *iamRevisionIndex { return &ies.index }
func (ies *IAMEtcdStore) rlock() *iamCache {
ies.RLock()
return ies.iamCache
@@ -103,6 +107,7 @@ func (ies *IAMEtcdStore) saveIAMConfig(ctx context.Context, item any, itemPath s
if err != nil {
return err
}
plain := data
if GlobalKMS != nil {
data, err = config.EncryptBytes(GlobalKMS, data, kms.Context{
minioMetaBucket: path.Join(minioMetaBucket, itemPath),
@@ -111,24 +116,28 @@ func (ies *IAMEtcdStore) saveIAMConfig(ctx context.Context, item any, itemPath s
return err
}
}
return saveKeyEtcd(ctx, ies.client, itemPath, data, opts...)
if err := saveKeyEtcd(ctx, ies.client, itemPath, data, opts...); err != nil {
return err
}
ies.index.observe(itemPath, plain)
return nil
}
func getIAMConfig(item any, data []byte, itemPath string) error {
data, err := decryptData(data, itemPath)
func (ies *IAMEtcdStore) decodeIAMConfig(item any, data []byte, path string) error {
data, err := decryptData(data, path)
if err != nil {
return err
}
json := jsoniter.ConfigCompatibleWithStandardLibrary
ies.index.observe(path, data)
return json.Unmarshal(data, item)
}
func (ies *IAMEtcdStore) loadIAMConfig(ctx context.Context, item any, path string) error {
data, err := readKeyEtcd(ctx, ies.client, path)
data, err := ies.loadIAMConfigBytes(ctx, path)
if err != nil {
return err
}
return getIAMConfig(item, data, path)
return json.Unmarshal(data, item)
}
func (ies *IAMEtcdStore) loadIAMConfigBytes(ctx context.Context, path string) ([]byte, error) {
@@ -136,11 +145,19 @@ func (ies *IAMEtcdStore) loadIAMConfigBytes(ctx context.Context, path string) ([
if err != nil {
return nil, err
}
return decryptData(data, path)
data, err = decryptData(data, path)
if err == nil {
ies.index.observe(path, data)
}
return data, err
}
func (ies *IAMEtcdStore) deleteIAMConfig(ctx context.Context, path string) error {
return deleteKeyEtcd(ctx, ies.client, path)
if err := deleteKeyEtcd(ctx, ies.client, path); err != nil {
return err
}
ies.index.forget(path)
return nil
}
func (ies *IAMEtcdStore) loadPolicyDocWithRetry(ctx context.Context, policy string, m map[string]PolicyDoc, _ int) error {
@@ -162,6 +179,9 @@ func (ies *IAMEtcdStore) loadPolicyDoc(ctx context.Context, policy string, m map
return err
}
if p.Deleted {
return errNoSuchPolicy
}
m[policy] = p
return nil
}
@@ -181,7 +201,11 @@ func (ies *IAMEtcdStore) getPolicyDocKV(ctx context.Context, kvs *mvccpb.KeyValu
return err
}
ies.index.observe(string(kvs.Key), data)
policy := extractPathPrefixAndSuffix(string(kvs.Key), iamConfigPoliciesPrefix, path.Base(string(kvs.Key)))
if p.Deleted {
return errNoSuchPolicy
}
m[policy] = p
return nil
}
@@ -207,7 +231,7 @@ func (ies *IAMEtcdStore) loadPolicyDocs(ctx context.Context, m map[string]Policy
func (ies *IAMEtcdStore) getUserKV(ctx context.Context, userkv *mvccpb.KeyValue, userType IAMUserType, m map[string]UserIdentity, basePrefix string) error {
var u UserIdentity
err := getIAMConfig(&u, userkv.Value, string(userkv.Key))
err := ies.decodeIAMConfig(&u, userkv.Value, string(userkv.Key))
if err != nil {
if err == errConfigNotFound {
return errNoSuchUser
@@ -219,10 +243,14 @@ func (ies *IAMEtcdStore) getUserKV(ctx context.Context, userkv *mvccpb.KeyValue,
}
func (ies *IAMEtcdStore) addUser(ctx context.Context, user string, userType IAMUserType, u UserIdentity, m map[string]UserIdentity) error {
if u.Deleted {
if userType == stsUser && !u.ExpiresAt.IsZero() && UTCNow().After(u.ExpiresAt) {
bestEffortIAMExpiration(ctx, ies, getUserIdentityPath(user, userType))
}
return errNoSuchUser
}
if u.Credentials.IsExpired() {
// Delete expired identity.
deleteKeyEtcd(ctx, ies.client, getUserIdentityPath(user, userType))
deleteKeyEtcd(ctx, ies.client, getMappedPolicyPath(user, userType, false))
bestEffortIAMExpiration(ctx, ies, getUserIdentityPath(user, userType))
return nil
}
if u.Credentials.AccessKey == "" {
@@ -231,16 +259,17 @@ func (ies *IAMEtcdStore) addUser(ctx context.Context, user string, userType IAMU
if u.Credentials.SessionToken != "" {
jwtClaims, err := extractJWTClaims(u)
if err != nil {
if u.Credentials.IsTemp() {
// We should delete such that the client can re-request
// for the expiring credentials.
deleteKeyEtcd(ctx, ies.client, getUserIdentityPath(user, userType))
deleteKeyEtcd(ctx, ies.client, getMappedPolicyPath(user, userType, false))
}
// A temporarily unavailable signing key is not proof of expiration.
return nil
}
u.Credentials.Claims = jwtClaims.Map()
}
if err := checkIAMParentRevision(ctx, ies, u.Credentials); err != nil {
if errors.Is(err, errIAMStaleUpdate) {
return errNoSuchUser
}
return err
}
if u.Credentials.Description == "" {
u.Credentials.Description = u.Credentials.Comment
}
@@ -258,6 +287,9 @@ func (ies *IAMEtcdStore) loadSecretKey(ctx context.Context, user string, userTyp
}
return "", err
}
if u.Deleted {
return "", errNoSuchUser
}
return u.Credentials.SecretKey, nil
}
@@ -274,6 +306,7 @@ func (ies *IAMEtcdStore) loadUser(ctx context.Context, user string, userType IAM
}
func (ies *IAMEtcdStore) loadUsers(ctx context.Context, userType IAMUserType, m map[string]UserIdentity) error {
ctx = withIAMExpirationCleanup(ctx)
var basePrefix string
switch userType {
case svcUser:
@@ -312,6 +345,9 @@ func (ies *IAMEtcdStore) loadGroup(ctx context.Context, group string, m map[stri
}
return err
}
if gi.Deleted {
return errNoSuchGroup
}
m[group] = gi
return nil
}
@@ -349,13 +385,16 @@ func (ies *IAMEtcdStore) loadMappedPolicy(ctx context.Context, name string, user
}
return err
}
if !ies.index.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), p) {
return errNoSuchPolicy
}
m.Store(name, p)
return nil
}
func getMappedPolicy(kv *mvccpb.KeyValue, m *xsync.MapOf[string, MappedPolicy], basePrefix string) error {
func (ies *IAMEtcdStore) getMappedPolicy(kv *mvccpb.KeyValue, m *xsync.MapOf[string, MappedPolicy], basePrefix string) error {
var p MappedPolicy
err := getIAMConfig(&p, kv.Value, string(kv.Key))
err := ies.decodeIAMConfig(&p, kv.Value, string(kv.Key))
if err != nil {
if err == errConfigNotFound {
return errNoSuchPolicy
@@ -363,6 +402,9 @@ func getMappedPolicy(kv *mvccpb.KeyValue, m *xsync.MapOf[string, MappedPolicy],
return err
}
name := extractPathPrefixAndSuffix(string(kv.Key), basePrefix, ".json")
if !ies.index.mappingAllowed(string(kv.Key), p) {
return errNoSuchPolicy
}
m.Store(name, p)
return nil
}
@@ -392,61 +434,13 @@ func (ies *IAMEtcdStore) loadMappedPolicies(ctx context.Context, userType IAMUse
// Parse all policies mapping to create the proper data model
for _, kv := range r.Kvs {
if err = getMappedPolicy(kv, m, basePrefix); err != nil && !errors.Is(err, errNoSuchPolicy) {
if err = ies.getMappedPolicy(kv, m, basePrefix); err != nil && !errors.Is(err, errNoSuchPolicy) {
return err
}
}
return nil
}
func (ies *IAMEtcdStore) savePolicyDoc(ctx context.Context, policyName string, p PolicyDoc) error {
return ies.saveIAMConfig(ctx, &p, getPolicyDocPath(policyName))
}
func (ies *IAMEtcdStore) saveMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool, mp MappedPolicy, opts ...options) error {
return ies.saveIAMConfig(ctx, mp, getMappedPolicyPath(name, userType, isGroup), opts...)
}
func (ies *IAMEtcdStore) saveUserIdentity(ctx context.Context, name string, userType IAMUserType, u UserIdentity, opts ...options) error {
return ies.saveIAMConfig(ctx, u, getUserIdentityPath(name, userType), opts...)
}
func (ies *IAMEtcdStore) saveGroupInfo(ctx context.Context, name string, gi GroupInfo) error {
return ies.saveIAMConfig(ctx, gi, getGroupInfoPath(name))
}
func (ies *IAMEtcdStore) deletePolicyDoc(ctx context.Context, name string) error {
err := ies.deleteIAMConfig(ctx, getPolicyDocPath(name))
if err == errConfigNotFound {
err = errNoSuchPolicy
}
return err
}
func (ies *IAMEtcdStore) deleteMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool) error {
err := ies.deleteIAMConfig(ctx, getMappedPolicyPath(name, userType, isGroup))
if err == errConfigNotFound {
err = errNoSuchPolicy
}
return err
}
func (ies *IAMEtcdStore) deleteUserIdentity(ctx context.Context, name string, userType IAMUserType) error {
err := ies.deleteIAMConfig(ctx, getUserIdentityPath(name, userType))
if err == errConfigNotFound {
err = errNoSuchUser
}
return err
}
func (ies *IAMEtcdStore) deleteGroupInfo(ctx context.Context, name string) error {
err := ies.deleteIAMConfig(ctx, getGroupInfoPath(name))
if err == errConfigNotFound {
err = errNoSuchGroup
}
return err
}
func (ies *IAMEtcdStore) watch(ctx context.Context, keyPath string) <-chan iamWatchEvent {
ch := make(chan iamWatchEvent)
+164
View File
@@ -0,0 +1,164 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"maps"
"slices"
"time"
"github.com/minio/minio-go/v7/pkg/set"
)
type (
iamGroupGrantsKey struct{}
iamGroupMutationKey struct{}
iamGroupMutation struct {
Members []string
Remove bool
StatusOnly bool
}
)
// Merge the intended mutation with the record read under the distributed
// revision lock, not the older cache used to prepare the request.
func mergeIAMGroupMutation(ctx context.Context, previous GroupInfo, next *GroupInfo) {
op, ok := ctx.Value(iamGroupMutationKey{}).(iamGroupMutation)
if !ok || previous.Deleted || previous.Version == 0 {
return
}
members := set.CreateStringSet(previous.Members...)
grants := maps.Clone(previous.MemberGrants)
if grants == nil {
grants = make(map[string]time.Time)
}
switch {
case op.StatusOnly:
// Only the status changes.
case op.Remove:
for _, member := range op.Members {
members.Remove(member)
delete(grants, member)
}
next.Status = previous.Status
default:
requested := set.CreateStringSet(next.Members...)
for _, member := range op.Members {
if !requested.Contains(member) {
continue
}
at := next.MemberGrants[member]
if at.Before(grants[member]) {
continue
}
members.Add(member)
grants[member] = at
}
next.Status = previous.Status
}
next.Members, next.MemberGrants = members.ToSlice(), grants
slices.Sort(next.Members)
}
// A non-nil map is supplied by the versioned peer envelope, including for
// snapshots. Missing times are unknown, never the snapshot's newer timestamp.
func withIAMGroupGrants(ctx context.Context, grants map[string]time.Time) context.Context {
return context.WithValue(ctx, iamGroupGrantsKey{}, grants)
}
func (c *iamCache) effectiveGroupMembers(gi GroupInfo) []string {
var members []string
for _, member := range gi.Members {
if c.groupMemberAllowed(member, gi.MemberGrants[member], gi.RevokedBefore) {
members = append(members, member)
}
}
return members
}
func (c *iamCache) effectiveUserGroups(user string) []string {
var groups []string
for group := range c.iamUserGroupMemberships[user] {
gi, ok := c.iamGroupsMap[group]
r := c.revisions.get(getGroupInfoPath(group))
if r.RevokedBefore.After(gi.RevokedBefore) {
gi.RevokedBefore = r.RevokedBefore
}
if ok && !r.Deleted && c.groupMemberAllowed(user, gi.MemberGrants[user], gi.RevokedBefore) {
groups = append(groups, group)
}
}
return groups
}
func (c *iamCache) addGroupMembers(ctx context.Context, gi GroupInfo, members []string) (GroupInfo, error) {
grants, versioned := ctx.Value(iamGroupGrantsKey{}).(map[string]time.Time)
if boundary, ok := ctx.Value(iamRecordBoundaryKey{}).(time.Time); ok && boundary.After(gi.RevokedBefore) {
gi.RevokedBefore = boundary
}
origin, replicated := iamReplicationTime(ctx)
gi.Members = slices.Clone(gi.Members)
gi.MemberGrants = maps.Clone(gi.MemberGrants)
if gi.MemberGrants == nil {
gi.MemberGrants = make(map[string]time.Time)
}
current := set.CreateStringSet(gi.Members...)
gi.UpdatedAt = UTCNow()
if replicated {
gi.UpdatedAt = origin
}
for _, member := range members {
at := gi.UpdatedAt
r := c.userRevocation(member)
if replicated {
switch {
case versioned:
at = grants[member]
if at.After(origin) {
return gi, errInvalidArgument
}
case !r.RevokedBefore.IsZero() || r.Deleted || !gi.RevokedBefore.IsZero():
// Legacy snapshots cannot prove a post-revocation grant.
continue
case current.Contains(member):
continue
}
if !c.groupMemberAllowed(member, at, gi.RevokedBefore) {
continue
}
} else {
if current.Contains(member) && c.groupMemberAllowed(member, gi.MemberGrants[member], gi.RevokedBefore) {
continue // Editing the group is not reissuing every grant.
}
if !at.After(gi.RevokedBefore) {
at = gi.RevokedBefore.Add(time.Nanosecond)
}
if !at.After(r.RevokedBefore) {
at = r.RevokedBefore.Add(time.Nanosecond)
}
if !at.After(gi.MemberGrants[member]) {
at = gi.MemberGrants[member].Add(time.Nanosecond)
}
if at.After(gi.UpdatedAt) {
gi.UpdatedAt = at
}
}
u, ok := c.iamUsersMap[member]
if !ok {
return gi, errNoSuchUser
}
if u.Credentials.IsTemp() || u.Credentials.IsServiceAccount() {
return gi, errIAMActionNotAllowed
}
if previous := gi.MemberGrants[member]; previous.After(at) {
continue
}
current.Add(member)
gi.MemberGrants[member] = at
}
gi.Members = current.ToSlice()
slices.Sort(gi.Members)
return gi, nil
}
+75
View File
@@ -0,0 +1,75 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"testing"
"time"
)
func TestIAMHealingResumesAfterLeadershipLoss(t *testing.T) {
previous := globalLeaderLock
locks := make(chan LockContext)
globalLeaderLock = &sharedLock{lockContext: locks}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
c := &SiteReplicationSys{}
go func() { c.startHealRoutine(ctx, nil); close(done) }()
t.Cleanup(func() {
cancel()
select {
case <-done:
case locks <- LockContext{ctx: ctx}:
}
<-done
globalLeaderLock = previous
})
first, loseFirst := context.WithCancel(ctx)
defer loseFirst()
select {
case locks <- LockContext{ctx: first}:
case <-time.After(time.Second):
t.Fatal("healer did not acquire its first leader context")
}
loseFirst() // A transient quorum loss cancels the distributed lease.
select {
case <-done:
t.Fatal("healer permanently exited after temporary leadership loss")
case locks <- LockContext{ctx: ctx}:
case <-time.After(time.Second):
t.Fatal("healer did not wait for reacquired leadership")
}
// Shutdown must also interrupt the wait for leadership after lease loss.
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("healer did not stop with its owning context")
}
}
func TestIAMHealingLeadershipWaitCancels(t *testing.T) {
previous := globalLeaderLock
locks := make(chan LockContext)
globalLeaderLock = &sharedLock{lockContext: locks}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
c := &SiteReplicationSys{}
go func() { c.startHealRoutine(ctx, nil); close(done) }()
cancel()
t.Cleanup(func() {
select {
case <-done:
case locks <- LockContext{ctx: ctx}:
}
<-done
globalLeaderLock = previous
})
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("healer ignored shutdown while waiting for leadership")
}
}
+60
View File
@@ -0,0 +1,60 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
)
// Measures steady-state index traversal, sorting and the capability request.
// The network peer acknowledges real batches but performs no disk I/O; this
// benchmark deliberately does not claim durable catch-up throughput.
func BenchmarkIAMRevisionConvergedHealing(b *testing.B) {
for _, n := range []int{1000, 10000} {
b.Run(fmt.Sprint(n), func(b *testing.B) {
ctx, sys, _ := prepareIAMRevisionFixture(b)
_, err := sys.CreateUser(ctx, "benchmark-sync", madmin.AddOrUpdateUserReq{SecretKey: "valid-sync-password", Status: madmin.AccountEnabled})
mustIAM(b, err)
for i := range n {
at := UTCNow().Add(time.Duration(i) * time.Nanosecond)
data, err := json.Marshal(iamRevision{Deleted: true, UpdatedAt: at, RevokedBefore: at})
mustIAM(b, err)
sys.store.revisionIndex().observe(getUserIdentityPath(fmt.Sprintf("deleted-%06d", i), regUser), data)
}
var puts atomic.Int64
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/minio/health/live" {
w.WriteHeader(http.StatusOK)
return
}
if r.Method == http.MethodPut {
puts.Add(1)
}
_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "node-1", Instance: "benchmark-peer", Digest: "constant"}})
}))
defer server.Close()
c := &SiteReplicationSys{enabled: true, state: srState{ServiceAccountAccessKey: "benchmark-sync", Peers: map[string]madmin.PeerInfo{globalDeploymentID(): {DeploymentID: globalDeploymentID(), Name: "local"}, "remote": {DeploymentID: "remote", Name: "remote", Endpoint: server.URL}}}}
mustIAM(b, c.healIAMDeletions(ctx))
before := puts.Load()
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
mustIAM(b, c.healIAMDeletions(ctx))
}
b.StopTimer()
b.ReportMetric(float64(puts.Load()-before)/float64(b.N), "PUT/op")
if puts.Load() != before {
b.Fatal("steady-state healing replayed acknowledged records")
}
})
}
}
+56 -61
View File
@@ -45,6 +45,7 @@ type IAMObjectStore struct {
sync.RWMutex
*iamCache
index iamRevisionIndex
usersSysType UsersSysType
@@ -52,13 +53,17 @@ type IAMObjectStore struct {
}
func newIAMObjectStore(objAPI ObjectLayer, usersSysType UsersSysType) *IAMObjectStore {
return &IAMObjectStore{
store := &IAMObjectStore{
iamCache: newIamCache(),
objAPI: objAPI,
usersSysType: usersSysType,
}
store.revisions = &store.index
return store
}
func (iamOS *IAMObjectStore) revisionIndex() *iamRevisionIndex { return &iamOS.index }
func (iamOS *IAMObjectStore) rlock() *iamCache {
iamOS.RLock()
return iamOS.iamCache
@@ -87,6 +92,7 @@ func (iamOS *IAMObjectStore) saveIAMConfig(ctx context.Context, item any, objPat
if err != nil {
return err
}
plain := data
if GlobalKMS != nil {
data, err = config.EncryptBytes(GlobalKMS, data, kms.Context{
minioMetaBucket: path.Join(minioMetaBucket, objPath),
@@ -95,7 +101,11 @@ func (iamOS *IAMObjectStore) saveIAMConfig(ctx context.Context, item any, objPat
return err
}
}
return saveConfig(ctx, iamOS.objAPI, objPath, data)
if err := saveConfig(ctx, iamOS.objAPI, objPath, data); err != nil {
return err
}
iamOS.index.observe(objPath, plain)
return nil
}
func decryptData(data []byte, objPath string) ([]byte, error) {
@@ -133,6 +143,7 @@ func (iamOS *IAMObjectStore) loadIAMConfigBytesWithMetadata(ctx context.Context,
if err != nil {
return nil, meta, err
}
iamOS.index.observe(objPath, data)
return data, meta, nil
}
@@ -146,7 +157,11 @@ func (iamOS *IAMObjectStore) loadIAMConfig(ctx context.Context, item any, objPat
}
func (iamOS *IAMObjectStore) deleteIAMConfig(ctx context.Context, path string) error {
return deleteConfig(ctx, iamOS.objAPI, path)
if err := deleteConfig(ctx, iamOS.objAPI, path); err != nil {
return err
}
iamOS.index.forget(path)
return nil
}
func (iamOS *IAMObjectStore) loadPolicyDocWithRetry(ctx context.Context, policy string, m map[string]PolicyDoc, retries int) error {
@@ -171,6 +186,10 @@ func (iamOS *IAMObjectStore) loadPolicyDocWithRetry(ctx context.Context, policy
return err
}
if p.Deleted {
return errNoSuchPolicy
}
if p.Version == 0 {
// This means that policy was in the old version (without any
// timestamp info). We fetch the mod time of the file and save
@@ -200,6 +219,10 @@ func (iamOS *IAMObjectStore) loadPolicy(ctx context.Context, policy string) (Pol
return p, err
}
if p.Deleted {
return PolicyDoc{}, errNoSuchPolicy
}
if p.Version == 0 {
// This means that policy was in the old version (without any
// timestamp info). We fetch the mod time of the file and save
@@ -245,6 +268,9 @@ func (iamOS *IAMObjectStore) loadSecretKey(ctx context.Context, user string, use
}
return "", err
}
if u.Deleted {
return "", errNoSuchUser
}
return u.Credentials.SecretKey, nil
}
@@ -258,10 +284,15 @@ func (iamOS *IAMObjectStore) loadUserIdentity(ctx context.Context, user string,
return u, err
}
if u.Deleted {
if userType == stsUser && !u.ExpiresAt.IsZero() && UTCNow().After(u.ExpiresAt) {
bestEffortIAMExpiration(ctx, iamOS, getUserIdentityPath(user, userType))
}
return UserIdentity{}, errNoSuchUser
}
if u.Credentials.IsExpired() {
// Delete expired identity - ignoring errors here.
iamOS.deleteIAMConfig(ctx, getUserIdentityPath(user, userType))
iamOS.deleteIAMConfig(ctx, getMappedPolicyPath(user, userType, false))
bestEffortIAMExpiration(ctx, iamOS, getUserIdentityPath(user, userType))
return u, errNoSuchUser
}
@@ -272,16 +303,18 @@ func (iamOS *IAMObjectStore) loadUserIdentity(ctx context.Context, user string,
if u.Credentials.SessionToken != "" {
jwtClaims, err := extractJWTClaims(u)
if err != nil {
if u.Credentials.IsTemp() {
// We should delete such that the client can re-request
// for the expiring credentials.
iamOS.deleteIAMConfig(ctx, getUserIdentityPath(user, userType))
iamOS.deleteIAMConfig(ctx, getMappedPolicyPath(user, userType, false))
}
return u, errNoSuchUser
// During startup the site signing key may not be available yet.
// Reject this load without deleting a credential that has not expired.
return UserIdentity{}, errNoSuchUser
}
u.Credentials.Claims = jwtClaims.Map()
}
if err := checkIAMParentRevision(ctx, iamOS, u.Credentials); err != nil {
if errors.Is(err, errIAMStaleUpdate) {
return UserIdentity{}, errNoSuchUser
}
return UserIdentity{}, err
}
if u.Credentials.Description == "" {
u.Credentials.Description = u.Credentials.Comment
@@ -320,6 +353,7 @@ func (iamOS *IAMObjectStore) loadUser(ctx context.Context, user string, userType
}
func (iamOS *IAMObjectStore) loadUsers(ctx context.Context, userType IAMUserType, m map[string]UserIdentity) error {
ctx = withIAMExpirationCleanup(ctx)
var basePrefix string
switch userType {
case svcUser:
@@ -354,6 +388,9 @@ func (iamOS *IAMObjectStore) loadGroup(ctx context.Context, group string, m map[
}
return err
}
if g.Deleted {
return errNoSuchGroup
}
m[group] = g
return nil
}
@@ -391,6 +428,9 @@ func (iamOS *IAMObjectStore) loadMappedPolicyWithRetry(ctx context.Context, name
goto retry
}
if !iamOS.index.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), p) {
return errNoSuchPolicy
}
m.Store(name, p)
return nil
}
@@ -405,6 +445,9 @@ func (iamOS *IAMObjectStore) loadMappedPolicyInternal(ctx context.Context, name
}
return p, err
}
if !iamOS.index.mappingAllowed(getMappedPolicyPath(name, userType, isGroup), p) {
return MappedPolicy{}, errNoSuchPolicy
}
return p, nil
}
@@ -824,54 +867,6 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
return nil
}
func (iamOS *IAMObjectStore) savePolicyDoc(ctx context.Context, policyName string, p PolicyDoc) error {
return iamOS.saveIAMConfig(ctx, &p, getPolicyDocPath(policyName))
}
func (iamOS *IAMObjectStore) saveMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool, mp MappedPolicy, opts ...options) error {
return iamOS.saveIAMConfig(ctx, mp, getMappedPolicyPath(name, userType, isGroup), opts...)
}
func (iamOS *IAMObjectStore) saveUserIdentity(ctx context.Context, name string, userType IAMUserType, u UserIdentity, opts ...options) error {
return iamOS.saveIAMConfig(ctx, u, getUserIdentityPath(name, userType), opts...)
}
func (iamOS *IAMObjectStore) saveGroupInfo(ctx context.Context, name string, gi GroupInfo) error {
return iamOS.saveIAMConfig(ctx, gi, getGroupInfoPath(name))
}
func (iamOS *IAMObjectStore) deletePolicyDoc(ctx context.Context, name string) error {
err := iamOS.deleteIAMConfig(ctx, getPolicyDocPath(name))
if err == errConfigNotFound {
err = errNoSuchPolicy
}
return err
}
func (iamOS *IAMObjectStore) deleteMappedPolicy(ctx context.Context, name string, userType IAMUserType, isGroup bool) error {
err := iamOS.deleteIAMConfig(ctx, getMappedPolicyPath(name, userType, isGroup))
if err == errConfigNotFound {
err = errNoSuchPolicy
}
return err
}
func (iamOS *IAMObjectStore) deleteUserIdentity(ctx context.Context, name string, userType IAMUserType) error {
err := iamOS.deleteIAMConfig(ctx, getUserIdentityPath(name, userType))
if err == errConfigNotFound {
err = errNoSuchUser
}
return err
}
func (iamOS *IAMObjectStore) deleteGroupInfo(ctx context.Context, name string) error {
err := iamOS.deleteIAMConfig(ctx, getGroupInfoPath(name))
if err == errConfigNotFound {
err = errNoSuchGroup
}
return err
}
// Lists objects in the minioMetaBucket at the given path prefix. All returned
// items have the pathPrefix removed from their names.
func listIAMConfigItems(ctx context.Context, objAPI ObjectLayer, pathPrefix string) <-chan itemOrErr[string] {
+133
View File
@@ -0,0 +1,133 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"os"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/grid"
"github.com/pgsty/silo-pkg/v3/policy"
)
// Two independent IAM caches share the same real object backend, as sibling
// nodes do. Deliver the actual peer handler only after the source committed.
func TestIAMPeerDeleteNotificationReloadsCommittedState(t *testing.T) {
for _, name := range []string{"deleted", "recreated", "recreated_without_grant"} {
recreate := name != "deleted"
t.Run(name, func(t *testing.T) {
resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
obj, disk, err := prepareFS(ctx)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(disk)
defer obj.Shutdown(ctx)
defer resetTestGlobals()
globalObjLayerMutex.Lock()
globalObjectAPI = obj
globalObjLayerMutex.Unlock()
must := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
source := globalIAMSys
const user = "peer-reload-user"
req := madmin.AddOrUpdateUserReq{SecretKey: "original-test-password", Status: madmin.AccountEnabled}
_, err = source.CreateUser(ctx, user, req)
must(err)
_, err = source.PolicyDBSet(ctx, user, "readwrite", regUser, false)
must(err)
_, err = source.AddUsersToGroup(ctx, "peer-reload-group", []string{user})
must(err)
_, err = source.PolicyDBSet(ctx, "peer-reload-group", "readwrite", regUser, true)
must(err)
svc, _, err := source.NewServiceAccount(ctx, user, nil, newServiceAccountOpts{accessKey: "peer-reload-service", secretKey: "service-test-password"})
must(err)
signingKey, err := getTokenSigningKey()
must(err)
sts, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: user}, signingKey)
must(err)
sts.ParentUser = user
_, err = source.SetTempUser(ctx, sts.AccessKey, sts, "")
must(err)
siblingStore := &IAMStoreSys{IAMStorageAPI: newIAMObjectStore(obj, MinIOUsersSysType)}
must(siblingStore.LoadIAMCache(ctx, true))
must(siblingStore.UserNotificationHandler(ctx, sts.AccessKey, stsUser))
for _, key := range []string{user, svc.AccessKey, sts.AccessKey} {
if _, ok := siblingStore.GetUser(key); !ok {
t.Fatalf("fixture did not load %s", key)
}
}
must(source.DeleteUser(ctx, user, false))
if recreate {
req.SecretKey = "recreated-test-password"
_, err = source.CreateUser(ctx, user, req)
must(err)
if name == "recreated" {
_, err = source.PolicyDBSet(ctx, user, "readonly", regUser, false)
must(err)
}
}
sibling := &IAMSys{store: siblingStore, usersSysType: MinIOUsersSysType}
globalIAMSys = sibling
defer func() { globalIAMSys = source }()
server := &peerRESTServer{}
for range 2 {
_, remoteErr := server.DeleteUserHandler(grid.NewMSSWith(map[string]string{peerRESTUser: user}))
if remoteErr != nil {
t.Fatal(remoteErr)
}
}
if recreate {
u, ok := siblingStore.GetUser(user)
if !ok || u.Credentials.SecretKey != req.SecretKey {
t.Fatal("delayed deletion notification removed the recreated user")
}
loaded := make(map[string]UserIdentity)
must(source.store.loadUser(ctx, user, regUser, loaded))
if loaded[user].Credentials.SecretKey != req.SecretKey {
t.Fatal("notification changed the persisted recreated identity")
}
if allowed := sibling.IsAllowed(policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}); allowed != (name == "recreated") {
t.Fatal("notification did not load the recreated user's current grant")
}
}
if sibling.IsAllowed(policy.Args{AccountName: user, Action: policy.PutObjectAction, BucketName: "bucket", ObjectName: "object"}) {
t.Fatal("notification retained an old direct or group grant")
}
for _, key := range []string{svc.AccessKey, sts.AccessKey} {
if _, ok := siblingStore.GetUser(key); ok {
t.Fatalf("notification retained a revoked child: %s", key)
}
}
if !recreate {
for _, key := range []string{user} {
if _, ok := siblingStore.GetUser(key); ok {
t.Fatalf("notification retained a revoked cached identity: %s", key)
}
}
if sibling.IsAllowed(policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "bucket", ObjectName: "object"}) {
t.Fatal("notification retained the user's old grant")
}
cache := siblingStore.rlock()
member := cache.iamUserGroupMemberships[user].Contains("peer-reload-group")
siblingStore.runlock()
if member {
t.Fatal("notification retained the deleted user's group membership")
}
}
})
}
}
+131
View File
@@ -0,0 +1,131 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"fmt"
"os"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
)
// Uses APIs shared with the pre-revision tree so the same benchmark can be
// overlaid on that tree for a comparable local baseline.
func prepareIAMPerformanceFixture(b *testing.B) (context.Context, *IAMSys) {
b.Helper()
resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background())
disks, err := getRandomDisks(1)
if err != nil {
b.Fatal(err)
}
obj, _, err := initObjectLayer(ctx, mustGetPoolEndpoints(0, disks...))
if err != nil {
b.Fatal(err)
}
initAllSubsystems(ctx)
globalIAMSys.initStore(obj, nil)
if err := globalIAMSys.Load(ctx, true); err != nil {
b.Fatal(err)
}
b.Cleanup(func() { cancel(); obj.Shutdown(context.Background()); os.RemoveAll(disks[0]); resetTestGlobals() })
return ctx, globalIAMSys
}
func BenchmarkIAMCachedCredential(b *testing.B) {
for _, kind := range []string{"user", "service", "sts"} {
b.Run(kind, func(b *testing.B) {
ctx, sys := prepareIAMPerformanceFixture(b)
const parent = "benchmark-parent"
_, err := sys.CreateUser(ctx, parent, madmin.AddOrUpdateUserReq{SecretKey: "benchmark-user-password", Status: madmin.AccountEnabled})
if err != nil {
b.Fatal(err)
}
key := parent
if kind == "service" {
c, _, err := sys.NewServiceAccount(ctx, parent, nil, newServiceAccountOpts{accessKey: "benchmark-service", secretKey: "benchmark-service-password"})
if err != nil {
b.Fatal(err)
}
key = c.AccessKey
}
if kind == "sts" {
secret, err := getTokenSigningKey()
if err != nil {
b.Fatal(err)
}
c, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
if err != nil {
b.Fatal(err)
}
c.ParentUser = parent
if _, err := sys.SetTempUser(ctx, c.AccessKey, c, ""); err != nil {
b.Fatal(err)
}
key = c.AccessKey
}
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
if _, ok := sys.store.GetUser(key); !ok {
b.Fatal("credential missing")
}
}
})
}
}
func BenchmarkIAMSetTempUser(b *testing.B) {
ctx, sys := prepareIAMPerformanceFixture(b)
const parent = "benchmark-sts-parent"
_, err := sys.CreateUser(ctx, parent, madmin.AddOrUpdateUserReq{SecretKey: "benchmark-user-password", Status: madmin.AccountEnabled})
if err != nil {
b.Fatal(err)
}
secret, err := getTokenSigningKey()
if err != nil {
b.Fatal(err)
}
cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{"exp": UTCNow().Add(time.Hour).Unix(), parentClaim: parent}, secret)
if err != nil {
b.Fatal(err)
}
cred.ParentUser = parent
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
if _, err := sys.SetTempUser(ctx, cred.AccessKey, cred, "readwrite"); err != nil {
b.Fatal(err)
}
}
}
// Run with -benchtime=1x. Preparation is outside the timer; each measured load
// sees a fresh set of expired reusable service-account records.
func BenchmarkIAMColdLoadExpiredServices(b *testing.B) {
for _, count := range []int{100, 1000} {
b.Run(fmt.Sprint(count), func(b *testing.B) {
ctx, sys := prepareIAMPerformanceFixture(b)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
b.StopTimer()
for j := 0; j < count; j++ {
key := fmt.Sprintf("expired-benchmark-%d-%d", i, j)
u := UserIdentity{Version: 1, UpdatedAt: UTCNow().Add(-2 * time.Hour), Credentials: auth.Credentials{AccessKey: key, SecretKey: "expired-benchmark-password", ParentUser: "absent-idp-parent", Expiration: UTCNow().Add(-time.Hour), Status: auth.AccountOn}}
if err := sys.store.saveIAMConfig(ctx, &u, getUserIdentityPath(key, svcUser)); err != nil {
b.Fatal(err)
}
}
b.StartTimer()
if err := sys.store.LoadIAMCache(ctx, true); err != nil {
b.Fatal(err)
}
}
})
}
}
+168
View File
@@ -0,0 +1,168 @@
package cmd
import (
"context"
"errors"
"os"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/pgsty/silo-pkg/v3/policy"
)
func TestReviewIAMRevokedUserReplay(t *testing.T) {
resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
obj, disk, err := prepareFS(ctx)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(disk)
defer obj.Shutdown(ctx)
defer resetTestGlobals()
user := "review-revoked-user"
req := madmin.AddOrUpdateUserReq{SecretKey: "review-valid-password", Status: madmin.AccountEnabled}
created, err := globalIAMSys.CreateUser(ctx, user, req)
if err != nil {
t.Fatal(err)
}
policyAt, err := globalIAMSys.PolicyDBSet(ctx, user, "readwrite", regUser, false)
if err != nil {
t.Fatal(err)
}
args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "review-bucket", ObjectName: "review-object"}
if !globalIAMSys.IsAllowed(args) {
t.Fatal("seed must allow object read")
}
if err := globalIAMSys.DeleteUser(ctx, user, false); err != nil {
t.Fatal(err)
}
if err := globalIAMSys.store.LoadIAMCache(ctx, false); err != nil {
t.Fatal(err)
}
if globalIAMSys.IsAllowed(args) {
t.Fatal("deletion did not remove initial permission")
}
if err := globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, created); err != nil {
t.Fatal(err)
}
if _, err := globalIAMSys.GetUserInfo(ctx, user); !errors.Is(err, errNoSuchUser) {
t.Errorf("revoked user restored by an older replicated create, GetUserInfo error = %v", err)
}
if err := globalSiteReplicationSys.PeerPolicyMappingHandler(ctx, &madmin.SRPolicyMapping{UserOrGroup: user, UserType: int(regUser), Policy: "readwrite"}, policyAt); err != nil {
t.Fatal(err)
}
if globalIAMSys.IsAllowed(args) {
t.Error("older replicated identity and policy events restored revoked S3 read permission")
}
}
func TestReviewIAMSourceTimestampOrder(t *testing.T) {
resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
obj, disk, err := prepareFS(ctx)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(disk)
defer obj.Shutdown(ctx)
defer resetTestGlobals()
user := "review-ordered-user"
req := madmin.AddOrUpdateUserReq{SecretKey: "review-valid-password", Status: madmin.AccountEnabled}
origin := UTCNow().Add(-time.Hour)
if err := globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin); err != nil {
t.Fatal(err)
}
if err := globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(time.Minute)); err != nil {
t.Fatal(err)
}
if _, err := globalIAMSys.GetUserInfo(ctx, user); !errors.Is(err, errNoSuchUser) {
t.Fatalf("newer source deletion skipped after delayed creation, GetUserInfo error = %v", err)
}
}
// A user's old group grant must not return after deletion and deliberate recreation.
func TestR3CandidateOldGroupReplayAfterRecreation(t *testing.T) {
resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
obj, disk, err := prepareFS(ctx)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(disk)
defer obj.Shutdown(ctx)
defer resetTestGlobals()
must := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
user, group := "r3-group-member", "r3-granting-group"
origin := UTCNow().Add(-time.Hour)
req := madmin.AddOrUpdateUserReq{SecretKey: "valid-r3-user-password", Status: madmin.AccountEnabled}
peer := &globalSiteReplicationSys
must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin))
add := &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}
must(peer.PeerGroupInfoChangeHandler(ctx, add, origin.Add(time.Minute)))
must(peer.PeerPolicyMappingHandler(ctx, &madmin.SRPolicyMapping{UserOrGroup: group, IsGroup: true, UserType: int(regUser), Policy: "readwrite"}, origin.Add(time.Minute)))
args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "r3-bucket", ObjectName: "probe"}
if !globalIAMSys.IsAllowed(args) {
t.Fatal("fixture must grant through group")
}
must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(2*time.Minute)))
must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin.Add(3*time.Minute)))
must(globalIAMSys.store.LoadIAMCache(ctx, false))
if globalIAMSys.IsAllowed(args) {
t.Fatal("recreation must start without deleted group membership")
}
must(peer.PeerGroupInfoChangeHandler(ctx, add, origin.Add(time.Minute)))
must(globalIAMSys.store.LoadIAMCache(ctx, false))
if globalIAMSys.IsAllowed(args) {
t.Fatal("old group event restored the deleted user's read grant after recreation and durable reload")
}
}
// The user delete is also a revocation of its earlier group memberships.
func TestR3CandidateLateDeleteRetainsOldGroupGrant(t *testing.T) {
resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
obj, disk, err := prepareFS(ctx)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(disk)
defer obj.Shutdown(ctx)
defer resetTestGlobals()
must := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
user, group := "r3-late-member", "r3-late-group"
origin := UTCNow().Add(-time.Hour)
req := madmin.AddOrUpdateUserReq{SecretKey: "valid-r3-user-password", Status: madmin.AccountEnabled}
peer := &globalSiteReplicationSys
must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin))
must(peer.PeerGroupInfoChangeHandler(ctx, &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: group, Members: []string{user}}}, origin.Add(time.Minute)))
must(peer.PeerPolicyMappingHandler(ctx, &madmin.SRPolicyMapping{UserOrGroup: group, IsGroup: true, UserType: int(regUser), Policy: "readwrite"}, origin.Add(time.Minute)))
args := policy.Args{AccountName: user, Action: policy.GetObjectAction, BucketName: "r3-bucket", ObjectName: "probe"}
if !globalIAMSys.IsAllowed(args) {
t.Fatal("fixture must grant through group")
}
must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, UserReq: &req}, origin.Add(3*time.Minute)))
must(peer.PeerIAMUserChangeHandler(ctx, &madmin.SRIAMUser{AccessKey: user, IsDeleteReq: true}, origin.Add(2*time.Minute)))
must(globalIAMSys.store.LoadIAMCache(ctx, false))
if _, ok := globalIAMSys.GetUser(ctx, user); !ok {
t.Fatal("newer identity must survive")
}
if globalIAMSys.IsAllowed(args) {
t.Fatal("late user deletion retained the older group grant on the recreated identity")
}
}
+321
View File
@@ -0,0 +1,321 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"strings"
"sync/atomic"
"time"
"github.com/minio/madmin-go/v3"
xhttp "github.com/minio/minio/internal/http"
"github.com/pgsty/silo-pkg/v3/policy"
)
const (
iamRevisionProtocol = 1
iamRevisionPeerPath = "/v3/site-replication/peer/iam-revisions"
iamUserBoundaryType = "silo-user-revocation"
iamGroupBoundaryType = "silo-group-revocation"
maxIAMRevisionBatch = 128
)
var iamRevisionInstance = mustGetUUID()
type iamUserBoundary struct {
User string `json:"user"`
Before time.Time `json:"before"`
}
type iamGroupBoundary struct {
Group string `json:"group"`
Before time.Time `json:"before"`
}
// The server owns this additive protocol, without changing the client SDK or
// overloading a policy/document field. Old servers reject the dedicated route
// before applying any change that would lose revocation or member metadata.
type iamReplicationItem struct {
madmin.SRIAMItem
GroupGrants map[string]time.Time `json:"groupGrants,omitempty"`
GroupSnapshot bool `json:"groupSnapshot,omitempty"`
UserRevocation *iamUserBoundary `json:"userRevocation,omitempty"`
GroupRevocation *iamGroupBoundary `json:"groupRevocation,omitempty"`
RevokedBefore time.Time `json:"revokedBefore,omitempty"`
}
type iamRevisionBatch struct {
Version int `json:"version"`
Items []iamReplicationItem `json:"items"`
}
type iamRevisionStatus struct {
Version int `json:"version"`
Node string `json:"node"`
Instance string `json:"instance"`
Digest string `json:"digest"`
}
type iamRevisionResponse struct {
iamRevisionStatus
Errors []string `json:"errors,omitempty"`
}
type iamRevisionBatchError struct{ failures []string }
func (e *iamRevisionBatchError) Error() string {
return "IAM revision batch: " + strings.Join(e.failures, "; ")
}
type iamRevisionProgress struct {
Instances map[string]string
Acknowledged map[string]string
}
type iamRevisionMetrics struct {
healFailures atomic.Uint64
healLastSuccess atomic.Int64
healDurationMillis atomic.Int64
}
func iamRevisionDigest(items map[string]iamRevision) string {
paths := make([]string, 0, len(items))
for path := range items {
paths = append(paths, path)
}
sort.Strings(paths)
h := sha256.New()
for _, path := range paths {
r := items[path]
fmt.Fprintf(h, "%q %s %t %s\n", path, r.timestamp().UTC().Format(time.RFC3339Nano), r.Deleted, r.RevokedBefore.UTC().Format(time.RFC3339Nano))
}
return hex.EncodeToString(h.Sum(nil))
}
func (store *IAMStoreSys) iamRevisionStatus() iamRevisionStatus {
node := globalLocalNodeName
if node == "" {
node = "local"
}
return iamRevisionStatus{Version: iamRevisionProtocol, Node: node, Instance: iamRevisionInstance, Digest: store.revisionIndex().digest()}
}
func executeIAMRevisionRequest(ctx context.Context, client *madmin.AdminClient, method string, batch *iamRevisionBatch) (status iamRevisionStatus, err error) {
var content []byte
if batch != nil {
content, err = json.Marshal(batch)
if err != nil {
return status, err
}
}
resp, err := client.ExecuteMethod(ctx, method, madmin.RequestData{RelPath: iamRevisionPeerPath, QueryValues: url.Values{"api-version": {madmin.SiteReplAPIVersion}}, Content: content})
if resp != nil {
defer xhttp.DrainBody(resp.Body)
}
if err != nil {
return status, err
}
if resp.StatusCode != http.StatusOK {
var remote madmin.ErrorResponse
if json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&remote) == nil && remote.Code != "" {
return status, remote
}
return status, fmt.Errorf("IAM revision protocol requires upgraded peers: %s", resp.Status)
}
var response iamRevisionResponse
if err = json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&response); err != nil {
return status, err
}
status = response.iamRevisionStatus
if status.Version != iamRevisionProtocol || status.Node == "" || status.Instance == "" || status.Digest == "" {
return status, errors.New("peer did not acknowledge the IAM revision protocol")
}
if len(response.Errors) != 0 {
return status, &iamRevisionBatchError{failures: response.Errors}
}
return status, nil
}
type (
iamRecordBoundaryKey struct{}
iamGroupSnapshotKey struct{}
)
func (c *SiteReplicationSys) replicationItem(ctx context.Context, item madmin.SRIAMItem) (iamReplicationItem, error) {
out := iamReplicationItem{SRIAMItem: item}
if item.Type == madmin.SRIAMItemSvcAcc && item.SvcAccChange != nil {
var key string
if item.SvcAccChange.Create != nil {
key = item.SvcAccChange.Create.AccessKey
} else if item.SvcAccChange.Update != nil {
key = item.SvcAccChange.Update.AccessKey
}
if key != "" {
r, err := loadIAMRevision(ctx, globalIAMSys.store, getUserIdentityPath(key, svcUser))
if err != nil {
return out, err
}
if r.Deleted || r.timestamp().After(item.UpdatedAt) {
return out, errIAMStaleUpdate
}
out.RevokedBefore = r.RevokedBefore
}
}
if item.Type == madmin.SRIAMItemGroupInfo && item.GroupInfo != nil && !item.GroupInfo.UpdateReq.IsRemove {
out.GroupSnapshot = true
var gi GroupInfo
if err := globalIAMSys.store.loadIAMConfig(ctx, &gi, getGroupInfoPath(item.GroupInfo.UpdateReq.Group)); err != nil {
return out, err
}
// The matching persisted snapshot carries member grant times. If a
// later write won before sending, propagate that whole newer state.
if gi.Deleted {
return out, errIAMStaleUpdate
}
out.UpdatedAt = gi.UpdatedAt
out.RevokedBefore = gi.RevokedBefore
out.GroupInfo = &madmin.SRGroupInfo{UpdateReq: madmin.GroupAddRemove{Group: item.GroupInfo.UpdateReq.Group, Status: madmin.GroupStatus(gi.Status)}}
cache := globalIAMSys.store.rlock()
out.GroupInfo.UpdateReq.Members = cache.effectiveGroupMembers(gi)
globalIAMSys.store.runlock()
out.GroupGrants = make(map[string]time.Time, len(out.GroupInfo.UpdateReq.Members))
for _, member := range out.GroupInfo.UpdateReq.Members {
out.GroupGrants[member] = gi.MemberGrants[member]
}
}
if item.Type == madmin.SRIAMItemGroupInfo && item.GroupInfo != nil && item.GroupInfo.UpdateReq.IsRemove && len(item.GroupInfo.UpdateReq.Members) == 0 {
r, err := loadIAMRevision(ctx, globalIAMSys.store, getGroupInfoPath(item.GroupInfo.UpdateReq.Group))
if err != nil {
return out, err
}
if !r.Deleted && !r.RevokedBefore.IsZero() {
out.Type, out.GroupInfo = iamGroupBoundaryType, nil
out.GroupRevocation = &iamGroupBoundary{Group: item.GroupInfo.UpdateReq.Group, Before: r.RevokedBefore}
out.UpdatedAt = r.RevokedBefore
}
}
if item.Type == madmin.SRIAMItemIAMUser && item.IAMUser != nil {
r, err := loadIAMRevision(ctx, globalIAMSys.store, getUserIdentityPath(item.IAMUser.AccessKey, regUser))
if err != nil {
return out, err
}
if item.IAMUser.IsDeleteReq && !r.Deleted && !r.RevokedBefore.IsZero() {
out.Type = iamUserBoundaryType
out.IAMUser = nil
out.UserRevocation = &iamUserBoundary{User: item.IAMUser.AccessKey, Before: r.RevokedBefore}
out.UpdatedAt = r.RevokedBefore
} else if !item.IAMUser.IsDeleteReq {
if r.Deleted || r.timestamp().After(item.UpdatedAt) {
return out, errIAMStaleUpdate
}
out.RevokedBefore = r.RevokedBefore
}
}
return out, nil
}
func applyIAMReplicationItem(ctx context.Context, item iamReplicationItem) error {
if item.GroupInfo != nil {
if item.GroupSnapshot {
ctx = context.WithValue(ctx, iamGroupSnapshotKey{}, true)
}
// A nil map also explicitly denotes unknown legacy grants. Do not
// turn an unrelated group edit into a new grant after a revocation.
ctx = withIAMGroupGrants(ctx, item.GroupGrants)
}
if !item.RevokedBefore.IsZero() {
if item.RevokedBefore.After(item.UpdatedAt) {
return errSRInvalidRequest(errInvalidArgument)
}
ctx = context.WithValue(ctx, iamRecordBoundaryKey{}, item.RevokedBefore)
}
switch item.Type {
case iamUserBoundaryType:
if item.UserRevocation == nil || item.UserRevocation.User == "" || item.UserRevocation.Before.IsZero() {
return errSRInvalidRequest(errInvalidArgument)
}
return iamReplicationError(globalIAMSys.DeleteUser(withIAMReplicationTime(ctx, item.UserRevocation.Before), item.UserRevocation.User, true))
case iamGroupBoundaryType:
if item.GroupRevocation == nil || item.GroupRevocation.Group == "" || item.GroupRevocation.Before.IsZero() {
return errSRInvalidRequest(errInvalidArgument)
}
_, err := globalIAMSys.RemoveUsersFromGroup(withIAMReplicationTime(ctx, item.GroupRevocation.Before), item.GroupRevocation.Group, nil)
return iamReplicationError(err)
case madmin.SRIAMItemPolicy:
if len(item.Policy) == 0 {
return globalSiteReplicationSys.PeerAddPolicyHandler(ctx, item.Name, nil, item.UpdatedAt)
}
p, err := policy.ParseConfig(bytes.NewReader(item.Policy))
if err != nil {
return err
}
if p.IsEmpty() {
p = nil
}
return globalSiteReplicationSys.PeerAddPolicyHandler(ctx, item.Name, p, item.UpdatedAt)
case madmin.SRIAMItemSvcAcc:
return globalSiteReplicationSys.PeerSvcAccChangeHandler(ctx, item.SvcAccChange, item.UpdatedAt)
case madmin.SRIAMItemPolicyMapping:
return globalSiteReplicationSys.PeerPolicyMappingHandler(ctx, item.PolicyMapping, item.UpdatedAt)
case madmin.SRIAMItemSTSAcc:
return globalSiteReplicationSys.PeerSTSAccHandler(ctx, item.STSCredential, item.UpdatedAt)
case madmin.SRIAMItemIAMUser:
return globalSiteReplicationSys.PeerIAMUserChangeHandler(ctx, item.IAMUser, item.UpdatedAt)
case madmin.SRIAMItemGroupInfo:
return globalSiteReplicationSys.PeerGroupInfoChangeHandler(ctx, item.GroupInfo, item.UpdatedAt)
default:
return errSRInvalidRequest(errInvalidArgument)
}
}
func (a adminAPIHandlers) SRPeerIAMRevisions(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
if obj, _ := validateAdminReq(ctx, w, r, policy.SiteReplicationOperationAction); obj == nil {
return
}
var failures []string
if r.Method == http.MethodPut {
var batch iamRevisionBatch
if err := parseJSONBody(ctx, r.Body, &batch, ""); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
if batch.Version != iamRevisionProtocol || len(batch.Items) == 0 || len(batch.Items) > maxIAMRevisionBatch {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, errSRInvalidRequest(errInvalidArgument)), r.URL)
return
}
for i, item := range batch.Items {
if err := applyIAMReplicationItem(ctx, item); err != nil {
failures = append(failures, fmt.Sprintf("item %d (%s): %v", i, item.Type, err))
}
}
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: globalIAMSys.store.iamRevisionStatus(), Errors: failures})
}
// A site endpoint can balance requests across nodes sharing durable IAM state.
// Switching between known node incarnations preserves ACKs; a new incarnation
// conservatively invalidates them so restoring an old backend cannot inherit
// acknowledgements from before the restore.
func (p *iamRevisionProgress) observePeer(status iamRevisionStatus) {
if p.Instances == nil {
p.Instances = make(map[string]string)
}
if p.Instances[status.Node] != status.Instance || p.Acknowledged == nil {
p.Instances[status.Node] = status.Instance
p.Acknowledged = make(map[string]string)
}
}

Some files were not shown because too many files have changed in this diff Show More