Compare commits

...

256 Commits

Author SHA1 Message Date
Feng Ruohang 41822710e4 deps: embed the Console bucket-search navigation repair
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-25 22:13:27 +08:00
Feng Ruohang 79eca5e920 deps: select the Console candidate with native keyboard regression coverage
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-25 21:50:10 +08:00
Feng Ruohang 475640e7e1 deps: embed the repaired Console 2.5.0 candidate
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-25 21:33:34 +08:00
Feng Ruohang 62124c779c docs: include coordinated Console dependency attributions
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-25 21:06:21 +08:00
Feng Ruohang 030c6d740e deps: integrate pkg 3.15.1 and Console 2.5.0 candidate for acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-25 21:01:22 +08:00
Feng Ruohang 5f1f4178ae fix: return current bucket inventory from account info
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 6921f8c186)
2026-10-25 21:01:09 +08:00
Feng Ruohang 0828861bd0 iam: restore native policy immediately after resetting overrides
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 1e3ab56b06)
2026-10-25 21:01:09 +08:00
Feng Ruohang d16f5ed9c8 fix(ilm): keep tier data when expiry only adds a delete marker (#253)
Preserve transitioned version data when lifecycle expiration only creates a delete marker, and apply suspended-versioning semantics consistently.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 17:46:10 +08:00
Feng Ruohang 6d890fc860 Merge pull request #255: preserve FULL_OBJECT checksum type in replication
Forward FULL_OBJECT checksum type for multipart CRC replication so the target validates the whole-object checksum correctly. Restore the existing single-PUT replication path for server-side copies retaining these checksums.

Fixes #254

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 17:39:38 +08:00
Feng Ruohang aafe4f8886 fix(pools): preserve authoritative copies across failed deletes (#257)
Preserve the latest logical object when cleanup of another pool fails, report each bulk error with its original replication identity, and support retries after partial physical deletion. Reuse existing tier-aware reconciliation to avoid premature remote-object collection.

Includes the original contribution and signed-off maintainer fixes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 16:04:09 +08:00
Feng Ruohang a2073d2514 fix(pools): preserve shared tier references during failed deletes
Reuse deleteObjectReconciled for tiered bulk entries and unversioned single deletes. Its reference-aware cleanup avoids scheduling remote contents for garbage collection while another copy still needs them, including restored copies. Keep the normal local-data bulk path unchanged.

Extend existing tier GC regressions to bulk and unversioned deletes and include them in the multi-pool race CI step.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 15:47:51 +08:00
Feng Ruohang 3c82748e14 fix(pools): retain authoritative copies until bulk cleanup succeeds
Resolve each addressed version under the shared write lock, keep bulk I/O parallel within cleanup and authoritative-deletion stages, and preserve replication response identity on errors. Treat reliable absence as successful physical cleanup on retries without accepting it for marker or replication-state writes.

Cover execution-time faults, partial cleanup recovery, mixed authoritative pools, response identity, null marker ordering, metadata uncertainty, cancellation, and concurrent PUT. Run the delete consistency regressions under race in CI.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 15:36:07 +08:00
Feng Ruohang 1e94c5a30d Merge remote-tracking branch 'origin/main' into codex/pr257-delete-safety
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 15:13:21 +08:00
Feng Ruohang 000e653038 ci: use runner Python for repository card checks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 12:42:50 +08:00
Feng Ruohang 945dc16060 Update SFTP and Go dependencies with object-operation regression coverage
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-24 11:50:39 +08:00
Denis Babchuk a30270d9ba fix(pools): put a delete marker into a pool that can take it
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-10-23 22:29:49 +03:00
Denis Babchuk 1841fce4c4 fix(pools): keep the latest copy when a delete fails in one pool
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-10-23 21:28:46 +03:00
Denis Babchuk f2b8d4b9cb fix(replication): forward FULL_OBJECT checksum type for multipart replicas
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-10-23 17:52:58 +03:00
Denis Babchuk 8ed5c90dcd fix(ilm): keep tier data when expiry only adds a delete marker
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-10-23 17:36:21 +03:00
Feng Ruohang 789a2327ba docs: replace the single-node report with the maintained migration entry point
Keep the original PR #228 report at a fixed commit link with contributor credit. Direct operators to the bilingual migration guide and its recovery requirements, and record the merged contribution.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 19:27:20 +08:00
Feng Ruohang e3e0c6215d Merge remote-tracking branch 'origin/main'
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 19:16:00 +08:00
Feng Ruohang b91e1c6f2a Merge pull request #228 from yupoet/docs/single-node-upgrade-from-archived-minio
Preserve the single-node migration report contributed by @yupoet.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 19:15:50 +08:00
Feng Ruohang 969a6a42d3 build: package Windows SILO archives as ZIP
Use ZIP for Windows amd64 and arm64. Validate the platform archive matrix, executable names, documentation, SBOMs and checksum coverage in the release pipeline.

Validated with local GoReleaser snapshots, six SPDX SBOMs and packaging regression checks.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 16:13:15 +08:00
Feng Ruohang 1b5477f3df fix(iam): isolate policy reloads and propagate KMS cancellation
Release the authentication-cache lock while retaining IAM write ordering. Carry request cancellation through IAM and SSE-S3 list decryption, and publish policy replacements only after a complete successful load.

Preserve caller deadlines for slow policy queries and keep the existing etcd read budget scoped to its RPC. Cover concurrent refresh, updates, revocation, cancellation, and deadline compatibility.

Validation: 36 focused and 23 adversarial scenarios each passed three race runs; real 32-second object/etcd probes passed. Full suite passed with the Go DNS resolver after reproducing native DNS failures on the baseline; existing etcd suites, make verifiers, and make build also passed.
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 15:50:12 +08:00
Feng Ruohang a07d655dc6 Merge PR #250 while preserving local dependency updates
Retain the merged upstream LDAP fix and original contributor history. Keep the existing Go/dependency refresh and published-release documentation; combine the changelog credit without repeating the Go 1.27.3 note.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 11:22:52 +08:00
Feng Ruohang d43adc0f2b Merge pull request #250 from 910patin/fix/ldap-long-dn
Restore long-DN LDAP login while preserving IAM revision read errors.

Retain the original contribution by @910patin, add the bounded ASCII component guard and regression tests, and require the patched Go 1.27.3 toolchain. DCO, Go CI, VulnCheck and release snapshot validation passed on the final PR head.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 11:21:50 +08:00
Feng Ruohang 3df51549bb build: require Go 1.27.3 for patched certificate verification
The PR CI found GO-2026-6809 in the remote main baseline using Go 1.27.2. Require the patched toolchain already used for local acceptance; keep module versions and the separate dependency refresh unchanged.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 11:04:31 +08:00
Feng Ruohang 324bb308dd docs: credit LDAP long-DN fix contributor
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 10:53:41 +08:00
Feng Ruohang 625c11e7e1 fix(iam): preserve revision errors for storable record names
Only classify a valid IAM key as absent when a length error accompanies an ASCII path component longer than 255 bytes. Normalized LDAP DNs satisfy this condition, while total-path errors and Unicode names can refer to existing revocation records.

Cover component boundaries, wrapped and storage errors, and real revocation records moved beneath a deep Darwin drive path.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 10:35:31 +08:00
Feng Ruohang 4059c25e22 docs: reconcile published releases and unreleased dependencies
Replace stale draft status with the published 20261018 source boundary and
summarize the subsequent Go, dependency, and compatibility changes. Keep
historical details in the canonical English and Chinese release records.

Validated new links and anchors and the rebrand compatibility guard.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 02:15:22 +08:00
Feng Ruohang 011b0df5f5 build: update Go to 1.27.3 and refresh dependencies
Update the server and five debugging modules, regenerate dependency credits,
and refresh static curl and the native Red Hat CA mitigation.

Preserve Kafka's default broker protocol and terminal widths. Copy the
returned DARE plaintext when unsealing legacy keys, and cover old encrypted
keys, Kafka producer exchanges, and MinKMS TLS identity with regression tests.
Keep the published PGSTY component references for reproducible builds.

Validated with make test, targeted race tests, module integrity and tidy
checks, and govulncheck v1.9.0 source analysis. Rechecked the frozen source
and artifact hashes behind the earlier whole-stack compatibility evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-23 02:15:16 +08:00
GL 9faeb7a114 fix(iam): treat IAM record names too long for the drive as absent
Since RELEASE.2026-09-16T00-00-00Z, LDAP users whose normalized DN is
longer than 255 bytes cannot log in. NormDN hex-escapes every non-ASCII
byte (go-ldap DN.String()), so one Cyrillic letter takes six characters.
IAM looks up records named after the DN: the parent revision check added
with cmd/iam-revision.go reads config/iam/users/<DN>/identity.json, and
PolicyDBGet reads config/iam/policydb/sts-users/<DN>.json when the
mapping is not cached.

A drive rejects such an element in one of two ways. checkPathLength
refuses more than 255 counted characters, but outside Windows it never
counts a backslash, so an escaped DN usually passes it. The filesystem
then fails with ENAMETOOLONG once the parent directory exists.
toObjectErr reports both as ObjectNameInvalid, and the callers treat
only errConfigNotFound as "no record". As a result,
AssumeRoleWithLDAPIdentity, access-key creation and STS loading fail.

Such a record cannot have been stored. When the object layer's own name
validation (checkBucketAndObjectNames) accepts the path,
ObjectNameInvalid can only mean a length rejection, so report it as
errConfigNotFound. Invalid names and storage errors are returned
unchanged, and saving such a record still fails. The etcd store has no
such limit and is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: GL <g@lopatin.info>
2026-10-19 18:49:20 +03:00
Feng Ruohang 856f8f4c55 docs: refresh release and contributor acknowledgments
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-19 07:52:26 +08:00
Feng Ruohang 0cd388c666 fix(iam): persist and propagate scoped STS revocation
Enumerate durable STS identities for parent/type revocation, retain the
operation cutoff, and notify sibling caches for each committed deletion.
Repair missed notifications during periodic object/etcd IAM refresh.

Carry immutable STS tombstones and expiry through the existing IAM revision
protocol and deletion healing. Delayed creation cannot undo revocation,
while unrelated and newly issued keys stay valid. Keep partial failures
visible while propagating the successfully committed deletions.

Cover uncached credentials, warm sibling caches, dropped notifications,
object/etcd backends, reordered replay, concurrent issuance and partial
persistence failure. No new dependencies or authentication-path I/O.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 22:39:12 +08:00
Feng Ruohang 1748f62c7e fix(pools): preserve delete markers and recheck migration cleanup
Move solitary and null delete markers instead of dropping them, including
idempotent UUID marker retries. Check that a reader's source version still
exists before moving it, and preserve the established read winner for
conflicting timestamp ties.

Retire only the versions that were successfully moved while holding the
pooled object lock. Recheck the source and retained copies before cleanup,
so concurrent deletes and uncertain destination reads cannot revive data.
Count remaining markers when verifying decommission completion.

Add two/three-pool walker, history, purge, concurrent-delete, read-fault,
and timestamp-tie regressions.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 22:39:12 +08:00
Denis Babchuk 60c7179ddb fix(pools): do not let data movement overwrite a newer copy
Apply PR #248 from bca40a4ac9.

Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 22:39:07 +08:00
Feng Ruohang 7007c164a5 build: remove ripgrep from rebrand verification
Use the existing Git tool to scan source files, including untracked files and exported source trees. Preserve hidden-file exclusions and worktree ignore rules, and reject scanner diagnostics before filtering matches. Remove the CI installation step.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 09:50:18 +08:00
Feng Ruohang 51b7de6a0d ci: install ripgrep for rebrand verification
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 08:46:30 +08:00
Feng Ruohang 4733e1423c fix(multipart): defer persistent listing from public requests
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 08:36:58 +08:00
Feng Ruohang 4189869e43 Merge multipart parent cleanup fix
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 08:22:31 +08:00
Feng Ruohang 0f9199e0c3 fix(multipart): remove empty upload parents promptly
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 08:21:35 +08:00
Feng Ruohang bb60871f57 build: update silo-pkg to v3.14.3
Adopt the published policy-evaluation optimization and record the source update separately from existing component releases.

Verified with repository checks, builds and relevant race tests. No component release is created.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 07:52:13 +08:00
Feng Ruohang 2c820895d7 fix(buildscripts): fail rebrand verification when ripgrep is missing
Check the existing ripgrep prerequisite before searches whose conditional
status handling would otherwise hide command-not-found failures. Preserve
untracked-file coverage and verification from a source archive without Git.

Addresses the missing-tool finding recorded in commit
4fb3a7fa3b175727db987f3ba2523c84f74d4168 without its scan-scope changes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 07:23:05 +08:00
Feng Ruohang 2080b3d031 test(listing): cover mixed object and directory recovery groups
Extend the pending-directory collision test with absent recovery and a smaller
key. Add all six normal-object, directory, and pending-object channel orders,
checking complete-group recovery and preservation of trailing entries.

Test changes adapted from commit 557bc274a32330321fa5e5ac7636068c7bd98eb0.
Retain the complete-group directory handling from 3dd53eb81f.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 07:23:05 +08:00
Feng Ruohang 9954392605 fix(iam): normalize bulk self access-key listings
Use the existing self-service authorization helper for policyless regular
users and normalize explicit service-account and STS aliases to their parent.
Keep explicit denials and cross-user authorization intact. Assert decrypted
identity contents and negative boundaries in the existing handler tests.

Adapted from commit 5d0b692d8cd387622a3219e72430dffaaf5a25ce.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-18 07:23:05 +08:00
Feng Ruohang 3dd53eb81f fix(listing): preserve directories after absent object recovery
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 1445fc1d85d5166de24c7a378cee3479cfc95a43)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 21:25:01 +08:00
Feng Ruohang 86fb2b833f Prepare SILO 20261018 with the coordinated October dependency stack
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 19:28:33 +08:00
Feng Ruohang fbcb511978 Merge pull request #239: confirm unresolved LIST groups
Confirm eligible unresolved LIST groups without changing S3 parameters, storage formats or ordinary writer routing. Preserve original PR #221 commits and authorship.

All eight PR checks, local full/race tests and the nonpublishing snapshot pipeline pass. Current-source normal and rolling correctness pass. The strict 5% performance qualification remains inconclusive and is documented separately from the source-merge decision; no production release is authorized.

Closes #218.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 11:57:20 +08:00
Feng Ruohang 87c1c6b3dc fix(listing): confirm current groups and bound recovery lock waits
Distinguish all-drive native absence from read uncertainty and confirm current pool targets before omitting a recovered name. Cover both existing single-pool writer lock domains, release before encoding or output, and preserve cache terminal errors without blocking the HTTP error response on remote publication.

Keep ordinary entries free of recovery I/O and locks. Bound the selective short lock-conflict wait, preserve existing retry intervals and quorum, and make retries cancelable with late-grant cleanup coverage. Restore the original raw listing callback and unchanged scanner/healing/migration callers now that recovery errors originate in the merger.

Add deterministic deletion, pool ordering, cache, lock-routing and cancellation regressions to race CI. Focused cmd/dsync/lsync race suites and verifiers pass; complete current-main integration and bare-binary performance validation are recorded in PR #239.

Refs: https://github.com/pgsty/silo/issues/218
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 11:18:08 +08:00
Feng Ruohang 1e0e5ac53c Merge current main before completing listing recovery
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 11:09:12 +08:00
Feng Ruohang 7e9454456e Merge pull request #243: unify durable multipart listing
Use the existing persistent scanner for public multipart listing, tolerate confirmed unidentified historical records individually, and retire the migration mode while preserving default Abort availability.

Keep capacity and delayed-create boundaries explicit. Related to #79; this does not close its remaining lifecycle and capacity work. Companion documentation: pgsty/silo.pgsty.com#43.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 10:15:39 +08:00
Feng Ruohang 4ed86dfe62 Unify durable multipart listing without a migration mode
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-17 08:56:50 +08:00
Feng Ruohang 4e55fc7db4 Merge pull request #242 from pgsty/codex/fix-iam-admin-authorization
Enforce Admin self-service denials, session credential boundaries, root credential protection, and safe service-account issuance/cleanup.

Accepted commit 2616d43fbc after the complete IAM/STS integration selection, focused race tests, build/verifiers, and all eight hosted checks passed.

Related to #222; release and deployment remain separate.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-16 22:03:00 +08:00
Feng Ruohang 2616d43fbc fix(iam): enforce admin credential authorization boundaries
Honor self-service authorization denials, prevent session-limited credential delegation and updates, and protect root credential mutations under built-in IAM. Preserve empty-policy inheritance and damaged-account cleanup, normalize new service policy claims at issuance, and add signed Admin API and LDAP regressions.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-16 21:49:59 +08:00
Feng Ruohang 2a4db6a4f9 fix(listing): snapshot asynchronous cache cleanup inputs
Capture the metacache peer before launching cleanup and pass listing
options by value. The parent can clear its cache ID without changing the
background RPC lookup or racing with its reads of shared request state.

Publish complete immutable disk snapshots through atomic.Value in the
listing fixtures instead of replacing getDisks after requests start.
Add a real peer RPC regression for cleanup using the original cache ID;
reverting the options copy reproduces both the race and an empty RPC ID.

Verified the final 18-test listing race suite, 30 repetitions of the real
peer regression, formatting checks and make lint. Prior runtime evidence
retains its original e90cb206 source identity.

Refs: https://github.com/pgsty/silo/issues/218
Refs: https://github.com/pgsty/silo/pull/239
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-16 01:23:29 +08:00
Feng Ruohang e90cb2067d fix(listing): preserve readable keys across pool and metadata disagreements
Continue PR #221 while retaining its original commits and authorship.
Route latest-only fallback reads through the namespace lock used by normal
writes, propagate terminal scan errors before cache completion, and retain
delete markers until the cross-pool merge has resolved them.

Recount narrowly eligible latest null-version metadata when mixed erasure
layouts prevent selection. Keep each EC group separate and preserve the
requested quorum, strict resolution, and version-listing behavior.

Add regression coverage for locking, terminal errors, cache completion,
delete markers, mixed EC layouts, and HTTP behavior; run focused race tests
in CI. Verified with make test, focused race tests, and local distributed
single- and multi-pool rolling-restart workloads with exact final oracles.

Refs: https://github.com/pgsty/silo/issues/218
Refs: https://github.com/pgsty/silo/pull/221
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-16 00:58:46 +08:00
Feng Ruohang 00a375c4e2 Merge pull request #238 from pgsty/codex/checksum-error-followup
Embed the Console versioning error fix at e5b165e4c0ee and add regression coverage for the algorithm-specific checksum messages merged in #229. Go CI, vulnerability checks, DCO and release-pipeline validation passed; the actual embedded Console and S3 checksum responses were also verified locally.

Fixes #231.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 21:47:26 +08:00
Feng Ruohang ab4c97e607 fix: embed Console versioning error handling and cover checksum messages
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 21:33:02 +08:00
Feng Ruohang 721dbfd1b7 Merge pull request #229 from yupoet/fix/putobject-checksum-error-message-226
Preserve the checksum rejection status and code while naming the checksum algorithm in PutObject and UploadPart error messages.

Validated on the branch updated to main: all six Go CI jobs and VulnCheck passed. Additional local coverage checked all five algorithms, wrapped errors, trailing checksums, multipart count mismatches and the generic fallback.

The maintainer explicitly accepted this contribution without the original author's DCO trailer as a one-time exception. Original authorship and the original commit are retained; no sign-off has been added on the author's behalf.

Fixes #226

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 21:22:45 +08:00
Feng Ruohang 2926474688 Merge branch 'main' into fix/putobject-checksum-error-message-226 2026-10-15 21:10:14 +08:00
Feng Ruohang a7370aa9ac Merge pull request #236 from clown15/fix-cleanup-stale-uploads-modtime
Parse native multipart directory timestamps before the Base64 compatibility path. Cover ambiguous plaintext IDs, cleanup metadata reads, expiration, and conservative legacy fallback.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:31:35 +08:00
Feng Ruohang a592d69f45 Merge pull request #232 from julienlau/julienlau/fix/commit-metadata-missing-503
Classify missing final-commit metadata as a retryable write failure, preserving storage RPC and quorum behavior. Cover real staged-metadata loss, API responses, successful quorums, and rollback boundaries.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:30:01 +08:00
Feng Ruohang bfc7906064 fix: prefer native multipart timestamps during cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:19:37 +08:00
Feng Ruohang eeff112491 test: verify commit metadata loss and write quorum boundaries
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:18:45 +08:00
Feng Ruohang a70156e45b Merge remote-tracking branch 'origin/main' into codex/pr236-maintainer-completion
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:12:09 +08:00
Feng Ruohang 99245e3ef8 Merge remote-tracking branch 'origin/main' into codex/pr232-maintainer-completion
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:12:09 +08:00
Feng Ruohang 7a3ca0f821 build: gate releases on Object Lock regressions (#237)
Run the existing Object Lock authorization and overwrite race regressions before release state changes, builds, packages, and uploads.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 13:20:15 +08:00
Feng Ruohang d94b3b3c92 Merge pull request #233 from julienlau/julienlau/fix/stale-metadata-leaks
Fix stale metadata in legacy xl.meta headers and scanner responses.

Keep scanner response resets local and cover reused legacy headers and both scanner response variants.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 10:01:39 +08:00
Feng Ruohang 86c5a92eef fix: keep scanner response resets local and strengthen regression tests
Keep the legacy header resets and clear the unused NSScanner response
field at each construction site. Remove the generic stream Recycler call:
Recycler transfers resource ownership and cannot be treated as a reset
hook followed by another insertion into the handler's pool.

Exercise the actual NSScanner handler across repeated scans with dirty
response allocations, so the regression test does not depend on sync.Pool
reusing an object. Check both response variants after wire serialization.
Cover legacy DecodeMsg and reused indexed metadata headers as well.

Retain the original contributor's commits and DCO sign-offs.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 09:48:16 +08:00
Feng Ruohang cdc3b8cb10 Merge remote-tracking branch 'origin/main' into codex/pr-233-maintainer-fix
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 09:42:46 +08:00
Feng Ruohang 0d3ba5d4cb Merge pull request #235 from pgsty/codex/retention-preflight-hardening
Retain the verified Object Lock write-lock defenses and the coordinated bilingual repair documentation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 23:42:33 +08:00
Feng Ruohang 71a1631f13 fix: reject object lock preflight read failures
Return storage read failures before accepting an object overwrite. Keep the null-version and multipart target checks under the destination write lock, and add slow-body, read-fault, legacy-upload and recovery-collision regressions to the Object Lock race gate.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 23:28:36 +08:00
clown15 2d57b975d1 cmd: fix fast-path modtime parsing in cleanupStaleUploadsOnDisk
Upload ID directories on disk are named as '<UUID>x<UnixNano>'.
Attempting to decode them as Base64 fails unconditionally, causing
the cleanup routine to bypass the timestamp check and fall back to
disk.ReadVersion on every active upload.

Directly parse the timestamp suffix from the plaintext directory
name to restore O(1) stale upload detection, while preserving Base64
decoding and ReadVersion fallback for legacy uploads.

Signed-off-by: clown15 <thsdhrhd15@naver.com>
2026-10-14 08:55:51 +00:00
Feng Ruohang 14042b5aab Merge pull request #234 from pgsty/codex/fix-retention-write-bypasses
fix: enforce retention authorization and immutable version writes

Include Go 1.27.2 security fixes and a compatible linter.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 16:02:58 +08:00
Feng Ruohang 38a5ab549c build: use golangci-lint compatible with Go 1.27.2
Pin golangci-lint v2.14.0, whose x/tools v0.50.0 decoder supports the
version 5 export data emitted by Go 1.27.2. The previous v2.13.1 binary
failed to import internal/cpu in Linux CI before running the checks.
Keep the existing linter selection and configuration.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 15:48:51 +08:00
Feng Ruohang fa565ec9fc build: update Go to 1.27.2 and x/net to 0.60.0
Use the October security fixes for reachable standard-library and HTTP/2
vulnerabilities found by govulncheck. Align the module requirement, build
dependency check and digest-pinned Docker build stages on Go 1.27.2.
Update only golang.org/x/net and its checksums; module tidying and credit
regeneration introduce no other dependency or license changes.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 15:39:25 +08:00
Feng Ruohang 3196480da9 fix: enforce retention authorization and immutable version writes
Close SN-2026-015 by requiring PutObjectRetention with request-derived
condition keys before reading bucket or object state. Weakening unexpired
GOVERNANCE retention also requires the bypass header and permission. Treat
retention removal as zero remaining days and avoid duration saturation for
distant dates.

Close SN-2026-016 by requiring replication trust for caller-selected target
version IDs and checking persisted multipart targets and locked null
versions under the destination write lock, including across storage pools.
Preserve versioning transitions, trusted replication and batch Snowball
version IDs, and fail closed when lock state cannot be read.

Add authorization, concurrent lock-update, multipart transition, cross-pool,
configuration-failure and retention-boundary regression coverage. Validate
with make test, focused race tests and the original independent probes.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit fe3935476b929e03193d7496a7448d760e635096)
2026-10-14 15:25:19 +08:00
Julien Laurenceau cb85461c42 test: cover renameData commit-metadata-missing conversion
Drive renameData() with stub disks to assert that an in-quorum
errCommitMetaMissing is converted to errErasureWriteQuorum (HTTP 503)
and that an in-quorum errFileNotFound still surfaces unchanged so
PutObject() keeps returning IncompleteBody for client disconnects.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:24:40 +02:00
Julien Laurenceau ba6e8c8d69 fix: reset Signature when unmarshaling legacy V1 xl.meta headers
The V1 header wire format has no Signature field, so decoding into a
reused header leaked the previous object's signature. Reset it along
with EcN/EcM so a decode fully replaces the header contents.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:12:49 +02:00
Julien Laurenceau a54d0c4354 fix: reset pooled NSScanner RPC responses to avoid stale usage data
NSScanner responses stream either a dataUsageEntry update or the
final dataUsageCache. The grid framework recycles stream responses
into a pool without resetting them, and the handler set only one
field per response - so a recycled response still carried the Final
(or Update) payload of an earlier scan. A remote node reading such a
response accepted the stale final cache, corrupting bucket usage
data across scanner cycles.

Honor Recycler types when pooling stream responses, implement it for
nsScannerResp, and clear the unset field at the response
construction sites for defense in depth.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:02:02 +02:00
Julien Laurenceau 00d4a206f7 fix: zero EcM when unmarshaling legacy xl.meta V2 headers
The legacy header decode paths zeroed EcN twice (z.EcN, z.EcN = 0, 0)
instead of EcN and EcM, so EcM retained its previous value when the
same structure was reused across decodes. Decoding an xl.meta written
by an older release would then leak the erasure metadata (data/parity
block counts) of a previously decoded object into it, potentially
corrupting quorum and healing decisions.

Also zero the fields on the V1 header path, which had no reset at
all.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:02:02 +02:00
Julien Laurenceau f480086b6b fix: return 503 instead of 404 when commit metadata is missing
When the staged xl.meta disappears between the data rename and the
meta-file commit inside RenameData(), the affected drives return
errFileNotFound. An in-quorum errFileNotFound is reduced by
reduceWriteQuorumErrs() and previously surfaced as:

- IncompleteBody (400) from PutObject(), misattributing the failure
  to a client disconnect, or
- ObjectNotFound (404 NoSuchKey) from CompleteMultipartUpload(),
  telling clients the object is absent so they skip retries.

Map errFileNotFound at the commit step to a dedicated internal
errCommitMetaMissing, convert it to errErasureWriteQuorum in
renameData(), and map it to InsufficientWriteQuorum (HTTP 503) in
toObjectErr() so clients retry. Below-quorum outcomes already return
InsufficientWriteQuorum and are unchanged, as is the IncompleteBody
mapping for staging failures of client-disconnected uploads.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 16:11:06 +02:00
Paris Yu 968c30238c fix: name the checksum algorithm in PutObject/UploadPart mismatch errors
hash.ChecksumMismatch now carries the requested algorithm (CRC32, CRC32C,
SHA1, SHA256, CRC64NVME), and toAPIError renders it S3-style:

  The CRC32 you specified did not match the calculated checksum.

instead of the generic "The provided 'x-amz-checksum' header does not match
what was computed.", for all four mismatch construction sites (streaming
reader, trailing-checksum decode, and both Checksum.Matches paths). Mirrors
the wording already shipped for CompleteMultipartUpload in #225.

Fixes #226
2026-09-26 20:37:20 +08:00
Paris Yu fec1e0804a docs: add field-validated single-node upgrade guide from archived MinIO community edition
The compatibility audit does not cover the single-node/single-drive (FS mode)
case explicitly. This documents a production upgrade from the final community
release (RELEASE.2025-10-15T17-29-55Z) to Silo RELEASE.2026-09-16T00-00-00Z:
why standalone deployments are low-risk (schema retention, no multi-node REST
concern, IAM tightening not applicable), the validated procedure with layered
verification, field results (~5.9k objects, zero migration, zero hardening
rejections, full app regression passed), and the post-upgrade behavior changes
to review per client (TLS defaults, If-Match on DeleteObject, multipart listing
cap, TRUSTED_PROXIES default).
2026-09-26 19:41:43 +08:00
Feng Ruohang bcb5fc9566 Merge pull request #225 from cbornet/error-messages
Align some checksum error messages with S3

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-26 09:03:50 +08:00
cbornet 394d9db212 Align some checksum error messages with S3
Signed-off-by: cbornet <cbornet@hotmail.com>
2026-09-25 17:44:52 +02:00
mr javad seydi 8c784f6133 fix(listing): narrow the list fallback per review
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-23 19:03:00 +03:30
mr javad seydi 258b167b1c fix(listing): resolve undecidable list entries against the locked read path
Refs: https://github.com/pgsty/silo/issues/218

Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-23 18:23:15 +03:30
Feng Ruohang 69d5e92791 Merge pull request #220 from AleksaMCode/fix/lambda-targetidset-isempty
Fix `TargetIDSet.IsEmpty` and add regression test
2026-09-23 08:07:09 +08:00
AleksaMCode efd5698597 lambda/event: fix TargetIDSet.IsEmpty and add regression test
Signed-off-by: AleksaMCode <aleksamcode@gmail.com>
2026-09-17 22:06:44 +02:00
Feng Ruohang 2fde3cf535 docs: update ec description 2026-09-17 09:42:26 +08:00
Feng Ruohang 2a4d51406b test: build the purge-response fixture ARN at runtime
The rebrand compatibility guard tracks every literal policy value that
names the upstream brand. Construct the replication ARN the way the
other replication fixtures do instead of adding a new literal.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 23:50:09 +08:00
Feng Ruohang 0596685ae7 docs: record the delete-marker purge, listing quorum and migration tag repairs
Describe the three correctness repairs merged after the 20260916 candidate
was prepared, together with their known remaining limits (#217, #218).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 23:44:44 +08:00
Feng Ruohang 358ab38fb0 fix: report a purged version as a delete marker only for stored markers
The exact-version purge path copies the looked-up DeleteMarker flag
into the DELETE response so that removing a delete marker keeps its
x-amz-delete-marker header. That flag is also set for a data version
whose purge is pending, because the lookup exposes such a version as
deleted for visibility. Retrying the purge after the replication
configuration was removed therefore answered with
x-amz-delete-marker: true and raised ObjectRemovedDeleteMarkerCreated
for a data version.

Only a stored marker carries no erasure layout; use that to decide the
response identity. Found by the adversarial review of the purge change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 22:53:12 +08:00
Feng Ruohang 254b19ac07 fix(replication): recheck queued delete-marker creations under the lock
A delete-marker creation task carries the marker as it looked when it
was queued by the DELETE handler, a GET/HEAD/LIST heal, the scanner or
MRF. Tasks from several frontends serialize on the per-object
replication lock, so a task can run after the user has purged that
marker and the purge has already reached the targets. The target no
longer holds the marker, so the queued creation recreated it there with
the original VersionID and mtime. This is the late-create sequence seen
in the 2026-09-16 three-site runs: purge 204 on both targets, then about
85 ms later a replicated creation for the same VersionID.

Re-read the source version under the replication lock before sending a
creation. A missing version, a non-marker version, or a version under
purge makes the task stale; it is dropped without touching the source.
A read that cannot confirm either way is retried through MRF instead of
being treated as absence.

Creations already on the wire, replays from other sites, and cleanup of
minority residue after a crash are not covered by this check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 22:35:08 +08:00
Feng Ruohang eb4f5e5b31 Fix exact-version purges and delete-marker metadata healing
Require write-quorum absence for missing purge retries, aggregate removed and already-absent votes only for physical purges, and resolve receiver purges by version across pools. Preserve marker metadata through creation and healing while retaining DELETE response semantics.

Add real-disk quorum, pool, callback, metadata, heal and outbound replication regressions. Synchronize capacity fixture installation and restoration with concurrent IAM readers.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 22ba4d426677aa07470927fc349a43afd87e19b6)
2026-09-16 22:24:27 +08:00
Feng Ruohang 8d06424b12 fix: retain quorate null versions after newer minorities
Recount only when the existing selection is below quorum and each original
nonempty stream has one ordinary null version with identical EC headers.
Reuse the existing header grouping and leave history pruning unchanged.

Add deterministic signed LIST, complete-metadata permutation, excluded-history,
object-read, scanner/heal, Walk and migration-consumer regressions.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 9804e4deb4d5a18cca640be72bff9b47a415f7de)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 21:50:03 +08:00
Feng Ruohang fced863036 fix(storage): preserve tags during pool migration
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 20:22:37 +08:00
Feng Ruohang 5af0865aab Merge pull request #214 from pgsty/codex/release-prep-20260916
Prepare the SILO 20260916000000 source candidate with released component pins, CPU metrics synchronization, regression coverage and release tooling updates.

All 11 checks passed on e7963381ba. Signed release artifacts, image publication and deployment remain separate deliverables tracked by #203.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 19:30:07 +08:00
Feng Ruohang 83821f0f1f Merge pull request #216 from pgsty/codex/docs-published-design-links-20260916
docs: link retained guides to published design records
2026-09-16 18:50:28 +08:00
Feng Ruohang bf053386a4 docs: link compression and federation to maintained design records
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 18:26:03 +08:00
Feng Ruohang e7963381ba test: synchronize tag replication capacity fixtures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:37:47 +08:00
Feng Ruohang 37c0edc7ca test: use canonical octal permissions in LZ4 fixture
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:33:58 +08:00
Feng Ruohang a2fe70424e release: prepare SILO 20260916000000 candidate
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:30:29 +08:00
Feng Ruohang 027d43b4eb fix: synchronize CPU resource metrics reads
Hold resourceMetricsMapMu.RLock while the v3 CPU collector reads both the
subsystem map and its nested idle/iowait metrics. The unsynchronized reads
were inherited from upstream commit f7b665347 (minio/minio#19560).

Add value and concurrent Prometheus Gather regression tests, run them in
the targeted race CI job, and record the existing CPU metric names in the
compatibility inventory.

Fixes #210

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:09:19 +08:00
Feng Ruohang f99ed829b5 Merge pull request #213 from pgsty/codex/pr198-release-compat
fix: preserve multipart defaults and rollback compatibility
2026-09-16 16:42:56 +08:00
Feng Ruohang 956a1a8e33 Merge pull request #212 from pgsty/codex/docs-migration-cleanup
docs: consolidate repository entry points and retire migrated work records
2026-09-16 16:39:40 +08:00
Feng Ruohang 82f0a9828e fix: preserve multipart defaults and rollback compatibility
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:30:43 +08:00
Feng Ruohang 584b5a3a8f docs: link the canonical recovery runbooks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:19:56 +08:00
Feng Ruohang 39f49d548f Merge remote-tracking branch 'origin/main' into codex/docs-migration-cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>

# Conflicts:
#	CONTRIBUTORS.md
2026-09-16 16:13:31 +08:00
Feng Ruohang a168576adb Merge pull request #208 from pgsty/codex/conditional-put-release-notes
docs: describe conditional PUT behavior and tested recovery guidance
2026-09-16 16:09:22 +08:00
Feng Ruohang e85c4c8dfb docs: preserve reviewed contributor records and daily card references
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:09:00 +08:00
Feng Ruohang 0e3c43778e Merge pull request #211 from pgsty/codex/daily-repository-cards
ci: update README repository cards daily at 00:00 UTC
2026-09-16 15:40:09 +08:00
Feng Ruohang 9101fe78db ci: refresh README repository cards daily at 00:00 UTC
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 15:27:48 +08:00
Feng Ruohang 82948d6306 Merge pull request #198 from mrjavadseydi/fix/issue-79-list-multipart-uploads
Retain the original contributor commit and integrate the reviewed durable listing, cancellation confirmation and migration fixes. Capacity acceptance and delayed creation-write fencing remain open in issue #79.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:40:10 +08:00
Feng Ruohang dfb4b2a1d2 fix: return HTTP 503 when multipart scan capacity is exhausted
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:24:52 +08:00
Feng Ruohang 143f6970d8 fix: make multipart discovery and cancellation explicit and bounded
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:16:37 +08:00
Feng Ruohang ea5dac5a99 Merge current main into multipart listing contribution
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 12:51:20 +08:00
Feng Ruohang 3c26a8b0b5 Merge pull request #209 from pgsty/codex/credit-console-share-reporter
fix: integrate the bounded Console sharing proxy and credit its reporter
2026-09-16 12:19:29 +08:00
Feng Ruohang 2fabd436c0 fix: select the bounded Console sharing proxy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 12:05:00 +08:00
Feng Ruohang 07c68d6054 docs: credit Jiri Pejchal for the Console sharing report
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:48:38 +08:00
Feng Ruohang 15090dc4fd docs: retire migrated working material and clarify documentation ownership
Remove migrated investigation and security documents, repair their incoming links, and align the English/Chinese READMEs with current module and release boundaries. Track AGENTS.md as the shared repository guide and make CLAUDE.md import it; keep working artifacts outside the repository.

Publish pgsty/silo.pgsty.com commit c7682185e2832b981629e1c17aef74fc778c6ca1 before publishing this cleanup: the new documentation routes are not live yet.

Validation: make rebrand-guard; 92 Markdown files checked for deletion-induced broken relative paths; 199 source-to-site references and anchors resolve in the paired site build; git diff --check.
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:22:48 +08:00
Feng Ruohang e791640dac docs: describe merged conditional PUT behavior and recovery guidance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:21:16 +08:00
Feng Ruohang 9b4ae82a29 Merge pull request #207 from pgsty/codex/conditional-put-pools
fix(pools): evaluate cross-pool PUT conditions against the current object
2026-09-16 11:17:02 +08:00
Feng Ruohang 4620be394b test: synchronize conditional PUT disk fixtures
Replace unsynchronized getDisks swaps with backing disk-list updates under
erasureDisksMu, matching the existing GetDisks reader lock. Apply the same
helper to capacity and read-fault adapters while preserving nested restore
ordering.

Add a regression that overlaps fixture changes with the real IAM Walk
reader, and run the conditional PUT suite under the race detector in CI.
The regression reproduces the old fixture race; ten fixed race iterations
pass without warnings. Production conditional PUT behavior is unchanged.

Refs #199

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 10:53:24 +08:00
Feng Ruohang 5e7d603083 fix(pools): evaluate cross-pool PUT conditions against the current object
Multi-pool PUT selected a destination by capacity and evaluated
If-Match/If-None-Match only against that destination's local object
state. An empty or stale destination could accept a stale ETag or
If-None-Match:* while another pool held the current object, replacing
it; a current ETag could instead be rejected with 412 or 404.

Under PUT's existing pools-layer object lock, resolve the comparison
object with objectPoolInfos (including draining pools), treat a latest
delete marker as absence, fail closed on unreadable pool metadata, and
clear an accepted callback before destination dispatch. Replica and
data-movement callbacks keep their addressed-version semantics and
metadata reconciliation.

Reproduced on 40220bd836 and RELEASE.2026-09-03T13-18-01Z with six
signed HTTP scenarios: four defect cases failed, two controls passed.

Refs #199

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:21:21 +08:00
Feng Ruohang fb7c406ddc Merge pull request #206 from pgsty/codex/main-consolidation-20260916
test: restore valid credentials in integration fixtures
2026-09-16 08:18:56 +08:00
Feng Ruohang 416826f61f Merge pull request #205 from pgsty/codex/release-notes-followups
docs: complete September correctness and upgrade notes
2026-09-16 08:16:23 +08:00
Feng Ruohang a2e2f3ee82 test: restore valid credentials in integration fixtures
Port the shell-fixture changes from ebc9937d97b27871dcc4bb91d4b5771d3550b76a. Match the existing minimum secret length consistently across startup, aliases and helper commands.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:08:08 +08:00
Feng Ruohang 70c7ec4a9f docs: link reviewed runbook sources before website deployment
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:04:53 +08:00
Feng Ruohang 2b722b7e92 docs: complete September correctness and upgrade notes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 07:51:11 +08:00
mr javad seydi 4cbb074ccd fix: make multipart upload listing S3-compatible
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-15 23:17:55 +03:30
Feng Ruohang 40220bd836 Merge pull request #196 from pgsty/codex/merge-r4-r8
Complete R5, R6 and R8 on the main baseline containing R4 and R7.

Preserve the individual signed repairs, independent Opus 5 Max review, and integration validation evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 01:14:06 +08:00
Feng Ruohang df0dfa0a34 docs: record R4-R8 integration review and validation
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:58:31 +08:00
Feng Ruohang 80684fed59 chore: align integrated repair tests with contribution checks
Use the actual author and AGPL-3.0-or-later notices for new R6/R8 tests, preserving all test bodies and the Linux build tag. Record the existing replication ARN prefix used by the new R6 fixture in the compatibility inventory; no wire behavior changes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:44:34 +08:00
Feng Ruohang 055030ea53 fix(http): honor configured request header deadlines
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 0d48d32d7e038ae1ea5966f3d7e0cb86780a6311)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:56 +08:00
Feng Ruohang aea3882c95 docs: record R6 integration verification
(cherry picked from commit d38edb2c46182d3a8fa96e040604493d20a4b478)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:56 +08:00
Feng Ruohang 0c61128d23 fix(replication): retry marker purges through persisted MRF
(cherry picked from commit cf381a7151ef25fc95ace5fedcd767fa19410de2)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:55 +08:00
Feng Ruohang 680eac66e4 fix(replication): preserve ordered tag deletions
Persist, send and reconcile empty tag states together with their revision
across COPY, PUT and multipart replication. Advance local tag mutations
under the existing locks and preserve current tags during replication ACK.

Cover signed HTTP, persistent single/multiple pool state, KMS, SSE-C key
rotation, ordering, retry and duplicate requests. Record real Opus plan
consensus, implementation review and local verification evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 115fe8b12329d147adbaf817faa1737392ecbf9b)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:55 +08:00
Feng Ruohang 9f3037e941 Merge pull request #194 from pgsty/codex/r7-replication-content-encoding
fix(replication): preserve normalized replica metadata
2026-09-16 00:31:03 +08:00
Feng Ruohang 5f00f6762f Merge main after R4 validation into R7 candidate
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:21:21 +08:00
Feng Ruohang af2b1794d3 Merge pull request #193 from pgsty/codex/r4-kms-tag-timestamp
fix(replication): preserve SSE-KMS tag timestamps
2026-09-16 00:20:10 +08:00
Feng Ruohang d371f77dcc docs: record final Opus 5 Max R7 implementation review
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:10:31 +08:00
Feng Ruohang 022722a7a7 chore: align R4 contribution notices and record final review
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:08:09 +08:00
Feng Ruohang 03027727d1 fix(replication): preserve SSE-KMS tag timestamps
Carry the already-parsed source tagging timestamp through the KMS options
constructor so replica COPY can apply newer tag updates on explicitly or
automatically encrypted destinations.

Cover all option encryption modes and signed COPY persistence for newer,
stale, duplicate and timestamp-less updates, including bucket defaults.
Preserve the real Opus 5.0/max plan review, consensus and local validation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:03:52 +08:00
Feng Ruohang 4fcdf37ce6 fix(replication): preserve normalized replica metadata
Restore only the six replication-specific metadata fields after trust
validation, so streaming uploads retain their actual content encoding and
Snowball entries do not inherit ordinary metadata from the outer archive.

Include helper, authenticated PUT/COPY/multipart and Snowball regressions,
plus the R7 investigation, actual Opus 5 consensus and local verification.
The production change is based on PR #187 by Mikhail Khadarenka.

Co-authored-by: Mikhail Khadarenka <chodorenko@gmail.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:00:17 +08:00
Feng Ruohang 9ebe81c1b3 Merge pull request #192 from pgsty/codex/iam-revision-tombstones
fix(iam): retain revocation versions through replay and recovery
2026-09-15 23:14:22 +08:00
Feng Ruohang e5f5c9e7f6 Merge pull request #191 from pgsty/codex/iam-peer-delete-reload
fix(iam): reload committed state on peer deletion notifications
2026-09-15 23:10:38 +08:00
Feng Ruohang 7b4cacc392 chore: align IAM file notices with contribution policy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:59:04 +08:00
Feng Ruohang a0dd7dae9b fix(iam): resume site healing after leadership changes
Keep one healing loop per process across replication configuration reloads. Reacquire leadership after a lease is canceled and allow shutdown while waiting, so temporary quorum loss cannot permanently stop revocation propagation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:59:04 +08:00
Feng Ruohang 709d50a916 fix(iam): persist revocations across site replay and recovery
Retain source-ordered tombstones and parent grant boundaries across both IAM backends, cache reloads, and deliberate identity recreation. Reconcile deletions through a versioned, bounded replication protocol with restart-aware acknowledgements.

Cover inherited group grants, STS retention, same-key service recreation, absolute expiration, and failures after the durable commit. Document coordinated upgrades and the remaining consistency boundaries.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:59:04 +08:00
Feng Ruohang dff81f293b chore: align IAM test notice with contribution policy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:58:52 +08:00
Feng Ruohang f653a6ea03 fix(iam): reload committed state on peer deletion notifications
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 22:58:52 +08:00
Feng Ruohang 47d239f84f Merge pull request #190 from pgsty/codex/fix-pool-multipart-preconditions
fix(storage): evaluate multipart preconditions across pools
2026-09-15 21:57:26 +08:00
Feng Ruohang e069fe9d92 fix(storage): evaluate multipart preconditions across pools
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 21:13:37 +08:00
Feng Ruohang d848fb52b5 Merge pull request #189 from pgsty/codex/fix-pool-tag-reconciliation
fix(storage): preserve tags during pool reconciliation
2026-09-15 19:38:19 +08:00
Feng Ruohang 3ce8319251 fix(storage): preserve tags during pool reconciliation
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 19:26:58 +08:00
Feng Ruohang 9df0f4abaf Merge pull request #188 from pgsty/codex/remove-access-tiering-final
Remove access-frequency pool tiering and preserve independent multi-pool correctness fixes. Reconcile ordinary addressed-version DELETE across pools, retaining existing quorum and compatibility boundaries.

Verified delivery head: 4d0693cb8c. All 11 final CI checks and three qualified Linux upgrade runs passed. Introduction, retirement decisions and historical unresolved observations are documented.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 15:09:48 +08:00
Feng Ruohang 4d0693cb8c docs: record controlled retirement evidence and qualified upgrade runs
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 14:56:20 +08:00
Feng Ruohang bf59e3f222 docs: record retirement execution and blocked Linux acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 12:16:45 +08:00
Feng Ruohang 41aa846097 style(storage): satisfy callback selection lint rule
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:31:32 +08:00
Feng Ruohang 4093fa0d78 docs: record access tiering introduction and retirement decisions
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:02 +08:00
Feng Ruohang 13bf126ebd fix(storage): reuse resolved copies for version DELETE callbacks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:02 +08:00
Feng Ruohang 1cf529ce8a fix(storage): reconcile ordinary version DELETE across pools
Delete every copy of an explicitly addressed UUID, null version or delete
marker under the pool lock. Preserve retention and replication callbacks,
report unreadable pools and cleanup failures, and keep movement, incoming
replication, expiration and free-version cleanup on their existing paths.

Retain the separately developed general DELETE repair and replace its
access-mover-only coverage with a real interrupted rebalance copy followed
by HTTP deletion. Cover unqualified directory-marker DELETE and document
the existing pool-order-dependent 503 behavior that this makes consistent.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:01 +08:00
Feng Ruohang 9b76a21675 revert: remove access-frequency ILM tiering (#60)
Reverse the first-parent diff of a3df317ae0,
including the feature branch compatibility and mover follow-up fixes.
Retain the independent multi-pool correctness fixes from #178 and migrate
their shared test fixture away from access-tier code.

Tolerate retired ILM keys and XML, read old v9 statistics while writing v8,
and document migration without moving objects or rewriting their metadata.
Include regression coverage using a historical scanner/writer v9 fixture.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-15 11:27:01 +08:00
Feng Ruohang 89637554d6 Merge pull request #182 from pgsty/codex/docs-current-state-20260913
docs: align release notes and current component status
2026-09-13 10:50:22 +08:00
Feng Ruohang 2dd1e00da4 docs: align release notes and current component status
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 10:39:57 +08:00
Feng Ruohang 5d955b5b74 Merge pull request #181 from pgsty/codex/deps-release-20260913
Coordinate September dependency releases and replace vulnerable bundled curl
2026-09-13 10:14:57 +08:00
Feng Ruohang f7808a172c deps: pin the coordinated September 13 SILO stack
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:54:51 +08:00
Feng Ruohang acc9b514f5 Follow the curl builder rename in delivery verification
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:08:38 +08:00
Feng Ruohang 31ba01c5d5 Refresh Go dependencies and build current static curl for both architectures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-13 09:01:09 +08:00
Feng Ruohang 48ec10312f Merge pull request #180 from pgsty/codex/issue-77-metadata-convergence
fix(replication): preserve bucket metadata source times and converge deletions
2026-09-12 18:07:36 +08:00
Feng Ruohang 114dc10529 docs: archive issue 77 design, adversarial reviews and acceptance evidence
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:52:25 +08:00
Feng Ruohang 461e9a7210 test(replication): satisfy diagnostic regression style checks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:17:18 +08:00
Feng Ruohang fcbb93e895 fix(replication): diagnose unusable metadata without a heal source
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 17:09:34 +08:00
Feng Ruohang 62cf066ff5 fix(replication): recover physical creation time and align policy status
An independent adversarial review of the bucket metadata convergence
work found three defects it had introduced.

GetBucketInfo overwrote the physical creation probe with cached
metadata, which a bucket that never held a configuration legitimately
lacks. The new creation-time requirement then failed every policy, tag,
SSE, quota, versioning and Object Lock write on such a bucket, with no
operator recovery path, and initial synchronization skipped it silently.
Return the physical result unchanged when metadata is not requested, as
ListBuckets already does, recover the time during initial
synchronization, and pass it to MakeBucketHook so peers adopt the same
bucket generation.

Replication status compared parsed policies statement by statement while
heal compares the canonical key. An upgraded peer that stored an
equivalent statement order was therefore reported as mismatched forever,
and heal never had anything to write. Compare the key heal compares;
per-site presence counting is unchanged.

Heal diagnostics shared one log key across four conditions, so a real
peer RPC failure could be deduplicated away by an earlier message, and
they were logged at error level for the normal transient of a peer that
does not have the bucket yet. Give each reason its own key at warning
level, report only a field state that exists and still cannot be
ordered, and diagnose nothing when no site holds a state to propagate.

The recovery test now runs against the real ObjectLayer; the stub it
replaced returned the expected time and hid the defect. The policy
status test uses a statement order the canonical encoder reorders, and
adoption coverage is extended past a real field time.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 14:01:35 +08:00
Feng Ruohang 1ee64a8d89 fix(replication): gate metadata tombstone export during rolling upgrades
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang 01aaef2b50 fix(replication): converge bucket metadata using deterministic source states
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang bcc62afe3d fix(replication): apply bucket metadata source times under the metadata lock
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 13:50:55 +08:00
Feng Ruohang 5c57658163 Merge pull request #179 from pgsty/codex/federation-copy-fixes
fix(federation): preserve committed copy times and reject raw SSE-C replicas
2026-09-12 02:00:57 +08:00
Feng Ruohang f175e98c34 fix(federation): bind copy timestamps to committed writes
Return the committed object or part time on federation write responses and
capture it for CopyObject and UploadPartCopy without a follow-up read.
Keep successful writes compatible with targets that do not supply a time.

Reject authenticated raw SSE-C replica CopyObject across deployments before
forwarding. Extend the existing SSE fixtures to verify empty objects,
stored checksums, KMS contexts and multipart sources.

Refs: #169, #168, #171
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-12 01:44:22 +08:00
Feng Ruohang 12f631b502 Merge pull request #178 from pgsty/codex/multipool-correctness-20260911
fix(storage): reconcile multi-pool writes and conditional deletes
2026-09-11 20:34:58 +08:00
Feng Ruohang ccb676e60c fix(storage): preserve shared tier references during pool cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:23:40 +08:00
Feng Ruohang 51d41345f7 test: record Linux restart and OIDC release acceptance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:05:56 +08:00
Feng Ruohang e59a3d938e fix(storage): serialize and reconcile multi-pool object updates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 20:05:56 +08:00
Feng Ruohang b32f2d9dd0 Merge pull request #177 from pgsty/codex/release-consolidation-20260911
Fix signed payloads, Object Lock and federated copies; secure AMQP
2026-09-11 17:08:34 +08:00
Feng Ruohang d63c92e393 build(deps): secure AMQP frames and select merged Console
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:56:33 +08:00
Feng Ruohang 68127c5a63 build(deps): embed the consolidated Console source
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:40:42 +08:00
Feng Ruohang c4b5e1cb45 fix(auth): enforce header-only presigned payload checksums
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang 9a303f5096 fix(object): preserve retention and federated copy destination state
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang 87d8b5967f fix(auth): align signed request and policy condition semantics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:24:42 +08:00
Feng Ruohang f760046c44 Merge remote-tracking branch 'origin/main' into codex/release-consolidation-20260911
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-11 16:13:42 +08:00
Feng Ruohang 93e7ef4bcc Merge pull request #175 from pgsty/codex/upstream-sdk-password-20260910
fix(iam)!: split self-service password permissions and update SDK stack
2026-09-10 17:51:08 +08:00
Feng Ruohang a4229b366f build(deps): align final coordinated SILO source pins
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 17:39:01 +08:00
Feng Ruohang 420340bc14 docs(iam): explain breaking password-policy semantics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 16:58:21 +08:00
Ayush Sharma e7e87402ed fix: forward the legal hold as an Object Lock header on federated CopyObject
A cross-deployment CopyObject that requests
`x-amz-object-lock-legal-hold: ON` answered 200 while the destination
carried no hold. The resolved value reached the remote as ordinary user
metadata, `X-Amz-Meta-X-Amz-Object-Lock-Legal-Hold`, so nothing applied it.
Retention requested on the same copy survived, which is what made the loss
easy to miss.

The federation branch passes the resolved metadata map straight to
`Core.PutObject` as `PutObjectOptions.UserMetadata`. minio-go's `Header()`
writes the typed lock fields first, then prefixes every UserMetadata key it
does not recognise with `x-amz-meta-`; `supportedHeaders` covers
`x-amz-object-lock-mode` and `x-amz-object-lock-retain-until-date` but not
`x-amz-object-lock-legal-hold`, and `isAmzHeader` does not match it either.
Retention therefore arrives as real headers and the hold does not. The
high-level `validate()` that would have rejected the key never runs, because
`Core.PutObject` goes straight to the low-level PUT.

Carry the hold on the typed `LegalHold` option and forward a cloned map with
the raw key removed. The clone matters twice: typed fields are written before
the UserMetadata loop, so a leftover raw key would add a bogus `x-amz-meta-`
entry beside the correct header, and the proxy's own response and event
metadata are rebuilt from the resolved values rather than the forwarding map,
which no longer carries the hold.

Retention stays in the map deliberately. It already passes through as a
standard header, and moving it to the typed `RetainUntilDate` field would
format with `time.RFC3339` and truncate a retain-until date to whole seconds.

The new test asserts the wire: the remote must receive
`X-Amz-Object-Lock-Legal-Hold` and never the `x-amz-meta-` spelling, and the
destination version must actually store the hold. It fails without the change
with "legal hold forwarded as user metadata [ON]".

Fixes #166

Signed-off-by: Ayush Sharma <72848455+Aeirx@users.noreply.github.com>
2026-09-10 13:15:10 +05:30
Feng Ruohang a164e1dda1 fix(iam): separate password changes and refresh coordinated SDK dependencies
Use ChangeMyPassword for the authenticated user and keep CreateUser for other users. Coordinate silo-pkg b3760f56ec23, mcli fa22b40b4eb7, Console 1b95b6cec652 and upstream minio-go 78bfa91607c2. Add legacy-policy and SDK streaming regressions plus upgrade guidance.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 15:16:21 +08:00
Feng Ruohang 2f61325d4a Merge pull request #174 from pgsty/codex/contributor-orenyomtov-20260910
docs: credit @orenyomtov for the SN-2026-011 report
2026-09-10 11:34:52 +08:00
Feng Ruohang a6145e1d2e docs: credit @orenyomtov for the SN-2026-011 report
Add Oren Yomtov (github.com/orenyomtov) to the contributor wall in
README.md, README_ZH.md, and CONTRIBUTORS.md, and to the Issue reports
table, for the private disclosure of the unsigned-header CopyObject
cross-object read fixed as SN-2026-011 (#173). Community contributor
count 40 -> 41.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 11:24:12 +08:00
Feng Ruohang 5232546690 Merge pull request #173 from pgsty/codex/unsigned-amz-header-copy-20260909
fix(auth): reject unsigned x-amz-* headers to close CopyObject confused-deputy (SN-2026-011)
2026-09-10 10:10:49 +08:00
Feng Ruohang 0c46cb641b docs(security): record SN-2026-011 (unsigned x-amz-* header CopyObject)
Ledger entry for the confused-deputy fix in 123325430: an unsigned
x-amz-copy-source header turned a presigned or signed PUT into a
server-side copy of any object the signing key can read. Reported by
Oren Yomtov; inherited from upstream minio/minio; CVE requested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 09:57:26 +08:00
Feng Ruohang 1233254309 fix(auth): reject unsigned x-amz-* headers to close CopyObject confused-deputy
A presigned or signed PUT authorized for a single object could be turned
into a server-side copy of any object the signing key can read by adding
an unsigned x-amz-copy-source header, executed as the signer. SigV4
verification only walked the signed-headers list, never the headers that
actually arrived; the meta-header check matched only X-Amz-Meta- and ran
only on the presigned path, so an unsigned x-amz-* header outside the
list was never seen while the router still dispatched the PUT to
CopyObjectHandler.

Reject any x-amz-* request header not covered by the signed headers, on
both the presigned (doesPresignedSignatureMatch) and Authorization-header
(doesSignatureMatch) paths, matching AWS S3. The check tests membership
in the signed set rather than value equality, so a header whose first
value is empty (e.g. {"", "/src/secret"}) cannot slip through.
X-Amz-Content-Sha256 is exempt (payload hash: read from the query for
presigned requests and bound into the string-to-sign for signed ones, so
it is self-protected) and X-Amz-Signature-Age is exempt (an internal
scratch header written after verification, so repeated verification of
the same request stays idempotent). The synthesized X-Amz-Tagging header
in PutObjectTagging is now injected after signature verification.

Tests that previously added x-amz-copy-source and friends after signing
(relying on the vulnerable behavior) now re-sign, mirroring real S3
clients. Adds checkUnsignedHeaders unit cases and TestPresignedVerifyIdempotent.

Reported by Oren Yomtov. Inherited unchanged from upstream minio/minio.
Tracked as SN-2026-011.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-10 09:57:01 +08:00
Feng Ruohang 8a2fe9b7a0 Merge pull request #164 from pgsty/codex/main-consolidation-20260909
fix: honor TLS defaults and accept valid bucket metadata reloads
2026-09-09 19:49:08 +08:00
Feng Ruohang f26ee6bf0a test: cover metadata reload with equal maximum timestamps
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:35:13 +08:00
Feng Ruohang d69c4ccfe4 Merge TLS default key exchange compatibility fixes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:31:35 +08:00
Feng Ruohang 086e619505 Merge accepted bucket metadata reload correction
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 19:31:10 +08:00
Feng Ruohang 48e1846525 fix(tls): honor Go key exchange defaults across transports
Remove the eight explicit curve overrides so Go 1.27 honors tlsmlkem=0
across Server listeners, node links and outbound transports. Remove the
unused shared curve option and add wire-level regression coverage.

Document CA trust and TLS upgrade behavior, retain the investigation
artifacts, and exclude their synthetic routes from the rebrand guard.
The product compatibility baseline remains unchanged.

Validation: focused race tests, HTTP tests, lint, compatibility guard
positive/negative controls, and a fresh Linux build with three isolated
OIDC integration scenarios all pass.

Adversarial review: Claude Code Fable 5.1, max effort.
Final verdict: APPROVE FOR COMMIT.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 18:50:12 +08:00
Feng Ruohang bcc8871b1d Merge pull request #163 from pgsty/fix/issue-158-federated-copy-sse
fix: forward plaintext on federated CopyObject of SSE objects (#158)
2026-09-09 18:02:45 +08:00
Feng Ruohang 25cb3511c9 test: cover multipart SSE sources on federated CopyObject
A multipart SSE-S3 source is encrypted per part, so its logical size is
the sum of the parts' decrypted sizes and the decrypting reader crosses
a part boundary. Copy such a source across the federation to a plain
and to an SSE-S3 destination and check the destination plaintext and
single-encryption size.

The test router registers routes in endpoint order and the plain
PutObject route has no query matcher, so the multipart endpoints are
listed first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 17:51:32 +08:00
Feng Ruohang cfefc049c1 fix: send the SSE-KMS context as a JSON object on federated copies
putOptsFromReq handed the parsed kms.Context straight to
encrypt.NewSSEKMS. kms.Context implements encoding.TextMarshaler, so the
SDK serialized it as a JSON string, and a request without a context
still produced one because the nil Context is a typed nil inside the
interface value and marshals to "{}". The receiving ParseHTTP rejects
both forms, so every federated CopyObject to an SSE-KMS destination
failed with InvalidArgument once the forwarded stream was correct.

Pass a plain map, or nothing when no context was requested, and cover
SSE-KMS destinations with and without an explicit context.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 17:51:32 +08:00
Feng Ruohang af56d17630 fix: forward plaintext on federated CopyObject of SSE objects (#158)
The legacy etcd bucket-federation branch of CopyObjectHandler reads its
source through getObjectNInfo, which yields the decrypted and
decompressed bytes, but it also ran the destination encryption locally
and then forwarded that stream to the remote PutObject with the source's
stored size and the destination SSE option. SSE to plain and plain to
SSE therefore failed on a Content-Length mismatch, while SSE to SSE
matched by coincidence: the remote encrypted the ciphertext a second
time and stored an unreadable object, and a destination GET returned
the inner ciphertext with HTTP 200.

The remote write owns the destination's storage transformations, so
hand it the logical bytes at their logical size and let it encrypt
exactly once. Compression was already excluded on this branch; apply
the same rule to encryption, size the forwarded reader by actualSize,
and declare that size on the forwarded PutObject.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 17:51:32 +08:00
Feng Ruohang d1105bbb3d Merge pull request #162 from pgsty/codex/replication-reliability-20260909
fix(replication): complete purges, expose MRF drops, and cancel resyncs reliably
2026-09-09 14:55:35 +08:00
Feng Ruohang 66fe61ff65 test: reuse replication compatibility fixtures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:44:02 +08:00
Feng Ruohang a1141a43f2 style: format replication regression fixture
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:24 +08:00
Feng Ruohang 702f113f51 fix(replication): scope resync cancellation and drain worker lifecycle
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:02 +08:00
Feng Ruohang 63aace4099 fix(replication): expose bounded MRF queue drops
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:02 +08:00
Feng Ruohang 9d7094b770 fix(replication): complete single-object delete marker purges
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 14:37:02 +08:00
Feng Ruohang 450dcb8484 Merge pull request #161 from pgsty/codex/server-dependency-refresh-20260909
build: refresh maintained SILO stack dependencies
2026-09-09 12:04:16 +08:00
Feng Ruohang 4074d00b96 build: refresh maintained SILO stack dependencies
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-09 11:49:13 +08:00
Feng Ruohang 75ba0ce402 fix: drop the broken bucket-metadata reload publication guard (issue #105 T3)
The #105 T3 change (PR #156) tried to keep the resident metadata cache
monotonic by guarding peer-reload publication on lastUpdate(). But
lastUpdate() is the max of per-config timestamps and cannot order whole
records: a node caching {policy@20, CORS@10} that receives a newer CORS@15
still has lastUpdate()==20, so the guard rejects the legitimately-newer
record and the periodic refresh (same comparator) cannot repair it. A
paused reload could also resurrect deleted resident state.

Per the maintainer decision, revert the reload publication to its original
unconditional (acceptable-until-refresh) behavior:
- remove setReloaded and restore the plain Set plus notification/target
  registry updates in LoadBucketMetadataHandler;
- restore refreshBucketsMetadataLoop's own lastUpdate() staleness check and
  globalEventNotifier.set / globalBucketTargetSys.set publication;
- restore the unconditional GetConfig cache-miss publication;
- document the known freshness limitation at the reload site (the periodic
  refresh is best-effort and cannot repair an equal-maximum-timestamp
  divergence).

The T1 lifecycle merge-under-lock (UpdateExpiryLCConfig) and both T2 fixes
(DeleteBucket takes metadata.lock before deleting; saveMetadata and
loadBucketMetadataParseUnderLock recheck physical bucket existence) are
kept fully intact.

Tests:
- drop the T3 reproductions (overlapping-reload resident-cache test and the
  peer-reload-preserves-current-targets publication test);
- add lockBucketMetadataAcquireHook, a nil-in-production atomic test hook in
  the shared metadata.lock path, so tests can deterministically observe a
  caller (notably DeleteBucket, whose lock is taken through its
  erasureServerPools receiver and is invisible to an injected object layer)
  reaching the lock;
- rewrite the T2 delete-race ghost test to hold metadata.lock MID-SAVE (past
  saveMetadata's existence recheck) and synchronize on the delete's actual
  lock attempt via the hook, so it isolates the lock-before-delete fix:
  removing only DeleteBucket's metadata.lock (recheck kept) now fails it;
- rewrite the cancellation test to observe the delete's actual lock attempt,
  then cancel and await its error while still holding the lock, so a
  scheduling-delayed delete stopped by the canceled context can no longer
  pass on a broken tree.

Refs #105. Follows #156.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 16:13:54 +08:00
Feng Ruohang da142327f2 Merge pull request #159 from pgsty/fix/issue-99-100-followups
fix: repair federated CopyObject checksum edge cases (empty body, inherited, multipart-suffix)
2026-09-08 15:58:22 +08:00
Feng Ruohang a3df317ae0 Merge pull request #60 from mrjavadseydi/feat/access-based-ilm
[ILM] Relocate hot objects across server pools by GET frequency
2026-09-08 15:42:15 +08:00
Feng Ruohang 2c50d11f72 fix: correct federated CopyObject checksum edge cases (#99 follow-ups)
Three residual checksum defects in the legacy etcd federation branch of
CopyObjectHandler, found by post-merge review of #157.

1. Empty-source 500 regression. A checksum-less object gains the S3 default
   CRC-64NVME (WantServerSideChecksumType is set), but minio-go streams no
   trailing checksum for a 0-byte body (contentLength == 0), so the remote
   computed none, federatedChecksumValue was empty, hash.NewChecksumWithType
   returned nil, and the handler returned 500 -- so every empty-object
   federated copy failed. For a 0-byte source, forward the empty-content digest
   as an ordinary checksum request header instead, so the remote validates,
   persists and returns it, matching the local path (e.g. CRC32 "AAAAAA==").

2. Inherited full-object checksum dropped. When the source already carries a
   full-object checksum, the local path sets dstOpts.WantChecksum, not
   WantServerSideChecksumType (only multipart-composite sources are promoted).
   The federated branch inspected only WantServerSideChecksumType, so a
   checksum-bearing source's checksum was silently discarded on a federated
   copy that requested no algorithm. Forward WantChecksum.Encoded (always a
   plain digest) as a checksum header so the remote validates and persists it,
   and bind the returned value, matching local persistence.

3. Multipart-suffixed remote value accepted. The bind accepted a value like
   "NSRBwg==-0": NewChecksumWithType parses the "-N" as ChecksumMultipart with
   WantParts 0 and the length-only validator passes, so the destination was
   returned as COMPOSITE. A single forwarded PutObject must yield a full-object
   digest, so reject a multipart-marked parsed value in addition to the
   existing nil (missing/malformed) rejection.

A forwarded checksum request header is stripped from objInfo.UserDefined so it
is not mistaken for object metadata.

Out of scope: the SSE federated-copy corruption (srcInfo.Reader/Size mismatch
for encrypted sources) predates this work and is filed separately.

New federated regressions cover empty source with requested and default
checksum (200 + correct value + persisted), an inherited full-object checksum
preserved without a requested algorithm, and a multipart-suffixed remote value
rejected. Red/green verified for each against the merged code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:38:11 +08:00
Feng Ruohang 5ac33e1583 test: make access move failure recovery deterministic
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:28:33 +08:00
Feng Ruohang d57c4e8407 Merge remote-tracking branch 'origin/main' into codex/access-tiering-ci-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:08:34 +08:00
Feng Ruohang 374de0fa32 fix: make access tier moves preserve versions and isolate writes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 15:08:34 +08:00
Feng Ruohang 80e23dc9f2 Merge pull request #156 from pgsty/codex/bucket-metadata-merge-20260908
fix: preserve bucket metadata across concurrent updates and reloads
2026-09-08 15:03:52 +08:00
Feng Ruohang 2cc0e3c6ed test: construct notification fixtures with the event ARN type
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:52:48 +08:00
Feng Ruohang f9da3b919d fix: order bucket deletion and metadata publication safely
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:43:15 +08:00
Feng Ruohang 1309853f57 Merge remote-tracking branch 'origin/main' into codex/access-tiering-ci-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:34:51 +08:00
Feng Ruohang 39b8e6c30a Merge remote-tracking branch 'origin/main' into codex/bucket-metadata-merge-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 14:24:22 +08:00
Feng Ruohang 9a6e1477f4 ci: record access tiering compatibility identifiers
Record the ten documented MINIO_ILM_ACCESS settings, the internal object
metadata stamp, and the tracker storage-path suffix introduced by this PR.
The guard places the /ilm/access string in its routes set, but the value
is a component of the tracker object prefix, not a public HTTP endpoint.

Keep all existing compatibility entries. The guard, delivery rebrand check,
and Docker entrypoint compatibility tests pass with the refreshed manifest.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 13:18:50 +08:00
Feng Ruohang 49375ed2d3 Merge pull request #157 from pgsty/codex/federated-copy-merge-20260908
fix: preserve metadata and checksums in federated object copies
2026-09-08 13:16:30 +08:00
Feng Ruohang f817b5261c Merge pull request #151 from nikitapogromsky/fix/idempotent-add-system-target
logger: make AddSystemTarget idempotent
2026-09-08 13:14:30 +08:00
Feng Ruohang 885bd2c20a fix: reject missing remote checksums on federated copies
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 13:05:45 +08:00
Feng Ruohang 89b75e7913 Merge branch 'main' into feat/access-based-ilm 2026-09-08 13:04:25 +08:00
Feng Ruohang 079ebb1926 fix: serialize logger initialization and publish the console target safely
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:58:21 +08:00
Feng Ruohang 04c29aac11 Merge branch 'audit/issue-105-bucketmeta-races' into codex/bucket-metadata-merge-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:55:51 +08:00
Feng Ruohang 95e7a190b3 Merge branch 'fix/issue-99-100-federated-copyobject' into codex/federated-copy-merge-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:55:51 +08:00
Feng Ruohang 8e2392e48f Merge remote-tracking branch 'origin/main' into codex/logger-idempotency-20260908
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 12:55:50 +08:00
Feng Ruohang 3abe0d95a5 Merge pull request #149 from pgsty/codex/embedded-console-compat
fix: restore embedded Console proxy and WebSocket configuration
2026-09-08 10:09:21 +08:00
Feng Ruohang 9c6c9805de fix: select the validated Console mainline for embedding
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-08 10:00:45 +08:00
nikitapogromsky 5cb900bfad logger: make AddSystemTarget idempotent
Subscribe re-registered the console target on every console-log subscription, producing duplicate minio_logger_webhook_* series on each /minio/metrics/v3 scrape. Fixes #150

Signed-off-by: nikitapogromsky <129324283+nikitapogromsky@users.noreply.github.com>
2026-09-07 12:03:39 +03:00
Feng Ruohang 0af5d22286 fix: restore embedded Console proxy and WebSocket configuration
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 13:36:24 +08:00
Feng Ruohang f1687f402b fix: scope rebrand checks to the retired repository
Match the exact retired repository while preserving links to distinct repositories and historical issue titles in contributor credits. Continue rejecting live links to the retired repository in those credits.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 11:49:23 +08:00
Feng Ruohang ce606df2c4 fix: align contribution tooling with SILO AGPL policy
Clarify SILO contribution ownership and preserve prior copyright notices. Consolidate issue templates and route the legacy credits command through the maintained generator.

Validation: make rebrand-guard; bash -n update-credits.sh; regenerated credits match CREDITS; template and link checks.
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 11:33:21 +08:00
Feng Ruohang fd44dc4e9b docs: include the latest merged community contribution
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 11:01:04 +08:00
Feng Ruohang 479745e764 docs: credit contributors across the SILO repositories
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:58:47 +08:00
Feng Ruohang cc1c54475f Merge pull request #146 from pgsty/fix/issue-108-console-loopback-tls
fix: keep embedded Console login working over loopback TLS (#108)
2026-09-07 10:53:45 +08:00
Feng Ruohang e7654d470c fix: close residual bucket-metadata races (issue #105 audit)
Audit of the three deferred #105 follow-ups. Each reproduces with a
deterministic red test in cmd/bucket-metadata-race_test.go, and each fix is
the minimal change that turns its test green while preserving the
<bucket>.lck -> metadata.lock -> .metadata.bin lock order established by #103.

1. Lifecycle expiry merge lost update (persistent). PeerBucketLCConfigHandler
   and healBucketILMExpiry read the current lifecycle with an unlocked
   GetConfigFromDisk, merged the replicated expiry rules with the local
   transition rules, then wrote the pre-computed blob via Update. Any lifecycle
   transition change committed between the merge read and the merge write was
   silently lost. New BucketMetadataSys.UpdateExpiryLCConfig performs the read,
   merge, and save under one metadata.lock; mergeExpiryWithLCConfig now takes
   the locked snapshot and validates object-lock retention from it instead of
   re-reading (avoids a re-entrant metadata load under the lock).

2. DeleteBucket ghost .metadata.bin (persistent). DeleteBucket took only
   <bucket>.lck while config writers take only metadata.lock, so a writer that
   was mid-save could re-create .metadata.bin after the prefix purge. The purge
   now runs under metadata.lock, with a best-effort unlocked fallback so a
   delete is never blocked from completing.

3. Overlapping peer reloads publishing a stale resident cache (freshness only;
   the persisted record stays correct). LoadBucketMetadataHandler and the
   GetConfig cache-miss path published with an unconditional Set, so a reload
   that read an older revision could overwrite a newer resident record until the
   next refresh. New BucketMetadataSys.setReloaded (and a matching GetConfig
   guard) refuses to regress a newer resident record, mirroring
   refreshBucketsMetadataLoop.

Verification: go build -tags kqueue,dev ./...; go vet ./cmd; gofmt clean;
rebrand-guard baseline unchanged; go test -tags kqueue,dev ./cmd (207s) green;
new tests plus the #103 metadata suite green under -race.

Refs #105. Parent #102. Foundation #103.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:33:11 +08:00
Feng Ruohang 4b25f7e819 fix: return and persist checksum on federated CopyObject (#99)
The legacy etcd federation branch of CopyObjectHandler forwards the copied
bytes with minio-go Core.PutObject but never asked the remote for a checksum
and discarded any it returned, so a cross-deployment whole-object copy that
requested a checksum returned 200 with an empty checksum, and a checksum-less
source did not gain the S3 default CRC-64NVME that the local path assigns. The
request was neither honored nor rejected. This is the whole-object counterpart
of #72, which repaired the same class of defect for federated UploadPartCopy.

When a server-side checksum is wanted -- explicitly requested, inherited from a
multipart source, or the CRC-64NVME default for a checksum-less object, all
already captured in dstOpts.WantServerSideChecksumType -- the forwarded
PutObject now streams a trailing checksum of that type, so the remote computes
and persists it and echoes it in the response. The value the remote reports for
that exact write is bound into objInfo.Checksum, matching how the local
CopyObject path carries checksums into the CopyObjectResult. Reading the value
from the same UploadInfo that produced the ETag keeps the pair bound to one
write.

Only the requested algorithm is returned; a malformed or absent remote value
leaves objInfo.Checksum unset, so an ordinary copy that wanted no checksum
still returns none. Two small mapping helpers convert between the server's
hash.ChecksumType and the minio-go request type and response field.

New end-to-end tests drive the real federation branch through
getRemoteInstanceClient and minio-go into a second in-process deployment and
assert that CRC32/CRC32C/SHA256/CRC64NVME and the no-algorithm default are all
returned in the CopyObjectResult and persisted on the destination, and that a
requested algorithm never leaks other algorithms into the response.

Fixes #99

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:32:59 +08:00
Feng Ruohang 711b092f86 fix: keep embedded Console login working over loopback TLS (#108)
The embedded Console reaches the S3/STS API at https://127.0.0.1:<port>
(minioConfigToConsoleFeatures), a loopback endpoint whose TLS certificate is
not expected to carry a 127.0.0.1 SAN. silo-console v2.3.x began verifying
every outbound TLS peer, so the Console's STS AssumeRole handshake to that
loopback endpoint now fails certificate validation and BOTH local and LDAP
logins fail with a generic "invalid login". The failure happens in the Console
HTTP client before any request reaches a server auth/STS/LDAP handler, so no
server-side auth error is logged, matching the report.

Restore the documented loopback bypass by opting the embedded Console into its
endpoint-scoped CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY switch whenever the server
falls back to the 127.0.0.1 endpoint under TLS. The exemption is scoped to that
single loopback origin inside Console; every other HTTPS peer (IdP, Prometheus,
webhooks) stays verified, preserving the v2.3.x hardening. An explicitly
configured endpoint is reached under its own verified name and is never
exempted. initConsoleServer unsets CONSOLE_* before re-deriving them, so the
switch cannot be supplied by the operator on the embedded path; the server must
assert it.

Fixes #108

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:32:59 +08:00
Feng Ruohang 2bc103b80c fix: strip all reserved metadata on federated CopyObject (#100)
The legacy etcd federation branch of CopyObjectHandler forwards the copied
source metadata to the remote deployment with minio-go Core.PutObject, after
removing only two reserved keys (compression and actual-size). Every small
object is stored inline, so its stored metadata also carries
X-Minio-Internal-inline-data; the remote's setRequestLimitMiddleware rejects
any request bearing a reserved-prefix header (containsReservedMetadata), so
the forwarded write failed with 400 InvalidArgument "Your metadata headers
are not supported." for the default COPY metadata directive.

A plain federated PutObject must not carry any internal storage metadata, so
strip the whole reserved-prefix class before forwarding instead of an
enumerated subset. Enumerating a third key would only defer the next leak:
besides inline-data, replication bookkeeping (replica/replication status and
timestamps) is added to the same map earlier in the handler and would be
rejected just the same. None of these keys is required by the remote for a
correct plain PutObject; they are internal storage details the remote sets
for itself. The stripping uses stringsHasPrefixFold, matching the remote's
own case-insensitive detection. Ordinary user metadata (x-amz-meta-*) is
untouched and still copied.

The pre-existing UUID ETag on the federated write (no Content-MD5 is sent) is
out of scope and left unchanged, as recorded in the issue.

A new end-to-end test drives the real federation branch through
getRemoteInstanceClient and minio-go into a second in-process deployment,
copying an inline source with the default COPY directive. It asserts the copy
now succeeds, that no reserved-prefix header reaches the remote on any
forwarded request, and that copied user metadata survives.

Fixes #100

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-07 10:32:59 +08:00
Feng Ruohang ad873c7357 Merge pull request #132 from mrjavadseydi/fix/issue-106-bucket-quota-metrics
fix: report effective bucket quotas in metrics
2026-09-07 09:54:01 +08:00
mr javad seydi 0db4bf3b00 fix: report effective bucket quotas in metrics
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-05 21:23:46 +03:30
mr javad seydi 7a060cab1e feat(ilm): relocate hot objects across server pools by GET frequency
Keep NVMe/HDD pool pairs useful without remote tiering: promote objects that
are read often, demote previously moved objects once they go idle, and leave
the feature off until operators set a two-pool topology.

Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-08-15 13:48:22 +03:30
324 changed files with 42226 additions and 4312 deletions
-54
View File
@@ -1,54 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage
assignees: ''
---
## NOTE
Silo issues are handled by community maintainers on a best-effort basis. There
is no SLA, SLO, or emergency production-support channel. Follow the local
[Code of Conduct](../code_of_conduct.md) when participating. Report suspected
vulnerabilities through the private process in [SECURITY.md](../SECURITY.md),
not in a public issue.
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and include relevant Silo logs with secrets and credentials removed -->
1.
2.
3.
4.
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally include the Silo version, commit id, or PR that caused this regression. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`silo --version`):
* Server setup and configuration:
* Operating System and version (`uname -a`):
+8
View File
@@ -7,6 +7,14 @@ assignees: ''
---
Report bugs in the PGSTY SILO server (`pgsty/silo`) here. Community maintainers
handle reports on a best-effort basis. There is no SLA, SLO, or emergency
production-support channel. Follow the
[Code of Conduct](https://github.com/pgsty/silo/blob/main/code_of_conduct.md).
Report suspected vulnerabilities privately through
[SECURITY.md](https://github.com/pgsty/silo/blob/main/SECURITY.md).
For patches, see the [contribution guide](https://github.com/pgsty/silo/blob/main/CONTRIBUTING.md).
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
@@ -7,6 +7,9 @@ assignees: ''
---
Suggest improvements to the PGSTY SILO server (`pgsty/silo`) here.
For patches, see the [contribution guide](https://github.com/pgsty/silo/blob/main/CONTRIBUTING.md).
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+7 -3
View File
@@ -1,10 +1,14 @@
## Contribution Licensing (no CLA, inbound=outbound, DCO required)
This project does not use a CLA; contributions are accepted inbound=outbound.
This pull request contributes to PGSTY SILO (`pgsty/silo`). Code contributions
are accepted under AGPL-3.0-or-later, the same license as the server.
This project does not use a CLA or require a separate Apache-2.0 license grant.
By submitting this pull request I represent that I have the right to contribute
the changes, which are licensed under this repository's
the code changes under this repository's
[GNU Affero General Public License v3.0 or later](https://www.gnu.org/licenses/agpl-3.0.html)
and remain my copyright. Every commit must carry a DCO `Signed-off-by` trailer
and retain copyright in my original work. Existing copyright and license
notices remain intact; separately licensed material keeps its applicable terms.
Every commit must carry a DCO `Signed-off-by` trailer
(`git commit -s`) certifying the
[Developer Certificate of Origin](https://developercertificate.org/) — see
[CONTRIBUTING.md](https://github.com/pgsty/silo/blob/main/CONTRIBUTING.md).
+4
View File
@@ -27,6 +27,10 @@ archives:
ids:
- silo
name_template: "silo_{{ .Env.PKG_VERSION }}_{{ .Os }}_{{ .Arch }}"
formats: ["tar.gz"]
format_overrides:
- goos: windows
formats: ["zip"]
# Explicit so the license materials cannot silently drop out of the
# binary archives: GoReleaser's default file globs would miss NOTICE.
# CREDITS stays out deliberately -- at 1.8MB it would dominate the
+1 -1
View File
@@ -140,7 +140,7 @@ jobs:
mkdir -p "${context}/dockerscripts"
tar -xzf "${archive}" -C "${context}" silo
cp Dockerfile.goreleaser Dockerfile.distroless LICENSE NOTICE CREDITS "${context}/"
cp dockerscripts/docker-entrypoint.sh dockerscripts/download-static-curl.sh \
cp dockerscripts/docker-entrypoint.sh dockerscripts/build-static-curl.sh \
"${context}/dockerscripts/"
done
+28
View File
@@ -90,6 +90,34 @@ jobs:
- name: Run S3 Select tests under race detector
run: go test -race ./internal/s3select/... -count=1
- name: Run conditional PUT tests under race detector
run: go test -race ./cmd -run '^Test(PoolsConditionalPut|SinglePoolConditionalPutHTTP)' -count=1 -timeout=5m
- name: Run multi-pool delete consistency tests under race detector
run: go test -race ./cmd -run '^Test(Pools(Delete|BulkDelete|RetiringCopyPreservesSharedTierObject)|BulkDeleteQuorumEvidence)' -count=1 -timeout=5m
- name: Run multipart listing, cancellation and cleanup tests under race detector
env:
MINIO_API_REQUESTS_MAX: "10000"
run: go test -race ./cmd -run '^Test(Multipart|PaginateMultipartUploads|ListMultipartUploads|CleanupStaleUploads|ExtractModTime)' -count=1 -timeout=5m
- name: Run listing metadata recovery, pagination and cancellation tests under race detector
run: go test -race ./cmd -run '^Test(ListFallback|ListingCache|ListingMerge|ListingRecovery|ListingPending|ListingEntry|ListingSecondLock|StreamMetadataParts|LocalLockerListing|DistributedListing|MergeXLV2LatestNullCrossParity|CountXLV2MixedECGroups|ListCrossParity)' -count=1 -timeout=5m
- name: Run lock cancellation and late-grant cleanup tests under race detector
run: go test -race ./internal/dsync ./internal/lsync -run '^Test(DRWMutex|LRWMutex)' -count=1 -timeout=5m
- name: Run CPU metrics tests under race detector
run: go test -race ./cmd -run '^TestLoadCPUMetrics' -count=1 -timeout=5m
- name: Run tag replication tests under race detector
run: go test -race ./cmd -run '^TestAPITagging' -count=1 -timeout=5m
- name: Run Object Lock authorization and overwrite tests under race detector
env:
MINIO_API_REQUESTS_MAX: "10000"
run: go test -race -tags kqueue,dev ./cmd -run '^Test(APIPutObjectRetention|APIVersionAddressedWrite|PutObjectRetention|DirectoryObjectLock|ObjectLockOverwrite|APICopyObjectPreservesVersionAfterSuspension|MultipartNullObjectLock|PoolsObjectLockOverwrite|MultipartVersioningTransitions|PlanReview|RecoveredMultipartUpload)' -count=1 -timeout=5m
crosscompile:
name: Cross Compile
runs-on: ubuntu-latest
+9 -3
View File
@@ -70,6 +70,12 @@ jobs:
buildscripts/verify-rebrand.sh
dockerscripts/docker-entrypoint_test.sh
# Keep this gate aligned with the Object Lock race regression in go.yml.
- name: Gate release on Object Lock authorization and overwrite tests under race detector
env:
MINIO_API_REQUESTS_MAX: "10000"
run: go test -race -tags kqueue,dev ./cmd -run '^Test(APIPutObjectRetention|APIVersionAddressedWrite|PutObjectRetention|DirectoryObjectLock|ObjectLockOverwrite|APICopyObjectPreservesVersionAfterSuspension|MultipartNullObjectLock|PoolsObjectLockOverwrite|MultipartVersioningTransitions|PlanReview|RecoveredMultipartUpload)' -count=1 -timeout=5m
- name: Compute release variables
env:
# Passed through the environment, never interpolated into the script
@@ -132,9 +138,9 @@ jobs:
cosign-release: v3.1.2
- name: Build Draft release with GoReleaser
uses: goreleaser/goreleaser-action@v7
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
with:
version: "~> v2"
version: v2.18.3
args: release --clean --skip=validate --config .github/goreleaser.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -150,7 +156,7 @@ jobs:
- name: Install nFPM
run: |
set -euo pipefail
go install github.com/goreleaser/nfpm/v2/cmd/nfpm@v2.47.0
go install github.com/goreleaser/nfpm/v2/cmd/nfpm@v2.48.0
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
- name: Build nFPM packages
+102
View File
@@ -0,0 +1,102 @@
name: Repository Cards
on:
schedule:
# 00:00 UTC = 08:00 Asia/Shanghai. GitHub may queue scheduled runs.
- cron: "0 0 * * *"
workflow_dispatch:
push:
branches: [main]
paths:
- .github/workflows/repository-cards.yml
- .github/silo.svg
- buildscripts/repository-cards/**
pull_request:
branches: [main]
paths:
- .github/workflows/repository-cards.yml
- .github/silo.svg
- buildscripts/repository-cards/**
permissions:
contents: read
concurrency:
group: repository-cards-${{ github.event.pull_request.number || 'publish' }}
cancel-in-progress: false
jobs:
check:
name: Validate repository cards
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Use the runner's Python in an isolated environment
run: |
python3 -m venv "$RUNNER_TEMP/repository-cards-venv"
"$RUNNER_TEMP/repository-cards-venv/bin/python" -m pip install -r buildscripts/repository-cards/requirements.txt
echo "$RUNNER_TEMP/repository-cards-venv/bin" >> "$GITHUB_PATH"
- run: python -m unittest discover -s buildscripts/repository-cards -p 'test_*.py' -v
publish:
name: Update README images
needs: check
if: github.repository == 'pgsty/silo' && github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write
issues: read
pull-requests: read
env:
OUTPUT_BRANCH: codex/repository-cards
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- name: Use the runner's Python in an isolated environment
run: |
python3 -m venv "$RUNNER_TEMP/repository-cards-venv"
"$RUNNER_TEMP/repository-cards-venv/bin/python" -m pip install -r buildscripts/repository-cards/requirements.txt
echo "$RUNNER_TEMP/repository-cards-venv/bin" >> "$GITHUB_PATH"
- name: Load the generated-assets branch
shell: bash
run: |
set -euo pipefail
artifacts="$RUNNER_TEMP/repository-cards"
if git ls-remote --exit-code --heads origin "$OUTPUT_BRANCH"; then
git fetch --depth=1 origin "$OUTPUT_BRANCH"
git worktree add --detach "$artifacts" FETCH_HEAD
else
status=$?
# Exit 2 means no matching ref; transport/auth failures must stop.
if [ "$status" -ne 2 ]; then exit "$status"; fi
git worktree add --detach "$artifacts" HEAD
git -C "$artifacts" checkout --orphan "$OUTPUT_BRANCH"
git -C "$artifacts" rm -rf .
fi
- name: Refresh contributor and star cards
env:
GH_TOKEN: ${{ github.token }}
run: python buildscripts/repository-cards/update.py --output "$RUNNER_TEMP/repository-cards"
- name: Publish changed assets
shell: bash
run: |
set -euo pipefail
cd "$RUNNER_TEMP/repository-cards"
git add README.md history.json curated.json contributors.json \
contributors-light.svg contributors-dark.svg \
star-history-light.svg star-history-dark.svg
if git diff --cached --quiet; then
echo "Repository cards are already current."
exit 0
fi
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git commit -s -m "chore: update repository cards $(date -u +%F)"
# A normal push preserves history and refuses concurrent overwrites.
git push origin "HEAD:refs/heads/$OUTPUT_BRANCH"
+55 -10
View File
@@ -4,13 +4,15 @@ on:
workflow_dispatch:
pull_request:
paths:
- "go.mod"
- "go.sum"
- ".github/goreleaser.yml"
- ".github/nfpm.yml"
- "Dockerfile.goreleaser"
- "Dockerfile.distroless"
- "cmd/healthcheck-main.go"
- "cmd/main.go"
- "dockerscripts/download-static-curl.sh"
- "dockerscripts/build-static-curl.sh"
- "dockerscripts/docker-entrypoint.sh"
- "dockerscripts/docker-entrypoint_test.sh"
- "silo.service"
@@ -41,6 +43,28 @@ permissions:
contents: read
jobs:
curl:
name: Static curl (${{ matrix.arch }})
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- arch: amd64
runner: ubuntu-latest
- arch: arm64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v7
- name: Build and exercise curl in an empty runtime
run: |
docker build --build-arg TARGETARCH=${{ matrix.arch }} \
--target curl-runtime -f Dockerfile.goreleaser -t silo-curl-test .
docker run --rm silo-curl-test --version | tee curl-version.txt
grep -F 'curl 8.23.0 ' curl-version.txt
grep -F 'HTTP2' curl-version.txt
docker run --rm silo-curl-test --fail --silent --show-error \
--connect-timeout 15 --max-time 60 https://curl.se/robots.txt
validate:
runs-on: ubuntu-latest
steps:
@@ -71,9 +95,9 @@ jobs:
echo "LDFLAGS: ${LDFLAGS}"
- name: GoReleaser config check
uses: goreleaser/goreleaser-action@v7
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
with:
version: "~> v2"
version: v2.18.3
args: check --config .github/goreleaser.yml
- name: Validate Helm chart and legacy upgrade identity
@@ -85,9 +109,9 @@ jobs:
syft-version: v1.50.0
- name: Build snapshot artifacts
uses: goreleaser/goreleaser-action@v7
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
with:
version: "~> v2"
version: v2.18.3
# A pull-request snapshot has no trusted release identity. Exercise
# the SBOM/checksum pipeline here, and reserve keyless signing for
# the tag-triggered release workflow with GitHub OIDC.
@@ -96,13 +120,34 @@ jobs:
LDFLAGS: ${{ env.LDFLAGS }}
PKG_VERSION: ${{ env.PKG_VERSION }}
- name: Verify archive SBOM and checksum coverage
- name: Verify archive formats, documentation, SBOMs, and checksums
run: |
set -euo pipefail
mapfile -t archives < <(find dist -maxdepth 1 -type f -name 'silo_*.tar.gz' | sort)
mapfile -t sboms < <(find dist -maxdepth 1 -type f -name 'silo_*.tar.gz.sbom.json' | sort)
mapfile -t archives < <(find dist -maxdepth 1 -type f \( -name 'silo_*.tar.gz' -o -name 'silo_*.zip' \) | sort)
mapfile -t sboms < <(find dist -maxdepth 1 -type f \( -name 'silo_*.tar.gz.sbom.json' -o -name 'silo_*.zip.sbom.json' \) | sort)
test "${#archives[@]}" -eq 6
test "${#sboms[@]}" -eq 6
for os in linux darwin windows; do
format=tar.gz
binary=silo
if [ "${os}" = windows ]; then
format=zip
binary=silo.exe
fi
for arch in amd64 arm64; do
archive="dist/silo_${PKG_VERSION}_${os}_${arch}.${format}"
test -s "${archive}"
jq -e '.spdxVersion | startswith("SPDX-")' "${archive}.sbom.json" >/dev/null
if [ "${format}" = zip ]; then
listing="$(unzip -Z1 "${archive}")"
else
listing="$(tar -tzf "${archive}")"
fi
for member in "${binary}" LICENSE NOTICE README.md; do
grep -Fx "${member}" <<< "${listing}"
done
done
done
manifest="dist/silo_${PKG_VERSION}_checksums.txt"
test -s "${manifest}"
(
@@ -119,7 +164,7 @@ jobs:
- name: Install package validation tools
run: |
set -euo pipefail
go install github.com/goreleaser/nfpm/v2/cmd/nfpm@v2.47.0
go install github.com/goreleaser/nfpm/v2/cmd/nfpm@v2.48.0
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
sudo apt-get update
sudo apt-get install --yes rpm binutils
@@ -490,7 +535,7 @@ jobs:
bash -n buildscripts/package/lifecycle_test.sh
buildscripts/package/lifecycle_test.sh
bash -n dockerscripts/docker-entrypoint_test.sh
bash -n dockerscripts/download-static-curl.sh
bash -n dockerscripts/build-static-curl.sh
dockerscripts/docker-entrypoint_test.sh
go run ./buildscripts/rebrand-guard
buildscripts/verify-rebrand.sh
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
- name: Install govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@v1.7.0
go install golang.org/x/vuln/cmd/govulncheck@v1.9.0
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
- name: Run govulncheck
+2 -2
View File
@@ -62,10 +62,10 @@ dist/
.claude/
.codex/
AGENTS.md
CLAUDE.md
_bmad/
_bmad-output/
# Investigation and editorial working files belong outside this repository.
docs/investigations/
docs/security/
docs/rebranding.md
.release-sign/
+84
View File
@@ -0,0 +1,84 @@
# SILO Repository Guide
## Project map
- Server: `pgsty/silo` (this checkout, default branch `main`)
- Console: `pgsty/silo-console` (usual sibling checkout `../silo-console`)
- Client: `pgsty/mc`, shipped as `mcli` (usual sibling checkout `../mc`)
- Shared packages: `pgsty/silo-pkg` (usual sibling checkout `../silo-pkg`)
- Documentation: `pgsty/silo.pgsty.com` (usual sibling checkout `../silo.pgsty.com`),
published at <https://silo.pgsty.com/>
The server executable, package, systemd service, and container image use `silo`;
the public Docker image is `docker.io/pgsty/silo`. The server module remains
`github.com/minio/minio`. Consult the current `go.mod` and release configuration
for selected component versions and compatibility replacements.
## Supported stack and compatibility policy
The maintained, release-gating product graph is the coordinated PGSTY stack:
`silo` + `silo-console` + `mc` + `silo-pkg`.
Keep compatibility with upstream MinIO/MC on a best-effort basis. Preserve
inexpensive wire, configuration, CLI, migration, and import compatibility when
it helps users, and document known differences. Do not infer from source
lineage, MinIO-compatible protocols, retained `MINIO_*`/`MC_*` names, or an
inherited upstream test that unmodified upstream MinIO is a supported release
target.
Upstream-only compatibility checks may remain as advisory evidence, but they
must not force a downgrade of a maintained SILO component, a fork-only API
workaround, or a release block. Make upstream compatibility a hard gate only
when the user explicitly requests that scope.
## Dependency selection
- When PGSTY maintains a component, import and require it directly under its own
module path. In particular, prefer `github.com/pgsty/silo-pkg/v3` over
`github.com/minio/pkg/v3` in maintained SILO source.
- Do not use an upstream package merely to make unmodified upstream MinIO or MC
compile. Unavoidable transitive upstream modules should be documented and
kept separate from the maintained product dependency.
- `github.com/minio/minio-go/v7` is the explicit exception: use the verified
upstream module/commit while it contains the required fixes; do not recreate
a SILO fork without a concrete functional divergence.
- Check the current `go.mod` before changing versions. Coordinate breaking
import-path changes across package, client, Console, and server releases.
## Documentation and delivery
The companion site owns product, operations, migration, security-advisory,
design, and release documentation. Maintain English and Chinese content in
that repository and link to its canonical URLs from this one. The site's
`content/docs/` is the documentation entry point; detailed pages live under
`content/operations/`, `administration/`, `reference/`, `compatibility/`,
`about/`, and `blog/`, following the existing structure.
Keep repository entry points and contributor instructions here. Retained
upstream documents, examples, and test fixtures under `docs/` may be updated
when a code or tooling change requires it; do not build a second documentation
site in this tree.
Investigation plans, prompts, AI session transcripts, execution logs, temporary
reports, and private security material belong outside the checkout, for example
in a task-specific directory under `~/tmp/`. Do not recreate
`docs/investigations/`, `docs/security/`, or `docs/rebranding.md`, or move their
working material to another repository directory. Extract reusable, verified
knowledge into the companion site; keep private evidence outside public Git.
Compatibility pages may continue to describe similarities with upstream, but
must label that compatibility as best effort. The supported and tested server
for Console and mcli administration is `pgsty/silo`.
Before removing or moving documentation, check repository links and scripts,
the companion site's links and anchors, and references from this guide. Use
fixed commit URLs for historical evidence that should survive deletion from
the current tree. Run the site's `make check` for site changes and this
repository's `make rebrand-guard` for documentation cleanup that changes the
identifier inventory. When new public URLs are involved, publish the site
before publishing source changes that depend on those URLs.
Keep changes in the repository that owns the affected surface. Treat every
repository commit, tag, release, image, documentation update, and deployment as
a separate deliverable. Follow `CONTRIBUTING.md`, including DCO sign-off
(`git commit -s`). `CLAUDE.md` imports this guide so both agents use one policy.
+140
View File
@@ -0,0 +1,140 @@
# Changelog
## Unreleased
- Fix LDAP login for long normalized DNs without hiding unreadable IAM
revocation records. Only length errors for valid keys with an ASCII path
component over 255 bytes are treated as absent; long-DN writes still fail.
Thanks to [@910patin](https://github.com/910patin) for the original fix and
reproduction in [#250](https://github.com/pgsty/silo/pull/250).
- Build the server and standalone debugging tools with Go 1.27.3 and refresh
Go dependencies, including silo-pkg v3.15.1, the published mcli 20261025
source and upstream minio-go `ab93db7ba435` for multipart, read/seek and
batch-delete fixes. Retain output-format and NetBSD compatibility selections.
- Embed the unreleased Console 2.5.0 candidate `258717be077b`, including its
GUI update, diagnostic WebSocket closing handshake and complete health-report
decoding. This selects a source commit, not a new Console release.
- Read current bucket inventory in account-info responses so Console refreshes
immediately reflect successful creation and deletion. This management endpoint
now performs a live list instead of returning a cached inventory.
- Restore the native built-in policy immediately after resetting an override.
Peer and etcd notifications reload committed state, so delayed delete events
preserve subsequent edits. Keep deletion revisions, policy-in-use checks and
custom-policy deletion behavior; storage failures retain the previous policy.
- Preserve Kafka's default broker protocol 2.1 and terminal-width behavior;
fix legacy DARE v1 key unsealing and verify DARE v1/v2 and MinKMS TLS compatibility.
- Update bundled curl to 8.23.0 and require patched static libraries. Apply
[Red Hat's compromised-CA mitigation](https://access.redhat.com/security/vulnerabilities/RHSB-2026-012);
the root blocklist also rejects legitimate chains under the affected roots.
These source changes are not included in the published 20261018 artifacts.
## RELEASE.2026-10-18T00-00-00Z
Package version: `20261018000000.0.0`; published on 2026-10-18 from source
`0cd388c666f8211f552604ae73448a066e9c25ce`. This section covers the changes
after the public 20260916 source, `2a4d51406b7ed87af5fe6fe0f801f3290f96eb3c`.
Full [English release notes](https://silo.pgsty.com/blog/release/silo-20261018/)
and [Chinese release notes](https://silo.pgsty.com/zh/blog/release/silo-20261018/)
cover the upgrade contract, evidence and remaining limits.
- **Object Lock (#234, #235):** require `s3:PutObjectRetention` with request-body
conditions before object lookup; weakening active GOVERNANCE additionally
requires bypass intent and permission. Protect caller-addressed write versions,
directory null versions and persisted multipart targets under the write lock;
fail closed on unreadable lock state. See SN-2026-015/016 in the
[advisory ledger](https://silo.pgsty.com/about/security-advisories/).
- **IAM Admin authorization (#242, `995439260`):** enforce explicit self denies,
session restrictions and plugin decisions; prevent restricted delegation and
non-root mutation of root-derived credentials under built-in authorization.
Preserve empty-policy inheritance and damaged-key cleanup. Normalize bulk
self-list aliases and allow ordinary policyless self inspection. See
[SN-2026-017](https://silo.pgsty.com/blog/security/20261018-release-hardening/#sn-2026-017).
- **Scoped STS revocation (`0cd388c66`):** enumerate durable parent/type-selected
tokens, propagate each committed tombstone to sibling nodes and sites, and
recover missed notifications through existing refresh/healing. Tokens issued
only at an isolated site are outside the requesting cluster's snapshot;
propagation is eventual and newly issued keys remain valid.
- **Pool migration (#248, `60c7179dd`, `1748f62c7`):** prevent stale moved
payloads from overwriting newer copies, preserve null/UUID delete markers,
recheck source versions and clean only confirmed moved versions under the
object lock. Added per-pool reads and per-version cleanup have maintenance
costs; earlier data loss is not reconstructed.
- **LIST recovery (#239, `3dd53eb81`, `2080b3d03`):** confirm eligible uncertain
latest-object groups under the appropriate read locks, distinguish absence
from failure, preserve directory/common-prefix entries and terminal pagination
errors, and make lock backoff cancelable. Resolved entries keep their fast path;
version history, quorum and disk formats retain their contracts. The strict
sub-5% performance target remains unconfirmed in the recorded comparison.
- **Multipart listing (`4733e1423`):** withdraw #243's default persistent scan.
Public listing uses node-local cache or exact-key lookup; restart omissions,
prefix and cache-pagination limits remain. The retired strict switch is ignored
with a startup warning. **Former strict users lose durable listing and strict
cancellation.** Keep identity metadata, path/wrong-target safety, default
best-effort Abort and peer notification. Diagnostic `multipart-preflight`
reports `mode=legacy`; its budgets do not gate public operations. #79 remains
open, including delayed creation after cancellation.
- **Multipart cleanup (#236, `0f9199e0c`):** parse native timestamps before the
Base64 fallback and remove empty hash parents without a second expiry wait.
Keep active/unreadable uploads; default expiry/check intervals stay 24h/6h.
- **Commit errors (#232):** missing staged `xl.meta` at final commit becomes
HTTP 503 `SlowDownWrite`; preserve this classification across storage RPC and
healing. Real incomplete bodies retain their existing errors.
- **Reused metadata (#233):** clear absent legacy `EcM`/`Signature` and locally
reset mutually exclusive scanner response fields without changing disk/RPC formats.
- **Checksum and Console errors (#225, #229, #238):** align Content-MD5 and
algorithm-specific messages, retaining the generic unknown-algorithm message;
embed Console's wrapped-error and `InvalidBucketState` handling.
- **Lambda and erasure documentation (#220, `2fde3cf53`):** fix inverted
`TargetIDSet.IsEmpty`; document parity/quorum per object and erasure set.
- **Coordinated dependencies:** Console v2.4.2 (`ef2dd2c41d21`), mcli 20261018
(`fa3b8bf72458`), silo-pkg v3.14.3, upstream minio-go v7.4.1 and Go 1.27.2.
Standalone Console/mcli still declare pkg v3.14.2. The Server package update
speeds admin-action classification without changing policy behavior. Refresh
Go x/*, go-jose v4.1.6, LDAP v3.4.15, compress v1.20.3, msgp v1.6.5, etcd v3.7.2,
fwd v1.2.1 and cpuid v2.4.1. OIDC RSA JWKs above 4096 bits are rejected.
Keep the NetBSD go-systemd v22.6.0 replacement; parallel SDK GET/Compose remain opt-in.
- **Build and delivery:** gate releases on Object Lock regressions; use
Git for rebrand scanning without a ripgrep dependency (`7007c164a`). Refresh UBI 9 pins,
GoReleaser v2.18.3, nFPM v2.48.0, golangci-lint v2.15.0 and govulncheck v1.9.0.
Update bundled client hashes/helper and the Helm client image to mcli 20261018.
The chart's Server default stays at the public 20260916 image until publication.
Upgrade every writer and replication site; follow the
[IAM recovery guide](https://silo.pgsty.com/operations/replication/iam-upgrade/).
Old strict multipart users must review the capability loss before upgrading.
#217's multi-site delete-marker late creation and crash residue remain unresolved.
Source tests, draft artifacts, public images and production rollout are separate
acceptance boundaries; historical 20261015 test counts do not certify this source.
## RELEASE.2026-09-16T00-00-00Z
Package version: `20260916000000.0.0`; published source:
`2a4d51406b7ed87af5fe6fe0f801f3290f96eb3c`.
This release includes signed-header and payload-policy hardening (SN-2026-011/012),
durable IAM revocations (SN-2026-013), the bounded Console sharing proxy
(SN-2026-014), password-permission separation, multi-pool conditions/deletion/tag
repairs, encrypted/federated copy repairs, delete-marker retry fixes, opt-in strict
multipart discovery, Go-default TLS key exchange, absolute HTTP/1 header deadlines
and synchronized CPU metrics. It embeds Console v2.4.1, mcli 20260916 and pkg v3.14.1.
The complete release, contributor credits, historical artifact acceptance and
upgrade requirements are preserved in the canonical
[English](https://silo.pgsty.com/blog/release/silo-20260916/) and
[Chinese](https://silo.pgsty.com/zh/blog/release/silo-20260916/) release records.
## RELEASE.2026-09-03T13-18-01Z
Published source: `9b11dc9469e650815b775cb47b039610644f5da4`.
[Complete release notes](https://silo.pgsty.com/blog/release/silo-20260903/) ·
[GitHub release](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z)
This release ships Go 1.27.1, silo-pkg v3.13.2, upstream minio-go `0e78d3f18efe`,
mcli 20260903 and embedded Console source `464a59d73ada` (v2.3.0 version identity).
Installing the newer standalone mcli or Console does not replace components
inside this existing Server binary or image.
Earlier releases: [release archive](https://github.com/pgsty/silo/releases).
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
+24 -8
View File
@@ -77,11 +77,22 @@ test evidence, compatibility notes, and documentation impact. Public product
documentation is owned by the separate
[`pgsty/silo.pgsty.com`](https://github.com/pgsty/silo.pgsty.com) repository.
## Contributor recognition
Every human issue or pull-request author is recognized in [CONTRIBUTORS.md](CONTRIBUTORS.md),
including open issues, draft PRs, and PRs closed without merging. Merged fixes,
adopted proposals, and reports that lead to fixes receive greater prominence;
first participation guides the remaining order. Work incorporated through a
later PR retains credit without changing the original PR's recorded status.
Security disclosures are credited with the reporter's agreement. DCO sign-off
applies to code commits, not to opening an issue.
## Licensing of Contributions
Silo is licensed under the [GNU AGPL v3.0 or later](LICENSE). Its core is
Copyright (c) MinIO, Inc.; the combined work can never be relicensed, and this
fork does not try to.
Code contributions to PGSTY SILO (`pgsty/silo`) are accepted under the
[GNU AGPL v3.0 or later](LICENSE), the same license as the server. Submit issues
and pull requests to this repository's maintainers. No separate Apache-2.0
license grant to SILO or upstream MinIO maintainers is required.
* **No CLA.** We do not ask you to sign a Contributor License Agreement and we
do not take your copyright. Contributions are accepted inbound=outbound: you
@@ -112,15 +123,20 @@ fork does not try to.
`Signed-off-by` trailers) and add your own sign-off as the person passing it
along. Never import code from a proprietary distribution.
* **File headers.** Files derived from upstream keep the original MinIO
copyright header unchanged. New files added by this fork use the dual
header, followed by the standard AGPL boilerplate:
* **File headers.** Preserve existing copyright and license notices in inherited
and third-party files. New original files name their actual copyright holders
and use AGPL-3.0-or-later. Use a header such as the following, then append the
standard AGPL boilerplate:
```
// Copyright (c) 2015-2025 MinIO, Inc.
// Copyright (c) 2025-2026 PGSTY
// Copyright (c) 2026 Your Name
```
* **Separately licensed material.** Documentation contributions in `docs/`
follow its existing [CC BY 4.0 license](docs/LICENSE). Third-party components
and earlier Apache-2.0 contributions retain their original licenses and
attribution; this policy does not relicense earlier work.
* **Squash merges** must keep the `Signed-off-by:` trailers in the resulting
commit message.
+193 -72
View File
File diff suppressed because one or more lines are too long
+835 -276
View File
File diff suppressed because it is too large Load Diff
+31 -11
View File
@@ -1,4 +1,15 @@
FROM golang:1.27.1-alpine AS build
FROM golang:1.27.3-alpine@sha256:d530cf60723abf1ffae0cd16bb546b379ec0550d2f3815e5b16ed69dbc00fba4 AS curl-build
ARG TARGETARCH
COPY dockerscripts/build-static-curl.sh /build/build-static-curl
RUN /bin/sh /build/build-static-curl
# Exercise the exact shipped curl without a dynamic loader or shared libraries.
FROM scratch AS curl-runtime
COPY --from=curl-build /go/bin/curl /curl
COPY --from=curl-build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/curl"]
FROM golang:1.27.3-alpine@sha256:d530cf60723abf1ffae0cd16bb546b379ec0550d2f3815e5b16ed69dbc00fba4 AS build
ARG TARGETARCH
@@ -6,9 +17,9 @@ ENV GOPATH=/go
ENV CGO_ENABLED=0
ARG MC_REPO=pgsty/mc
ARG MC_VERSION=RELEASE.2026-09-03T07-13-05Z
ARG MC_AMD64_SHA256=cd7fcd449bb6b52e2eb727431ba6975b1e5d90df011a75869020ea9ac9e2b2a8
ARG MC_ARM64_SHA256=7962afc37c3e60e5758b19e819cb62d2f340ee655fad7b067e2ac9bc5716c2e8
ARG MC_VERSION=RELEASE.2026-10-18T00-00-00Z
ARG MC_AMD64_SHA256=b5392a642dbc9685b517ee901f032d1d5e88f41704436f0f08dde6060b577db3
ARG MC_ARM64_SHA256=17f3eb694ed34e155b331b1215832140749656d8d4b60e2e3a4d0573b8f320d7
RUN apk add -U --no-cache \
ca-certificates \
@@ -56,18 +67,26 @@ RUN apk add -U --no-cache \
chmod +x /go/bin/mcli && \
ln -sf mcli /go/bin/mc
COPY dockerscripts/download-static-curl.sh /build/download-static-curl
RUN chmod +x /build/download-static-curl && \
/build/download-static-curl
FROM registry.access.redhat.com/ubi9/ubi:latest AS certs
FROM registry.access.redhat.com/ubi9/ubi:latest@sha256:16e9e43d99e181463513cc84a514859b1abb9460f1c4d9e208da4b4e51cd5cc3 AS certs
# Native root blocklist mitigation: https://access.redhat.com/security/vulnerabilities/RHSB-2026-012
RUN dnf -y install ca-certificates && \
if [ -d /etc/pki/ca-trust/source/blocklist ]; then \
blocklist=/etc/pki/ca-trust/source/blocklist; \
else \
blocklist=/etc/pki/ca-trust/source/blacklist; \
fi && \
trust extract --overwrite --format=pem-bundle \
--filter='pkcs11:object=Comodo%20AAA%20Services%20root;type=cert' \
"$blocklist/Comodo_AAA_Services_root.pem" && \
trust extract --overwrite --format=pem-bundle \
--filter='pkcs11:object=emSign%20ECC%20Root%20CA%20-%20G3;type=cert' \
"$blocklist/emSign_ECC_Root_CA_-_G3.pem" && \
update-ca-trust && \
cp /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /tmp/ca-certificates.crt && \
dnf clean all && \
rm -rf /var/cache/dnf
FROM registry.access.redhat.com/ubi9/ubi-micro:latest
FROM registry.access.redhat.com/ubi9/ubi-micro:latest@sha256:5be877a2a280ae36eb9ab80ca4efb28bbf77c57abbba38137cdc37a03b8465e6
LABEL org.opencontainers.image.title="Silo" \
org.opencontainers.image.description="S3-Interface Libre Object Storage" \
@@ -88,7 +107,8 @@ ENV MINIO_ACCESS_KEY_FILE=access_key \
COPY --from=certs /tmp/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY silo /usr/bin/silo
COPY --from=build /go/bin/mcli /usr/bin/mcli
COPY --from=build /go/bin/curl* /usr/bin/
COPY --from=curl-build /go/bin/curl /usr/bin/curl
COPY --from=curl-build /go/share/curl /licenses/curl
COPY dockerscripts/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
COPY LICENSE /licenses/LICENSE
COPY NOTICE /licenses/NOTICE
+2 -2
View File
@@ -4,7 +4,7 @@ LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
GOOS ?= $(shell go env GOOS)
GOARCH ?= $(shell go env GOARCH)
GOLANGCI_VERSION ?= v2.13.1
GOLANGCI_VERSION ?= v2.15.0
VERSION ?= $(shell git describe --tags)
REPO ?= docker.io/pgsty
@@ -216,7 +216,7 @@ docker: checks build-debugging ## builds the local Linux Silo container image
--ldflags "$(LDFLAGS)" -o "$$context/silo"; \
mkdir -p "$$context/dockerscripts"; \
cp Dockerfile.goreleaser LICENSE NOTICE CREDITS "$$context/"; \
cp dockerscripts/docker-entrypoint.sh dockerscripts/download-static-curl.sh \
cp dockerscripts/docker-entrypoint.sh dockerscripts/build-static-curl.sh \
"$$context/dockerscripts/"; \
docker build -q --no-cache --platform linux/$(GOARCH) -t $(TAG) --build-arg TARGETARCH=$(GOARCH) \
-f "$$context/Dockerfile.goreleaser" "$$context"
+65 -64
View File
@@ -35,6 +35,20 @@
> [!NOTE]
> Renamed from `pgsty/minio` to `pgsty/silo`, default branch `master` → `main`, on 2026-08-06. Artifacts under the original MinIO identity stay published on the archived [`minio`](https://github.com/pgsty/silo/tree/minio) branch and in releases up to [`RELEASE.2026-08-04T00-00-00Z`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z).
## SILO 20261018000000
[SILO 20261018000000](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-10-18T00-00-00Z) is available as platform archives, Linux packages, and standard or distroless Docker images for amd64 and arm64. The release tag is `RELEASE.2026-10-18T00-00-00Z`; the package version is `20261018000000.0.0`.
This release strengthens Object Lock and IAM authorization, makes scoped STS revocation durable across processes and sites, protects concurrent updates during pool migration, and improves LIST recovery, multipart cleanup, and S3 error handling. It includes Console v2.4.2 and silo-pkg v3.14.3, and is built with Go 1.27.2. Use mcli 20261018 for administration.
Read the [release notes](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-10-18T00-00-00Z) before upgrading, especially the removal of strict multipart listing, coordinated server upgrades, and measured performance costs. See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the people behind the code, reports, and documentation.
Current source is unreleased: Go 1.27.3, silo-pkg v3.15.1, mcli 20261025,
upstream minio-go `ab93db7ba435`, and the Console 2.5.0 candidate `258717be077b`.
It includes the Server IAM cache repair needed by Console's **Reset to default**
action. See [CHANGELOG.md](CHANGELOG.md#unreleased); the published artifacts above
retain their original component versions.
## Overview
PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
@@ -54,7 +68,7 @@ docker run -d --name silo -p 9000:9000 -p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=change-me-long-password \
-v "$PWD/data:/data" \
docker.io/pgsty/silo:latest server /data --console-address ":9001"
docker.io/pgsty/silo:RELEASE.2026-10-18T00-00-00Z server /data --console-address ":9001"
```
<p align="center">
@@ -85,80 +99,67 @@ Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub
## Compatibility
The S3 API, `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, the `github.com/minio/*` import paths, and the on-disk format (including `.minio.sys`) are preserved and held in place by a CI compatibility check. Only Silo-owned delivery surfaces change: the `silo` executable, package, service, Helm chart, and container image — no `minio` binary alias is installed.
Silo preserves S3 and storage-format compatibility, including existing `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, and `.minio.sys` data. CI guards selected compatibility identifiers; release notes document intentional security and behavior changes. Silo-owned delivery surfaces use the `silo` executable, package, service, Helm chart, and container image; no `minio` server binary alias is installed.
The supported release stack is `pgsty/silo` + `pgsty/silo-console` + `pgsty/mc` + `pgsty/silo-pkg`; compatibility with unmodified upstream MinIO/MC is best effort. The Server, Console, and client retain their historical module paths where needed, while maintained code imports `github.com/pgsty/silo-pkg/v3` directly. The SDK `github.com/minio/minio-go/v7` is an explicit upstream dependency. See the current [go.mod](go.mod) for versions and replacements.
Every divergence from upstream is listed in the code-verified [compatibility audit](https://silo.pgsty.com/compatibility/server/). Treat each release as a downstream upgrade: pin versions, read the [release notes](https://silo.pgsty.com/tags/silo/), and keep a rollback path.
### TLS and Go upgrades
Server 20260916 restored Go's default key exchange; Server 20261018 retains it with Go 1.27.2.
TLS key exchange follows Go's defaults across the S3 listener, node links,
replication, identity providers, etcd, and external HTTP services. If an endpoint
cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges
for the process; certificate verification remains enabled. This option does not
disable ML-DSA signatures or resolve every TLS reset. Prefer updating the
incompatible endpoint before removing the temporary setting.
If only the new SecP hybrids cause problems, `GODEBUG=tlssecpmlkem=0` disables
those groups while retaining X25519MLKEM768.
For builds targeting Go 1.27, setting either `SSL_CERT_FILE` or `SSL_CERT_DIR`
on macOS replaces Keychain trust with on-disk roots and Go's verifier. Stale or
incomplete CA paths can break previously trusted connections; unset inherited
values to restore Keychain trust. Explicit certificates in the configured `CAs`
directory remain additive to the selected root pool.
Go 1.27 binaries require macOS 13 or later. See the
[Go release notes](https://go.dev/doc/go1.27) and the
[Go 1.27 TLS and OIDC discovery guide](https://silo.pgsty.com/blog/design/go127-tls-oidc-discovery/).
## Documentation ownership
User documentation is maintained at [silo.pgsty.com](https://silo.pgsty.com/docs/),
with source in [pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com).
The remaining `docs/` tree contains inherited references, examples, and tooling
fixtures. Investigation logs, AI work records, and temporary reports are kept
outside this repository; reusable findings belong in the companion site.
See [AGENTS.md](AGENTS.md) for repository ownership and maintenance rules.
## Security & Contributing
Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); every fix ships with a public [advisory](https://silo.pgsty.com/blog/security/). Contributions are accepted inbound=outbound under AGPL-3.0-or-later with no CLA — only DCO sign-off (`git commit -s`) is required; see [`CONTRIBUTING.md`](CONTRIBUTING.md).
## Contributors
The cards highlight community changes merged into `main`; the avatar wall includes every other
community member who has opened an issue or pull request.
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
<table>
<tr>
<td align="center" width="150">
<a href="https://github.com/h5vx"><img src="https://github.com/h5vx.png?size=100" width="72" alt="h5vx"><br><sub><b>@h5vx</b></sub></a><br><sub>Per-bucket CORS · <a href="https://github.com/pgsty/silo/pull/71">#71</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/ycjlin"><img src="https://github.com/ycjlin.png?size=100" width="72" alt="ycjlin"><br><sub><b>@ycjlin</b></sub></a><br><sub>Missing-bucket listing · <a href="https://github.com/pgsty/silo/pull/37">#37</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/Dansyuqri"><img src="https://github.com/Dansyuqri.png?size=100" width="72" alt="Dansyuqri"><br><sub><b>@Dansyuqri</b></sub></a><br><sub>Multipart checksum type · <a href="https://github.com/pgsty/silo/pull/57">#57</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/pinginfo"><img src="https://github.com/pinginfo.png?size=100" width="72" alt="pinginfo"><br><sub><b>@pinginfo</b></sub></a><br><sub>Notification streaming · <a href="https://github.com/pgsty/silo/pull/34">#34</a></sub>
</td>
</tr>
<tr>
<td align="center" width="150">
<a href="https://github.com/ZouhairCharef"><img src="https://github.com/ZouhairCharef.png?size=100" width="72" alt="ZouhairCharef"><br><sub><b>@ZouhairCharef</b></sub></a><br><sub>CVE-2026-34986 · <a href="https://github.com/pgsty/silo/pull/18">#18</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/mfredenhagen"><img src="https://github.com/mfredenhagen.png?size=100" width="72" alt="mfredenhagen"><br><sub><b>@mfredenhagen</b></sub></a><br><sub>CVE-2026-39883 · <a href="https://github.com/pgsty/silo/pull/19">#19</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/waterkip"><img src="https://github.com/waterkip.png?size=100" width="72" alt="waterkip"><br><sub><b>@waterkip</b></sub></a><br><sub>Documentation links · <a href="https://github.com/pgsty/silo/pull/41">#41</a></sub>
</td>
</tr>
</table>
<p>
<a href="https://github.com/magicxor"><img src="https://github.com/magicxor.png?size=64" width="44" alt="magicxor" title="@magicxor"></a>
<a href="https://github.com/davinkevin"><img src="https://github.com/davinkevin.png?size=64" width="44" alt="davinkevin" title="@davinkevin"></a>
<a href="https://github.com/lem21h"><img src="https://github.com/lem21h.png?size=64" width="44" alt="lem21h" title="@lem21h"></a>
<a href="https://github.com/sulin37392"><img src="https://github.com/sulin37392.png?size=64" width="44" alt="sulin37392" title="@sulin37392"></a>
<a href="https://github.com/metaneutrons"><img src="https://github.com/metaneutrons.png?size=64" width="44" alt="metaneutrons" title="@metaneutrons"></a>
<a href="https://github.com/mrjavadseydi"><img src="https://github.com/mrjavadseydi.png?size=64" width="44" alt="mrjavadseydi" title="@mrjavadseydi"></a>
<a href="https://github.com/mosesdd"><img src="https://github.com/mosesdd.png?size=64" width="44" alt="mosesdd" title="@mosesdd"></a>
<a href="https://github.com/Xavier-777"><img src="https://github.com/Xavier-777.png?size=64" width="44" alt="Xavier-777" title="@Xavier-777"></a>
<a href="https://github.com/jiadzh"><img src="https://github.com/jiadzh.png?size=64" width="44" alt="jiadzh" title="@jiadzh"></a>
<a href="https://github.com/TLINDEN"><img src="https://github.com/TLINDEN.png?size=64" width="44" alt="TLINDEN" title="@TLINDEN"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://github.com/AntonOfTheWoods.png?size=64" width="44" alt="AntonOfTheWoods" title="@AntonOfTheWoods"></a>
<a href="https://github.com/zylpsrs"><img src="https://github.com/zylpsrs.png?size=64" width="44" alt="zylpsrs" title="@zylpsrs"></a>
<a href="https://github.com/nsanitate"><img src="https://github.com/nsanitate.png?size=64" width="44" alt="nsanitate" title="@nsanitate"></a>
<a href="https://github.com/makinikm"><img src="https://github.com/makinikm.png?size=64" width="44" alt="makinikm" title="@makinikm"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://github.com/spaceg00se-r.png?size=64" width="44" alt="spaceg00se-r" title="@spaceg00se-r"></a>
<a href="https://github.com/heroes1412"><img src="https://github.com/heroes1412.png?size=64" width="44" alt="heroes1412" title="@heroes1412"></a>
<a href="https://github.com/vampywiz17"><img src="https://github.com/vampywiz17.png?size=64" width="44" alt="vampywiz17" title="@vampywiz17"></a>
<a href="https://github.com/chalukyaj"><img src="https://github.com/chalukyaj.png?size=64" width="44" alt="chalukyaj" title="@chalukyaj"></a>
<a href="https://github.com/cbornet"><img src="https://github.com/cbornet.png?size=64" width="44" alt="cbornet" title="@cbornet"></a>
<a href="https://github.com/jvasile"><img src="https://github.com/jvasile.png?size=64" width="44" alt="jvasile" title="@jvasile"></a>
<a href="https://github.com/Kesavaambati"><img src="https://github.com/Kesavaambati.png?size=64" width="44" alt="Kesavaambati" title="@Kesavaambati"></a>
<a href="https://github.com/redfoxfox"><img src="https://github.com/redfoxfox.png?size=64" width="44" alt="redfoxfox" title="@redfoxfox"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://github.com/kuldeep-link11.png?size=64" width="44" alt="kuldeep-link11" title="@kuldeep-link11"></a>
<a href="https://github.com/meesudzu"><img src="https://github.com/meesudzu.png?size=64" width="44" alt="meesudzu" title="@meesudzu"></a>
<a href="https://github.com/pmezhuev"><img src="https://github.com/pmezhuev.png?size=64" width="44" alt="pmezhuev" title="@pmezhuev"></a>
<a href="https://github.com/kh0mka"><img src="https://github.com/kh0mka.png?size=64" width="44" alt="kh0mka" title="@kh0mka"></a>
<a href="https://github.com/bagutzu"><img src="https://github.com/bagutzu.png?size=64" width="44" alt="bagutzu" title="@bagutzu"></a>
<a href="https://github.com/liuhaodongliu990-cmyk"><img src="https://github.com/liuhaodongliu990-cmyk.png?size=64" width="44" alt="liuhaodongliu990-cmyk" title="@liuhaodongliu990-cmyk"></a>
<a href="https://github.com/sargarass"><img src="https://github.com/sargarass.png?size=64" width="44" alt="sargarass" title="@sargarass"></a>
<a href="https://github.com/mumu-lab"><img src="https://github.com/mumu-lab.png?size=64" width="44" alt="mumu-lab" title="@mumu-lab"></a>
</p>
Every human issue or pull-request author is part of the SILO community, including open and unmerged work. We also recognize security reports, including reports sent by email, with the reporter's permission. Merged fixes, adopted proposals, and actionable reports receive priority, with first participation guiding the remaining order. Gold rings highlight reviewed significant contributions.
GitHub does not generate a contributor graph for forks, so [`CONTRIBUTORS.md`](CONTRIBUTORS.md) — not the Insights page — is this project's attribution record. It names all 37 community contributors alongside the change or report they contributed.
<a href="CONTRIBUTORS.md">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-light.svg" alt="SILO community contributors">
</picture>
</a>
[View contribution notes and actual PR status](CONTRIBUTORS.md).
## Star History
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-light.svg" alt="SILO GitHub star history">
</picture>
## Background
+47 -63
View File
@@ -35,6 +35,19 @@
> [!NOTE]
> 2026-08-06,本仓库由 `pgsty/minio` 更名为 `pgsty/silo`,默认分支由 `master` 更名为 `main`。以原 MinIO 形态维持的归档构件仍位于归档的 [`minio`](https://github.com/pgsty/silo/tree/minio) 分支,以及截止 [`RELEASE.2026-08-04T00-00-00Z`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z) 的历次发布中。
## SILO 20261018000000
[SILO 20261018000000](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-10-18T00-00-00Z) 已正式发布,提供各平台归档包、Linux 软件包,以及 amd64/arm64 标准版和 distroless Docker 镜像。版本标签为 `RELEASE.2026-10-18T00-00-00Z`,包版本为 `20261018000000.0.0`。
本版加强 Object Lock 与 IAM 鉴权,实现跨进程和站点的持久 STS 撤销,保护多池迁移中的并发更新,并改进 LIST 恢复、分片清理与 S3 错误处理。内置 Console v2.4.2、silo-pkg v3.14.3,使用 Go 1.27.2 构建,配套管理客户端为 mcli 20261018。
升级前请阅读[发行说明](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-10-18T00-00-00Z),特别是 strict 分片列表的移除、服务端协调升级要求与实测性能成本。代码、问题报告和文档贡献者见 [CONTRIBUTORS.md](CONTRIBUTORS.md)。
当前源码尚未发布:使用 Go 1.27.3、silo-pkg v3.15.1、mcli 20261025、上游
minio-go `ab93db7ba435`,并内嵌 Console 2.5.0 候选提交 `258717be077b`。
其中包含 Console **Reset to default(恢复默认)** 所需的 Server IAM 缓存修复。
详见 [CHANGELOG.md](CHANGELOG.md#unreleased);上述已发布制品仍保留原有组件版本。
## 概述
上游停止社区发行后,Silo 为开源 MinIO 服务端维护一条持续可用的版本线:构建、软件包、多架构镜像、安全修复与完整 Web 控制台。Pigsty 在生产环境中用它承载 PostgreSQL 备份存储。
@@ -54,7 +67,7 @@ docker run -d --name silo -p 9000:9000 -p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=change-me-long-password \
-v "$PWD/data:/data" \
docker.io/pgsty/silo:latest server /data --console-address ":9001"
docker.io/pgsty/silo:RELEASE.2026-10-18T00-00-00Z server /data --console-address ":9001"
```
<p align="center">
@@ -85,79 +98,50 @@ docker exec silo mcli mb local/demo && docker exec silo mcli ls local
## 兼容性
S3 API、`MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*` 路由、`github.com/minio/*` 导入路径与磁盘格式(含 `.minio.sys`)原样保留,并由 CI 兼容性门禁冻结。只有 Silo 自有交付面改名:`silo` 可执行文件、软件包、服务、Helm Chart 与容器镜像 —— 原生交付物不会安装 `minio` 二进制别名。
Silo 保留 S3 与存储格式兼容性,包括既有 `MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*` 路由与 `.minio.sys` 数据。CI 守卫检查选定的兼容性标识,有意的安全与行为变化在发布说明中记录。Silo 自有交付面使用 `silo` 可执行文件、软件包、服务、Helm Chart 与容器镜像;原生交付物不会安装 `minio` 服务端二进制别名。
正式支持和发布验收的组合为 `pgsty/silo` + `pgsty/silo-console` + `pgsty/mc` + `pgsty/silo-pkg`,对未修改的上游 MinIO/MC 尽最大努力保持兼容。Server、Console 与客户端按需保留历史模块路径,维护源码直接导入 `github.com/pgsty/silo-pkg/v3`;SDK `github.com/minio/minio-go/v7` 是明确保留的上游依赖。具体版本与 replace 以当前 [go.mod](go.mod) 为准。
与上游的全部分歧,以逐项核验代码的[兼容性审计](https://silo.pgsty.com/zh/compatibility/server/)形式维护。每个版本仍应视为下游升级:锁定版本,阅读[版本说明](https://silo.pgsty.com/zh/tags/silo/),并保留回滚路径。
### TLS 与 Go 升级
Server 20260916 已恢复 Go 默认密钥交换策略,20261018 在 Go 1.27.2 下保留该行为。
`GODEBUG=tlsmlkem=0`、`tlssecpmlkem=0` 的适用范围、macOS 根证书来源变化,
以及 OIDC discovery 的诊断方法见 [Go 1.27 TLS 与 OIDC 指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
## 文档归属
用户文档统一维护在 [silo.pgsty.com](https://silo.pgsty.com/zh/docs/),源码位于
[pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com)。本仓库保留的 `docs/`
主要是继承的参考材料、示例和工具测试夹具。调查日志、AI 工作记录与临时报告放在仓库外;
可复用的结论应整理进伴生文档站。仓库职责与维护规则见 [AGENTS.md](AGENTS.md)。
## 安全与贡献
请按照 [`SECURITY.md`](SECURITY.md) 私密报告漏洞;每项修复都会发布公开[安全公告](https://silo.pgsty.com/zh/blog/security/)。本项目不要求签署 CLA:贡献按 AGPL-3.0-or-later(inbound=outbound)接收,只需 DCO 签署(`git commit -s`),详见 [`CONTRIBUTING.md`](CONTRIBUTING.md)。
## 贡献者
上方卡片展示已合入 `main` 的社区贡献;下方头像墙收录其他所有提交过 issue 或 PR 的社区成员。
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
<table>
<tr>
<td align="center" width="150">
<a href="https://github.com/h5vx"><img src="https://github.com/h5vx.png?size=100" width="72" alt="h5vx"><br><sub><b>@h5vx</b></sub></a><br><sub>单桶 CORS · <a href="https://github.com/pgsty/silo/pull/71">#71</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/ycjlin"><img src="https://github.com/ycjlin.png?size=100" width="72" alt="ycjlin"><br><sub><b>@ycjlin</b></sub></a><br><sub>缺失桶列举语义 · <a href="https://github.com/pgsty/silo/pull/37">#37</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/Dansyuqri"><img src="https://github.com/Dansyuqri.png?size=100" width="72" alt="Dansyuqri"><br><sub><b>@Dansyuqri</b></sub></a><br><sub>分片上传校验类型 · <a href="https://github.com/pgsty/silo/pull/57">#57</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/pinginfo"><img src="https://github.com/pinginfo.png?size=100" width="72" alt="pinginfo"><br><sub><b>@pinginfo</b></sub></a><br><sub>桶通知流式输出 · <a href="https://github.com/pgsty/silo/pull/34">#34</a></sub>
</td>
</tr>
<tr>
<td align="center" width="150">
<a href="https://github.com/ZouhairCharef"><img src="https://github.com/ZouhairCharef.png?size=100" width="72" alt="ZouhairCharef"><br><sub><b>@ZouhairCharef</b></sub></a><br><sub>CVE-2026-34986 · <a href="https://github.com/pgsty/silo/pull/18">#18</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/mfredenhagen"><img src="https://github.com/mfredenhagen.png?size=100" width="72" alt="mfredenhagen"><br><sub><b>@mfredenhagen</b></sub></a><br><sub>CVE-2026-39883 · <a href="https://github.com/pgsty/silo/pull/19">#19</a></sub>
</td>
<td align="center" width="150">
<a href="https://github.com/waterkip"><img src="https://github.com/waterkip.png?size=100" width="72" alt="waterkip"><br><sub><b>@waterkip</b></sub></a><br><sub>文档链接修正 · <a href="https://github.com/pgsty/silo/pull/41">#41</a></sub>
</td>
</tr>
</table>
<p>
<a href="https://github.com/magicxor"><img src="https://github.com/magicxor.png?size=64" width="44" alt="magicxor" title="@magicxor"></a>
<a href="https://github.com/davinkevin"><img src="https://github.com/davinkevin.png?size=64" width="44" alt="davinkevin" title="@davinkevin"></a>
<a href="https://github.com/lem21h"><img src="https://github.com/lem21h.png?size=64" width="44" alt="lem21h" title="@lem21h"></a>
<a href="https://github.com/sulin37392"><img src="https://github.com/sulin37392.png?size=64" width="44" alt="sulin37392" title="@sulin37392"></a>
<a href="https://github.com/metaneutrons"><img src="https://github.com/metaneutrons.png?size=64" width="44" alt="metaneutrons" title="@metaneutrons"></a>
<a href="https://github.com/mrjavadseydi"><img src="https://github.com/mrjavadseydi.png?size=64" width="44" alt="mrjavadseydi" title="@mrjavadseydi"></a>
<a href="https://github.com/mosesdd"><img src="https://github.com/mosesdd.png?size=64" width="44" alt="mosesdd" title="@mosesdd"></a>
<a href="https://github.com/Xavier-777"><img src="https://github.com/Xavier-777.png?size=64" width="44" alt="Xavier-777" title="@Xavier-777"></a>
<a href="https://github.com/jiadzh"><img src="https://github.com/jiadzh.png?size=64" width="44" alt="jiadzh" title="@jiadzh"></a>
<a href="https://github.com/TLINDEN"><img src="https://github.com/TLINDEN.png?size=64" width="44" alt="TLINDEN" title="@TLINDEN"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://github.com/AntonOfTheWoods.png?size=64" width="44" alt="AntonOfTheWoods" title="@AntonOfTheWoods"></a>
<a href="https://github.com/zylpsrs"><img src="https://github.com/zylpsrs.png?size=64" width="44" alt="zylpsrs" title="@zylpsrs"></a>
<a href="https://github.com/nsanitate"><img src="https://github.com/nsanitate.png?size=64" width="44" alt="nsanitate" title="@nsanitate"></a>
<a href="https://github.com/makinikm"><img src="https://github.com/makinikm.png?size=64" width="44" alt="makinikm" title="@makinikm"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://github.com/spaceg00se-r.png?size=64" width="44" alt="spaceg00se-r" title="@spaceg00se-r"></a>
<a href="https://github.com/heroes1412"><img src="https://github.com/heroes1412.png?size=64" width="44" alt="heroes1412" title="@heroes1412"></a>
<a href="https://github.com/vampywiz17"><img src="https://github.com/vampywiz17.png?size=64" width="44" alt="vampywiz17" title="@vampywiz17"></a>
<a href="https://github.com/chalukyaj"><img src="https://github.com/chalukyaj.png?size=64" width="44" alt="chalukyaj" title="@chalukyaj"></a>
<a href="https://github.com/cbornet"><img src="https://github.com/cbornet.png?size=64" width="44" alt="cbornet" title="@cbornet"></a>
<a href="https://github.com/jvasile"><img src="https://github.com/jvasile.png?size=64" width="44" alt="jvasile" title="@jvasile"></a>
<a href="https://github.com/Kesavaambati"><img src="https://github.com/Kesavaambati.png?size=64" width="44" alt="Kesavaambati" title="@Kesavaambati"></a>
<a href="https://github.com/redfoxfox"><img src="https://github.com/redfoxfox.png?size=64" width="44" alt="redfoxfox" title="@redfoxfox"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://github.com/kuldeep-link11.png?size=64" width="44" alt="kuldeep-link11" title="@kuldeep-link11"></a>
<a href="https://github.com/meesudzu"><img src="https://github.com/meesudzu.png?size=64" width="44" alt="meesudzu" title="@meesudzu"></a>
<a href="https://github.com/pmezhuev"><img src="https://github.com/pmezhuev.png?size=64" width="44" alt="pmezhuev" title="@pmezhuev"></a>
<a href="https://github.com/kh0mka"><img src="https://github.com/kh0mka.png?size=64" width="44" alt="kh0mka" title="@kh0mka"></a>
<a href="https://github.com/bagutzu"><img src="https://github.com/bagutzu.png?size=64" width="44" alt="bagutzu" title="@bagutzu"></a>
<a href="https://github.com/liuhaodongliu990-cmyk"><img src="https://github.com/liuhaodongliu990-cmyk.png?size=64" width="44" alt="liuhaodongliu990-cmyk" title="@liuhaodongliu990-cmyk"></a>
<a href="https://github.com/sargarass"><img src="https://github.com/sargarass.png?size=64" width="44" alt="sargarass" title="@sargarass"></a>
<a href="https://github.com/mumu-lab"><img src="https://github.com/mumu-lab.png?size=64" width="44" alt="mumu-lab" title="@mumu-lab"></a>
</p>
每位 issue 或 PR 的作者,以及同意公开署名的安全报告者,都是 SILO 社区的一员,包括尚未合并的工作。已合并的修复、被采纳的方案和有效报告优先展示,其余参考首次参与时间;金色圆环突出经过审核的显著贡献。
GitHub 不为 fork 仓库生成贡献者图表,因此 [`CONTRIBUTORS.md`](CONTRIBUTORS.md)(而非 Insights 页面)才是本项目的署名记录,其中逐一记录了全部 37 位社区贡献者对应的改动或报告。
<a href="CONTRIBUTORS.md">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-light.svg" alt="SILO 社区贡献者">
</picture>
</a>
[查看贡献记录与实际 PR 状态](CONTRIBUTORS.md)。
## Star History
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-light.svg" alt="SILO GitHub 星标历史">
</picture>
## 背景
+3 -3
View File
@@ -7,7 +7,7 @@ Silo-specific fixes or release notes.
## Supported Versions
Security fixes are tracked on the active development branch and summarized in
[docs/security/advisories.md](docs/security/advisories.md). Only the current
[the security advisory ledger](https://silo.pgsty.com/about/security-advisories/). Only the current
Silo release line is supported unless an advisory says otherwise.
## Inherited Fix Evidence
@@ -26,7 +26,7 @@ separately even when the fork preserves the original commit object and SHA.
The inherited [service-account](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/admin-handlers-users_test.go#L211-L212)
and [STS](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/sts-handlers_test.go#L45-L46)
regression groups remain part of `go test ./cmd`; see the
[canonical ledger](docs/security/advisories.md#inherited-upstream-advisory-baseline)
[canonical ledger](https://silo.pgsty.com/about/security-advisories/#inherited)
for the operator-facing record.
## Reporting a Vulnerability
@@ -42,4 +42,4 @@ For vulnerabilities in this fork:
## Disclosure Process
Fork-specific fixes and user-visible upgrade notes are published in [docs/security/advisories.md](docs/security/advisories.md). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
Fork-specific fixes and user-visible upgrade notes are published in [the security advisory ledger](https://silo.pgsty.com/about/security-advisories/). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
+1 -1
View File
@@ -34,4 +34,4 @@ Based on the report, the Silo maintainers investigate:
If the vulnerability exists in this fork itself, the maintainers will, when
feasible, fix the issue or implement reasonable countermeasures such that the
vulnerability can no longer be exploited. Fork-specific upgrade notes and
security advisories are published in `docs/security/advisories.md`.
security advisories are published in [the security advisory ledger](https://silo.pgsty.com/about/security-advisories/).
+1 -1
View File
@@ -7,7 +7,7 @@ _init() {
## Minimum required versions for build dependencies
GIT_VERSION="1.0"
GO_VERSION="1.27.1"
GO_VERSION="1.27.3"
OSX_VERSION="10.8"
KNAME=$(uname -s)
ARCH=$(uname -m)
+1 -1
View File
@@ -40,7 +40,7 @@ if [ -n "${MCLI_BIN:-}" ]; then
exit 0
fi
release=${MCLI_RELEASE:-RELEASE.2026-09-03T07-13-05Z}
release=${MCLI_RELEASE:-RELEASE.2026-10-18T00-00-00Z}
version_hyphen=${release#RELEASE.}
package_version=$(printf '%s\n' "${version_hyphen}" | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/\1\2\3\4\5\6.0.0/')
if [ "${package_version}" = "${version_hyphen}" ]; then
@@ -132,8 +132,8 @@
"MINIO_API_DELETE_CLEANUP_INTERVAL",
"MINIO_API_DISABLE_ODIRECT",
"MINIO_API_GZIP_OBJECTS",
"MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH",
"MINIO_API_LIST_QUORUM",
"MINIO_API_MULTIPART_LISTING",
"MINIO_API_OBJECT_MAX_VERSIONS",
"MINIO_API_ODIRECT",
"MINIO_API_REMOTE_TRANSPORT_DEADLINE",
@@ -502,6 +502,7 @@
"MINIO_SITE_COMMENT",
"MINIO_SITE_NAME",
"MINIO_SITE_REGION",
"MINIO_SITE_REPLICATION_METADATA_TOMBSTONES",
"MINIO_STORAGE_CLASS_COMMENT",
"MINIO_STORAGE_CLASS_INLINE_BLOCK",
"MINIO_STORAGE_CLASS_OPTIMIZE",
@@ -570,6 +571,14 @@
"minio_resource_stats",
"minio_s3",
"minio_stats",
"minio_system_cpu_avg_idle",
"minio_system_cpu_avg_iowait",
"minio_system_cpu_load",
"minio_system_cpu_load_perc",
"minio_system_cpu_nice",
"minio_system_cpu_steal",
"minio_system_cpu_system",
"minio_system_cpu_user",
"minio_test"
],
"headers": [
@@ -594,6 +603,7 @@
"x-minio-internal-encrypted-multipart",
"x-minio-internal-encryptedmultipart",
"x-minio-internal-erasure-upgraded",
"x-minio-internal-ilm-atier",
"x-minio-internal-inline-data",
"x-minio-internal-objectlock-legalhold-timestamp",
"x-minio-internal-replica-status",
@@ -616,6 +626,7 @@
"x-minio-internal-transitioned-object",
"x-minio-internal-xyz",
"x-minio-key",
"x-minio-last-modified",
"x-minio-lifecycleconfig-updatedat",
"x-minio-meta",
"x-minio-meta-appid",
@@ -624,7 +635,6 @@
"x-minio-origin-endpoint",
"x-minio-prefixes-total",
"x-minio-read-quorum",
"x-minio-replication",
"x-minio-replication-actual-object-size",
"x-minio-replication-delete-status",
"x-minio-replication-deletemarker-status",
@@ -763,6 +773,7 @@
"/metrics/v3",
"/minio/grid/",
"/minio/grid/lock/",
"/multipart-preflight",
"/netperf",
"/notification",
"/oauth2/callback",
@@ -826,6 +837,7 @@
"/site-replication/peer/bucket-ops",
"/site-replication/peer/edit",
"/site-replication/peer/iam-item",
"/site-replication/peer/iam-revisions",
"/site-replication/peer/idp-settings",
"/site-replication/peer/join",
"/site-replication/peer/remove",
@@ -874,6 +886,7 @@
"/v2/metrics/cluster",
"/v2/metrics/node",
"/v2/metrics/resource",
"/v3/site-replication/peer/iam-revisions",
"/var/vcap/bosh",
"/verifybinary",
"/version",
@@ -901,6 +914,7 @@
".minio.sys/config/config.json",
".minio.sys/config/hello.txt",
".minio.sys/config/iam/${username}/identity.json",
".minio.sys/config/ilm/access",
".minio.sys/format.json",
".minio.sys/multipart",
".minio.sys/multipart/bucket/object/uploads.json",
@@ -928,6 +942,7 @@
"arn:minio:kms:::this-is-disregarded",
"arn:minio:kms:::xyz-test-key",
"arn:minio:replication:",
"arn:minio:replication::",
"arn:minio:replication::8320b6d18f9032b4700f1f03b50d8d1853de8f22cab86931ee794e12f190852c:destinationbucket",
"arn:minio:replication:::",
"arn:minio:replication:::dest-bucket",
@@ -1059,6 +1074,7 @@
"cmd/metrics.go=\"Version of current MinIO server instance\"",
"cmd/metrics.go=\"minio\"",
"cmd/object-api-utils.go=\".minio.sys\"",
"cmd/object-handlers-common.go=\"X-Minio-Last-Modified\"",
"cmd/object-handlers.go=\"minio-federated\"",
"cmd/object-handlers.go=\"minio.metadata.\"",
"cmd/object-handlers.go=\"minio.versionId\"",
+1
View File
@@ -115,6 +115,7 @@ func collect(repo string) (manifest, error) {
fset := token.NewFileSet()
for _, rel := range files {
// Migration notes and guard fixtures contain archived identifiers.
if rel == "SILO_REBRANDING_MIGRATION.md" ||
strings.HasPrefix(rel, "buildscripts/rebrand-guard/") ||
strings.HasPrefix(rel, "buildscripts/helm-migration-guard/") {
+1
View File
@@ -0,0 +1 @@
__pycache__/
+163
View File
@@ -0,0 +1,163 @@
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Pure, self-contained SVG rendering for SILO's README cards."""
from datetime import date, timedelta
from html import escape
import math
import xml.etree.ElementTree as ET
themes = {
'light': dict(bg='#ffffff', wash='#f2f7fc', edge='#d9e3ee', ink='#16222e',
muted='#62758a', blue='#1d588c', copper='#b4762e', grid='#e5edf5',
line='#2b6ca3', ring='#dce5ef', field='#f7f9fc', label='#3d4e61'),
'dark': dict(bg='#101923', wash='#152738', edge='#2b3c50', ink='#e8eef6',
muted='#93a3b8', blue='#7fb8e8', copper='#e0a35c', grid='#263749',
line='#5da2dd', ring='#3a4e63', field='#0b1119', label='#b6c2d2'),
}
def read_emblem(path):
emblem = ET.parse(path).getroot()
body = ''.join(ET.tostring(child, encoding='unicode') for child in emblem
if child.tag.rsplit('}', 1)[-1] in ('defs', 'g'))
return '\n'.join(line.rstrip() for line in body.splitlines()).strip()
def txt(x, y, value, size=14, color=None, weight=400, anchor='start', mono=False, spacing=None):
family = 'Menlo,Consolas,monospace' if mono else 'Arial,Helvetica,sans-serif'
extra = f' letter-spacing="{spacing}"' if spacing is not None else ''
return (f'<text x="{x}" y="{y}" font-family="{family}" font-size="{size}" '
f'font-weight="{weight}" fill="{color}" text-anchor="{anchor}"{extra}>'
f'{escape(str(value))}</text>')
def start(height, theme, title, description, emblem_body):
t = themes[theme]
return [f'<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="{height}" '
f'viewBox="0 0 1000 {height}" role="img" aria-labelledby="title desc">',
f'<title id="title">{escape(title)}</title><desc id="desc">{escape(description)}</desc>',
'<defs><linearGradient id="surface" x1="0" y1="1" x2="1" y2="0">'
f'<stop offset="0" stop-color="{t["bg"]}"/>'
f'<stop offset="1" stop-color="{t["wash"]}"/></linearGradient>'
'<linearGradient id="accent" x1="0" y1="0" x2="1" y2="0">'
f'<stop offset="0" stop-color="{t["blue"]}"/>'
f'<stop offset="1" stop-color="{t["copper"]}"/></linearGradient>'
'<linearGradient id="area" x1="0" y1="0" x2="0" y2="1">'
f'<stop offset="0" stop-color="{t["line"]}" stop-opacity=".22"/>'
f'<stop offset="1" stop-color="{t["line"]}" stop-opacity=".015"/>'
'</linearGradient></defs>',
f'<rect x=".75" y=".75" width="998.5" height="{height-1.5}" rx="22" '
f'fill="url(#surface)" stroke="{t["edge"]}" stroke-width="1.5"/>',
f'<svg x="40" y="25" width="25" height="25" viewBox="230 213 570 570">{emblem_body}</svg>']
def heading(parts, t, eyebrow, title, subtitle, value, value_label):
parts.extend([
txt(76, 43, eyebrow, 11, t['muted'], 600, mono=True, spacing=1.7),
txt(40, 94, title, 32, t['ink'], 700),
txt(41, 123, subtitle, 14, t['muted']),
txt(958, 89, f'{value:,}', 45, t['ink'], 700, anchor='end'),
txt(957, 114, value_label, 10, t['muted'], 600, anchor='end', mono=True, spacing=1.5),
f'<path d="M40 146 H960" stroke="{t["edge"]}"/>',
])
def contributors(theme, people, snapshot, emblem_body):
height = 206 + 76 * math.ceil(len(people) / 10)
t = themes[theme]
parts = start(height, theme, f'SILO community — {len(people)} contributors',
f'The existing SILO community roll, including code, proposals and reports across related projects. '
f'Gold rings retain the existing significant-contribution designation. Snapshot {snapshot}.', emblem_body)
heading(parts, t, 'SILO / COMMUNITY', 'Contributors',
'Code, proposals & reports across SILO and related projects', len(people), 'COMMUNITY CONTRIBUTORS')
for row in range(math.ceil(len(people) / 10)):
group = people[row * 10:(row + 1) * 10]
row_width = len(group) * 91
for col, person in enumerate(group):
x = (1000 - row_width) / 2 + col * 91 + 45.5
y = 199 + row * 76
identifier = f'avatar-{row}-{col}'
featured = bool(person.get('featured'))
parts.append(f'<g><title>@{escape(person["handle"])} — {escape(person["what"])}</title>')
parts.append(f'<defs><clipPath id="{identifier}"><circle cx="{x}" cy="{y}" r="29"/></clipPath></defs>')
if featured:
parts.append(f'<circle cx="{x}" cy="{y}" r="34" fill="{t["copper"]}" opacity=".09"/>')
if person.get('avatarDataUrl'):
parts.append(f'<image x="{x-29}" y="{y-29}" width="58" height="58" '
f'clip-path="url(#{identifier})" href="{escape(person["avatarDataUrl"])}"/>')
else:
parts.append(f'<circle cx="{x}" cy="{y}" r="29" fill="{t["ring"]}"/>')
parts.append(txt(x, y + 9, person['handle'][0].upper(), 26, t['ink'], 700, 'middle'))
parts.append(f'<circle cx="{x}" cy="{y}" r="30.5" fill="none" '
f'stroke="{t["copper"] if featured else t["ring"]}" stroke-width="{2 if featured else 1.25}"/></g>')
parts.extend([
f'<path d="M40 {height-42} H960" stroke="{t["edge"]}"/>',
f'<circle cx="47" cy="{height-21}" r="4" fill="none" stroke="{t["copper"]}" stroke-width="1.5"/>',
txt(61, height-17, 'Gold rings mark significant contributions', 12, t['muted']),
txt(959, height-17, f'AS OF {snapshot}', 10, t['muted'], 500, 'end', mono=True, spacing=.6),
'</svg>',
])
return ''.join(parts)
def stars(theme, history, snapshot, emblem_body):
points = history['points']
star_count = points[-1]['stars']
t = themes[theme]
provenance = ('Initial history reconstructed · Daily totals since ' + history['bootstrap']['through'] + ' · UTC'
if history['bootstrap']['reconstructed'] else 'Observed daily star totals · UTC')
parts = start(558, theme, f'SILO star history — {star_count:,} stars',
f'GitHub repository pgsty/silo. {star_count:,} stars as of {snapshot}. ' +
provenance, emblem_body)
heading(parts, t, 'SILO / GITHUB', 'Star History', 'pgsty/silo', star_count, 'GITHUB STARS')
left, right, top, bottom = 76, 958, 177, 440
begin = date.fromisoformat(points[0]['date'])
end = date.fromisoformat(points[-1]['date'])
days = max(1, (end - begin).days)
maximum = max(500, math.ceil(max(p['stars'] for p in points) / 500) * 500)
tick_step = 10 ** max(0, int(math.log10(maximum)))
xy = lambda day, n: (left + (right-left)*(date.fromisoformat(day)-begin).days / days,
bottom-(bottom-top)*n/maximum)
for value in range(0, maximum+1, tick_step):
y = xy(points[0]['date'], value)[1]
parts.append(f'<path d="M{left} {y:.2f} H{right}" stroke="{t["grid"]}" stroke-dasharray="4 6"/>')
parts.append(txt(left-16, round(y+4, 2), f'{value / 1000:g}k' if value >= 1000 else str(value), 12, t['muted'], anchor='end'))
dates = sorted({begin + timedelta(days=round((end-begin).days*i/5)) for i in range(6)})
ticks = [(d.isoformat(), d.strftime('%b %Y') if days > 90 else d.strftime('%b %d')) for d in dates]
for day, label in ticks:
x = xy(day, 0)[0]
parts.append(f'<path d="M{x:.2f} {top} V{bottom}" stroke="{t["grid"]}" stroke-opacity=".65"/>')
anchor = 'start' if day == points[0]['date'] else 'end' if day == snapshot else 'middle'
parts.append(txt(round(x, 2), 466, label, 12, t['muted'], anchor=anchor))
coords = [xy(p['date'], p['stars']) for p in points]
line = 'M' + ' L'.join(f'{x:.2f} {y:.2f}' for x, y in coords)
area = line + f' L{coords[-1][0]:.2f} {bottom} L{left} {bottom} Z'
parts.extend([
f'<path d="{area}" fill="url(#area)"/>',
f'<path d="{line}" fill="none" stroke="url(#accent)" stroke-width="3" '
'stroke-linecap="round" stroke-linejoin="round"/>',
f'<path d="M{left} {bottom} H{right}" stroke="{t["edge"]}"/>',
])
x, y = coords[-1]
parts.extend([
f'<circle cx="{x}" cy="{y}" r="10" fill="{t["copper"]}" opacity=".12"/>',
f'<circle cx="{x}" cy="{y}" r="5" fill="{t["copper"]}" stroke="{t["bg"]}" stroke-width="2"/>',
f'<path d="M40 491 H960" stroke="{t["edge"]}"/>',
txt(40, 516, f'{begin:%b %Y} — {end:%b %Y}'.upper(), 10, t['muted'], 500, mono=True, spacing=.7),
txt(959, 516, f'SNAPSHOT {snapshot}', 10, t['muted'], 500, 'end', mono=True, spacing=.6),
txt(40, 539, provenance, 11, t['muted']),
'</svg>',
])
return ''.join(parts)
@@ -0,0 +1 @@
PyYAML==6.0.3
+178
View File
@@ -0,0 +1,178 @@
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Regression checks for historical accuracy, contributor scope and SVG safety."""
import base64
import json
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
from urllib.error import URLError
import xml.etree.ElementTree as ET
import render
import update
NS = {'s': 'http://www.w3.org/2000/svg'}
PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+a7mgAAAAASUVORK5CYII=')
def person(handle='Alice', group='reports', featured=False):
return {'handle': handle, 'group': group, 'featured': featured,
'what': 'A reviewed contribution', 'firstContribution': '2026-09-01'}
def history():
return {'repository': 'pgsty/silo',
'bootstrap': {'through': '2026-09-15', 'reconstructed': True},
'points': [{'date': '2026-09-14', 'stars': 100}, {'date': '2026-09-15', 'stars': 105}]}
class HistoryTests(unittest.TestCase):
def test_new_day_preserves_old_counts_and_unstars(self):
before = history()
after = update.update_history(before, '2026-09-16', 103)
self.assertEqual(after['points'][:-1], before['points'])
self.assertEqual(after['points'][-1], {'date': '2026-09-16', 'stars': 103})
self.assertEqual(before, history())
def test_same_day_rerun_replaces_instead_of_appending(self):
first = update.update_history(history(), '2026-09-15', 107)
self.assertEqual(len(first['points']), 2)
self.assertEqual(first, update.update_history(first, '2026-09-15', 107))
def test_missing_days_are_not_invented(self):
result = update.update_history(history(), '2026-09-18', 106)
self.assertEqual([p['date'] for p in result['points']], ['2026-09-14', '2026-09-15', '2026-09-18'])
def test_rejects_wrong_repository_and_corrupt_history(self):
cases = []
wrong = history(); wrong['repository'] = 'someone/else'; cases.append(wrong)
duplicate = history(); duplicate['points'].append(duplicate['points'][-1]); cases.append(duplicate)
unordered = history(); unordered['points'].reverse(); cases.append(unordered)
negative = history(); negative['points'][0]['stars'] = -1; cases.append(negative)
future = history(); future['points'][-1]['date'] = '2026-09-20'; cases.append(future)
for case in cases:
with self.subTest(case=case), self.assertRaises(ValueError):
update.update_history(case, '2026-09-16', 100)
def test_first_run_has_no_fabricated_history(self):
result = update.update_history(None, '2026-09-16', 10)
self.assertFalse(result['bootstrap']['reconstructed'])
self.assertEqual(result['points'], [{'date': '2026-09-16', 'stars': 10}])
class ContributorTests(unittest.TestCase):
def test_retries_truncated_json_before_using_it(self):
with patch('update.request', side_effect=[b'{"partial":', b'{"ok":true}']), patch('update.time.sleep'):
self.assertEqual(update.GitHub('').get('repos/pgsty/silo'), {'ok': True})
def test_paginates_past_one_full_page(self):
class API(update.GitHub):
def __init__(self): self.calls = []
def get(self, path):
self.calls.append(path)
return list(range(100)) if 'page=1&' in path else [100]
api = API()
self.assertEqual(len(list(api.issues('pgsty/silo'))), 101)
self.assertIn('state=all', api.calls[0])
self.assertIn('page=2&', api.calls[1])
def test_bots_deduplication_unmerged_work_and_reviewed_credit(self):
def issue(login, kind='issue', user_type='User'):
item = {'user': {'login': login, 'type': user_type, 'avatar_url': ''}, 'created_at': '2026-09-02T00:00:00Z'}
if kind != 'issue': item['pull_request'] = {'merged_at': None if kind == 'open' else '2026-09-03T00:00:00Z'}
return item
class API:
def issues(self, _repo):
return [issue('alice'), issue('Bob', 'open'), issue('Bob', 'merged'),
issue('Carol', 'open'), issue('Copilot'), issue('robot', user_type='Bot')]
curated = {'repositories': ['pgsty/silo', 'pgsty/mc'], 'bots': ['Copilot'],
'people': [person('Alice', featured=True), person('Reporter'), person('Copilot')]}
result = update.collect_people(API(), curated)
self.assertEqual({p['handle'] for p in result}, {'Alice', 'Bob', 'Carol', 'Reporter'})
self.assertEqual(result[0]['handle'], 'Bob')
self.assertEqual(result[1]['handle'], 'Carol')
self.assertTrue(next(p for p in result if p['handle'] == 'Alice')['featured'])
self.assertEqual(next(p for p in result if p['handle'] == 'Bob')['group'], 'code')
self.assertFalse(next(p for p in result if p['handle'] == 'Carol')['featured'])
def test_newer_reviewed_preview_survives_until_site_catches_up(self):
remote = {'updated': '2026-09-16T03:00:00+00:00'}
cached = {'updated': '2026-09-16T04:00:00+00:00'}
self.assertIs(update.select_curated(remote, cached), cached)
newer = {'updated': '2026-09-17T03:00:00+00:00'}
self.assertIs(update.select_curated(newer, cached), newer)
def test_avatar_failure_reuses_raster_cache(self):
previous = {**person(), 'avatarDataUrl': update.raster_data_url(PNG)}
with patch('update.request', side_effect=URLError('unavailable')):
result = update.add_avatars(None, [{**person(), 'avatarUrl': 'https://avatars.githubusercontent.com/u/1'}], [previous])
self.assertEqual(result[0]['avatarDataUrl'], previous['avatarDataUrl'])
with self.assertRaises(ValueError): update.raster_data_url(b'<svg onload="bad()"/>')
with self.assertRaises(ValueError): update.cached_avatar({'avatarDataUrl': 'data:image/svg+xml;base64,PHN2Zy8+'})
def test_fetch_failure_leaves_published_assets_untouched(self):
class API:
def get(self, path):
if path == 'repos/pgsty/silo': return {'full_name': 'pgsty/silo', 'stargazers_count': 106}
raise URLError('roster unavailable')
with tempfile.TemporaryDirectory() as directory:
out = Path(directory)
original = json.dumps(history())
(out / 'history.json').write_text(original)
(out / 'contributors-light.svg').write_text('previous image')
with self.assertRaises(URLError): update.refresh(out, API(), Path('.'))
self.assertEqual((out / 'history.json').read_text(), original)
self.assertEqual((out / 'contributors-light.svg').read_text(), 'previous image')
class RenderTests(unittest.TestCase):
def test_real_emblem_generates_clean_xml(self):
emblem = render.read_emblem(Path(__file__).resolve().parents[2] / '.github/silo.svg')
svg = render.contributors('light', [person()], '2026-09-16', emblem)
ET.fromstring(svg)
self.assertTrue(all(line == line.rstrip() for line in svg.splitlines()))
def test_all_avatars_fit_when_the_roster_grows(self):
people = [{**person(f'person-{i}'), 'avatarDataUrl': update.raster_data_url(PNG)} for i in range(151)]
for theme in ('light', 'dark'):
root = ET.fromstring(render.contributors(theme, people, '2026-09-16', ''))
images = root.findall('.//s:image', NS)
self.assertEqual(len(images), 151)
footer = float(root.attrib['height']) - 42
self.assertTrue(all(float(i.attrib['y']) + float(i.attrib['height']) < footer for i in images))
self.assertTrue(all(i.attrib['href'].startswith('data:image/png;base64,') for i in images))
def test_untrusted_text_is_escaped(self):
data = [{**person(), 'what': '<script>alert("x")</script> & contributions'}]
svg = render.contributors('light', data, '2026-09-16', '')
root = ET.fromstring(svg)
self.assertEqual(root.findall('.//s:script', NS), [])
self.assertIn('&lt;script&gt;', svg)
def test_single_point_and_decreasing_star_history_render(self):
for data in (update.update_history(None, '2026-09-16', 0), update.update_history(history(), '2026-09-16', 90)):
for theme in ('light', 'dark'):
svg = render.stars(theme, data, '2026-09-16', '')
ET.fromstring(svg)
self.assertNotIn('nan', svg.lower())
self.assertNotIn('inf', svg.lower())
if __name__ == '__main__':
unittest.main()
+295
View File
@@ -0,0 +1,295 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Refresh the generated-asset checkout; publishing is handled by the workflow."""
import argparse
import base64
from concurrent.futures import ThreadPoolExecutor
from datetime import date, datetime, timezone
import json
from http.client import IncompleteRead
import os
from pathlib import Path
import re
import sys
import time
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse
from urllib.request import Request, urlopen
import xml.etree.ElementTree as ET
import yaml
import render
REPOSITORY = 'pgsty/silo'
SOURCE = 'repos/pgsty/silo.pgsty.com/contents/data/home/contributors.yaml?ref=main'
GROUPS = ('code', 'proposed', 'reports')
HANDLE = re.compile(r'[A-Za-z0-9][A-Za-z0-9-]{0,38}\Z')
def request(url, token='', limit=8 * 1024 * 1024):
headers = {'User-Agent': 'silo-repository-cards', 'Accept': 'application/vnd.github+json'}
if urlparse(url).netloc == 'api.github.com':
headers['X-GitHub-Api-Version'] = '2022-11-28'
if token:
headers['Authorization'] = f'Bearer {token}'
for attempt in range(3):
try:
with urlopen(Request(url, headers=headers), timeout=25) as response:
data = response.read(limit + 1)
if len(data) > limit:
raise ValueError('Response exceeds the size limit')
expected = response.headers.get('Content-Length')
if expected is not None and len(data) != int(expected):
raise URLError('Incomplete response body')
return data
except HTTPError as exc:
if exc.code < 500 or attempt == 2:
raise
except (URLError, TimeoutError, IncompleteRead):
if attempt == 2:
raise
time.sleep(attempt + 1)
class GitHub:
def __init__(self, token):
self.token = token
def get(self, path):
for attempt in range(3):
try:
return json.loads(request('https://api.github.com/' + path, self.token))
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
if attempt == 2:
raise ValueError(f'Incomplete or invalid GitHub JSON: {path}') from exc
time.sleep(attempt + 1)
def issues(self, repository):
page = 1
while True:
batch = self.get(f'repos/{repository}/issues?state=all&per_page=100&page={page}&sort=created&direction=asc')
if not isinstance(batch, list):
raise ValueError(f'Invalid issues response for {repository}')
yield from batch
if len(batch) < 100:
return
page += 1
def curated_snapshot(data, revision):
updated = str(data['updated'])
datetime.fromisoformat(updated)
repositories = [item['repo'] for item in data['repositories']]
if not repositories or any(not re.fullmatch(r'pgsty/[A-Za-z0-9_.-]+', repo) for repo in repositories):
raise ValueError('Invalid contributor repository scope')
people = []
for group in GROUPS:
for entry in data[group]:
if not HANDLE.fullmatch(entry['handle']):
raise ValueError('Invalid GitHub contributor handle')
people.append({
'handle': entry['handle'], 'group': group,
'featured': bool(entry.get('featured')), 'what': entry['what'],
'firstContribution': str(entry.get('firstContribution', '9999-12-31')),
})
if not people or len({p['handle'].lower() for p in people}) != len(people):
raise ValueError('Empty or duplicate contributor roster')
return {'updated': updated, 'revision': revision, 'repositories': repositories,
'bots': data.get('bots', ['Copilot', 'dependabot[bot]']), 'people': people}
def select_curated(remote, cached):
# The initial, approved preview can contain reviewed credit not published by
# the companion site yet. Keep that newer snapshot until the site catches up.
if cached and datetime.fromisoformat(cached['updated']) > datetime.fromisoformat(remote['updated']):
return cached
return remote
def collect_people(api, curated):
bots = {name.lower() for name in curated['bots']}
people = {p['handle'].lower(): dict(p) for p in curated['people']
if p['handle'].lower() not in bots and not p['handle'].lower().endswith('[bot]')}
order = {p['handle'].lower(): index for index, p in enumerate(curated['people'])}
for repository in curated['repositories']:
print(f'Reading issue and PR authors: {repository}', flush=True)
for issue in api.issues(repository):
user = issue.get('user') or {}
handle = user.get('login', '')
key = handle.lower()
if user.get('type') != 'User' or key in bots or key.endswith('[bot]'):
continue
if not HANDLE.fullmatch(handle):
raise ValueError('Invalid issue author')
pr = issue.get('pull_request')
group = 'code' if pr and pr.get('merged_at') else 'proposed' if pr else 'reports'
first = issue['created_at'][:10]
date.fromisoformat(first)
person = people.setdefault(key, {
'handle': handle, 'group': group, 'featured': False,
'what': 'Contributed an issue or pull request to SILO and related projects',
'firstContribution': first,
})
person['avatarUrl'] = user.get('avatar_url', '')
person['firstContribution'] = min(person['firstContribution'], first)
if GROUPS.index(group) < GROUPS.index(person['group']):
person['group'] = group
if not people:
raise ValueError('No human contributors were collected')
return sorted(people.values(), key=lambda p: (
GROUPS.index(p['group']), not p['featured'],
order.get(p['handle'].lower(), len(order)), p['firstContribution'], p['handle'].lower()))
def raster_data_url(data):
if data.startswith(b'\x89PNG\r\n\x1a\n'):
mime = 'image/png'
elif data.startswith(b'\xff\xd8\xff'):
mime = 'image/jpeg'
elif data.startswith((b'GIF87a', b'GIF89a')):
mime = 'image/gif'
elif data[:4] == b'RIFF' and data[8:12] == b'WEBP':
mime = 'image/webp'
else:
raise ValueError('Avatar is not a raster image')
return f'data:{mime};base64,' + base64.b64encode(data).decode('ascii')
def cached_avatar(person):
value = person.get('avatarDataUrl', '')
if not value:
return ''
prefix, encoded = value.split(',', 1)
if prefix not in ('data:image/png;base64', 'data:image/jpeg;base64', 'data:image/gif;base64', 'data:image/webp;base64'):
raise ValueError('Invalid cached avatar format')
raw = base64.b64decode(encoded, validate=True)
if len(raw) > 512 * 1024 or raster_data_url(raw) != value:
raise ValueError('Invalid cached avatar')
return value
def add_avatars(api, people, previous):
cached = {p['handle'].lower(): cached_avatar(p) for p in previous}
def update(person):
person = dict(person)
try:
url = person.pop('avatarUrl', '') or api.get('users/' + person['handle'])['avatar_url']
parsed = urlparse(url)
if parsed.scheme != 'https' or parsed.netloc != 'avatars.githubusercontent.com':
raise ValueError('Unexpected avatar host')
data = request(url + ('&' if '?' in url else '?') + 's=96', limit=512 * 1024)
person['avatarDataUrl'] = raster_data_url(data)
except (HTTPError, URLError, TimeoutError, IncompleteRead, ValueError, KeyError) as exc:
person.pop('avatarUrl', None)
person['avatarDataUrl'] = cached.get(person['handle'].lower(), '')
print(f'Avatar fallback for @{person["handle"]}: {type(exc).__name__}', file=sys.stderr)
return person
with ThreadPoolExecutor(max_workers=6) as pool:
return list(pool.map(update, people))
def update_history(history, day, stars):
date.fromisoformat(day)
if type(stars) is not int or stars < 0:
raise ValueError('Invalid repository star count')
if history is None:
history = {'repository': REPOSITORY, 'bootstrap': {'through': day, 'reconstructed': False}, 'points': []}
if history['repository'] != REPOSITORY:
raise ValueError('Star history belongs to a different repository')
date.fromisoformat(history['bootstrap']['through'])
dates = []
for point in history['points']:
date.fromisoformat(point['date'])
if type(point['stars']) is not int or point['stars'] < 0:
raise ValueError('Invalid historical star count')
dates.append(point['date'])
if dates != sorted(set(dates)) or any(d > day for d in dates):
raise ValueError('History contains duplicate, unordered, or future dates')
# Replace today's observation, preserve previous days, and allow unstars.
points = [dict(p) for p in history['points'] if p['date'] != day]
points.append({'date': day, 'stars': stars})
return {**history, 'points': points}
def read_json(path, default=None):
return json.loads(path.read_text()) if path.exists() else default
def refresh(output, api, source_root):
day = datetime.now(timezone.utc).date().isoformat()
metadata = api.get('repos/' + REPOSITORY)
if metadata['full_name'].lower() != REPOSITORY:
raise ValueError('Unexpected repository metadata')
history = update_history(read_json(output / 'history.json'), day, metadata['stargazers_count'])
source = api.get(SOURCE)
reviewed = yaml.safe_load(base64.b64decode(source['content'], validate=False))
curated = select_curated(curated_snapshot(reviewed, source['sha']), read_json(output / 'curated.json'))
people = collect_people(api, curated)
previous = read_json(output / 'contributors.json', {}).get('people', [])
people = add_avatars(api, people, previous)
emblem = render.read_emblem(source_root / '.github/silo.svg')
payloads = {}
for theme in ('light', 'dark'):
payloads[f'contributors-{theme}.svg'] = render.contributors(theme, people, day, emblem) + '\n'
payloads[f'star-history-{theme}.svg'] = render.stars(theme, history, day, emblem) + '\n'
for svg in payloads.values():
ET.fromstring(svg)
for name, data in {
'history.json': history,
'curated.json': curated,
'contributors.json': {'repository': REPOSITORY, 'updated': day, 'people': people},
}.items():
payloads[name] = json.dumps(data, indent=2, ensure_ascii=False) + '\n'
payloads['README.md'] = f'''# SILO repository cards
Generated by [Repository Cards](https://github.com/pgsty/silo/actions/workflows/repository-cards.yml)
at 00:00 UTC daily (08:00 Asia/Shanghai). GitHub may queue scheduled runs.
Snapshot: {day}. {metadata['stargazers_count']:,} stars; {len(people)} community contributors.
- `contributors-light.svg` / `contributors-dark.svg`: human issue and PR authors across the SILO project scope, plus reviewed acknowledgements. Bots are excluded. Gold rings follow the reviewed companion-site roster; new authors are collected automatically.
- `star-history-light.svg` / `star-history-dark.svg`: initial history reconstructed from the then-current stargazers; later points are daily observed totals, including decreases. Missing days are not fabricated.
- `curated.json`: a cache of reviewed contributor credit from `pgsty/silo.pgsty.com/data/home/contributors.yaml`. The approved initial preview may be newer than the published site; a newer reviewed snapshot is retained until the site catches up.
- `contributors.json`: generated contributor data and embedded raster avatars. Failed avatar refreshes use the previous image, or an initial when no image is available.
- `history.json`: persistent daily totals. Keep this file when regenerating images.
The SVGs are self-contained. Source and instructions live on the default branch;
this branch contains generated assets only. Do not merge it into `main`.
'''
# Collect and validate everything before touching the publication checkout.
output.mkdir(parents=True, exist_ok=True)
for filename, text in payloads.items():
(output / filename).write_text(text)
print(f'{day}: {len(people)} contributors; {metadata["stargazers_count"]:,} stars; {len(history["points"])} history points')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output', type=Path, required=True)
args = parser.parse_args()
configured = os.environ.get('GITHUB_REPOSITORY', REPOSITORY)
if configured.lower() != REPOSITORY:
raise SystemExit('This workflow is scoped to pgsty/silo')
refresh(args.output, GitHub(os.environ.get('GH_TOKEN', '')), Path(__file__).resolve().parents[2])
if __name__ == '__main__':
main()
+1 -1
View File
@@ -113,7 +113,7 @@ helm_run template my-release "${new_chart}" \
go run ./buildscripts/helm-migration-guard "${old_render}" "${new_render}"
helm_run package "${new_chart}" --destination "${output_dir}" >/dev/null
test -s "${work_dir}/silo-7.0.2.tgz"
test -s "${work_dir}/silo-7.0.3.tgz"
if find "${work_dir}" -maxdepth 1 -type f -name 'minio-*.tgz' | grep -q .; then
echo "Helm packaging emitted a legacy MinIO chart name" >&2
exit 1
+54 -21
View File
@@ -11,6 +11,33 @@ fail() {
exit 1
}
# Git is already a build prerequisite. --no-index also scans untracked files
# and exported source trees. As before, honor .gitignore only in a worktree.
search_ignore=--no-exclude-standard
if git rev-parse --show-toplevel >/dev/null 2>&1; then
search_ignore=--exclude-standard
fi
search_errors="$(mktemp)"
trap 'rm -f "${search_errors}"' EXIT
search_text() {
local status=0
git --no-pager grep --no-index "${search_ignore}" --no-color -nIHE "$@" 2>"${search_errors}" || status=$?
# Git can return 1 for both no matches and a file read error. Check stderr
# as well, before callers filter matches through their allowlists.
if [ "${status}" -gt 1 ] || [ -s "${search_errors}" ]; then
cat "${search_errors}" >&2
fail "could not scan source files"
fi
}
# Keep hidden files and build output out of the repository-wide scans.
source_excludes=(
':(exclude,glob)**/.*'
':(exclude,glob)**/.*/**'
':(exclude)dist/**'
)
require_file() {
[ -f "$1" ] || fail "missing required file: $1"
}
@@ -36,7 +63,7 @@ for file in \
buildscripts/verify-helm-migration.sh \
Dockerfile.goreleaser \
Dockerfile.distroless \
dockerscripts/download-static-curl.sh \
dockerscripts/build-static-curl.sh \
dockerscripts/docker-entrypoint.sh \
helm/silo/Chart.yaml \
helm/silo/values.yaml \
@@ -101,7 +128,7 @@ require_text Dockerfile.goreleaser "Published checksum drift"
require_text Dockerfile.distroless 'COPY --chmod=0755 silo /usr/bin/silo'
require_text Dockerfile.distroless 'ENTRYPOINT ["/usr/bin/silo"]'
require_text Dockerfile.distroless '"/usr/bin/silo", "healthcheck", "ready"'
require_text dockerscripts/download-static-curl.sh "sha256sum -c"
require_text dockerscripts/build-static-curl.sh "sha256sum -c"
require_text helm/silo/Chart.yaml "name: silo"
require_text helm/silo/values.yaml "repository: pgsty/silo"
require_text helm/silo/templates/deployment.yaml "/usr/bin/docker-entrypoint.sh silo server"
@@ -152,21 +179,27 @@ if grep -Ev '^[[:space:]]*(#|$)' silo.env | grep -q '='; then
fail "silo.env must not contain active assignments that shadow /etc/default/minio"
fi
if rg -n 'pgsty/minio:' .github/workflows Dockerfile.goreleaser helm/silo; then
delivery_hits="$(search_text 'pgsty/minio:' -- .github/workflows Dockerfile.goreleaser helm/silo \
':(exclude,glob).github/workflows/**/.*' ':(exclude,glob).github/workflows/**/.*/**' \
':(exclude,glob)helm/silo/**/.*' ':(exclude,glob)helm/silo/**/.*/**')"
if [ -n "${delivery_hits}" ]; then
printf '%s\n' "${delivery_hits}" >&2
fail "an active delivery surface still publishes the frozen pgsty/minio image"
fi
# The repository and its default branch are pgsty/silo and main. The invariant
# is that the old name is never a live target, not that it is never spoken: the
# READMEs have to name it to explain the rename and to point at the archived
# artifacts, which is the opposite of stranding a reader on it.
# artifacts, which is the opposite of stranding a reader on it. CONTRIBUTORS.md
# also quotes historical issue titles.
#
# So two rules. First, no live URL may resolve to the old repository anywhere,
# READMEs included.
stale_repo_url="$(rg -n -e 'github\.com/pgsty/minio' -e 'hub\.docker\.com/r/pgsty/minio' \
--glob '!.git/**' --glob '!dist/**' \
--glob '!SILO_REBRANDING_MIGRATION.md' \
--glob '!buildscripts/rebrand-guard/compat-baseline.json' . |
# READMEs and CONTRIBUTORS.md included.
old_repo_pattern='pgsty/minio(\.git)?([^[:alnum:]_.-]|$)'
stale_repo_url="$(search_text -e "github\.com/${old_repo_pattern}" -e "hub\.docker\.com/r/${old_repo_pattern}" -- . \
"${source_excludes[@]}" ':(exclude)SILO_REBRANDING_MIGRATION.md' \
':(exclude)buildscripts/rebrand-guard/compat-baseline.json')"
stale_repo_url="$(printf '%s\n' "${stale_repo_url}" |
sed 's#^\./##' | grep -v '^buildscripts/verify-rebrand\.sh:' || true)"
if [ -n "${stale_repo_url}" ]; then
printf '%s\n' "${stale_repo_url}" >&2
@@ -175,35 +208,35 @@ fi
# Second, the bare name may only appear where it is deliberate: the pinned
# pre-rebrand image digest in the upgrade test, the two guards that refuse a
# legacy image, and the two READMEs that document the rename and the archived
# minio branch.
repo_guard_allowlist='^(buildscripts/minio-upgrade\.sh|buildscripts/verify-rebrand\.sh|buildscripts/helm-migration-guard/main\.go|README\.md|README_ZH\.md):'
stale_repo="$(rg -n 'pgsty/minio' --glob '!.git/**' --glob '!dist/**' \
--glob '!SILO_REBRANDING_MIGRATION.md' \
--glob '!buildscripts/rebrand-guard/compat-baseline.json' . |
# legacy image, the two READMEs that document the rename and the archived
# minio branch, and historical issue titles in CONTRIBUTORS.md.
repo_guard_allowlist='^(buildscripts/minio-upgrade\.sh|buildscripts/verify-rebrand\.sh|buildscripts/helm-migration-guard/main\.go|README\.md|README_ZH\.md|CONTRIBUTORS\.md):'
stale_repo="$(search_text "${old_repo_pattern}" -- . "${source_excludes[@]}" \
':(exclude)SILO_REBRANDING_MIGRATION.md' ':(exclude)buildscripts/rebrand-guard/compat-baseline.json')"
stale_repo="$(printf '%s\n' "${stale_repo}" |
sed 's#^\./##' | grep -Ev "${repo_guard_allowlist}" || true)"
if [ -n "${stale_repo}" ]; then
printf '%s\n' "${stale_repo}" >&2
fail "a source reference still names the pre-rename pgsty/minio repository"
fi
stale_branch="$(rg -n 'pgsty/silo/(blob/|tree/|raw/)?master' \
--glob '!.git/**' --glob '!dist/**' . || true)"
stale_branch="$(search_text 'pgsty/silo/(blob/|tree/|raw/)?master' -- . "${source_excludes[@]}")"
if [ -n "${stale_branch}" ]; then
printf '%s\n' "${stale_branch}" >&2
fail "a link still targets the retired master branch; raw and Actions URLs do not follow a branch rename"
fi
for workflow in .github/workflows/go.yml .github/workflows/vulncheck.yml; do
if rg -q '^\s+- master$' "${workflow}"; then
stale_filter="$(search_text '^[[:space:]]+- master$' -- "${workflow}")"
if [ -n "${stale_filter}" ]; then
fail "${workflow} still filters on master and would go silently dormant on main"
fi
require_text "${workflow}" " - main"
done
network_hits="$(rg -n --glob '*.go' --glob '!**/*_test.go' \
'https?://[^"`[:space:]]*(dl\.min\.io|subnet\.min\.io|api\.min\.io|slack\.min\.io|play\.min\.io)' \
cmd internal || true)"
network_hits="$(search_text \
'https?://[^"`[:space:]]*(dl\.min\.io|subnet\.min\.io|api\.min\.io|slack\.min\.io|play\.min\.io)' -- \
':(glob)cmd/**/*.go' ':(glob)internal/**/*.go' ':(exclude,glob)**/*_test.go' "${source_excludes[@]}")"
network_hits="$(printf '%s\n' "${network_hits}" | grep -Ev '^[^:]+:[0-9]+:[[:space:]]*//' || true)"
if [ -n "${network_hits}" ]; then
printf '%s\n' "${network_hits}" >&2
+80
View File
@@ -0,0 +1,80 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/cachevalue"
)
func TestAccountInfoReflectsBucketMutations(t *testing.T) {
ctx := t.Context()
bed, err := prepareAdminErasureTestBed(ctx)
if err != nil {
t.Fatal(err)
}
defer bed.TearDown()
previousCache := listBucketsCache
defer func() { listBucketsCache = previousCache }()
// Hold a real pre-mutation snapshot past the assertions so the regression
// cannot disappear merely because a slow test outlasted the cache TTL.
freezeListing := func() {
t.Helper()
buckets, err := bed.objLayer.ListBuckets(ctx, BucketOptions{})
if err != nil {
t.Fatal(err)
}
listBucketsCache = cachevalue.NewFromFunc(time.Hour, cachevalue.Opts{}, func(context.Context) ([]BucketInfo, error) {
return buckets, nil
})
if _, err := listBucketsCache.GetWithCtx(ctx); err != nil {
t.Fatal(err)
}
}
const bucket = "account-info-freshness"
assertPresent := func(want bool) {
t.Helper()
req, err := buildAdminRequest(nil, http.MethodGet, "/accountinfo", 0, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("account info: %d %s", rec.Code, rec.Body.String())
}
var info madmin.AccountInfo
if err := json.Unmarshal(rec.Body.Bytes(), &info); err != nil {
t.Fatal(err)
}
found := false
for _, item := range info.Buckets {
if item.Name == bucket {
found = true
}
}
if found != want {
t.Errorf("account info bucket present = %v, want %v", found, want)
}
}
freezeListing()
if err := bed.objLayer.MakeBucket(ctx, bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
assertPresent(true)
freezeListing()
if err := bed.objLayer.DeleteBucket(ctx, bucket, DeleteBucketOptions{}); err != nil {
t.Fatal(err)
}
assertPresent(false)
}
+34 -10
View File
@@ -76,7 +76,7 @@ func (a adminAPIHandlers) PutBucketQuotaConfigHandler(w http.ResponseWriter, r *
return
}
quotaConfig, err := parseBucketQuota(bucket, data)
_, err = parseBucketQuota(bucket, data)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -94,9 +94,6 @@ func (a adminAPIHandlers) PutBucketQuotaConfigHandler(w http.ResponseWriter, r *
Quota: data,
UpdatedAt: updatedAt,
}
if quotaConfig.Size == 0 && quotaConfig.Quota == 0 {
bucketMeta.Quota = nil
}
// Call site replication hook.
replLogIf(ctx, globalSiteReplicationSys.BucketMetaHook(ctx, bucketMeta))
@@ -438,7 +435,7 @@ func (a adminAPIHandlers) ExportBucketMetadataHandler(w http.ResponseWriter, r *
writeErrorResponse(ctx, w, exportError(ctx, err, cfgFile, bucket), r.URL)
return
}
configData, err := json.Marshal(config)
configData, err := canonicalBucketPolicy(config)
if err != nil {
writeErrorResponse(ctx, w, exportError(ctx, err, cfgFile, bucket), r.URL)
return
@@ -922,7 +919,7 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
continue
}
configData, err := json.Marshal(bucketPolicy)
configData, err := canonicalBucketPolicy(bucketPolicy)
if err != nil {
rpt.SetStatus(bucket, fileName, err)
continue
@@ -1081,18 +1078,39 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
continue
}
var merged BucketMetadata
var commitAt time.Time
err := func() error {
lockCtx, unlock, err := lockBucketMetadata(ctx, objectAPI, bucket)
if err != nil {
return err
}
defer unlock()
merged, err = loadBucketMetadataParse(lockCtx, objectAPI, bucket, true)
merged, err = loadBucketMetadataParse(lockCtx, objectAPI, bucket, false)
if err != nil {
return err
}
if err := ensureBucketMetadataCreated(lockCtx, objectAPI, &merged); err != nil {
return err
}
commitAt = UTCNow()
for _, file := range replicatedBucketConfigs {
if _, ok := fields[file]; ok {
commitAt = localBucketConfigUpdatedAt(merged, file, commitAt)
}
}
applyImportedBucketMetadata(&merged, *meta, fields)
return globalBucketMetadataSys.saveMetadata(lockCtx, objectAPI, merged)
for _, file := range replicatedBucketConfigs {
if _, ok := fields[file]; !ok {
continue
}
data, at := replicatedBucketConfig(&merged, file)
payload, _, err := bucketConfigPayload(bucket, file, *data, len(merged.ObjectLockConfigXML) != 0)
if err != nil {
return err
}
*data, *at = payload, commitAt
}
return globalBucketMetadataSys.saveMetadata(lockCtx, objectAPI, &merged)
}()
if err != nil {
rpt.SetStatus(bucket, "", err)
@@ -1100,7 +1118,7 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
}
*meta = merged
globalNotificationSys.LoadBucketMetadata(bgContext(ctx), bucket)
hook := madmin.SRBucketMeta{Bucket: bucket, UpdatedAt: updatedAt}
hook := madmin.SRBucketMeta{Bucket: bucket, UpdatedAt: commitAt}
var hookNeeded bool
if _, ok := fields[bucketQuotaConfigFile]; ok {
hook.Quota = meta.QuotaConfigJSON
@@ -1108,7 +1126,7 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
}
if _, ok := fields[bucketPolicyConfig]; ok {
hook.Policy = meta.PolicyConfigJSON
hookNeeded = true
hookNeeded = hookNeeded || len(hook.Policy) != 0
}
if _, ok := fields[bucketVersioningConfig]; ok {
hook.Versioning = enc(meta.VersioningConfigXML)
@@ -1129,6 +1147,12 @@ func (a adminAPIHandlers) ImportBucketMetadataHandler(w http.ResponseWriter, r *
if hookNeeded {
err = globalSiteReplicationSys.BucketMetaHook(ctx, hook)
}
if _, ok := fields[bucketPolicyConfig]; ok && len(meta.PolicyConfigJSON) == 0 {
// An omitted bulk Policy cannot express deletion.
err = errors.Join(err, globalSiteReplicationSys.BucketMetaHook(ctx, madmin.SRBucketMeta{
Type: madmin.SRBucketMetaTypePolicy, Bucket: bucket, UpdatedAt: commitAt,
}))
}
if _, ok := fields[bucketCorsConfig]; ok {
// CORS carries its own timestamp, so it replicates through the
// dedicated event rather than the shared bucket metadata hook. It
+3 -1
View File
@@ -249,8 +249,10 @@ func (a adminAPIHandlers) AddServiceAccountLDAP(w http.ResponseWriter, r *http.R
// In case of LDAP/OIDC we need to set `opts.claims` to ensure
// it is associated with the LDAP/OIDC user properly.
// The new credential's policy comes from the request, not the caller.
for k, v := range cred.Claims {
if k == expClaim {
switch k {
case expClaim, policy.SessionPolicyName, sessionPolicyNameExtracted, iamPolicyClaimNameSA():
continue
}
opts.claims[k] = v
+556
View File
@@ -0,0 +1,556 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
polplugin "github.com/minio/minio/internal/config/policy/plugin"
"github.com/minio/mux"
xnet "github.com/pgsty/silo-pkg/v3/net"
"github.com/pgsty/silo-pkg/v3/policy"
)
func iamAdminFixture(t *testing.T) (context.Context, *IAMSys) {
t.Helper()
ctx, sys, obj := prepareIAMRevisionFixture(t)
globalObjLayerMutex.Lock()
globalObjectAPI = obj
globalObjLayerMutex.Unlock()
return ctx, sys
}
func iamAdminUser(ctx context.Context, t *testing.T, sys *IAMSys, name, policyName string) auth.Credentials {
t.Helper()
cred := auth.Credentials{AccessKey: name, SecretKey: "review-user-valid-secret"}
_, err := sys.CreateUser(ctx, name, madmin.AddOrUpdateUserReq{SecretKey: cred.SecretKey, Status: madmin.AccountEnabled})
mustIAM(t, err)
if policyName != "" {
_, err = sys.PolicyDBSet(ctx, name, policyName, regUser, false)
mustIAM(t, err)
}
return cred
}
func iamAdminServiceAccount(ctx context.Context, t *testing.T, sys *IAMSys, parent, key, document string) auth.Credentials {
t.Helper()
var sp *policy.Policy
var err error
if document != "" {
sp, err = policy.ParseConfig(strings.NewReader(document))
mustIAM(t, err)
}
cred, _, err := sys.NewServiceAccount(ctx, parent, nil, newServiceAccountOpts{accessKey: key, secretKey: "review-service-valid-secret", sessionPolicy: sp})
mustIAM(t, err)
return cred
}
func iamAdminRequest(t *testing.T, cred auth.Credentials, method, target string, payload any) *http.Request {
t.Helper()
var body []byte
if payload != nil {
raw, err := json.Marshal(payload)
mustIAM(t, err)
body, err = madmin.EncryptData(cred.SecretKey, raw)
mustIAM(t, err)
}
headers := map[string]string{}
if cred.IsTemp() && !cred.IsServiceAccount() {
headers["X-Amz-Security-Token"] = cred.SessionToken
}
req, err := newTestSignedRequestV4(method, "http://localhost"+target, int64(len(body)), bytes.NewReader(body), cred.AccessKey, cred.SecretKey, headers)
mustIAM(t, err)
mustIAM(t, req.ParseForm())
return req
}
func iamAdminCall(t *testing.T, cred auth.Credentials, method, target string, payload any, vars map[string]string, handler http.HandlerFunc) *httptest.ResponseRecorder {
t.Helper()
req := iamAdminRequest(t, cred, method, target, payload)
req = mux.SetURLVars(req, vars)
rec := httptest.NewRecorder()
req = req.WithContext(newContext(req, rec, "IAMAdminAuthorization"))
handler(rec, req)
return rec
}
func assertIAMAdminStatus(t *testing.T, operation string, rec *httptest.ResponseRecorder, want int) {
t.Helper()
if rec.Code != want {
t.Errorf("%s status=%d want=%d body=%s", operation, rec.Code, want, rec.Body.String())
return
}
if want == http.StatusForbidden || want == http.StatusNotFound {
var response APIErrorResponse
mustIAM(t, json.Unmarshal(rec.Body.Bytes(), &response))
wantCode := errorCodes.ToAPIErr(ErrAccessDenied).Code
if want == http.StatusNotFound {
wantCode = errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound).Code
}
if response.Code != wantCode {
t.Errorf("%s error code=%s want=%s", operation, response.Code, wantCode)
}
}
}
func iamAdminPlugin(t *testing.T, allow bool) {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]bool{"result": allow})
}))
t.Cleanup(server.Close)
u, err := xnet.ParseHTTPURL(server.URL)
mustIAM(t, err)
previous := newGlobalAuthZPluginFn()
setGlobalAuthZPlugin(polplugin.New(polplugin.Args{URL: u, CloseRespFn: func(r io.ReadCloser) { r.Close() }}))
t.Cleanup(func() { setGlobalAuthZPlugin(previous) })
}
func TestIAMAdminRootEmptyPolicy(t *testing.T) {
for _, doc := range []string{"", "null", "{}", `{"Statement":null}`} {
t.Run("policy="+doc, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "review-root-empty", doc)
sibling := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "review-root-sibling", "")
req := iamAdminRequest(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil)
checked, owner, code := checkKeyValid(req, caller.AccessKey)
if code != ErrNone {
t.Fatalf("checkKeyValid: %v", code)
}
active, _ := isAllowedBySessionPolicyForServiceAccount(policy.Args{Claims: checked.Claims})
t.Logf("owner=%v parsed_active_session_policy=%v", owner, active)
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "create", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-root-new", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccount), 200)
assertIAMAdminStatus(t, "update", iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+sibling.AccessKey, madmin.UpdateServiceAccountReq{NewName: "review"}, map[string]string{"accessKey": sibling.AccessKey}, api.UpdateServiceAccount), 204)
assertIAMAdminStatus(t, "delete", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+sibling.AccessKey, nil, map[string]string{"accessKey": sibling.AccessKey}, api.DeleteServiceAccount), 204)
assertIAMAdminStatus(t, "revoke", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 204)
})
}
}
func TestIAMAdminEmptyPolicyDelegation(t *testing.T) {
for _, parentType := range []string{"root", "user"} {
for _, document := range []string{"null", "{}", `{"Statement":null}`} {
t.Run(parentType+"/"+document, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := globalActiveCred
if parentType == "user" {
parent = iamAdminUser(ctx, t, sys, "empty-policy-parent", "readwrite")
}
sp, err := policy.ParseConfig(strings.NewReader(document))
mustIAM(t, err)
caller, _, err := sys.NewServiceAccount(ctx, parent.AccessKey, nil, newServiceAccountOpts{
accessKey: "empty-policy-caller", secretKey: "empty-policy-caller-secret", sessionPolicy: sp,
claims: map[string]any{"identity-attribute": "preserved"},
})
mustIAM(t, err)
const limitedPolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:ListServiceAccounts"]}]}`
for _, tc := range []struct {
policy string
listStatus int
allowObject bool
}{
{"", http.StatusOK, true},
{"null", http.StatusOK, true},
{"{}", http.StatusOK, true},
{`{"Statement":null}`, http.StatusOK, true},
{limitedPolicy, http.StatusOK, false},
{`{"Version":"2012-10-17"}`, http.StatusForbidden, false},
{`{"Statement":[]}`, http.StatusForbidden, false},
} {
expires := UTCNow().Add(time.Hour)
req := madmin.AddServiceAccountReq{Expiration: &expires}
if tc.policy != "" {
req.Policy = json.RawMessage(tc.policy)
}
api := adminAPIHandlers{}
rec := iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", req, nil, api.AddServiceAccount)
assertIAMAdminStatus(t, "create expiring child", rec, http.StatusOK)
data, err := madmin.DecryptData(caller.SecretKey, rec.Body)
mustIAM(t, err)
var response madmin.AddServiceAccountResp
mustIAM(t, json.Unmarshal(data, &response))
child, ok := sys.store.GetUser(response.Credentials.AccessKey)
if !ok {
t.Fatal("new service account missing")
}
// Exercise the issued key through authentication and authorization.
assertIAMAdminStatus(t, "use expiring child policy="+tc.policy, iamAdminCall(t, child.Credentials, "GET", "/minio/admin/v3/list-service-accounts", nil, nil, api.ListServiceAccounts), tc.listStatus)
signed := iamAdminRequest(t, child.Credentials, "GET", "/example/object", nil)
checked, owner, code := checkKeyValid(signed, child.Credentials.AccessKey)
if code != ErrNone {
t.Fatalf("child authentication: %v", code)
}
if checked.Claims["identity-attribute"] != "preserved" {
t.Fatal("creation lost the caller's identity context")
}
allowed := sys.IsAllowed(policy.Args{AccountName: checked.AccessKey, Claims: checked.Claims, IsOwner: owner, Action: policy.GetObjectAction, BucketName: "example", ObjectName: "object"})
if allowed != tc.allowObject {
t.Errorf("GetObject allowed=%v with requested policy=%s", allowed, tc.policy)
}
}
})
}
}
}
func TestIAMAdminCorruptServiceCleanup(t *testing.T) {
for _, status := range []string{auth.AccountOn, auth.AccountOff, "expired"} {
t.Run(status, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "review-admin", "consoleAdmin")
parent := iamAdminUser(ctx, t, sys, "review-parent", "readwrite")
svc := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-corrupt", "")
if status == auth.AccountOff {
_, err := sys.UpdateServiceAccount(ctx, svc.AccessKey, updateServiceAccountOpts{status: status})
mustIAM(t, err)
}
cache := sys.store.lock()
u := cache.iamUsersMap[svc.AccessKey]
u.Credentials.SessionToken = "broken-session-token"
if status == "expired" {
u.Credentials.Expiration = UTCNow().Add(-time.Minute)
}
cache.iamUsersMap[svc.AccessKey] = u
sys.store.unlock()
_, _, err := sys.GetServiceAccount(ctx, svc.AccessKey)
if err == nil {
t.Fatal("fault injection failed")
}
raw, exists := sys.store.GetUser(svc.AccessKey)
_, loginOK := sys.GetUser(ctx, svc.AccessKey)
t.Logf("metadata_exists=%v parent_known=%v GetUser_ok=%v", exists, raw.Credentials.ParentUser == parent.AccessKey, loginOK)
api := adminAPIHandlers{}
outsider := iamAdminUser(ctx, t, sys, "cleanup-outsider", "readwrite")
assertIAMAdminStatus(t, "unauthorized cleanup", iamAdminCall(t, outsider, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.DeleteServiceAccount), 404)
assertIAMAdminStatus(t, "admin cleanup", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.DeleteServiceAccount), 204)
if _, ok := sys.store.GetUser(svc.AccessKey); ok {
t.Fatal("deleted service account remains in the IAM cache")
}
})
}
}
func TestIAMAdminPolicylessSelf(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "review-policyless", "")
svc := iamAdminServiceAccount(ctx, t, sys, caller.AccessKey, "rv-policyless-child", "")
other := iamAdminUser(ctx, t, sys, "bulk-other", "readwrite")
_ = iamAdminServiceAccount(ctx, t, sys, other.AccessKey, "bulk-other-child", "")
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "list", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil, nil, api.ListServiceAccounts), 200)
rec := iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-access-keys-bulk?listType=all", nil, nil, api.ListAccessKeysBulk)
assertIAMAdminStatus(t, "list bulk self", rec, 200)
data, err := madmin.DecryptData(caller.SecretKey, rec.Body)
mustIAM(t, err)
var listed map[string]madmin.ListAccessKeysResp
mustIAM(t, json.Unmarshal(data, &listed))
entry, ok := listed[caller.AccessKey]
if len(listed) != 1 || !ok || len(entry.ServiceAccounts) != 1 || entry.ServiceAccounts[0].AccessKey != svc.AccessKey || len(entry.STSKeys) != 0 {
t.Fatalf("policyless bulk self returned unexpected identities: %+v", listed)
}
assertIAMAdminStatus(t, "info service", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.InfoServiceAccount), 200)
assertIAMAdminStatus(t, "info access key", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-access-key?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.InfoAccessKey), 200)
assertIAMAdminStatus(t, "delete", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.DeleteServiceAccount), 204)
assertIAMAdminStatus(t, "revoke", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 204)
assertIAMAdminStatus(t, "list another user", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts?user="+globalActiveCred.AccessKey, nil, nil, api.ListServiceAccounts), 403)
assertIAMAdminStatus(t, "revoke another user", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true&user="+globalActiveCred.AccessKey, nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
}
func TestIAMAdminExpiringInheritedPolicy(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "review-legacy-parent", "consoleAdmin")
document := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:CreateServiceAccount","admin:UpdateServiceAccount"]}]}`
// Model a credential persisted by the old handler: it copied its caller's
// policy claims but set the newly created service's type to inherited.
expires := UTCNow().Add(time.Hour)
caller, _, err := sys.NewServiceAccount(ctx, parent.AccessKey, nil, newServiceAccountOpts{
accessKey: "rv-exp-inherited", secretKey: "review-service-valid-secret", expiration: &expires,
claims: map[string]any{policy.SessionPolicyName: base64.StdEncoding.EncodeToString([]byte(document))},
})
mustIAM(t, err)
req := iamAdminRequest(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil)
checked, owner, code := checkKeyValid(req, caller.AccessKey)
if code != ErrNone {
t.Fatalf("checkKeyValid: %v", code)
}
args := policy.Args{AccountName: caller.AccessKey, Claims: checked.Claims, IsOwner: owner, Action: policy.CreateServiceAccountAdminAction, DenyOnly: true}
if !sys.IsAllowed(args) {
t.Fatal("fixture cannot create")
}
args.Action = policy.CreateUserAdminAction
args.DenyOnly = false
if sys.IsAllowed(args) {
t.Fatal("fixture not constrained")
}
t.Logf("service=%v temp=%v inherited=%v; create allowed, create-user denied", checked.IsServiceAccount(), checked.IsTemp(), checked.Claims[iamPolicyClaimNameSA()] == inheritedPolicyType)
api := adminAPIHandlers{}
rec := iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-escaped", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccount)
if rec.Code == 200 {
child, ok := sys.GetUser(ctx, "review-escaped")
if !ok {
t.Fatal("new key missing")
}
childReq := iamAdminRequest(t, child.Credentials, "GET", "/minio/admin/v3/list-service-accounts", nil)
childChecked, childOwner, childCode := checkKeyValid(childReq, child.Credentials.AccessKey)
if childCode != ErrNone {
t.Fatalf("child auth=%v", childCode)
}
allowed := sys.IsAllowed(policy.Args{AccountName: childChecked.AccessKey, Claims: childChecked.Claims, IsOwner: childOwner, Action: policy.CreateUserAdminAction})
t.Errorf("constrained inherited service created a new key; new key CreateUser allowed=%v", allowed)
} else {
assertIAMAdminStatus(t, "reject derived creation", rec, 403)
}
updated := iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+caller.AccessKey, madmin.UpdateServiceAccountReq{NewName: "metadata-only"}, map[string]string{"accessKey": caller.AccessKey}, api.UpdateServiceAccount)
if updated.Code == 204 {
after, ok := sys.GetUser(ctx, caller.AccessKey)
if !ok {
t.Fatal("updated key missing")
}
afterReq := iamAdminRequest(t, after.Credentials, "GET", "/minio/admin/v3/list-service-accounts", nil)
afterChecked, afterOwner, afterCode := checkKeyValid(afterReq, caller.AccessKey)
if afterCode != ErrNone {
t.Fatalf("updated auth=%v", afterCode)
}
allowed := sys.IsAllowed(policy.Args{AccountName: caller.AccessKey, Claims: afterChecked.Claims, IsOwner: afterOwner, Action: policy.CreateUserAdminAction})
t.Errorf("constrained inherited service updated itself; CreateUser allowed after update=%v", allowed)
} else {
assertIAMAdminStatus(t, "reject inherited self update", updated, 403)
}
}
func TestIAMAdminSessionPolicyCannotDelegate(t *testing.T) {
for _, temp := range []bool{false, true} {
t.Run(map[bool]string{false: "service", true: "sts"}[temp], func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "rv-parent", "consoleAdmin")
doc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:CreateServiceAccount","admin:UpdateServiceAccount"]}]}`
var caller auth.Credentials
if !temp {
caller = iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "review-restricted", doc)
} else {
var err error
caller, err = auth.GetNewCredentialsWithMetadata(map[string]any{parentClaim: parent.AccessKey, "exp": UTCNow().Add(time.Hour).Unix(), policy.SessionPolicyName: base64.StdEncoding.EncodeToString([]byte(doc))}, globalActiveCred.SecretKey)
mustIAM(t, err)
caller.ParentUser = parent.AccessKey
_, err = sys.SetTempUser(ctx, caller.AccessKey, caller, "")
mustIAM(t, err)
}
sibling := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-sibling", "")
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "restrict create", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-bad-new", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccount), 403)
assertIAMAdminStatus(t, "restrict LDAP create", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/idp/ldap/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-bad-new", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccountLDAP), 403)
assertIAMAdminStatus(t, "restrict update", iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+sibling.AccessKey, madmin.UpdateServiceAccountReq{NewSecretKey: "review-known-new-secret"}, map[string]string{"accessKey": sibling.AccessKey}, api.UpdateServiceAccount), 403)
if _, ok := sys.store.GetUser("review-bad-new"); ok {
t.Fatal("denied creation persisted a credential")
}
u, ok := sys.store.GetUser(sibling.AccessKey)
if !ok || u.Credentials.SecretKey != sibling.SecretKey {
t.Fatal("denied update changed the target credential")
}
})
}
}
func TestIAMAdminSelfDenials(t *testing.T) {
for _, mode := range []string{"parent_deny", "group_deny", "session_implicit_deny", "session_explicit_deny", "plugin_deny"} {
t.Run(mode, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "rv-self-parent", "readwrite")
target := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-self-target", "")
caller := parent
doc := `{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["admin:ListServiceAccounts","admin:RemoveServiceAccount"]}]}`
switch mode {
case "parent_deny", "group_deny":
p, err := policy.ParseConfig(strings.NewReader(doc))
mustIAM(t, err)
_, err = sys.SetPolicy(ctx, "review-self-deny", *p)
mustIAM(t, err)
if mode == "group_deny" {
_, err = sys.AddUsersToGroup(ctx, "self-deny-group", []string{parent.AccessKey})
mustIAM(t, err)
_, err = sys.PolicyDBSet(ctx, "self-deny-group", "review-self-deny", regUser, true)
} else {
_, err = sys.PolicyDBSet(ctx, parent.AccessKey, "readwrite,review-self-deny", regUser, false)
}
mustIAM(t, err)
case "session_implicit_deny":
caller = iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-self-caller", `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::example/*"]}]}`)
case "session_explicit_deny":
caller = iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-self-caller", doc)
case "plugin_deny":
iamAdminPlugin(t, false)
}
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "list", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil, nil, api.ListServiceAccounts), 403)
assertIAMAdminStatus(t, "list bulk self", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-access-keys-bulk?listType=all", nil, nil, api.ListAccessKeysBulk), 403)
assertIAMAdminStatus(t, "info service", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.InfoServiceAccount), 403)
assertIAMAdminStatus(t, "info access key", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-access-key?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.InfoAccessKey), 403)
assertIAMAdminStatus(t, "delete", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 403)
assertIAMAdminStatus(t, "revoke", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
})
}
}
func TestIAMAdminNonRootCannotManageRoot(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "rv-console-admin", "consoleAdmin")
target := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "rv-root-target", "")
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "create root", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "rv-new-root", SecretKey: "review-service-valid-secret", TargetUser: globalActiveCred.AccessKey}, nil, api.AddServiceAccount), 403)
assertIAMAdminStatus(t, "update root", iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+target.AccessKey, madmin.UpdateServiceAccountReq{NewSecretKey: "review-known-new-secret"}, map[string]string{"accessKey": target.AccessKey}, api.UpdateServiceAccount), 403)
assertIAMAdminStatus(t, "delete root", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 404)
assertIAMAdminStatus(t, "revoke root", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true&user="+globalActiveCred.AccessKey, nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
// Restricted root-derived credentials and the site replicator are not root
// administrators, even when their effective policy allows these actions.
limited := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "limited-root", `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:*"]}]}`)
assertIAMAdminStatus(t, "limited root delete", iamAdminCall(t, limited, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 404)
assertIAMAdminStatus(t, "limited root revoke", iamAdminCall(t, limited, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
replicator, _, err := sys.NewServiceAccount(ctx, globalActiveCred.AccessKey, nil, newServiceAccountOpts{accessKey: siteReplicatorSvcAcc, secretKey: "replicator-valid-secret", allowSiteReplicatorAccount: true})
mustIAM(t, err)
assertIAMAdminStatus(t, "replicator create root", iamAdminCall(t, replicator, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{}, nil, api.AddServiceAccount), 403)
// An external plugin remains authoritative, including for root targets.
iamAdminPlugin(t, true)
assertIAMAdminStatus(t, "plugin update root", iamAdminCall(t, limited, "POST", "/minio/admin/v3/update-service-account?accessKey="+target.AccessKey, madmin.UpdateServiceAccountReq{NewName: "plugin-authorized"}, map[string]string{"accessKey": target.AccessKey}, api.UpdateServiceAccount), 204)
assertIAMAdminStatus(t, "plugin create root", iamAdminCall(t, limited, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{}, nil, api.AddServiceAccount), 200)
}
func TestIAMAdminDeleteCorruptServiceReplicates(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "cleanup-admin", "consoleAdmin")
parent := iamAdminUser(ctx, t, sys, "cleanup-parent", "readwrite")
target := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "cleanup-target", "")
cache := sys.store.lock()
u := cache.iamUsersMap[target.AccessKey]
u.Credentials.SessionToken = "broken-session-token"
u.Credentials.Status = auth.AccountOff
cache.iamUsersMap[target.AccessKey] = u
sys.store.unlock()
var deletions atomic.Int32
peer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/minio/health/live" {
return
}
var batch iamRevisionBatch
if err := json.NewDecoder(r.Body).Decode(&batch); err != nil {
t.Error(err)
w.WriteHeader(http.StatusBadRequest)
return
}
if len(batch.Items) != 1 || batch.Items[0].SvcAccChange == nil || batch.Items[0].SvcAccChange.Delete == nil || batch.Items[0].SvcAccChange.Delete.AccessKey != target.AccessKey {
t.Error("replication did not carry the service account deletion")
} else {
deletions.Add(1)
}
_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "peer", Instance: "peer-boot", Digest: "deleted"}})
}))
t.Cleanup(peer.Close)
globalSiteReplicatorCred.RLock()
previousSecret := globalSiteReplicatorCred.secretKey
globalSiteReplicatorCred.RUnlock()
globalSiteReplicatorCred.Set(caller.SecretKey)
t.Cleanup(func() { globalSiteReplicatorCred.Set(previousSecret) })
globalSiteReplicationSys.Lock()
globalSiteReplicationSys.enabled = true
globalSiteReplicationSys.state = srState{ServiceAccountAccessKey: caller.AccessKey, Peers: map[string]madmin.PeerInfo{
globalDeploymentID(): {DeploymentID: globalDeploymentID(), Name: "local"},
"peer": {DeploymentID: "peer", Name: "peer", Endpoint: peer.URL},
}}
globalSiteReplicationSys.Unlock()
t.Cleanup(func() {
globalSiteReplicationSys.Lock()
globalSiteReplicationSys.enabled = false
globalSiteReplicationSys.Unlock()
})
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "replicated cleanup", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 204)
if deletions.Load() != 1 {
t.Fatalf("expected one immediate deletion notification, got %d", deletions.Load())
}
}
func TestIAMAdminListSelfAliases(t *testing.T) {
for _, kind := range []string{"service", "sts"} {
t.Run(kind, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "alias-parent", "readwrite")
target := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "alias-target", "")
serviceCaller := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "alias-caller", "")
caller := serviceCaller
other := iamAdminUser(ctx, t, sys, "alias-other", "readwrite")
_ = iamAdminServiceAccount(ctx, t, sys, other.AccessKey, "alias-other-child", "")
if kind == "sts" {
var err error
caller, err = auth.GetNewCredentialsWithMetadata(map[string]any{parentClaim: parent.AccessKey, "exp": UTCNow().Add(time.Hour).Unix()}, globalActiveCred.SecretKey)
mustIAM(t, err)
caller.ParentUser = parent.AccessKey
_, err = sys.SetTempUser(ctx, caller.AccessKey, caller, "")
mustIAM(t, err)
}
api := adminAPIHandlers{}
for _, alias := range []string{"", caller.AccessKey, parent.AccessKey} {
rec := iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts?user="+alias, nil, nil, api.ListServiceAccounts)
assertIAMAdminStatus(t, "list self alias", rec, 200)
data, err := madmin.DecryptData(caller.SecretKey, rec.Body)
mustIAM(t, err)
var response madmin.ListServiceAccountsResp
mustIAM(t, json.Unmarshal(data, &response))
found := false
for _, account := range response.Accounts {
found = found || account.AccessKey == target.AccessKey
}
if !found {
t.Fatalf("self alias %q did not list the parent's service accounts", alias)
}
query := url.Values{"listType": {madmin.AccessKeyListAll}}
if alias != "" {
query.Set("users", alias)
}
bulk := iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-access-keys-bulk?"+query.Encode(), nil, nil, api.ListAccessKeysBulk)
assertIAMAdminStatus(t, "list bulk self alias", bulk, 200)
data, err = madmin.DecryptData(caller.SecretKey, bulk.Body)
mustIAM(t, err)
var listed map[string]madmin.ListAccessKeysResp
mustIAM(t, json.Unmarshal(data, &listed))
entry, ok := listed[parent.AccessKey]
if len(listed) != 1 || !ok || len(entry.ServiceAccounts) != 2 {
t.Fatalf("bulk self alias %q returned unexpected identities: %+v", alias, listed)
}
want := map[string]bool{target.AccessKey: true, serviceCaller.AccessKey: true}
for _, account := range entry.ServiceAccounts {
if !want[account.AccessKey] {
t.Fatalf("bulk self alias %q returned unexpected or duplicate service account %s", alias, account.AccessKey)
}
delete(want, account.AccessKey)
}
if kind == "sts" {
if len(entry.STSKeys) != 1 || entry.STSKeys[0].AccessKey != caller.AccessKey {
t.Fatalf("bulk self alias %q returned unexpected STS accounts: %+v", alias, entry.STSKeys)
}
} else if len(entry.STSKeys) != 0 {
t.Fatalf("service caller returned unexpected STS accounts: %+v", entry.STSKeys)
}
if bytes.Contains(data, []byte(serviceCaller.SecretKey)) || bytes.Contains(data, []byte(`"secretKey"`)) || bytes.Contains(data, []byte(`"sessionToken"`)) {
t.Fatal("bulk listing returned credential secrets")
}
}
})
}
}
+151 -87
View File
@@ -502,11 +502,15 @@ func (a adminAPIHandlers) AddUser(w http.ResponseWriter, r *http.Request) {
}
checkDenyOnly := accessKey == cred.AccessKey
action := policy.Action(policy.CreateUserAdminAction)
if checkDenyOnly {
action = policy.ChangeMyPasswordAdminAction
}
if !globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.CreateUserAdminAction,
Action: action,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
@@ -710,8 +714,10 @@ func (a adminAPIHandlers) AddServiceAccount(w http.ResponseWriter, r *http.Reque
// In case of LDAP/OIDC we need to set `opts.claims` to ensure
// it is associated with the LDAP/OIDC user properly.
// The new credential's policy comes from the request, not the caller.
for k, v := range cred.Claims {
if k == expClaim {
switch k {
case expClaim, policy.SessionPolicyName, sessionPolicyNameExtracted, iamPolicyClaimNameSA():
continue
}
opts.claims[k] = v
@@ -851,10 +857,9 @@ func (a adminAPIHandlers) UpdateServiceAccount(w http.ResponseWriter, r *http.Re
// Permission checks:
//
// 1. Any type of account (i.e. access keys (previously/still called service
// accounts), STS accounts, internal IDP accounts, etc) with the
// policy.UpdateServiceAccountAdminAction permission can update any service
// account.
// 1. Updating requires policy.UpdateServiceAccountAdminAction. Under built-in
// IAM, session-limited callers cannot update credentials, and only unrestricted
// root credentials can manage root's service accounts.
//
// 2. We would like to let a user update their own access keys, however it
// is currently blocked pending a re-design. Users are still able to delete
@@ -870,6 +875,11 @@ func (a adminAPIHandlers) UpdateServiceAccount(w http.ResponseWriter, r *http.Re
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if newGlobalAuthZPluginFn() == nil && (hasActiveSessionPolicy(cred) ||
(svcAccount.ParentUser == globalActiveCred.AccessKey && !canManageRootCredentials(cred))) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
var sp *policy.Policy
if len(updateReq.NewPolicy) > 0 {
@@ -947,23 +957,21 @@ func (a adminAPIHandlers) InfoServiceAccount(w http.ResponseWriter, r *http.Requ
return
}
if !globalIAMSys.IsAllowed(policy.Args{
requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
DenyOnly: requestUser == svcAccount.ParentUser,
}) {
requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
if requestUser != svcAccount.ParentUser {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
// if session policy is nil or empty, then it is implied policy
@@ -1035,29 +1043,27 @@ func (a adminAPIHandlers) ListServiceAccounts(w http.ResponseWriter, r *http.Req
return
}
var targetAccount string
// If listing is requested for a specific user (who is not the request
// sender), check that the user has permissions.
selfUser := cred.AccessKey
if cred.ParentUser != "" {
selfUser = cred.ParentUser
}
user := r.Form.Get("user")
if user != "" && user != cred.AccessKey {
if !globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
}) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
targetAccount = user
} else {
targetAccount = cred.AccessKey
if cred.ParentUser != "" {
targetAccount = cred.ParentUser
}
selfOnly := user == "" || user == cred.AccessKey || user == selfUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
DenyOnly: selfOnly,
}) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
targetAccount := user
if selfOnly {
targetAccount = selfUser
}
serviceAccounts, err := globalIAMSys.ListServiceAccounts(ctx, targetAccount)
@@ -1106,7 +1112,7 @@ func (a adminAPIHandlers) DeleteServiceAccount(w http.ResponseWriter, r *http.Re
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil || globalNotificationSys == nil {
if objectAPI == nil || globalNotificationSys == nil || !globalIAMSys.Initialized() {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL)
return
}
@@ -1127,35 +1133,41 @@ func (a adminAPIHandlers) DeleteServiceAccount(w http.ResponseWriter, r *http.Re
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidArgument), r.URL)
return
}
// We do not care if service account is readable or not at this point,
// since this is a delete call we shall allow it to be deleted if possible.
svcAccount, _, err := globalIAMSys.GetServiceAccount(ctx, serviceAccount)
if errors.Is(err, errNoSuchServiceAccount) {
// Deletion needs the stored identity, not a valid token or session policy.
// Keep disabled, expired and damaged service accounts removable, and retain
// their parent for authorization and the site replication hook below.
u, ok := globalIAMSys.store.GetUser(serviceAccount)
if !ok || !u.Credentials.IsServiceAccount() {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL)
return
}
svcAccount := u.Credentials
adminPrivilege := globalIAMSys.IsAllowed(policy.Args{
parentUser := cred.AccessKey
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
isSelf := parentUser == svcAccount.ParentUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.RemoveServiceAccountAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
})
if !adminPrivilege {
parentUser := cred.AccessKey
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
if svcAccount.ParentUser != "" && parentUser != svcAccount.ParentUser {
// The service account belongs to another user but return not
// found error to mitigate brute force attacks. or the
// serviceAccount doesn't exist.
DenyOnly: isSelf,
}) {
if !isSelf {
// Hide the existence of another user's service account.
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL)
return
}
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if newGlobalAuthZPluginFn() == nil && svcAccount.ParentUser == globalActiveCred.AccessKey && !canManageRootCredentials(cred) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL)
return
}
if err := globalIAMSys.DeleteServiceAccount(ctx, serviceAccount, true); err != nil {
@@ -1225,7 +1237,7 @@ func (a adminAPIHandlers) ListAccessKeysBulk(w http.ResponseWriter, r *http.Requ
}
}
if !globalIAMSys.IsAllowed(policy.Args{
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction,
@@ -1238,12 +1250,12 @@ func (a adminAPIHandlers) ListAccessKeysBulk(w http.ResponseWriter, r *http.Requ
return
}
if selfOnly && len(users) == 0 {
if selfOnly {
selfUser := cred.AccessKey
if cred.ParentUser != "" {
selfUser = cred.ParentUser
}
users = append(users, selfUser)
users = []string{selfUser}
}
var checkedUserList []string
@@ -1435,7 +1447,10 @@ func (a adminAPIHandlers) AccountInfoHandler(w http.ResponseWriter, r *http.Requ
return buckets[i].Name < buckets[j].Name
})
} else {
buckets, err = objectAPI.ListBuckets(ctx, BucketOptions{Cached: true})
// Console refreshes this inventory immediately after bucket mutations.
// A stale cached list can otherwise hide a creation or retain a deletion
// indefinitely in the client's snapshot.
buckets, err = objectAPI.ListBuckets(ctx, BucketOptions{})
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
@@ -2042,26 +2057,30 @@ func (a adminAPIHandlers) RevokeTokens(w http.ResponseWriter, r *http.Request) {
return
}
adminPrivilege := globalIAMSys.IsAllowed(policy.Args{
parentUser := cred.AccessKey
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
isSelfTarget := isTokenSelfRevoke || user == parentUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.RemoveServiceAccountAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
})
if !adminPrivilege || isTokenSelfRevoke {
parentUser := cred.AccessKey
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
if !isTokenSelfRevoke && user != parentUser {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
DenyOnly: isSelfTarget,
}) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if isSelfTarget {
user = parentUser
}
if newGlobalAuthZPluginFn() == nil && user == globalActiveCred.AccessKey && !canManageRootCredentials(cred) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
// Infer token revoke type from the request if requestor is STS.
if isTokenSelfRevoke && tokenRevokeType == "" && !fullRevoke {
@@ -2108,29 +2127,22 @@ func (a adminAPIHandlers) InfoAccessKey(w http.ResponseWriter, r *http.Request)
u, ok := globalIAMSys.GetUser(ctx, accessKey)
targetCred := u.Credentials
if !globalIAMSys.IsAllowed(policy.Args{
requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
isSelf := ok && targetCred.ParentUser != "" && requestUser == targetCred.ParentUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
DenyOnly: isSelf,
}) {
// If requested user does not exist and requestor is not allowed to list service accounts, return access denied.
if !ok {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
if requestUser != targetCred.ParentUser {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if !ok {
@@ -2904,6 +2916,44 @@ func addExpirationToCondValues(exp *time.Time, condValues map[string][]string) e
return nil
}
// hasActiveSessionPolicy follows the same credential dispatch as IAMSys.IsAllowed.
// Expiring service accounts also satisfy IsTemp and are evaluated as STS there.
// Callers use this only after authorization under built-in IAM.
func hasActiveSessionPolicy(cred auth.Credentials) bool {
args := policy.Args{Claims: cred.Claims}
if cred.IsTemp() {
hasSP, _ := isAllowedBySessionPolicy(args)
return hasSP
}
if cred.IsServiceAccount() {
if cred.Claims[iamPolicyClaimNameSA()] == inheritedPolicyType {
return false
}
hasSP, _ := isAllowedBySessionPolicyForServiceAccount(args)
return hasSP
}
return false
}
// canManageRootCredentials is an additional guard after built-in IAM authorization.
// The signature's owner flag is insufficient: historical empty session policies
// clear it even though the service account still inherits root's permissions.
func canManageRootCredentials(cred auth.Credentials) bool {
return cred.AccessKey == globalActiveCred.AccessKey ||
(cred.ParentUser == globalActiveCred.AccessKey && cred.AccessKey != siteReplicatorSvcAcc && !hasActiveSessionPolicy(cred))
}
// isAllowedServiceAccountAction preserves regular users' self inspection and
// cleanup even without a policy attachment, while enforcing explicit denies.
// Derived credentials and external authorization plugins use full IAM evaluation.
func isAllowedServiceAccountAction(cred auth.Credentials, args policy.Args) bool {
if args.DenyOnly && !args.IsOwner && !cred.IsTemp() && !cred.IsServiceAccount() && newGlobalAuthZPluginFn() == nil {
policies, err := globalIAMSys.PolicyDBGet(args.AccountName, args.Groups...)
return err == nil && globalIAMSys.GetCombinedPolicy(policies...).IsAllowed(args)
}
return globalIAMSys.IsAllowed(args)
}
func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.Credentials, newServiceAccountOpts, madmin.AddServiceAccountReq, string, APIError) {
ctx := r.Context()
@@ -2992,6 +3042,14 @@ func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.
}) {
return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", errorCodes.ToAPIErr(ErrAccessDenied)
}
effectiveTarget := targetUser
if (targetUser == cred.AccessKey || targetUser == cred.ParentUser) && cred.ParentUser != "" {
effectiveTarget = cred.ParentUser
}
if newGlobalAuthZPluginFn() == nil && (hasActiveSessionPolicy(cred) ||
(effectiveTarget == globalActiveCred.AccessKey && !canManageRootCredentials(cred))) {
return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", errorCodes.ToAPIErr(ErrAccessDenied)
}
var sp *policy.Policy
if len(createReq.Policy) > 0 {
@@ -2999,6 +3057,12 @@ func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.
if err != nil {
return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", toAdminAPIErr(ctx, err)
}
// Empty service policies mean inheritance, including for expiring keys.
// Normalize the replication request as well as the local credential.
if sp.Version == "" && sp.Statements == nil && sp.ID == "" {
sp = nil
createReq.Policy = nil
}
}
opts.sessionPolicy = sp
+102
View File
@@ -243,6 +243,7 @@ func TestIAMInternalIDPServerSuite(t *testing.T) {
suite.SetUpSuite(c)
suite.TestUserCreate(c)
suite.TestUserPasswordActionAuthorization(c)
suite.TestUserStatusActionAuthorization(c)
suite.TestGroupStatusActionAuthorization(c)
suite.TestUserPolicyEscalationBug(c)
@@ -356,6 +357,106 @@ func (s *TestSuiteIAM) TestUserCreate(c *check) {
}
}
func (s *TestSuiteIAM) TestUserPasswordActionAuthorization(c *check) {
for _, tt := range []struct {
name string
statements string
self bool
other bool
}{
{"readonly", "", true, false},
{"consolereadonly", "", true, false},
{"password grant", `{"Effect":"Allow","Action":"admin:ChangeMyPassword"}`, true, false},
{"legacy CreateUser deny", `{"Effect":"Deny","Action":"admin:CreateUser","Resource":"arn:aws:s3:::*"}`, true, false},
{"password deny", `{"Effect":"Deny","Action":"admin:ChangeMyPassword"}`, false, false},
{"user admin", `{"Effect":"Allow","Action":"admin:CreateUser"}`, true, true},
{"user admin with password deny", `{"Effect":"Allow","Action":"admin:CreateUser"},{"Effect":"Deny","Action":"admin:ChangeMyPassword"}`, false, true},
{"password deny overrides grant", `{"Effect":"Allow","Action":"admin:ChangeMyPassword"},{"Effect":"Deny","Action":"admin:ChangeMyPassword"}`, false, false},
{"wildcard deny", `{"Effect":"Deny","Action":"admin:*"}`, false, false},
} {
c.Run(tt.name, func(t *testing.T) {
c := &check{t, s.serverType}
ctx, cancel := context.WithTimeout(context.Background(), testDefaultTimeout)
defer cancel()
var users []string
policyName := tt.name
defer func() {
for _, user := range users {
if err := s.adm.RemoveUser(ctx, user); err != nil {
c.Errorf("remove test user: %v", err)
}
}
if tt.statements != "" {
if err := s.adm.RemoveCannedPolicy(ctx, policyName); err != nil {
c.Errorf("remove test policy: %v", err)
}
}
}()
createUser := func() (string, string) {
accessKey, secretKey := mustGenerateCredentials(c)
if err := s.adm.SetUser(ctx, accessKey, secretKey, madmin.AccountEnabled); err != nil {
c.Fatalf("create test user: %v", err)
}
users = append(users, accessKey)
return accessKey, secretKey
}
client := func(accessKey, secretKey string) *madmin.AdminClient {
adm, err := madmin.New(s.endpoint, accessKey, secretKey, s.secure)
if err != nil {
c.Fatal(err)
}
adm.SetCustomTransport(s.TestSuiteCommon.client.Transport)
return adm
}
if tt.statements != "" {
policyName = getRandomBucketName()
doc := []byte(`{"Version":"2012-10-17","Statement":[` + tt.statements + `]}`)
if err := s.adm.AddCannedPolicy(ctx, policyName, doc); err != nil {
c.Fatalf("save test policy: %v", err)
}
}
accessKey, secretKey := createUser()
if _, err := s.adm.AttachPolicy(ctx, madmin.PolicyAssociationReq{
User: accessKey, Policies: []string{policyName},
}); err != nil {
c.Fatalf("attach test policy: %v", err)
}
adm := client(accessKey, secretKey)
_, newSecretKey := mustGenerateCredentials(c)
err := adm.SetUser(ctx, accessKey, newSecretKey, madmin.AccountEnabled)
if tt.self {
if err != nil {
c.Fatalf("change own password: %v", err)
}
if _, err = adm.AccountInfo(ctx, madmin.AccountOpts{}); err == nil {
c.Fatal("old password still authenticates")
}
adm = client(accessKey, newSecretKey)
} else if err == nil || madmin.ToErrorResponse(err).Code != "AccessDenied" {
c.Fatalf("self password change: expected AccessDenied, got %v", err)
}
if _, err := adm.AccountInfo(ctx, madmin.AccountOpts{}); err != nil {
c.Fatalf("current password no longer authenticates: %v", err)
}
target, _ := createUser()
newUser, newUserSecret := mustGenerateCredentials(c)
for _, key := range []string{target, newUser} {
err := adm.SetUser(ctx, key, newUserSecret, madmin.AccountEnabled)
if tt.other {
if err != nil {
c.Fatalf("create or update another user: %v", err)
}
if key == newUser {
users = append(users, newUser)
}
} else if err == nil || madmin.ToErrorResponse(err).Code != "AccessDenied" {
c.Fatalf("create or update another user: expected AccessDenied, got %v", err)
}
}
})
}
}
func (s *TestSuiteIAM) TestUserStatusActionAuthorization(c *check) {
ctx, cancel := context.WithTimeout(context.Background(), testDefaultTimeout)
defer cancel()
@@ -946,6 +1047,7 @@ func (s *TestSuiteIAM) TestCannedPolicies(c *check) {
defaultPolicies := []string{
"readwrite",
"readonly",
"consolereadonly",
"writeonly",
"diagnostics",
"consoleAdmin",
+2
View File
@@ -163,6 +163,7 @@ func registerAdminRouter(router *mux.Router, enableConfigOps bool) {
// StorageInfo operations
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/storageinfo").HandlerFunc(adminMiddleware(adminAPI.StorageInfoHandler, traceAllFlag))
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/multipart-preflight").HandlerFunc(adminMiddleware(adminAPI.MultipartPreflightHandler, traceAllFlag))
// DataUsageInfo operations
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/datausageinfo").HandlerFunc(adminMiddleware(adminAPI.DataUsageInfoHandler, traceAllFlag))
// Metrics operation
@@ -388,6 +389,7 @@ func registerAdminRouter(router *mux.Router, enableConfigOps bool) {
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/join").HandlerFunc(adminMiddleware(adminAPI.SRPeerJoin))
adminRouter.Methods(http.MethodPut).Path(adminVersion+"/site-replication/peer/bucket-ops").HandlerFunc(adminMiddleware(adminAPI.SRPeerBucketOps)).Queries("bucket", "{bucket:.*}").Queries("operation", "{operation:.*}")
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/iam-item").HandlerFunc(adminMiddleware(adminAPI.SRPeerReplicateIAMItem))
adminRouter.Methods(http.MethodGet, http.MethodPut).Path(adminVersion + "/site-replication/peer/iam-revisions").HandlerFunc(adminMiddleware(adminAPI.SRPeerIAMRevisions))
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/peer/bucket-meta").HandlerFunc(adminMiddleware(adminAPI.SRPeerReplicateBucketItem))
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/site-replication/peer/idp-settings").HandlerFunc(adminMiddleware(adminAPI.SRPeerGetIDPSettings))
adminRouter.Methods(http.MethodPut).Path(adminVersion + "/site-replication/edit").HandlerFunc(adminMiddleware(adminAPI.SiteReplicationEdit))
+45
View File
@@ -0,0 +1,45 @@
// Copyright (c) 2026 PGSTY
//
// This file is part of SILO Object Storage.
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"fmt"
"net/http"
"testing"
"github.com/minio/minio/internal/hash"
)
func TestChecksumMismatchAPIMessage(t *testing.T) {
for _, algorithm := range []string{"CRC32", "CRC32C", "SHA1", "SHA256", "CRC64NVME", ""} {
t.Run(algorithm, func(t *testing.T) {
mismatch := hash.ChecksumMismatch{Algorithm: algorithm, Want: "expected", Got: "actual"}
want := errorCodes.ToAPIErr(ErrContentChecksumMismatch).Description
if algorithm != "" {
want = fmt.Sprintf("The %s you specified did not match the calculated checksum.", algorithm)
}
for _, err := range []error{mismatch, fmt.Errorf("upload: %w", mismatch)} {
got := toAPIError(t.Context(), err)
if got.HTTPStatusCode != http.StatusBadRequest || got.Code != "XAmzContentChecksumMismatch" || got.Description != want {
t.Fatalf("toAPIError(%v) = %+v, want 400 XAmzContentChecksumMismatch: %s", err, got, want)
}
}
})
}
}
+31 -1
View File
@@ -450,6 +450,9 @@ const (
ErrAdminNoSecretKey
ErrIAMNotInitialized
ErrMultipartListingLegacy // Reserved: legacy uploads no longer block listing.
ErrMultipartListingIdentity
ErrSlowDown
apiErrCodeEnd // This is used only for the testing code
)
@@ -547,7 +550,7 @@ var errorCodes = errorCodeMap{
},
ErrBadDigest: {
Code: "BadDigest",
Description: "The Content-Md5 you specified did not match what we received.",
Description: "The Content-MD5 you specified did not match what we received.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrEntityTooSmall: {
@@ -1336,6 +1339,21 @@ var errorCodes = errorCodeMap{
Description: "IAM sub-system not initialized yet, please try again.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMultipartListingLegacy: {
Code: "MultipartListingNotReady",
Description: "Legacy multipart uploads prevent a complete listing. Upgrade all writers, drain old uploads and run the multipart preflight check.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMultipartListingIdentity: {
Code: "MultipartListingMetadataInvalid",
Description: "Multipart upload metadata is inconsistent. Run the multipart preflight check to locate the affected storage set.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrSlowDown: {
Code: "SlowDown",
Description: "Please reduce your request rate",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrBucketMetadataNotInitialized: {
Code: "XMinioBucketMetadataNotInitialized",
Description: "Bucket metadata not initialized yet, please try again.",
@@ -2173,6 +2191,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
err = unwrapAll(err)
switch err {
case errMultipartListingIdentity:
apiErr = ErrMultipartListingIdentity
case errCompleteMultipartChecksumMismatch, errCompleteMultipartChecksumTypeMismatch:
apiErr = ErrBadDigest
case errMissingPartChecksum:
@@ -2303,6 +2323,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
}
switch err.(type) {
case SlowDown:
apiErr = ErrSlowDown
case StorageFull:
apiErr = ErrStorageFull
case hash.BadDigest:
@@ -2481,6 +2503,14 @@ func toAPIError(ctx context.Context, err error) APIError {
case errors.Is(err, errMissingPartChecksum):
apiErr.Description = strings.TrimPrefix(err.Error(), errMissingPartChecksum.Error()+": ")
}
// PutObject/UploadPart checksum mismatches name the algorithm in the
// message, matching S3 (e.g. "The CRC32 you specified did not match the
// calculated checksum."). Falls through to the generic table description
// when the algorithm is unknown.
var checksumMismatch hash.ChecksumMismatch
if errors.As(err, &checksumMismatch) && checksumMismatch.Algorithm != "" {
apiErr.Description = fmt.Sprintf("The %s you specified did not match the calculated checksum.", checksumMismatch.Algorithm)
}
switch apiErr.Code {
case "NotImplemented":
apiErr = APIError{
+1 -1
View File
@@ -41,7 +41,7 @@ import (
const (
maxObjectList = 1000 // Limit number of objects in a listObjectsResponse/listObjectsVersionsResponse.
maxDeleteList = 1000 // Limit number of objects deleted in a delete call.
maxUploadsList = 10000 // Limit number of uploads in a listUploadsResponse.
maxUploadsList = 1000 // Limit number of uploads in a listUploadsResponse.
maxPartsList = 10000 // Limit number of parts in a listPartsResponse.
)
File diff suppressed because one or more lines are too long
+4 -56
View File
@@ -28,13 +28,10 @@ import (
"mime"
"net/http"
"net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
"github.com/minio/minio/internal/etag"
"github.com/minio/minio/internal/hash"
xhttp "github.com/minio/minio/internal/http"
@@ -632,18 +629,11 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
return ErrInvalidDigest
}
// Extract either 'X-Amz-Content-Sha256' header or 'X-Amz-Content-Sha256' query parameter (if V4 presigned)
// Do not verify 'X-Amz-Content-Sha256' if skipSHA256.
// Honor the selected header/query checksum, including the header fallback
// for a presigned request. STS separately hashes its body for its signature.
var contentSHA256 []byte
if skipSHA256 := skipContentSha256Cksum(r); !skipSHA256 && isRequestPresignedSignatureV4(r) {
if sha256Sum, ok := r.Form[xhttp.AmzContentSha256]; ok && len(sha256Sum) > 0 {
contentSHA256, err = hex.DecodeString(sha256Sum[0])
if err != nil {
return ErrContentSHA256Mismatch
}
}
} else if _, ok := r.Header[xhttp.AmzContentSha256]; !skipSHA256 && ok {
contentSHA256, err = hex.DecodeString(r.Header.Get(xhttp.AmzContentSha256))
if !skipContentSha256Cksum(r) {
contentSHA256, err = hex.DecodeString(getContentSha256Cksum(r, serviceS3))
if err != nil || len(contentSHA256) == 0 {
return ErrContentSHA256Mismatch
}
@@ -740,48 +730,6 @@ func setAuthMiddleware(h http.Handler) http.Handler {
})
}
func isPutRetentionAllowed(bucketName, objectName string, retDays int, retDate time.Time, retMode objectlock.RetMode, byPassSet bool, r *http.Request, cred auth.Credentials, owner bool) (s3Err APIErrorCode) {
var retSet bool
if cred.AccessKey == "" {
return ErrAccessDenied
}
conditions := getConditionValues(r, "", cred)
conditions["object-lock-mode"] = []string{string(retMode)}
conditions["object-lock-retain-until-date"] = []string{retDate.UTC().Format(time.RFC3339)}
if retDays > 0 {
conditions["object-lock-remaining-retention-days"] = []string{strconv.Itoa(retDays)}
}
if retMode == objectlock.RetGovernance && byPassSet {
byPassSet = globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.BypassGovernanceRetentionAction,
BucketName: bucketName,
ObjectName: objectName,
ConditionValues: conditions,
IsOwner: owner,
Claims: cred.Claims,
})
}
if globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.PutObjectRetentionAction,
BucketName: bucketName,
ConditionValues: conditions,
ObjectName: objectName,
IsOwner: owner,
Claims: cred.Claims,
}) {
retSet = true
}
if byPassSet || retSet {
return ErrNone
}
return ErrAccessDenied
}
// isPutActionAllowed - check if PUT operation is allowed on the resource, this
// call verifies bucket policies and IAM policies, supports multi user
// checks etc.
+5 -1
View File
@@ -583,7 +583,11 @@ func toObjectInfo(bucket, object string, objInfo minio.ObjectInfo) ObjectInfo {
func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLayer, remoteClnt *minio.Client, entries []ObjectInfo, prefix string) error {
input := make(chan minio.SnowballObject, 1)
opts := minio.SnowballOptions{
Opts: minio.PutObjectOptions{},
Opts: minio.PutObjectOptions{
Internal: minio.AdvancedPutOptions{
ReplicationRequest: true,
},
},
InMemory: *r.Source.Snowball.InMemory,
Compress: *r.Source.Snowball.Compress,
SkipErrs: *r.Source.Snowball.SkipErrs,
-19
View File
@@ -467,25 +467,6 @@ func testAdversarialHealCorsPropagatesNewerEqualValueTimestamp(obj ObjectLayer,
}
}
func TestAdversarialBucketMetadataComparisonIsBase64CaseSensitive(t *testing.T) {
upper := "QQ=="
lower := "qQ=="
upperBytes, err := base64.StdEncoding.Strict().DecodeString(upper)
if err != nil {
t.Fatal(err)
}
lowerBytes, err := base64.StdEncoding.Strict().DecodeString(lower)
if err != nil {
t.Fatal(err)
}
if string(upperBytes) == string(lowerBytes) {
t.Fatal("test inputs unexpectedly decode to the same bytes")
}
if isBucketMetadataEqual(&upper, &lower) {
t.Fatal("different decoded payloads were treated as equal")
}
}
func TestSiteReplicationStatusDetectsCorsTimestampMismatch(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
+6 -8
View File
@@ -277,14 +277,6 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
return
}
if keyMarker != "" {
// Marker not common with prefix is not implemented.
if !HasPrefix(keyMarker, prefix) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrNotImplemented), r.URL)
return
}
}
listMultipartsInfo, err := objectAPI.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
@@ -1281,6 +1273,12 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
return
}
if isDirObject(object) {
if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, false); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
}
opts.WantChecksum = checksum
fanOutOpts := fanOutOptions{Checksum: checksum}
+3 -3
View File
@@ -425,7 +425,7 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
shouldPass: true,
},
// Test case - 4.
// Setting Invalid prefix and marker combination.
// A key marker outside the prefix is valid and produces an empty page.
{
bucket: bucketName,
prefix: "asia",
@@ -435,8 +435,8 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
maxUploads: "0",
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
expectedRespStatus: http.StatusNotImplemented,
shouldPass: false,
expectedRespStatus: http.StatusOK,
shouldPass: true,
},
// Test case - 5.
// Invalid upload id and marker combination.
+19 -7
View File
@@ -632,13 +632,25 @@ func expireTransitionedObject(ctx context.Context, objectAPI ObjectLayer, oi *Ob
}
// Delete remote object from warm-tier
err := deleteObjectFromRemoteTier(ctx, oi.TransitionedObject.Name, oi.TransitionedObject.VersionID, oi.TransitionedObject.Tier)
if err == nil {
// Skip adding free version since we successfully deleted the
// remote object
opts.SkipFreeVersion = true
} else {
transitionLogIf(ctx, err)
//
// Unless the expiry only puts a delete marker on top of the version, as
// for an object that is not transitioned: with versioning enabled, or
// suspended and the version is not the null version, the version stays
// as a noncurrent version and must remain readable. Its remote object is
// removed later, through the free version added when the version itself
// is deleted.
opts.VersionSuspended = globalBucketVersioningSys.PrefixSuspended(oi.Bucket, oi.Name)
keepRemote := opts.VersionID == "" &&
(opts.Versioned || (opts.VersionSuspended && oi.VersionID != "" && oi.VersionID != nullVersionID))
if !keepRemote {
err := deleteObjectFromRemoteTier(ctx, oi.TransitionedObject.Name, oi.TransitionedObject.VersionID, oi.TransitionedObject.Tier)
if err == nil {
// Skip adding free version since we successfully deleted the
// remote object
opts.SkipFreeVersion = true
} else {
transitionLogIf(ctx, err)
}
}
// Now, delete object from hot-tier namespace
+183
View File
@@ -19,11 +19,17 @@ package cmd
import (
"bytes"
"context"
"io"
"net/http"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/bucket/lifecycle"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
xhttp "github.com/minio/minio/internal/http"
)
@@ -245,3 +251,180 @@ func TestValidateTransitionTier(t *testing.T) {
}
}
}
// tierExpiryTestBackend is a remote tier held in memory.
type tierExpiryTestBackend struct {
mu sync.Mutex
objects map[string][]byte
removed []string
}
func (b *tierExpiryTestBackend) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
return b.PutWithMeta(ctx, object, r, length, nil)
}
func (b *tierExpiryTestBackend) PutWithMeta(_ context.Context, object string, r io.Reader, _ int64, _ map[string]string) (remoteVersionID, error) {
data, err := io.ReadAll(r)
if err != nil {
return "", err
}
b.mu.Lock()
defer b.mu.Unlock()
b.objects[object] = data
return "", nil
}
func (b *tierExpiryTestBackend) Get(_ context.Context, object string, _ remoteVersionID, opts WarmBackendGetOpts) (io.ReadCloser, error) {
b.mu.Lock()
defer b.mu.Unlock()
data, ok := b.objects[object]
if !ok {
return nil, ObjectNotFound{Bucket: "tier", Object: object}
}
if opts.length > 0 {
data = data[opts.startOffset : opts.startOffset+opts.length]
}
return io.NopCloser(bytes.NewReader(data)), nil
}
func (b *tierExpiryTestBackend) Remove(_ context.Context, object string, _ remoteVersionID) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.objects, object)
b.removed = append(b.removed, object)
return nil
}
func (b *tierExpiryTestBackend) InUse(context.Context) (bool, error) { return false, nil }
// The expiry of a transitioned latest version that only puts a delete marker
// on top of it must keep its remote object, so the version stays readable as
// a noncurrent version. The remote object goes when the version is deleted.
func TestExpireTransitionedObjectKeepsNoncurrentData(t *testing.T) {
z, _ := migrationTestPools(t)
ctx := t.Context()
tier := &tierExpiryTestBackend{objects: map[string][]byte{}}
prevTier := globalTierConfigMgr
globalTierConfigMgr = NewTierConfigMgr()
globalTierConfigMgr.drivercache["WARM"] = tier
t.Cleanup(func() { globalTierConfigMgr = prevTier })
const expire = `<LifecycleConfiguration><Rule><ID>exp</ID><Status>Enabled</Status><Filter><Prefix></Prefix></Filter>` +
`<Expiration><Date>2020-01-01T00:00:00Z</Date></Expiration></Rule></LifecycleConfiguration>`
testCases := []struct {
name string
versioning string // "", Enabled or Suspended
nullLatest bool // the latest version is the null version
keep bool // the latest version survives as noncurrent
want []string // versions after the expiry, newest first
}{
{name: "versioned", versioning: "Enabled", keep: true, want: []string{"DM", "v2", "v1"}},
{name: "suspended", versioning: "Suspended", keep: true, want: []string{"DM", "v2", "v1"}},
{name: "suspended-null", versioning: "Suspended", nullLatest: true, want: []string{"DM", "v1"}},
{name: "unversioned", nullLatest: true, want: nil},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
bucket := "tier-expiry-" + tc.name
if err := z.MakeBucket(ctx, bucket, MakeBucketOptions{VersioningEnabled: tc.versioning != ""}); err != nil {
t.Fatal(err)
}
meta, err := globalBucketMetadataSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
if tc.versioning == "Suspended" {
meta.VersioningConfigXML = []byte("<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>")
}
meta.LifecycleConfigXML = []byte(expire)
if err := meta.Save(ctx, z); err != nil {
t.Fatal(err)
}
globalBucketMetadataSys.Set(bucket, meta)
// tiered writes a version whose data is in the remote tier.
labels := map[string]string{}
tiered := func(label string, opts ObjectOptions) ObjectInfo {
t.Helper()
oi := putConsistencyObject(t, z, bucket, "obj", 0, "obj "+label, opts)
vid := migrationTestVersion(oi)
infos, errs := readAllFileInfo(ctx, z.serverPools[0].getHashedSet("obj").getDisks(), "", bucket, "obj", vid, false, false)
if errs[0] != nil {
t.Fatal(errs[0])
}
fi := infos[0]
fi.TransitionStatus, fi.TransitionTier, fi.TransitionedObjName = lifecycle.TransitionComplete, "WARM", bucket+"/"+label
if _, err := tier.Put(ctx, fi.TransitionedObjName, strings.NewReader("obj "+label), -1); err != nil {
t.Fatal(err)
}
if err := z.serverPools[0].DecomTieredObject(ctx, bucket, "obj", fi, ObjectOptions{}); err != nil {
t.Fatal(err)
}
labels[vid] = label
return oi
}
var latest ObjectInfo
if tc.versioning != "" {
tiered("v1", ObjectOptions{Versioned: true})
}
if tc.nullLatest {
latest = tiered("v2", ObjectOptions{})
} else {
latest = tiered("v2", ObjectOptions{Versioned: true})
}
// Expire the latest version as the expiry worker does for the scanner.
getOpts := ObjectOptions{Versioned: tc.versioning == "Enabled", VersionSuspended: tc.versioning == "Suspended"}
oi, err := z.GetObjectInfo(ctx, bucket, "obj", getOpts)
if err != nil {
t.Fatal(err)
}
lc, err := globalLifecycleSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
evt := evalActionFromLifecycle(ctx, *lc, objectlock.Retention{}, nil, oi)
if evt.Action != lifecycle.DeleteAction || oi.TransitionedObject.Status != lifecycle.TransitionComplete {
t.Fatalf("fixture: action %v, transition status %q", evt.Action, oi.TransitionedObject.Status)
}
if err := expireTransitionedObject(ctx, z, &oi, evt, lcEventSrc_Scanner); err != nil {
t.Errorf("expiry: %v", err)
}
var got []string
for _, set := range z.serverPools[0].sets {
err := set.listObjectsToDecommission(ctx, decomBucketInfo{Name: bucket}, func(entry metaCacheEntry) {
fivs, err := entry.fileInfoVersions(bucket)
if err != nil {
t.Error(err)
return
}
for _, fi := range fivs.Versions {
if fi.Deleted {
got = append(got, "DM")
} else {
got = append(got, labels[migrationTestVersion(fi.ToObjectInfo(bucket, "obj", true))])
}
}
})
if err != nil {
t.Fatal(err)
}
}
if !slices.Equal(got, tc.want) {
t.Errorf("versions after expiry: got %v, want %v", got, tc.want)
}
remote := bucket + "/v2"
if removed := slices.Contains(tier.removed, remote); removed == tc.keep {
t.Errorf("remote object of the expired version removed: %v, want %v", removed, !tc.keep)
}
if tc.keep {
body := migrationTestBody(t, z.GetObjectNInfo, bucket, "obj", migrationTestVersion(latest))
if body != "obj v2" {
t.Errorf("v2 reads %q after the expiry, want %q", body, "obj v2")
}
}
})
}
}
+111
View File
@@ -0,0 +1,111 @@
// Copyright 2026 PGSTY contributors.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"errors"
"fmt"
"net/http"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/auth"
)
func TestDeleteBucketMetadataLockCancellation(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testDeleteBucketMetadataLockCancellation})
}
func testDeleteBucketMetadataLockCancellation(obj ObjectLayer, instanceType, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
_, unlock, err := lockBucketMetadata(t.Context(), obj, bucket)
if err != nil {
t.Fatal(err)
}
release := sync.OnceFunc(unlock)
defer release()
// Observe DeleteBucket's ACTUAL metadata.lock attempt. Set the hook after
// our own acquisition above so it only trips on the delete.
delAtLock := make(chan struct{})
var once sync.Once
hook := func(b string) {
if b == bucket {
once.Do(func() { close(delAtLock) })
}
}
lockBucketMetadataAcquireHook.Store(&hook)
defer lockBucketMetadataAcquireHook.Store(nil)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() { done <- obj.DeleteBucket(ctx, bucket, DeleteBucketOptions{Force: true, NoLock: true}) }()
select {
case <-delAtLock:
// Fixed tree: the delete reached metadata.lock and is blocking on the
// lock we hold. Cancel it and confirm it fails WITHOUT deleting, while we
// still hold the lock (release stays deferred until after the checks).
cancel()
if err := <-done; err == nil {
t.Errorf("%s: canceled deletion succeeded", instanceType)
}
if _, err := obj.GetBucketInfo(t.Context(), bucket, BucketOptions{}); err != nil {
t.Errorf("%s: bucket disappeared while metadata.lock was held: %v", instanceType, err)
}
if _, err := readBucketMetadata(t.Context(), obj, bucket); err != nil {
t.Errorf("%s: canceled deletion removed metadata: %v", instanceType, err)
}
case err := <-done:
// Broken tree: the delete finished without ever taking metadata.lock,
// i.e. it did not serialize the destructive operation behind the lock.
t.Errorf("%s: delete bypassed metadata.lock (err=%v)", instanceType, err)
}
}
func TestQueuedMetadataUpdateAfterDelete(t *testing.T) {
defer DetectTestLeak(t)()
for _, expiry := range []bool{false, true} {
t.Run(fmt.Sprintf("expiry=%v", expiry), func(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instanceType, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
previous := newObjectLayerFn()
barrier := &lcMergeBarrier{ObjectLayer: obj, bucket: bucket, mAtLock: make(chan struct{}), mProceed: make(chan struct{})}
setObjectLayer(barrier)
defer setObjectLayer(previous)
release := sync.OnceFunc(func() { close(barrier.mProceed) })
defer release()
ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
defer cancel()
ctx = context.WithValue(ctx, lcMergeWriterKey{}, "M")
done := make(chan error, 1)
go func() {
if expiry {
done <- globalBucketMetadataSys.UpdateExpiryLCConfig(ctx, bucket, nil, UTCNow())
return
}
_, err := globalBucketMetadataSys.Update(ctx, bucket, bucketTaggingConfig, []byte(`<Tagging><TagSet/></Tagging>`))
done <- err
}()
select {
case <-barrier.mAtLock:
case <-ctx.Done():
t.Fatal("writer did not reach metadata.lock")
}
if err := obj.DeleteBucket(t.Context(), bucket, DeleteBucketOptions{Force: true}); err != nil {
t.Fatal(err)
}
release()
if err := <-done; !isErrBucketNotFound(err) {
t.Errorf("%s: queued update should reject a deleted bucket, got %v", instanceType, err)
}
if _, err := readBucketMetadata(t.Context(), obj, bucket); !errors.Is(err, errConfigNotFound) && !isErrBucketNotFound(err) {
t.Errorf("%s: queued update recreated metadata: %v", instanceType, err)
}
}})
})
}
}
+278
View File
@@ -0,0 +1,278 @@
// Copyright (c) 2015-2026 MinIO, Inc.
// Copyright (c) 2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
package cmd
import (
"context"
"encoding/base64"
"errors"
"net/http"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/auth"
)
// ---------------------------------------------------------------------------
// Target 1 (issue #105): a higher-level lifecycle XML merge racing another
// bucket-metadata transition outside BucketMetadataSys.Delete.
//
// Before the fix, PeerBucketLCConfigHandler / healBucketILMExpiry read the
// current lifecycle document without metadata.lock, merged the replicated expiry
// rules with the local transition rules, and then persisted the merged blob with
// BucketMetadataSys.Update. Update re-read the record under metadata.lock but
// overwrote LifecycleConfigXML wholesale with the pre-computed blob, so any
// lifecycle transition change committed between the merge read and the merge
// write was silently lost on disk. BucketMetadataSys.UpdateExpiryLCConfig now
// performs the read, merge, and save under a single metadata.lock. This test
// drives PeerBucketLCConfigHandler and asserts the concurrent change survives.
// ---------------------------------------------------------------------------
type lcMergeWriterKey struct{}
// lcMergeBarrier pauses the merge writer (context value "M") exactly when it
// tries to take metadata.lock for its persisting Update. By that point the
// merge has already read the stale lifecycle document, so the test can commit a
// concurrent transition change before releasing the merge write.
type lcMergeBarrier struct {
ObjectLayer
bucket string
mAtLock chan struct{}
mProceed chan struct{}
mOnce sync.Once
}
func (o *lcMergeBarrier) metadataLock() string {
return pathJoin(bucketMetaPrefix, o.bucket, "metadata.lock")
}
func (o *lcMergeBarrier) NewNSLock(bucket string, objects ...string) RWLocker {
lock := o.ObjectLayer.NewNSLock(bucket, objects...)
if bucket != minioMetaBucket || len(objects) != 1 || objects[0] != o.metadataLock() {
return lock
}
return metadataObservedRWLocker{RWLocker: lock, onLock: func(ctx context.Context) {
if ctx.Value(lcMergeWriterKey{}) != "M" {
return
}
o.mOnce.Do(func() { close(o.mAtLock) })
select {
case <-o.mProceed:
case <-ctx.Done():
}
}}
}
func TestLifecycleExpiryMergeRaceLosesConcurrentTransition(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testLifecycleExpiryMergeRaceLosesConcurrentTransition,
})
}
func testLifecycleExpiryMergeRaceLosesConcurrentTransition(obj ObjectLayer, instanceType, bucket string,
_ http.Handler, _ auth.Credentials, t *testing.T,
) {
// Revision N: a single transition-only rule.
baseXML := []byte(`<LifecycleConfiguration><Rule><ID>keep</ID><Filter><Prefix>data/</Prefix></Filter><Status>Enabled</Status><Transition><Days>30</Days><StorageClass>WARM</StorageClass></Transition></Rule></LifecycleConfiguration>`)
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketLifecycleConfig, baseXML); err != nil {
t.Fatalf("%s: seed lifecycle: %v", instanceType, err)
}
// A replicated expiry-only rule arriving from a peer site.
expXML := `<LifecycleConfiguration><Rule><ID>expire</ID><Filter><Prefix>tmp/</Prefix></Filter><Status>Enabled</Status><Expiration><Days>7</Days></Expiration></Rule></LifecycleConfiguration>`
expLCConfig := base64.StdEncoding.EncodeToString([]byte(expXML))
previousObjectAPI := newObjectLayerFn()
barrier := &lcMergeBarrier{
ObjectLayer: obj,
bucket: bucket,
mAtLock: make(chan struct{}),
mProceed: make(chan struct{}),
}
setObjectLayer(barrier)
defer setObjectLayer(previousObjectAPI)
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
mCtx := context.WithValue(ctx, lcMergeWriterKey{}, "M")
mDone := make(chan error, 1)
go func() {
mDone <- globalSiteReplicationSys.PeerBucketLCConfigHandler(mCtx, bucket, &expLCConfig, UTCNow())
}()
// Wait until the merge writer has read revision N and is about to persist.
select {
case <-barrier.mAtLock:
case err := <-mDone:
t.Fatalf("%s: merge writer finished before persisting: %v", instanceType, err)
case <-ctx.Done():
t.Fatalf("%s: merge writer never reached metadata.lock: %v", instanceType, ctx.Err())
}
// Concurrent local lifecycle transition change commits revision N+1.
concurrentXML := []byte(`<LifecycleConfiguration><Rule><ID>keep</ID><Filter><Prefix>data/</Prefix></Filter><Status>Enabled</Status><Transition><Days>10</Days><StorageClass>COLD</StorageClass></Transition></Rule></LifecycleConfiguration>`)
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketLifecycleConfig, concurrentXML); err != nil {
t.Fatalf("%s: concurrent transition update: %v", instanceType, err)
}
// Release the merge write so it lands after the concurrent commit.
close(barrier.mProceed)
if err := <-mDone; err != nil {
t.Fatalf("%s: merge writer failed: %v", instanceType, err)
}
// The persisted lifecycle must contain both the replicated expiry rule and
// the concurrent transition change.
cfg, _, err := globalBucketMetadataSys.GetLifecycleConfig(bucket)
if err != nil {
t.Fatalf("%s: read merged lifecycle: %v", instanceType, err)
}
var keep, expire bool
for i := range cfg.Rules {
switch cfg.Rules[i].ID {
case "keep":
keep = true
if cfg.Rules[i].Transition.Days != 10 || cfg.Rules[i].Transition.StorageClass != "COLD" {
t.Fatalf("%s: lifecycle merge overwrote the concurrent transition change: got Days=%d StorageClass=%q, want Days=10 StorageClass=COLD",
instanceType, cfg.Rules[i].Transition.Days, cfg.Rules[i].Transition.StorageClass)
}
case "expire":
expire = true
}
}
if !expire {
t.Fatalf("%s: merged lifecycle dropped the replicated expiry rule: %+v", instanceType, cfg.Rules)
}
if !keep {
t.Fatalf("%s: merged lifecycle dropped the transition rule entirely: %+v", instanceType, cfg.Rules)
}
}
// ---------------------------------------------------------------------------
// Target 2 (issue #105): DeleteBucket racing an in-flight metadata writer must
// not resurrect a ghost .metadata.bin record.
//
// Before the fix, erasureServerPools.DeleteBucket took only <bucket>.lck and
// purged the metadata prefix while config writers (updateAndParse) took only
// metadata.lock, so a writer already MID-SAVE (holding metadata.lock, past
// saveMetadata's existence recheck) could persist .metadata.bin after the purge.
// DeleteBucket now takes metadata.lock before deleting, so it waits for that
// writer and then purges whatever the writer wrote.
//
// This test isolates the DeleteBucket-lock fix specifically: the writer holds
// metadata.lock and is paused at the .metadata.bin PutObject, so the earlier
// saveMetadata existence recheck cannot save it — only serializing the delete
// behind the writer can. Removing just DeleteBucket's metadata.lock (keeping the
// recheck) therefore makes this test fail. The delete's ACTUAL metadata.lock
// attempt is observed with lockBucketMetadataAcquireHook (its lock is taken
// through the erasureServerPools receiver, invisible to the object-layer
// barrier), so the handshake is deterministic with no timing assumption.
// ---------------------------------------------------------------------------
func TestDeleteBucketResurrectsGhostMetadata(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testDeleteBucketResurrectsGhostMetadata,
})
}
func testDeleteBucketResurrectsGhostMetadata(obj ObjectLayer, instanceType, bucket string,
_ http.Handler, _ auth.Credentials, t *testing.T,
) {
previousObjectAPI := newObjectLayerFn()
// Writer A holds metadata.lock and pauses at the .metadata.bin PutObject,
// i.e. already past saveMetadata's existence recheck and mid-save.
barrier := &metadataRMWBarrierObjectLayer{
ObjectLayer: obj,
bucket: bucket,
aReady: make(chan struct{}),
aRelease: make(chan struct{}),
bLockAttempt: make(chan struct{}),
}
setObjectLayer(barrier)
defer setObjectLayer(previousObjectAPI)
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
aCtx := context.WithValue(ctx, metadataRMWWriterKey{}, "A")
policyJSON := []byte(`{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::` + bucket + `/*"}]}`)
aReleased := sync.OnceFunc(func() { close(barrier.aRelease) })
defer aReleased()
aDone := make(chan error, 1)
go func() {
_, err := globalBucketMetadataSys.Update(aCtx, bucket, bucketPolicyConfig, policyJSON)
aDone <- err
}()
select {
case <-barrier.aReady:
case err := <-aDone:
t.Fatalf("%s: writer A finished before persisting: %v", instanceType, err)
case <-ctx.Done():
t.Fatalf("%s: writer A never reached metadata save: %v", instanceType, ctx.Err())
}
// A now holds metadata.lock mid-save. Observe DeleteBucket's ACTUAL
// metadata.lock attempt via the acquire hook, set only now so A's earlier
// acquisition does not trip it.
delAtLock := make(chan struct{})
var once sync.Once
hook := func(b string) {
if b == bucket {
once.Do(func() { close(delAtLock) })
}
}
lockBucketMetadataAcquireHook.Store(&hook)
defer lockBucketMetadataAcquireHook.Store(nil)
delDone := make(chan error, 1)
go func() {
delDone <- obj.DeleteBucket(ctx, bucket, DeleteBucketOptions{Force: true})
}()
select {
case <-delAtLock:
// Fixed tree: DeleteBucket reached metadata.lock and blocks on A. Release
// A so it finishes its save and unlocks; the delete then acquires the
// lock and purges the record A wrote.
aReleased()
if err := <-aDone; err != nil {
t.Fatalf("%s: writer A save failed while holding metadata.lock: %v", instanceType, err)
}
if err := <-delDone; err != nil {
t.Fatalf("%s: delete bucket: %v", instanceType, err)
}
case err := <-delDone:
// Broken tree: DeleteBucket purged without taking metadata.lock. Release
// A so its mid-save PutObject recreates .metadata.bin (the ghost).
if err != nil {
t.Fatalf("%s: delete bucket: %v", instanceType, err)
}
aReleased()
if err := <-aDone; err != nil {
t.Fatalf("%s: writer A save failed: %v", instanceType, err)
}
}
// After DeleteBucket, no .metadata.bin record may remain on disk.
if meta, err := readBucketMetadata(ctx, obj, bucket); err == nil {
t.Fatalf("%s: ghost .metadata.bin resurrected after DeleteBucket: name=%q created=%s policyLen=%d",
instanceType, meta.Name, meta.Created, len(meta.PolicyConfigJSON))
} else if !errors.Is(err, errConfigNotFound) && !isErrBucketNotFound(err) && !errors.Is(err, errVolumeNotFound) {
t.Fatalf("%s: unexpected error reading deleted bucket metadata: %v", instanceType, err)
}
}
+54
View File
@@ -0,0 +1,54 @@
// Copyright 2026 PGSTY contributors.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"net/http"
"testing"
"time"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/grid"
)
func TestPeerMetadataReloadWithEqualMaximumTimestamp(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testPeerMetadataReloadWithEqualMaximumTimestamp})
}
func testPeerMetadataReloadWithEqualMaximumTimestamp(obj ObjectLayer, instanceType, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
disk, err := loadBucketMetadata(t.Context(), obj, bucket)
if err != nil {
t.Fatal(err)
}
disk.TaggingConfigXML = []byte(`<Tagging><TagSet><Tag><Key>revision</Key><Value>new</Value></Tag></TagSet></Tagging>`)
disk.TaggingConfigUpdatedAt = UTCNow()
// An unrelated configuration has the greatest timestamp in both records.
disk.PolicyConfigUpdatedAt = disk.TaggingConfigUpdatedAt.Add(time.Hour)
if err := disk.Save(t.Context(), obj); err != nil {
t.Fatal(err)
}
resident := disk
resident.TaggingConfigXML = []byte(`<Tagging><TagSet><Tag><Key>revision</Key><Value>old</Value></Tag></TagSet></Tagging>`)
resident.TaggingConfigUpdatedAt = disk.TaggingConfigUpdatedAt.Add(-time.Minute)
if err := resident.parseAllConfigs(t.Context(), obj); err != nil {
t.Fatal(err)
}
if !resident.lastUpdate().Equal(disk.lastUpdate()) {
t.Fatal("fixture must have equal maximum timestamps")
}
globalBucketMetadataSys.Set(bucket, resident)
args := grid.MSS{peerRESTBucket: bucket}
if _, err := (&peerRESTServer{}).LoadBucketMetadataHandler(&args); err != nil {
t.Fatal(err)
}
tagging, _, err := globalBucketMetadataSys.GetTaggingConfig(bucket)
if err != nil {
t.Fatal(err)
}
if got := tagging.ToMap()["revision"]; got != "new" {
t.Errorf("%s: peer reload retained tag %q despite a newer tagging configuration", instanceType, got)
}
}
+330
View File
@@ -0,0 +1,330 @@
// Copyright (c) 2015-2026 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"encoding/json"
"errors"
"sort"
"time"
"github.com/minio/minio-go/v7/pkg/tags"
bucketsse "github.com/minio/minio/internal/bucket/encryption"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
"github.com/minio/minio/internal/bucket/versioning"
"github.com/pgsty/silo-pkg/v3/policy"
)
// Only these fields share the site-replication source-time ordering contract.
// Bulk apply/import process Object Lock before Versioning, whose effective
// document depends on it. Periodic heal retains its existing type order.
var replicatedBucketConfigs = [...]string{
objectLockConfig, bucketVersioningConfig, bucketPolicyConfig,
bucketTaggingConfig, bucketSSEConfig, bucketQuotaConfigFile,
}
func replicatedBucketConfig(meta *BucketMetadata, file string) (*[]byte, *time.Time) {
switch file {
case bucketPolicyConfig:
return &meta.PolicyConfigJSON, &meta.PolicyConfigUpdatedAt
case bucketTaggingConfig:
return &meta.TaggingConfigXML, &meta.TaggingConfigUpdatedAt
case bucketSSEConfig:
return &meta.EncryptionConfigXML, &meta.EncryptionConfigUpdatedAt
case bucketQuotaConfigFile:
return &meta.QuotaConfigJSON, &meta.QuotaConfigUpdatedAt
case bucketVersioningConfig:
return &meta.VersioningConfigXML, &meta.VersioningConfigUpdatedAt
case objectLockConfig:
return &meta.ObjectLockConfigXML, &meta.ObjectLockConfigUpdatedAt
}
return nil, nil
}
// Callers that only need to know whether a file is under the contract must not
// probe replicatedBucketConfig with a throwaway BucketMetadata.
func isReplicatedBucketConfig(file string) bool {
for _, replicated := range replicatedBucketConfigs {
if replicated == file {
return true
}
}
return false
}
func bucketConfigUpdateOnly(file string) bool {
return file == bucketVersioningConfig || file == objectLockConfig
}
// Reuse the persistence rule before comparison, so an accepted Versioning
// event and the document Save actually writes have the same comparison key.
func effectiveBucketVersioning(data []byte, lockEnabled bool) []byte {
if lockEnabled {
config, err := versioning.ParseConfig(bytes.NewReader(data))
if err != nil || !config.Enabled() || config.PrefixesExcluded() {
return enabledBucketVersioningConfig
}
}
return data
}
// A parsed policy still contains map-backed sets with nondeterministic Marshal
// order. Sort every set array recursively, including statements and conditions.
// RawMessage keeps integer values intact; decoding through float64 would not.
func canonicalBucketPolicyJSON(data json.RawMessage) (json.RawMessage, error) {
data = bytes.TrimSpace(data)
if len(data) == 0 {
return nil, nil
}
switch data[0] {
case '{':
var obj map[string]json.RawMessage
if err := json.Unmarshal(data, &obj); err != nil {
return nil, err
}
for key, value := range obj {
var err error
obj[key], err = canonicalBucketPolicyJSON(value)
if err != nil {
return nil, err
}
}
return json.Marshal(obj)
case '[':
var arr []json.RawMessage
if err := json.Unmarshal(data, &arr); err != nil {
return nil, err
}
for i := range arr {
var err error
arr[i], err = canonicalBucketPolicyJSON(arr[i])
if err != nil {
return nil, err
}
}
sort.Slice(arr, func(i, j int) bool { return bytes.Compare(arr[i], arr[j]) < 0 })
return json.Marshal(arr)
default:
var compact bytes.Buffer
if err := json.Compact(&compact, data); err != nil {
return nil, err
}
return compact.Bytes(), nil
}
}
// Encode the validated policy fields explicitly: BPStatement's required
// Action/Resource tags otherwise try to marshal empty sets for the supported
// NotAction/NotResource alternatives. This stays within the existing schema.
func canonicalBucketPolicy(cfg *policy.BucketPolicy) ([]byte, error) {
if cfg.IsEmpty() {
return nil, nil
}
doc := map[string]any{"Version": cfg.Version}
if cfg.ID != "" {
doc["ID"] = cfg.ID
}
statements := make([]map[string]any, 0, len(cfg.Statements))
for _, st := range cfg.Statements {
statement := map[string]any{"Effect": st.Effect, "Principal": st.Principal}
if st.SID != "" {
statement["Sid"] = st.SID
}
if len(st.Actions) != 0 {
statement["Action"] = st.Actions
}
if len(st.NotActions) != 0 {
statement["NotAction"] = st.NotActions
}
if len(st.Resources) != 0 {
statement["Resource"] = st.Resources
}
if len(st.NotResources) != 0 {
statement["NotResource"] = st.NotResources
}
if len(st.Conditions) != 0 {
statement["Condition"] = st.Conditions
}
statements = append(statements, statement)
}
doc["Statement"] = statements
data, err := json.Marshal(doc)
if err != nil {
return nil, err
}
return canonicalBucketPolicyJSON(data)
}
// Validate with the same parsers as Save. Policy's established empty-policy
// semantics are deletion; a parsed zero quota is still a live document.
func bucketConfigPayload(bucket, file string, data []byte, lockEnabled bool) ([]byte, []byte, error) {
if len(data) == 0 {
return nil, nil, nil
}
var err error
key := data
switch file {
case bucketPolicyConfig:
var cfg *policy.BucketPolicy
cfg, err = policy.ParseBucketPolicyConfig(bytes.NewReader(data), bucket)
if err == nil {
if cfg.IsEmpty() {
return nil, nil, nil
}
key, err = canonicalBucketPolicy(cfg)
}
case bucketQuotaConfigFile:
cfg, parseErr := parseBucketQuota(bucket, data)
err = parseErr
if err == nil {
key, err = json.Marshal(cfg)
}
case bucketTaggingConfig:
_, err = tags.ParseBucketXML(bytes.NewReader(data))
case bucketSSEConfig:
_, err = bucketsse.ParseBucketSSEConfig(bytes.NewReader(data))
case objectLockConfig:
_, err = objectlock.ParseObjectLockConfig(bytes.NewReader(data))
case bucketVersioningConfig:
data = effectiveBucketVersioning(data, lockEnabled)
key = data
_, err = versioning.ParseConfig(bytes.NewReader(data))
}
return data, key, err
}
type bucketConfigState struct {
data, key []byte
at time.Time
real, valid bool
}
func newBucketConfigState(bucket, file string, data []byte, at, created time.Time, lockEnabled bool) (bucketConfigState, error) {
data, key, err := bucketConfigPayload(bucket, file, data, lockEnabled)
if err != nil {
return bucketConfigState{}, err
}
if at.IsZero() {
at = created
}
valid := !created.IsZero() && !at.Before(created)
modified := valid && at.After(created)
if bucketConfigUpdateOnly(file) && len(data) == 0 {
modified = false
}
return bucketConfigState{data: data, key: key, at: at, real: modified, valid: valid}, nil
}
func (s bucketConfigState) candidate() bool {
return s.valid && (s.real || len(s.data) != 0)
}
func compareBucketConfigStates(a, b bucketConfigState) int {
if a.valid != b.valid {
if a.valid {
return 1
}
return -1
}
if a.real != b.real {
if a.real {
return 1
}
return -1
}
if a.real {
if n := a.at.Compare(b.at); n != 0 {
return n
}
// At equal source time a real deletion wins, preventing resurrection.
if (len(a.data) == 0) != (len(b.data) == 0) {
if len(a.data) == 0 {
return 1
}
return -1
}
}
return bytes.Compare(a.key, b.key)
}
func localBucketConfigUpdatedAt(meta BucketMetadata, file string, now time.Time) time.Time {
_, at := replicatedBucketConfig(&meta, file)
for _, lower := range []time.Time{meta.Created, *at} {
if !now.After(lower) {
now = lower.Add(time.Nanosecond)
}
}
return now.UTC()
}
func ensureBucketMetadataCreated(ctx context.Context, obj ObjectLayer, meta *BucketMetadata) error {
if !meta.Created.IsZero() {
return nil
}
info, err := obj.GetBucketInfo(ctx, meta.Name, BucketOptions{NoMetadata: true})
if err != nil {
return err
}
if info.Created.IsZero() {
return errors.New("bucket metadata creation time is unknown")
}
meta.Created = info.Created.UTC()
return nil
}
// applyBucketConfig runs under metadata.lock, on freshly loaded metadata. It
// changes only the selected field; the caller persists once after all checks.
func applyBucketConfig(meta *BucketMetadata, file string, data []byte, at time.Time) (bool, error) {
if bucketConfigUpdateOnly(file) && len(data) == 0 {
return false, nil
}
current, currentAt := replicatedBucketConfig(meta, file)
lockEnabled := len(meta.ObjectLockConfigXML) != 0
incoming, err := newBucketConfigState(meta.Name, file, data, at, meta.Created, lockEnabled)
if err != nil {
return false, err
}
if !incoming.candidate() {
return false, nil
}
local, err := newBucketConfigState(meta.Name, file, *current, *currentAt, meta.Created, lockEnabled)
if err != nil {
return false, err
}
if compareBucketConfigStates(incoming, local) <= 0 {
return false, nil
}
*current, *currentAt = bytes.Clone(incoming.data), incoming.at.UTC()
return true, nil
}
func rebaseBucketConfigDefaults(meta *BucketMetadata, oldCreated time.Time) {
if meta.Created.Equal(oldCreated) {
return
}
// These six fields alone use Created to distinguish a baseline from a
// tombstone. Preserve actual source times when adopting an existing bucket.
for _, file := range replicatedBucketConfigs {
_, at := replicatedBucketConfig(meta, file)
if at.IsZero() || at.Equal(oldCreated) {
*at = meta.Created
}
}
}
+160 -51
View File
@@ -24,6 +24,7 @@ import (
"fmt"
"math/rand"
"sync"
"sync/atomic"
"time"
"github.com/minio/madmin-go/v3"
@@ -125,19 +126,42 @@ func (sys *BucketMetadataSys) Set(bucket string, meta BucketMetadata) {
}
}
func (sys *BucketMetadataSys) updateAndParse(ctx context.Context, bucket string, configFile string, configData []byte, parse, lifecycleDelete bool) (updatedAt time.Time, err error) {
// bucketMetadataUpdate returns the committed snapshot to the caller. meta and
// updatedAt hold the saved state only when changed is true. Local writes always
// change state, because localBucketConfigUpdatedAt is strictly greater than the
// current field time, so their handlers can broadcast meta without rechecking.
type bucketMetadataUpdate struct {
meta BucketMetadata
updatedAt time.Time
changed bool
}
func (sys *BucketMetadataSys) updateAndParse(ctx context.Context, bucket, configFile string, configData []byte, parse, lifecycleDelete bool) (time.Time, error) {
result, err := sys.updateAndParseMetadata(ctx, bucket, configFile, configData, parse, lifecycleDelete, nil)
return result.updatedAt, err
}
func (sys *BucketMetadataSys) updateAndParseMetadata(ctx context.Context, bucket string, configFile string, configData []byte, parse, lifecycleDelete bool, sourceTime *time.Time) (result bucketMetadataUpdate, err error) {
objAPI := newObjectLayerFn()
if objAPI == nil {
return updatedAt, errServerNotInitialized
return result, errServerNotInitialized
}
if isMinioMetaBucketName(bucket) {
return updatedAt, errInvalidArgument
return result, errInvalidArgument
}
// Load deletions without parsed caches (notably quota), and compare the
// six replicated fields against the raw document under the same lock.
if isReplicatedBucketConfig(configFile) {
parse = false
if bucketConfigUpdateOnly(configFile) && len(configData) == 0 {
return result, nil
}
}
notifyCtx := ctx
ctx, unlock, err := lockBucketMetadata(ctx, objAPI, bucket)
if err != nil {
return updatedAt, err
return result, err
}
err = func() error {
@@ -157,55 +181,73 @@ func (sys *BucketMetadataSys) updateAndParse(ctx context.Context, bucket string,
return err
}
}
updatedAt = UTCNow()
switch configFile {
case bucketPolicyConfig:
meta.PolicyConfigJSON = configData
meta.PolicyConfigUpdatedAt = updatedAt
case bucketNotificationConfig:
meta.NotificationConfigXML = configData
meta.NotificationConfigUpdatedAt = updatedAt
case bucketLifecycleConfig:
meta.LifecycleConfigXML = configData
meta.LifecycleConfigUpdatedAt = updatedAt
case bucketSSEConfig:
meta.EncryptionConfigXML = configData
meta.EncryptionConfigUpdatedAt = updatedAt
case bucketTaggingConfig:
meta.TaggingConfigXML = configData
meta.TaggingConfigUpdatedAt = updatedAt
case bucketQuotaConfigFile:
meta.QuotaConfigJSON = configData
meta.QuotaConfigUpdatedAt = updatedAt
case objectLockConfig:
meta.ObjectLockConfigXML = configData
meta.ObjectLockConfigUpdatedAt = updatedAt
case bucketVersioningConfig:
meta.VersioningConfigXML = configData
meta.VersioningConfigUpdatedAt = updatedAt
case bucketReplicationConfig:
meta.ReplicationConfigXML = configData
meta.ReplicationConfigUpdatedAt = updatedAt
case bucketTargetsFile:
meta.BucketTargetsConfigJSON, meta.BucketTargetsConfigMetaJSON, err = encryptBucketMetadata(ctx, meta.Name, configData, kms.Context{
bucket: meta.Name,
bucketTargetsFile: bucketTargetsFile,
})
if err != nil {
return fmt.Errorf("Error encrypting bucket target metadata %w", err)
updatedAt := UTCNow()
if isReplicatedBucketConfig(configFile) {
if err := ensureBucketMetadataCreated(ctx, objAPI, &meta); err != nil {
var at time.Time
if sourceTime != nil {
at = *sourceTime
}
logBucketConfigReplication(ctx, bucket, configFile, "indeterminate", at, meta.Created, err.Error())
return err
}
if sourceTime == nil || sourceTime.IsZero() {
updatedAt = localBucketConfigUpdatedAt(meta, configFile, updatedAt)
if sourceTime != nil {
logBucketConfigReplication(ctx, bucket, configFile, "legacy-zero", *sourceTime, meta.Created, "assigned local source time")
}
} else {
updatedAt = sourceTime.UTC()
}
if updatedAt.Before(meta.Created) {
logBucketConfigReplication(ctx, bucket, configFile, "before-created", updatedAt, meta.Created, "peer event")
return nil
}
changed, err := applyBucketConfig(&meta, configFile, configData, updatedAt)
if err != nil {
return err
}
if !changed {
return nil
}
} else {
switch configFile {
case bucketNotificationConfig:
meta.NotificationConfigXML = configData
meta.NotificationConfigUpdatedAt = updatedAt
case bucketLifecycleConfig:
meta.LifecycleConfigXML = configData
meta.LifecycleConfigUpdatedAt = updatedAt
case bucketReplicationConfig:
meta.ReplicationConfigXML = configData
meta.ReplicationConfigUpdatedAt = updatedAt
case bucketTargetsFile:
meta.BucketTargetsConfigJSON, meta.BucketTargetsConfigMetaJSON, err = encryptBucketMetadata(ctx, meta.Name, configData, kms.Context{
bucket: meta.Name,
bucketTargetsFile: bucketTargetsFile,
})
if err != nil {
return fmt.Errorf("Error encrypting bucket target metadata %w", err)
}
meta.BucketTargetsConfigUpdatedAt = updatedAt
meta.BucketTargetsConfigMetaUpdatedAt = updatedAt
default:
return fmt.Errorf("Unknown bucket %s metadata update requested %s", bucket, configFile)
}
meta.BucketTargetsConfigUpdatedAt = updatedAt
meta.BucketTargetsConfigMetaUpdatedAt = updatedAt
default:
return fmt.Errorf("Unknown bucket %s metadata update requested %s", bucket, configFile)
}
return sys.saveMetadata(ctx, objAPI, meta)
if err := sys.saveMetadata(ctx, objAPI, &meta); err != nil {
return err
}
result = bucketMetadataUpdate{meta: meta, updatedAt: updatedAt, changed: true}
return nil
}()
if err != nil {
return updatedAt, err
return result, err
}
globalNotificationSys.LoadBucketMetadata(bgContext(notifyCtx), bucket) // Do not use caller context here
return updatedAt, nil
if result.changed {
globalNotificationSys.LoadBucketMetadata(bgContext(notifyCtx), bucket)
}
return result, nil
}
func (sys *BucketMetadataSys) save(ctx context.Context, meta BucketMetadata) error {
@@ -218,7 +260,7 @@ func (sys *BucketMetadataSys) save(ctx context.Context, meta BucketMetadata) err
return errInvalidArgument
}
if err := sys.saveMetadata(ctx, objAPI, meta); err != nil {
if err := sys.saveMetadata(ctx, objAPI, &meta); err != nil {
return err
}
@@ -228,11 +270,16 @@ func (sys *BucketMetadataSys) save(ctx context.Context, meta BucketMetadata) err
// saveMetadata persists and publishes metadata locally. Callers performing a
// read-modify-write must hold metadata.lock and release it before peer fan-out.
func (sys *BucketMetadataSys) saveMetadata(ctx context.Context, objAPI ObjectLayer, meta BucketMetadata) error {
func (sys *BucketMetadataSys) saveMetadata(ctx context.Context, objAPI ObjectLayer, meta *BucketMetadata) error {
// A writer may have queued for metadata.lock before DeleteBucket completed.
// Recheck the physical bucket under that lock, before recreating metadata.
if _, err := objAPI.GetBucketInfo(ctx, meta.Name, BucketOptions{NoMetadata: true}); err != nil {
return err
}
if err := meta.Save(ctx, objAPI); err != nil {
return err
}
sys.Set(meta.Name, meta)
sys.Set(meta.Name, *meta)
return nil
}
@@ -240,8 +287,19 @@ func lockBucketMetadata(ctx context.Context, objectAPI ObjectLayer, bucket strin
return lockBucketMetadataWithTimeout(ctx, objectAPI, bucket, globalOperationTimeout)
}
// lockBucketMetadataAcquireHook, when set, is invoked at the start of every
// metadata.lock acquisition, immediately before the blocking Lock() call. It is
// nil in production (a single atomic load, no behavior change) and exists only
// so tests can deterministically observe a caller reaching the metadata lock —
// notably DeleteBucket, whose lock is taken through its erasureServerPools
// receiver and is therefore invisible to an injected object layer.
var lockBucketMetadataAcquireHook atomic.Pointer[func(bucket string)]
func lockBucketMetadataWithTimeout(ctx context.Context, objectAPI ObjectLayer, bucket string, timeout *dynamicTimeout) (context.Context, func(), error) {
lock := objectAPI.NewNSLock(minioMetaBucket, pathJoin(bucketMetaPrefix, bucket, "metadata.lock"))
if hook := lockBucketMetadataAcquireHook.Load(); hook != nil {
(*hook)(bucket)
}
lkctx, err := lock.GetLock(ctx, timeout)
if err != nil {
return nil, nil, err
@@ -292,6 +350,57 @@ func (sys *BucketMetadataSys) Update(ctx context.Context, bucket string, configF
return sys.updateAndParse(ctx, bucket, configFile, configData, true, false)
}
// UpdateExpiryLCConfig merges a replicated ILM expiry configuration with the
// bucket's current lifecycle document and persists the merged result while
// holding metadata.lock across the read, merge, and save. The site-replication
// expiry heal and peer-apply paths must use this instead of computing the merge
// from an unlocked GetConfigFromDisk read and then writing it with Update: that
// two-step sequence drops any lifecycle transition change committed in between
// (issue #105). Lock order stays <bucket>.lck -> metadata.lock -> .metadata.bin;
// the merge and save run under metadata.lock and the peer fan-out runs after it
// is released.
func (sys *BucketMetadataSys) UpdateExpiryLCConfig(ctx context.Context, bucket string, expLCConfig *string, updatedAt time.Time) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
if isMinioMetaBucketName(bucket) {
return errInvalidArgument
}
notifyCtx := ctx
ctx, unlock, err := lockBucketMetadata(ctx, objAPI, bucket)
if err != nil {
return err
}
err = func() error {
defer unlock()
meta, err := loadBucketMetadataParse(ctx, objAPI, bucket, true)
if err != nil {
if !globalIsErasure && !globalIsDistErasure && errors.Is(err, errVolumeNotFound) {
// Only single drive mode needs this fallback.
meta = newBucketMetadata(bucket)
} else {
return err
}
}
configData, err := mergeExpiryWithLCConfig(bucket, meta, expLCConfig, updatedAt)
if err != nil {
return err
}
meta.LifecycleConfigXML = configData
meta.LifecycleConfigUpdatedAt = UTCNow()
return sys.saveMetadata(ctx, objAPI, &meta)
}()
if err != nil {
return err
}
globalNotificationSys.LoadBucketMetadata(bgContext(notifyCtx), bucket) // Do not use caller context here
return nil
}
// Get metadata for a bucket.
// If no metadata exists errConfigNotFound is returned and a new metadata is returned.
// Only a shallow copy is returned, so referenced data should not be modified,
+4 -9
View File
@@ -303,6 +303,9 @@ func loadBucketMetadataParseUnderLock(ctx context.Context, objectAPI ObjectLayer
return newBucketMetadata(bucket), fmt.Errorf("%w: %v", errBucketMetadataMigrationLockUnavailable, err)
}
defer unlock()
if _, err := objectAPI.GetBucketInfo(ctx, bucket, BucketOptions{NoMetadata: true}); err != nil {
return newBucketMetadata(bucket), err
}
return loadBucketMetadataParse(ctx, objectAPI, bucket, parse)
}
@@ -385,15 +388,7 @@ func (b *BucketMetadata) parseAllConfigs(ctx context.Context, objectAPI ObjectLa
} else {
b.objectLockConfig = nil
}
if b.objectLockConfig != nil {
// Object Lock requires every object to be versioned. Whatever the lock
// document contains, a suspended or prefix-excluded versioning document
// is replaced by plain Enabled versioning; Save persists the result.
config, versioningErr := versioning.ParseConfig(bytes.NewReader(b.VersioningConfigXML))
if versioningErr != nil || !config.Enabled() || config.PrefixesExcluded() {
b.VersioningConfigXML = enabledBucketVersioningConfig
}
}
b.VersioningConfigXML = effectiveBucketVersioning(b.VersioningConfigXML, b.objectLockConfig != nil)
if len(b.VersioningConfigXML) != 0 {
b.versioningConfig, err = versioning.ParseConfig(bytes.NewReader(b.VersioningConfigXML))
+25
View File
@@ -39,3 +39,28 @@ func TestBucketMetadataCorsRoundTrip(t *testing.T) {
t.Fatalf("CorsConfigUpdatedAt not preserved")
}
}
// A persisted retired extension must not prevent the whole bucket's metadata
// from loading, including unrelated versioning and ordinary lifecycle rules.
func TestBucketMetadataRetiredAccessTiering(t *testing.T) {
meta := newBucketMetadata("retired-access")
meta.LifecycleConfigXML = []byte(`<LifecycleConfiguration><AccessTierQuota>500GiB</AccessTierQuota><Rule><ID>access</ID><Status>Enabled</Status><Filter><Prefix>logs/</Prefix></Filter><AccessTransition><Window>10m</Window><PromoteAfterAccesses>10</PromoteAfterAccesses></AccessTransition></Rule><Rule><ID>ordinary</ID><Status>Enabled</Status><Filter><Prefix>expired/</Prefix></Filter><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>`)
meta.VersioningConfigXML = []byte(`<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Enabled</Status></VersioningConfiguration>`)
data, err := meta.MarshalMsg(nil)
if err != nil {
t.Fatal(err)
}
got := newBucketMetadata(meta.Name)
if _, err := got.UnmarshalMsg(data); err != nil {
t.Fatal(err)
}
if err := got.parseAllConfigs(t.Context(), nil); err != nil {
t.Fatalf("bucket metadata failed to load: %v", err)
}
if got.lifecycleConfig == nil || got.lifecycleConfig.HasActiveRules("logs/") || !got.lifecycleConfig.HasActiveRules("expired/") {
t.Fatal("unexpected lifecycle behavior")
}
if got.versioningConfig == nil || !got.versioningConfig.Enabled() {
t.Fatal("unrelated versioning lost")
}
}
+126 -66
View File
@@ -20,8 +20,8 @@ package cmd
import (
"context"
"errors"
"math"
"net/http"
"strconv"
"strings"
"time"
@@ -164,76 +164,129 @@ func enforceRetentionBypassForDelete(ctx context.Context, r *http.Request, bucke
return nil
}
// enforceRetentionBypassForPut enforces whether an existing object under governance can be overwritten
// with governance bypass headers set in the request.
// Objects under site wide WORM cannot be overwritten.
// For objects in "Governance" mode, overwrite is allowed if a) object retention date is past OR
// governance bypass headers are set and user has governance bypass permissions.
// Objects in compliance mode can be overwritten only if retention date is being extended. No mode change is permitted.
// isRetentionActionAllowed evaluates action for a PutObjectRetention request
// with the object-lock condition keys derived from the requested retention.
// The keys come from the request, not from the stored version, so the
// handler can decide s3:PutObjectRetention before reading any object state.
func isRetentionActionAllowed(action policy.Action, bucket, object string, objRetention *objectlock.ObjectRetention, r *http.Request, cred auth.Credentials, owner bool) bool {
if cred.AccessKey == "" {
return false
}
t, err := objectlock.UTCNowNTP()
if err != nil {
return false
}
conditions := getConditionValues(r, "", cred)
conditions["object-lock-mode"] = []string{string(objRetention.Mode)}
conditions["object-lock-retain-until-date"] = []string{objRetention.RetainUntilDate.UTC().Format(time.RFC3339)}
// Relative days from now, for "object-lock-remaining-retention-days".
// Clearing retention (<Retention/>) has a zero RetainUntilDate, which must
// evaluate as 0 remaining days rather than a large positive math.Abs delta
// from year 0001.
days := 0
if !objRetention.RetainUntilDate.IsZero() && objRetention.RetainUntilDate.After(t) {
// Time.Sub saturates after about 292 years. Count seconds instead so
// maximum-retention policies also apply to distant RFC3339 dates.
seconds := objRetention.RetainUntilDate.Unix() - t.Unix()
days = int(seconds / 86400)
if seconds%86400 != 0 || objRetention.RetainUntilDate.Nanosecond() > t.Nanosecond() {
days++
}
}
conditions["object-lock-remaining-retention-days"] = []string{strconv.Itoa(days)}
return globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: action,
BucketName: bucket,
ConditionValues: conditions,
ObjectName: object,
IsOwner: owner,
Claims: cred.Claims,
})
}
// enforceRetentionBypassForPut enforces the stored retention of oi against a
// PutObjectRetention request. The caller must already hold
// s3:PutObjectRetention; this only decides changes that depend on the stored
// state. Expired or absent retention may be changed freely. Unexpired
// GOVERNANCE retention may be shortened, removed or have its mode changed only
// with the bypass header and s3:BypassGovernanceRetention. Unexpired
// COMPLIANCE retention may only be extended.
func enforceRetentionBypassForPut(ctx context.Context, r *http.Request, oi ObjectInfo, objRetention *objectlock.ObjectRetention, cred auth.Credentials, owner bool) error {
byPassSet := objectlock.IsObjectLockGovernanceBypassSet(r.Header)
ret := objectlock.GetObjectRetentionMeta(oi.UserDefined)
if !ret.Mode.Valid() {
return nil
}
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
if ret.RetainUntilDate.Before(t) {
return nil
}
// Pass in relative days from current time, to additionally
// to verify "object-lock-remaining-retention-days" policy if any.
days := int(math.Ceil(math.Abs(objRetention.RetainUntilDate.Sub(t).Hours()) / 24))
ret := objectlock.GetObjectRetentionMeta(oi.UserDefined)
if ret.Mode.Valid() {
// Retention has expired you may change whatever you like.
if ret.RetainUntilDate.Before(t) {
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time,
objRetention.Mode, byPassSet, r, cred,
owner)
if apiErr == ErrAccessDenied {
return errAuthentication
}
return nil
weakens := objRetention.Mode != ret.Mode || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time)
if !weakens {
return nil
}
switch ret.Mode {
case objectlock.RetGovernance:
// https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-bypass-governance
if !objectlock.IsObjectLockGovernanceBypassSet(r.Header) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
switch ret.Mode {
case objectlock.RetGovernance:
govPerm := isPutRetentionAllowed(oi.Bucket, oi.Name, days,
objRetention.RetainUntilDate.Time, objRetention.Mode,
byPassSet, r, cred, owner)
// Governance mode retention period cannot be shortened, if x-amz-bypass-governance is not set.
if !byPassSet {
if objRetention.Mode != objectlock.RetGovernance || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
}
if govPerm == ErrAccessDenied {
return errAuthentication
}
return nil
case objectlock.RetCompliance:
// Compliance retention mode cannot be changed or shortened.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html#object-lock-retention-modes
if objRetention.Mode != objectlock.RetCompliance || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time, objRetention.Mode,
false, r, cred, owner)
if apiErr == ErrAccessDenied {
return errAuthentication
}
return nil
if !isRetentionActionAllowed(policy.BypassGovernanceRetentionAction, oi.Bucket, oi.Name, objRetention, r, cred, owner) {
return errAuthentication
}
return nil
} // No pre-existing retention metadata present.
default:
// Compliance retention mode cannot be changed or shortened.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html#object-lock-retention-modes
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
}
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time,
objRetention.Mode, byPassSet, r, cred, owner)
if apiErr == ErrAccessDenied {
return errAuthentication
// checkObjectLockedForOverwrite re-checks under the object write lock whether an
// in-place write to the null version (such as a directory object) would
// overwrite a locked version in an Object Lock bucket.
func checkObjectLockedForOverwrite(ctx context.Context, bucket, object string, getObjectInfoFn GetObjectInfoFn) error {
if isMinioMetaBucketName(bucket) {
return nil
}
retentionCfg, err := globalBucketObjectLockSys.Get(bucket)
if err != nil {
return err
}
if !retentionCfg.LockEnabled {
return nil
}
objInfo, err := getObjectInfoFn(ctx, bucket, object, ObjectOptions{
VersionID: nullVersionID,
NoLock: true,
NoAuditLog: true,
})
if err != nil {
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || errors.As(err, &MethodNotAllowed{}) {
return nil
}
return err
}
if objectlock.GetObjectLegalHoldMeta(objInfo.UserDefined).Status == objectlock.LegalHoldOn {
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined)
if r.Mode == objectlock.RetCompliance || r.Mode == objectlock.RetGovernance {
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
if r.RetainUntilDate.After(t) {
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
}
return nil
}
@@ -274,16 +327,24 @@ func checkPutObjectLockAllowed(ctx context.Context, rq *http.Request, bucket, ob
if err != nil {
return mode, retainDate, legalHold, toAPIErrorCode(ctx, err)
}
if isDirObject(object) && opts.VersionID == "" {
opts.VersionID = nullVersionID
}
if opts.VersionID != "" && !replicaTrusted {
if objInfo, err := getObjectInfoFn(ctx, bucket, object, opts); err == nil {
objInfo, err := getObjectInfoFn(ctx, bucket, object, opts)
if err != nil {
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) && !errors.As(err, &MethodNotAllowed{}) {
return mode, retainDate, legalHold, toAPIErrorCode(ctx, err)
}
} else {
r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined)
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return mode, retainDate, legalHold, ErrObjectLocked
}
if r.Mode == objectlock.RetCompliance && r.RetainUntilDate.After(t) {
if (r.Mode == objectlock.RetCompliance || r.Mode == objectlock.RetGovernance) && r.RetainUntilDate.After(t) {
return mode, retainDate, legalHold, ErrObjectLocked
}
mode = r.Mode
@@ -334,11 +395,10 @@ func checkPutObjectLockAllowed(ctx context.Context, rq *http.Request, bucket, ob
return mode, retainDate, legalHold, ErrObjectLocked
}
if !legalHoldRequested && retentionCfg.LockEnabled {
// inherit retention from bucket configuration
return retentionCfg.Mode, objectlock.RetentionDate{Time: t.Add(retentionCfg.Validity)}, legalHold, ErrNone
}
return "", objectlock.RetentionDate{}, legalHold, ErrNone
// Inherit retention from the bucket configuration. A legal-hold header
// on the same request, ON or OFF, is independent of retention and must
// not suppress the default (#165).
return retentionCfg.Mode, objectlock.RetentionDate{Time: t.Add(retentionCfg.Validity)}, legalHold, ErrNone
}
return mode, retainDate, legalHold, ErrNone
}
+5 -6
View File
@@ -19,7 +19,6 @@ package cmd
import (
"bytes"
"encoding/json"
"io"
"net/http"
@@ -100,13 +99,13 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
return
}
configData, err := json.Marshal(bucketPolicy)
configData, err := canonicalBucketPolicy(bucketPolicy)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
updatedAt, err := globalBucketMetadataSys.Update(ctx, bucket, bucketPolicyConfig, configData)
result, err := globalBucketMetadataSys.updateAndParseMetadata(ctx, bucket, bucketPolicyConfig, configData, false, false, nil)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -116,8 +115,8 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
replLogIf(ctx, globalSiteReplicationSys.BucketMetaHook(ctx, madmin.SRBucketMeta{
Type: madmin.SRBucketMetaTypePolicy,
Bucket: bucket,
Policy: bucketPolicyBytes,
UpdatedAt: updatedAt,
Policy: result.meta.PolicyConfigJSON,
UpdatedAt: result.updatedAt,
}))
// Success.
@@ -200,7 +199,7 @@ func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *ht
return
}
configData, err := json.Marshal(config)
configData, err := canonicalBucketPolicy(config)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
+26 -7
View File
@@ -255,13 +255,32 @@ func getConditionValuesWithTags(r *http.Request, lc string, cred auth.Credential
}
cloneHeader := r.Header.Clone()
signatureAge := cloneHeader.Get("x-amz-signature-age")
cloneHeader.Del("x-amz-signature-age")
// The presigned V4 verifier overwrites this internal scratch header after
// validating the signature. Ignore a value supplied on every other request
// type, where it would otherwise synthesize s3:signatureAge.
if authType == authTypePresigned && signatureAge != "" {
args["signatureAge"] = []string{signatureAge}
// s3:signatureAge is derived from the presigned X-Amz-Date rather than from
// anything the verifier writes back: PutObject and UploadPart authorize
// before they verify the signature, so a post-verification value is not yet
// available on the first evaluation. The date is bound by the signature
// (doesPresignedSignatureMatch rebuilds and compares it), so a forged date
// only changes the authorization outcome of a request that then fails
// verification. A date that does not parse leaves the key absent; the
// verifier rejects the request as ErrMalformedPresignedDate.
if authType == authTypePresigned {
if signedDate, err := time.Parse(iso8601Format, r.Form.Get(xhttp.AmzDate)); err == nil {
args["signatureAge"] = []string{strconv.FormatInt(currTime.Sub(signedDate).Milliseconds(), 10)}
}
}
// s3:x-amz-content-sha256 must name the payload hash the request is actually
// verified and enforced against, and only one such value. Presence of the
// header controls whether the key exists at all (AWS documents that the
// query-string form does not populate it), but the value comes from the same
// selection getContentSha256Cksum makes for verification: the presigned query
// value takes precedence over the header, and a repeated header contributes
// only its first value. Exposing every raw header value instead let a
// request satisfy a policy with a value the verifier never checked.
if _, ok := cloneHeader[xhttp.AmzContentSha256]; ok {
args[xhttp.AmzContentSha256] = []string{getContentSha256Cksum(r, serviceS3)}
cloneHeader.Del(xhttp.AmzContentSha256)
}
userTags := cloneHeader.Get(xhttp.AmzObjectTagging)
+34 -6
View File
@@ -23,8 +23,10 @@ import (
"net/url"
"os"
"slices"
"strconv"
"strings"
"testing"
"time"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/handlers"
@@ -579,8 +581,20 @@ func TestGetConditionValuesRejectsAbsentInternalKeys(t *testing.T) {
}
}
// s3:signatureAge is derived from the presigned X-Amz-Date, which the signature
// binds. A client header under the former scratch name must never supply it on
// any auth type, and a presign whose date is missing or malformed leaves the
// key absent (the verifier then rejects the request).
func TestGetConditionValuesOnlyAcceptsPresignedSignatureAge(t *testing.T) {
const signatureAgeHeader = "x-amz-signature-age"
signedDate := UTCNow().Add(-90 * time.Second)
presignQuery := func(date string) string {
q := url.Values{xhttp.AmzCredential: {"access/20260803/us-east-1/s3/aws4_request"}}
if date != "" {
q.Set(xhttp.AmzDate, date)
}
return "http://minio.local/bkt/obj?" + q.Encode()
}
for _, tc := range []struct {
name string
@@ -602,19 +616,33 @@ func TestGetConditionValuesOnlyAcceptsPresignedSignatureAge(t *testing.T) {
},
},
{
name: "presigned verifier value",
target: "http://minio.local/bkt/obj?" + url.Values{
xhttp.AmzCredential: {"access/20260803/us-east-1/s3/aws4_request"},
}.Encode(),
name: "presigned client header without date",
target: presignQuery(""),
headers: map[string]string{signatureAgeHeader: "250"},
},
{
name: "presigned malformed date",
target: presignQuery("yesterday"),
},
{
name: "presigned signed date",
target: presignQuery(signedDate.Format(iso8601Format)),
headers: map[string]string{signatureAgeHeader: "250"},
want: true,
},
} {
t.Run(tc.name, func(t *testing.T) {
got := condValuesForRequest(t, tc.target, tc.headers)
_, ok := got["signatureAge"]
v, ok := got["signatureAge"]
if ok != tc.want {
t.Fatalf("signatureAge presence: expected %v, got %v", tc.want, got["signatureAge"])
t.Fatalf("signatureAge presence: expected %v, got %v", tc.want, v)
}
if !tc.want {
return
}
age, err := strconv.ParseInt(strings.Join(v, ""), 10, 64)
if err != nil || age < (90*time.Second).Milliseconds() || age > (2*time.Minute).Milliseconds() {
t.Fatalf("signatureAge = %v, want about 90s derived from X-Amz-Date rather than the client header", v)
}
})
}
+12 -8
View File
@@ -43,6 +43,17 @@ func NewBucketQuotaSys() *BucketQuotaSys {
return &BucketQuotaSys{}
}
// getBucketQuotaSize returns the effective enforced hard-quota size.
func getBucketQuotaSize(quota *madmin.BucketQuota) uint64 {
if quota == nil || quota.Type != madmin.HardQuota {
return 0
}
if quota.Size > 0 {
return quota.Size
}
return quota.Quota
}
var bucketStorageCache = cachevalue.New[DataUsageInfo]()
// Init initialize bucket quota.
@@ -110,14 +121,7 @@ func (sys *BucketQuotaSys) enforceQuotaHard(ctx context.Context, bucket string,
return err
}
var quotaSize uint64
if q != nil && q.Type == madmin.HardQuota {
if q.Size > 0 {
quotaSize = q.Size
} else if q.Quota > 0 {
quotaSize = q.Quota
}
}
quotaSize := getBucketQuotaSize(q)
if quotaSize > 0 {
if uint64(size) >= quotaSize { // check if file size already exceeds the quota
return BucketQuotaExceeded{Bucket: bucket}
+77
View File
@@ -0,0 +1,77 @@
// Copyright (c) 2015-2025 MinIO, Inc.
// Copyright (c) 2025-2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"testing"
"github.com/minio/madmin-go/v3"
)
func TestGetBucketQuotaSize(t *testing.T) {
tests := []struct {
name string
quota *madmin.BucketQuota
want uint64
}{
{name: "nil"},
{name: "empty", quota: &madmin.BucketQuota{}},
{name: "current size", quota: &madmin.BucketQuota{Type: madmin.HardQuota, Size: 1024}, want: 1024},
{name: "legacy quota", quota: &madmin.BucketQuota{Type: madmin.HardQuota, Quota: 2048}, want: 2048},
{name: "size takes precedence", quota: &madmin.BucketQuota{Type: madmin.HardQuota, Size: 1024, Quota: 2048}, want: 1024},
{name: "missing type", quota: &madmin.BucketQuota{Size: 1024}},
{name: "unsupported type", quota: &madmin.BucketQuota{Type: "fifo", Size: 1024}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := getBucketQuotaSize(tt.quota); got != tt.want {
t.Fatalf("getBucketQuotaSize() = %d, want %d", got, tt.want)
}
})
}
}
func TestIsBktQuotaCfgReplicated(t *testing.T) {
hardQuota := func(size, legacy uint64) *madmin.BucketQuota {
return &madmin.BucketQuota{Type: madmin.HardQuota, Size: size, Quota: legacy}
}
tests := []struct {
name string
quotas []*madmin.BucketQuota
want bool
}{
{name: "none configured", quotas: []*madmin.BucketQuota{nil, nil}, want: true},
{name: "missing from one site", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), nil}},
{name: "matching size", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), hardQuota(1024, 0)}, want: true},
{name: "different size", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), hardQuota(2048, 0)}},
{name: "equivalent representations", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), hardQuota(0, 1024)}, want: true},
{name: "different typeless size", quotas: []*madmin.BucketQuota{{Size: 1024}, {Size: 2048}}},
{name: "different type", quotas: []*madmin.BucketQuota{hardQuota(1024, 0), {Type: "fifo", Size: 1024}}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isBktQuotaCfgReplicated(len(tt.quotas), tt.quotas); got != tt.want {
t.Fatalf("isBktQuotaCfgReplicated() = %v, want %v", got, tt.want)
}
})
}
}
+257
View File
@@ -0,0 +1,257 @@
// Copyright (c) 2026 Denis Babchuk
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"maps"
"net/http"
"strings"
"testing"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
"github.com/minio/minio/internal/hash"
xhttp "github.com/minio/minio/internal/http"
)
// checksumTestObject builds the xl.meta checksum of an object the way
// PutObject and CompleteMultipartUpload store it.
func checksumTestObject(t *testing.T, alg hash.ChecksumType, full bool, parts ...[]byte) FileInfo {
t.Helper()
fi := FileInfo{Volume: "bucket", Name: "object"}
if len(parts) == 1 {
fi.Size = int64(len(parts[0]))
fi.Checksum = hash.NewChecksumFromData(alg, parts[0]).AppendTo(nil, nil)
return fi
}
var combined, all []byte
for i, p := range parts {
combined = append(combined, hash.NewChecksumFromData(alg, p).Raw...)
all = append(all, p...)
fi.Parts = append(fi.Parts, ObjectPartInfo{Number: i + 1, Size: int64(len(p)), ActualSize: int64(len(p))})
fi.Size += int64(len(p))
}
typ := alg | hash.ChecksumMultipart | hash.ChecksumIncludesMultipart
if full {
typ |= hash.ChecksumFullObject
}
cs := hash.NewChecksumFromData(typ, combined)
if typ.FullObjectRequested() {
// Full object CRC, as merged from the parts on completion.
whole := hash.NewChecksumFromData(alg, all)
cs = &hash.Checksum{Type: typ, Raw: whole.Raw, Encoded: whole.Encoded}
}
fi.Checksum = cs.AppendTo(nil, combined)
return fi
}
// TestGetCRCMetaChecksumType checks the checksum headers that replication
// sends for an object: a FULL_OBJECT checksum type only for multipart full
// object CRCs, where the target needs it to create a matching upload.
func TestGetCRCMetaChecksumType(t *testing.T) {
p1, p2 := []byte("first part"), []byte("second part")
enc := func(alg hash.ChecksumType, b []byte) string { return hash.NewChecksumFromData(alg, b).Encoded }
whole := append(append([]byte{}, p1...), p2...)
composite := func(alg hash.ChecksumType) string {
var combined []byte
combined = append(combined, hash.NewChecksumFromData(alg, p1).Raw...)
combined = append(combined, hash.NewChecksumFromData(alg, p2).Raw...)
return enc(alg, combined) + "-2"
}
full := map[string]string{xhttp.AmzChecksumType: xhttp.AmzChecksumTypeFullObject}
tests := []struct {
name string
fi FileInfo
alg hash.ChecksumType
isMP bool
value string
extra map[string]string
}{
{"single-part SHA256", checksumTestObject(t, hash.ChecksumSHA256, false, p1), hash.ChecksumSHA256, false, enc(hash.ChecksumSHA256, p1), nil},
{"single-part CRC32", checksumTestObject(t, hash.ChecksumCRC32, false, p1), hash.ChecksumCRC32, false, enc(hash.ChecksumCRC32, p1), nil},
{"single-part CRC64NVME", checksumTestObject(t, hash.ChecksumCRC64NVME, false, p1), hash.ChecksumCRC64NVME, false, enc(hash.ChecksumCRC64NVME, p1), nil},
{"multipart COMPOSITE CRC32C", checksumTestObject(t, hash.ChecksumCRC32C, false, p1, p2), hash.ChecksumCRC32C, true, composite(hash.ChecksumCRC32C), nil},
{"multipart COMPOSITE SHA256", checksumTestObject(t, hash.ChecksumSHA256, false, p1, p2), hash.ChecksumSHA256, true, composite(hash.ChecksumSHA256), nil},
{"multipart FULL_OBJECT CRC32", checksumTestObject(t, hash.ChecksumCRC32, true, p1, p2), hash.ChecksumCRC32, true, enc(hash.ChecksumCRC32, whole), full},
{"multipart FULL_OBJECT CRC32C", checksumTestObject(t, hash.ChecksumCRC32C, true, p1, p2), hash.ChecksumCRC32C, true, enc(hash.ChecksumCRC32C, whole), full},
{"multipart CRC64NVME", checksumTestObject(t, hash.ChecksumCRC64NVME, false, p1, p2), hash.ChecksumCRC64NVME, true, enc(hash.ChecksumCRC64NVME, whole), full},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
oi := tt.fi.ToObjectInfo("bucket", "object", true)
want := map[string]string{tt.alg.Key(): tt.value, xhttp.AmzChecksumAlgo: tt.alg.String()}
maps.Copy(want, tt.extra)
got, isMP := getCRCMeta(oi, 0, nil)
if isMP != tt.isMP {
t.Errorf("isMP: got %t, want %t", isMP, tt.isMP)
}
if !maps.Equal(got, want) {
t.Errorf("object headers: got %v, want %v", got, want)
}
if !tt.isMP {
return
}
// Part headers carry the part checksum and never a type.
want = map[string]string{tt.alg.Key(): enc(tt.alg, p2), xhttp.AmzChecksumAlgo: tt.alg.String()}
if got, _ := getCRCMeta(oi, 2, nil); !maps.Equal(got, want) {
t.Errorf("part 2 headers: got %v, want %v", got, want)
}
})
}
}
// TestReplicateObjectChecksumTypes replicates objects with checksums through
// the S3 API of a test server, and checks that the replica gets the same
// checksum as the source.
func TestReplicateObjectChecksumTypes(t *testing.T) {
ctx := context.Background()
// Leave no object layer behind: unit tests that run later in the package
// rely on newObjectLayerFn() returning nil.
defer resetTestGlobals()
ts := StartTestServer(t, ErasureSDStr)
defer ts.Stop()
clnt, err := minio.New(strings.TrimPrefix(ts.Server.URL, "http://"), &minio.Options{
Creds: credentials.NewStaticV4(ts.AccessKey, ts.SecretKey, ""),
})
if err != nil {
t.Fatal(err)
}
core := &minio.Core{Client: clnt}
for _, bucket := range []string{"src", "dst"} {
if err := clnt.MakeBucket(ctx, bucket, minio.MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
if err := clnt.EnableVersioning(ctx, bucket); err != nil {
t.Fatal(err)
}
}
small := []byte("single part object")
p1 := bytes.Repeat([]byte("a"), 5<<20)
p2 := []byte("last part")
tests := []struct {
name string
alg hash.ChecksumType
multipart bool
full bool
copyOf string // server-side copy of an object created above
}{
{"put-SHA256", hash.ChecksumSHA256, false, false, ""},
{"put-CRC32", hash.ChecksumCRC32, false, false, ""},
{"mp-COMPOSITE-CRC32C", hash.ChecksumCRC32C, true, false, ""},
{"mp-COMPOSITE-SHA256", hash.ChecksumSHA256, true, false, ""},
{"mp-FULL_OBJECT-CRC32", hash.ChecksumCRC32, true, true, ""},
{"mp-FULL_OBJECT-CRC32C", hash.ChecksumCRC32C, true, true, ""},
{"mp-CRC64NVME", hash.ChecksumCRC64NVME, true, false, ""},
// The copy keeps the multipart FULL_OBJECT checksum of its source
// but is not multipart, so it replicates with a single PUT.
{"copy-FULL_OBJECT-CRC32", hash.ChecksumCRC32, false, true, "mp-FULL_OBJECT-CRC32"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
object := tt.name
// Upload the source object like an S3 client does.
switch {
case tt.copyOf != "":
_, err := core.CopyObject(ctx, "src", tt.copyOf, "src", object, nil, minio.CopySrcOptions{}, minio.PutObjectOptions{})
if err != nil {
t.Fatal(err)
}
case !tt.multipart:
_, err := core.PutObject(ctx, "src", object, bytes.NewReader(small), int64(len(small)), "", "", minio.PutObjectOptions{
UserMetadata: map[string]string{tt.alg.Key(): hash.NewChecksumFromData(tt.alg, small).Encoded},
})
if err != nil {
t.Fatal(err)
}
default:
meta := map[string]string{xhttp.AmzChecksumAlgo: tt.alg.String()}
if tt.full {
meta[xhttp.AmzChecksumType] = xhttp.AmzChecksumTypeFullObject
}
uploadID, err := core.NewMultipartUpload(ctx, "src", object, minio.PutObjectOptions{UserMetadata: meta})
if err != nil {
t.Fatal(err)
}
var parts []minio.CompletePart
for i, p := range [][]byte{p1, p2} {
h := http.Header{}
h.Set(tt.alg.Key(), hash.NewChecksumFromData(tt.alg, p).Encoded)
pi, err := core.PutObjectPart(ctx, "src", object, uploadID, i+1, bytes.NewReader(p), int64(len(p)), minio.PutObjectPartOptions{CustomHeader: h})
if err != nil {
t.Fatal(err)
}
parts = append(parts, minio.CompletePart{
PartNumber: pi.PartNumber, ETag: pi.ETag,
ChecksumCRC32: pi.ChecksumCRC32, ChecksumCRC32C: pi.ChecksumCRC32C,
ChecksumSHA1: pi.ChecksumSHA1, ChecksumSHA256: pi.ChecksumSHA256,
ChecksumCRC64NVME: pi.ChecksumCRC64NVME,
})
}
if _, err := core.CompleteMultipartUpload(ctx, "src", object, uploadID, parts, minio.PutObjectOptions{}); err != nil {
t.Fatal(err)
}
}
// Replicate it the way replicateObject does.
gr, err := ts.Obj.GetObjectNInfo(ctx, "src", object, nil, http.Header{}, ObjectOptions{Versioned: true, ReplicationRequest: true})
if err != nil {
t.Fatal(err)
}
defer gr.Close()
objInfo := gr.ObjInfo
putOpts, isMP, err := putReplicationOpts(ctx, "", objInfo)
if err != nil {
t.Fatal(err)
}
if isMP != tt.multipart {
t.Fatalf("multipart replication: got %t, want %t", isMP, tt.multipart)
}
if isMP {
err = replicateObjectWithMultipart(ctx, core, "dst", object, gr, objInfo, putOpts)
} else {
size, serr := objInfo.GetActualSize()
if serr != nil {
t.Fatal(serr)
}
_, err = core.PutObject(ctx, "dst", object, gr, size, "", "", putOpts)
}
if err != nil {
t.Fatalf("replication failed: %v", err)
}
src, err := clnt.StatObject(ctx, "src", object, minio.StatObjectOptions{Checksum: true})
if err != nil {
t.Fatal(err)
}
dst, err := clnt.StatObject(ctx, "dst", object, minio.StatObjectOptions{Checksum: true})
if err != nil {
t.Fatal(err)
}
checksums := func(oi minio.ObjectInfo) string {
return strings.Join([]string{oi.ChecksumMode, oi.ChecksumCRC32, oi.ChecksumCRC32C, oi.ChecksumSHA1, oi.ChecksumSHA256, oi.ChecksumCRC64NVME}, "|")
}
if checksums(src) == "|||||" || checksums(dst) != checksums(src) {
t.Errorf("replica checksum %q, source %q", checksums(dst), checksums(src))
}
})
}
}
+7 -4
View File
@@ -418,6 +418,9 @@ func getReplicationState(rinfos replicatedInfos, prevState ReplicationState, vID
for _, rinfo := range rinfos.Targets {
if rinfo.ResyncTimestamp != "" {
if rs.ResetStatusesMap == nil {
rs.ResetStatusesMap = make(map[string]string)
}
rs.ResetStatusesMap[targetResetHeader(rinfo.Arn)] = rinfo.ResyncTimestamp
}
}
@@ -640,10 +643,10 @@ type VersionPurgeStatusType = replication.VersionPurgeStatusType
type replicationResyncer struct {
// map of bucket to their resync status
statusMap map[string]BucketReplicationResyncStatus
workerSize int
resyncCancelCh chan struct{}
workerCh chan struct{}
statusMap map[string]BucketReplicationResyncStatus
workerSize int
cancelResyncs map[resyncOpts]context.CancelCauseFunc
workerCh chan struct{}
sync.RWMutex
}
+323 -141
View File
@@ -425,6 +425,7 @@ func checkReplicateDelete(ctx context.Context, bucket string, dobj ObjectToDelet
// the mere presence or absence of the target version.
func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, objectAPI ObjectLayer) replicatedInfos {
var replicationStatus replication.StatusType
isPurge := dobj.isVersionPurge()
bucket := dobj.Bucket
versionID := dobj.DeleteMarkerVersionID
if versionID == "" {
@@ -484,6 +485,7 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
lk := objectAPI.NewNSLock(bucket, "/[replicate]/"+dobj.ObjectName)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
dobj.RetryCount++
globalReplicationPool.Get().queueMRFSave(dobj.ToMRFEntry())
sendEvent(eventArgs{
BucketName: bucket,
@@ -502,6 +504,35 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
if !isPurge && dobj.DeleteMarkerVersionID != "" {
// A creation task carries the marker as it looked when it was queued
// by the DELETE handler, a GET/HEAD/LIST heal, the scanner or MRF.
// While it waited for this lock, another frontend may have purged the
// marker and replicated that purge. The targets no longer hold the
// marker, so the queued creation would recreate it from the stale
// snapshot. Confirm the source version under the lock first.
switch deleteMarkerCreationState(ctx, objectAPI, dobj) {
case creationStale:
return replicatedInfos{}
case creationUnverified:
dobj.RetryCount++
globalReplicationPool.Get().queueMRFSave(dobj.ToMRFEntry())
sendEvent(eventArgs{
BucketName: bucket,
Object: ObjectInfo{
Bucket: bucket,
Name: dobj.ObjectName,
VersionID: versionID,
DeleteMarker: dobj.DeleteMarker,
},
UserAgent: "Internal: [Replication]",
Host: globalLocalNodeName,
EventName: event.ObjectReplicationNotTracked,
})
return replicatedInfos{}
}
}
rinfos := replicatedInfos{Targets: make([]replicatedTargetInfo, 0, len(dsc.targetsMap))}
var wg sync.WaitGroup
var mu sync.Mutex
@@ -546,25 +577,38 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
replicationStatus = rinfos.ReplicationStatus()
prevStatus := dobj.DeleteMarkerReplicationStatus()
if dobj.VersionID != "" {
prevStatus = replication.StatusType(dobj.VersionPurgeStatus())
replicationStatus = replication.StatusType(rinfos.VersionPurgeStatus())
if isPurge {
prevStatus = purgeReplicationStatus(dobj.VersionPurgeStatus())
replicationStatus = purgeReplicationStatus(rinfos.VersionPurgeStatus())
}
// to decrement pending count later.
for _, rinfo := range rinfos.Targets {
if rinfo.ReplicationStatus != rinfo.PrevReplicationStatus {
globalReplicationStats.Load().Update(dobj.Bucket, rinfo, replicationStatus,
prevStatus)
status, previous := rinfo.ReplicationStatus, rinfo.PrevReplicationStatus
if isPurge {
status = purgeReplicationStatus(rinfo.VersionPurgeStatus)
previous = purgeReplicationStatus(dobj.ReplicationState.PurgeTargets[rinfo.Arn])
}
if status != previous {
globalReplicationStats.Load().Update(dobj.Bucket, rinfo, status, previous)
}
}
eventName := event.ObjectReplicationComplete
if replicationStatus == replication.Failed {
eventName = event.ObjectReplicationFailed
dobj.RetryCount++
globalReplicationPool.Get().queueMRFSave(dobj.ToMRFEntry())
}
drs := getReplicationState(rinfos, dobj.ReplicationState, dobj.VersionID)
if isPurge {
// A purge must not rewrite the marker's creation/replica metadata.
// Multiple serialized empty target statuses can parse as nonempty,
// so explicitly send an empty creation update to the metadata writer.
drs.ReplicationStatusInternal = ""
drs.Targets = nil
drs.ReplicaStatus = ""
}
if replicationStatus != prevStatus {
drs.ReplicationTimeStamp = UTCNow()
}
@@ -606,6 +650,7 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
}
func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationInfo, tgt *TargetClient) (rinfo replicatedTargetInfo) {
isPurge := dobj.isVersionPurge()
versionID := dobj.DeleteMarkerVersionID
if versionID == "" {
versionID = dobj.VersionID
@@ -615,42 +660,50 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
rinfo.OpType = dobj.OpType
rinfo.endpoint = tgt.EndpointURL().Host
rinfo.secure = tgt.EndpointURL().Scheme == "https"
// A purge leaves ReplicationStatus empty: the metadata writer interprets
// that as preserving the entire creation-status block, including targets
// outside this fan-out. Only VersionPurgeStatus records the purge outcome.
defer func() {
if rinfo.ReplicationStatus == replication.Completed && tgt.ResetID != "" && dobj.OpType == replication.ExistingObjectReplicationType {
completed := rinfo.ReplicationStatus == replication.Completed
if isPurge {
completed = rinfo.VersionPurgeStatus == replication.VersionPurgeComplete
}
if completed && tgt.ResetID != "" && dobj.OpType == replication.ExistingObjectReplicationType {
rinfo.ResyncTimestamp = fmt.Sprintf("%s;%s", UTCNow().Format(http.TimeFormat), tgt.ResetID)
}
}()
if dobj.VersionID == "" && rinfo.PrevReplicationStatus == replication.Completed && dobj.OpType != replication.ExistingObjectReplicationType {
if !isPurge && rinfo.PrevReplicationStatus == replication.Completed && dobj.OpType != replication.ExistingObjectReplicationType {
rinfo.ReplicationStatus = rinfo.PrevReplicationStatus
return rinfo
}
if dobj.VersionID != "" && rinfo.VersionPurgeStatus == replication.VersionPurgeComplete {
if isPurge && rinfo.VersionPurgeStatus == replication.VersionPurgeComplete {
return rinfo
}
if globalBucketTargetSys.isOffline(tgt.EndpointURL()) {
replLogOnceIf(ctx, fmt.Errorf("remote target is offline for bucket:%s arn:%s", dobj.Bucket, tgt.ARN), "replication-target-offline-delete-"+tgt.ARN)
rinfo.Err = fmt.Errorf("remote target is offline for bucket:%s arn:%s", dobj.Bucket, tgt.ARN)
replLogOnceIf(ctx, rinfo.Err, "replication-target-offline-delete-"+tgt.ARN)
sendEvent(eventArgs{
BucketName: dobj.Bucket,
Object: ObjectInfo{
Bucket: dobj.Bucket,
Name: dobj.ObjectName,
VersionID: dobj.VersionID,
VersionID: versionID,
DeleteMarker: dobj.DeleteMarker,
},
UserAgent: "Internal: [Replication]",
Host: globalLocalNodeName,
EventName: event.ObjectReplicationNotTracked,
})
if dobj.VersionID == "" {
rinfo.ReplicationStatus = replication.Failed
} else {
if isPurge {
rinfo.VersionPurgeStatus = replication.VersionPurgeFailed
} else {
rinfo.ReplicationStatus = replication.Failed
}
return rinfo
}
// early return if already replicated delete marker for existing object replication/ healing delete markers
if dobj.DeleteMarkerVersionID != "" {
if !isPurge && dobj.DeleteMarkerVersionID != "" {
toi, err := tgt.StatObject(ctx, tgt.Bucket, dobj.ObjectName, minio.StatObjectOptions{
VersionID: versionID,
Internal: minio.AdvancedGetOptions{
@@ -662,16 +715,10 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
switch {
case isErrMethodNotAllowed(serr):
// delete marker already replicated
if dobj.VersionID == "" && rinfo.VersionPurgeStatus.Empty() {
rinfo.ReplicationStatus = replication.Completed
return rinfo
}
rinfo.ReplicationStatus = replication.Completed
return rinfo
case isErrObjectNotFound(serr), isErrVersionNotFound(serr):
// version being purged is already not found on target.
if !rinfo.VersionPurgeStatus.Empty() {
rinfo.VersionPurgeStatus = replication.VersionPurgeComplete
return rinfo
}
// The marker still needs to be created on the target.
case isErrReadQuorum(serr), isErrWriteQuorum(serr):
// destination has some quorum issues, perform removeObject() anyways
// to complete the operation.
@@ -691,7 +738,7 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
rmErr := tgt.RemoveObject(ctx, tgt.Bucket, dobj.ObjectName, minio.RemoveObjectOptions{
VersionID: versionID,
Internal: minio.AdvancedRemoveOptions{
ReplicationDeleteMarker: dobj.DeleteMarkerVersionID != "",
ReplicationDeleteMarker: !isPurge && dobj.DeleteMarkerVersionID != "",
ReplicationMTime: dobj.DeleteMarkerMTime.Time,
ReplicationStatus: minio.ReplicationStatusReplica,
ReplicationRequest: true, // always set this to distinguish between `mc mirror` replication and serverside
@@ -699,20 +746,20 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
})
if rmErr != nil {
rinfo.Err = rmErr
if dobj.VersionID == "" {
rinfo.ReplicationStatus = replication.Failed
} else {
if isPurge {
rinfo.VersionPurgeStatus = replication.VersionPurgeFailed
} else {
rinfo.ReplicationStatus = replication.Failed
}
replLogIf(ctx, fmt.Errorf("unable to replicate delete marker to %s: %s/%s(%s): %w", tgt.EndpointURL(), tgt.Bucket, dobj.ObjectName, versionID, rmErr))
if rmErr != nil && minio.IsNetworkOrHostDown(rmErr, true) && !globalBucketTargetSys.isOffline(tgt.EndpointURL()) {
globalBucketTargetSys.markOffline(tgt.EndpointURL())
}
} else {
if dobj.VersionID == "" {
rinfo.ReplicationStatus = replication.Completed
} else {
if isPurge {
rinfo.VersionPurgeStatus = replication.VersionPurgeComplete
} else {
rinfo.ReplicationStatus = replication.Completed
}
}
return rinfo
@@ -781,6 +828,18 @@ func (m caseInsensitiveMap) Lookup(key string) (string, bool) {
return "", false
}
// replicationTaggingTimestamp carries a recorded removal even when tags are
// empty. Only legacy nonempty tags use ModTime; absence is not a tombstone.
func replicationTaggingTimestamp(objInfo ObjectInfo) (time.Time, error) {
if stamp, ok := caseInsensitiveMap(objInfo.UserDefined).Lookup(ReservedMetadataPrefixLower + TaggingTimestamp); ok {
return time.Parse(time.RFC3339Nano, stamp)
}
if objInfo.UserTags != "" {
return objInfo.ModTime, nil
}
return time.Time{}, nil
}
func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo) (putOpts minio.PutObjectOptions, isMP bool, err error) {
meta := make(map[string]string)
isSSEC := crypto.SSEC.IsEncrypted(objInfo.UserDefined)
@@ -850,17 +909,12 @@ func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo) (put
tag, _ := tags.ParseObjectTags(objInfo.UserTags)
if tag != nil {
putOpts.UserTags = tag.ToMap()
// set tag timestamp in opts
tagTimestamp := objInfo.ModTime
if tagTmstampStr, ok := objInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]; ok {
tagTimestamp, err = time.Parse(time.RFC3339Nano, tagTmstampStr)
if err != nil {
return putOpts, false, err
}
}
putOpts.Internal.TaggingTimestamp = tagTimestamp
}
}
putOpts.Internal.TaggingTimestamp, err = replicationTaggingTimestamp(objInfo)
if err != nil {
return putOpts, false, err
}
lkMap := caseInsensitiveMap(objInfo.UserDefined)
if lang, ok := lkMap.Lookup(xhttp.ContentLanguage); ok {
@@ -1003,6 +1057,12 @@ func getReplicationAction(oi1 ObjectInfo, oi2 minio.ObjectInfo, opType replicati
if (oi2.UserTagCount > 0 && !reflect.DeepEqual(oi2Map, t.ToMap())) || (oi2.UserTagCount != len(t.ToMap())) {
return replicateMetadata
}
// HEAD does not report the tag revision. Equal values can hide a newer
// deletion or re-addition, so scheduled metadata/heal work must deliver it.
// Completed objects are still excluded by the existing scanner gates.
if _, ok := caseInsensitiveMap(oi1.UserDefined).Lookup(ReservedMetadataPrefixLower + TaggingTimestamp); ok {
return replicateMetadata
}
// Compare only necessary headers
compareKeys := []string{
@@ -1269,9 +1329,6 @@ func replicateObject(ctx context.Context, ri ReplicateObjectInfo, objectAPI Obje
oi.UserDefined[targetResetHeader(rinfo.Arn)] = rinfo.ResyncTimestamp
}
}
if ri.UserTags != "" {
oi.UserDefined[xhttp.AmzObjectTagging] = ri.UserTags
}
return dsc, nil
},
}
@@ -1689,14 +1746,11 @@ applyAction:
if _, ok := lkMap.Lookup(xhttp.AmzObjectLockRetainUntilDate); ok {
dstOpts.Internal.RetentionTimestamp = objInfo.ModTime
}
if objInfo.UserTags != "" {
dstOpts.Internal.TaggingTimestamp = objInfo.ModTime
}
if tagTmStr, ok := lkMap.Lookup(ReservedMetadataPrefixLower + TaggingTimestamp); ok {
ondiskTimestamp, err := time.Parse(time.RFC3339, tagTmStr)
if err == nil {
dstOpts.Internal.TaggingTimestamp = ondiskTimestamp
}
dstOpts.Internal.TaggingTimestamp, rinfo.Err = replicationTaggingTimestamp(objInfo)
if rinfo.Err != nil {
rinfo.ReplicationStatus = replication.Failed
replLogIf(ctx, fmt.Errorf("invalid tagging timestamp for object %s/%s(%s): %w", bucket, object, objInfo.VersionID, rinfo.Err))
return rinfo
}
if retTmStr, ok := lkMap.Lookup(ReservedMetadataPrefixLower + ObjectLockRetentionTimestamp); ok {
ondiskTimestamp, err := time.Parse(time.RFC3339, retTmStr)
@@ -1916,11 +1970,61 @@ func filterReplicationStatusMetadata(metadata map[string]string) map[string]stri
// DeletedObjectReplicationInfo has info on deleted object
type DeletedObjectReplicationInfo struct {
DeletedObject
Bucket string
EventType string
OpType replication.Type
ResetID string
TargetArn string
Bucket string
EventType string
OpType replication.Type
ResetID string
TargetArn string
RetryCount int
}
// isVersionPurge also recognizes the old marker-shaped purge task. Use the
// operation's state, rather than one target's possibly missing purge entry.
func (di DeletedObjectReplicationInfo) isVersionPurge() bool {
return di.VersionID != "" || di.DeleteMarkerVersionID != "" && !di.VersionPurgeStatus().Empty()
}
// creationState is the outcome of re-reading a queued delete-marker creation
// against the source.
type creationState int
const (
creationCurrent creationState = iota
creationStale
creationUnverified
)
// deleteMarkerCreationState re-reads the marker a queued creation task refers
// to. The version being absent, no longer a marker, or under a version purge
// makes the creation stale. A failed read is not absence: the caller retries
// later instead of guessing.
func deleteMarkerCreationState(ctx context.Context, objectAPI ObjectLayer, dobj DeletedObjectReplicationInfo) creationState {
oi, err := objectAPI.GetObjectInfo(ctx, dobj.Bucket, dobj.ObjectName, ObjectOptions{
VersionID: dobj.DeleteMarkerVersionID,
Versioned: globalBucketVersioningSys.PrefixEnabled(dobj.Bucket, dobj.ObjectName),
VersionSuspended: globalBucketVersioningSys.Suspended(dobj.Bucket),
})
switch {
case isErrObjectNotFound(err), isErrVersionNotFound(err):
return creationStale
case err != nil && !isErrMethodNotAllowed(err):
return creationUnverified
}
if !oi.DeleteMarker || oi.VersionID != dobj.DeleteMarkerVersionID {
return creationStale
}
if !oi.VersionPurgeStatus.Empty() || oi.VersionPurgeStatusInternal != "" {
return creationStale
}
return creationCurrent
}
// Purge metadata uses COMPLETE; operation statistics and audit use COMPLETED.
func purgeReplicationStatus(status VersionPurgeStatusType) replication.StatusType {
if replication.StatusType(status) == replication.CompletedLegacy {
return replication.Completed
}
return replication.StatusType(status)
}
// ToMRFEntry returns the relevant info needed by MRF
@@ -1930,9 +2034,10 @@ func (di DeletedObjectReplicationInfo) ToMRFEntry() MRFReplicateEntry {
versionID = di.VersionID
}
return MRFReplicateEntry{
Bucket: di.Bucket,
Object: di.ObjectName,
versionID: versionID,
Bucket: di.Bucket,
Object: di.ObjectName,
versionID: versionID,
RetryCount: di.RetryCount,
}
}
@@ -2354,9 +2459,9 @@ func (p *ReplicationPool) queueReplicaTask(ri ReplicateObjectInfo) {
switch ri.OpType {
case replication.HealReplicationType, replication.ExistingObjectReplicationType:
ch = p.mrfReplicaCh
healCh = p.getWorkerCh(ri.Name, ri.Bucket, ri.Size)
healCh = p.getWorkerCh(ri.Bucket, ri.Name, ri.Size)
default:
ch = p.getWorkerCh(ri.Name, ri.Bucket, ri.Size)
ch = p.getWorkerCh(ri.Bucket, ri.Name, ri.Size)
}
if ch == nil && healCh == nil {
return
@@ -2424,6 +2529,7 @@ func (p *ReplicationPool) queueReplicaDeleteTask(doi DeletedObjectReplicationInf
case <-p.ctx.Done():
case ch <- doi:
default:
doi.RetryCount++
p.queueMRFSave(doi.ToMRFEntry())
p.mu.RLock()
prio := p.priority
@@ -2964,10 +3070,10 @@ const (
func newresyncer() *replicationResyncer {
rs := replicationResyncer{
statusMap: make(map[string]BucketReplicationResyncStatus),
workerSize: resyncWorkerCnt,
resyncCancelCh: make(chan struct{}, resyncWorkerCnt),
workerCh: make(chan struct{}, resyncWorkerCnt),
statusMap: make(map[string]BucketReplicationResyncStatus),
workerSize: resyncWorkerCnt,
cancelResyncs: make(map[resyncOpts]context.CancelCauseFunc),
workerCh: make(chan struct{}, resyncWorkerCnt),
}
for i := 0; i < rs.workerSize; i++ {
rs.workerCh <- struct{}{}
@@ -2975,13 +3081,67 @@ func newresyncer() *replicationResyncer {
return &rs
}
var errResyncCanceled = errors.New("replication resync canceled")
// Registration and cancellation share the status lock. A queued run therefore
// cannot miss cancellation between publishing its status and taking a slot.
func (s *replicationResyncer) registerResync(parent context.Context, opts resyncOpts) (context.Context, context.CancelCauseFunc, bool) {
s.Lock()
defer s.Unlock()
st, ok := s.statusMap[opts.bucket].TargetsMap[opts.arn]
if !ok || st.ResyncID != opts.resyncID {
return nil, nil, false
}
if _, running := s.cancelResyncs[opts]; running {
return nil, nil, false
}
ctx, cancel := context.WithCancelCause(parent)
if s.cancelResyncs == nil {
s.cancelResyncs = make(map[resyncOpts]context.CancelCauseFunc)
}
s.cancelResyncs[opts] = cancel
if st.ResyncStatus == ResyncCanceled {
cancel(errResyncCanceled)
}
return ctx, cancel, true
}
func (s *replicationResyncer) cancelResyncID(resyncID string) {
s.Lock()
defer s.Unlock()
for bucket, m := range s.statusMap {
for arn, st := range m.TargetsMap {
if st.ResyncID == resyncID && (st.ResyncStatus == ResyncPending || st.ResyncStatus == ResyncStarted) {
st.ResyncStatus = ResyncCanceled
st.LastUpdate = UTCNow()
m.TargetsMap[arn] = st
m.LastUpdate = st.LastUpdate
}
}
s.statusMap[bucket] = m
}
for opts, cancel := range s.cancelResyncs {
if opts.resyncID == resyncID {
cancel(errResyncCanceled)
}
}
}
// mark status of replication resync on remote target for the bucket
func (s *replicationResyncer) markStatus(status ResyncStatusType, opts resyncOpts, objAPI ObjectLayer) {
func (s *replicationResyncer) markStatus(status ResyncStatusType, opts resyncOpts, objAPI ObjectLayer) ResyncStatusType {
s.Lock()
defer s.Unlock()
m := s.statusMap[opts.bucket]
st := m.TargetsMap[opts.arn]
st, ok := m.TargetsMap[opts.arn]
if !ok || st.ResyncID != opts.resyncID {
return NoResync
}
// A cancel may win the lock after the finalizer checked its context.
// Persist that cancellation, never a stale Started/Completed result.
if st.ResyncStatus == ResyncCanceled {
status = ResyncCanceled
}
st.LastUpdate = UTCNow()
st.ResyncStatus = status
m.TargetsMap[opts.arn] = st
@@ -2991,14 +3151,18 @@ func (s *replicationResyncer) markStatus(status ResyncStatusType, opts resyncOpt
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
saveResyncStatus(ctx, opts.bucket, m, objAPI)
return status
}
// update replication resync stats for bucket's remote target
func (s *replicationResyncer) incStats(ts TargetReplicationResyncStatus, opts resyncOpts) {
func (s *replicationResyncer) incStats(ts TargetReplicationResyncStatus, opts resyncOpts) bool {
s.Lock()
defer s.Unlock()
m := s.statusMap[opts.bucket]
st := m.TargetsMap[opts.arn]
st, ok := m.TargetsMap[opts.arn]
if !ok || st.ResyncID != opts.resyncID || st.ResyncStatus == ResyncCanceled {
return false
}
st.Object = ts.Object
st.ReplicatedCount += ts.ReplicatedCount
st.FailedCount += ts.FailedCount
@@ -3007,6 +3171,7 @@ func (s *replicationResyncer) incStats(ts TargetReplicationResyncStatus, opts re
m.TargetsMap[opts.arn] = st
m.LastUpdate = UTCNow()
s.statusMap[opts.bucket] = m
return true
}
// resyncResults consumes the per-object outcomes produced by the resync worker
@@ -3023,8 +3188,9 @@ type resyncResults struct {
// result into the bucket's resync status.
func (s *replicationResyncer) newResyncResults(opts resyncOpts) *resyncResults {
return startResyncResults(func(r TargetReplicationResyncStatus) {
s.incStats(r, opts)
globalSiteResyncMetrics.updateMetric(r, opts.resyncID)
if s.incStats(r, opts) {
globalSiteResyncMetrics.updateMetric(r, opts.resyncID)
}
})
}
@@ -3063,29 +3229,22 @@ func (rr *resyncResults) finish(workers []chan ReplicateObjectInfo, workerWg *sy
}
// sendResyncResult delivers a worker's computed per-object result to ch,
// returning false if the worker must stop first. On the resync-cancel signal it
// records the abort - the already-computed result is dropped - so
// finalResyncStatus can downgrade a Completed run; on ctx cancellation it stops
// without recording, since finalResyncStatus's parent-context check covers that.
func (s *replicationResyncer) sendResyncResult(ctx context.Context, ch chan<- TargetReplicationResyncStatus, st TargetReplicationResyncStatus, workerAborted *atomic.Bool) bool {
// returning false if the run's context was canceled first.
func (s *replicationResyncer) sendResyncResult(ctx context.Context, ch chan<- TargetReplicationResyncStatus, st TargetReplicationResyncStatus) bool {
select {
case <-ctx.Done():
return false
case <-s.resyncCancelCh:
workerAborted.Store(true)
return false
case ch <- st:
return true
}
}
// finalResyncStatus downgrades a Completed status to Failed when the run could
// not have observed every object: the parent context was canceled (workers then
// return without sending their computed result) or a worker dropped a result on
// the resync-cancel signal. Without this a persisted Completed would misrepresent
// an incomplete resync.
func finalResyncStatus(status ResyncStatusType, ctxErr error, workerAborted bool) ResyncStatusType {
if status == ResyncCompleted && (ctxErr != nil || workerAborted) {
// User cancellation is terminal; interrupted completion remains retryable.
func finalResyncStatus(status ResyncStatusType, cause error) ResyncStatusType {
if errors.Is(cause, errResyncCanceled) {
return ResyncCanceled
}
if status == ResyncCompleted && cause != nil {
return ResyncFailed
}
return status
@@ -3156,26 +3315,31 @@ func objectNeedsResyncForARN(roi ReplicateObjectInfo, arn string) bool {
// resyncBucket resyncs all qualifying objects as per replication rules for the target
// ARN
func (s *replicationResyncer) resyncBucket(ctx context.Context, objectAPI ObjectLayer, heal bool, opts resyncOpts) {
ctx, cancel, registered := s.registerResync(ctx, opts)
if !registered {
return
}
defer func() {
cancel(nil)
s.Lock()
delete(s.cancelResyncs, opts)
s.Unlock()
}()
select {
case <-s.workerCh: // block till a worker is available
case <-ctx.Done():
if errors.Is(context.Cause(ctx), errResyncCanceled) {
status := s.markStatus(ResyncCanceled, opts, objectAPI)
globalSiteResyncMetrics.incBucket(opts, status)
}
return
}
resyncStatus := ResyncFailed
// workerAborted records that a worker dropped an already-computed result on
// the resync-cancel signal. With a canceled parent context (which makes
// workers return without sending their result), it means a Completed run did
// not actually observe every object - see finalResyncStatus below.
var workerAborted atomic.Bool
defer func() {
// Downgrade a Completed status whose counts are incomplete, so the
// persisted status is not a misleading Completed. Runs after results.finish
// drains (LIFO) and before markStatus persists - markStatus uses its own
// background context, so a parent cancellation during the drain would
// otherwise still record Completed.
resyncStatus = finalResyncStatus(resyncStatus, ctx.Err(), workerAborted.Load())
s.markStatus(resyncStatus, opts, objectAPI)
// Runs after workers/results drain and before our own deferred cancel.
resyncStatus = finalResyncStatus(resyncStatus, context.Cause(ctx))
resyncStatus = s.markStatus(resyncStatus, opts, objectAPI)
globalSiteResyncMetrics.incBucket(opts, resyncStatus)
s.workerCh <- struct{}{}
}()
@@ -3210,8 +3374,8 @@ func (s *replicationResyncer) resyncBucket(ctx context.Context, objectAPI Object
return
}
// mark resync status as resync started
if !heal {
s.markStatus(ResyncStarted, opts, objectAPI)
if !heal && s.markStatus(ResyncStarted, opts, objectAPI) != ResyncStarted {
return
}
// Walk through all object versions - Walk() is always in ascending order needed to ensure
@@ -3237,7 +3401,14 @@ func (s *replicationResyncer) resyncBucket(ctx context.Context, objectAPI Object
// cannot race the last incStats. Registered after the markStatus finalizer, so
// LIFO runs finish first.
results := s.newResyncResults(opts)
defer results.finish(workers, &wg)
defer func() {
if resyncStatus != ResyncCompleted {
cancel(nil)
}
// Exactly one finish: success drains with a live context; errors stop
// blocked workers first. Both drain counts before persisting status.
results.finish(workers, &wg)
}()
for i := range resyncParallelRoutines {
wg.Add(1)
workers[i] = make(chan ReplicateObjectInfo, 100)
@@ -3248,7 +3419,6 @@ func (s *replicationResyncer) resyncBucket(ctx context.Context, objectAPI Object
select {
case <-ctx.Done():
return
case <-s.resyncCancelCh:
default:
}
traceFn := s.trace(tgt.ResetID, fmt.Sprintf("%s/%s (%s)", opts.bucket, roi.Name, roi.VersionID))
@@ -3295,26 +3465,29 @@ func (s *replicationResyncer) resyncBucket(ctx context.Context, objectAPI Object
}
}
traceFn(traceSize, traceErr)
if !s.sendResyncResult(ctx, results.ch, st, &workerAborted) {
if !s.sendResyncResult(ctx, results.ch, st) {
return
}
}
}(ctx, i)
}
for res := range objInfoCh {
walkLoop:
for {
var res itemOrErr[ObjectInfo]
select {
case <-ctx.Done():
return
case item, ok := <-objInfoCh:
if !ok {
break walkLoop
}
res = item
}
if res.Err != nil {
resyncStatus = ResyncFailed
replLogIf(ctx, res.Err)
return
}
select {
case <-s.resyncCancelCh:
resyncStatus = ResyncCanceled
return
case <-ctx.Done():
return
default:
}
if heal && lastCheckpoint != "" && lastCheckpoint != res.Item.Name {
continue
}
@@ -3328,14 +3501,11 @@ func (s *replicationResyncer) resyncBucket(ctx context.Context, objectAPI Object
if !objectNeedsResyncForARN(roi, opts.arn) {
continue
}
h := xxh3.HashString(roi.Bucket + roi.Name)
select {
case <-s.resyncCancelCh:
return
case <-ctx.Done():
return
default:
h := xxh3.HashString(roi.Bucket + roi.Name)
workers[h%uint64(resyncParallelRoutines)] <- roi
case workers[h%uint64(resyncParallelRoutines)] <- roi:
}
}
resyncStatus = ResyncCompleted
@@ -3363,9 +3533,9 @@ func (s *replicationResyncer) start(ctx context.Context, objAPI ObjectLayer, opt
if len(tgtArns) == 0 {
return fmt.Errorf("arn %s specified for resync not found in replication config", opts.arn)
}
globalReplicationPool.Get().resyncer.RLock()
data, ok := globalReplicationPool.Get().resyncer.statusMap[opts.bucket]
globalReplicationPool.Get().resyncer.RUnlock()
s.Lock()
defer s.Unlock()
data, ok := s.statusMap[opts.bucket]
if !ok {
data, err = loadBucketResyncMetadata(ctx, opts.bucket, objAPI)
if err != nil {
@@ -3386,23 +3556,14 @@ func (s *replicationResyncer) start(ctx context.Context, objAPI ObjectLayer, opt
ResyncStatus: ResyncPending,
Bucket: opts.bucket,
}
data.TargetsMap = data.cloneTgtStats()
data.TargetsMap[opts.arn] = status
if err = saveResyncStatus(ctx, opts.bucket, data, objAPI); err != nil {
return err
}
globalReplicationPool.Get().resyncer.Lock()
defer globalReplicationPool.Get().resyncer.Unlock()
brs, ok := globalReplicationPool.Get().resyncer.statusMap[opts.bucket]
if !ok {
brs = BucketReplicationResyncStatus{
Version: resyncMetaVersion,
TargetsMap: make(map[string]TargetReplicationResyncStatus),
}
}
brs.TargetsMap[opts.arn] = status
globalReplicationPool.Get().resyncer.statusMap[opts.bucket] = brs
go globalReplicationPool.Get().resyncer.resyncBucket(GlobalContext, objAPI, false, opts)
s.statusMap[opts.bucket] = data
go s.resyncBucket(GlobalContext, objAPI, false, opts)
return nil
}
@@ -3476,6 +3637,9 @@ func (p *ReplicationPool) loadResync(ctx context.Context, buckets []string, objA
// Make sure only one node running resync on the cluster.
ctx, cancel := globalLeaderLock.GetLock(ctx)
defer cancel()
var workers sync.WaitGroup
// Keep the merged leader context alive until every resumed run exits.
defer workers.Wait()
for index := range buckets {
bucket := buckets[index]
@@ -3489,19 +3653,24 @@ func (p *ReplicationPool) loadResync(ctx context.Context, buckets []string, objA
}
p.resyncer.Lock()
p.resyncer.statusMap[bucket] = meta
p.resyncer.Unlock()
if current, ok := p.resyncer.statusMap[bucket]; ok {
// A concurrent start/cancel is newer than the disk snapshot.
meta = current
} else {
p.resyncer.statusMap[bucket] = meta
}
tgts := meta.cloneTgtStats()
p.resyncer.Unlock()
for arn, st := range tgts {
switch st.ResyncStatus {
case ResyncFailed, ResyncStarted, ResyncPending:
go p.resyncer.resyncBucket(ctx, objAPI, true, resyncOpts{
opts := resyncOpts{
bucket: bucket,
arn: arn,
resyncID: st.ResyncID,
resyncBefore: st.ResyncBeforeDate,
})
}
workers.Go(func() { p.resyncer.resyncBucket(ctx, objAPI, true, opts) })
}
}
}
@@ -3724,9 +3893,10 @@ func queueReplicationHeal(ctx context.Context, bucket string, oi ObjectInfo, rcf
DeleteMarkerMTime: DeleteMarkerMTime{roi.ModTime},
DeleteMarker: roi.DeleteMarker,
},
Bucket: roi.Bucket,
OpType: replication.HealReplicationType,
EventType: ReplicateHealDelete,
Bucket: roi.Bucket,
OpType: replication.HealReplicationType,
EventType: ReplicateHealDelete,
RetryCount: retryCount,
}
// heal delete marker replication failure or versioned delete replication failure
if roi.ReplicationStatus == replication.Pending ||
@@ -3820,6 +3990,7 @@ func (p *ReplicationPool) queueMRFSave(entry MRFReplicateEntry) {
if entry.RetryCount > mrfRetryLimit { // let scanner catch up if retry count exceeded
atomic.AddUint64(&p.stats.mrfStats.TotalDroppedCount, 1)
atomic.AddUint64(&p.stats.mrfStats.TotalDroppedBytes, uint64(entry.sz))
replLogOnceIf(GlobalContext, errors.New("Replication MRF retry limit reached; further repair is deferred to the scanner"), "replication-mrf-retry-limit", logger.WarningKind)
return
}
@@ -3834,6 +4005,7 @@ func (p *ReplicationPool) queueMRFSave(entry MRFReplicateEntry) {
default:
atomic.AddUint64(&p.stats.mrfStats.TotalDroppedCount, 1)
atomic.AddUint64(&p.stats.mrfStats.TotalDroppedBytes, uint64(entry.sz))
replLogOnceIf(GlobalContext, errors.New("Replication MRF queue is full; dropped entries will need scanner repair"), "replication-mrf-queue-full", logger.WarningKind)
}
}
}
@@ -4007,7 +4179,12 @@ func (p *ReplicationPool) queueMRFHeal() error {
VersionID: vID,
})
cancel()
if err != nil {
// A versioned marker lookup returns its metadata with a 405. Only
// accept that error with a real, matching marker identity.
validMarker := isErrMethodNotAllowed(err) && oi.DeleteMarker &&
vID != "" && oi.VersionID == vID && !oi.ModTime.IsZero() &&
oi.Bucket == e.Bucket && oi.Name != "" && oi.Name == decodeDirObject(e.Object)
if err != nil && !validMarker || oi.Name == "" {
continue
}
@@ -4073,6 +4250,11 @@ func getCRCMeta(oi ObjectInfo, partNum int, h http.Header) (cs map[string]string
if cktype.IsSet() {
meta[cktype.Key()] = v
meta[xhttp.AmzChecksumAlgo] = cktype.String()
// Without the type the target creates a COMPOSITE multipart
// upload and rejects the full object CRC on completion.
if isMP && cktype.CanMerge() && cs[xhttp.AmzChecksumType] == xhttp.AmzChecksumTypeFullObject {
meta[xhttp.AmzChecksumType] = xhttp.AmzChecksumTypeFullObject
}
}
}
return meta, isMP
+19 -26
View File
@@ -26,7 +26,6 @@ import (
"path"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/synctest"
"time"
@@ -328,20 +327,19 @@ func TestReplicationValidationObjectUsesRulePrefix(t *testing.T) {
func newTestResyncer(bucket, arn string) (*replicationResyncer, resyncOpts) {
s := &replicationResyncer{
statusMap: map[string]BucketReplicationResyncStatus{},
resyncCancelCh: make(chan struct{}, resyncWorkerCnt),
statusMap: map[string]BucketReplicationResyncStatus{},
}
brs := newBucketResyncStatus(bucket)
brs.TargetsMap[arn] = TargetReplicationResyncStatus{ResyncStatus: ResyncStarted}
brs.TargetsMap[arn] = TargetReplicationResyncStatus{ResyncStatus: ResyncStarted, ResyncID: "reset-" + bucket}
s.statusMap[bucket] = brs
return s, resyncOpts{bucket: bucket, arn: arn, resyncID: "reset-" + bucket}
}
// TestResyncBucketFinalize round-trips the terminal status through a real
// ObjectLayer: a clean run persists Completed with every result, while a run
// whose parent context was canceled during the drain, or in which a worker
// dropped a result on the cancel signal, is downgraded to Failed so a persisted
// Completed never misrepresents an incomplete resync.
// whose parent context was canceled during the drain is downgraded to Failed;
// a user-canceled run persists Canceled. Completed never misrepresents an
// incomplete resync.
func TestResyncBucketFinalize(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
@@ -354,9 +352,9 @@ func TestResyncBucketFinalize(t *testing.T) {
// persistTerminal applies resyncBucket's finalizer logic (finalResyncStatus
// then markStatus, which persists) and reads the status back the way the
// resync status API does.
persistTerminal := func(t *testing.T, s *replicationResyncer, opts resyncOpts, status ResyncStatusType, ctxErr error, aborted bool) TargetReplicationResyncStatus {
persistTerminal := func(t *testing.T, s *replicationResyncer, opts resyncOpts, status ResyncStatusType, cause error) TargetReplicationResyncStatus {
t.Helper()
s.markStatus(finalResyncStatus(status, ctxErr, aborted), opts, objAPI)
s.markStatus(finalResyncStatus(status, cause), opts, objAPI)
brs, err := loadBucketResyncMetadata(ctx, opts.bucket, objAPI)
if err != nil {
t.Fatalf("load persisted resync metadata: %v", err)
@@ -376,7 +374,7 @@ func TestResyncBucketFinalize(t *testing.T) {
var wg sync.WaitGroup // no producer workers for this case
results.finish(nil, &wg)
st := persistTerminal(t, s, opts, ResyncCompleted, nil, false)
st := persistTerminal(t, s, opts, ResyncCompleted, nil)
if st.ResyncStatus != ResyncCompleted {
t.Fatalf("persisted status = %s, want Completed", st.ResyncStatus)
}
@@ -398,29 +396,24 @@ func TestResyncBucketFinalize(t *testing.T) {
cctx, ccancel := context.WithCancel(context.Background())
ccancel()
st := persistTerminal(t, s, opts, ResyncCompleted, cctx.Err(), false)
st := persistTerminal(t, s, opts, ResyncCompleted, context.Cause(cctx))
if st.ResyncStatus != ResyncFailed {
t.Fatalf("persisted status = %s, want Failed (parent canceled during drain)", st.ResyncStatus)
}
})
// 3. A worker dropped a computed result on the resync-cancel token (parent
// still alive) -> sendResyncResult records the abort and Completed is
// downgraded to Failed.
t.Run("worker abort downgrades to failed", func(t *testing.T) {
// 3. A user-canceled worker cannot report a completed resync.
t.Run("user cancel persists canceled", func(t *testing.T) {
s, opts := newTestResyncer("finalize-worker-abort", "arn1")
s.resyncCancelCh <- struct{}{} // cancel token waiting
ch := make(chan TargetReplicationResyncStatus) // no reader: the send would block
var aborted atomic.Bool
if s.sendResyncResult(context.Background(), ch, TargetReplicationResyncStatus{Object: "dropped", ReplicatedCount: 1}, &aborted) {
t.Fatal("sendResyncResult reported success despite the cancel token")
ctx, cancel := context.WithCancelCause(context.Background())
cancel(errResyncCanceled)
ch := make(chan TargetReplicationResyncStatus)
if s.sendResyncResult(ctx, ch, TargetReplicationResyncStatus{Object: "dropped", ReplicatedCount: 1}) {
t.Fatal("sendResyncResult reported success after cancellation")
}
if !aborted.Load() {
t.Fatal("worker abort was not recorded")
}
st := persistTerminal(t, s, opts, ResyncCompleted, nil, aborted.Load())
if st.ResyncStatus != ResyncFailed {
t.Fatalf("persisted status = %s, want Failed (worker dropped a result)", st.ResyncStatus)
st := persistTerminal(t, s, opts, ResyncCompleted, context.Cause(ctx))
if st.ResyncStatus != ResyncCanceled {
t.Fatalf("persisted status = %s, want Canceled", st.ResyncStatus)
}
})
}
+6 -2
View File
@@ -319,7 +319,9 @@ func (r *ReplicationStats) getNodeQueueStats(bucket string) (qs ReplQNodeStats)
qs.QStats = r.qCache.getBucketStats(bucket)
qs.TgtXferStats = make(map[string]map[RMetricName]XferStats)
qs.MRFStats = ReplicationMRFStats{
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
TotalDroppedCount: atomic.LoadUint64(&r.mrfStats.TotalDroppedCount),
TotalDroppedBytes: atomic.LoadUint64(&r.mrfStats.TotalDroppedBytes),
}
r.RLock()
@@ -410,7 +412,9 @@ func (r *ReplicationStats) getNodeQueueStatsSummary() (qs ReplQNodeStats) {
qs.XferStats = make(map[RMetricName]XferStats)
qs.QStats = r.qCache.getSiteStats()
qs.MRFStats = ReplicationMRFStats{
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
LastFailedCount: atomic.LoadUint64(&r.mrfStats.LastFailedCount),
TotalDroppedCount: atomic.LoadUint64(&r.mrfStats.TotalDroppedCount),
TotalDroppedBytes: atomic.LoadUint64(&r.mrfStats.TotalDroppedBytes),
}
r.RLock()
defer r.RUnlock()
+3 -3
View File
@@ -97,7 +97,7 @@ func (api objectAPIHandlers) PutBucketVersioningHandler(w http.ResponseWriter, r
return
}
updatedAt, err := globalBucketMetadataSys.Update(ctx, bucket, bucketVersioningConfig, configData)
result, err := globalBucketMetadataSys.updateAndParseMetadata(ctx, bucket, bucketVersioningConfig, configData, false, false, nil)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -107,12 +107,12 @@ func (api objectAPIHandlers) PutBucketVersioningHandler(w http.ResponseWriter, r
//
// We encode the xml bytes as base64 to ensure there are no encoding
// errors.
cfgStr := base64.StdEncoding.EncodeToString(configData)
cfgStr := base64.StdEncoding.EncodeToString(result.meta.VersioningConfigXML)
replLogIf(ctx, globalSiteReplicationSys.BucketMetaHook(ctx, madmin.SRBucketMeta{
Type: madmin.SRBucketMetaTypeVersionConfig,
Bucket: bucket,
Versioning: &cfgStr,
UpdatedAt: updatedAt,
UpdatedAt: result.updatedAt,
}))
writeSuccessResponseHeadersOnly(w)
+58 -7
View File
@@ -56,6 +56,7 @@ import (
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/color"
"github.com/minio/minio/internal/config"
"github.com/minio/minio/internal/config/api"
"github.com/minio/minio/internal/handlers"
"github.com/minio/minio/internal/kms"
"github.com/minio/minio/internal/logger"
@@ -120,15 +121,40 @@ func init() {
const consolePrefix = "CONSOLE_"
// consoleMinIOServerEnv derives the CONSOLE_MINIO_SERVER value the embedded
// Console uses to reach the S3/STS API, and whether TLS verification of that
// endpoint must be skipped. With no explicit endpoint configured the Console
// reaches the API over the loopback address, whose TLS certificate is not
// expected to carry a 127.0.0.1 SAN; because Console verifies outbound TLS by
// default, the loopback origin has to be exempted or embedded login (local and
// LDAP alike) fails at the STS handshake. The exemption is endpoint-scoped in
// Console, so every other HTTPS peer stays verified. An explicitly configured
// endpoint is always reached under its own verified name and is never exempted.
func consoleMinIOServerEnv(endpoint string, isTLS bool, port string) (server string, skipVerify bool) {
if endpoint != "" {
return endpoint, false
}
return fmt.Sprintf("%s://127.0.0.1:%s", getURLScheme(isTLS), port), isTLS
}
func minioConfigToConsoleFeatures() {
os.Setenv("CONSOLE_PBKDF_SALT", globalDeploymentID())
os.Setenv("CONSOLE_PBKDF_PASSPHRASE", globalDeploymentID())
if globalMinioEndpoint != "" {
os.Setenv("CONSOLE_MINIO_SERVER", globalMinioEndpoint)
consoleServer, skipVerify := consoleMinIOServerEnv(globalMinioEndpoint, globalIsTLS, globalMinioPort)
os.Setenv("CONSOLE_MINIO_SERVER", consoleServer)
if skipVerify {
// The embedded Console reaches the loopback S3/STS endpoint above, whose
// certificate is not expected to carry a 127.0.0.1 SAN. Console verifies
// outbound TLS by default (silo-console v2.3.x), so opt into the
// endpoint-scoped compatibility switch to preserve the documented loopback
// bypass; every other HTTPS peer (IdP, Prometheus, webhooks, ...) stays
// verified. initConsoleServer unsets CONSOLE_* before calling this, so the
// switch cannot be supplied by the operator on the embedded path.
os.Setenv("CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY", "on")
} else {
// Explicitly set 127.0.0.1 so Console will automatically bypass TLS verification to the local S3 API.
// This will save users from providing a certificate with IP or FQDN SAN that points to the local host.
os.Setenv("CONSOLE_MINIO_SERVER", fmt.Sprintf("%s://127.0.0.1:%s", getURLScheme(globalIsTLS), globalMinioPort))
// An explicitly configured endpoint is reached under its own verified name;
// never let a loopback exemption apply to it.
os.Unsetenv("CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY")
}
if value := env.Get(config.EnvMinIOLogQueryURL, ""); value != "" {
os.Setenv("CONSOLE_LOG_QUERY_URL", value)
@@ -232,11 +258,31 @@ func buildOpenIDConsoleConfig() consoleoauth2.OpenIDPCfg {
return m
}
func initConsoleServer() (*consoleapi.Server, error) {
// unset all console_ environment variables.
// resetConsoleEnvironment preserves the embedded Console's supported resource
// settings verbatim. Server derives all other Console settings itself.
func resetConsoleEnvironment() {
for _, cenv := range env.List(consolePrefix) {
switch cenv {
case consoleapi.ConsoleWSMaxConnections,
consoleapi.ConsoleWSMaxConnectionsPerClient,
consoleapi.ConsoleWSMaxAnonymousConnections,
consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient:
continue
}
os.Unsetenv(cenv)
}
}
func initConsoleServer() (*consoleapi.Server, error) {
resetConsoleEnvironment()
// Validate explicitly: ConfigureAPI logs errors, but embedded Console logs
// are normally silenced. Return configuration failures to Server startup.
if err := consoleapi.ConfigureEmbeddedSourceIPTrust(); err != nil {
return nil, err
}
if err := consoleapi.ConfigureWebSocketLimits(); err != nil {
return nil, err
}
// enable all console environment variables
minioConfigToConsoleFeatures()
@@ -400,6 +446,7 @@ func buildServerCtxt(ctx *cli.Context, ctxt *serverCtxt) (err error) {
ctxt.SendBufSize = ctx.Int("send-buf-size")
ctxt.RecvBufSize = ctx.Int("recv-buf-size")
ctxt.IdleTimeout = ctx.Duration("idle-timeout")
ctxt.ReadHeaderTimeout = ctx.Duration("read-header-timeout")
ctxt.UserTimeout = ctx.Duration("conn-user-timeout")
if conf := ctx.String("config"); len(conf) > 0 {
@@ -835,6 +882,9 @@ func serverHandleEnvVars() {
logger.Warning("%s=%s is ignored: Silo in-place updates are permanently disabled", config.EnvUpdate, updateSetting)
}
globalInplaceUpdateDisabled = true
if env.IsSet(api.EnvAPIMultipartListing) {
logger.Warning("%s is ignored: multipart listing uses the legacy cache/exact-key behavior; persistent listing is deferred", api.EnvAPIMultipartListing)
}
// Check if the supported credential env vars,
// "MINIO_ROOT_USER" and "MINIO_ROOT_PASSWORD" are provided
@@ -855,6 +905,7 @@ func serverHandleEnvVars() {
}
globalEnableSyncBoot = env.Get("MINIO_SYNC_BOOT", config.EnableOff) == config.EnableOn
globalSiteReplicationMetadataTombstones = env.Get("MINIO_SITE_REPLICATION_METADATA_TOMBSTONES", config.EnableOff) == config.EnableOn
}
func loadRootCredentials() auth.Credentials {
+136
View File
@@ -26,6 +26,7 @@ import (
"strings"
"testing"
consoleapi "github.com/minio/console/api"
"github.com/minio/minio/internal/config"
)
@@ -372,3 +373,138 @@ func TestConfigEnvFileNamedTargetDiscovery(t *testing.T) {
t.Fatalf("named target %q not discovered from %s: %v", "my-hook", key, targets)
}
}
// TestConsoleMinIOServerEnv locks in the loopback TLS exemption that keeps
// embedded Console login working (issue #108) while ensuring an explicitly
// configured endpoint is never silently exempted from TLS verification.
func TestConsoleMinIOServerEnv(t *testing.T) {
tests := []struct {
name string
endpoint string
isTLS bool
port string
wantServer string
wantSkipVerify bool
}{
{
name: "loopback TLS is exempted so embedded login works",
isTLS: true,
port: "9000",
wantServer: "https://127.0.0.1:9000",
wantSkipVerify: true,
},
{
name: "loopback plain HTTP needs no exemption",
isTLS: false,
port: "9000",
wantServer: "http://127.0.0.1:9000",
},
{
name: "explicit https endpoint stays verified",
endpoint: "https://silo.example:9000",
isTLS: true,
port: "9000",
wantServer: "https://silo.example:9000",
},
{
name: "explicit http endpoint stays verified",
endpoint: "http://silo.example:9000",
isTLS: false,
port: "9000",
wantServer: "http://silo.example:9000",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
server, skipVerify := consoleMinIOServerEnv(tt.endpoint, tt.isTLS, tt.port)
if server != tt.wantServer {
t.Fatalf("server = %q, want %q", server, tt.wantServer)
}
if skipVerify != tt.wantSkipVerify {
t.Fatalf("skipVerify = %v, want %v", skipVerify, tt.wantSkipVerify)
}
})
}
}
// The startup path clears process environment, so preserve all existing Console
// variables, including ones unrelated to this test, before exercising it.
func preserveConsoleEnvironment(t *testing.T) {
t.Helper()
for _, entry := range os.Environ() {
if strings.HasPrefix(entry, consolePrefix) {
name, value, _ := strings.Cut(entry, "=")
t.Setenv(name, value)
}
}
}
func TestResetConsoleEnvironment(t *testing.T) {
preserveConsoleEnvironment(t)
settings := map[string]string{
consoleapi.ConsoleWSMaxConnections: "2048",
consoleapi.ConsoleWSMaxConnectionsPerClient: "512",
consoleapi.ConsoleWSMaxAnonymousConnections: "128",
consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient: "16",
}
for key, value := range settings {
t.Setenv(key, value)
}
decoys := []string{"CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY", "CONSOLE_MINIO_SERVER", "CONSOLE_PBKDF_SALT", "CONSOLE_TRUSTED_PROXIES", "CONSOLE_WS_MAX_UNKNOWN"}
for _, key := range decoys {
t.Setenv(key, "operator-value")
}
resetConsoleEnvironment()
for key, want := range settings {
if got, present := os.LookupEnv(key); !present || got != want {
t.Errorf("%s = %q, present = %v; want %q", key, got, present, want)
}
}
for _, key := range decoys {
if _, present := os.LookupEnv(key); present {
t.Errorf("unsupported override %s survived", key)
}
}
for _, raw := range []string{"", " 16 ", "env://missing-limit"} {
t.Setenv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient, raw)
resetConsoleEnvironment()
if got, present := os.LookupEnv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient); !present || got != raw {
t.Fatalf("raw value %q was changed to %q (present = %v)", raw, got, present)
}
}
os.Unsetenv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient)
resetConsoleEnvironment()
if _, present := os.LookupEnv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient); present {
t.Fatal("unset setting became present")
}
}
func TestInitConsoleServerConfigurationErrors(t *testing.T) {
for _, tt := range []struct {
name, proxy, limit, want string
}{
{"proxy error precedes limit error", "proxy.internal", "bad", "MINIO_API_TRUSTED_PROXIES"},
{"blank limit", "", "", "CONSOLE_WS_MAX_ANONYMOUS_CONNECTIONS_PER_CLIENT"},
{"non-integer limit", "", "bad", "CONSOLE_WS_MAX_ANONYMOUS_CONNECTIONS_PER_CLIENT"},
{"out-of-range limit", "", "0", "CONSOLE_WS_MAX_ANONYMOUS_CONNECTIONS_PER_CLIENT"},
{"inconsistent limits", "", "256", "must be less than"},
} {
t.Run(tt.name, func(t *testing.T) {
// Restore the process-wide library configuration after environment cleanup.
t.Cleanup(func() {
_ = consoleapi.ConfigureEmbeddedSourceIPTrust()
_ = consoleapi.ConfigureWebSocketLimits()
})
preserveConsoleEnvironment(t)
t.Setenv(consoleapi.EnvMinIOTrustedProxies, tt.proxy)
t.Setenv(consoleapi.ConsoleWSMaxConnections, "1024")
t.Setenv(consoleapi.ConsoleWSMaxConnectionsPerClient, "256")
t.Setenv(consoleapi.ConsoleWSMaxAnonymousConnections, "64")
t.Setenv(consoleapi.ConsoleWSMaxAnonymousConnectionsPerClient, tt.limit)
server, err := initConsoleServer()
if err == nil || !strings.Contains(err.Error(), tt.want) || server != nil {
t.Fatalf("initConsoleServer() = %v, %v; want nil server and %q error", server, err, tt.want)
}
})
}
}
+1 -1
View File
@@ -91,7 +91,7 @@ func readConfigWithoutMigrate(ctx context.Context, objAPI ObjectLayer) (config.C
return nil, err
}
data, err = decryptData(data, configFile)
data, err = decryptData(ctx, data, configFile)
if err != nil {
return nil, err
}
+3 -3
View File
@@ -68,7 +68,7 @@ func listServerConfigHistory(ctx context.Context, objAPI ObjectLayer, withData b
continue
}
data, err = decryptData(data, obj.Name)
data, err = decryptData(ctx, data, obj.Name)
if err != nil {
// ignore history file that cannot be loaded.
continue
@@ -110,7 +110,7 @@ func readServerConfigHistory(ctx context.Context, objAPI ObjectLayer, uuidKV str
return nil, err
}
return decryptData(data, historyFile)
return decryptData(ctx, data, historyFile)
}
func saveServerConfigHistory(ctx context.Context, objAPI ObjectLayer, kv []byte) error {
@@ -162,7 +162,7 @@ func readServerConfig(ctx context.Context, objAPI ObjectLayer, data []byte) (con
return nil, err
}
data, err = decryptData(data, configFile)
data, err = decryptData(ctx, data, configFile)
if err != nil {
lookupConfigs(srvCfg, objAPI)
return nil, err
+89
View File
@@ -0,0 +1,89 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"encoding/json"
"os"
"reflect"
"testing"
)
// Read a real pre-removal scanner cache with nonzero hot-tier bytes. A v8
// payload with a relabeled header would not exercise the ignored hts field.
func TestDataUsageCacheReadV9(t *testing.T) {
raw, err := os.ReadFile("testdata/data-usage-v9/data-usage-v9.bin")
if err != nil {
t.Fatal(err)
}
if len(raw) == 0 || raw[0] != 9 {
t.Fatal("fixture is not v9")
}
expected, err := os.ReadFile("testdata/data-usage-v9/data-usage-v9.json")
if err != nil {
t.Fatal(err)
}
var want, got dataUsageCache
if err := json.Unmarshal(expected, &want); err != nil {
t.Fatal(err)
}
if err := got.deserialize(bytes.NewReader(raw)); err != nil {
t.Fatal(err)
}
if !got.Info.LastUpdate.Equal(want.Info.LastUpdate) {
t.Fatal("cache timestamp changed")
}
// msgp restores local time while JSON preserves the UTC representation.
want.Info.LastUpdate = got.Info.LastUpdate
if !reflect.DeepEqual(got, want) {
t.Fatalf("v9 ordinary cache fields changed:\ngot: %#v\nwant: %#v", got, want)
}
flat := got.flatten(*got.root())
if flat.Size != 74962 || flat.Objects != 3 || flat.Versions != 5 || flat.DeleteMarkers != 2 {
t.Fatalf("statistics changed: %+v", flat)
}
if flat.AllTierStats == nil || flat.AllTierStats.Tiers["COLD"] != (tierStats{TotalSize: 65536, NumVersions: 1, NumObjects: 1}) {
t.Fatalf("remote tier statistics changed: %+v", flat.AllTierStats)
}
buckets := []BucketInfo{{Name: "v9-bucket"}}
if gotInfo, wantInfo := got.dui(dataUsageRoot, buckets), want.dui(dataUsageRoot, buckets); !reflect.DeepEqual(gotInfo, wantInfo) {
t.Fatalf("bucket usage aggregation changed: %+v != %+v", gotInfo, wantInfo)
}
var buf bytes.Buffer
if err := got.serializeTo(&buf); err != nil {
t.Fatal(err)
}
if buf.Bytes()[0] != 8 {
t.Fatalf("wrote cache version %d, want 8", buf.Bytes()[0])
}
var roundtrip dataUsageCache
if err := roundtrip.deserialize(&buf); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, roundtrip) {
t.Fatal("v8 round trip lost ordinary fields")
}
if err := roundtrip.deserialize(bytes.NewReader(raw[:len(raw)/2])); err == nil {
t.Fatal("truncated v9 cache accepted")
}
raw[0] = 10
if err := roundtrip.deserialize(bytes.NewReader(raw)); err == nil {
t.Fatal("unknown cache version accepted")
}
}
+2 -1
View File
@@ -981,6 +981,7 @@ func (d *dataUsageCache) save(ctx context.Context, store objectIO, name string)
// and write new data with the new version.
const (
dataUsageCacheVerCurrent = 8
dataUsageCacheVerV9 = 9 // Retired access-tier cache; only adds the ignored "hts" entry key.
dataUsageCacheVerV7 = 7
dataUsageCacheVerV6 = 6
dataUsageCacheVerV5 = 5
@@ -1182,7 +1183,7 @@ func (d *dataUsageCache) deserialize(r io.Reader) error {
}
return nil
case dataUsageCacheVerCurrent:
case dataUsageCacheVerCurrent, dataUsageCacheVerV9:
// Zstd compressed.
dec, err := zstd.NewReader(r, zstd.WithDecoderConcurrency(2))
if err != nil {
@@ -0,0 +1,109 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"net/http"
"net/http/httptest"
"testing"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
)
// An exact-version DELETE without a replication decision physically purges
// the version. Its response must describe the stored version: a data version
// is not a delete marker even while pending-purge metadata makes the lookup
// expose it as deleted, and a stored marker stays a marker.
func TestDeleteMarkerPurgeResponseIdentity(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, endpoints: []string{"DeleteObject"}, objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
defer replicationTestCapacity(obj)()
ctx := t.Context()
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
// Seed the purge state the DELETE handler would record for a target.
arn := "arn:minio:replication::" + mustGetUUID() + ":bucket"
pendingPurge := ReplicationState{
VersionPurgeStatusInternal: arn + "=PENDING;",
PurgeTargets: map[string]VersionPurgeStatusType{arn: replication.VersionPurgePending},
}
for _, tc := range []struct {
name string
marker, purged bool
}{
{name: "data"},
{name: "data-pending-purge", purged: true},
{name: "marker", marker: true},
{name: "marker-pending-purge", marker: true, purged: true},
} {
name := "purge-response-" + mustGetUUID()
oi, err := obj.PutObject(ctx, bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
version := oi.VersionID
if tc.marker {
opts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow(), ReplicationRequest: true}
opts.SetReplicaStatus(replication.Replica)
if _, err := obj.DeleteObject(ctx, bucket, name, opts); err != nil {
t.Fatal(err)
}
version = opts.VersionID
}
if tc.purged {
if _, err := obj.DeleteObject(ctx, bucket, name, ObjectOptions{Versioned: true, VersionID: version, DeleteReplication: pendingPurge}); err != nil {
t.Fatal(err)
}
}
before, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{Versioned: true, VersionID: version})
if tc.marker || tc.purged {
if !isErrMethodNotAllowed(err) || !before.DeleteMarker {
t.Fatalf("%s: seeded version not exposed as deleted: %+v %v", tc.name, before, err)
}
} else if err != nil || before.DeleteMarker {
t.Fatalf("%s: seeded data version: %+v %v", tc.name, before, err)
}
if tc.purged && before.VersionPurgeStatus != replication.VersionPurgePending {
t.Fatalf("%s: purge state not seeded: %+v", tc.name, before)
}
req, err := newTestSignedRequestV4(http.MethodDelete, "/"+bucket+"/"+name+"?versionId="+version, 0, nil, creds.AccessKey, creds.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusNoContent {
t.Fatalf("%s: status=%d: %s", tc.name, rec.Code, rec.Body.String())
}
// The handler sets these headers by map key, not canonical name.
if got := rec.Header()[xhttp.AmzVersionID]; len(got) != 1 || got[0] != version {
t.Fatalf("%s: response version %q, want %q", tc.name, got, version)
}
if got := len(rec.Header()[xhttp.AmzDeleteMarker]) == 1 && rec.Header()[xhttp.AmzDeleteMarker][0] == "true"; got != tc.marker {
t.Fatalf("%s: x-amz-delete-marker=%v for a stored marker=%v", tc.name, got, tc.marker)
}
if _, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{Versioned: true, VersionID: version}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("%s: version not purged: %v", tc.name, err)
}
t.Logf("%s %s: purged with x-amz-delete-marker=%q", backend, tc.name, rec.Header()[xhttp.AmzDeleteMarker])
}
}})
}
+524
View File
@@ -0,0 +1,524 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"errors"
"fmt"
"maps"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/once"
)
func TestDeleteMarkerPurgeIntent(t *testing.T) {
version := mustGetUUID()
for _, tc := range []struct {
name string
edit func(*ObjectOptions)
want bool
}{
{"ordinary", func(*ObjectOptions) {}, true},
{"receiver", func(o *ObjectOptions) { o.ReplicationRequest = true; o.SetReplicaStatus(replication.Replica) }, true},
{"untrusted-replica", func(o *ObjectOptions) { o.SetReplicaStatus(replication.Replica) }, false},
{"complete-purge", func(o *ObjectOptions) {
o.DeleteReplication.VersionPurgeStatusInternal = string(replication.VersionPurgeComplete)
}, true},
{"pending-purge", func(o *ObjectOptions) { o.DeleteReplication.VersionPurgeStatusInternal = "PENDING" }, false},
{"failed-purge", func(o *ObjectOptions) { o.DeleteReplication.VersionPurgeStatusInternal = "FAILED" }, false},
{"unknown-purge", func(o *ObjectOptions) { o.DeleteReplication.VersionPurgeStatusInternal = "future-state" }, false},
{"pending-creation", func(o *ObjectOptions) { o.DeleteReplication.ReplicationStatusInternal = "PENDING" }, false},
{"failed-creation", func(o *ObjectOptions) { o.DeleteReplication.ReplicationStatusInternal = "FAILED" }, false},
{"completed-creation", func(o *ObjectOptions) { o.DeleteReplication.ReplicationStatusInternal = "COMPLETED" }, false},
{"create-marker", func(o *ObjectOptions) { o.DeleteMarker = true }, false},
{"empty", func(o *ObjectOptions) { o.VersionID = "" }, false},
{"null", func(o *ObjectOptions) { o.VersionID = nullVersionID }, false},
{"invalid", func(o *ObjectOptions) { o.VersionID = "invalid" }, false},
{"zero-uuid", func(o *ObjectOptions) { o.VersionID = emptyUUID }, false},
{"movement", func(o *ObjectOptions) { o.DataMovement = true }, false},
{"free-version", func(o *ObjectOptions) { o.InclFreeVersions = true }, false},
{"expiration", func(o *ObjectOptions) { o.Expiration.Expire = true }, false},
{"transition", func(o *ObjectOptions) { o.Transition.Status = "complete" }, false},
{"restored-expiration", func(o *ObjectOptions) { o.Transition.ExpireRestored = true }, false},
} {
t.Run(tc.name, func(t *testing.T) {
o := ObjectOptions{VersionID: version, Versioned: true}
tc.edit(&o)
if got := o.isVersionPurge(); got != tc.want {
t.Fatalf("purge=%v want=%v", got, tc.want)
}
})
}
}
// Every other StorageAPI method panics through the nil embedding: a proof may
// read metadata, but must never delete, heal, or ask a different storage API.
type absenceProofDisk struct {
StorageAPI
t *testing.T
err error
reads *atomic.Int32
deletes *atomic.Int32
}
func (d absenceProofDisk) ReadVersion(_ context.Context, _, _, _, _ string, opts ReadOptions) (FileInfo, error) {
if opts.ReadData || opts.Healing {
d.t.Error("absence proof requested data/healing")
}
d.reads.Add(1)
return FileInfo{}, d.err
}
func (d absenceProofDisk) DeleteVersion(_ context.Context, _, _ string, _ FileInfo, _ bool, _ DeleteOptions) error {
if d.deletes == nil {
d.t.Error("read-only absence proof attempted a deletion")
} else {
d.deletes.Add(1)
}
return d.err
}
func TestVersionPurgeAbsenceProof(t *testing.T) {
for _, tc := range []struct {
name string
errs []error
quorum, all bool
}{
{"all-absent", []error{errFileNotFound, errFileVersionNotFound, errFileNotFound, errFileVersionNotFound}, true, true},
{"majority-absent", []error{errFileNotFound, errFileVersionNotFound, errFileNotFound, nil}, true, false},
{"half-absent", []error{errFileNotFound, errFileVersionNotFound, errDiskNotFound, errDiskNotFound}, false, false},
{"corrupt", []error{errFileNotFound, errFileVersionNotFound, errFileCorrupt, errFileCorrupt}, false, false},
{"permission", []error{errFileNotFound, errFileVersionNotFound, errDiskAccessDenied, errVolumeAccessDenied}, false, false},
{"volume-missing", []error{errFileNotFound, errFileVersionNotFound, errVolumeNotFound, errVolumeNotFound}, false, false},
} {
t.Run(tc.name, func(t *testing.T) {
var reads atomic.Int32
disks := make([]StorageAPI, len(tc.errs))
for i, err := range tc.errs {
disks[i] = absenceProofDisk{t: t, err: err, reads: &reads}
}
er := erasureObjects{getDisks: func() []StorageAPI { return disks }}
oldQueue := globalMRFState.opCh
globalMRFState.opCh = make(chan PartialOperation, 1)
defer func() { globalMRFState.opCh = oldQueue }()
all, err := er.confirmVersionAbsent(t.Context(), "bucket", "object", mustGetUUID())
if (err == nil) != tc.quorum || all != tc.all || reads.Load() != int32(len(disks)) || len(globalMRFState.opCh) != 0 {
t.Fatalf("proof all=%v error=%v reads=%d queued=%d", all, err, reads.Load(), len(globalMRFState.opCh))
}
})
}
}
func TestVersionPurgeAggregation(t *testing.T) {
for _, pure := range []bool{false, true} {
for _, fault := range []error{nil, errFileCorrupt, errDiskAccessDenied} {
t.Run(fmt.Sprintf("purge_%v_fault_%v", pure, fault), func(t *testing.T) {
var calls atomic.Int32
disks := make([]StorageAPI, 4)
for i, err := range []error{nil, errFileNotFound, errFileVersionNotFound, fault} {
disks[i] = absenceProofDisk{t: t, err: err, deletes: &calls}
}
er := erasureObjects{getDisks: func() []StorageAPI { return disks }}
err := er.deleteObjectVersion(t.Context(), "bucket", "object", FileInfo{VersionID: mustGetUUID()}, false, pure)
if (err == nil) != pure || calls.Load() != 4 {
t.Fatalf("aggregation error=%v calls=%d", err, calls.Load())
}
})
}
}
}
func TestDeleteMarkerPurgeOmittedPool(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
const name = "partially-absent"
er := z.serverPools[1].getHashedSet(name)
_, opts := seedPurgeMarker(t, er, bucket, name, false)
opts.ReplicationRequest = false
opts.DeleteReplication = ReplicationState{}
// Pool 0 appears absent at read quorum, but half of it cannot be read.
missing := z.serverPools[0].getHashedSet(name)
original := missing.getDisks
disks := append([]StorageAPI(nil), original()...)
clear(disks[len(disks)/2:])
missing.getDisks = func() []StorageAPI { return disks }
defer func() { missing.getDisks = original }()
_, err := z.DeleteObject(t.Context(), bucket, name, opts)
if !isErrWriteQuorum(err) {
t.Errorf("omitted pool acknowledged deletion: %T %v", err, err)
}
if countPurgeMarkers(t, er.getDisks(), bucket, name, opts.VersionID) != 16 {
t.Fatal("mutated known copies before checking omitted pool")
}
}
func TestDeleteMarkerPurgeReceivingPool(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
for _, duplicate := range []bool{false, true} {
t.Run(fmt.Sprintf("duplicate_%v", duplicate), func(t *testing.T) {
name := "receiver-" + mustGetUUID()
_, opts := seedPurgeMarker(t, z.serverPools[0].getHashedSet(name), bucket, name, false)
if duplicate {
create := opts
create.DeleteMarker = true
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, name, create); err != nil {
t.Fatal(err)
}
} else {
// Latest-key routing selects pool 1, but the addressed marker is
// in pool 0. A replica purge addresses a version, not the latest.
putConsistencyObject(t, z, bucket, name, 1, "newer-data", ObjectOptions{Versioned: true})
}
if _, err := z.DeleteObject(t.Context(), bucket, name, opts); err != nil {
t.Errorf("replica purge did not find its addressed version: %v", err)
}
for i, pool := range z.serverPools {
if got := countPurgeMarkers(t, pool.getHashedSet(name).getDisks(), bucket, name, opts.VersionID); got != 0 {
t.Errorf("replica purge left %d marker copies in pool %d", got, i)
}
}
})
}
}
func TestDeleteMarkerPurgeCallbackIntent(t *testing.T) {
for _, loseCopies := range []bool{false, true} {
t.Run(fmt.Sprintf("missing_after_callback_%v", loseCopies), func(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
const name = "callback-pending"
er := z.serverPools[0].getHashedSet(name)
_, opts := seedPurgeMarker(t, er, bucket, name, false)
original := er.getDisks
all := append([]StorageAPI(nil), original()...)
defer func() { er.getDisks = original }()
opts.ReplicationRequest = false
opts.DeleteReplication = ReplicationState{}
metadata, retention := 0, 0
opts.EvalRetentionBypassFn = func(oi ObjectInfo, err error) error {
retention++
if !oi.DeleteMarker || !isErrMethodNotAllowed(err) {
t.Fatalf("retention callback lost marker: %+v %v", oi, err)
}
return nil
}
opts.EvalMetadataFn = func(*ObjectInfo, error) (ReplicateDecision, error) {
metadata++
if loseCopies {
// Inject a disk-state change after preflight, before the set's
// metadata-update read. It returns NotFound at read quorum.
for _, disk := range all[:8] {
if err := disk.DeleteVersion(t.Context(), bucket, name, FileInfo{VersionID: opts.VersionID}, false, DeleteOptions{}); err != nil {
t.Fatal(err)
}
}
partial := make([]StorageAPI, len(all))
copy(partial, all[:8])
er.getDisks = func() []StorageAPI { return partial }
}
decision := ReplicateDecision{}
decision.Set(newReplicateTargetDecision("arn1", true, false))
return decision, nil
}
_, err := z.DeleteObject(t.Context(), bucket, name, opts)
if loseCopies {
if !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Errorf("pending update misclassified as physical purge: %T %v", err, err)
}
} else if err != nil {
t.Errorf("pending update failed: %v", err)
}
want := 16
if loseCopies {
want = 8
}
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != want || metadata != 1 || retention != 1 {
t.Errorf("pending update: copies=%d want=%d metadata=%d retention=%d", got, want, metadata, retention)
}
if !loseCopies {
fi, err := all[0].ReadVersion(t.Context(), "", bucket, name, opts.VersionID, ReadOptions{})
if err != nil || fi.VersionPurgeStatus() != replication.VersionPurgePending {
t.Errorf("pending state not persisted: %+v %v", fi.ReplicationState, err)
}
}
})
}
}
func TestDeleteMarkerReceivingMetadataUpdate(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
for _, state := range []VersionPurgeStatusType{replication.VersionPurgePending, replication.VersionPurgeFailed} {
t.Run(string(state), func(t *testing.T) {
name := "update-" + mustGetUUID()
_, opts := seedPurgeMarker(t, z.serverPools[0].getHashedSet(name), bucket, name, false)
create := opts
create.DeleteMarker = true
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, name, create); err != nil {
t.Fatal(err)
}
opts.DeleteReplication = ReplicationState{VersionPurgeStatusInternal: string(state)}
if _, err := z.DeleteObject(t.Context(), bucket, name, opts); err != nil {
t.Fatal(err)
}
for i, pool := range z.serverPools {
if got := countPurgeMarkers(t, pool.getHashedSet(name).getDisks(), bucket, name, opts.VersionID); got != 16 {
t.Errorf("metadata update removed marker copies in pool %d: %d", i, got)
}
}
})
}
}
func TestVersionPurgeRetentionGate(t *testing.T) {
z, er, all, bucket := markerPurgeFixture(t, 4)
data, _ := seedPurgeMarker(t, er, bucket, "protected-data", true)
denied := errors.New("retention denied")
opts := ObjectOptions{Versioned: true, VersionID: data, EvalRetentionBypassFn: func(oi ObjectInfo, err error) error {
if err != nil || oi.VersionID != data || oi.DeleteMarker {
t.Errorf("wrong retained version: %+v error=%v", oi, err)
}
return denied
}}
_, err := z.DeleteObject(t.Context(), bucket, "protected-data", opts)
if !errors.Is(err, denied) {
t.Fatalf("retention gate: %v", err)
}
for _, disk := range all {
if _, err := disk.ReadVersion(t.Context(), "", bucket, "protected-data", data, ReadOptions{}); err != nil {
t.Errorf("protected version changed: %v", err)
}
}
}
func markerPurgeFixture(t *testing.T, n int) (*erasureServerPools, *erasureObjects, []StorageAPI, string) {
t.Helper()
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure(ctx, n)
if err != nil {
cancel()
t.Fatal(err)
}
restore := replicationTestCapacity(obj)
t.Cleanup(func() {
cancel()
restore()
obj.Shutdown(context.Background())
removeRoots(dirs)
})
bucket := getRandomBucketName()
if err := obj.MakeBucket(ctx, bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
er := z.serverPools[0].sets[0]
original := er.getDisks
t.Cleanup(func() { er.getDisks = original })
return z, er, append([]StorageAPI(nil), original()...), bucket
}
func seedPurgeMarker(t *testing.T, er *erasureObjects, bucket, name string, withData bool) (string, ObjectOptions) {
t.Helper()
dataVersion := ""
if withData {
oi, err := er.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
dataVersion = oi.VersionID
}
opts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, ReplicationRequest: true, MTime: UTCNow().Add(-time.Hour), NoAuditLog: true}
opts.SetReplicaStatus(replication.Replica)
if _, err := er.DeleteObject(t.Context(), bucket, name, opts); err != nil {
t.Fatal(err)
}
opts.DeleteMarker = false
return dataVersion, opts
}
func countPurgeMarkers(t *testing.T, disks []StorageAPI, bucket, name, version string) int {
t.Helper()
n := 0
for _, disk := range disks {
fi, err := disk.ReadVersion(t.Context(), "", bucket, name, version, ReadOptions{})
if err == nil && fi.Deleted {
n++
} else if err != errFileNotFound && err != errFileVersionNotFound {
t.Fatalf("unexpected disk version: deleted=%v error=%v", fi.Deleted, err)
}
}
return n
}
func TestDeleteMarkerPurgeQuorum(t *testing.T) {
for _, tc := range []struct{ disks, online int }{{4, 2}, {16, 7}, {16, 8}, {16, 9}} {
t.Run(fmt.Sprintf("%d_disks_%d_online", tc.disks, tc.online), func(t *testing.T) {
z, er, all, bucket := markerPurgeFixture(t, tc.disks)
const name = "marker"
dataVersion, opts := seedPurgeMarker(t, er, bucket, name, true)
partial := make([]StorageAPI, len(all))
copy(partial, all[:tc.online])
er.getDisks = func() []StorageAPI { return partial }
_, first := z.DeleteObject(t.Context(), bucket, name, opts)
_, retry := z.DeleteObject(t.Context(), bucket, name, opts)
if tc.online <= tc.disks/2 {
if !isErrWriteQuorum(first) || !isErrWriteQuorum(retry) {
t.Errorf("below write quorum: first=%T %v, retry=%T %v", first, first, retry, retry)
}
} else if first != nil || (retry != nil && !isErrVersionNotFound(retry) && !isErrObjectNotFound(retry)) {
t.Errorf("write quorum available: first=%v retry=%v", first, retry)
}
want := tc.disks - tc.online
if tc.online < tc.disks/2 {
want = tc.disks
}
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != want {
t.Errorf("partial purge markers=%d want=%d", got, want)
}
er.getDisks = func() []StorageAPI { return all }
_, err := z.DeleteObject(t.Context(), bucket, name, opts)
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Errorf("full-online retry: %v", err)
}
if tc.online <= tc.disks/2 {
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != 0 {
t.Errorf("full-online retry recreated/retained %d marker copies", got)
}
}
// A quorum-confirmed missing retry can leave minority residue. Exercise
// the real dangling-version healer after every disk has returned.
_, _ = er.HealObject(t.Context(), bucket, name, opts.VersionID, madmin.HealOpts{Remove: true})
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != 0 {
t.Errorf("marker copies after retry and heal=%d", got)
}
oi, err := z.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{Versioned: true})
if err != nil || oi.DeleteMarker || oi.VersionID != dataVersion {
t.Errorf("underlying version not visible: %+v error=%v", oi, err)
}
})
}
}
func TestDeleteMarkerPurgeMissingKey(t *testing.T) {
for _, pooled := range []bool{false, true} {
t.Run(fmt.Sprintf("pooled_%v", pooled), func(t *testing.T) {
z, er, all, bucket := markerPurgeFixture(t, 4)
_, opts := seedPurgeMarker(t, er, bucket, "marker-only", false)
er.getDisks = func() []StorageAPI { return []StorageAPI{all[0], all[1], nil, nil} }
deleteFn := er.DeleteObject
if pooled {
deleteFn = z.DeleteObject
}
_, _ = deleteFn(t.Context(), bucket, "marker-only", opts)
_, err := deleteFn(t.Context(), bucket, "marker-only", opts)
if !isErrWriteQuorum(err) {
t.Errorf("missing retry with only 2/4 absence votes returned %T %v", err, err)
}
})
}
}
func TestDeleteMarkerHealReplicationIdentity(t *testing.T) {
obj, er, all, bucket := markerPurgeFixture(t, 4)
const name = "healed-marker"
_, opts := seedPurgeMarker(t, er, bucket, name, true)
original, err := all[3].ReadVersion(t.Context(), "", bucket, name, opts.VersionID, ReadOptions{})
if err != nil {
t.Fatal(err)
}
for _, disk := range all[:2] {
if err := disk.DeleteVersion(t.Context(), bucket, name, FileInfo{VersionID: opts.VersionID}, false, DeleteOptions{}); err != nil {
t.Fatal(err)
}
}
if _, err := er.HealObject(t.Context(), bucket, name, opts.VersionID, madmin.HealOpts{}); err != nil {
t.Fatal(err)
}
for i, disk := range all {
fi, err := disk.ReadVersion(t.Context(), "", bucket, name, opts.VersionID, ReadOptions{})
if err != nil || !maps.Equal(fi.Metadata, original.Metadata) {
t.Errorf("disk %d lost marker metadata: before=%v after=%v error=%v", i, original.Metadata, fi.Metadata, err)
}
}
// Only the healed half remains readable. Read actual disk state before
// invoking scanner replication, rather than constructing an ObjectInfo.
er.getDisks = func() []StorageAPI { return []StorageAPI{all[0], all[1], nil, nil} }
oi, err := er.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{VersionID: opts.VersionID, Versioned: true})
if !oi.DeleteMarker || !isErrMethodNotAllowed(err) {
t.Fatalf("healed marker unreadable: %+v %v", oi, err)
}
er.getDisks = func() []StorageAPI { return all }
var outbound atomic.Int32
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodDelete {
outbound.Add(1)
if r.Header.Get(xhttp.MinIOSourceDeleteMarker) != "true" || r.URL.Query().Get("versionId") != opts.VersionID {
t.Errorf("unexpected outbound request: %s %s", r.Method, r.URL)
}
w.WriteHeader(http.StatusNoContent)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer remote.Close()
client, err := minio.New(strings.TrimPrefix(remote.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
arn := "arn:minio:replication::" + mustGetUUID() + ":bucket"
target := &TargetClient{Client: client, ARN: arn, Bucket: "target"}
oldTargets := globalBucketTargetSys
globalBucketTargetSys = &BucketTargetSys{arnRemotesMap: map[string]arnTarget{arn: {Client: target, lastRefresh: UTCNow()}}, targetsMap: map[string][]madmin.BucketTarget{bucket: {{Arn: arn, TargetBucket: "target"}}}, hc: map[string]epHealth{client.EndpointURL().Host: {Online: true}}}
defer func() { globalBucketTargetSys = oldTargets }()
cfg := configs[0]
cfg.RoleArn = arn
meta, err := globalBucketMetadataSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
meta.replicationConfig = &cfg
globalBucketMetadataSys.Set(bucket, meta)
worker := make(chan ReplicationWorkerOperation, 1)
oldPool := globalReplicationPool
globalReplicationPool = once.NewSingleton[ReplicationPool]()
globalReplicationPool.Set(&ReplicationPool{ctx: t.Context(), objLayer: obj, workers: []chan ReplicationWorkerOperation{worker}, stats: globalReplicationStats.Load(), mrfSaveCh: make(chan MRFReplicateEntry, 1)})
defer func() { globalReplicationPool = oldPool }()
roi := queueReplicationHeal(t.Context(), bucket, oi, replicationConfig{Config: &cfg, remotes: &madmin.BucketTargets{Targets: []madmin.BucketTarget{{Arn: arn, TargetBucket: "target"}}}}, 0)
select {
case op := <-worker:
d := op.(DeletedObjectReplicationInfo)
result := replicateDeleteToTarget(t.Context(), d, target)
t.Errorf("healed replica scheduled for creation: version=%s existing=%v result=%+v", d.DeleteMarkerVersionID, roi.ExistingObjResync.mustResync(), result)
default:
}
if outbound.Load() != 0 {
t.Errorf("healed replica sent %d new marker creations", outbound.Load())
}
}
+5
View File
@@ -1068,6 +1068,11 @@ func (er erasureObjects) HealObject(ctx context.Context, bucket, object, version
newReqInfo = logger.NewReqInfo("", "", globalDeploymentID(), "", "Heal", bucket, object)
}
healCtx := logger.SetReqInfo(GlobalContext, newReqInfo)
if opts.NoLock {
// The caller owns the namespace lock. Stop if that lock's context is
// canceled instead of continuing metadata writes after losing it.
healCtx = logger.SetReqInfo(ctx, newReqInfo)
}
// Healing directories handle it separately.
if HasSuffix(object, SlashSeparator) {
+91
View File
@@ -20,6 +20,7 @@ package cmd
import (
"context"
"encoding/hex"
"errors"
"fmt"
"math/rand"
"reflect"
@@ -110,6 +111,96 @@ func TestReduceErrs(t *testing.T) {
}
}
// TestReduceWriteQuorumErrsCommitMetaMissing verifies that an in-quorum
// errCommitMetaMissing reduces to the sentinel, which renameData() converts
// to errErasureWriteQuorum, while below-quorum reduces directly to
// errErasureWriteQuorum.
func TestReduceWriteQuorumErrsCommitMetaMissing(t *testing.T) {
errs := make([]error, 16)
for i := range 9 {
errs[i] = errCommitMetaMissing
}
if err := reduceWriteQuorumErrs(t.Context(), errs, objectOpIgnoredErrs, 9); !errors.Is(err, errCommitMetaMissing) {
t.Errorf("expected %s, got %s", errCommitMetaMissing, err)
}
// The sentinel must be distinguishable from errFileNotFound so that
// commit failures are never mapped to ObjectNotFound (404).
if errors.Is(errCommitMetaMissing, errFileNotFound) {
t.Error("errCommitMetaMissing must not match errFileNotFound")
}
// Below quorum the reduction must yield a write-quorum failure (503);
// eight sentinels against eight committed drives is not a quorum.
belowQuorum := make([]error, 16)
for i := range 8 {
belowQuorum[i] = errCommitMetaMissing
}
if err := reduceWriteQuorumErrs(t.Context(), belowQuorum, objectOpIgnoredErrs, 9); !errors.Is(err, errErasureWriteQuorum) {
t.Errorf("expected %s, got %s", errErasureWriteQuorum, err)
}
}
// renameDataFailingDisk is a StorageAPI stub returning a fixed error
// from RenameData, enough to drive renameData() in unit tests.
type renameDataFailingDisk struct {
StorageAPI
err error
undoWrite bool
}
func (d *renameDataFailingDisk) RenameData(context.Context, string, string, FileInfo, string, string, RenameOptions) (RenameDataResp, error) {
return RenameDataResp{}, d.err
}
func (d *renameDataFailingDisk) DeleteVersion(_ context.Context, _, _ string, _ FileInfo, _ bool, opts DeleteOptions) error {
d.undoWrite = opts.UndoWrite
return nil
}
// TestRenameDataCommitMetaMissing verifies that renameData() converts an
// in-quorum errCommitMetaMissing into errErasureWriteQuorum (HTTP 503),
// while an in-quorum errFileNotFound keeps its meaning so that
// PutObject() still reports IncompleteBody for client disconnects.
func TestRenameDataCommitMetaMissing(t *testing.T) {
for _, tc := range []struct {
name string
failures int
diskErr error
wantErr error
}{
{"all_commit_metadata_missing", 16, errCommitMetaMissing, errErasureWriteQuorum},
{"quorum_commit_metadata_missing", 9, errCommitMetaMissing, errErasureWriteQuorum},
{"neither_side_has_quorum", 8, errCommitMetaMissing, errErasureWriteQuorum},
{"successful_write_quorum", 7, errCommitMetaMissing, nil},
{"staging_failure", 16, errFileNotFound, errFileNotFound},
} {
t.Run(tc.name, func(t *testing.T) {
metadata := make([]FileInfo, 16)
disks := make([]StorageAPI, 16)
for i := range disks {
metadata[i] = FileInfo{Deleted: true} // valid without erasure info
disk := &renameDataFailingDisk{}
if i < tc.failures {
disk.err = tc.diskErr
}
disks[i] = disk
}
_, _, _, err := renameData(t.Context(), disks, minioMetaTmpBucket, "tmp", metadata, "bucket", "object", 9)
if !errors.Is(err, tc.wantErr) {
t.Fatalf("expected %v, got %v", tc.wantErr, err)
}
for i, disk := range disks {
// Failed writes undo only the drives that committed. A
// successful quorum must never be rolled back.
wantUndo := tc.wantErr != nil && i >= tc.failures
if got := disk.(*renameDataFailingDisk).undoWrite; got != wantUndo {
t.Errorf("disk %d: UndoWrite = %v, want %v", i, got, wantUndo)
}
}
})
}
}
// TestHashOrder - test order of ints in array
func TestHashOrder(t *testing.T) {
testCases := []struct {
+234
View File
@@ -0,0 +1,234 @@
// Copyright (c) 2026 clown15
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"encoding/base64"
"fmt"
"net/url"
"os"
"path/filepath"
"testing"
"time"
)
func TestExtractModTimeFromUploadIDDir(t *testing.T) {
now := time.Unix(1792056747, 123456789).UTC()
nano := now.UnixNano()
uuid := "d16ad42a-6fe5-4a3b-8092-69983142238f"
testCases := []struct {
name string
dirName string
expectedMod time.Time
wantZero bool
}{
{
name: "Modern Plaintext Format (<UUID>x<UnixNano>)",
dirName: fmt.Sprintf("%sx%d", uuid, nano),
expectedMod: time.Unix(0, nano),
wantZero: false,
},
{
// This valid UUID decodes to bytes containing 'x'. Checking
// for that byte before parsing plaintext loses the timestamp.
name: "Plaintext Also Decodes To Base64 Containing x",
dirName: fmt.Sprintf("4eacd477-e733-4851-b821-6fd42785ff87x%d", nano),
expectedMod: now,
},
{
name: "Legacy Format Without Timestamp (<UUID>)",
dirName: uuid,
expectedMod: time.Time{},
wantZero: true,
},
{
name: "Base64 Encoded Format",
dirName: base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("%sx%d", uuid, nano))),
expectedMod: time.Unix(0, nano),
wantZero: false,
},
{
name: "Encoded Format With Deployment Prefix",
dirName: base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("%s.%sx%d", uuid, uuid, nano))),
expectedMod: now,
},
{
name: "Malformed Suffix (<UUID>xinvalid)",
dirName: fmt.Sprintf("%sxnotanumber", uuid),
expectedMod: time.Time{},
wantZero: true,
},
{
name: "Timestamp Overflow",
dirName: uuid + "x9223372036854775808",
wantZero: true,
},
{
name: "Malformed Plaintext UUID",
dirName: fmt.Sprintf("not-a-uuidx%d", nano),
wantZero: true,
},
{
name: "Empty Directory Name",
dirName: "",
expectedMod: time.Time{},
wantZero: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
got := extractModTimeFromUploadIDDir(tc.dirName)
if tc.wantZero {
if !got.IsZero() {
t.Fatalf("expected zero time for %q, got %v", tc.dirName, got)
}
return
}
if !got.Equal(tc.expectedMod) {
t.Fatalf("extractModTimeFromUploadIDDir(%q) = %v; want %v", tc.dirName, got, tc.expectedMod)
}
})
}
}
type multipartCleanupDisk struct {
StorageAPI
root string
modTime time.Time
readErr error
reads int
}
func (d *multipartCleanupDisk) Endpoint() Endpoint {
return Endpoint{URL: &url.URL{Path: d.root}}
}
func (d *multipartCleanupDisk) ReadVersion(context.Context, string, string, string, string, ReadOptions) (FileInfo, error) {
d.reads++
return FileInfo{ModTime: d.modTime}, d.readErr
}
func (d *multipartCleanupDisk) StatVol(ctx context.Context, volume string) (VolInfo, error) {
return (&xlStorage{drivePath: d.root}).StatVol(ctx, volume)
}
func TestCleanupStaleUploadsOnDisk(t *testing.T) {
const uuid = "d16ad42a-6fe5-4a3b-8092-69983142238f"
expiry := globalAPIConfig.getStaleUploadsExpiry()
for _, tc := range []struct {
name, prefix string
age time.Duration
legacy bool
readErr error
wantReads int
wantDeleted bool
}{
{name: "recent", prefix: uuid, age: expiry / 2, readErr: errFileNotFound},
{name: "plaintext_base64_collision", prefix: "4eacd477-e733-4851-b821-6fd42785ff87", age: expiry / 2, readErr: errFileNotFound},
{name: "expired", prefix: uuid, age: 2 * expiry, readErr: errFileNotFound, wantDeleted: true},
{name: "legacy_recent", prefix: uuid, age: expiry / 2, legacy: true, wantReads: 1},
{name: "legacy_expired", prefix: uuid, age: 2 * expiry, legacy: true, wantReads: 1, wantDeleted: true},
{name: "legacy_metadata_missing", prefix: uuid, age: 2 * expiry, legacy: true, readErr: errFileNotFound, wantReads: 1},
{name: "malformed_metadata_missing", prefix: "not-a-uuid", age: 2 * expiry, readErr: errFileNotFound, wantReads: 1},
} {
t.Run(tc.name, func(t *testing.T) {
modTime := time.Now().Add(-tc.age)
id := tc.prefix
if !tc.legacy {
id = fmt.Sprintf("%sx%d", tc.prefix, modTime.UnixNano())
}
disk := &multipartCleanupDisk{root: t.TempDir(), modTime: modTime, readErr: tc.readErr}
uploadDir := filepath.Join(disk.root, minioMetaMultipartBucket, "sha", id)
for _, dir := range []string{uploadDir, filepath.Join(disk.root, minioMetaTmpDeletedBucket)} {
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
}
// An old parent must not bypass a skipped upload's age or read error.
if err := os.Chtimes(filepath.Dir(uploadDir), modTime, modTime); err != nil {
t.Fatal(err)
}
erasureObjects{}.cleanupStaleUploadsOnDisk(t.Context(), disk)
if disk.reads != tc.wantReads {
t.Errorf("ReadVersion calls = %d, want %d", disk.reads, tc.wantReads)
}
_, err := os.Stat(uploadDir)
if tc.wantDeleted {
if !os.IsNotExist(err) {
t.Errorf("expired upload must be removed: %v", err)
}
if _, err := os.Stat(filepath.Dir(uploadDir)); !os.IsNotExist(err) {
t.Errorf("empty hash directory must be removed in the same pass: %v", err)
}
} else if err != nil {
t.Errorf("upload must be preserved: %v", err)
}
})
}
}
func TestCleanupStaleUploadsEmptyHashDirectory(t *testing.T) {
for _, age := range []time.Duration{0, 2 * globalAPIConfig.getStaleUploadsExpiry()} {
t.Run(age.String(), func(t *testing.T) {
disk := &multipartCleanupDisk{root: t.TempDir()}
shaDir := filepath.Join(disk.root, minioMetaMultipartBucket, "sha")
for _, dir := range []string{shaDir, filepath.Join(disk.root, minioMetaTmpDeletedBucket)} {
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
}
modTime := time.Now().Add(-age)
if err := os.Chtimes(shaDir, modTime, modTime); err != nil {
t.Fatal(err)
}
erasureObjects{}.cleanupStaleUploadsOnDisk(t.Context(), disk)
if _, err := os.Stat(shaDir); !os.IsNotExist(err) {
t.Fatalf("empty hash directory must be removed regardless of age: %v", err)
}
})
}
}
func TestCleanupStaleUploadsSharedHashDirectory(t *testing.T) {
disk := &multipartCleanupDisk{root: t.TempDir(), readErr: errFileNotFound}
shaDir := filepath.Join(disk.root, minioMetaMultipartBucket, "sha")
now := time.Now()
expiredID := fmt.Sprintf("d16ad42a-6fe5-4a3b-8092-69983142238fx%d", now.Add(-2*globalAPIConfig.getStaleUploadsExpiry()).UnixNano())
recentID := fmt.Sprintf("4eacd477-e733-4851-b821-6fd42785ff87x%d", now.UnixNano())
for _, id := range []string{expiredID, recentID} {
if err := os.MkdirAll(filepath.Join(shaDir, id), 0o755); err != nil {
t.Fatal(err)
}
}
partPath := filepath.Join(shaDir, recentID, "part.1")
const part = "recent upload part"
if err := os.WriteFile(partPath, []byte(part), 0o600); err != nil {
t.Fatal(err)
}
erasureObjects{}.cleanupStaleUploadsOnDisk(t.Context(), disk)
if _, err := os.Stat(filepath.Join(shaDir, expiredID)); !os.IsNotExist(err) {
t.Errorf("expired upload must be removed: %v", err)
}
if got, err := os.ReadFile(partPath); err != nil || string(got) != part {
t.Errorf("recent upload must remain intact: got %q, err %v", got, err)
}
}
+2 -2
View File
@@ -253,7 +253,7 @@ func testAPICompleteMultipartFullObjectChecksumMismatch(obj ObjectLayer, instanc
if apiErr.Code != "BadDigest" {
t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code)
}
if want := "The CRC32 checksum you specified did not match the calculated checksum."; apiErr.Message != want {
if want := "The CRC32 you specified did not match the calculated checksum."; apiErr.Message != want {
t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message)
}
@@ -361,7 +361,7 @@ func testAPICompleteMultipartCompositeChecksumMismatch(obj ObjectLayer, instance
if apiErr.Code != "BadDigest" {
t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code)
}
if want := "The CRC32 checksum you specified did not match the calculated checksum."; apiErr.Message != want {
if want := "The CRC32 you specified did not match the calculated checksum."; apiErr.Message != want {
t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message)
}
if _, err := obj.GetObjectInfo(t.Context(), bucketName, objectName, ObjectOptions{}); err == nil {
@@ -0,0 +1,170 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"errors"
"fmt"
"sync/atomic"
"testing"
)
// Run explicitly with -benchtime=1x. Fixture creation is excluded from timing;
// each request still discovers real on-disk metadata. The 16-drive/10k case
// deliberately exceeds the existing aggregate directory-entry budget.
func BenchmarkMultipartDiagnosticScanCapacity(b *testing.B) {
for _, tc := range []struct {
drives, uploads int
mixed, limited bool
}{
{4, 1_000, false, false},
{4, 10_000, false, false},
{4, 10_000, true, false},
{16, 1_000, false, false},
{16, 10_000, false, true},
} {
b.Run(fmt.Sprintf("drives=%d/uploads=%d/mixed=%t", tc.drives, tc.uploads, tc.mixed), func(b *testing.B) {
obj, dirs, err := prepareErasure(b.Context(), tc.drives)
if err != nil {
b.Fatal(err)
}
z := obj.(*erasureServerPools)
b.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket, other = "multipart-capacity", "multipart-unrelated"
for _, name := range []string{bucket, other} {
if err := z.MakeBucket(b.Context(), name, MakeBucketOptions{}); err != nil {
b.Fatal(err)
}
}
for n := range tc.uploads {
target := bucket
if tc.mixed && n%2 != 0 {
target = other
}
key := fmt.Sprintf("dir/%05d", n)
mp, err := z.NewMultipartUpload(b.Context(), target, key, ObjectOptions{})
if err != nil {
b.Fatal(err)
}
if tc.mixed && n%10 == 0 {
rewriteMultipartIdentity(b, z.serverPools[0].getHashedSet(key), target, key, mp.UploadID, removeMultipartIdentity)
}
}
z.mpCache.Clear()
var directoryCalls, metadataCalls atomic.Int64
for _, set := range z.serverPools[0].sets {
original := set.getDisks
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, disk := range disks {
disks[i] = multipartListingCountingDisk{disk, &directoryCalls, &metadataCalls}
}
return disks
}
b.Cleanup(func() { set.getDisks = original })
}
b.ResetTimer()
for range b.N {
got, err := z.scanMultipartUploadsForTest(b.Context(), bucket, "dir/", "", "", "", 1000)
if tc.limited {
var limited SlowDown
if !errors.As(err, &limited) || len(got.Uploads) != 0 {
b.Fatalf("overflow must return SlowDown without a partial page: %+v %v", got, err)
}
} else if err != nil || len(got.Uploads) != 1000 {
b.Fatalf("capacity listing: count=%d err=%v", len(got.Uploads), err)
}
}
b.StopTimer()
b.ReportMetric(float64(directoryCalls.Load())/float64(b.N), "ListDir/op")
b.ReportMetric(float64(metadataCalls.Load())/float64(b.N), "ReadVersion/op")
})
}
}
// The public path is measured with a populated cache as well as after cache
// loss. An empty post-restart response alone is not capacity acceptance.
func BenchmarkMultipartPublicListingCapacity(b *testing.B) {
for _, tc := range []struct {
drives, uploads int
sameKey bool
}{
{4, 10_000, false},
{16, 3_126, false},
{16, 10_000, true},
} {
b.Run(fmt.Sprintf("drives=%d/uploads=%d/same-key=%t", tc.drives, tc.uploads, tc.sameKey), func(b *testing.B) {
obj, dirs, err := prepareErasure(b.Context(), tc.drives)
if err != nil {
b.Fatal(err)
}
z := obj.(*erasureServerPools)
b.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket, other = "multipart-public-capacity", "multipart-public-other"
for _, name := range []string{bucket, other} {
if err := z.MakeBucket(b.Context(), name, MakeBucketOptions{}); err != nil {
b.Fatal(err)
}
}
wantCached, wantExact := 0, 1
for n := range tc.uploads {
target, key := bucket, fmt.Sprintf("dir/%05d", n)
if tc.sameKey {
key = "dir/00000"
} else if n%2 != 0 {
target = other
}
if target == bucket {
wantCached++
}
if _, err := z.NewMultipartUpload(b.Context(), target, key, ObjectOptions{}); err != nil {
b.Fatal(err)
}
}
if tc.sameKey {
wantExact = 1000
}
if tc.drives == 16 && !tc.sameKey {
_, err := z.scanMultipartUploadsForTest(b.Context(), bucket, "", "", "", "", 1000)
var limited SlowDown
if !errors.As(err, &limited) {
b.Fatalf("fixture did not reproduce the previous global budget failure: %v", err)
}
}
var directoryCalls, metadataCalls atomic.Int64
for _, set := range z.serverPools[0].sets {
original := set.getDisks
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, disk := range disks {
disks[i] = multipartListingCountingDisk{disk, &directoryCalls, &metadataCalls}
}
return disks
}
b.Cleanup(func() { set.getDisks = original })
}
measure := func(name, prefix string, want int) {
b.Run(name, func(b *testing.B) {
directoryCalls.Store(0)
metadataCalls.Store(0)
b.ResetTimer()
for range b.N {
got, err := z.ListMultipartUploads(b.Context(), bucket, prefix, "", "", "", 1000)
if err != nil || len(got.Uploads) != want {
b.Fatalf("public listing: count=%d want=%d err=%v", len(got.Uploads), want, err)
}
}
b.StopTimer()
b.ReportMetric(float64(directoryCalls.Load())/float64(b.N), "ListDir/op")
b.ReportMetric(float64(metadataCalls.Load())/float64(b.N), "ReadVersion/op")
})
}
measure("populated-cache", "", wantCached)
z.mpCache.Clear()
measure("lost-cache", "", 0)
measure("known-key", "dir/00000", wantExact)
})
}
}
+404
View File
@@ -0,0 +1,404 @@
// Copyright (c) 2026 mr javad seydi and Ruohang Feng
//
// This file is part of Silo Object Storage stack.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/pgsty/silo-pkg/v3/policy"
)
const multipartScanEntryLimit = 100_000
var (
multipartScanSlots = make(chan struct{}, 2)
errMultipartListingIdentity = errors.New("multipart upload identity is invalid")
)
// One budget and admission slot cover the entire request, including all pools.
// A slot is released only after the scan workers have actually stopped.
type multipartScan struct {
ctx context.Context
cancel context.CancelFunc
remaining atomic.Int64
metadataSlots chan struct{}
preflight bool
sets []multipartScanSet
}
type multipartScanSet struct {
Pool int `json:"pool"`
Set int `json:"set"`
Drives int `json:"drives"`
ScannedDrives int `json:"scannedDrives"`
UncoveredDrives []int `json:"uncoveredDrives,omitempty"`
Candidates int `json:"candidates"`
LegacyUploads int `json:"legacyUploads"`
OldestLegacy time.Time `json:"oldestLegacy,omitempty"`
Error string `json:"error,omitempty"`
}
type multipartPreflightReport struct {
Ready bool `json:"ready"`
Complete bool `json:"complete"`
Mode string `json:"mode"`
ScannedEntries int64 `json:"scannedEntries"`
LegacyUploads int `json:"legacyUploads"`
Sets []multipartScanSet `json:"sets"`
}
func startMultipartScan(ctx context.Context, preflight bool) (*multipartScan, error) {
select {
case multipartScanSlots <- struct{}{}:
case <-ctx.Done():
return nil, ctx.Err()
default:
return nil, SlowDown{}
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
s := &multipartScan{ctx: ctx, cancel: cancel, preflight: preflight, metadataSlots: make(chan struct{}, multipartMetadataScanConcurrency)}
s.remaining.Store(multipartScanEntryLimit)
return s, nil
}
func (s *multipartScan) close() {
s.cancel()
<-multipartScanSlots
}
func (s *multipartScan) listDir(disk StorageAPI, bucket, dir string) ([]string, error) {
if err := s.ctx.Err(); err != nil {
return nil, SlowDown{}
}
remaining := s.remaining.Load()
if remaining <= 0 {
return nil, SlowDown{}
}
// Request one extra entry to detect overflow, never a silently partial page.
entries, err := disk.ListDir(s.ctx, bucket, minioMetaMultipartBucket, dir, int(remaining)+1)
if errors.Is(err, errFileNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
if s.remaining.Add(-int64(len(entries))) < 0 {
return nil, SlowDown{}
}
return entries, nil
}
func (s *multipartScan) listUploadDirs(disk StorageAPI, bucket string) ([]string, error) {
hashDirs, err := s.listDir(disk, bucket, "")
if err != nil {
return nil, err
}
var candidates []string
for _, hashDir := range hashDirs {
if !strings.HasSuffix(hashDir, SlashSeparator) {
continue
}
hashDir = strings.TrimSuffix(hashDir, SlashSeparator)
uploadDirs, err := s.listDir(disk, bucket, hashDir)
if err != nil {
return nil, err
}
for _, uploadDir := range uploadDirs {
if strings.HasSuffix(uploadDir, SlashSeparator) {
candidates = append(candidates, pathJoin(hashDir, strings.TrimSuffix(uploadDir, SlashSeparator)))
}
}
}
return candidates, nil
}
// Identity is immutable at hash/uploadUUID. Check the hash before using even
// a single source drive to exclude another bucket. Missing/bad identities must
// fall back to the full metadata read; they cannot prove absence.
func (er erasureObjects) multipartIdentity(fi FileInfo, shaDir string) (string, string, bool) {
bucket, object := fi.Metadata[multipartMetaBucket], fi.Metadata[multipartMetaObject]
return bucket, object, bucket != "" && object != "" && IsValidBucketName(bucket) &&
IsValidObjectPrefix(object) && er.getMultipartSHADir(bucket, object) == shaDir
}
// Only the absence of both fields identifies the historical format. Empty,
// partial or invalid identities must not hide a damaged new upload as legacy.
func multipartLegacyIdentity(fi FileInfo) bool {
_, bucketPresent := fi.Metadata[multipartMetaBucket]
_, objectPresent := fi.Metadata[multipartMetaObject]
return !bucketPresent && !objectPresent
}
func (er erasureObjects) readMultipartUploadCandidate(s *multipartScan, bucket, candidate string, source StorageAPI) (MultipartInfo, bool, bool, error) {
if s.ctx.Err() != nil {
return MultipartInfo{}, false, false, SlowDown{}
}
shaDir, uploadUUID, ok := strings.Cut(candidate, SlashSeparator)
if !ok || shaDir == "" || uploadUUID == "" || strings.Contains(uploadUUID, SlashSeparator) {
return MultipartInfo{}, false, false, errMultipartListingIdentity
}
if !s.preflight && bucket != "" && source != nil {
fi, err := source.ReadVersion(s.ctx, bucket, minioMetaMultipartBucket, candidate, "", ReadOptions{})
if err == nil {
storedBucket, _, valid := er.multipartIdentity(fi, shaDir)
if valid && storedBucket != bucket {
return MultipartInfo{}, false, false, nil
}
}
}
if s.ctx.Err() != nil {
return MultipartInfo{}, false, false, SlowDown{}
}
select {
case s.metadataSlots <- struct{}{}:
defer func() { <-s.metadataSlots }()
case <-s.ctx.Done():
return MultipartInfo{}, false, false, SlowDown{}
}
disks := er.getDisks()
metadata, errs := readAllFileInfo(s.ctx, disks, bucket, minioMetaMultipartBucket, candidate, "", false, false)
if s.preflight {
// The diagnostic counts even a single readable legacy copy. It does not
// gate listing; unreadable copies still prevent a complete diagnosis.
var oldest MultipartInfo
legacy := false
_, nativeID := multipartUploadTime(uploadUUID)
for i, err := range errs {
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
continue
}
if err != nil {
return MultipartInfo{}, false, false, err
}
fi := metadata[i]
storedBucket, storedObject, valid := er.multipartIdentity(fi, shaDir)
isLegacy := multipartLegacyIdentity(fi)
if !isLegacy && (!valid || !nativeID) {
return MultipartInfo{}, false, false, errMultipartListingIdentity
}
if isLegacy {
info := multipartUploadInfo(storedBucket, storedObject, uploadUUID, fi.ModTime)
if !legacy || info.Initiated.Before(oldest.Initiated) {
oldest = info
}
legacy = true
}
}
if legacy {
return oldest, false, true, nil
}
}
readQuorum, _, err := objectQuorumFromMeta(s.ctx, metadata, errs, er.defaultParityCount)
if err != nil {
return MultipartInfo{}, false, false, err
}
_, modTime, etag := listOnlineDisks(disks, metadata, errs, readQuorum)
if err := reduceReadQuorumErrs(s.ctx, errs, objectOpIgnoredErrs, readQuorum); err != nil {
return MultipartInfo{}, false, false, err
}
fi, err := pickValidFileInfo(s.ctx, metadata, modTime, etag, readQuorum)
if err != nil {
return MultipartInfo{}, false, false, err
}
// The generic metadata quorum does not compare these identity fields.
// Contradictory readable copies cannot certify a historical upload: picking
// a copy with missing fields must not silently hide a new-format record.
legacy := multipartLegacyIdentity(fi)
storedBucket, storedObject, valid := er.multipartIdentity(fi, shaDir)
for i, other := range metadata {
if errs[i] != nil || !other.IsValid() {
continue
}
if multipartLegacyIdentity(other) != legacy ||
(!legacy && (other.Metadata[multipartMetaBucket] != storedBucket || other.Metadata[multipartMetaObject] != storedObject)) {
return MultipartInfo{}, false, false, fmt.Errorf("%w: inconsistent copies at %s", errMultipartListingIdentity, candidate)
}
}
info := multipartUploadInfo(storedBucket, storedObject, uploadUUID, fi.ModTime)
if legacy {
return info, false, true, nil
}
if !valid {
return MultipartInfo{}, false, false, fmt.Errorf("%w: %s", errMultipartListingIdentity, candidate)
}
if _, ok := multipartUploadTime(uploadUUID); !ok {
return MultipartInfo{}, false, false, fmt.Errorf("%w: %s", errMultipartListingIdentity, candidate)
}
if bucket != "" && bucket != storedBucket {
return MultipartInfo{}, false, false, nil
}
return info, true, false, nil
}
func (er erasureObjects) scanMultipartUploads(s *multipartScan, bucket string, poolIdx, setIdx int) ([]MultipartInfo, error) {
disks := er.getDisks()
report := multipartScanSet{Pool: poolIdx, Set: setIdx, Drives: er.setDriveCount}
var resultErr error
defer func() {
if resultErr != nil {
report.Error = resultErr.Error()
}
s.sets = append(s.sets, report)
}()
var candidateMu sync.Mutex
candidates := make(map[string]StorageAPI)
errs := make([]error, len(disks))
var wg sync.WaitGroup
for i, disk := range disks {
wg.Add(1)
go func() {
defer wg.Done()
if disk == nil || !disk.IsOnline() {
errs[i] = errDiskNotFound
return
}
paths, err := s.listUploadDirs(disk, bucket)
errs[i] = err
if err != nil {
return
}
candidateMu.Lock()
defer candidateMu.Unlock()
for _, p := range paths {
candidates[p] = disk
}
}()
}
wg.Wait()
for i, err := range errs {
if err == nil {
report.ScannedDrives++
} else {
report.UncoveredDrives = append(report.UncoveredDrives, i)
if _, limited := err.(SlowDown); limited {
resultErr = err
}
}
}
report.Candidates = len(candidates)
// A successful scan must intersect every metadata read quorum, including
// records left with R copies by a partially failed cancellation.
if report.ScannedDrives < er.setDriveCount/2+1 && resultErr == nil {
resultErr = toObjectErr(errErasureReadQuorum, bucket)
}
if resultErr != nil {
return nil, resultErr
}
type candidate struct {
path string
source StorageAPI
}
jobs := make(chan candidate)
var mu sync.Mutex
var uploads []MultipartInfo
for range min(16, len(candidates)) {
wg.Add(1)
go func() {
defer wg.Done()
for c := range jobs {
mu.Lock()
stopped := resultErr != nil
mu.Unlock()
if stopped || s.ctx.Err() != nil {
continue
}
upload, found, legacy, err := er.readMultipartUploadCandidate(s, bucket, c.path, c.source)
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
continue
}
mu.Lock()
switch {
case err != nil && resultErr == nil:
resultErr = err
case legacy:
report.LegacyUploads++
if report.OldestLegacy.IsZero() || upload.Initiated.Before(report.OldestLegacy) {
report.OldestLegacy = upload.Initiated
}
case found && !s.preflight:
uploads = append(uploads, upload)
}
mu.Unlock()
}
}()
}
dispatch:
for p, source := range candidates {
select {
case jobs <- candidate{p, source}:
case <-s.ctx.Done():
break dispatch
}
}
close(jobs)
wg.Wait()
if s.ctx.Err() != nil {
resultErr = SlowDown{}
}
return uploads, resultErr
}
// multipartPreflight scans all pools, sets and drives, independent of caches.
// A majority suffices for normal listing; a complete diagnosis requires every
// drive to have been inspected. Ready describes the absence of legacy copies,
// not whether listing is enabled.
func (z *erasureServerPools) multipartPreflight(ctx context.Context) (multipartPreflightReport, error) {
s, err := startMultipartScan(ctx, true)
if err != nil {
return multipartPreflightReport{}, err
}
defer s.close()
report := multipartPreflightReport{Complete: true, Mode: "legacy"}
for p, pool := range z.serverPools {
for i, set := range pool.sets {
_, _ = set.scanMultipartUploads(s, "", p, i)
}
}
report.Sets = s.sets
for _, set := range s.sets {
report.LegacyUploads += set.LegacyUploads
if set.Error != "" || set.ScannedDrives != set.Drives {
report.Complete = false
}
}
report.ScannedEntries = multipartScanEntryLimit - s.remaining.Load()
report.Ready = report.Complete && report.LegacyUploads == 0
return report, nil
}
// MultipartPreflightHandler is a read-only storage-admin diagnostic. It never
// accepts an arbitrary deletion path or changes upload lifetime settings.
func (a adminAPIHandlers) MultipartPreflightHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
obj, _ := validateAdminReq(ctx, w, r, policy.StorageInfoAdminAction)
if obj == nil {
return
}
z, ok := obj.(*erasureServerPools)
if !ok {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrNotImplemented), r.URL)
return
}
report, err := z.multipartPreflight(ctx)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
b, err := json.Marshal(report)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
writeSuccessResponseJSON(w, b)
}
+824
View File
@@ -0,0 +1,824 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/minio/minio/internal/config/storageclass"
)
// The persistent scanner is retained for diagnostics. This test-only wrapper
// preserves its pagination coverage without exposing it through public listing.
func (z *erasureServerPools) scanMultipartUploadsForTest(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
return ListMultipartsInfo{}, toObjectErr(err, bucket)
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
var uploads []MultipartInfo
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
poolUploads, err := pool.scanMultipartUploadsForTest(scan, bucket, idx)
if err != nil {
return ListMultipartsInfo{}, err
}
uploads = append(uploads, poolUploads...)
}
// Confirmed historical records lack a recoverable bucket/key and are
// omitted by the scan. They do not block listing identifiable uploads.
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
func (s *erasureSets) scanMultipartUploadsForTest(scan *multipartScan, bucket string, poolIdx int) ([]MultipartInfo, error) {
var uploads []MultipartInfo
for i, set := range s.sets {
setUploads, err := set.scanMultipartUploads(scan, bucket, poolIdx, i)
if err != nil {
return nil, err
}
uploads = append(uploads, setUploads...)
}
return uploads, nil
}
// Check that a successful HTTP abort removes the cached upload. The restored
// cache path still ignores page limits; this is a documented legacy limitation.
func TestMultipartListingAbortHTTPUpdatesCache(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads", "AbortMultipart"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
var firstID, secondID string
for attempt := 0; attempt < 32; attempt++ {
one, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
two, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if one.UploadID < two.UploadID {
firstID, secondID = one.UploadID, two.UploadID
break
}
for _, id := range []string{one.UploadID, two.UploadID} {
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", id, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
}
if firstID == "" {
t.Fatal("could not construct increasing upload IDs")
}
if _, err := z.NewMultipartUpload(t.Context(), bucket, "b", ObjectOptions{}); err != nil {
t.Fatal(err)
}
request := func(method, u string) *httptest.ResponseRecorder {
t.Helper()
req, err := newTestSignedRequestV4(method, u, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
return rec
}
list := func(keyMarker, uploadMarker string, limit int) ListMultipartUploadsResponse {
t.Helper()
rec := request(http.MethodGet, getListMultipartUploadsURLWithParams("", bucket, "", keyMarker, uploadMarker, "", strconv.Itoa(limit)))
if rec.Code != http.StatusOK {
t.Fatalf("list HTTP %d: %s", rec.Code, rec.Body.String())
}
var result ListMultipartUploadsResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &result); err != nil {
t.Fatal(err)
}
return result
}
first := list("", "", 1)
if len(first.Uploads) != 3 || first.Uploads[0].UploadID != firstID || first.IsTruncated {
t.Fatalf("unexpected legacy cache view: %+v", first)
}
rec := request(http.MethodDelete, getAbortMultipartUploadURL("", bucket, "a", firstID))
if rec.Code != http.StatusNoContent {
t.Fatalf("abort HTTP %d: %s", rec.Code, rec.Body.String())
}
rest := list(first.NextKeyMarker, first.NextUploadIDMarker, 10)
var keys []string
for _, upload := range rest.Uploads {
keys = append(keys, upload.Key)
}
t.Logf("GET cache=200; DELETE upload=204; GET cache=200 keys=%v truncated=%v", keys, rest.IsTruncated)
if _, err := z.GetMultipartInfo(t.Context(), bucket, "a", secondID, ObjectOptions{}); err != nil {
t.Fatalf("remaining upload is not valid: %v", err)
}
if len(rest.Uploads) != 2 || rest.Uploads[0].UploadID != secondID {
t.Fatalf("valid remaining upload for key a is missing from cache: %+v", rest.Uploads)
}
}
type multipartListingFaultDisk struct {
StorageAPI
read func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error)
delete func(context.Context, string, string, DeleteOptions) error
}
func (d multipartListingFaultDisk) ReadVersion(ctx context.Context, original, volume, object, version string, opts ReadOptions) (FileInfo, error) {
if d.read != nil {
return d.read(ctx, original, volume, object, version, opts)
}
return d.StorageAPI.ReadVersion(ctx, original, volume, object, version, opts)
}
func (d multipartListingFaultDisk) Delete(ctx context.Context, volume, object string, opts DeleteOptions) error {
if d.delete != nil {
return d.delete(ctx, volume, object, opts)
}
return d.StorageAPI.Delete(ctx, volume, object, opts)
}
func TestMultipartListingLegacyPreflight(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
const other = "multipart-legacy-other"
if err := z.MakeBucket(t.Context(), other, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
old, err := z.NewMultipartUpload(t.Context(), other, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
fi, metadata, err := set.checkUploadIDExists(t.Context(), other, "old", old.UploadID, true)
if err != nil {
t.Fatal(err)
}
for i := range metadata {
delete(metadata[i].Metadata, multipartMetaBucket)
delete(metadata[i].Metadata, multipartMetaObject)
}
if _, err = writeAllMetadata(t.Context(), set.getDisks(), other, minioMetaMultipartBucket,
set.getUploadIDDir(other, "old", old.UploadID), metadata, fi.WriteQuorum(set.defaultWQuorum())); err != nil {
t.Fatal(err)
}
if _, err = z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
z.mpCache.Clear()
for _, target := range []string{bucket, other} {
got, err := z.scanMultipartUploadsForTest(t.Context(), target, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
if target == bucket {
requireMultipartUploadKeys(t, got, "a")
} else {
requireMultipartUploadKeys(t, got)
}
}
// Diagnostic discovery cannot recover an unidentified historical key.
exact, err := z.scanMultipartUploadsForTest(t.Context(), other, "old", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, exact)
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || !report.Complete || report.LegacyUploads != 1 || report.Mode != "legacy" {
t.Fatalf("legacy preflight: %+v %v", report, err)
}
if err = z.AbortMultipartUpload(t.Context(), other, "old", old.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
report, err = z.multipartPreflight(t.Context())
if err != nil || !report.Ready || report.LegacyUploads != 0 {
t.Fatalf("drained preflight: %+v %v", report, err)
}
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
}
func TestMultipartDiagnosticIdentityFallback(t *testing.T) {
for _, kind := range []string{"old", "partial", "corrupt", "read-failure", "wrong-bucket"} {
t.Run(kind, func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
if _, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
var first atomic.Bool
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(ctx context.Context, b, v, p, version string, opts ReadOptions) (FileInfo, error) {
fi, err := d.ReadVersion(ctx, b, v, p, version, opts)
if v == minioMetaMultipartBucket && first.CompareAndSwap(false, true) {
if kind == "read-failure" {
return FileInfo{}, errDiskNotFound
}
if kind == "corrupt" {
return FileInfo{}, errFileCorrupt
}
fi.Metadata = cloneMSS(fi.Metadata)
if kind == "old" || kind == "partial" {
delete(fi.Metadata, multipartMetaBucket)
if kind == "old" {
delete(fi.Metadata, multipartMetaObject)
}
} else {
fi.Metadata[multipartMetaBucket] = "another-bucket"
}
}
return fi, err
}}
}
return disks
}
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
})
}
}
func TestMultipartAbortPoolsAndRetry(t *testing.T) {
z, bucket := consistencyPools(t)
mp, err := z.serverPools[1].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
emptySet := z.serverPools[0].getHashedSet("a")
original := emptySet.getDisks
t.Cleanup(func() { emptySet.getDisks = original })
emptySet.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error) {
return FileInfo{}, errDiskNotFound
}}
}
return disks
}
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("unknown pool must not acknowledge cancellation: %v", err)
}
if _, err := z.serverPools[1].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("earlier pool error changed the later pool: %v", err)
}
emptySet.getDisks = original
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("retry after pool recovery: %v", err)
}
mp, err = z.serverPools[1].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if _, err = z.serverPools[1].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err == nil {
t.Fatal("empty first pool hid the actual upload")
}
}
func TestMultipartAbortRetryBelowReadQuorum(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
disks[3] = nil
d := disks[2]
disks[2] = multipartListingFaultDisk{StorageAPI: d, delete: func(ctx context.Context, v, p string, opts DeleteOptions) error {
if err := d.Delete(ctx, v, p, opts); err != nil {
return err
}
return context.DeadlineExceeded // operation finished, acknowledgement lost
}}
return disks
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("best-effort cleanup must retain read-quorum availability: %v", err)
}
set.getDisks = original
var invalid InvalidUploadID
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("minority remnant must retain absent-upload result: %v", err)
}
}
func TestMultipartListingMarkerHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
key, marker string
status int
}{
{"", "not-base64=", 200},
{"a", "not-base64=", 404},
{"a", base64.RawURLEncoding.EncodeToString([]byte("not-native")), 400},
{"a", multipartListingTestID(time.Unix(100, 0), 1), 200},
} {
u := getListMultipartUploadsURLWithParams("", bucket, "", tc.key, tc.marker, "", "10")
req, err := newTestSignedRequestV4(http.MethodGet, u, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != tc.status {
t.Fatalf("key=%q marker=%q: %d %s", tc.key, tc.marker, rec.Code, rec.Body.String())
}
}
}
func TestMultipartPreflightAdminHTTP(t *testing.T) {
bed, err := prepareAdminErasureTestBed(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { bed.done(); bed.objLayer.Shutdown(context.Background()); removeRoots(bed.erasureDirs) })
const target = "/minio/admin/v3/multipart-preflight"
rec := httptest.NewRecorder()
bed.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, target, nil))
if rec.Code != 403 {
t.Fatalf("anonymous preflight: %d", rec.Code)
}
req, err := newTestSignedRequestV4(http.MethodGet, target, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec = httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != 200 {
t.Fatalf("admin preflight: %d %s", rec.Code, rec.Body.String())
}
var report multipartPreflightReport
if err = json.Unmarshal(rec.Body.Bytes(), &report); err != nil || !report.Ready || !report.Complete {
t.Fatalf("preflight: %+v %v", report, err)
}
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
rec = httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("busy admin preflight: %d %s", rec.Code, rec.Body.String())
}
}
type multipartLateCreateDisk struct {
StorageAPI
late chan<- func() error
}
func (d multipartLateCreateDisk) WriteMetadata(_ context.Context, original, volume, object string, fi FileInfo) error {
if volume == minioMetaMultipartBucket {
d.late <- func() error { return d.StorageAPI.WriteMetadata(context.Background(), original, volume, object, fi) }
return context.DeadlineExceeded
}
return d.StorageAPI.WriteMetadata(context.Background(), original, volume, object, fi)
}
// This characterization is deliberately NOT an assertion of terminal abort
// correctness. It preserves an executable example of the separately scoped
// late-creation-write limitation. Change the expectation when fencing is added.
func TestMultipartAbortLateCreateBoundary(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), 16)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
saved := globalStorageClass
globalStorageClass.Update(storageclass.Config{Standard: storageclass.StorageClass{Parity: 8}})
t.Cleanup(func() { globalStorageClass.Update(saved) })
const bucket = "multipart-late-create"
if err = z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
set := z.serverPools[0].getHashedSet("a")
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
late := make(chan func() error, 16)
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 9; i < 16; i++ {
disks[i] = multipartLateCreateDisk{disks[i], late}
}
return disks
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if len(late) != 7 {
t.Fatalf("expected seven timed-out creation writes, got %d", len(late))
}
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
disks[8] = nil
return disks
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
for range 7 {
if err = (<-late)(); err != nil {
t.Fatal(err)
}
}
set.getDisks = original
_, metadata, err := set.checkUploadIDExists(t.Context(), bucket, "a", mp.UploadID, false)
if err != nil {
t.Fatalf("late creation boundary changed; review and remove the documented limitation: %v", err)
}
count := 0
for _, fi := range metadata {
if fi.IsValid() {
count++
}
}
if count != 8 {
t.Fatalf("expected eight resurrected copies, got %d", count)
}
t.Log("KNOWN UNRESOLVED BOUNDARY: seven delayed creation writes plus one offline copy restore a readable upload after acknowledged cancellation")
}
type multipartListingCountingDisk struct {
StorageAPI
directoryCalls *atomic.Int64
metadataCalls *atomic.Int64
}
func (d multipartListingCountingDisk) ListDir(ctx context.Context, original, volume, dir string, count int) ([]string, error) {
if volume == minioMetaMultipartBucket {
d.directoryCalls.Add(1)
}
return d.StorageAPI.ListDir(ctx, original, volume, dir, count)
}
func (d multipartListingCountingDisk) ReadVersion(ctx context.Context, original, volume, object, version string, opts ReadOptions) (FileInfo, error) {
if volume == minioMetaMultipartBucket {
d.metadataCalls.Add(1)
}
return d.StorageAPI.ReadVersion(ctx, original, volume, object, version, opts)
}
// Counts storage API calls; this is not a deployment throughput benchmark.
func TestMultipartDiagnosticScanCosts(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), 4)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket, otherBucket = "r9-scan-target", "r9-scan-unrelated"
for _, name := range []string{bucket, otherBucket} {
if err := z.MakeBucket(t.Context(), name, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
}
if _, err := z.NewMultipartUpload(t.Context(), bucket, "dir/target", ObjectOptions{}); err != nil {
t.Fatal(err)
}
var directoryCalls, metadataCalls atomic.Int64
for _, pool := range z.serverPools {
for _, set := range pool.sets {
original := set.getDisks
set.getDisks = func() []StorageAPI {
disks := original()
wrapped := make([]StorageAPI, len(disks))
for i, disk := range disks {
if disk != nil {
wrapped[i] = multipartListingCountingDisk{disk, &directoryCalls, &metadataCalls}
}
}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
}
}
measure := func(label string) (int64, int64) {
t.Helper()
directoryCalls.Store(0)
metadataCalls.Store(0)
result, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "dir/", "", "", "", 1)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, result, "dir/target")
d, m := directoryCalls.Load(), metadataCalls.Load()
t.Logf("%s: max-uploads=1 returned=%d ListDir=%d ReadVersion=%d", label, len(result.Uploads), d, m)
return d, m
}
_, baseline := measure("no unrelated uploads")
for n := 0; n < 32; n++ {
if _, err := z.NewMultipartUpload(t.Context(), otherBucket, fmt.Sprintf("other/%03d", n), ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
_, loaded := measure("32 uploads in another bucket")
_, repeated := measure("same query repeated")
if loaded != baseline+32 || repeated != loaded {
t.Fatalf("unexpected scan accounting: baseline=%d loaded=%d repeated=%d", baseline, loaded, repeated)
}
}
func multipartListingTestID(created time.Time, n int) string {
return base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("00000000-0000-4000-8000-000000000000.00000000-0000-4000-8000-%012xx%d", n, created.UnixNano())))
}
func multipartListingFixture(t *testing.T) (*erasureServerPools, *erasureObjects, string) {
t.Helper()
obj, dirs, err := prepareErasure(t.Context(), 4)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket = "multipart-listing-test"
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
return z, z.serverPools[0].getHashedSet("a"), bucket
}
func TestMultipartListingMissingMarkers(t *testing.T) {
base := time.Unix(100, 0)
sameTime := []string{multipartListingTestID(base.Add(time.Second), 1), multipartListingTestID(base.Add(time.Second), 2), multipartListingTestID(base.Add(time.Second), 3)}
slices.Sort(sameTime)
uploads := []MultipartInfo{
{Bucket: "bucket", Object: "a", UploadID: multipartListingTestID(base, 1), Initiated: base},
{Bucket: "bucket", Object: "a", UploadID: sameTime[1], Initiated: base.Add(time.Second)},
{Bucket: "bucket", Object: "b", UploadID: multipartListingTestID(base, 4), Initiated: base},
}
for _, tc := range []struct {
name string
created time.Time
id int
want []string
}{
{"before", base.Add(-time.Second), 0, []string{"a", "a", "b"}},
{"between", base.Add(time.Second / 2), 2, []string{"a", "b"}},
{"same-time-before", base.Add(time.Second), 2, []string{"a", "b"}},
{"same-time-after", base.Add(time.Second), 4, []string{"b"}},
{"after", base.Add(2 * time.Second), 5, []string{"b"}},
} {
t.Run(tc.name, func(t *testing.T) {
marker := multipartListingTestID(tc.created, tc.id)
if tc.name == "same-time-before" {
marker = sameTime[0]
}
if tc.name == "same-time-after" {
marker = sameTime[2]
}
if err := checkListMultipartArgs(t.Context(), "bucket", "", "a", marker, ""); err != nil {
t.Fatal(err)
}
got := paginateMultipartUploads(uploads, "", "a", marker, "", 10)
if !slices.Equal(multipartUploadKeys(got.Uploads), tc.want) {
t.Fatalf("got %v want %v", multipartUploadKeys(got.Uploads), tc.want)
}
})
}
}
func TestMultipartDiagnosticCoverage(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
disks := original()
// Leave a readable two-copy record, simulating a partially failed abort.
for _, d := range disks[:2] {
if err = d.Delete(t.Context(), minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
set.getDisks = func() []StorageAPI { return []StorageAPI{disks[0], disks[1], nil, nil} }
_, err = z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("incomplete discovery returned success: %v", err)
}
set.getDisks = func() []StorageAPI { return []StorageAPI{nil, disks[1], disks[2], disks[3]} }
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || report.Complete || len(report.Sets[0].UncoveredDrives) != 1 {
t.Fatalf("offline upgrade preflight: %+v %v", report, err)
}
}
func TestMultipartDiagnosticBudgetAndAdmission(t *testing.T) {
_, set, bucket := multipartListingFixture(t)
if _, err := set.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
scan.remaining.Store(1)
_, err = set.scanMultipartUploads(scan, bucket, 0, 0)
var limited SlowDown
if !errors.As(err, &limited) {
t.Fatalf("budget: %v", err)
}
if apiErr := toAPIError(t.Context(), err); apiErr.HTTPStatusCode != http.StatusServiceUnavailable || apiErr.Code != "SlowDown" {
t.Fatalf("budget error mapping: %+v", apiErr)
}
second, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer second.close()
if third, err := startMultipartScan(t.Context(), false); err == nil {
third.close()
t.Fatal("third scan admitted")
}
}
func TestMultipartPublicListingAdmissionHTTP(t *testing.T) {
z, _, _ := multipartListingFixture(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
req, err := newTestSignedRequestV4(http.MethodGet,
getListMultipartUploadsURLWithParams("", bucket, "", "", "", "", "1"),
0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
var response ListMultipartUploadsResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &response); err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusOK || len(response.Uploads) != 0 {
t.Fatalf("diagnostic admission blocked public listing: %d %s", rec.Code, rec.Body.String())
}
}
func TestMultipartListingPreflightMinorityLegacy(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
path := set.getUploadIDDir(bucket, "old", mp.UploadID)
disks := set.getDisks()
fi, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, path, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
if err = disks[0].WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, path, fi); err != nil {
t.Fatal(err)
}
for _, d := range disks[1:] {
if err = d.Delete(t.Context(), minioMetaMultipartBucket, path, DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || !report.Complete || report.LegacyUploads != 1 || report.Mode != "legacy" {
t.Fatalf("minority legacy copy must prevent readiness: %+v %v", report, err)
}
}
func TestMultipartDiagnosticCancellationRetainsAdmission(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
for i := range 40 {
if _, err := z.NewMultipartUpload(t.Context(), bucket, fmt.Sprintf("key-%02d", i), ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
entered := make(chan struct{}, 40)
release := make(chan struct{})
var once sync.Once
defer once.Do(func() { close(release) })
var reads atomic.Int32
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(ctx context.Context, b, v, p, version string, opts ReadOptions) (FileInfo, error) {
if v != minioMetaMultipartBucket {
return d.ReadVersion(ctx, b, v, p, version, opts)
}
reads.Add(1)
entered <- struct{}{}
// Model an RPC which does not return immediately on cancellation.
<-release
return FileInfo{}, ctx.Err()
}}
}
return disks
}
second, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer second.close()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() {
_, err := z.scanMultipartUploadsForTest(ctx, bucket, "", "", "", "", 10)
done <- err
}()
for range 16 {
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("identity workers did not start")
}
}
cancel()
if extra, err := startMultipartScan(t.Context(), false); err == nil {
extra.close()
t.Fatal("canceled request released admission while RPCs were still running")
}
start := time.Now()
once.Do(func() { close(release) })
select {
case err := <-done:
if err == nil {
t.Fatal("canceled scan returned a successful partial list")
}
case <-time.After(2 * time.Second):
t.Fatal("scan did not stop after blocked RPCs returned")
}
if got := reads.Load(); got != 16 {
t.Fatalf("scheduled more identity/metadata reads after cancellation: %d", got)
}
t.Logf("16 identity RPCs bounded; admission held until return; cancellation settled in %s", time.Since(start))
}
+236
View File
@@ -0,0 +1,236 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/base64"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
"github.com/minio/minio/internal/config/storageclass"
"github.com/minio/minio/internal/dsync"
"github.com/minio/minio/internal/grid"
xnet "github.com/pgsty/silo-pkg/v3/net"
)
func TestMultipartAbortDefaultAvailability(t *testing.T) {
for _, drives := range []int{4, 16} {
for _, offline := range []int{0, 1, drives / 2} {
t.Run(fmt.Sprintf("drives=%d/offline=%d", drives, offline), func(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), drives)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
previous := globalStorageClass
globalStorageClass.Update(storageclass.Config{Standard: storageclass.StorageClass{Parity: drives / 2}})
t.Cleanup(func() { globalStorageClass.Update(previous) })
const bucket, key = "multipart-legacy-availability", "keep-available"
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
set := z.serverPools[0].getHashedSet(key)
original := set.getDisks
disks := original()
set.getDisks = func() []StorageAPI {
visible := append([]StorageAPI(nil), disks...)
clear(visible[:offline])
return visible
}
t.Cleanup(func() { set.getDisks = original })
if err := z.AbortMultipartUpload(t.Context(), bucket, key, mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("released read-quorum availability regressed: %v", err)
}
for _, disk := range disks[offline:] {
_, err := disk.ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, key, mp.UploadID), "", ReadOptions{})
if !errors.Is(err, errFileNotFound) {
t.Fatalf("online replica not cleaned: %v", err)
}
}
})
}
}
}
func TestMultipartAbortDefaultPoolOrder(t *testing.T) {
for _, owner := range []int{0, 1} {
t.Run(fmt.Sprintf("owner=%d", owner), func(t *testing.T) {
z, bucket := consistencyPools(t)
mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
emptySet := z.serverPools[1-owner].getHashedSet("a")
original := emptySet.getDisks
t.Cleanup(func() { emptySet.getDisks = original })
emptySet.getDisks = func() []StorageAPI { return make([]StorageAPI, emptySet.setDriveCount) }
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if owner == 0 && err != nil {
t.Fatalf("unrelated later pool blocked cancellation: %v", err)
}
if owner == 1 {
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("unknown earlier pool must retain released error behavior: %v", err)
}
if _, err := z.serverPools[owner].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("later pool visited despite earlier error: %v", err)
}
emptySet.getDisks = original
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
})
}
}
func TestMultipartAbortWrongTargetKeepsCache(t *testing.T) {
z, _, bucket := multipartListingFixture(t)
const other = "multipart-other-bucket"
if err := z.MakeBucket(t.Context(), other, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, "valid-key", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, target := range [][2]string{{bucket, "wrong-key"}, {other, "valid-key"}} {
err := z.AbortMultipartUpload(t.Context(), target[0], target[1], mp.UploadID, ObjectOptions{})
var invalid InvalidUploadID
if !errors.As(err, &invalid) {
t.Fatalf("wrong target result: %v", err)
}
if _, present := z.mpCache.Load(mp.UploadID); !present {
t.Fatal("wrong-target abort evicted another upload")
}
if _, err := z.GetMultipartInfo(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("wrong-target abort harmed valid upload: %v", err)
}
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("repeat abort: %v", err)
}
}
func TestMultipartAbortRejectsUnsafeID(t *testing.T) {
z, _, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "keep", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, suffix := range []string{".", "..", "../other", "a/b", "a\\b", ""} {
id := base64.RawURLEncoding.EncodeToString([]byte("deployment." + suffix))
err := z.AbortMultipartUpload(t.Context(), bucket, "keep", id, ObjectOptions{})
var invalid InvalidUploadID
if !errors.As(err, &invalid) {
t.Fatalf("unsafe ID accepted: suffix=%q err=%v", suffix, err)
}
if _, err := z.GetMultipartInfo(t.Context(), bucket, "keep", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("valid upload harmed by rejected ID: %v", err)
}
}
}
func TestMultipartAbortPeerNotification(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
tg, err := grid.SetupTestGrid(2)
if err != nil {
t.Fatal(err)
}
t.Cleanup(tg.Cleanup)
var notifications atomic.Int32
if err := cleanupUploadIDCacheMetaRPC.Register(tg.Managers[1], func(*grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
notifications.Add(1)
return grid.NoPayload{}, nil
}); err != nil {
t.Fatal(err)
}
host, err := xnet.ParseHost(strings.TrimPrefix(tg.Hosts[1], "http://"))
if err != nil {
t.Fatal(err)
}
previous := globalNotificationSys
globalNotificationSys = &NotificationSys{peerClients: []*peerRESTClient{{
host: host,
gridConn: func() *grid.Connection { return tg.Managers[0].Connection(tg.Hosts[1]) },
}}}
t.Cleanup(func() { globalNotificationSys = previous })
// Use the real distributed lock implementation: Unlock cancels its derived
// context. Local locks do not, and would hide a broken notification context.
oldMutex, oldLockers := set.nsMutex, set.getLockers
lockers := []dsync.NetLocker{newLocker(), newLocker()}
set.nsMutex = newNSLock(true)
set.getLockers = func() ([]dsync.NetLocker, string) { return lockers, "multipart-test" }
t.Cleanup(func() { set.nsMutex, set.getLockers = oldMutex, oldLockers })
for range 4 {
mp, err := z.NewMultipartUpload(t.Context(), bucket, "valid", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
before := notifications.Load()
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "wrong", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("wrong target: %v", err)
}
if notifications.Load() != before {
t.Fatal("wrong-target abort sent a destructive peer notification")
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if notifications.Load() != before+1 {
t.Fatal("successful abort lost peer notification after unlocking")
}
}
}
func TestMultipartAbortBestEffortCleanup(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
disks := original()
set.getDisks = func() []StorageAPI {
wrapped := append([]StorageAPI(nil), disks...)
wrapped[0] = multipartListingFaultDisk{StorageAPI: disks[0], delete: func(context.Context, string, string, DeleteOptions) error { return errFaultyDisk }}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("read-quorum cancellation was tightened: %v", err)
}
// Best-effort deletion may leave a remnant. Retrying after recovery cleans
// it, but the read quorum no longer recognizes it as an active upload.
if _, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), "", ReadOptions{}); err != nil {
t.Fatalf("fault fixture did not leave a remnant: %v", err)
}
set.getDisks = original
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("minority retry must preserve default absent-upload result: %v", err)
}
if _, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), "", ReadOptions{}); !errors.Is(err, errFileNotFound) {
t.Fatalf("best-effort retry did not clean remnant: %v", err)
}
}
+106
View File
@@ -0,0 +1,106 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/xml"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
)
type multipartRootScanGuard struct {
StorageAPI
roots *atomic.Int64
}
func (d multipartRootScanGuard) ListDir(ctx context.Context, original, volume, dir string, count int) ([]string, error) {
if volume == minioMetaMultipartBucket && dir == "" {
d.roots.Add(1)
return nil, errFaultyDisk
}
return d.StorageAPI.ListDir(ctx, original, volume, dir, count)
}
// A busy or unreadable diagnostic namespace must not become a prerequisite for
// ordinary listing, including the lower-level exact-object entry points.
func TestMultipartPublicListingIndependentOfDiagnostics(t *testing.T) {
for _, busy := range []bool{false, true} {
t.Run(map[bool]string{false: "unreadable-root", true: "busy-diagnostics"}[busy], func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
const key = "dir/known"
mp, err := z.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
var roots atomic.Int64
for _, pool := range z.serverPools {
for _, set := range pool.sets {
original := set.getDisks
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, disk := range disks {
if disk != nil {
disks[i] = multipartRootScanGuard{disk, &roots}
}
}
return disks
}
t.Cleanup(func() { set.getDisks = original })
}
}
if busy {
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), true)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
}
for _, prefix := range []string{"", key, "dir/"} {
req, err := newTestSignedRequestV4(http.MethodGet,
getListMultipartUploadsURLWithParams("", bucket, prefix, "", "", "", "1"),
0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("prefix %q: HTTP %d %s", prefix, rec.Code, rec.Body.String())
}
var got ListMultipartUploadsResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
if prefix == "dir/" {
if len(got.Uploads) != 0 {
t.Fatalf("partial prefix must retain the documented exact-key limitation: %+v", got)
}
} else if len(got.Uploads) != 1 || got.Uploads[0].UploadID != mp.UploadID {
t.Fatalf("known upload missing: %+v", got)
}
}
for _, pool := range z.serverPools {
for _, list := range []func(context.Context, string, string, string, string, string, int) (ListMultipartsInfo, error){
pool.ListMultipartUploads, pool.getHashedSet(key).ListMultipartUploads,
} {
if _, err := list(t.Context(), bucket, key, "", "", "", 10); err != nil {
t.Fatalf("lower-level exact listing: %v", err)
}
}
}
if roots.Load() != 0 {
t.Fatalf("public listing attempted %d global multipart scans", roots.Load())
}
})
}
}
+256
View File
@@ -0,0 +1,256 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"bytes"
"context"
"errors"
"net/http"
"slices"
"testing"
)
func rewriteMultipartIdentity(t testing.TB, set *erasureObjects, bucket, key, id string, change func(map[string]string)) {
t.Helper()
fi, metadata, err := set.checkUploadIDExists(t.Context(), bucket, key, id, true)
if err != nil {
t.Fatal(err)
}
for i := range metadata {
change(metadata[i].Metadata)
}
if _, err := writeAllMetadata(t.Context(), set.getDisks(), bucket, minioMetaMultipartBucket,
set.getUploadIDDir(bucket, key, id), metadata, fi.WriteQuorum(set.defaultWQuorum())); err != nil {
t.Fatal(err)
}
}
func removeMultipartIdentity(metadata map[string]string) {
delete(metadata, multipartMetaBucket)
delete(metadata, multipartMetaObject)
}
// The retained diagnostic scanner handles mixed records independently of a
// stale cache. Public HTTP behavior is tested separately.
func TestMultipartDiagnosticMixed(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
for _, key := range []string{"dir/old", "dir/a", "dir/sub/b", "outside"} {
mp, err := z.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if key == "dir/old" {
rewriteMultipartIdentity(t, set, bucket, key, mp.UploadID, removeMultipartIdentity)
}
}
z.mpCache.Clear()
z.mpCache.Store("stale", MultipartInfo{Bucket: bucket, Object: "ghost", UploadID: "stale"})
list := func(prefix, keyMarker, uploadMarker, delimiter string) ListMultipartUploadsResponse {
t.Helper()
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, prefix, keyMarker, uploadMarker, delimiter, 1)
if err != nil {
t.Fatal(err)
}
return generateListMultipartUploadsResponse(bucket, got, "")
}
var keys []string
keyMarker, uploadMarker := "", ""
for page := 0; ; page++ {
if page >= 4 {
t.Fatal("pagination did not terminate")
}
got := list("", keyMarker, uploadMarker, "")
if len(got.Uploads) != 1 {
t.Fatalf("page size: %+v", got)
}
keys = append(keys, got.Uploads[0].Key)
if !got.IsTruncated {
break
}
keyMarker, uploadMarker = got.NextKeyMarker, got.NextUploadIDMarker
}
if !slices.Equal(keys, []string{"dir/a", "dir/sub/b", "outside"}) {
t.Fatalf("mixed full listing: %v", keys)
}
first := list("dir/", "", "", "/")
if len(first.Uploads) != 1 || first.Uploads[0].Key != "dir/a" || !first.IsTruncated {
t.Fatalf("prefix first page: %+v", first)
}
rest := list("dir/", first.NextKeyMarker, first.NextUploadIDMarker, "/")
if len(rest.Uploads) != 0 || rest.IsTruncated || len(rest.CommonPrefixes) != 1 || rest.CommonPrefixes[0].Prefix != "dir/sub/" {
t.Fatalf("prefix continuation: %+v", rest)
}
if got := list("dir/old", "", "", ""); len(got.Uploads) != 0 || got.IsTruncated {
t.Fatalf("historical exact-looking prefix: %+v", got)
}
}
func TestMultipartDiagnosticInvalidIdentity(t *testing.T) {
for _, kind := range []string{"missing-bucket", "missing-object", "empty-bucket", "empty-object", "empty-both", "wrong-hash", "read-failure"} {
t.Run(kind, func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
rewriteMultipartIdentity(t, set, bucket, "a", mp.UploadID, func(metadata map[string]string) {
switch kind {
case "missing-bucket":
delete(metadata, multipartMetaBucket)
case "missing-object":
delete(metadata, multipartMetaObject)
case "empty-bucket":
metadata[multipartMetaBucket] = ""
case "empty-object":
metadata[multipartMetaObject] = ""
case "empty-both":
metadata[multipartMetaBucket], metadata[multipartMetaObject] = "", ""
case "wrong-hash":
metadata[multipartMetaObject] = "wrong-key"
case "read-failure":
removeMultipartIdentity(metadata)
}
})
if kind == "read-failure" {
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, disk := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: disk, read: func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error) {
return FileInfo{}, errFaultyDisk
}}
}
return disks
}
}
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if err == nil || len(got.Uploads) != 0 || toAPIError(t.Context(), err).HTTPStatusCode != http.StatusServiceUnavailable {
t.Fatalf("invalid/unreadable record returned a successful page: %+v %v", got, err)
}
if kind != "read-failure" && !errors.Is(err, errMultipartListingIdentity) {
t.Fatalf("identity error: %v", err)
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Complete || report.Ready || report.LegacyUploads != 0 {
t.Fatalf("invalid/unreadable record counted as historical: %+v %v", report, err)
}
})
}
}
func TestMultipartListingRestart(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure(ctx, 4)
if err != nil {
cancel()
t.Fatal(err)
}
t.Cleanup(func() { cancel(); obj.Shutdown(context.Background()); removeRoots(dirs) })
const bucket, key = "multipart-restart", "dir/new"
if err := obj.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
cancel()
if err := obj.Shutdown(context.Background()); err != nil {
t.Fatal(err)
}
reopened, _, err := initObjectLayer(t.Context(), mustGetPoolEndpoints(0, dirs...))
if err != nil {
t.Fatal(err)
}
obj = reopened
if _, cached := obj.(*erasureServerPools).mpCache.Load(mp.UploadID); cached {
t.Fatal("restart unexpectedly preserved the upload cache")
}
for _, prefix := range []string{"", "dir/"} {
got, err := obj.ListMultipartUploads(t.Context(), bucket, prefix, "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got)
}
got, err := obj.ListMultipartUploads(t.Context(), bucket, key, "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, key)
if got.Uploads[0].UploadID != mp.UploadID {
t.Fatal("restart changed upload identity")
}
part, err := obj.PutObjectPart(t.Context(), bucket, key, mp.UploadID, 1,
mustGetPutObjReader(t, bytes.NewBufferString("part"), 4, "", ""), ObjectOptions{})
if err != nil {
t.Fatalf("continue known upload after restart: %v", err)
}
if _, err := obj.CompleteMultipartUpload(t.Context(), bucket, key, mp.UploadID,
[]CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{}); err != nil {
t.Fatalf("complete known upload after restart: %v", err)
}
}
func TestMultipartDiagnosticConflictingIdentityCopies(t *testing.T) {
for _, oldCopies := range []int{1, 3} {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
path := set.getUploadIDDir(bucket, "a", mp.UploadID)
for _, disk := range set.getDisks()[:oldCopies] {
fi, err := disk.ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, path, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
removeMultipartIdentity(fi.Metadata)
if err := disk.WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, path, fi); err != nil {
t.Fatal(err)
}
}
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if !errors.Is(err, errMultipartListingIdentity) || len(got.Uploads) != 0 {
t.Fatalf("%d contradictory copies returned a successful page: %+v %v", oldCopies, got, err)
}
}
}
func TestMultipartDiagnosticUnsupportedStoredID(t *testing.T) {
for _, historical := range []bool{false, true} {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
oldPath := set.getUploadIDDir(bucket, "a", mp.UploadID)
unsupportedPath := pathJoin(set.getMultipartSHADir(bucket, "a"), "unsupported-id")
for _, disk := range set.getDisks() {
fi, err := disk.ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, oldPath, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
if historical {
removeMultipartIdentity(fi.Metadata)
}
if err := disk.WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, unsupportedPath, fi); err != nil {
t.Fatal(err)
}
if err := disk.Delete(t.Context(), minioMetaMultipartBucket, oldPath, DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
got, err := z.scanMultipartUploadsForTest(t.Context(), bucket, "", "", "", "", 10)
if historical {
if err != nil || len(got.Uploads) != 0 {
t.Fatalf("unidentified historical ID: %+v %v", got, err)
}
} else if !errors.Is(err, errMultipartListingIdentity) {
t.Fatalf("invalid new ID silently treated as historical: %+v %v", got, err)
}
}
}
@@ -157,6 +157,13 @@ func copyPartWithoutChecksumHTTP(t *testing.T, apiRouter http.Handler, creds aut
if sourceRange != "" {
req.Header.Set(xhttp.AmzCopySourceRange, sourceRange)
}
// Re-sign so the copy-source x-amz-* headers are covered by the signature,
// as real S3 clients send them; the verifier rejects unsigned x-amz-*.
if creds.AccessKey != "" && creds.SecretKey != "" {
if err := signRequestV4(req, creds.AccessKey, creds.SecretKey); err != nil {
t.Fatalf("failed to re-sign UploadPartCopy request: %v", err)
}
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
+266 -68
View File
@@ -31,6 +31,7 @@ import (
"sync"
"time"
"github.com/google/uuid"
"github.com/klauspost/readahead"
"github.com/minio/minio-go/v7/pkg/set"
"github.com/minio/minio/internal/config/storageclass"
@@ -44,6 +45,15 @@ import (
"github.com/pgsty/silo-pkg/v3/sync/errgroup"
)
const (
multipartMetaBucket = ReservedMetadataPrefixLower + "multipart-v1-bucket"
multipartMetaObject = ReservedMetadataPrefixLower + "multipart-v1-object"
// ponytail: keep scan concurrency fixed until the multipart-list benchmark
// establishes a better adaptive limit.
multipartMetadataScanConcurrency = 4
)
func (er erasureObjects) getUploadIDDir(bucket, object, uploadID string) string {
uploadUUID := uploadID
uploadBytes, err := base64.RawURLEncoding.DecodeString(uploadID)
@@ -164,34 +174,46 @@ func (er erasureObjects) deleteAll(ctx context.Context, bucket, prefix string) {
wg.Wait()
}
// extractModTimeFromUploadIDDir extracts creation time from the upload ID directory on disk.
// On-disk directory names are of the form <UUID>x<UnixNano>.
// Parse the native format first: plaintext UUIDs can also be valid Base64.
// Preserve the previous encoded-format parser for compatibility.
func extractModTimeFromUploadIDDir(uploadIDDir string) time.Time {
if modTime, ok := multipartUploadTime(uploadIDDir); ok {
return modTime
}
if b64, err := base64.RawURLEncoding.DecodeString(uploadIDDir); err == nil {
if split := strings.Split(string(b64), "x"); len(split) == 2 {
if t, err := strconv.ParseInt(split[1], 10, 64); err == nil {
return time.Unix(0, t)
}
}
}
return time.Time{}
}
// Remove the old multipart uploads on the given disk.
func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk StorageAPI) {
drivePath := disk.Endpoint().Path
readDirFn(pathJoin(drivePath, minioMetaMultipartBucket), func(shaDir string, typ os.FileMode) error {
hasSkippedUploads := false
readDirFn(pathJoin(drivePath, minioMetaMultipartBucket, shaDir), func(uploadIDDir string, typ os.FileMode) error {
uploadIDPath := pathJoin(shaDir, uploadIDDir)
var modTime time.Time
// Upload IDs are of the form base64_url(<UUID>x<UnixNano>), we can extract the time from the UUID.
if b64, err := base64.RawURLEncoding.DecodeString(uploadIDDir); err == nil {
if split := strings.Split(string(b64), "x"); len(split) == 2 {
t, err := strconv.ParseInt(split[1], 10, 64)
if err == nil {
modTime = time.Unix(0, t)
}
}
}
modTime := extractModTimeFromUploadIDDir(uploadIDDir)
// Fallback for older uploads without time in the ID.
if modTime.IsZero() {
wait := deleteMultipartCleanupSleeper.Timer(ctx)
fi, err := disk.ReadVersion(ctx, "", minioMetaMultipartBucket, uploadIDPath, "", ReadOptions{})
if err != nil {
hasSkippedUploads = true
return nil
}
modTime = fi.ModTime
wait()
}
if time.Since(modTime) < globalAPIConfig.getStaleUploadsExpiry() {
hasSkippedUploads = true
return nil
}
w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
@@ -204,24 +226,15 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
return nil
})
})
// Get the modtime of the shaDir.
vi, err := disk.StatVol(ctx, pathJoin(minioMetaMultipartBucket, shaDir))
if err != nil {
return nil
}
// Modtime is returned in the Created field. See (*xlStorage).StatVol
if time.Since(vi.Created) < globalAPIConfig.getStaleUploadsExpiry() {
if hasSkippedUploads {
return nil
}
w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
return w.Run(func() error {
wait := deleteMultipartCleanupSleeper.Timer(ctx)
pathUUID := mustGetUUID()
targetPath := pathJoin(drivePath, minioMetaTmpDeletedBucket, pathUUID)
// We are not deleting shaDir recursively here, if shaDir is empty
// and its older then we can happily delete it.
Rename(pathJoin(drivePath, minioMetaMultipartBucket, shaDir), targetPath)
// Removing an upload updates the parent's mtime. Remove the parent
// immediately if it is empty; other uploads keep it in place.
_ = Remove(pathJoin(drivePath, minioMetaMultipartBucket, shaDir))
wait()
return nil
})
@@ -251,14 +264,152 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
})
}
// ListMultipartUploads - lists all the pending multipart
// uploads for a particular object in a bucket.
//
// Implements minimal S3 compatible ListMultipartUploads API. We do
// not support prefix based listing, this is a deliberate attempt
// towards simplification of multipart APIs.
// The resulting ListMultipartsInfo structure is unmarshalled directly as XML.
func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
func multipartUploadInfo(bucket, object, uploadUUID string, fallback time.Time) MultipartInfo {
initiated := fallback
if parsed, ok := multipartUploadTime(uploadUUID); ok {
initiated = parsed
}
return MultipartInfo{
Bucket: bucket,
Object: object,
UploadID: base64.RawURLEncoding.EncodeToString(fmt.Appendf(nil, "%s.%s", globalDeploymentID(), uploadUUID)),
Initiated: initiated,
}
}
// multipartUploadTime is shared by stored records and continuation markers.
// The time is part of the immutable upload ID, so a removed marker still
// identifies the same ordering boundary.
func multipartUploadTime(uploadUUID string) (time.Time, bool) {
if len(uploadUUID) < 38 || uploadUUID[36] != 'x' {
return time.Time{}, false
}
if _, err := uuid.Parse(uploadUUID[:36]); err != nil {
return time.Time{}, false
}
ns, err := strconv.ParseInt(uploadUUID[37:], 10, 64)
if err != nil || ns <= 0 || strconv.FormatInt(ns, 10) != uploadUUID[37:] {
return time.Time{}, false
}
return time.Unix(0, ns), true
}
func multipartMarkerTime(uploadID string) (time.Time, bool) {
b, err := base64.RawURLEncoding.DecodeString(uploadID)
if err != nil {
return time.Time{}, false
}
_, uploadUUID, ok := strings.Cut(string(b), ".")
if !ok {
return time.Time{}, false
}
return multipartUploadTime(uploadUUID)
}
type multipartListEntry struct {
upload *MultipartInfo
commonPrefix string
}
// paginateMultipartUploads applies the S3 ordering, prefix, delimiter, marker,
// and page rules exactly once after all pools and sets have been merged.
func paginateMultipartUploads(uploads []MultipartInfo, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) ListMultipartsInfo {
if maxUploads > maxUploadsList {
maxUploads = maxUploadsList
}
result := ListMultipartsInfo{
MaxUploads: maxUploads,
KeyMarker: keyMarker,
UploadIDMarker: uploadIDMarker,
Prefix: prefix,
Delimiter: delimiter,
}
deduplicated := make([]MultipartInfo, 0, len(uploads))
seenUploads := make(map[string]struct{}, len(uploads))
for _, upload := range uploads {
identity := upload.Bucket + "\x00" + upload.Object + "\x00" + upload.UploadID
if _, ok := seenUploads[identity]; ok {
continue
}
seenUploads[identity] = struct{}{}
deduplicated = append(deduplicated, upload)
}
sort.Slice(deduplicated, func(i, j int) bool {
if deduplicated[i].Object != deduplicated[j].Object {
return deduplicated[i].Object < deduplicated[j].Object
}
if !deduplicated[i].Initiated.Equal(deduplicated[j].Initiated) {
return deduplicated[i].Initiated.Before(deduplicated[j].Initiated)
}
return deduplicated[i].UploadID < deduplicated[j].UploadID
})
markerTime, _ := multipartMarkerTime(uploadIDMarker)
seenPrefixes := make(map[string]struct{})
entries := make([]multipartListEntry, 0, len(deduplicated))
for i := range deduplicated {
upload := &deduplicated[i]
if !strings.HasPrefix(upload.Object, prefix) {
continue
}
if keyMarker != "" {
switch strings.Compare(upload.Object, keyMarker) {
case -1:
continue
case 0:
if uploadIDMarker == "" || upload.Initiated.Before(markerTime) ||
(upload.Initiated.Equal(markerTime) && upload.UploadID <= uploadIDMarker) {
continue
}
}
}
if delimiter != "" {
remainder := strings.TrimPrefix(upload.Object, prefix)
if i := strings.Index(remainder, delimiter); i >= 0 {
commonPrefix := prefix + remainder[:i+len(delimiter)]
if keyMarker != "" && commonPrefix <= keyMarker {
continue
}
if _, ok := seenPrefixes[commonPrefix]; ok {
continue
}
seenPrefixes[commonPrefix] = struct{}{}
entries = append(entries, multipartListEntry{commonPrefix: commonPrefix})
continue
}
}
entries = append(entries, multipartListEntry{upload: upload})
}
if maxUploads <= 0 {
return result
}
pageSize := min(maxUploads, len(entries))
for _, entry := range entries[:pageSize] {
if entry.upload != nil {
result.Uploads = append(result.Uploads, *entry.upload)
continue
}
result.CommonPrefixes = append(result.CommonPrefixes, entry.commonPrefix)
}
result.IsTruncated = pageSize < len(entries)
if result.IsTruncated && pageSize > 0 {
last := entries[pageSize-1]
if last.upload != nil {
result.NextKeyMarker = last.upload.Object
result.NextUploadIDMarker = last.upload.UploadID
} else {
result.NextKeyMarker = last.commonPrefix
}
}
return result
}
// listMultipartUploadsExact preserves the hashed exact-object lookup used by
// multipart write placement and nonempty public listing prefixes.
func (er erasureObjects) listMultipartUploadsExact(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
auditObjectErasureSet(ctx, "ListMultipartUploads", object, &er)
result.MaxUploads = maxUploads
@@ -311,20 +462,16 @@ func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, objec
if populatedUploadIDs.Contains(uploadID) {
continue
}
// If present, use time stored in ID.
startTime := time.Now()
if split := strings.Split(uploadID, "x"); len(split) == 2 {
t, err := strconv.ParseInt(split[1], 10, 64)
if err == nil {
startTime = time.Unix(0, t)
var fallback time.Time
if _, ok := multipartUploadTime(uploadID); !ok {
fi, err := disk.ReadVersion(ctx, bucket, minioMetaMultipartBucket,
pathJoin(er.getMultipartSHADir(bucket, object), uploadID), "", ReadOptions{})
if err != nil {
return result, toObjectErr(err, bucket, object)
}
fallback = fi.ModTime
}
uploads = append(uploads, MultipartInfo{
Bucket: bucket,
Object: object,
UploadID: base64.RawURLEncoding.EncodeToString(fmt.Appendf(nil, "%s.%s", globalDeploymentID(), uploadID)),
Initiated: startTime,
})
uploads = append(uploads, multipartUploadInfo(bucket, object, uploadID, fallback))
populatedUploadIDs.Add(uploadID)
}
@@ -365,6 +512,13 @@ func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, objec
return result, nil
}
func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
return er.listMultipartUploadsExact(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
}
// newMultipartUpload - wrapper for initializing a new multipart
// request; returns a unique upload id.
//
@@ -402,6 +556,8 @@ func (er erasureObjects) newMultipartUpload(ctx context.Context, bucket string,
}
userDefined := cloneMSS(opts.UserDefined)
userDefined[multipartMetaBucket] = bucket
userDefined[multipartMetaObject] = object
if opts.PreserveETag != "" {
userDefined["etag"] = opts.PreserveETag
}
@@ -1155,17 +1311,10 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
return oi, toObjectErr(err, bucket, object, uploadID)
}
// A trusted SSE-C replica completion re-orders the Object Lock carried in the
// upload metadata against the version it is about to replace, read on this
// erasure set under the write lock held above, so a hold or retention that
// reached the version after this upload was initiated is not rolled back at
// completion (issue #120). Scoped to SSE-C uploads, the only ones this issue
// routes through completion.
//
// Scope: correct for a single erasure set. A multi-pool deployment (duplicate
// versions across pools, ModTime ties, cross-pool lock authority) is out of
// scope and tracked in pgsty/silo#133.
if opts.ReplicaLockReconcile && crypto.SSEC.IsEncrypted(fi.Metadata) {
// Reconcile against the upload's persisted version, under the object lock.
// Multi-pool callers supply a resolver spanning all pools, including those
// draining their contents; the upload itself stays in its original pool.
if opts.ReplicaLockReconcile {
// A persisted upload records the null version as an empty VersionID; look
// it up as the null version so the reconcile reads the addressed version's
// stored lock, not the latest version's.
@@ -1173,7 +1322,11 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
if lookupVersionID == "" {
lookupVersionID = nullVersionID
}
curr, gerr := er.getObjectInfo(ctx, bucket, object, ObjectOptions{
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
curr, gerr := getObjectInfo(ctx, bucket, object, ObjectOptions{
VersionID: lookupVersionID,
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
@@ -1182,6 +1335,7 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
switch {
case gerr == nil:
reconcileStoredObjectLock(fi.Metadata, storedObjectLockState(curr.UserDefined))
reconcileStoredObjectTags(fi.Metadata, curr.UserTags, curr.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
case isErrVersionNotFound(gerr) || isErrObjectNotFound(gerr):
// No existing version to order against: keep the upload's own accepted
// lock, including a pre-upgrade upload that persisted values without
@@ -1461,6 +1615,8 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
// Remove superfluous internal headers.
delete(fi.Metadata, hash.MinIOMultipartChecksum)
delete(fi.Metadata, hash.MinIOMultipartChecksumType)
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
// Save the final object size and modtime.
fi.Size = objectSize
@@ -1531,6 +1687,32 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
defer lk.Unlock(lkctx)
}
if !opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement {
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
if fi.VersionID == "" || fi.VersionID == nullVersionID {
if err := checkObjectLockedForOverwrite(ctx, bucket, object, getObjectInfo); err != nil {
return oi, toObjectErr(err, bucket, object)
}
} else {
_, gerr := getObjectInfo(ctx, bucket, object, ObjectOptions{
VersionID: fi.VersionID,
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoLock: true,
})
switch {
case gerr == nil, errors.As(gerr, &MethodNotAllowed{}):
return oi, toObjectErr(errAuthentication, bucket, object)
case isErrVersionNotFound(gerr) || isErrObjectNotFound(gerr):
default:
return oi, toObjectErr(gerr, bucket, object)
}
}
}
er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename().
defer func() {
@@ -1588,22 +1770,38 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
return fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
}
// AbortMultipartUpload - aborts an ongoing multipart operation
// signified by the input uploadID. This is an atomic operation
// doesn't require clients to initiate multiple such requests.
//
// All parts are purged from all disks and reference to the uploadID
// would be removed from the system, rollback is not possible on this
// operation.
func (er erasureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (err error) {
// abortMultipartUpload retains read-quorum validation and best-effort cleanup.
// It does not fence creation writes still executing after a storage timeout.
func (er erasureObjects) abortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (bool, error) {
if !opts.NoAuditLog {
auditObjectErasureSet(ctx, "AbortMultipartUpload", object, &er)
}
// Cleanup all uploaded parts.
b, err := base64.RawURLEncoding.DecodeString(uploadID)
if err != nil {
return false, MalformedUploadID{UploadID: uploadID}
}
_, internalID, ok := strings.Cut(string(b), ".")
if !ok || internalID == "" || internalID == "." || internalID == ".." || strings.ContainsAny(internalID, "/\\") {
return false, InvalidUploadID{Bucket: bucket, Object: object, UploadID: uploadID}
}
defer er.deleteAll(ctx, minioMetaMultipartBucket, er.getUploadIDDir(bucket, object, uploadID))
// Validates if upload ID exists.
_, _, err = er.checkUploadIDExists(ctx, bucket, object, uploadID, false)
return toObjectErr(err, bucket, object, uploadID)
err = toObjectErr(err, bucket, object, uploadID)
if _, absent := err.(InvalidUploadID); absent {
return false, nil
}
return err == nil, err
}
// AbortMultipartUpload cancels an upload using read-quorum validation.
// Offline part data may still need stale-upload cleanup after its drives return.
func (er erasureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (err error) {
found, err := er.abortMultipartUpload(ctx, bucket, object, uploadID, opts)
if err != nil {
return err
}
if !found {
return InvalidUploadID{Bucket: bucket, Object: object, UploadID: uploadID}
}
return nil
}
+153 -16
View File
@@ -133,6 +133,11 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
return fi.ToObjectInfo(srcBucket, srcObject, srcOpts.Versioned || srcOpts.VersionSuspended), toObjectErr(errMethodNotAllowed, srcBucket, srcObject)
}
if dstOpts.ReplicaLockReconcile {
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(fi.Metadata))
reconcileStoredObjectTags(srcInfo.UserDefined, fi.Metadata[xhttp.AmzObjectTagging], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
}
filterOnlineDisksInplace(fi, metaArr, onlineDisks)
versionID := srcInfo.VersionID
@@ -145,6 +150,17 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
}
}
if !dstOpts.ReplicationRequest && !dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement &&
(versionID == "" || versionID == nullVersionID) {
getObjectInfo := er.getObjectInfo
if dstOpts.replicaObjectInfo != nil {
getObjectInfo = dstOpts.replicaObjectInfo
}
if err := checkObjectLockedForOverwrite(ctx, dstBucket, dstObject, getObjectInfo); err != nil {
return oi, toObjectErr(err, dstBucket, dstObject)
}
}
modTime = UTCNow() // We only preserve modTime if dstOpts.MTime is true.
// in all other cases mtime is latest.
@@ -713,6 +729,57 @@ func readAllXL(ctx context.Context, disks []StorageAPI, bucket, object string, r
return pickLatestQuorumFilesInfo(ctx, rawFileInfos, errs, bucket, object, readData, inclFreeVers)
}
// readListingFileInfo is the read-only, latest-version metadata path for listing
// recovery. Ordinary GET keeps its fast-read, healing and dangling semantics.
// Absence is proved before decoding can replace a native storage error: every
// configured slot must have returned FileNotFound in this read.
func (er erasureObjects) readListingFileInfo(ctx context.Context, bucket, object string) (FileInfo, bool, error) {
disks := er.getDisks()
if len(disks) == 0 || len(disks) != er.setDriveCount {
return FileInfo{}, false, listingReadQuorumError(bucket, object)
}
// An offline slot is not evidence of absence, even if a test or transport
// implementation could return a cached not-found response for it.
online := append([]StorageAPI(nil), disks...)
for i, disk := range online {
if disk != nil && !disk.IsOnline() {
online[i] = nil
}
}
raw, errs := readAllRawFileInfo(ctx, online, bucket, object, false)
if err := ctx.Err(); err != nil {
return FileInfo{}, false, err
}
absent := true
for _, err := range errs {
absent = absent && err == errFileNotFound
}
if absent {
return FileInfo{}, true, nil
}
// Keep the same latest-version and actual object parity quorum as GET,
// without its opportunistic dangling deletion or generic NotFound mapping.
meta, errs := pickLatestQuorumFilesInfo(ctx, raw, errs, bucket, object, false, false)
quorum, _, err := objectQuorumFromMeta(ctx, meta, errs, er.defaultParityCount)
if err == nil {
err = reduceReadQuorumErrs(ctx, errs, objectOpIgnoredErrs, quorum)
}
if err != nil {
return FileInfo{}, false, listingReadQuorumError(bucket, object)
}
_, modTime, etag := listOnlineDisks(disks, meta, errs, quorum)
fi, err := pickValidFileInfo(ctx, meta, modTime, etag, quorum)
if err != nil || (!fi.Deleted && len(fi.Erasure.Distribution) != len(disks)) {
return FileInfo{}, false, listingReadQuorumError(bucket, object)
}
return fi, false, nil
}
func listingReadQuorumError(bucket, object string) error {
return InsufficientReadQuorum{Bucket: bucket, Object: decodeDirObject(object), Err: errErasureReadQuorum, Type: RQInconsistentMeta}
}
func (er erasureObjects) getObjectFileInfo(ctx context.Context, bucket, object string, opts ObjectOptions, readData bool) (FileInfo, []FileInfo, []StorageAPI, error) {
rawArr := make([]RawFileInfo, er.setDriveCount)
metaArr := make([]FileInfo, er.setDriveCount)
@@ -1067,6 +1134,12 @@ func renameData(ctx context.Context, disks []StorageAPI, srcBucket, srcEntry str
errs := g.Wait()
err := reduceWriteQuorumErrs(ctx, errs, objectOpIgnoredErrs, writeQuorum)
if errors.Is(err, errCommitMetaMissing) {
// Commit metadata went missing on a quorum of drives; the write
// did not commit. Return a 503 class error so clients retry
// instead of treating the object as absent (404).
err = errErasureWriteQuorum
}
if err != nil {
dg := errgroup.WithNErrs(len(disks))
for index, nerr := range errs {
@@ -1280,7 +1353,11 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
opts.NoLock = true
}
obj, err := er.getObjectInfo(ctx, bucket, object, opts)
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
obj, err := getObjectInfo(ctx, bucket, object, opts)
// A destination read that fails for a reason other than not-found must not
// be taken as a passed precondition or as absent lock state.
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
@@ -1297,17 +1374,12 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
}
}
// Order this trusted SSE-C replica's Object Lock against the addressed
// version's stored state, read on this erasure set under the write lock,
// so a value that lost the ordering cannot overwrite a newer one committed
// after the handler decided (issue #120). Only reconcile against an
// existing version; on not-found the write's own accepted lock is kept.
//
// Scope: correct for a single erasure set. A multi-pool deployment
// (duplicate versions across pools, ModTime ties, cross-pool lock
// authority) is out of scope and tracked in pgsty/silo#133.
// Re-read under the write lock, using the pools-layer resolver when
// present. A missing version keeps the incoming accepted state; an
// existing version contributes independently ordered lock and tags.
if opts.ReplicaLockReconcile && err == nil {
reconcileStoredObjectLock(opts.UserDefined, storedObjectLockState(obj.UserDefined))
reconcileStoredObjectTags(opts.UserDefined, obj.UserTags, obj.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
}
}
@@ -1590,6 +1662,17 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
defer lk.Unlock(lkctx)
}
if !opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement &&
(fi.VersionID == "" || fi.VersionID == nullVersionID) {
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
if err := checkObjectLockedForOverwrite(ctx, bucket, object, getObjectInfo); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
}
// Rename the successfully written temporary object to final location.
onlineDisks, versions, oldDataDir, err := renameData(ctx, onlineDisks, minioMetaTmpBucket, tempObj, partsMetadata, bucket, object, writeQuorum)
if err != nil {
@@ -1653,7 +1736,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
return fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
}
func (er erasureObjects) deleteObjectVersion(ctx context.Context, bucket, object string, fi FileInfo, forceDelMarker bool) error {
func (er erasureObjects) deleteObjectVersion(ctx context.Context, bucket, object string, fi FileInfo, forceDelMarker, purge bool) error {
disks := er.getDisks()
// Assume (N/2 + 1) quorum for Delete()
// this is a theoretical assumption such that
@@ -1669,7 +1752,13 @@ func (er erasureObjects) deleteObjectVersion(ctx context.Context, bucket, object
if disks[index] == nil {
return errDiskNotFound
}
return disks[index].DeleteVersion(ctx, bucket, object, fi, forceDelMarker, DeleteOptions{})
err := disks[index].DeleteVersion(ctx, bucket, object, fi, forceDelMarker, DeleteOptions{})
// Physical removal and reliable already-absent replies are the same
// outcome. Creation and metadata updates must not use this quorum.
if purge && (err == errFileNotFound || err == errFileVersionNotFound) {
return nil
}
return err
}, index)
}
// return errors if any during deletion
@@ -1802,6 +1891,23 @@ func (er erasureObjects) DeleteObjects(ctx context.Context, bucket string, objec
if err == nil {
continue
}
if err == errFileNotFound || err == errFileVersionNotFound {
// A retry after partial physical deletion may mix removals
// with already-absent replies. Both prove the desired outcome.
// Marker creation and replication-state updates still require
// successful writes; absence cannot acknowledge those operations.
physical := true
for _, v := range dedupVersions[i].Versions {
if v.Deleted || v.MarkDeleted || v.ReplicationState.ReplicationStatusInternal != "" ||
v.ReplicationState.VersionPurgeStatusInternal != "" || v.ReplicationState.ReplicaStatus != "" {
physical = false
break
}
}
if physical {
continue
}
}
for _, v := range dedupVersions[i].Versions {
delObjErrs[index][v.Idx] = err
}
@@ -2019,6 +2125,11 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
if opts.DeleteMarker {
versionFound = false
} else if !tryDel {
if opts.isVersionPurge() && (isErrObjectNotFound(gerr) || isErrVersionNotFound(gerr)) {
if err := er.checkPurgeAbsent(ctx, bucket, object, opts.VersionID); err != nil {
return objInfo, err
}
}
return objInfo, gerr
}
}
@@ -2101,6 +2212,20 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
}
}
if opts.DataMovement && opts.DeleteMarker {
// Moving a null marker replaces any older null data/marker. Moving
// a UUID marker preserves it on retries instead of purging it.
deleteMarker = true
markDelete = opts.VersionID != nullVersionID
}
purge := opts.isVersionPurge()
if purge {
// A delete marker describes the version being removed; it is not an
// instruction to create that marker on disks which already lack it.
markDelete, deleteMarker = false, false
}
modTime := opts.MTime
if opts.MTime.IsZero() {
modTime = UTCNow()
@@ -2147,7 +2272,7 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
// delete marker. Add delete marker, since we don't have
// any version specified explicitly. Or if a particular
// version id needs to be replicated.
if err = er.deleteObjectVersion(ctx, bucket, object, fi, opts.DeleteMarker); err != nil {
if err = er.deleteObjectVersion(ctx, bucket, object, fi, opts.DeleteMarker, false); err != nil {
return objInfo, toObjectErr(err, bucket, object)
}
oi := fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended)
@@ -2170,11 +2295,19 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
if opts.SkipFreeVersion {
dfi.SetSkipTierFreeVersion()
}
if err = er.deleteObjectVersion(ctx, bucket, object, dfi, opts.DeleteMarker); err != nil {
if err = er.deleteObjectVersion(ctx, bucket, object, dfi, opts.DeleteMarker, purge); err != nil {
return objInfo, toObjectErr(err, bucket, object)
}
return dfi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
oi := dfi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended)
if purge {
// Preserve the DELETE response's identity without reusing it as a disk
// instruction to create a marker. The lookup also exposes a data
// version pending purge as deleted for visibility; only a stored
// marker, which carries no erasure layout, is reported as one.
oi.DeleteMarker = goi.DeleteMarker && goi.DataBlocks == 0
}
return oi, nil
}
// Send the successful but partial upload/delete, however ignore
@@ -2323,7 +2456,11 @@ func (er erasureObjects) PutObjectTags(ctx context.Context, bucket, object strin
fi.Metadata[xhttp.AmzObjectTagging] = tags
fi.ReplicationState = opts.PutReplicationState()
stamp := monotonicTaggingTimestamp(opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
maps.Copy(fi.Metadata, opts.UserDefined)
if stamp != "" {
fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
}
if err = er.updateObjectMeta(ctx, bucket, object, fi, onlineDisks); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
@@ -2459,7 +2596,7 @@ func (er erasureObjects) TransitionObject(ctx context.Context, bucket, object st
storageDisks := er.getDisks()
if err = er.deleteObjectVersion(ctx, bucket, object, fi, false); err != nil {
if err = er.deleteObjectVersion(ctx, bucket, object, fi, false, false); err != nil {
eventName = event.ObjectTransitionFailed
}
+643
View File
@@ -0,0 +1,643 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"maps"
"sort"
"strings"
"time"
madmin "github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/bucket/lifecycle"
xhttp "github.com/minio/minio/internal/http"
"github.com/pgsty/silo-pkg/v3/sync/errgroup"
)
// objectPoolInfos reads the addressed version in every pool, including pools
// draining their contents. The caller must hold the pools-layer object lock.
// An unreadable pool may contain a newer version or metadata; it is not absence.
func (z *erasureServerPools) objectPoolInfos(ctx context.Context, bucket, object string, opts ObjectOptions) ([]PoolObjInfo, error) {
opts.NoLock = true
opts.CheckPrecondFn = nil
var copies []PoolObjInfo
for i, pool := range z.serverPools {
oi, err := pool.GetObjectInfo(ctx, bucket, object, opts)
if err == nil || (oi.DeleteMarker && (isErrObjectNotFound(err) || isErrMethodNotAllowed(err))) {
copies = append(copies, PoolObjInfo{Index: i, ObjInfo: oi})
continue
}
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return nil, err
}
}
sort.Slice(copies, func(i, j int) bool {
a, b := copies[i], copies[j]
if a.ObjInfo.ModTime.Equal(b.ObjInfo.ModTime) {
return a.Index < b.Index
}
return a.ObjInfo.ModTime.After(b.ObjInfo.ModTime)
})
if len(copies) == 0 {
if opts.VersionID != "" {
return nil, VersionNotFound{Bucket: bucket, Object: decodeDirObject(object), VersionID: opts.VersionID}
}
return nil, ObjectNotFound{Bucket: bucket, Object: decodeDirObject(object)}
}
return copies, nil
}
// checkDataMovementCopies rejects moving a version when another pool holds a
// newer copy of it, which reads return instead. A cross-pool overwrite can
// leave such a copy, and a client write can add one while the move copies.
// Equal timestamps normally identify an interrupted move. Conflicting ties
// retain the same pool-order winner as reads. While the caller holds the
// pools-layer object lock, the returned resolver serves the set layer's
// reconciliation of this version from the same reads. Without opts.NoLock
// it takes the read lock itself.
func (z *erasureServerPools) checkDataMovementCopies(ctx context.Context, bucket, object string, opts ObjectOptions) (GetObjectInfoFn, error) {
if opts.MTime.IsZero() {
// The set layer would stamp the moved copy as the newest one.
return nil, errInvalidArgument
}
if !opts.NoLock {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetRLock(ctx, globalOperationTimeout)
if err != nil {
return nil, err
}
ctx = lkctx.Context()
defer lk.RUnlock(lkctx)
}
versionID := opts.VersionID
if versionID == "" {
versionID = nullVersionID
}
copies, err := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{VersionID: versionID, NoAuditLog: true})
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return nil, err
}
// The snapshot may have been purged or replaced after the reader opened.
// In that case the walker must leave the current source state alone.
sourcePresent := false
for _, copy := range copies {
if copy.Index == opts.SrcPoolIdx && copy.ObjInfo.ModTime.Equal(opts.MTime) && copy.ObjInfo.DeleteMarker == opts.DeleteMarker {
sourcePresent = true
}
}
if !sourcePresent {
return nil, VersionNotFound{Bucket: bucket, Object: decodeDirObject(object), VersionID: versionID}
}
for _, copy := range copies {
if copy.Index != opts.SrcPoolIdx && copy.ObjInfo.ModTime.After(opts.MTime) {
return nil, DataMovementOverwriteErr{Bucket: bucket, Object: object, VersionID: opts.VersionID, Err: errDataMovementSuperseded}
}
}
if winner := copies[0]; winner.Index != opts.SrcPoolIdx && winner.ObjInfo.ModTime.Equal(opts.MTime) {
for _, copy := range copies {
if copy.Index == opts.SrcPoolIdx && (copy.ObjInfo.ETag != winner.ObjInfo.ETag || copy.ObjInfo.DeleteMarker != winner.ObjInfo.DeleteMarker || copy.ObjInfo.Size != winner.ObjInfo.Size) {
return nil, DataMovementOverwriteErr{Bucket: bucket, Object: object, VersionID: opts.VersionID, Err: errDataMovementSuperseded}
}
}
}
return func(ctx context.Context, b, o string, ropts ObjectOptions) (ObjectInfo, error) {
if ropts.VersionID == "" {
ropts.VersionID = nullVersionID
}
if b != bucket || o != object || ropts.VersionID != versionID {
return z.replicaObjectInfo(ctx, b, o, ropts)
}
return replicaCopiesInfo(copies, err, bucket, object, versionID)
}, nil
}
// removeMovedVersions retires only the source versions in the walker's
// snapshot. A concurrent delete can create a new marker in the source pool;
// deleting the whole prefix would remove that marker and expose moved data.
func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo) error {
object = encodeDirObject(object)
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
return err
}
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
for _, version := range versions {
vid := version.VersionID
if vid == "" {
vid = nullVersionID
}
copies, err := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{VersionID: vid, NoAuditLog: true})
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
continue
}
if err != nil {
return err
}
var src *ObjectInfo
var retained []PoolObjInfo
for _, copy := range copies {
if copy.Index == source {
src = &copy.ObjInfo
} else {
retained = append(retained, copy)
}
}
if src == nil {
continue
}
if !src.ModTime.Equal(version.ModTime) || src.DeleteMarker != version.Deleted {
return PreConditionFailed{}
}
// Successful movement is not enough if the destination changed again.
// Keep the source unless another copy still preserves its read result.
if len(retained) == 0 || retained[0].ObjInfo.ModTime.Before(src.ModTime) {
return PreConditionFailed{}
}
_, err = z.serverPools[source].DeleteObject(ctx, bucket, object, ObjectOptions{
VersionID: vid, NoLock: true, NoAuditLog: true,
SkipFreeVersion: true, // preserve the old prefix cleanup's tier-GC behavior
})
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return err
}
}
return nil
}
// Healing also commits object metadata. Both queued healing and drive healing
// must use the same namespace as pooled writes, rather than racing them under
// a destination set's independent namespace.
func (z *erasureServerPools) healObjectInPool(ctx context.Context, er *erasureObjects, bucket, object, versionID string, opts madmin.HealOpts) (madmin.HealResultItem, error) {
if !z.SinglePool() && !opts.NoLock {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
return madmin.HealResultItem{}, err
}
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
opts.NoLock = true
}
return er.HealObject(ctx, bucket, object, versionID, opts)
}
// mergePoolLockState resolves each independently ordered field. An ordered
// removal is a value in its own right; an absent unordered field is not one.
func mergePoolLockState(copies []PoolObjInfo) objectLockState {
state := storedObjectLockState(copies[0].ObjInfo.UserDefined)
for _, copy := range copies[1:] {
next := storedObjectLockState(copy.ObjInfo.UserDefined)
retentionTime, _ := time.Parse(time.RFC3339Nano, next.retentionTimestamp)
if state.retentionIsOlderThan(retentionTime) || (state.retentionTimestamp == "" && state.mode == "" && next.mode != "") {
state.mode, state.retainUntil, state.retentionTimestamp = next.mode, next.retainUntil, next.retentionTimestamp
}
holdTime, _ := time.Parse(time.RFC3339Nano, next.legalHoldTimestamp)
if state.legalHoldIsOlderThan(holdTime) || (state.legalHoldTimestamp == "" && state.legalHold == "" && next.legalHold != "") {
state.legalHold, state.legalHoldTimestamp = next.legalHold, next.legalHoldTimestamp
}
}
return state
}
func replaceObjectLockMetadata(metadata map[string]string, state objectLockState) {
for _, key := range []string{
strings.ToLower(xhttp.AmzObjectLockMode), strings.ToLower(xhttp.AmzObjectLockRetainUntilDate),
strings.ToLower(xhttp.AmzObjectLockLegalHold),
ReservedMetadataPrefixLower + ObjectLockRetentionTimestamp,
ReservedMetadataPrefixLower + ObjectLockLegalHoldTimestamp,
} {
// Metadata updates use maps.Copy at the set layer. Empty values also
// clear a previous value there, including timestamp-only removals.
if _, exists := metadata[key]; exists {
metadata[key] = ""
}
}
state.restoreRetention(metadata)
state.restoreLegalHold(metadata)
}
func mergedPoolObjectInfo(copies []PoolObjInfo) ObjectInfo {
oi := copies[0].ObjInfo
oi.UserDefined = maps.Clone(oi.UserDefined)
if oi.UserDefined == nil {
oi.UserDefined = make(map[string]string)
}
replaceObjectLockMetadata(oi.UserDefined, mergePoolLockState(copies))
for _, copy := range copies[1:] {
ts := copy.ObjInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
stamp, _ := time.Parse(time.RFC3339Nano, ts)
if olderThan(oi.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], stamp) {
oi.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = ts
oi.UserTags = copy.ObjInfo.UserTags
}
}
return oi
}
func (z *erasureServerPools) replicaObjectInfo(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
return replicaCopiesInfo(copies, err, bucket, object, opts.VersionID)
}
func replicaCopiesInfo(copies []PoolObjInfo, err error, bucket, object, versionID string) (ObjectInfo, error) {
if err != nil {
return ObjectInfo{}, err
}
if copies[0].ObjInfo.DeleteMarker {
return copies[0].ObjInfo, MethodNotAllowed{Bucket: bucket, Object: decodeDirObject(object), VersionID: versionID}
}
return mergedPoolObjectInfo(copies), nil
}
// metadataPoolInfos resolves an unqualified metadata request to one logical
// version before collecting its copies, rather than mixing different versions.
func (z *erasureServerPools) metadataPoolInfos(ctx context.Context, bucket, object string, opts ObjectOptions) ([]PoolObjInfo, error) {
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
if err != nil {
return nil, err
}
if copies[0].ObjInfo.DeleteMarker {
return nil, MethodNotAllowed{Bucket: bucket, Object: decodeDirObject(object), VersionID: opts.VersionID}
}
if opts.VersionID == "" {
opts.VersionID = copies[0].ObjInfo.VersionID
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
return z.objectPoolInfos(ctx, bucket, object, opts)
}
return copies, nil
}
func (z *erasureServerPools) updatePoolMetadata(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
copies, err := z.metadataPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
updated := mergedPoolObjectInfo(copies)
before := maps.Clone(updated.UserDefined)
if opts.EvalMetadataFn != nil {
if _, err := opts.EvalMetadataFn(&updated, nil); err != nil {
return ObjectInfo{}, err
}
}
changes := make(map[string]string)
for key, value := range updated.UserDefined {
if prior, ok := before[key]; !ok || prior != value {
changes[key] = value
}
}
for key := range before {
if _, ok := updated.UserDefined[key]; !ok {
changes[key] = ""
}
}
// Metadata callbacks may explicitly replace tags in the raw write map.
// Otherwise retain the merged value from the ObjectInfo read model.
tags, ok := updated.UserDefined[xhttp.AmzObjectTagging]
if !ok {
tags = updated.UserTags
}
state := storedObjectLockState(updated.UserDefined)
opts.VersionID = updated.VersionID
if opts.VersionID == "" {
opts.VersionID = nullVersionID
}
opts.NoLock = true
opts.EvalMetadataFn = func(oi *ObjectInfo, _ error) (ReplicateDecision, error) {
maps.Copy(oi.UserDefined, changes)
replaceObjectLockMetadata(oi.UserDefined, state)
// Reassemble the tag value and its ordering timestamp for storage.
if tags != "" || oi.UserTags != "" {
oi.UserDefined[xhttp.AmzObjectTagging] = tags
}
key := ReservedMetadataPrefixLower + TaggingTimestamp
if value, exists := updated.UserDefined[key]; exists || oi.UserDefined[key] != "" {
oi.UserDefined[key] = value
}
return ReplicateDecision{}, nil
}
var primary ObjectInfo
for _, copy := range copies {
oi, err := z.serverPools[copy.Index].PutObjectMetadata(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
if copy.Index == copies[0].Index {
primary = oi
}
}
return primary, nil
}
// Pass the stored tag value explicitly: ObjectInfo.UserDefined excludes it,
// whereas FileInfo.Metadata retains the raw storage key.
func reconcileStoredObjectTags(metadata map[string]string, storedTags, storedTimestamp string) {
key := ReservedMetadataPrefixLower + TaggingTimestamp
stamp, err := time.Parse(time.RFC3339Nano, storedTimestamp)
if err != nil {
return
}
incoming, err := time.Parse(time.RFC3339Nano, metadata[key])
if err != nil || !stamp.Before(incoming) {
metadata[key] = storedTimestamp
metadata[xhttp.AmzObjectTagging] = storedTags
}
}
// Local tagging mutations must advance the revision they overwrite, even when
// a request's clock or lock acquisition order is behind the stored revision.
// Replica writes use reconcileStoredObjectTags instead of minting a revision.
func monotonicTaggingTimestamp(incoming, stored string) string {
requested, err := time.Parse(time.RFC3339Nano, incoming)
if err != nil {
return incoming
}
current, err := time.Parse(time.RFC3339Nano, stored)
if err != nil || requested.After(current) {
return incoming
}
return current.Add(time.Nanosecond).UTC().Format(time.RFC3339Nano)
}
// A restored version still owns its tier reference even while IsRemote is
// false. Only the last copy of a reference may schedule its contents for GC.
func sharesTierObject(oi ObjectInfo, copies []PoolObjInfo) bool {
ref := oi.TransitionedObject
if ref.Status != lifecycle.TransitionComplete {
return false
}
for _, copy := range copies {
other := copy.ObjInfo.TransitionedObject
if other.Status == lifecycle.TransitionComplete && ref.Tier == other.Tier && ref.Name == other.Name && ref.VersionID == other.VersionID {
return true
}
}
return false
}
// retireReplicaCopies runs only after committing a replacement. Failures are
// returned to the caller, so a stale copy cannot be hidden behind a successful
// response. Data movement owns its source cleanup and does not use this helper.
func (z *erasureServerPools) retireReplicaCopies(ctx context.Context, bucket, object string, keep int, oi ObjectInfo) error {
versionID := oi.VersionID
if versionID == "" {
versionID = nullVersionID
}
copies, err := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{VersionID: versionID})
if err != nil {
return err
}
var retained []PoolObjInfo
for _, copy := range copies {
if copy.Index == keep {
retained = []PoolObjInfo{copy}
break
}
}
if len(retained) == 0 {
return VersionNotFound{Bucket: bucket, Object: decodeDirObject(object), VersionID: versionID}
}
for i, copy := range copies {
if copy.Index == keep {
continue
}
_, err := z.serverPools[copy.Index].DeleteObject(ctx, bucket, object,
ObjectOptions{
VersionID: versionID, NoLock: true, NoAuditLog: true,
SkipFreeVersion: sharesTierObject(copy.ObjInfo, retained) || sharesTierObject(copy.ObjInfo, copies[i+1:]),
})
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return err
}
}
return nil
}
// deleteObjectReconciled evaluates any condition once against the logical
// version, then removes all its copies under the same lock as pooled writers.
func (z *erasureServerPools) deleteObjectReconciled(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
if err != nil {
if opts.isVersionPurge() && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
if quorumErr := z.checkPurgeAbsent(ctx, bucket, object, opts.VersionID); quorumErr != nil {
return ObjectInfo{}, quorumErr
}
}
return ObjectInfo{}, err
}
primary := copies[0]
if opts.CheckPrecondFn != nil && opts.CheckPrecondFn(primary.ObjInfo) {
return ObjectInfo{}, PreConditionFailed{}
}
opts.CheckPrecondFn = nil
opts.NoLock = true
if opts.EvalRetentionBypassFn != nil || opts.EvalMetadataFn != nil {
logical := primary.ObjInfo
var gerr error
switch {
case logical.DeleteMarker:
// Markers can be deleted by version ID. Match the set layer's
// callback inputs instead of rejecting them as metadata updates.
gerr = toObjectErr(errMethodNotAllowed, bucket, object)
if opts.VersionID == "" || opts.DeleteMarker {
gerr = toObjectErr(errFileNotFound, bucket, object)
}
case opts.VersionID != "":
// An addressed version already resolved every copy above.
logical = mergedPoolObjectInfo(copies)
default:
versions, err := z.metadataPoolInfos(ctx, bucket, object, opts)
if err != nil {
return ObjectInfo{}, err
}
logical = mergedPoolObjectInfo(versions)
}
// Keep the retention gate first. These callbacks independently evaluate
// the logical version and run once before any deletion; the handler only
// sweeps metadata's transition state after a successful delete.
if opts.EvalRetentionBypassFn != nil {
if err := opts.EvalRetentionBypassFn(logical, gerr); err != nil {
return ObjectInfo{}, err
}
opts.EvalRetentionBypassFn = nil
}
if opts.EvalMetadataFn != nil {
decision, err := opts.EvalMetadataFn(&logical, gerr)
if err != nil {
return ObjectInfo{}, err
}
if decision.ReplicateAny() {
opts.SetDeleteReplicationState(decision, opts.VersionID)
}
opts.EvalMetadataFn = nil
}
}
if opts.isVersionPurge() {
// A missing pool may have only read-quorum absence. Verify every pool
// omitted by lookup before mutating the known copies.
for i, pool := range z.serverPools {
found := false
for _, copy := range copies {
found = found || copy.Index == i
}
if !found {
if err := pool.getHashedSet(object).checkPurgeAbsent(ctx, bucket, object, opts.VersionID); err != nil {
return ObjectInfo{}, err
}
}
}
}
if opts.VersionID == "" && (opts.Versioned || opts.VersionSuspended) {
// A single new delete marker hides the current version. Older versions
// remain history and must not be removed by an unqualified DELETE.
oi, err := z.serverPools[primary.Index].DeleteObject(ctx, bucket, object, opts)
oi.Name = decodeDirObject(object)
oi.replicationDecision = opts.DeleteReplication.ReplicateDecisionStr
return oi, err
}
// Retire non-authoritative copies first. If cleanup fails, retain the
// authoritative version and report the error instead of acknowledging a
// deletion that would expose an older copy.
for i := 1; i < len(copies); i++ {
candidate := copies[i]
deleteOpts := opts
deleteOpts.SkipFreeVersion = opts.SkipFreeVersion || sharesTierObject(candidate.ObjInfo, copies[:1]) || sharesTierObject(candidate.ObjInfo, copies[i+1:])
_, err := z.serverPools[candidate.Index].DeleteObject(ctx, bucket, object, deleteOpts)
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
return ObjectInfo{}, err
}
}
oi, err := z.serverPools[primary.Index].DeleteObject(ctx, bucket, object, opts)
oi.Name = decodeDirObject(object)
oi.replicationDecision = opts.DeleteReplication.ReplicateDecisionStr
return oi, err
}
// deleteObjectsReconciled removes data while retaining each item's authoritative
// copy until every other pool has completed its cleanup. The caller holds the
// pools-layer write locks and separates marker creation from this batch.
func (z *erasureServerPools) deleteObjectsReconciled(ctx context.Context, bucket string, objects []ObjectToDelete, opts ObjectOptions) ([]DeletedObject, []error) {
deleted := make([]DeletedObject, len(objects))
errs := make([]error, len(objects))
primary := make([]int, len(objects))
tiered := make([]bool, len(objects))
lookup := errgroup.WithNErrs(len(objects)).WithConcurrency(32)
for i, obj := range objects {
// Preserve the full request identity even if lookup or cleanup fails.
// The HTTP handler includes replication state in its response-index key.
deleted[i] = DeletedObject{
ObjectName: obj.ObjectName,
VersionID: obj.VersionID,
ReplicationState: obj.ReplicationState(),
}
primary[i] = -1
lookup.Go(func() error {
copies, err := z.objectPoolInfos(ctx, bucket, obj.ObjectName, ObjectOptions{VersionID: obj.VersionID, NoAuditLog: true})
if err == nil {
primary[i] = copies[0].Index
for _, copy := range copies {
tiered[i] = tiered[i] || copy.ObjInfo.TransitionedObject.Status == lifecycle.TransitionComplete
}
} else if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
if isErrReadQuorum(err) {
err = InsufficientWriteQuorum{Bucket: bucket, Object: decodeDirObject(obj.ObjectName)}
}
errs[i] = err
}
return nil
}, i)
}
lookup.Wait()
type poolBatch struct {
objects []ObjectToDelete
indices []int
deleted []DeletedObject
errs []error
}
for phase := 0; phase < 2; phase++ {
batches := make([]poolBatch, len(z.serverPools))
for i, obj := range objects {
if errs[i] != nil || tiered[i] {
continue
}
for p := range z.serverPools {
if (p == primary[i]) != (phase == 1) {
continue
}
batches[p].objects = append(batches[p].objects, obj)
batches[p].indices = append(batches[p].indices, i)
}
}
// Each phase still uses the set layer's bulk disk I/O and runs pools
// concurrently. Include apparently absent pools in cleanup: read-quorum
// absence is not proof that an offline drive holds no remaining copy.
group := errgroup.WithNErrs(len(z.serverPools))
for p, pool := range z.serverPools {
if len(batches[p].objects) == 0 {
continue
}
group.Go(func() error {
batches[p].deleted, batches[p].errs = pool.DeleteObjects(ctx, bucket, batches[p].objects, opts)
return nil
}, p)
}
group.Wait()
for _, batch := range batches {
for j, i := range batch.indices {
if errs[i] != nil {
continue
}
if batch.errs[j] != nil {
errs[i] = batch.errs[j]
} else if phase == 1 || primary[i] == -1 {
deleted[i] = batch.deleted[j]
}
}
}
}
// Tiered copies can share one remote object, including restored copies.
// Reuse the existing reconciliation which suppresses premature tier GC.
// Serializing this uncommon path also keeps duplicate key/version entries
// in one request from racing their metadata updates under the shared lock.
for i, obj := range objects {
if !tiered[i] || errs[i] != nil {
continue
}
deleteOpts := opts
deleteOpts.VersionID, deleteOpts.NoLock = obj.VersionID, true
deleteOpts.DeleteReplication = obj.ReplicationState()
if opts.PrefixEnabledFn != nil {
deleteOpts.Versioned = opts.PrefixEnabledFn(obj.ObjectName)
}
_, err := z.deleteObjectReconciled(ctx, bucket, obj.ObjectName, deleteOpts)
if err != nil && !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
errs[i] = err
} else {
deleted[i].found = true
}
}
return deleted, errs
}
File diff suppressed because it is too large Load Diff
+24 -34
View File
@@ -616,9 +616,15 @@ func (z *erasureServerPools) decommissionObject(ctx context.Context, idx int, bu
}
if objInfo.isMultipart() {
// Completion repeats this check under the object write lock.
if _, err := z.checkDataMovementCopies(ctx, bucket, encodeDirObject(objInfo.Name), ObjectOptions{
VersionID: objInfo.VersionID, MTime: objInfo.ModTime, SrcPoolIdx: idx,
}); err != nil {
return err
}
res, err := z.NewMultipartUpload(ctx, bucket, objInfo.Name, ObjectOptions{
VersionID: objInfo.VersionID,
UserDefined: objInfo.UserDefined,
UserDefined: migrationObjectMetadata(objInfo),
NoAuditLog: true,
SrcPoolIdx: idx,
DataMovement: true,
@@ -656,6 +662,7 @@ func (z *erasureServerPools) decommissionObject(ctx context.Context, idx int, bu
}
}
_, err = z.CompleteMultipartUpload(ctx, bucket, objInfo.Name, res.UploadID, parts, ObjectOptions{
VersionID: objInfo.VersionID,
SrcPoolIdx: idx,
DataMovement: true,
MTime: objInfo.ModTime,
@@ -681,7 +688,7 @@ func (z *erasureServerPools) decommissionObject(ctx context.Context, idx int, bu
SrcPoolIdx: idx,
VersionID: objInfo.VersionID,
MTime: objInfo.ModTime,
UserDefined: objInfo.UserDefined,
UserDefined: migrationObjectMetadata(objInfo),
PreserveETag: objInfo.ETag, // Preserve original ETag to ensure same metadata.
IndexCB: func() []byte {
return objInfo.Parts[0].Index // Preserve part Index to ensure decompression works.
@@ -825,27 +832,17 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
// to create the appropriate stack.
versionsSorter(fivs.Versions).reverse()
var decommissioned, expired int
var decommissioned int
for _, version := range fivs.Versions {
stopFn := globalDecommissionMetrics.log(decomMetricDecommissionObject, idx, bi.Name, version.Name, version.VersionID)
// Apply lifecycle rules on the objects that are expired.
if filterLifecycle(bi.Name, version.Name, version) {
expired++
decommissioned++
stopFn(version.Size, errors.New("ILM expired object/version will be skipped"))
continue
}
// any object with only single DEL marker we don't need
// to decommission, just skip it, this also includes
// any other versions that have already expired.
remainingVersions := len(fivs.Versions) - expired
if version.Deleted && remainingVersions == 1 && rcfg == nil {
decommissioned++
stopFn(version.Size, errors.New("DELETE marked object with no other non-current versions will be skipped"))
continue
}
// Preserve every marker: older versions may live in another pool.
versionID := version.VersionID
if versionID == "" {
versionID = nullVersionID
@@ -870,6 +867,9 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
SkipDecommissioned: true, // make sure we skip the decommissioned pool
NoAuditLog: true,
})
if isDataMovementSuperseded(err) {
err = nil
}
if err != nil {
// This can happen when rebalance stop races with ongoing rebalance workers.
// These rebalance failures can be ignored.
@@ -898,17 +898,19 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
// gr.Close() is ensured by decommissionObject().
for range 3 {
if version.IsRemote() {
if err := z.DecomTieredObject(ctx, bi.Name, version.Name, version, ObjectOptions{
err := z.DecomTieredObject(ctx, bi.Name, version.Name, version, ObjectOptions{
VersionID: versionID,
MTime: version.ModTime,
UserDefined: version.Metadata,
SrcPoolIdx: idx,
DataMovement: true,
}); err != nil {
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
ignore = true
stopFn(0, nil)
}
})
if isDataMovementSuperseded(err) {
err = nil
}
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
ignore = true
stopFn(0, nil)
}
if !ignore {
stopFn(version.Size, err)
@@ -949,7 +951,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
stopFn(version.Size, err)
continue
}
if err = z.decommissionObject(ctx, idx, bi.Name, gr); err != nil {
if err = z.decommissionObject(ctx, idx, bi.Name, gr); err != nil && !isDataMovementSuperseded(err) {
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
ignore = true
stopFn(0, nil)
@@ -979,15 +981,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
// if all versions were decommissioned, then we can delete the object versions.
if decommissioned == len(fivs.Versions) {
stopFn := globalDecommissionMetrics.log(decomMetricDecommissionRemoveObject, idx, bi.Name, entry.name)
_, err := set.DeleteObject(ctx,
bi.Name,
encodeDirObject(entry.name),
ObjectOptions{
DeletePrefix: true, // use prefix delete to delete all versions at once.
DeletePrefixObject: true, // use prefix delete on exact object (this is an optimization to avoid fan-out calls)
NoAuditLog: true,
},
)
err := z.removeMovedVersions(ctx, bi.Name, entry.name, idx, fivs.Versions)
stopFn(0, err)
auditLogDecom(ctx, "DecomDeleteObject", bi.Name, entry.name, "", err)
if err != nil {
@@ -1196,10 +1190,6 @@ func (z *erasureServerPools) checkAfterDecom(ctx context.Context, idx int) error
ignored++
continue
}
if version.Deleted {
ignored++
continue
}
}
versionsFound += len(fivs.Versions) - ignored
+5 -1
View File
@@ -23,6 +23,10 @@ import (
)
func prepareErasurePools() (ObjectLayer, []string, error) {
return prepareErasurePoolsWithContext(context.Background())
}
func prepareErasurePoolsWithContext(ctx context.Context) (ObjectLayer, []string, error) {
nDisks := 32
fsDirs, err := getRandomDisks(nDisks)
if err != nil {
@@ -32,7 +36,7 @@ func prepareErasurePools() (ObjectLayer, []string, error) {
pools := mustGetPoolEndpoints(0, fsDirs[:16]...)
pools = append(pools, mustGetPoolEndpoints(1, fsDirs[16:]...)...)
objLayer, _, err := initObjectLayer(context.Background(), pools)
objLayer, _, err := initObjectLayer(ctx, pools)
if err != nil {
removeRoots(fsDirs)
return nil, nil, err
+582
View File
@@ -0,0 +1,582 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"syscall"
"testing"
"time"
minio "github.com/minio/minio-go/v7"
)
func bulkDeletePools(t *testing.T, n int) (*erasureServerPools, string) {
t.Helper()
ctx, cancel := context.WithCancel(t.Context())
dirs, err := getRandomDisks(4 * n)
if err != nil {
cancel()
t.Fatal(err)
}
var endpoints EndpointServerPools
for i := range n {
endpoints = append(endpoints, mustGetPoolEndpoints(i, dirs[4*i:4*i+4]...)...)
}
obj, _, err := initObjectLayer(ctx, endpoints)
if err != nil {
cancel()
removeRoots(dirs)
t.Fatal(err)
}
z := obj.(*erasureServerPools)
previous := newObjectLayerFn()
setObjectLayer(z)
t.Cleanup(func() { cancel(); z.Shutdown(context.Background()); removeRoots(dirs); setObjectLayer(previous) })
bucket := "bulk-delete-consistency"
if err := z.MakeBucket(ctx, bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
return z, bucket
}
func bulkDeleteBody(t *testing.T, z *erasureServerPools, bucket, object, version string) string {
t.Helper()
r, err := z.GetObjectNInfo(t.Context(), bucket, object, nil, nil, ObjectOptions{VersionID: version})
if err != nil {
t.Fatal(err)
}
defer r.Close()
body, err := io.ReadAll(r)
if err != nil {
t.Fatal(err)
}
return string(body)
}
// All drives are online and all metadata reads succeed. Failure starts inside
// DeleteVersions, after any preflight. Count attempts to delete the latest copy
// instead of waiting for them: a safe implementation must never make that call.
type bulkDeleteFaultDisk struct {
StorageAPI
bucket, object, mode string
fail, latest bool
failed, attempted *atomic.Int32
}
func (d bulkDeleteFaultDisk) DeleteVersions(ctx context.Context, bucket string, versions []FileInfoVersions, opts DeleteOptions) []error {
if bucket != d.bucket {
return d.StorageAPI.DeleteVersions(ctx, bucket, versions, opts)
}
errs := make([]error, len(versions))
for i, v := range versions {
if v.Name != d.object {
errs[i] = d.StorageAPI.DeleteVersions(ctx, bucket, []FileInfoVersions{v}, opts)[0]
continue
}
if d.latest {
d.attempted.Add(1)
}
if !d.fail {
errs[i] = d.StorageAPI.DeleteVersions(ctx, bucket, []FileInfoVersions{v}, opts)[0]
continue
}
d.failed.Add(1)
switch d.mode {
case "eio":
errs[i] = syscall.EIO
case "disconnect":
errs[i] = errDiskNotFound
case "permission":
meta := filepath.Join(d.String(), bucket, v.Name, xlStorageFormatFile)
info, err := os.Stat(meta)
if err == nil {
err = os.Chmod(meta, 0)
}
if err != nil {
errs[i] = err
continue
}
errs[i] = d.StorageAPI.DeleteVersions(ctx, bucket, []FileInfoVersions{v}, opts)[0]
if err := os.Chmod(meta, info.Mode()); err != nil {
panic(err)
}
}
}
return errs
}
func requireBulkDeletePermissionFailure(t *testing.T) {
t.Helper()
probe := filepath.Join(t.TempDir(), "permission-probe")
if err := os.WriteFile(probe, []byte("metadata"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Chmod(probe, 0); err != nil {
t.Skipf("cannot inject filesystem permissions: %v", err)
}
_, err := os.ReadFile(probe)
if restoreErr := os.Chmod(probe, 0o600); restoreErr != nil {
t.Fatal(restoreErr)
}
if !os.IsPermission(err) {
t.Skip("this user/filesystem can read mode-000 files")
}
}
func TestPoolsBulkDeleteKeepsLatestOnIOFailure(t *testing.T) {
for _, mode := range []string{"eio", "disconnect", "permission"} {
for _, n := range []int{2, 3} {
for primary := range n {
for _, kind := range []string{"unversioned", "null", "uuid"} {
t.Run(fmt.Sprintf("%s/pools=%d/primary=%d/%s", mode, n, primary, kind), func(t *testing.T) {
if mode == "permission" {
requireBulkDeletePermissionFailure(t)
}
z, bucket := bulkDeletePools(t, n)
object, old := "target", (primary+1)%n
opts := ObjectOptions{MTime: UTCNow().Add(-time.Hour), MaxParity: true}
oldBody := "old"
if kind == "uuid" {
opts.Versioned, opts.VersionID, oldBody = true, mustGetUUID(), "latest"
}
putConsistencyObject(t, z, bucket, object, old, oldBody, opts)
opts.MTime = opts.MTime.Add(time.Minute)
latest := putConsistencyObject(t, z, bucket, object, primary, "latest", opts)
version := opts.VersionID
if kind == "null" {
version = nullVersionID
}
for _, key := range []string{"before", "after"} {
putConsistencyObject(t, z, bucket, key, primary, "unrelated", ObjectOptions{})
}
var failed, attempted atomic.Int32
var restore []func()
for _, p := range []int{old, primary} {
set := z.serverPools[p].getHashedSet(object)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
for i, disk := range disks {
disks[i] = bulkDeleteFaultDisk{StorageAPI: disk, bucket: bucket, object: object, mode: mode, fail: p == old, latest: p == primary, failed: &failed, attempted: &attempted}
}
set.getDisks = func() []StorageAPI { return disks }
restore = append(restore, func() { set.getDisks = original })
}
defer func() {
for _, f := range restore {
f()
}
}()
objects := []ObjectToDelete{{ObjectV: ObjectV{ObjectName: "before"}}, {ObjectV: ObjectV{ObjectName: object, VersionID: version}}, {ObjectV: ObjectV{ObjectName: "after"}}}
deleted, errs := z.DeleteObjects(t.Context(), bucket, objects, ObjectOptions{})
if failed.Load() != 4 || attempted.Load() != 0 {
t.Errorf("cleanup failures=%d; latest deletion attempts=%d, want 4/0", failed.Load(), attempted.Load())
}
if errs[1] == nil {
t.Error("cleanup failure was reported as success")
}
for _, i := range []int{0, 2} {
if errs[i] != nil || deleted[i].ObjectName != objects[i].ObjectName {
t.Errorf("unrelated item %d: %+v, %v", i, deleted[i], errs[i])
}
}
if body := bulkDeleteBody(t, z, bucket, object, version); body != "latest" {
t.Errorf("GET during failure=%q", body)
}
for _, f := range restore {
f()
}
if body := bulkDeleteBody(t, z, bucket, object, version); body != "latest" {
t.Errorf("GET after recovery=%q", body)
}
listed, err := z.ListObjectVersions(t.Context(), bucket, object, "", "", "", 100)
if err != nil || len(listed.Objects) != 1 || listed.Objects[0].ETag != latest.ETag {
t.Errorf("recovered versions=%+v, %v", listed.Objects, err)
}
_, retry := z.DeleteObjects(t.Context(), bucket, objects[1:2], ObjectOptions{})
if retry[0] != nil {
t.Fatal(retry[0])
}
_, err = z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: version})
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
t.Errorf("retry retained the object: %v", err)
}
})
}
}
}
}
}
func TestPoolsBulkDeleteReplicationIdentity(t *testing.T) {
z, _ := bulkDeletePools(t, 2)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{VersioningEnabled: true})
if err != nil {
t.Fatal(err)
}
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Error("unexpected outbound replication for a failed deletion")
w.WriteHeader(http.StatusNoContent)
}))
defer remote.Close()
client, err := minio.New(strings.TrimPrefix(remote.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
_, restoreReplication := r5ReplicationFixture(t, z, bucket, client)
defer restoreReplication()
versions := make(map[string]string)
for _, key := range []string{"first", "second"} {
v := putConsistencyObject(t, z, bucket, key, 0, "payload", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, key, 1, "payload", ObjectOptions{Versioned: true, VersionID: v.VersionID, MTime: v.ModTime})
versions[key] = v.VersionID
}
set := z.serverPools[0].getHashedSet("first")
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
disks[0], disks[1] = nil, nil
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
body := []byte(fmt.Sprintf("<Delete><Object><Key>first</Key><VersionId>%s</VersionId></Object><Object><Key>second</Key><VersionId>%s</VersionId></Object></Delete>", versions["first"], versions["second"]))
req, err := newTestSignedRequestV4(http.MethodPost, getMultiDeleteObjectURL("", bucket), int64(len(body)), bytes.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
var result struct {
Errors []struct {
Key, Code string
VersionID string `xml:"VersionId"`
} `xml:"Error"`
Deleted []struct{ Key string } `xml:"Deleted"`
}
if err := xml.Unmarshal(rec.Body.Bytes(), &result); err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusOK || len(result.Errors) != 2 || len(result.Deleted) != 0 {
t.Fatalf("lost response identity: HTTP %d %s", rec.Code, rec.Body.String())
}
for _, e := range result.Errors {
if versions[e.Key] == "" || e.VersionID != versions[e.Key] || e.Code != "SlowDownWrite" {
t.Errorf("incorrect error identity: %+v", e)
}
delete(versions, e.Key)
}
if len(versions) != 0 {
t.Errorf("missing errors: %v", versions)
}
}
func TestPoolsBulkDeleteNullBeforeMarker(t *testing.T) {
for _, n := range []int{1, 2, 3} {
t.Run(fmt.Sprintf("pools=%d", n), func(t *testing.T) {
z, bucket := bulkDeletePools(t, n)
object := "suspended"
history := putConsistencyObject(t, z, bucket, object, 0, "history", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
for p := range n {
putConsistencyObject(t, z, bucket, object, p, "null", ObjectOptions{MTime: UTCNow().Add(-time.Duration(p+1) * time.Minute)})
}
objects := []ObjectToDelete{{ObjectV: ObjectV{ObjectName: object}}, {ObjectV: ObjectV{ObjectName: object, VersionID: nullVersionID}}}
deleted, errs := z.DeleteObjects(t.Context(), bucket, objects, ObjectOptions{VersionSuspended: true})
if errs[0] != nil || errs[1] != nil || !deleted[0].DeleteMarker || deleted[1].VersionID != nullVersionID {
t.Fatalf("results=%+v errors=%v", deleted, errs)
}
info, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
if !info.DeleteMarker || !isErrObjectNotFound(err) {
t.Errorf("replacement null marker lost: %+v, %v", info, err)
}
if body := bulkDeleteBody(t, z, bucket, object, history.VersionID); body != "history" {
t.Errorf("UUID history changed: %q", body)
}
})
}
}
func TestPoolsBulkDeletePartialCleanupRetry(t *testing.T) {
for _, kind := range []string{"unversioned", "null", "uuid"} {
t.Run(kind, func(t *testing.T) {
z, bucket := bulkDeletePools(t, 2)
object := "partial-cleanup"
opts := ObjectOptions{MTime: UTCNow().Add(-time.Hour)}
oldBody := "old"
if kind == "uuid" {
opts.Versioned, opts.VersionID, oldBody = true, mustGetUUID(), "latest"
}
putConsistencyObject(t, z, bucket, object, 0, oldBody, opts)
opts.MTime = UTCNow()
putConsistencyObject(t, z, bucket, object, 1, "latest", opts)
version := opts.VersionID
if kind == "null" {
version = nullVersionID
}
set := z.serverPools[0].getHashedSet(object)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
var failed, attempted atomic.Int32
for i := range 2 {
disks[i] = bulkDeleteFaultDisk{StorageAPI: disks[i], bucket: bucket, object: object, mode: "eio", fail: true, failed: &failed, attempted: &attempted}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
objects := []ObjectToDelete{{ObjectV: ObjectV{ObjectName: object, VersionID: version}}}
_, errs := z.DeleteObjects(t.Context(), bucket, objects, ObjectOptions{})
if errs[0] == nil || failed.Load() != 2 {
t.Fatalf("did not partially fail cleanup: %v, failures=%d", errs, failed.Load())
}
set.getDisks = original
if got := bulkDeleteBody(t, z, bucket, object, version); got != "latest" {
t.Fatalf("latest lost after partial cleanup: %q", got)
}
_, errs = z.DeleteObjects(t.Context(), bucket, objects, ObjectOptions{})
if errs[0] != nil {
t.Fatalf("recovered retry must combine successful removals and already-absent replies: %v", errs[0])
}
_, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: version})
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
t.Errorf("retry retained object: %v", err)
}
})
}
}
func TestPoolsBulkDeleteDifferentPrimaries(t *testing.T) {
for _, n := range []int{2, 3} {
t.Run(fmt.Sprintf("pools=%d", n), func(t *testing.T) {
z, bucket := bulkDeletePools(t, n)
objects := make([]ObjectToDelete, n)
for primary := range n {
object := fmt.Sprintf("key-%d", primary)
objects[primary].ObjectName = object
putConsistencyObject(t, z, bucket, object, (primary+1)%n, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, primary, "latest", ObjectOptions{})
}
set := z.serverPools[0].getHashedSet(objects[n-1].ObjectName)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
var failed, attempted atomic.Int32
for i, disk := range disks {
disks[i] = bulkDeleteFaultDisk{StorageAPI: disk, bucket: bucket, object: objects[n-1].ObjectName, mode: "eio", fail: true, failed: &failed, attempted: &attempted}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
deleted, errs := z.DeleteObjects(t.Context(), bucket, objects, ObjectOptions{})
for i := range objects {
if deleted[i].ObjectName != objects[i].ObjectName {
t.Errorf("wrong result index %d: %+v", i, deleted[i])
}
if i == n-1 {
if errs[i] == nil || bulkDeleteBody(t, z, bucket, objects[i].ObjectName, "") != "latest" {
t.Errorf("failed item lost latest: %v", errs[i])
}
continue
}
_, err := z.GetObjectInfo(t.Context(), bucket, objects[i].ObjectName, ObjectOptions{})
if errs[i] != nil || !isErrObjectNotFound(err) {
t.Errorf("healthy item %d: delete=%v GET=%v", i, errs[i], err)
}
}
})
}
}
func TestPoolsBulkDeleteTieAndUnreadable(t *testing.T) {
for _, unreadable := range []bool{false, true} {
t.Run(fmt.Sprintf("unreadable=%t", unreadable), func(t *testing.T) {
z, bucket := bulkDeletePools(t, 3)
object := "same-time"
stamp := UTCNow().Add(-time.Hour)
for p := range 3 {
putConsistencyObject(t, z, bucket, object, p, fmt.Sprintf("pool-%d", p), ObjectOptions{MTime: stamp})
}
if got := bulkDeleteBody(t, z, bucket, object, ""); got != "pool-0" {
t.Fatalf("unexpected read winner: %q", got)
}
set := z.serverPools[2].getHashedSet(object)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
var failed, attempted atomic.Int32
for i, disk := range disks {
if unreadable {
disks[i] = consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
} else {
disks[i] = bulkDeleteFaultDisk{StorageAPI: disk, bucket: bucket, object: object, mode: "eio", fail: true, failed: &failed, attempted: &attempted}
}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
_, errs := z.DeleteObjects(t.Context(), bucket, []ObjectToDelete{{ObjectV: ObjectV{ObjectName: object}}}, ObjectOptions{})
set.getDisks = original
if errs[0] == nil || bulkDeleteBody(t, z, bucket, object, "") != "pool-0" {
t.Errorf("lost tie winner or unreadable-pool error: %v", errs[0])
}
})
}
}
type bulkDeletePauseDisk struct {
StorageAPI
bucket, object string
entered chan<- struct{}
release <-chan struct{}
}
func (d bulkDeletePauseDisk) DeleteVersions(ctx context.Context, bucket string, versions []FileInfoVersions, opts DeleteOptions) []error {
for _, v := range versions {
if bucket != d.bucket || v.Name != d.object {
continue
}
select {
case d.entered <- struct{}{}:
default:
}
select {
case <-d.release:
case <-ctx.Done():
}
errs := make([]error, len(versions))
for i := range errs {
errs[i] = syscall.EIO
}
return errs
}
return d.StorageAPI.DeleteVersions(ctx, bucket, versions, opts)
}
func TestPoolsBulkDeleteLockAndCancellation(t *testing.T) {
for _, canceled := range []bool{false, true} {
t.Run(fmt.Sprintf("cancel=%t", canceled), func(t *testing.T) {
z, bucket := bulkDeletePools(t, 2)
object := "locked"
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, 1, "latest", ObjectOptions{})
set := z.serverPools[0].getHashedSet(object)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
entered, release := make(chan struct{}, 4), make(chan struct{})
for i, disk := range disks {
disks[i] = bulkDeletePauseDisk{StorageAPI: disk, bucket: bucket, object: object, entered: entered, release: release}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() {
_, errs := z.DeleteObjects(ctx, bucket, []ObjectToDelete{{ObjectV: ObjectV{ObjectName: object}}}, ObjectOptions{})
done <- errs[0]
}()
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("old-copy cleanup did not start")
}
if canceled {
cancel()
if err := <-done; err == nil {
t.Fatal("canceled cleanup succeeded")
}
set.getDisks = original
if got := bulkDeleteBody(t, z, bucket, object, ""); got != "latest" {
t.Errorf("cancellation lost latest: %q", got)
}
return
}
putDone := make(chan error, 1)
reader := mustGetPutObjReader(t, bytes.NewBufferString("after-put"), 9, "", "")
go func() {
_, err := z.PutObject(t.Context(), bucket, object, reader, ObjectOptions{})
putDone <- err
}()
select {
case err := <-putDone:
t.Fatalf("PUT passed the held DELETE lock: %v", err)
case <-time.After(50 * time.Millisecond):
}
close(release)
if err := <-done; err == nil {
t.Error("cleanup failure was lost")
}
if err := <-putDone; err != nil {
t.Fatal(err)
}
if got := bulkDeleteBody(t, z, bucket, object, ""); got != "after-put" {
t.Errorf("concurrent PUT lost: %q", got)
}
})
}
}
type bulkDeleteReplyDisk struct {
StorageAPI
err error
}
func (d bulkDeleteReplyDisk) IsOnline() bool { return true }
func (d bulkDeleteReplyDisk) DeleteVersions(_ context.Context, _ string, versions []FileInfoVersions, _ DeleteOptions) []error {
errs := make([]error, len(versions))
for i := range errs {
errs[i] = d.err
}
return errs
}
func TestBulkDeleteQuorumEvidence(t *testing.T) {
for _, tc := range []struct {
name string
obj ObjectToDelete
opts ObjectOptions
ok bool
}{
{name: "unversioned", ok: true},
{name: "null", obj: ObjectToDelete{ObjectV: ObjectV{VersionID: nullVersionID}}, ok: true},
{name: "uuid", obj: ObjectToDelete{ObjectV: ObjectV{VersionID: mustGetUUID()}}, ok: true},
{name: "marker", opts: ObjectOptions{Versioned: true}},
{name: "suspended-marker", opts: ObjectOptions{VersionSuspended: true}},
{name: "pending-purge", obj: ObjectToDelete{ObjectV: ObjectV{VersionID: mustGetUUID()}, VersionPurgeStatuses: "target=PENDING;"}},
{name: "replicated-marker", obj: ObjectToDelete{DeleteMarkerReplicationStatus: "target=PENDING;"}, opts: ObjectOptions{Versioned: true}},
} {
t.Run(tc.name, func(t *testing.T) {
disks := []StorageAPI{
bulkDeleteReplyDisk{},
bulkDeleteReplyDisk{},
bulkDeleteReplyDisk{err: errFileNotFound},
bulkDeleteReplyDisk{err: errFileVersionNotFound},
}
er := erasureObjects{getDisks: func() []StorageAPI { return disks }}
tc.obj.ObjectName, tc.opts.NoAuditLog = "object", true
_, errs := er.DeleteObjects(t.Context(), "bucket", []ObjectToDelete{tc.obj}, tc.opts)
if (errs[0] == nil) != tc.ok {
t.Errorf("two writes + two absent replies: error=%v, want success=%t", errs[0], tc.ok)
}
})
}
}
@@ -0,0 +1,169 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"fmt"
"testing"
"time"
)
// Both walkers must preserve markers even when all older versions reside in
// another pool. Local version counts cannot prove that a marker is redundant.
func TestPoolsMigrationDeleteMarkers(t *testing.T) {
for _, kind := range []string{"rebalance", "decommission"} {
for _, pools := range []int{2, 3} {
for _, null := range []bool{false, true} {
for _, history := range []string{"none", "source", "destination"} {
t.Run(fmt.Sprintf("%s/pools=%d/null=%t/history=%s", kind, pools, null, history), func(t *testing.T) {
var z *erasureServerPools
var bucket string
if pools == 2 {
z, bucket = migrationTestPools(t)
} else {
z, bucket = migrationTestThreePools(t)
}
const object = "deleted"
old, stamp := UTCNow().Add(-time.Hour), UTCNow().Add(-time.Minute)
var prior ObjectInfo
if history != "none" {
pool := 0
if history == "destination" {
pool = 1
}
prior = putConsistencyObject(t, z, bucket, object, pool, "history", ObjectOptions{Versioned: true, MTime: old})
}
// A stale null copy is a normal consequence of cross-pool overwrites.
putConsistencyObject(t, z, bucket, object, 1, "stale", ObjectOptions{MTime: old})
vid := mustGetUUID()
if null {
vid = nullVersionID
}
opts := ObjectOptions{Versioned: true, VersionID: vid, DeleteMarker: true, MTime: stamp}
marker, err := z.serverPools[0].DeleteObject(t.Context(), bucket, object, opts)
mustIAM(t, err)
if !marker.DeleteMarker {
t.Fatal("fixture did not create marker")
}
if oi, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); !isErrObjectNotFound(err) || !oi.DeleteMarker {
t.Fatalf("fixture visible: %+v %v", oi, err)
}
if failed := migrationTestWalk(t, z, bucket, 0, kind); failed != 0 {
t.Fatalf("failed moves: %d", failed)
}
for attempt := 0; attempt < 2; attempt++ {
if oi, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); !isErrObjectNotFound(err) || !oi.DeleteMarker || !oi.ModTime.Equal(stamp) {
t.Fatalf("marker lost: %+v %v", oi, err)
}
if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: migrationTestVersion(marker)}); !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
t.Fatalf("source marker retained: %v", err)
}
if prior.VersionID != "" && migrationTestBody(t, z.GetObjectNInfo, bucket, object, prior.VersionID) != "history" {
t.Fatal("lost historical version")
}
// Repeating a completed walk must preserve the same state.
migrationTestWalk(t, z, bucket, 0, kind)
}
})
}
}
}
}
}
// A reader opened before a public purge must not recreate the purged version.
func TestPoolsMigrationDoesNotResurrectPurgedSource(t *testing.T) {
for _, kind := range []string{"rebalance", "decommission"} {
for _, multipart := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/multipart=%t", kind, multipart), func(t *testing.T) {
z, bucket := migrationTestPools(t)
move := migrationTestMover(t, z, 0, kind)
oi := migrationTestObject(t, z, bucket, "purged", 0, multipart, []string{"payload"}, ObjectOptions{Versioned: true})
gr := migrationTestReader(t, z, 0, oi)
_, err := z.DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: oi.VersionID, Versioned: true})
mustIAM(t, err)
err = move(t.Context(), 0, bucket, gr)
if err == nil {
t.Error("move accepted a purged source")
}
if oi, err := z.GetObjectInfo(t.Context(), bucket, oi.Name, ObjectOptions{}); !isErrObjectNotFound(err) {
t.Fatalf("purged version resurrected: %+v %v", oi, err)
}
})
}
}
}
// Cleanup must not erase a marker created after the data was copied. It must
// also fail closed if the destination becomes unreadable before source removal.
func TestPoolsMigrationCleanupRechecksSource(t *testing.T) {
for _, change := range []string{"delete-null", "delete-uuid", "unreadable-destination", "missing-destination"} {
t.Run(change, func(t *testing.T) {
z, bucket := migrationTestPools(t)
move := migrationTestMover(t, z, 0, "decommission")
oi := migrationTestObject(t, z, bucket, "cleanup", 0, false, []string{"payload"}, ObjectOptions{})
snapshot := []FileInfo{{VersionID: oi.VersionID, ModTime: oi.ModTime}}
mustIAM(t, move(t.Context(), 0, bucket, migrationTestReader(t, z, 0, oi)))
switch change {
case "delete-null", "delete-uuid":
_, err := z.DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionSuspended: change == "delete-null", Versioned: change == "delete-uuid"})
mustIAM(t, err)
case "unreadable-destination":
restore := conditionalPutSwapDisks(z.serverPools[1], oi.Name, func(d StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: d, bucket: bucket, object: oi.Name}
})
defer restore()
case "missing-destination":
_, err := z.serverPools[1].DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: nullVersionID})
mustIAM(t, err)
}
err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot)
if change != "delete-uuid" && err == nil {
t.Fatal("cleanup accepted changed or uncertain state")
}
if change == "delete-null" || change == "delete-uuid" {
if got, err := z.GetObjectInfo(t.Context(), bucket, oi.Name, ObjectOptions{}); !isErrObjectNotFound(err) || !got.DeleteMarker {
t.Fatalf("cleanup resurrected data: %v", err)
}
} else if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, oi.Name, ObjectOptions{}); err != nil {
t.Fatalf("cleanup lost source: %v", err)
}
})
}
}
func TestPoolsMigrationConflictingTimestampTie(t *testing.T) {
for _, kind := range []string{"rebalance", "decommission"} {
for source := range 2 {
t.Run(fmt.Sprintf("%s/source=%d", kind, source), func(t *testing.T) {
z, bucket := migrationTestPools(t)
stamp := UTCNow().Add(-time.Minute)
for i := range 2 {
putConsistencyObject(t, z, bucket, "tie", i, fmt.Sprintf("pool-%d", i), ObjectOptions{MTime: stamp})
}
if got := migrationTestBody(t, z.GetObjectNInfo, bucket, "tie", ""); got != "pool-0" {
t.Fatalf("fixture winner: %s", got)
}
migrationTestWalk(t, z, bucket, source, kind)
if got := migrationTestBody(t, z.GetObjectNInfo, bucket, "tie", ""); got != "pool-0" {
t.Fatalf("changed tie winner: %s", got)
}
})
}
}
}
+33
View File
@@ -0,0 +1,33 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import xhttp "github.com/minio/minio/internal/http"
// migrationObjectMetadata restores tags removed from the read representation.
// Keep the original revision, including ordered empty states, for the existing
// locked reconciliation at PUT or multipart completion. Moving is not a new
// tagging mutation, and the write must not modify the reader's metadata map.
func migrationObjectMetadata(oi ObjectInfo) map[string]string {
metadata := cloneMSS(oi.UserDefined)
delete(metadata, xhttp.AmzObjectTagging)
if oi.UserTags != "" {
metadata[xhttp.AmzObjectTagging] = oi.UserTags
}
return metadata
}

Some files were not shown because too many files have changed in this diff Show More