Compare commits

...

67 Commits

Author SHA1 Message Date
Feng Ruohang 71a1631f13 fix: reject object lock preflight read failures
Return storage read failures before accepting an object overwrite. Keep the null-version and multipart target checks under the destination write lock, and add slow-body, read-fault, legacy-upload and recovery-collision regressions to the Object Lock race gate.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 23:28:36 +08:00
Feng Ruohang 14042b5aab Merge pull request #234 from pgsty/codex/fix-retention-write-bypasses
fix: enforce retention authorization and immutable version writes

Include Go 1.27.2 security fixes and a compatible linter.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 16:02:58 +08:00
Feng Ruohang 38a5ab549c build: use golangci-lint compatible with Go 1.27.2
Pin golangci-lint v2.14.0, whose x/tools v0.50.0 decoder supports the
version 5 export data emitted by Go 1.27.2. The previous v2.13.1 binary
failed to import internal/cpu in Linux CI before running the checks.
Keep the existing linter selection and configuration.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 15:48:51 +08:00
Feng Ruohang fa565ec9fc build: update Go to 1.27.2 and x/net to 0.60.0
Use the October security fixes for reachable standard-library and HTTP/2
vulnerabilities found by govulncheck. Align the module requirement, build
dependency check and digest-pinned Docker build stages on Go 1.27.2.
Update only golang.org/x/net and its checksums; module tidying and credit
regeneration introduce no other dependency or license changes.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 15:39:25 +08:00
Feng Ruohang 3196480da9 fix: enforce retention authorization and immutable version writes
Close SN-2026-015 by requiring PutObjectRetention with request-derived
condition keys before reading bucket or object state. Weakening unexpired
GOVERNANCE retention also requires the bypass header and permission. Treat
retention removal as zero remaining days and avoid duration saturation for
distant dates.

Close SN-2026-016 by requiring replication trust for caller-selected target
version IDs and checking persisted multipart targets and locked null
versions under the destination write lock, including across storage pools.
Preserve versioning transitions, trusted replication and batch Snowball
version IDs, and fail closed when lock state cannot be read.

Add authorization, concurrent lock-update, multipart transition, cross-pool,
configuration-failure and retention-boundary regression coverage. Validate
with make test, focused race tests and the original independent probes.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit fe3935476b929e03193d7496a7448d760e635096)
2026-10-14 15:25:19 +08:00
Feng Ruohang bcb5fc9566 Merge pull request #225 from cbornet/error-messages
Align some checksum error messages with S3

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-26 09:03:50 +08:00
cbornet 394d9db212 Align some checksum error messages with S3
Signed-off-by: cbornet <cbornet@hotmail.com>
2026-09-25 17:44:52 +02:00
Feng Ruohang 69d5e92791 Merge pull request #220 from AleksaMCode/fix/lambda-targetidset-isempty
Fix `TargetIDSet.IsEmpty` and add regression test
2026-09-23 08:07:09 +08:00
AleksaMCode efd5698597 lambda/event: fix TargetIDSet.IsEmpty and add regression test
Signed-off-by: AleksaMCode <aleksamcode@gmail.com>
2026-09-17 22:06:44 +02:00
Feng Ruohang 2fde3cf535 docs: update ec description 2026-09-17 09:42:26 +08:00
Feng Ruohang 2a4d51406b test: build the purge-response fixture ARN at runtime
The rebrand compatibility guard tracks every literal policy value that
names the upstream brand. Construct the replication ARN the way the
other replication fixtures do instead of adding a new literal.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 23:50:09 +08:00
Feng Ruohang 0596685ae7 docs: record the delete-marker purge, listing quorum and migration tag repairs
Describe the three correctness repairs merged after the 20260916 candidate
was prepared, together with their known remaining limits (#217, #218).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 23:44:44 +08:00
Feng Ruohang 358ab38fb0 fix: report a purged version as a delete marker only for stored markers
The exact-version purge path copies the looked-up DeleteMarker flag
into the DELETE response so that removing a delete marker keeps its
x-amz-delete-marker header. That flag is also set for a data version
whose purge is pending, because the lookup exposes such a version as
deleted for visibility. Retrying the purge after the replication
configuration was removed therefore answered with
x-amz-delete-marker: true and raised ObjectRemovedDeleteMarkerCreated
for a data version.

Only a stored marker carries no erasure layout; use that to decide the
response identity. Found by the adversarial review of the purge change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 22:53:12 +08:00
Feng Ruohang 254b19ac07 fix(replication): recheck queued delete-marker creations under the lock
A delete-marker creation task carries the marker as it looked when it
was queued by the DELETE handler, a GET/HEAD/LIST heal, the scanner or
MRF. Tasks from several frontends serialize on the per-object
replication lock, so a task can run after the user has purged that
marker and the purge has already reached the targets. The target no
longer holds the marker, so the queued creation recreated it there with
the original VersionID and mtime. This is the late-create sequence seen
in the 2026-09-16 three-site runs: purge 204 on both targets, then about
85 ms later a replicated creation for the same VersionID.

Re-read the source version under the replication lock before sending a
creation. A missing version, a non-marker version, or a version under
purge makes the task stale; it is dropped without touching the source.
A read that cannot confirm either way is retried through MRF instead of
being treated as absence.

Creations already on the wire, replays from other sites, and cleanup of
minority residue after a crash are not covered by this check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 22:35:08 +08:00
Feng Ruohang eb4f5e5b31 Fix exact-version purges and delete-marker metadata healing
Require write-quorum absence for missing purge retries, aggregate removed and already-absent votes only for physical purges, and resolve receiver purges by version across pools. Preserve marker metadata through creation and healing while retaining DELETE response semantics.

Add real-disk quorum, pool, callback, metadata, heal and outbound replication regressions. Synchronize capacity fixture installation and restoration with concurrent IAM readers.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 22ba4d426677aa07470927fc349a43afd87e19b6)
2026-09-16 22:24:27 +08:00
Feng Ruohang 8d06424b12 fix: retain quorate null versions after newer minorities
Recount only when the existing selection is below quorum and each original
nonempty stream has one ordinary null version with identical EC headers.
Reuse the existing header grouping and leave history pruning unchanged.

Add deterministic signed LIST, complete-metadata permutation, excluded-history,
object-read, scanner/heal, Walk and migration-consumer regressions.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 9804e4deb4d5a18cca640be72bff9b47a415f7de)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 21:50:03 +08:00
Feng Ruohang fced863036 fix(storage): preserve tags during pool migration
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 20:22:37 +08:00
Feng Ruohang 5af0865aab Merge pull request #214 from pgsty/codex/release-prep-20260916
Prepare the SILO 20260916000000 source candidate with released component pins, CPU metrics synchronization, regression coverage and release tooling updates.

All 11 checks passed on e7963381ba. Signed release artifacts, image publication and deployment remain separate deliverables tracked by #203.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 19:30:07 +08:00
Feng Ruohang 83821f0f1f Merge pull request #216 from pgsty/codex/docs-published-design-links-20260916
docs: link retained guides to published design records
2026-09-16 18:50:28 +08:00
Feng Ruohang bf053386a4 docs: link compression and federation to maintained design records
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 18:26:03 +08:00
Feng Ruohang e7963381ba test: synchronize tag replication capacity fixtures
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:37:47 +08:00
Feng Ruohang 37c0edc7ca test: use canonical octal permissions in LZ4 fixture
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:33:58 +08:00
Feng Ruohang a2fe70424e release: prepare SILO 20260916000000 candidate
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:30:29 +08:00
Feng Ruohang 027d43b4eb fix: synchronize CPU resource metrics reads
Hold resourceMetricsMapMu.RLock while the v3 CPU collector reads both the
subsystem map and its nested idle/iowait metrics. The unsynchronized reads
were inherited from upstream commit f7b665347 (minio/minio#19560).

Add value and concurrent Prometheus Gather regression tests, run them in
the targeted race CI job, and record the existing CPU metric names in the
compatibility inventory.

Fixes #210

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 17:09:19 +08:00
Feng Ruohang f99ed829b5 Merge pull request #213 from pgsty/codex/pr198-release-compat
fix: preserve multipart defaults and rollback compatibility
2026-09-16 16:42:56 +08:00
Feng Ruohang 956a1a8e33 Merge pull request #212 from pgsty/codex/docs-migration-cleanup
docs: consolidate repository entry points and retire migrated work records
2026-09-16 16:39:40 +08:00
Feng Ruohang 82f0a9828e fix: preserve multipart defaults and rollback compatibility
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:30:43 +08:00
Feng Ruohang 584b5a3a8f docs: link the canonical recovery runbooks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:19:56 +08:00
Feng Ruohang 39f49d548f Merge remote-tracking branch 'origin/main' into codex/docs-migration-cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>

# Conflicts:
#	CONTRIBUTORS.md
2026-09-16 16:13:31 +08:00
Feng Ruohang a168576adb Merge pull request #208 from pgsty/codex/conditional-put-release-notes
docs: describe conditional PUT behavior and tested recovery guidance
2026-09-16 16:09:22 +08:00
Feng Ruohang e85c4c8dfb docs: preserve reviewed contributor records and daily card references
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 16:09:00 +08:00
Feng Ruohang 0e3c43778e Merge pull request #211 from pgsty/codex/daily-repository-cards
ci: update README repository cards daily at 00:00 UTC
2026-09-16 15:40:09 +08:00
Feng Ruohang 9101fe78db ci: refresh README repository cards daily at 00:00 UTC
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 15:27:48 +08:00
Feng Ruohang 82948d6306 Merge pull request #198 from mrjavadseydi/fix/issue-79-list-multipart-uploads
Retain the original contributor commit and integrate the reviewed durable listing, cancellation confirmation and migration fixes. Capacity acceptance and delayed creation-write fencing remain open in issue #79.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:40:10 +08:00
Feng Ruohang dfb4b2a1d2 fix: return HTTP 503 when multipart scan capacity is exhausted
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:24:52 +08:00
Feng Ruohang 143f6970d8 fix: make multipart discovery and cancellation explicit and bounded
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 13:16:37 +08:00
Feng Ruohang ea5dac5a99 Merge current main into multipart listing contribution
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 12:51:20 +08:00
Feng Ruohang 3c26a8b0b5 Merge pull request #209 from pgsty/codex/credit-console-share-reporter
fix: integrate the bounded Console sharing proxy and credit its reporter
2026-09-16 12:19:29 +08:00
Feng Ruohang 2fabd436c0 fix: select the bounded Console sharing proxy
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 12:05:00 +08:00
Feng Ruohang 07c68d6054 docs: credit Jiri Pejchal for the Console sharing report
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:48:38 +08:00
Feng Ruohang 15090dc4fd docs: retire migrated working material and clarify documentation ownership
Remove migrated investigation and security documents, repair their incoming links, and align the English/Chinese READMEs with current module and release boundaries. Track AGENTS.md as the shared repository guide and make CLAUDE.md import it; keep working artifacts outside the repository.

Publish pgsty/silo.pgsty.com commit c7682185e2832b981629e1c17aef74fc778c6ca1 before publishing this cleanup: the new documentation routes are not live yet.

Validation: make rebrand-guard; 92 Markdown files checked for deletion-induced broken relative paths; 199 source-to-site references and anchors resolve in the paired site build; git diff --check.
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:22:48 +08:00
Feng Ruohang e791640dac docs: describe merged conditional PUT behavior and recovery guidance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:21:16 +08:00
Feng Ruohang 9b4ae82a29 Merge pull request #207 from pgsty/codex/conditional-put-pools
fix(pools): evaluate cross-pool PUT conditions against the current object
2026-09-16 11:17:02 +08:00
Feng Ruohang 4620be394b test: synchronize conditional PUT disk fixtures
Replace unsynchronized getDisks swaps with backing disk-list updates under
erasureDisksMu, matching the existing GetDisks reader lock. Apply the same
helper to capacity and read-fault adapters while preserving nested restore
ordering.

Add a regression that overlaps fixture changes with the real IAM Walk
reader, and run the conditional PUT suite under the race detector in CI.
The regression reproduces the old fixture race; ten fixed race iterations
pass without warnings. Production conditional PUT behavior is unchanged.

Refs #199

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 10:53:24 +08:00
Feng Ruohang 5e7d603083 fix(pools): evaluate cross-pool PUT conditions against the current object
Multi-pool PUT selected a destination by capacity and evaluated
If-Match/If-None-Match only against that destination's local object
state. An empty or stale destination could accept a stale ETag or
If-None-Match:* while another pool held the current object, replacing
it; a current ETag could instead be rejected with 412 or 404.

Under PUT's existing pools-layer object lock, resolve the comparison
object with objectPoolInfos (including draining pools), treat a latest
delete marker as absence, fail closed on unreadable pool metadata, and
clear an accepted callback before destination dispatch. Replica and
data-movement callbacks keep their addressed-version semantics and
metadata reconciliation.

Reproduced on 40220bd836 and RELEASE.2026-09-03T13-18-01Z with six
signed HTTP scenarios: four defect cases failed, two controls passed.

Refs #199

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:21:21 +08:00
Feng Ruohang fb7c406ddc Merge pull request #206 from pgsty/codex/main-consolidation-20260916
test: restore valid credentials in integration fixtures
2026-09-16 08:18:56 +08:00
Feng Ruohang 416826f61f Merge pull request #205 from pgsty/codex/release-notes-followups
docs: complete September correctness and upgrade notes
2026-09-16 08:16:23 +08:00
Feng Ruohang a2e2f3ee82 test: restore valid credentials in integration fixtures
Port the shell-fixture changes from ebc9937d97b27871dcc4bb91d4b5771d3550b76a. Match the existing minimum secret length consistently across startup, aliases and helper commands.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:08:08 +08:00
Feng Ruohang 70c7ec4a9f docs: link reviewed runbook sources before website deployment
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:04:53 +08:00
Feng Ruohang 2b722b7e92 docs: complete September correctness and upgrade notes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 07:51:11 +08:00
mr javad seydi 4cbb074ccd fix: make multipart upload listing S3-compatible
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
2026-09-15 23:17:55 +03:30
Feng Ruohang 40220bd836 Merge pull request #196 from pgsty/codex/merge-r4-r8
Complete R5, R6 and R8 on the main baseline containing R4 and R7.

Preserve the individual signed repairs, independent Opus 5 Max review, and integration validation evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 01:14:06 +08:00
Feng Ruohang df0dfa0a34 docs: record R4-R8 integration review and validation
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:58:31 +08:00
Feng Ruohang 80684fed59 chore: align integrated repair tests with contribution checks
Use the actual author and AGPL-3.0-or-later notices for new R6/R8 tests, preserving all test bodies and the Linux build tag. Record the existing replication ARN prefix used by the new R6 fixture in the compatibility inventory; no wire behavior changes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:44:34 +08:00
Feng Ruohang 055030ea53 fix(http): honor configured request header deadlines
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 0d48d32d7e038ae1ea5966f3d7e0cb86780a6311)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:56 +08:00
Feng Ruohang aea3882c95 docs: record R6 integration verification
(cherry picked from commit d38edb2c46182d3a8fa96e040604493d20a4b478)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:56 +08:00
Feng Ruohang 0c61128d23 fix(replication): retry marker purges through persisted MRF
(cherry picked from commit cf381a7151ef25fc95ace5fedcd767fa19410de2)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:55 +08:00
Feng Ruohang 680eac66e4 fix(replication): preserve ordered tag deletions
Persist, send and reconcile empty tag states together with their revision
across COPY, PUT and multipart replication. Advance local tag mutations
under the existing locks and preserve current tags during replication ACK.

Cover signed HTTP, persistent single/multiple pool state, KMS, SSE-C key
rotation, ordering, retry and duplicate requests. Record real Opus plan
consensus, implementation review and local verification evidence.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 115fe8b12329d147adbaf817faa1737392ecbf9b)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:37:55 +08:00
Feng Ruohang 9f3037e941 Merge pull request #194 from pgsty/codex/r7-replication-content-encoding
fix(replication): preserve normalized replica metadata
2026-09-16 00:31:03 +08:00
Feng Ruohang 5f00f6762f Merge main after R4 validation into R7 candidate
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:21:21 +08:00
Feng Ruohang af2b1794d3 Merge pull request #193 from pgsty/codex/r4-kms-tag-timestamp
fix(replication): preserve SSE-KMS tag timestamps
2026-09-16 00:20:10 +08:00
Feng Ruohang d371f77dcc docs: record final Opus 5 Max R7 implementation review
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:10:31 +08:00
Feng Ruohang 022722a7a7 chore: align R4 contribution notices and record final review
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:08:09 +08:00
Feng Ruohang 03027727d1 fix(replication): preserve SSE-KMS tag timestamps
Carry the already-parsed source tagging timestamp through the KMS options
constructor so replica COPY can apply newer tag updates on explicitly or
automatically encrypted destinations.

Cover all option encryption modes and signed COPY persistence for newer,
stale, duplicate and timestamp-less updates, including bucket defaults.
Preserve the real Opus 5.0/max plan review, consensus and local validation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:03:52 +08:00
Feng Ruohang 4fcdf37ce6 fix(replication): preserve normalized replica metadata
Restore only the six replication-specific metadata fields after trust
validation, so streaming uploads retain their actual content encoding and
Snowball entries do not inherit ordinary metadata from the outer archive.

Include helper, authenticated PUT/COPY/multipart and Snowball regressions,
plus the R7 investigation, actual Opus 5 consensus and local verification.
The production change is based on PR #187 by Mikhail Khadarenka.

Co-authored-by: Mikhail Khadarenka <chodorenko@gmail.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 00:00:17 +08:00
Feng Ruohang 9ebe81c1b3 Merge pull request #192 from pgsty/codex/iam-revision-tombstones
fix(iam): retain revocation versions through replay and recovery
2026-09-15 23:14:22 +08:00
Feng Ruohang e5f5c9e7f6 Merge pull request #191 from pgsty/codex/iam-peer-delete-reload
fix(iam): reload committed state on peer deletion notifications
2026-09-15 23:10:38 +08:00
211 changed files with 12674 additions and 16306 deletions
+17
View File
@@ -90,6 +90,23 @@ jobs:
- name: Run S3 Select tests under race detector
run: go test -race ./internal/s3select/... -count=1
- name: Run conditional PUT tests under race detector
run: go test -race ./cmd -run '^Test(PoolsConditionalPut|SinglePoolConditionalPutHTTP)' -count=1 -timeout=5m
- name: Run multipart listing and cancellation tests under race detector
run: go test -race ./cmd -run '^Test(MultipartListing|MultipartAbort|PaginateMultipartUploads|ListMultipartUploads)' -count=1 -timeout=5m
- name: Run CPU metrics tests under race detector
run: go test -race ./cmd -run '^TestLoadCPUMetrics' -count=1 -timeout=5m
- name: Run tag replication tests under race detector
run: go test -race ./cmd -run '^TestAPITagging' -count=1 -timeout=5m
- name: Run Object Lock authorization and overwrite tests under race detector
env:
MINIO_API_REQUESTS_MAX: "10000"
run: go test -race -tags kqueue,dev ./cmd -run '^Test(APIPutObjectRetention|APIVersionAddressedWrite|PutObjectRetention|DirectoryObjectLock|ObjectLockOverwrite|APICopyObjectPreservesVersionAfterSuspension|MultipartNullObjectLock|PoolsObjectLockOverwrite|MultipartVersioningTransitions|PlanReview|RecoveredMultipartUpload)' -count=1 -timeout=5m
crosscompile:
name: Cross Compile
runs-on: ubuntu-latest
+2 -2
View File
@@ -132,9 +132,9 @@ jobs:
cosign-release: v3.1.2
- name: Build Draft release with GoReleaser
uses: goreleaser/goreleaser-action@v7
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
with:
version: "~> v2"
version: v2.18.1
args: release --clean --skip=validate --config .github/goreleaser.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+100
View File
@@ -0,0 +1,100 @@
name: Repository Cards
on:
schedule:
# 00:00 UTC = 08:00 Asia/Shanghai. GitHub may queue scheduled runs.
- cron: "0 0 * * *"
workflow_dispatch:
push:
branches: [main]
paths:
- .github/workflows/repository-cards.yml
- .github/silo.svg
- buildscripts/repository-cards/**
pull_request:
branches: [main]
paths:
- .github/workflows/repository-cards.yml
- .github/silo.svg
- buildscripts/repository-cards/**
permissions:
contents: read
concurrency:
group: repository-cards-${{ github.event.pull_request.number || 'publish' }}
cancel-in-progress: false
jobs:
check:
name: Validate repository cards
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.13"
- run: python -m pip install -r buildscripts/repository-cards/requirements.txt
- run: python -m unittest discover -s buildscripts/repository-cards -p 'test_*.py' -v
publish:
name: Update README images
needs: check
if: github.repository == 'pgsty/silo' && github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write
issues: read
pull-requests: read
env:
OUTPUT_BRANCH: codex/repository-cards
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.13"
- run: python -m pip install -r buildscripts/repository-cards/requirements.txt
- name: Load the generated-assets branch
shell: bash
run: |
set -euo pipefail
artifacts="$RUNNER_TEMP/repository-cards"
if git ls-remote --exit-code --heads origin "$OUTPUT_BRANCH"; then
git fetch --depth=1 origin "$OUTPUT_BRANCH"
git worktree add --detach "$artifacts" FETCH_HEAD
else
status=$?
# Exit 2 means no matching ref; transport/auth failures must stop.
if [ "$status" -ne 2 ]; then exit "$status"; fi
git worktree add --detach "$artifacts" HEAD
git -C "$artifacts" checkout --orphan "$OUTPUT_BRANCH"
git -C "$artifacts" rm -rf .
fi
- name: Refresh contributor and star cards
env:
GH_TOKEN: ${{ github.token }}
run: python buildscripts/repository-cards/update.py --output "$RUNNER_TEMP/repository-cards"
- name: Publish changed assets
shell: bash
run: |
set -euo pipefail
cd "$RUNNER_TEMP/repository-cards"
git add README.md history.json curated.json contributors.json \
contributors-light.svg contributors-dark.svg \
star-history-light.svg star-history-dark.svg
if git diff --cached --quiet; then
echo "Repository cards are already current."
exit 0
fi
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git commit -s -m "chore: update repository cards $(date -u +%F)"
# A normal push preserves history and refuses concurrent overwrites.
git push origin "HEAD:refs/heads/$OUTPUT_BRANCH"
+6 -4
View File
@@ -4,6 +4,8 @@ on:
workflow_dispatch:
pull_request:
paths:
- "go.mod"
- "go.sum"
- ".github/goreleaser.yml"
- ".github/nfpm.yml"
- "Dockerfile.goreleaser"
@@ -93,9 +95,9 @@ jobs:
echo "LDFLAGS: ${LDFLAGS}"
- name: GoReleaser config check
uses: goreleaser/goreleaser-action@v7
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
with:
version: "~> v2"
version: v2.18.1
args: check --config .github/goreleaser.yml
- name: Validate Helm chart and legacy upgrade identity
@@ -107,9 +109,9 @@ jobs:
syft-version: v1.50.0
- name: Build snapshot artifacts
uses: goreleaser/goreleaser-action@v7
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
with:
version: "~> v2"
version: v2.18.1
# A pull-request snapshot has no trusted release identity. Exercise
# the SBOM/checksum pipeline here, and reserve keyless signing for
# the tag-triggered release workflow with GitHub OIDC.
+2 -2
View File
@@ -62,10 +62,10 @@ dist/
.claude/
.codex/
AGENTS.md
CLAUDE.md
_bmad/
_bmad-output/
# Investigation and editorial working files belong outside this repository.
docs/investigations/
docs/security/
docs/rebranding.md
.release-sign/
+84
View File
@@ -0,0 +1,84 @@
# SILO Repository Guide
## Project map
- Server: `pgsty/silo` (this checkout, default branch `main`)
- Console: `pgsty/silo-console` (usual sibling checkout `../silo-console`)
- Client: `pgsty/mc`, shipped as `mcli` (usual sibling checkout `../mc`)
- Shared packages: `pgsty/silo-pkg` (usual sibling checkout `../silo-pkg`)
- Documentation: `pgsty/silo.pgsty.com` (usual sibling checkout `../silo.pgsty.com`),
published at <https://silo.pgsty.com/>
The server executable, package, systemd service, and container image use `silo`;
the public Docker image is `docker.io/pgsty/silo`. The server module remains
`github.com/minio/minio`. Consult the current `go.mod` and release configuration
for selected component versions and compatibility replacements.
## Supported stack and compatibility policy
The maintained, release-gating product graph is the coordinated PGSTY stack:
`silo` + `silo-console` + `mc` + `silo-pkg`.
Keep compatibility with upstream MinIO/MC on a best-effort basis. Preserve
inexpensive wire, configuration, CLI, migration, and import compatibility when
it helps users, and document known differences. Do not infer from source
lineage, MinIO-compatible protocols, retained `MINIO_*`/`MC_*` names, or an
inherited upstream test that unmodified upstream MinIO is a supported release
target.
Upstream-only compatibility checks may remain as advisory evidence, but they
must not force a downgrade of a maintained SILO component, a fork-only API
workaround, or a release block. Make upstream compatibility a hard gate only
when the user explicitly requests that scope.
## Dependency selection
- When PGSTY maintains a component, import and require it directly under its own
module path. In particular, prefer `github.com/pgsty/silo-pkg/v3` over
`github.com/minio/pkg/v3` in maintained SILO source.
- Do not use an upstream package merely to make unmodified upstream MinIO or MC
compile. Unavoidable transitive upstream modules should be documented and
kept separate from the maintained product dependency.
- `github.com/minio/minio-go/v7` is the explicit exception: use the verified
upstream module/commit while it contains the required fixes; do not recreate
a SILO fork without a concrete functional divergence.
- Check the current `go.mod` before changing versions. Coordinate breaking
import-path changes across package, client, Console, and server releases.
## Documentation and delivery
The companion site owns product, operations, migration, security-advisory,
design, and release documentation. Maintain English and Chinese content in
that repository and link to its canonical URLs from this one. The site's
`content/docs/` is the documentation entry point; detailed pages live under
`content/operations/`, `administration/`, `reference/`, `compatibility/`,
`about/`, and `blog/`, following the existing structure.
Keep repository entry points and contributor instructions here. Retained
upstream documents, examples, and test fixtures under `docs/` may be updated
when a code or tooling change requires it; do not build a second documentation
site in this tree.
Investigation plans, prompts, AI session transcripts, execution logs, temporary
reports, and private security material belong outside the checkout, for example
in a task-specific directory under `~/tmp/`. Do not recreate
`docs/investigations/`, `docs/security/`, or `docs/rebranding.md`, or move their
working material to another repository directory. Extract reusable, verified
knowledge into the companion site; keep private evidence outside public Git.
Compatibility pages may continue to describe similarities with upstream, but
must label that compatibility as best effort. The supported and tested server
for Console and mcli administration is `pgsty/silo`.
Before removing or moving documentation, check repository links and scripts,
the companion site's links and anchors, and references from this guide. Use
fixed commit URLs for historical evidence that should survive deletion from
the current tree. Run the site's `make check` for site changes and this
repository's `make rebrand-guard` for documentation cleanup that changes the
identifier inventory. When new public URLs are involved, publish the site
before publishing source changes that depend on those URLs.
Keep changes in the repository that owns the affected surface. Treat every
repository commit, tag, release, image, documentation update, and deployment as
a separate deliverable. Follow `CONTRIBUTING.md`, including DCO sign-off
(`git commit -s`). `CLAUDE.md` imports this guide so both agents use one policy.
+155 -21
View File
@@ -2,16 +2,48 @@
## Unreleased
The entries below describe source changes on main since the latest published Server.
Preparation target: `RELEASE.2026-09-16T00-00-00Z` (package version
`20260916000000.0.0`). The entries below describe the candidate changes since
the latest published Server.
**The latest published Server remains 20260903.** These changes are not in its
binaries, packages or images. See the [component matrix](https://silo.pgsty.com/compatibility/versions/)
and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-09-03T13-18-01Z...main).
### Authorization and security
- Synchronize CPU metrics reads with resource-metrics updates (#210), preventing
concurrent map access from terminating the server during Prometheus scraping.
Metric names, values and authentication requirements are unchanged.
- Restrict embedded Console's anonymous sharing proxy to object-content GETs
at the configured S3 origin, and reject every redirect. Internal metrics,
system paths and non-download S3 operations cannot be reached through it.
Normal public, presigned and versioned downloads remain available without a
new setting; a full sharing-disable switch is not introduced. See
[Console #56](https://github.com/pgsty/silo-console/pull/56) and the
[design record](https://github.com/pgsty/silo-console/issues/52).
Thanks to Jiri Pejchal (@jiri-pejchal) for the report.
- Persist IAM deletion revisions and parent revocation boundaries so stale site
events cannot restore deleted identities, policies or their older grants
(#191, #192). Peer deletion notifications reload committed storage; deliberate
recreation requires a newer revision, and credentials issued before the
parent's revocation remain invalid.
**Coordinated upgrade required:** upgrade every participating node and site.
Mixed old/new nodes sharing an IAM backend and rolling downgrade are
unsupported. Back up complete IAM storage and encryption material; an admin
export of live records omits deletion history. Reissue credentials for
recreated parents and explicitly reconcile pre-upgrade revocations whose
history is already lost. Restoring an older backup can lose later revocations;
keep affected sites isolated until reconciliation/rekeying is complete. See
[the operator runbook](https://silo.pgsty.com/operations/replication/iam-upgrade/).
- Enforce an absolute HTTP/1 request-header deadline through the connection
wrapper (#196). Repeated small reads no longer extend that deadline, and
`--read-header-timeout` / `MINIO_READ_HEADER_TIMEOUT` now reaches the HTTP
server. HTTP/1 request bodies retain the rolling idle timeout; this does not
impose a total upload/download duration. A shorter setting also constrains
TLS handshake reads. The wrapper's strict header mode is not applied to HTTP/2.
- Reject unsigned `x-amz-*` request headers that could turn a signed PUT into a
copy of another object accessible to the signer (SN-2026-011). The latest
public Server is affected; the fix is on main. See [the advisory ledger](docs/security/advisories.md).
public Server is affected; the fix is on main. See [the advisory ledger](https://silo.pgsty.com/about/security-advisories/).
- Align signed request fields with policy conditions and enforce header-only
presigned payload checksums. See [the signed-header review](https://silo.pgsty.com/blog/design/signed-header-coverage/).
- **Breaking policy semantics:** separate self-service `admin:ChangeMyPassword`
@@ -22,17 +54,114 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
### Object storage and replication
- Make `ListMultipartUploads` discover quorum-valid uploads from durable state
across pools, erasure sets and drives, then apply S3 prefix, delimiter,
marker, ordering and 1,000-entry pagination semantics globally (#198). New uploads
store their canonical bucket and key as reserved fields in the existing
quorum-written `xl.meta`; completion removes those upload-only fields. Native
markers remain usable after their upload is completed or canceled. Strict
listing returns a diagnostic 503 for legacy uploads or uncertain coverage;
the default remains the released exact-key/cache-based `legacy` behavior.
Opt into strict mode only through `MINIO_API_MULTIPART_LISTING=strict`, after
upgrading every writer, draining old uploads, checking the read-only admin
`multipart-preflight` report and validating scan capacity. The process-only
setting is not persisted into shared API configuration. Historical
`multipart_listing` keys are ignored and can be removed with a targeted
`mcli admin config reset ALIAS api multipart_listing` before rollback. Per-process
admission, directory-entry, worker and time budgets bound scan scheduling;
each page still scans durable state. See [issue #79](https://github.com/pgsty/silo/issues/79)
and its [design record](https://silo.pgsty.com/blog/design/list-multipart-uploads/).
Thanks to mr javad seydi (@mrjavadseydi) for the original implementation.
- Retain released read-quorum and best-effort multipart cancellation in default
legacy mode. Strict mode requires majority deletion acknowledgements and
permits retries below read quorum. When most drives were already empty,
failed deletion of an observed remnant now returns 503 instead of being
masked by empty-drive successes. Wrong-key or wrong-bucket cancellation
preserves the valid upload's cache entry; successful cancellation notifies
peers with the request context after the distributed lock is released.
The HTTP response for an absent
upload remains 204; this is not proof of physical cleanup. **Known boundary:**
delayed creation writes can still restore an upload after cancellation;
this change does not add a durable creation fence.
- Preserve object tags during multi-pool metadata reconciliation by reading the
resolved tag field together with its revision (#189). Previously, reconciliation
could replace existing tags with an empty value.
- Preserve the tag revision on SSE-KMS metadata replication (#193), and advance
tag revisions monotonically on local PUT/DELETE tagging (#196). Empty tags
participate in reconciliation as an ordered deletion, preventing older
events from restoring removed tags. SSE-C key rotation also retains the tag
revision. Malformed historical revisions can fail and retry; their missing
history is not reconstructed by the upgrade.
- Complete delete-marker version purges and preserve their identity and retry
state through MRF recovery (#196). Recovery accepts a 405 marker response only
when its version, bucket, object name and modification time match the task.
Purge audit status is normalized from `COMPLETE` to `COMPLETED`.
Thanks to Julien Laurenceau (@julienlau) for the investigation and proposed
fix in #184 that helped shape this follow-up.
- Restore only the six replication-specific metadata fields after ordinary
request metadata extraction (#194). This prevents transport-only `aws-chunked`
from being stored as Content-Encoding while preserving the signed-header
protections. Trusted Snowball entries no longer inherit the outer archive's
ordinary metadata. Thanks to Mikhail Khadarenka (@chodorenko) for the fix in #187.
**Existing data:** these repairs prevent new errors; they do not scan or rewrite
historical object metadata, recover lost tags or prove that old purge work has
converged. Follow the [read-only audit procedure](https://silo.pgsty.com/operations/replication/replica-metadata-audit/)
before planning any repair of stored state.
- Make exact-version delete-marker purges converge in replicated buckets
(`eb4f5e5b3`, `254b19ac0`, `358ab38fb`). A purge no longer creates the
marker on drives that lacked it; a retried purge of a missing version is
acknowledged only when a write-quorum majority of drives report it absent;
purge results merge removed and reliably absent replies; healing a marker
preserves its stored replication and purge metadata; and a queued marker
creation is re-checked against the source under the replication lock before
it is sent, so a purge that already reached the targets is not undone by a
stale task from another frontend, a GET/LIST heal, the scanner or MRF.
Purging a data version whose earlier purge is still pending reports it as a
data version. **Known limitations:** creations already in flight or replayed
from another site, and minority marker copies left by a crash after a
majority-acknowledged purge, are tracked in #217. See
[the replication reliability record](https://silo.pgsty.com/blog/design/replication-reliability/).
- Keep a null object version that has listing quorum when a newer minority of
drives sorts first (`8d06424b1`). The resolver recounts per header only when
the original selection lacks quorum, every non-empty drive stream holds
exactly one ordinary null version and all share the same erasure layout;
mixed histories keep their previous behavior. **Known limitation:** a
successful ListObjects can still omit readable keys during rolling restarts
with concurrent overwrites (#218). Do not run destination-deleting sync tools
against a listing taken during a rolling restart; list again once the
cluster is stable.
- Carry object tags through rebalance and decommission for ordinary and
multipart writes (`fced86303`). Both migration entry points restore the tags
and their revision fields when rewriting the object in the destination pool.
Tags dropped by earlier migrations are not recovered; audit tag-dependent
lifecycle and policy rules for pools migrated with an older build.
- Evaluate conditional multipart completion against the logical current object
across all pools while holding the existing object lock. A stale `If-Match`
can no longer replace newer data in another pool, and the current ETag is no
longer rejected because the upload resides next to an older copy. Conditions
are evaluated once; a current delete marker counts as an absent object.
**Availability change:** if metadata cannot be read from any pool, conditional
**Availability change:** if any pool's metadata cannot be read, conditional
completion fails even when another pool can still serve GET/HEAD. This also
applies when the unreadable pool may not hold the object: absence cannot be
verified. Retry after the pool recovers. Unconditional completion and the
single-pool path retain their existing behavior.
- Evaluate ordinary multi-pool conditional PUT against the logical current
object across all pools, including draining pools, under the existing object
lock (#207). A stale destination copy no longer accepts a stale ETag or rejects
the current one; a current delete marker is treated as absence.
**Availability change:** if any pool's object metadata cannot be verified,
the condition fails even when GET can use another pool; read-quorum failures
return 503. Restore readability or heal before retrying. Unconditional PUT,
single-pool conditions and internal replication retain their existing behavior.
A public condition with a destination `versionId` compares the current object
while preserving the requested write version. This change does not retire
stale copies in other pools, undo historical accepted overwrites or provide
a new global clock-ordering guarantee. The multipart-completion repair in #190
neither introduced nor repaired this separate PUT defect.
- Reconcile ordinary single-object version DELETE across all pools, including
null versions, delete markers and unqualified directory-marker DELETE. This
applies the deletion to every resolved pool copy under existing quorum
@@ -50,7 +179,7 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
its tracker, mover, scanner hooks, configuration, XML actions and metrics.
Accept and ignore retired configuration/XML and preserve ordinary statistics
when reading v9 caches. See [migration notes](docs/bucket/lifecycle/access-tiering-removal.md).
The [decision record](docs/investigations/access-tiering-revert.md) preserves
The [decision record](https://silo.pgsty.com/compatibility/access-tiering-removal/) preserves
the feature's introduction, subsequent fixes, rollback scope and review history.
- Preserve the independent multi-pool write, metadata, healing and conditional
deletion fixes from PR #178, including shared remote-tier reference protection.
@@ -61,10 +190,10 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
- Repair federated CopyObject checksums, destination timestamps, reserved
metadata, encrypted-object forwarding, legal hold and KMS context.
- Make resync counters, target selection, cancellation and worker lifetimes
reflect actual work; complete delete-marker purges and report bounded MRF drops.
reflect actual work, and report bounded MRF drops.
- Converge bucket metadata with deterministic source state, deletion tombstones,
creation time recovery and diagnostics. The mixed-version export gate requires
coordinated upgrades before tombstones are exported. See [the #77 record](docs/investigations/issue-77-current.md).
coordinated upgrades before tombstones are exported. See [the #77 record](https://silo.pgsty.com/blog/design/bucket-metadata-convergence/).
- Include per-bucket CORS in metadata export/import, close metadata publication
and logger races, and report effective bucket quotas in metrics.
@@ -72,23 +201,28 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
- Restore embedded Console login over loopback TLS, trusted-proxy handling and
all four WebSocket connection limits. Preserve Go TLS defaults across transports.
- Directly require `github.com/pgsty/silo-pkg/v3` v3.14.0; select Console
`v0.0.0-20260913015128-417559bb2c97` and MC
`v0.0.0-20260913012246-4f609a4da3bb` with explicit PGSTY replacements.
- Pin upstream minio-go `v7.3.1-0.20260910142817-60bd07042d49`; refresh Go x/*
modules and security fixes including bounded AMQP frame handling. Keep Go
1.27.1 and go-systemd v22.6.0's NetBSD compatibility replacement.
- Directly require `github.com/pgsty/silo-pkg/v3` v3.14.1; select released Console
v2.4.1 (`v0.0.0-20260916075814-1360e26d976d`) and mcli 20260916
(`v0.0.0-20260916070421-e952aa78f10a`) with explicit PGSTY replacements.
The embedded frontend identifies itself as Console v2.4.1.
- Pin upstream minio-go `v7.3.1-0.20260915093545-32e1f32cb176` to handle
CopyObject errors embedded in HTTP 200 responses. Update JWX to v3.3.0 for
JSON field-name escaping, strfmt to v0.27.2 for Go 1.27 hostname validation,
and LZ4 to v4.1.30 for frame-reader, partial-read and concurrency fixes.
Retain the earlier Go x/* and bounded AMQP frame updates, Go 1.27.1, and
go-systemd v22.6.0's NetBSD compatibility replacement.
- Refresh container base digests and build static curl 8.22.0 from verified
source for both Linux architectures. Pin the actual mcli 20260913 archives and
hashes. Helm's client image follows that release; its Server image still names
the latest published Server 20260903.
source for both Linux architectures. Pin the published mcli 20260916 archives
and hashes in the container and update the client installer default.
- Prepare Helm chart 7.0.3 with Server and client defaults for the September 16
batch. Publish the chart only after the corresponding Server image exists.
- Pin GoReleaser v2.18.1 and its action commit identically in snapshot and release
workflows. Dependency-only PRs now run the Test Release Pipeline too.
The dependency update passed the final candidate's Go, vulnerability and Test
Release workflows; native curl builds passed on both architectures. A local
ARM64 image passed startup, health, S3 transfer and embedded Console checks.
These checks do not publish a Server tag or production image and do not replace
cluster upgrade/rollback acceptance for the next release. Dated investigations
retain the exact source and runtime boundaries they tested.
Validation of earlier source revisions does not establish acceptance of this
candidate. Final source, package, image and multi-process checks are tracked
separately in [#203](https://github.com/pgsty/silo/issues/203). No Server release
or production rollout is implied by this preparation target.
## RELEASE.2026-09-03T13-18-01Z
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
+10
View File
@@ -77,6 +77,16 @@ test evidence, compatibility notes, and documentation impact. Public product
documentation is owned by the separate
[`pgsty/silo.pgsty.com`](https://github.com/pgsty/silo.pgsty.com) repository.
## Contributor recognition
Every human issue or pull-request author is recognized in [CONTRIBUTORS.md](CONTRIBUTORS.md),
including open issues, draft PRs, and PRs closed without merging. Merged fixes,
adopted proposals, and reports that lead to fixes receive greater prominence;
first participation guides the remaining order. Work incorporated through a
later PR retains credit without changing the original PR's recorded status.
Security disclosures are credited with the reporter's agreement. DCO sign-off
applies to code commits, not to opening an issue.
## Licensing of Contributions
Code contributions to PGSTY SILO (`pgsty/silo`) are accepted under the
+116 -120
View File
File diff suppressed because one or more lines are too long
+5 -5
View File
@@ -1,4 +1,4 @@
FROM golang:1.27.1-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS curl-build
FROM golang:1.27.2-alpine@sha256:f92b6ef800e499660581efdabdf25d9d817a9d124eaf900924f0504e7e27e12d AS curl-build
ARG TARGETARCH
COPY dockerscripts/build-static-curl.sh /build/build-static-curl
RUN /bin/sh /build/build-static-curl
@@ -9,7 +9,7 @@ COPY --from=curl-build /go/bin/curl /curl
COPY --from=curl-build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/curl"]
FROM golang:1.27.1-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS build
FROM golang:1.27.2-alpine@sha256:f92b6ef800e499660581efdabdf25d9d817a9d124eaf900924f0504e7e27e12d AS build
ARG TARGETARCH
@@ -17,9 +17,9 @@ ENV GOPATH=/go
ENV CGO_ENABLED=0
ARG MC_REPO=pgsty/mc
ARG MC_VERSION=RELEASE.2026-09-13T00-00-00Z
ARG MC_AMD64_SHA256=9d2a92de9c7b887d9b944fe9ddce68d23f1b6df3415092e737594e56593f5e2b
ARG MC_ARM64_SHA256=3d82e9ea6c601c4cb44fe5dd5f2ad1b7d7d64369378110f9ada9c524688a452a
ARG MC_VERSION=RELEASE.2026-09-16T00-00-00Z
ARG MC_AMD64_SHA256=4ba2814fd5507fbe6b4d237c359750b9119d28d7217495fa5b48002fcbd397ef
ARG MC_ARM64_SHA256=b7008ca2a1bc5735b6585981c59a3640a0daa152dc789df3d1a0d0438787de82
RUN apk add -U --no-cache \
ca-certificates \
+1 -1
View File
@@ -4,7 +4,7 @@ LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
GOOS ?= $(shell go env GOOS)
GOARCH ?= $(shell go env GOARCH)
GOLANGCI_VERSION ?= v2.13.1
GOLANGCI_VERSION ?= v2.14.0
VERSION ?= $(shell git describe --tags)
REPO ?= docker.io/pgsty
+33 -49
View File
@@ -38,7 +38,7 @@
## Current release and main branch
The latest published Server is [20260903](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z).
As of 2026-09-13, the main branch has newer security, storage, Console and
As of 2026-09-16, the main branch has newer security, storage, Console and
shared-package changes that have not shipped in a Server release. See
[CHANGELOG.md](CHANGELOG.md) and the [component version matrix](https://silo.pgsty.com/compatibility/versions/)
for the exact release/source boundary, including SN-2026-011 and password-policy migration.
@@ -93,13 +93,16 @@ Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub
## Compatibility
The S3 API, `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, the `github.com/minio/*` import paths, and the on-disk format (including `.minio.sys`) are preserved and held in place by a CI compatibility check. Only Silo-owned delivery surfaces change: the `silo` executable, package, service, Helm chart, and container image — no `minio` binary alias is installed.
Silo preserves S3 and storage-format compatibility, including existing `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, and `.minio.sys` data. CI guards selected compatibility identifiers; release notes document intentional security and behavior changes. Silo-owned delivery surfaces use the `silo` executable, package, service, Helm chart, and container image; no `minio` server binary alias is installed.
The supported release stack is `pgsty/silo` + `pgsty/silo-console` + `pgsty/mc` + `pgsty/silo-pkg`; compatibility with unmodified upstream MinIO/MC is best effort. The Server, Console, and client retain their historical module paths where needed, while maintained code imports `github.com/pgsty/silo-pkg/v3` directly. The SDK `github.com/minio/minio-go/v7` is an explicit upstream dependency. See the current [go.mod](go.mod) for versions and replacements.
Every divergence from upstream is listed in the code-verified [compatibility audit](https://silo.pgsty.com/compatibility/server/). Treat each release as a downstream upgrade: pin versions, read the [release notes](https://silo.pgsty.com/tags/silo/), and keep a rollback path.
### TLS and Go upgrades
TLS key exchange follows Go's defaults across the S3 listener, node links,
The following TLS repair is on main and is not included in Server 20260903.
With that repair, TLS key exchange follows Go's defaults across the S3 listener, node links,
replication, identity providers, etcd, and external HTTP services. If an endpoint
cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges
for the process; certificate verification remains enabled. This option does not
@@ -115,7 +118,16 @@ values to restore Keychain trust. Explicit certificates in the configured `CAs`
directory remain additive to the selected root pool.
Go 1.27 binaries require macOS 13 or later. See the
[Go release notes](https://go.dev/doc/go1.27) and the
[SILO stack investigation](docs/investigations/go127-stack.md).
[Go 1.27 TLS and OIDC discovery guide](https://silo.pgsty.com/blog/design/go127-tls-oidc-discovery/).
## Documentation ownership
User documentation is maintained at [silo.pgsty.com](https://silo.pgsty.com/docs/),
with source in [pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com).
The remaining `docs/` tree contains inherited references, examples, and tooling
fixtures. Investigation logs, AI work records, and temporary reports are kept
outside this repository; reusable findings belong in the companion site.
See [AGENTS.md](AGENTS.md) for repository ownership and maintenance rules.
## Security & Contributing
@@ -123,53 +135,25 @@ Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); e
## Contributors
**41 community contributors** build SILO, Console, mcli, shared packages, and related projects. The list includes maintainers and every human Issue or PR author, ordered by merged PRs, other PRs, then issue reports. Gold rings highlight significant contributions.
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
<p align="center">
<a href="https://github.com/Vonng"><img src="https://silo.pgsty.com/images/contributors/Vonng.svg" width="60" height="60" alt="@Vonng" title="@Vonng — Maintains SILO, Console, mcli, shared packages, releases, and documentation"></a>
<a href="https://github.com/h5vx"><img src="https://silo.pgsty.com/images/contributors/h5vx.svg" width="60" height="60" alt="@h5vx" title="@h5vx — Implemented per-bucket CORS configuration and enforcement"></a>
<a href="https://github.com/mrjavadseydi"><img src="https://silo.pgsty.com/images/contributors/mrjavadseydi.svg" width="60" height="60" alt="@mrjavadseydi" title="@mrjavadseydi — Fixed effective bucket quota metrics; proposed access-frequency ILM"></a>
<a href="https://github.com/Dansyuqri"><img src="https://silo.pgsty.com/images/contributors/Dansyuqri.svg" width="60" height="60" alt="@Dansyuqri" title="@Dansyuqri — Added ChecksumType to multipart completion responses"></a>
<a href="https://github.com/ycjlin"><img src="https://silo.pgsty.com/images/contributors/ycjlin.svg" width="60" height="60" alt="@ycjlin" title="@ycjlin — Fixed missing-bucket ListObjects semantics"></a>
<a href="https://github.com/pinginfo"><img src="https://silo.pgsty.com/images/contributors/pinginfo.svg" width="60" height="60" alt="@pinginfo" title="@pinginfo — Repaired bucket notification streaming"></a>
<a href="https://github.com/ZouhairCharef"><img src="https://silo.pgsty.com/images/contributors/ZouhairCharef.svg" width="60" height="60" alt="@ZouhairCharef" title="@ZouhairCharef — Patched CVE-2026-34986 in go-jose"></a>
<a href="https://github.com/mfredenhagen"><img src="https://silo.pgsty.com/images/contributors/mfredenhagen.svg" width="60" height="60" alt="@mfredenhagen" title="@mfredenhagen — Patched CVE-2026-39883 in OpenTelemetry"></a>
<a href="https://github.com/waterkip"><img src="https://silo.pgsty.com/images/contributors/waterkip.svg" width="60" height="60" alt="@waterkip" title="@waterkip — Repointed documentation links to the SILO portal"></a>
<a href="https://github.com/mikemikimike"><img src="https://silo.pgsty.com/images/contributors/mikemikimike.svg" width="60" height="60" alt="@mikemikimike" title="@mikemikimike — Contributed the replicated SSE-C plaintext part-size fix"></a>
<a href="https://github.com/metaneutrons"><img src="https://silo.pgsty.com/images/contributors/metaneutrons.svg" width="60" height="60" alt="@metaneutrons" title="@metaneutrons — Reported and proposed explicit-version delete authorization"></a>
<a href="https://github.com/magicxor"><img src="https://silo.pgsty.com/images/contributors/magicxor.svg" width="60" height="60" alt="@magicxor" title="@magicxor — Reported and proposed conditional DELETE support for If-Match"></a>
<a href="https://github.com/davinkevin"><img src="https://silo.pgsty.com/images/contributors/davinkevin.svg" width="60" height="60" alt="@davinkevin" title="@davinkevin — Proposed the distroless container image and dependency automation"></a>
<a href="https://github.com/lem21h"><img src="https://silo.pgsty.com/images/contributors/lem21h.svg" width="48" height="48" alt="@lem21h" title="@lem21h — Proposed robustness and goroutine improvements"></a>
<a href="https://github.com/sulin37392"><img src="https://silo.pgsty.com/images/contributors/sulin37392.svg" width="48" height="48" alt="@sulin37392" title="@sulin37392 — Proposed dependency updates"></a>
<a href="https://github.com/cbornet"><img src="https://silo.pgsty.com/images/contributors/cbornet.svg" width="60" height="60" alt="@cbornet" title="@cbornet — Reported multipart and streaming checksum defects and missing-bucket semantics"></a>
<a href="https://github.com/vampywiz17"><img src="https://silo.pgsty.com/images/contributors/vampywiz17.svg" width="60" height="60" alt="@vampywiz17" title="@vampywiz17 — Reported LDAP TLS and Console login regressions"></a>
<a href="https://github.com/orenyomtov"><img src="https://silo.pgsty.com/images/contributors/orenyomtov.svg" width="60" height="60" alt="@orenyomtov" title="@orenyomtov — Reported the unsigned-header CopyObject cross-object read (SN-2026-011)"></a>
<a href="https://github.com/mumu-lab"><img src="https://silo.pgsty.com/images/contributors/mumu-lab.svg" width="48" height="48" alt="@mumu-lab" title="@mumu-lab — Reported bucket quota metrics reading a deprecated field"></a>
<a href="https://github.com/jvasile"><img src="https://silo.pgsty.com/images/contributors/jvasile.svg" width="48" height="48" alt="@jvasile" title="@jvasile — Reported missing user, group, and defaults in Debian packages"></a>
<a href="https://github.com/pmezhuev"><img src="https://silo.pgsty.com/images/contributors/pmezhuev.svg" width="48" height="48" alt="@pmezhuev" title="@pmezhuev — Reported missing RPM package signatures"></a>
<a href="https://github.com/TLINDEN"><img src="https://silo.pgsty.com/images/contributors/TLINDEN.svg" width="48" height="48" alt="@TLINDEN" title="@TLINDEN — Reported the missing client in release tarballs"></a>
<a href="https://github.com/makinikm"><img src="https://silo.pgsty.com/images/contributors/makinikm.svg" width="48" height="48" alt="@makinikm" title="@makinikm — Reported the missing client in the container image"></a>
<a href="https://github.com/meesudzu"><img src="https://silo.pgsty.com/images/contributors/meesudzu.svg" width="48" height="48" alt="@meesudzu" title="@meesudzu — Requested the migration guide from upstream MinIO"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://silo.pgsty.com/images/contributors/kuldeep-link11.svg" width="48" height="48" alt="@kuldeep-link11" title="@kuldeep-link11 — Reported NATS JWT credentials and target reload issues"></a>
<a href="https://github.com/sargarass"><img src="https://silo.pgsty.com/images/contributors/sargarass.svg" width="48" height="48" alt="@sargarass" title="@sargarass — Reported ListMultipartUploads prefix and pagination semantics"></a>
<a href="https://github.com/liuhaodongliu990-cmyk"><img src="https://silo.pgsty.com/images/contributors/liuhaodongliu990-cmyk.svg" width="48" height="48" alt="@liuhaodongliu990-cmyk" title="@liuhaodongliu990-cmyk — Reported indeterminate progress for prefix downloads"></a>
<a href="https://github.com/Xavier-777"><img src="https://silo.pgsty.com/images/contributors/Xavier-777.svg" width="48" height="48" alt="@Xavier-777" title="@Xavier-777 — Reported Console lifecycle management and file preview gaps"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://silo.pgsty.com/images/contributors/spaceg00se-r.svg" width="48" height="48" alt="@spaceg00se-r" title="@spaceg00se-r — Requested cpuv1 support and reported a workflow token failure"></a>
<a href="https://github.com/kh0mka"><img src="https://silo.pgsty.com/images/contributors/kh0mka.svg" width="48" height="48" alt="@kh0mka" title="@kh0mka — Reported inter-node I/O timeouts in ReadFileStreamHandler"></a>
<a href="https://github.com/bagutzu"><img src="https://silo.pgsty.com/images/contributors/bagutzu.svg" width="48" height="48" alt="@bagutzu" title="@bagutzu — Requested KES-compatible external KMS and OpenBao support"></a>
<a href="https://github.com/DestroyLee"><img src="https://silo.pgsty.com/images/contributors/DestroyLee.svg" width="48" height="48" alt="@DestroyLee" title="@DestroyLee — Reported the missing documentation navigation"></a>
<a href="https://github.com/mosesdd"><img src="https://silo.pgsty.com/images/contributors/mosesdd.svg" width="48" height="48" alt="@mosesdd" title="@mosesdd — Requested a maintained Helm chart"></a>
<a href="https://github.com/zylpsrs"><img src="https://silo.pgsty.com/images/contributors/zylpsrs.svg" width="48" height="48" alt="@zylpsrs" title="@zylpsrs — Reported missing Console tiering and site replication"></a>
<a href="https://github.com/heroes1412"><img src="https://silo.pgsty.com/images/contributors/heroes1412.svg" width="48" height="48" alt="@heroes1412" title="@heroes1412 — Reported the unusable profiling option"></a>
<a href="https://github.com/redfoxfox"><img src="https://silo.pgsty.com/images/contributors/redfoxfox.svg" width="48" height="48" alt="@redfoxfox" title="@redfoxfox — Reported Chinese documentation availability"></a>
<a href="https://github.com/jiadzh"><img src="https://silo.pgsty.com/images/contributors/jiadzh.svg" width="48" height="48" alt="@jiadzh" title="@jiadzh — Requested Windows build guidance"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://silo.pgsty.com/images/contributors/AntonOfTheWoods.svg" width="48" height="48" alt="@AntonOfTheWoods" title="@AntonOfTheWoods — Asked for clarity on Helm chart and operator options"></a>
<a href="https://github.com/chalukyaj"><img src="https://silo.pgsty.com/images/contributors/chalukyaj.svg" width="48" height="48" alt="@chalukyaj" title="@chalukyaj — Proposed making the SILO Operator easier to discover"></a>
<a href="https://github.com/nsanitate"><img src="https://silo.pgsty.com/images/contributors/nsanitate.svg" width="48" height="48" alt="@nsanitate" title="@nsanitate — Proposed CNCF Sandbox governance"></a>
<a href="https://github.com/Kesavaambati"><img src="https://silo.pgsty.com/images/contributors/Kesavaambati.svg" width="48" height="48" alt="@Kesavaambati" title="@Kesavaambati — Asked about community support and image maintenance"></a>
</p>
Every human issue or pull-request author is part of the SILO community, including open and unmerged work. Merged fixes, adopted proposals, and actionable reports receive priority, with first participation guiding the remaining order. Gold rings highlight reviewed significant contributions.
[View the full contribution record](CONTRIBUTORS.md) for each person's proposals, fixes, and reports.
<a href="CONTRIBUTORS.md">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-light.svg" alt="SILO community contributors">
</picture>
</a>
[View contribution notes and actual PR status](CONTRIBUTORS.md).
## Star History
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-light.svg" alt="SILO GitHub star history">
</picture>
## Background
+34 -47
View File
@@ -38,7 +38,7 @@
## 当前发行版与主分支
最新已发布的 Server 仍为 [20260903](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z)。
截至 2026-09-13,主分支已合入更新的安全、存储、Console 与共享包改动,但尚未发布新 Server。
截至 2026-09-16,主分支已合入更新的安全、存储、Console 与共享包改动,但尚未发布新 Server。
准确的已发布/源码边界见 [CHANGELOG.md](CHANGELOG.md) 与[组件版本矩阵](https://silo.pgsty.com/zh/compatibility/versions/),
其中包括 SN-2026-011 修复状态与密码权限迁移要求。
@@ -92,63 +92,50 @@ docker exec silo mcli mb local/demo && docker exec silo mcli ls local
## 兼容性
S3 API、`MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*` 路由、`github.com/minio/*` 导入路径与磁盘格式(含 `.minio.sys`)原样保留,并由 CI 兼容性门禁冻结。只有 Silo 自有交付面改名:`silo` 可执行文件、软件包、服务、Helm Chart 与容器镜像 —— 原生交付物不会安装 `minio` 二进制别名。
Silo 保留 S3 与存储格式兼容性,包括既有 `MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*` 路由与 `.minio.sys` 数据。CI 守卫检查选定的兼容性标识,有意的安全与行为变化在发布说明中记录。Silo 自有交付面使用 `silo` 可执行文件、软件包、服务、Helm Chart 与容器镜像;原生交付物不会安装 `minio` 服务端二进制别名。
正式支持和发布验收的组合为 `pgsty/silo` + `pgsty/silo-console` + `pgsty/mc` + `pgsty/silo-pkg`,对未修改的上游 MinIO/MC 尽最大努力保持兼容。Server、Console 与客户端按需保留历史模块路径,维护源码直接导入 `github.com/pgsty/silo-pkg/v3`;SDK `github.com/minio/minio-go/v7` 是明确保留的上游依赖。具体版本与 replace 以当前 [go.mod](go.mod) 为准。
与上游的全部分歧,以逐项核验代码的[兼容性审计](https://silo.pgsty.com/zh/compatibility/server/)形式维护。每个版本仍应视为下游升级:锁定版本,阅读[版本说明](https://silo.pgsty.com/zh/tags/silo/),并保留回滚路径。
### TLS 与 Go 升级
Server 恢复 Go 默认密钥交换策略的修复已在 main,尚未包含在 Server 20260903。
`GODEBUG=tlsmlkem=0`、`tlssecpmlkem=0` 的适用范围、macOS 根证书来源变化,
以及 OIDC discovery 的诊断方法见 [Go 1.27 TLS 与 OIDC 指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
## 文档归属
用户文档统一维护在 [silo.pgsty.com](https://silo.pgsty.com/zh/docs/),源码位于
[pgsty/silo.pgsty.com](https://github.com/pgsty/silo.pgsty.com)。本仓库保留的 `docs/`
主要是继承的参考材料、示例和工具测试夹具。调查日志、AI 工作记录与临时报告放在仓库外;
可复用的结论应整理进伴生文档站。仓库职责与维护规则见 [AGENTS.md](AGENTS.md)。
## 安全与贡献
请按照 [`SECURITY.md`](SECURITY.md) 私密报告漏洞;每项修复都会发布公开[安全公告](https://silo.pgsty.com/zh/blog/security/)。本项目不要求签署 CLA:贡献按 AGPL-3.0-or-later(inbound=outbound)接收,只需 DCO 签署(`git commit -s`),详见 [`CONTRIBUTING.md`](CONTRIBUTING.md)。
## 贡献者
**41 位社区贡献者**共同建设 SILO、Console、mcli、公共包与相关项目。名单包含维护者,以及所有提出 Issue 或 PR 的真人作者;按已合并 PR、其他 PR、Issue 报告排序,黄圈标记显著贡献。
<!-- Generated by silo.pgsty.com/bin/contributors.py. -->
<p align="center">
<a href="https://github.com/Vonng"><img src="https://silo.pgsty.com/images/contributors/Vonng.svg" width="60" height="60" alt="@Vonng" title="@Vonng — 维护 SILO、Console、mcli、公共包、发行与文档"></a>
<a href="https://github.com/h5vx"><img src="https://silo.pgsty.com/images/contributors/h5vx.svg" width="60" height="60" alt="@h5vx" title="@h5vx — 实现单桶 CORS 配置与请求执行"></a>
<a href="https://github.com/mrjavadseydi"><img src="https://silo.pgsty.com/images/contributors/mrjavadseydi.svg" width="60" height="60" alt="@mrjavadseydi" title="@mrjavadseydi — 修复有效桶配额指标,并提交按访问频率分层的 ILM 方案"></a>
<a href="https://github.com/Dansyuqri"><img src="https://silo.pgsty.com/images/contributors/Dansyuqri.svg" width="60" height="60" alt="@Dansyuqri" title="@Dansyuqri — 为分片上传完成响应补充 ChecksumType"></a>
<a href="https://github.com/ycjlin"><img src="https://silo.pgsty.com/images/contributors/ycjlin.svg" width="60" height="60" alt="@ycjlin" title="@ycjlin — 修复缺失桶的 ListObjects 语义"></a>
<a href="https://github.com/pinginfo"><img src="https://silo.pgsty.com/images/contributors/pinginfo.svg" width="60" height="60" alt="@pinginfo" title="@pinginfo — 修复桶通知的流式输出"></a>
<a href="https://github.com/ZouhairCharef"><img src="https://silo.pgsty.com/images/contributors/ZouhairCharef.svg" width="60" height="60" alt="@ZouhairCharef" title="@ZouhairCharef — 修复 go-jose 中的 CVE-2026-34986"></a>
<a href="https://github.com/mfredenhagen"><img src="https://silo.pgsty.com/images/contributors/mfredenhagen.svg" width="60" height="60" alt="@mfredenhagen" title="@mfredenhagen — 修复 OpenTelemetry 中的 CVE-2026-39883"></a>
<a href="https://github.com/waterkip"><img src="https://silo.pgsty.com/images/contributors/waterkip.svg" width="60" height="60" alt="@waterkip" title="@waterkip — 将文档链接指向 SILO 门户"></a>
<a href="https://github.com/mikemikimike"><img src="https://silo.pgsty.com/images/contributors/mikemikimike.svg" width="60" height="60" alt="@mikemikimike" title="@mikemikimike — 提交 SSE-C 复制分片明文尺寸修复"></a>
<a href="https://github.com/metaneutrons"><img src="https://silo.pgsty.com/images/contributors/metaneutrons.svg" width="60" height="60" alt="@metaneutrons" title="@metaneutrons — 报告并提交显式版本删除鉴权方案"></a>
<a href="https://github.com/magicxor"><img src="https://silo.pgsty.com/images/contributors/magicxor.svg" width="60" height="60" alt="@magicxor" title="@magicxor — 报告并提交 DELETE If-Match 条件请求支持方案"></a>
<a href="https://github.com/davinkevin"><img src="https://silo.pgsty.com/images/contributors/davinkevin.svg" width="60" height="60" alt="@davinkevin" title="@davinkevin — 提交 distroless 容器镜像与依赖自动更新方案"></a>
<a href="https://github.com/lem21h"><img src="https://silo.pgsty.com/images/contributors/lem21h.svg" width="48" height="48" alt="@lem21h" title="@lem21h — 提交健壮性与 goroutine 改进"></a>
<a href="https://github.com/sulin37392"><img src="https://silo.pgsty.com/images/contributors/sulin37392.svg" width="48" height="48" alt="@sulin37392" title="@sulin37392 — 提交依赖更新"></a>
<a href="https://github.com/cbornet"><img src="https://silo.pgsty.com/images/contributors/cbornet.svg" width="60" height="60" alt="@cbornet" title="@cbornet — 报告分片与流式校验和缺陷及缺失桶语义问题"></a>
<a href="https://github.com/vampywiz17"><img src="https://silo.pgsty.com/images/contributors/vampywiz17.svg" width="60" height="60" alt="@vampywiz17" title="@vampywiz17 — 报告 LDAP TLS 与 Console 登录回归"></a>
<a href="https://github.com/orenyomtov"><img src="https://silo.pgsty.com/images/contributors/orenyomtov.svg" width="60" height="60" alt="@orenyomtov" title="@orenyomtov — 报告未签名头导致的 CopyObject 跨对象读取(SN-2026-011)"></a>
<a href="https://github.com/mumu-lab"><img src="https://silo.pgsty.com/images/contributors/mumu-lab.svg" width="48" height="48" alt="@mumu-lab" title="@mumu-lab — 报告桶配额指标读取已弃用字段的问题"></a>
<a href="https://github.com/jvasile"><img src="https://silo.pgsty.com/images/contributors/jvasile.svg" width="48" height="48" alt="@jvasile" title="@jvasile — 报告 Debian 包缺少用户、用户组与默认配置"></a>
<a href="https://github.com/pmezhuev"><img src="https://silo.pgsty.com/images/contributors/pmezhuev.svg" width="48" height="48" alt="@pmezhuev" title="@pmezhuev — 报告 RPM 包缺少 GPG 签名"></a>
<a href="https://github.com/TLINDEN"><img src="https://silo.pgsty.com/images/contributors/TLINDEN.svg" width="48" height="48" alt="@TLINDEN" title="@TLINDEN — 报告发布压缩包缺少客户端"></a>
<a href="https://github.com/makinikm"><img src="https://silo.pgsty.com/images/contributors/makinikm.svg" width="48" height="48" alt="@makinikm" title="@makinikm — 报告容器镜像缺少客户端"></a>
<a href="https://github.com/meesudzu"><img src="https://silo.pgsty.com/images/contributors/meesudzu.svg" width="48" height="48" alt="@meesudzu" title="@meesudzu — 提出从上游 MinIO 迁移的指南需求"></a>
<a href="https://github.com/kuldeep-link11"><img src="https://silo.pgsty.com/images/contributors/kuldeep-link11.svg" width="48" height="48" alt="@kuldeep-link11" title="@kuldeep-link11 — 报告 NATS JWT 凭据与通知目标重载问题"></a>
<a href="https://github.com/sargarass"><img src="https://silo.pgsty.com/images/contributors/sargarass.svg" width="48" height="48" alt="@sargarass" title="@sargarass — 报告 ListMultipartUploads 前缀与分页语义问题"></a>
<a href="https://github.com/liuhaodongliu990-cmyk"><img src="https://silo.pgsty.com/images/contributors/liuhaodongliu990-cmyk.svg" width="48" height="48" alt="@liuhaodongliu990-cmyk" title="@liuhaodongliu990-cmyk — 报告前缀下载进度显示异常"></a>
<a href="https://github.com/Xavier-777"><img src="https://silo.pgsty.com/images/contributors/Xavier-777.svg" width="48" height="48" alt="@Xavier-777" title="@Xavier-777 — 报告 Console 生命周期管理与文件预览缺失"></a>
<a href="https://github.com/spaceg00se-r"><img src="https://silo.pgsty.com/images/contributors/spaceg00se-r.svg" width="48" height="48" alt="@spaceg00se-r" title="@spaceg00se-r — 提出 cpuv1 支持需求并报告工作流令牌错误"></a>
<a href="https://github.com/kh0mka"><img src="https://silo.pgsty.com/images/contributors/kh0mka.svg" width="48" height="48" alt="@kh0mka" title="@kh0mka — 报告 ReadFileStreamHandler 节点间 I/O 超时"></a>
<a href="https://github.com/bagutzu"><img src="https://silo.pgsty.com/images/contributors/bagutzu.svg" width="48" height="48" alt="@bagutzu" title="@bagutzu — 提出兼容 KES 的外部 KMS 与 OpenBao 支持需求"></a>
<a href="https://github.com/DestroyLee"><img src="https://silo.pgsty.com/images/contributors/DestroyLee.svg" width="48" height="48" alt="@DestroyLee" title="@DestroyLee — 报告文档目录导航缺失"></a>
<a href="https://github.com/mosesdd"><img src="https://silo.pgsty.com/images/contributors/mosesdd.svg" width="48" height="48" alt="@mosesdd" title="@mosesdd — 提出维护 Helm Chart 的需求"></a>
<a href="https://github.com/zylpsrs"><img src="https://silo.pgsty.com/images/contributors/zylpsrs.svg" width="48" height="48" alt="@zylpsrs" title="@zylpsrs — 报告 Console 缺少分层与站点复制"></a>
<a href="https://github.com/heroes1412"><img src="https://silo.pgsty.com/images/contributors/heroes1412.svg" width="48" height="48" alt="@heroes1412" title="@heroes1412 — 报告性能分析选项不可用"></a>
<a href="https://github.com/redfoxfox"><img src="https://silo.pgsty.com/images/contributors/redfoxfox.svg" width="48" height="48" alt="@redfoxfox" title="@redfoxfox — 报告中文文档站点不可用"></a>
<a href="https://github.com/jiadzh"><img src="https://silo.pgsty.com/images/contributors/jiadzh.svg" width="48" height="48" alt="@jiadzh" title="@jiadzh — 提出 Windows 构建指导需求"></a>
<a href="https://github.com/AntonOfTheWoods"><img src="https://silo.pgsty.com/images/contributors/AntonOfTheWoods.svg" width="48" height="48" alt="@AntonOfTheWoods" title="@AntonOfTheWoods — 提出明确 Helm Chart 与 Operator 选项的需求"></a>
<a href="https://github.com/chalukyaj"><img src="https://silo.pgsty.com/images/contributors/chalukyaj.svg" width="48" height="48" alt="@chalukyaj" title="@chalukyaj — 提出改善 SILO Operator 可发现性的建议"></a>
<a href="https://github.com/nsanitate"><img src="https://silo.pgsty.com/images/contributors/nsanitate.svg" width="48" height="48" alt="@nsanitate" title="@nsanitate — 提出加入 CNCF Sandbox 的治理建议"></a>
<a href="https://github.com/Kesavaambati"><img src="https://silo.pgsty.com/images/contributors/Kesavaambati.svg" width="48" height="48" alt="@Kesavaambati" title="@Kesavaambati — 提出社区支持与容器镜像维护问题"></a>
</p>
每位 issue 或 PR 的作者都是 SILO 社区的一员,包括尚未合并的工作。已合并的修复、被采纳的方案和有效报告优先展示,其余参考首次参与时间;金色圆环突出经过审核的显著贡献。
[查看完整贡献记录](CONTRIBUTORS.md),了解每位贡献者的提案、修复与问题报告。
<a href="CONTRIBUTORS.md">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/contributors-light.svg" alt="SILO 社区贡献者">
</picture>
</a>
[查看贡献记录与实际 PR 状态](CONTRIBUTORS.md)。
## Star History
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-dark.svg">
<img src="https://raw.githubusercontent.com/pgsty/silo/codex/repository-cards/star-history-light.svg" alt="SILO GitHub 星标历史">
</picture>
## 背景
+3 -3
View File
@@ -7,7 +7,7 @@ Silo-specific fixes or release notes.
## Supported Versions
Security fixes are tracked on the active development branch and summarized in
[docs/security/advisories.md](docs/security/advisories.md). Only the current
[the security advisory ledger](https://silo.pgsty.com/about/security-advisories/). Only the current
Silo release line is supported unless an advisory says otherwise.
## Inherited Fix Evidence
@@ -26,7 +26,7 @@ separately even when the fork preserves the original commit object and SHA.
The inherited [service-account](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/admin-handlers-users_test.go#L211-L212)
and [STS](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/sts-handlers_test.go#L45-L46)
regression groups remain part of `go test ./cmd`; see the
[canonical ledger](docs/security/advisories.md#inherited-upstream-advisory-baseline)
[canonical ledger](https://silo.pgsty.com/about/security-advisories/#inherited)
for the operator-facing record.
## Reporting a Vulnerability
@@ -42,4 +42,4 @@ For vulnerabilities in this fork:
## Disclosure Process
Fork-specific fixes and user-visible upgrade notes are published in [docs/security/advisories.md](docs/security/advisories.md). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
Fork-specific fixes and user-visible upgrade notes are published in [the security advisory ledger](https://silo.pgsty.com/about/security-advisories/). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
+1 -1
View File
@@ -34,4 +34,4 @@ Based on the report, the Silo maintainers investigate:
If the vulnerability exists in this fork itself, the maintainers will, when
feasible, fix the issue or implement reasonable countermeasures such that the
vulnerability can no longer be exploited. Fork-specific upgrade notes and
security advisories are published in `docs/security/advisories.md`.
security advisories are published in [the security advisory ledger](https://silo.pgsty.com/about/security-advisories/).
+1 -1
View File
@@ -7,7 +7,7 @@ _init() {
## Minimum required versions for build dependencies
GIT_VERSION="1.0"
GO_VERSION="1.27.1"
GO_VERSION="1.27.2"
OSX_VERSION="10.8"
KNAME=$(uname -s)
ARCH=$(uname -m)
+1 -1
View File
@@ -40,7 +40,7 @@ if [ -n "${MCLI_BIN:-}" ]; then
exit 0
fi
release=${MCLI_RELEASE:-RELEASE.2026-09-13T00-00-00Z}
release=${MCLI_RELEASE:-RELEASE.2026-09-16T00-00-00Z}
version_hyphen=${release#RELEASE.}
package_version=$(printf '%s\n' "${version_hyphen}" | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/\1\2\3\4\5\6.0.0/')
if [ "${package_version}" = "${version_hyphen}" ]; then
@@ -132,8 +132,8 @@
"MINIO_API_DELETE_CLEANUP_INTERVAL",
"MINIO_API_DISABLE_ODIRECT",
"MINIO_API_GZIP_OBJECTS",
"MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH",
"MINIO_API_LIST_QUORUM",
"MINIO_API_MULTIPART_LISTING",
"MINIO_API_OBJECT_MAX_VERSIONS",
"MINIO_API_ODIRECT",
"MINIO_API_REMOTE_TRANSPORT_DEADLINE",
@@ -571,6 +571,14 @@
"minio_resource_stats",
"minio_s3",
"minio_stats",
"minio_system_cpu_avg_idle",
"minio_system_cpu_avg_iowait",
"minio_system_cpu_load",
"minio_system_cpu_load_perc",
"minio_system_cpu_nice",
"minio_system_cpu_steal",
"minio_system_cpu_system",
"minio_system_cpu_user",
"minio_test"
],
"headers": [
@@ -627,7 +635,6 @@
"x-minio-origin-endpoint",
"x-minio-prefixes-total",
"x-minio-read-quorum",
"x-minio-replication",
"x-minio-replication-actual-object-size",
"x-minio-replication-delete-status",
"x-minio-replication-deletemarker-status",
@@ -766,6 +773,7 @@
"/metrics/v3",
"/minio/grid/",
"/minio/grid/lock/",
"/multipart-preflight",
"/netperf",
"/notification",
"/oauth2/callback",
@@ -934,6 +942,7 @@
"arn:minio:kms:::this-is-disregarded",
"arn:minio:kms:::xyz-test-key",
"arn:minio:replication:",
"arn:minio:replication::",
"arn:minio:replication::8320b6d18f9032b4700f1f03b50d8d1853de8f22cab86931ee794e12f190852c:destinationbucket",
"arn:minio:replication:::",
"arn:minio:replication:::dest-bucket",
+1 -2
View File
@@ -115,9 +115,8 @@ func collect(repo string) (manifest, error) {
fset := token.NewFileSet()
for _, rel := range files {
// Investigation artifacts contain synthetic routes and archived configurations.
// Migration notes and guard fixtures contain archived identifiers.
if rel == "SILO_REBRANDING_MIGRATION.md" ||
strings.HasPrefix(rel, "docs/investigations/") ||
strings.HasPrefix(rel, "buildscripts/rebrand-guard/") ||
strings.HasPrefix(rel, "buildscripts/helm-migration-guard/") {
continue
+1
View File
@@ -0,0 +1 @@
__pycache__/
+163
View File
@@ -0,0 +1,163 @@
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Pure, self-contained SVG rendering for SILO's README cards."""
from datetime import date, timedelta
from html import escape
import math
import xml.etree.ElementTree as ET
themes = {
'light': dict(bg='#ffffff', wash='#f2f7fc', edge='#d9e3ee', ink='#16222e',
muted='#62758a', blue='#1d588c', copper='#b4762e', grid='#e5edf5',
line='#2b6ca3', ring='#dce5ef', field='#f7f9fc', label='#3d4e61'),
'dark': dict(bg='#101923', wash='#152738', edge='#2b3c50', ink='#e8eef6',
muted='#93a3b8', blue='#7fb8e8', copper='#e0a35c', grid='#263749',
line='#5da2dd', ring='#3a4e63', field='#0b1119', label='#b6c2d2'),
}
def read_emblem(path):
emblem = ET.parse(path).getroot()
body = ''.join(ET.tostring(child, encoding='unicode') for child in emblem
if child.tag.rsplit('}', 1)[-1] in ('defs', 'g'))
return '\n'.join(line.rstrip() for line in body.splitlines()).strip()
def txt(x, y, value, size=14, color=None, weight=400, anchor='start', mono=False, spacing=None):
family = 'Menlo,Consolas,monospace' if mono else 'Arial,Helvetica,sans-serif'
extra = f' letter-spacing="{spacing}"' if spacing is not None else ''
return (f'<text x="{x}" y="{y}" font-family="{family}" font-size="{size}" '
f'font-weight="{weight}" fill="{color}" text-anchor="{anchor}"{extra}>'
f'{escape(str(value))}</text>')
def start(height, theme, title, description, emblem_body):
t = themes[theme]
return [f'<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="{height}" '
f'viewBox="0 0 1000 {height}" role="img" aria-labelledby="title desc">',
f'<title id="title">{escape(title)}</title><desc id="desc">{escape(description)}</desc>',
'<defs><linearGradient id="surface" x1="0" y1="1" x2="1" y2="0">'
f'<stop offset="0" stop-color="{t["bg"]}"/>'
f'<stop offset="1" stop-color="{t["wash"]}"/></linearGradient>'
'<linearGradient id="accent" x1="0" y1="0" x2="1" y2="0">'
f'<stop offset="0" stop-color="{t["blue"]}"/>'
f'<stop offset="1" stop-color="{t["copper"]}"/></linearGradient>'
'<linearGradient id="area" x1="0" y1="0" x2="0" y2="1">'
f'<stop offset="0" stop-color="{t["line"]}" stop-opacity=".22"/>'
f'<stop offset="1" stop-color="{t["line"]}" stop-opacity=".015"/>'
'</linearGradient></defs>',
f'<rect x=".75" y=".75" width="998.5" height="{height-1.5}" rx="22" '
f'fill="url(#surface)" stroke="{t["edge"]}" stroke-width="1.5"/>',
f'<svg x="40" y="25" width="25" height="25" viewBox="230 213 570 570">{emblem_body}</svg>']
def heading(parts, t, eyebrow, title, subtitle, value, value_label):
parts.extend([
txt(76, 43, eyebrow, 11, t['muted'], 600, mono=True, spacing=1.7),
txt(40, 94, title, 32, t['ink'], 700),
txt(41, 123, subtitle, 14, t['muted']),
txt(958, 89, f'{value:,}', 45, t['ink'], 700, anchor='end'),
txt(957, 114, value_label, 10, t['muted'], 600, anchor='end', mono=True, spacing=1.5),
f'<path d="M40 146 H960" stroke="{t["edge"]}"/>',
])
def contributors(theme, people, snapshot, emblem_body):
height = 206 + 76 * math.ceil(len(people) / 10)
t = themes[theme]
parts = start(height, theme, f'SILO community — {len(people)} contributors',
f'The existing SILO community roll, including code, proposals and reports across related projects. '
f'Gold rings retain the existing significant-contribution designation. Snapshot {snapshot}.', emblem_body)
heading(parts, t, 'SILO / COMMUNITY', 'Contributors',
'Code, proposals & reports across SILO and related projects', len(people), 'COMMUNITY CONTRIBUTORS')
for row in range(math.ceil(len(people) / 10)):
group = people[row * 10:(row + 1) * 10]
row_width = len(group) * 91
for col, person in enumerate(group):
x = (1000 - row_width) / 2 + col * 91 + 45.5
y = 199 + row * 76
identifier = f'avatar-{row}-{col}'
featured = bool(person.get('featured'))
parts.append(f'<g><title>@{escape(person["handle"])} — {escape(person["what"])}</title>')
parts.append(f'<defs><clipPath id="{identifier}"><circle cx="{x}" cy="{y}" r="29"/></clipPath></defs>')
if featured:
parts.append(f'<circle cx="{x}" cy="{y}" r="34" fill="{t["copper"]}" opacity=".09"/>')
if person.get('avatarDataUrl'):
parts.append(f'<image x="{x-29}" y="{y-29}" width="58" height="58" '
f'clip-path="url(#{identifier})" href="{escape(person["avatarDataUrl"])}"/>')
else:
parts.append(f'<circle cx="{x}" cy="{y}" r="29" fill="{t["ring"]}"/>')
parts.append(txt(x, y + 9, person['handle'][0].upper(), 26, t['ink'], 700, 'middle'))
parts.append(f'<circle cx="{x}" cy="{y}" r="30.5" fill="none" '
f'stroke="{t["copper"] if featured else t["ring"]}" stroke-width="{2 if featured else 1.25}"/></g>')
parts.extend([
f'<path d="M40 {height-42} H960" stroke="{t["edge"]}"/>',
f'<circle cx="47" cy="{height-21}" r="4" fill="none" stroke="{t["copper"]}" stroke-width="1.5"/>',
txt(61, height-17, 'Gold rings mark significant contributions', 12, t['muted']),
txt(959, height-17, f'AS OF {snapshot}', 10, t['muted'], 500, 'end', mono=True, spacing=.6),
'</svg>',
])
return ''.join(parts)
def stars(theme, history, snapshot, emblem_body):
points = history['points']
star_count = points[-1]['stars']
t = themes[theme]
provenance = ('Initial history reconstructed · Daily totals since ' + history['bootstrap']['through'] + ' · UTC'
if history['bootstrap']['reconstructed'] else 'Observed daily star totals · UTC')
parts = start(558, theme, f'SILO star history — {star_count:,} stars',
f'GitHub repository pgsty/silo. {star_count:,} stars as of {snapshot}. ' +
provenance, emblem_body)
heading(parts, t, 'SILO / GITHUB', 'Star History', 'pgsty/silo', star_count, 'GITHUB STARS')
left, right, top, bottom = 76, 958, 177, 440
begin = date.fromisoformat(points[0]['date'])
end = date.fromisoformat(points[-1]['date'])
days = max(1, (end - begin).days)
maximum = max(500, math.ceil(max(p['stars'] for p in points) / 500) * 500)
tick_step = 10 ** max(0, int(math.log10(maximum)))
xy = lambda day, n: (left + (right-left)*(date.fromisoformat(day)-begin).days / days,
bottom-(bottom-top)*n/maximum)
for value in range(0, maximum+1, tick_step):
y = xy(points[0]['date'], value)[1]
parts.append(f'<path d="M{left} {y:.2f} H{right}" stroke="{t["grid"]}" stroke-dasharray="4 6"/>')
parts.append(txt(left-16, round(y+4, 2), f'{value / 1000:g}k' if value >= 1000 else str(value), 12, t['muted'], anchor='end'))
dates = sorted({begin + timedelta(days=round((end-begin).days*i/5)) for i in range(6)})
ticks = [(d.isoformat(), d.strftime('%b %Y') if days > 90 else d.strftime('%b %d')) for d in dates]
for day, label in ticks:
x = xy(day, 0)[0]
parts.append(f'<path d="M{x:.2f} {top} V{bottom}" stroke="{t["grid"]}" stroke-opacity=".65"/>')
anchor = 'start' if day == points[0]['date'] else 'end' if day == snapshot else 'middle'
parts.append(txt(round(x, 2), 466, label, 12, t['muted'], anchor=anchor))
coords = [xy(p['date'], p['stars']) for p in points]
line = 'M' + ' L'.join(f'{x:.2f} {y:.2f}' for x, y in coords)
area = line + f' L{coords[-1][0]:.2f} {bottom} L{left} {bottom} Z'
parts.extend([
f'<path d="{area}" fill="url(#area)"/>',
f'<path d="{line}" fill="none" stroke="url(#accent)" stroke-width="3" '
'stroke-linecap="round" stroke-linejoin="round"/>',
f'<path d="M{left} {bottom} H{right}" stroke="{t["edge"]}"/>',
])
x, y = coords[-1]
parts.extend([
f'<circle cx="{x}" cy="{y}" r="10" fill="{t["copper"]}" opacity=".12"/>',
f'<circle cx="{x}" cy="{y}" r="5" fill="{t["copper"]}" stroke="{t["bg"]}" stroke-width="2"/>',
f'<path d="M40 491 H960" stroke="{t["edge"]}"/>',
txt(40, 516, f'{begin:%b %Y} — {end:%b %Y}'.upper(), 10, t['muted'], 500, mono=True, spacing=.7),
txt(959, 516, f'SNAPSHOT {snapshot}', 10, t['muted'], 500, 'end', mono=True, spacing=.6),
txt(40, 539, provenance, 11, t['muted']),
'</svg>',
])
return ''.join(parts)
@@ -0,0 +1 @@
PyYAML==6.0.3
+178
View File
@@ -0,0 +1,178 @@
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Regression checks for historical accuracy, contributor scope and SVG safety."""
import base64
import json
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
from urllib.error import URLError
import xml.etree.ElementTree as ET
import render
import update
NS = {'s': 'http://www.w3.org/2000/svg'}
PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+a7mgAAAAASUVORK5CYII=')
def person(handle='Alice', group='reports', featured=False):
return {'handle': handle, 'group': group, 'featured': featured,
'what': 'A reviewed contribution', 'firstContribution': '2026-09-01'}
def history():
return {'repository': 'pgsty/silo',
'bootstrap': {'through': '2026-09-15', 'reconstructed': True},
'points': [{'date': '2026-09-14', 'stars': 100}, {'date': '2026-09-15', 'stars': 105}]}
class HistoryTests(unittest.TestCase):
def test_new_day_preserves_old_counts_and_unstars(self):
before = history()
after = update.update_history(before, '2026-09-16', 103)
self.assertEqual(after['points'][:-1], before['points'])
self.assertEqual(after['points'][-1], {'date': '2026-09-16', 'stars': 103})
self.assertEqual(before, history())
def test_same_day_rerun_replaces_instead_of_appending(self):
first = update.update_history(history(), '2026-09-15', 107)
self.assertEqual(len(first['points']), 2)
self.assertEqual(first, update.update_history(first, '2026-09-15', 107))
def test_missing_days_are_not_invented(self):
result = update.update_history(history(), '2026-09-18', 106)
self.assertEqual([p['date'] for p in result['points']], ['2026-09-14', '2026-09-15', '2026-09-18'])
def test_rejects_wrong_repository_and_corrupt_history(self):
cases = []
wrong = history(); wrong['repository'] = 'someone/else'; cases.append(wrong)
duplicate = history(); duplicate['points'].append(duplicate['points'][-1]); cases.append(duplicate)
unordered = history(); unordered['points'].reverse(); cases.append(unordered)
negative = history(); negative['points'][0]['stars'] = -1; cases.append(negative)
future = history(); future['points'][-1]['date'] = '2026-09-20'; cases.append(future)
for case in cases:
with self.subTest(case=case), self.assertRaises(ValueError):
update.update_history(case, '2026-09-16', 100)
def test_first_run_has_no_fabricated_history(self):
result = update.update_history(None, '2026-09-16', 10)
self.assertFalse(result['bootstrap']['reconstructed'])
self.assertEqual(result['points'], [{'date': '2026-09-16', 'stars': 10}])
class ContributorTests(unittest.TestCase):
def test_retries_truncated_json_before_using_it(self):
with patch('update.request', side_effect=[b'{"partial":', b'{"ok":true}']), patch('update.time.sleep'):
self.assertEqual(update.GitHub('').get('repos/pgsty/silo'), {'ok': True})
def test_paginates_past_one_full_page(self):
class API(update.GitHub):
def __init__(self): self.calls = []
def get(self, path):
self.calls.append(path)
return list(range(100)) if 'page=1&' in path else [100]
api = API()
self.assertEqual(len(list(api.issues('pgsty/silo'))), 101)
self.assertIn('state=all', api.calls[0])
self.assertIn('page=2&', api.calls[1])
def test_bots_deduplication_unmerged_work_and_reviewed_credit(self):
def issue(login, kind='issue', user_type='User'):
item = {'user': {'login': login, 'type': user_type, 'avatar_url': ''}, 'created_at': '2026-09-02T00:00:00Z'}
if kind != 'issue': item['pull_request'] = {'merged_at': None if kind == 'open' else '2026-09-03T00:00:00Z'}
return item
class API:
def issues(self, _repo):
return [issue('alice'), issue('Bob', 'open'), issue('Bob', 'merged'),
issue('Carol', 'open'), issue('Copilot'), issue('robot', user_type='Bot')]
curated = {'repositories': ['pgsty/silo', 'pgsty/mc'], 'bots': ['Copilot'],
'people': [person('Alice', featured=True), person('Reporter'), person('Copilot')]}
result = update.collect_people(API(), curated)
self.assertEqual({p['handle'] for p in result}, {'Alice', 'Bob', 'Carol', 'Reporter'})
self.assertEqual(result[0]['handle'], 'Bob')
self.assertEqual(result[1]['handle'], 'Carol')
self.assertTrue(next(p for p in result if p['handle'] == 'Alice')['featured'])
self.assertEqual(next(p for p in result if p['handle'] == 'Bob')['group'], 'code')
self.assertFalse(next(p for p in result if p['handle'] == 'Carol')['featured'])
def test_newer_reviewed_preview_survives_until_site_catches_up(self):
remote = {'updated': '2026-09-16T03:00:00+00:00'}
cached = {'updated': '2026-09-16T04:00:00+00:00'}
self.assertIs(update.select_curated(remote, cached), cached)
newer = {'updated': '2026-09-17T03:00:00+00:00'}
self.assertIs(update.select_curated(newer, cached), newer)
def test_avatar_failure_reuses_raster_cache(self):
previous = {**person(), 'avatarDataUrl': update.raster_data_url(PNG)}
with patch('update.request', side_effect=URLError('unavailable')):
result = update.add_avatars(None, [{**person(), 'avatarUrl': 'https://avatars.githubusercontent.com/u/1'}], [previous])
self.assertEqual(result[0]['avatarDataUrl'], previous['avatarDataUrl'])
with self.assertRaises(ValueError): update.raster_data_url(b'<svg onload="bad()"/>')
with self.assertRaises(ValueError): update.cached_avatar({'avatarDataUrl': 'data:image/svg+xml;base64,PHN2Zy8+'})
def test_fetch_failure_leaves_published_assets_untouched(self):
class API:
def get(self, path):
if path == 'repos/pgsty/silo': return {'full_name': 'pgsty/silo', 'stargazers_count': 106}
raise URLError('roster unavailable')
with tempfile.TemporaryDirectory() as directory:
out = Path(directory)
original = json.dumps(history())
(out / 'history.json').write_text(original)
(out / 'contributors-light.svg').write_text('previous image')
with self.assertRaises(URLError): update.refresh(out, API(), Path('.'))
self.assertEqual((out / 'history.json').read_text(), original)
self.assertEqual((out / 'contributors-light.svg').read_text(), 'previous image')
class RenderTests(unittest.TestCase):
def test_real_emblem_generates_clean_xml(self):
emblem = render.read_emblem(Path(__file__).resolve().parents[2] / '.github/silo.svg')
svg = render.contributors('light', [person()], '2026-09-16', emblem)
ET.fromstring(svg)
self.assertTrue(all(line == line.rstrip() for line in svg.splitlines()))
def test_all_avatars_fit_when_the_roster_grows(self):
people = [{**person(f'person-{i}'), 'avatarDataUrl': update.raster_data_url(PNG)} for i in range(151)]
for theme in ('light', 'dark'):
root = ET.fromstring(render.contributors(theme, people, '2026-09-16', ''))
images = root.findall('.//s:image', NS)
self.assertEqual(len(images), 151)
footer = float(root.attrib['height']) - 42
self.assertTrue(all(float(i.attrib['y']) + float(i.attrib['height']) < footer for i in images))
self.assertTrue(all(i.attrib['href'].startswith('data:image/png;base64,') for i in images))
def test_untrusted_text_is_escaped(self):
data = [{**person(), 'what': '<script>alert("x")</script> & contributions'}]
svg = render.contributors('light', data, '2026-09-16', '')
root = ET.fromstring(svg)
self.assertEqual(root.findall('.//s:script', NS), [])
self.assertIn('&lt;script&gt;', svg)
def test_single_point_and_decreasing_star_history_render(self):
for data in (update.update_history(None, '2026-09-16', 0), update.update_history(history(), '2026-09-16', 90)):
for theme in ('light', 'dark'):
svg = render.stars(theme, data, '2026-09-16', '')
ET.fromstring(svg)
self.assertNotIn('nan', svg.lower())
self.assertNotIn('inf', svg.lower())
if __name__ == '__main__':
unittest.main()
+295
View File
@@ -0,0 +1,295 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Feng Ruohang
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Refresh the generated-asset checkout; publishing is handled by the workflow."""
import argparse
import base64
from concurrent.futures import ThreadPoolExecutor
from datetime import date, datetime, timezone
import json
from http.client import IncompleteRead
import os
from pathlib import Path
import re
import sys
import time
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse
from urllib.request import Request, urlopen
import xml.etree.ElementTree as ET
import yaml
import render
REPOSITORY = 'pgsty/silo'
SOURCE = 'repos/pgsty/silo.pgsty.com/contents/data/home/contributors.yaml?ref=main'
GROUPS = ('code', 'proposed', 'reports')
HANDLE = re.compile(r'[A-Za-z0-9][A-Za-z0-9-]{0,38}\Z')
def request(url, token='', limit=8 * 1024 * 1024):
headers = {'User-Agent': 'silo-repository-cards', 'Accept': 'application/vnd.github+json'}
if urlparse(url).netloc == 'api.github.com':
headers['X-GitHub-Api-Version'] = '2022-11-28'
if token:
headers['Authorization'] = f'Bearer {token}'
for attempt in range(3):
try:
with urlopen(Request(url, headers=headers), timeout=25) as response:
data = response.read(limit + 1)
if len(data) > limit:
raise ValueError('Response exceeds the size limit')
expected = response.headers.get('Content-Length')
if expected is not None and len(data) != int(expected):
raise URLError('Incomplete response body')
return data
except HTTPError as exc:
if exc.code < 500 or attempt == 2:
raise
except (URLError, TimeoutError, IncompleteRead):
if attempt == 2:
raise
time.sleep(attempt + 1)
class GitHub:
def __init__(self, token):
self.token = token
def get(self, path):
for attempt in range(3):
try:
return json.loads(request('https://api.github.com/' + path, self.token))
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
if attempt == 2:
raise ValueError(f'Incomplete or invalid GitHub JSON: {path}') from exc
time.sleep(attempt + 1)
def issues(self, repository):
page = 1
while True:
batch = self.get(f'repos/{repository}/issues?state=all&per_page=100&page={page}&sort=created&direction=asc')
if not isinstance(batch, list):
raise ValueError(f'Invalid issues response for {repository}')
yield from batch
if len(batch) < 100:
return
page += 1
def curated_snapshot(data, revision):
updated = str(data['updated'])
datetime.fromisoformat(updated)
repositories = [item['repo'] for item in data['repositories']]
if not repositories or any(not re.fullmatch(r'pgsty/[A-Za-z0-9_.-]+', repo) for repo in repositories):
raise ValueError('Invalid contributor repository scope')
people = []
for group in GROUPS:
for entry in data[group]:
if not HANDLE.fullmatch(entry['handle']):
raise ValueError('Invalid GitHub contributor handle')
people.append({
'handle': entry['handle'], 'group': group,
'featured': bool(entry.get('featured')), 'what': entry['what'],
'firstContribution': str(entry.get('firstContribution', '9999-12-31')),
})
if not people or len({p['handle'].lower() for p in people}) != len(people):
raise ValueError('Empty or duplicate contributor roster')
return {'updated': updated, 'revision': revision, 'repositories': repositories,
'bots': data.get('bots', ['Copilot', 'dependabot[bot]']), 'people': people}
def select_curated(remote, cached):
# The initial, approved preview can contain reviewed credit not published by
# the companion site yet. Keep that newer snapshot until the site catches up.
if cached and datetime.fromisoformat(cached['updated']) > datetime.fromisoformat(remote['updated']):
return cached
return remote
def collect_people(api, curated):
bots = {name.lower() for name in curated['bots']}
people = {p['handle'].lower(): dict(p) for p in curated['people']
if p['handle'].lower() not in bots and not p['handle'].lower().endswith('[bot]')}
order = {p['handle'].lower(): index for index, p in enumerate(curated['people'])}
for repository in curated['repositories']:
print(f'Reading issue and PR authors: {repository}', flush=True)
for issue in api.issues(repository):
user = issue.get('user') or {}
handle = user.get('login', '')
key = handle.lower()
if user.get('type') != 'User' or key in bots or key.endswith('[bot]'):
continue
if not HANDLE.fullmatch(handle):
raise ValueError('Invalid issue author')
pr = issue.get('pull_request')
group = 'code' if pr and pr.get('merged_at') else 'proposed' if pr else 'reports'
first = issue['created_at'][:10]
date.fromisoformat(first)
person = people.setdefault(key, {
'handle': handle, 'group': group, 'featured': False,
'what': 'Contributed an issue or pull request to SILO and related projects',
'firstContribution': first,
})
person['avatarUrl'] = user.get('avatar_url', '')
person['firstContribution'] = min(person['firstContribution'], first)
if GROUPS.index(group) < GROUPS.index(person['group']):
person['group'] = group
if not people:
raise ValueError('No human contributors were collected')
return sorted(people.values(), key=lambda p: (
GROUPS.index(p['group']), not p['featured'],
order.get(p['handle'].lower(), len(order)), p['firstContribution'], p['handle'].lower()))
def raster_data_url(data):
if data.startswith(b'\x89PNG\r\n\x1a\n'):
mime = 'image/png'
elif data.startswith(b'\xff\xd8\xff'):
mime = 'image/jpeg'
elif data.startswith((b'GIF87a', b'GIF89a')):
mime = 'image/gif'
elif data[:4] == b'RIFF' and data[8:12] == b'WEBP':
mime = 'image/webp'
else:
raise ValueError('Avatar is not a raster image')
return f'data:{mime};base64,' + base64.b64encode(data).decode('ascii')
def cached_avatar(person):
value = person.get('avatarDataUrl', '')
if not value:
return ''
prefix, encoded = value.split(',', 1)
if prefix not in ('data:image/png;base64', 'data:image/jpeg;base64', 'data:image/gif;base64', 'data:image/webp;base64'):
raise ValueError('Invalid cached avatar format')
raw = base64.b64decode(encoded, validate=True)
if len(raw) > 512 * 1024 or raster_data_url(raw) != value:
raise ValueError('Invalid cached avatar')
return value
def add_avatars(api, people, previous):
cached = {p['handle'].lower(): cached_avatar(p) for p in previous}
def update(person):
person = dict(person)
try:
url = person.pop('avatarUrl', '') or api.get('users/' + person['handle'])['avatar_url']
parsed = urlparse(url)
if parsed.scheme != 'https' or parsed.netloc != 'avatars.githubusercontent.com':
raise ValueError('Unexpected avatar host')
data = request(url + ('&' if '?' in url else '?') + 's=96', limit=512 * 1024)
person['avatarDataUrl'] = raster_data_url(data)
except (HTTPError, URLError, TimeoutError, IncompleteRead, ValueError, KeyError) as exc:
person.pop('avatarUrl', None)
person['avatarDataUrl'] = cached.get(person['handle'].lower(), '')
print(f'Avatar fallback for @{person["handle"]}: {type(exc).__name__}', file=sys.stderr)
return person
with ThreadPoolExecutor(max_workers=6) as pool:
return list(pool.map(update, people))
def update_history(history, day, stars):
date.fromisoformat(day)
if type(stars) is not int or stars < 0:
raise ValueError('Invalid repository star count')
if history is None:
history = {'repository': REPOSITORY, 'bootstrap': {'through': day, 'reconstructed': False}, 'points': []}
if history['repository'] != REPOSITORY:
raise ValueError('Star history belongs to a different repository')
date.fromisoformat(history['bootstrap']['through'])
dates = []
for point in history['points']:
date.fromisoformat(point['date'])
if type(point['stars']) is not int or point['stars'] < 0:
raise ValueError('Invalid historical star count')
dates.append(point['date'])
if dates != sorted(set(dates)) or any(d > day for d in dates):
raise ValueError('History contains duplicate, unordered, or future dates')
# Replace today's observation, preserve previous days, and allow unstars.
points = [dict(p) for p in history['points'] if p['date'] != day]
points.append({'date': day, 'stars': stars})
return {**history, 'points': points}
def read_json(path, default=None):
return json.loads(path.read_text()) if path.exists() else default
def refresh(output, api, source_root):
day = datetime.now(timezone.utc).date().isoformat()
metadata = api.get('repos/' + REPOSITORY)
if metadata['full_name'].lower() != REPOSITORY:
raise ValueError('Unexpected repository metadata')
history = update_history(read_json(output / 'history.json'), day, metadata['stargazers_count'])
source = api.get(SOURCE)
reviewed = yaml.safe_load(base64.b64decode(source['content'], validate=False))
curated = select_curated(curated_snapshot(reviewed, source['sha']), read_json(output / 'curated.json'))
people = collect_people(api, curated)
previous = read_json(output / 'contributors.json', {}).get('people', [])
people = add_avatars(api, people, previous)
emblem = render.read_emblem(source_root / '.github/silo.svg')
payloads = {}
for theme in ('light', 'dark'):
payloads[f'contributors-{theme}.svg'] = render.contributors(theme, people, day, emblem) + '\n'
payloads[f'star-history-{theme}.svg'] = render.stars(theme, history, day, emblem) + '\n'
for svg in payloads.values():
ET.fromstring(svg)
for name, data in {
'history.json': history,
'curated.json': curated,
'contributors.json': {'repository': REPOSITORY, 'updated': day, 'people': people},
}.items():
payloads[name] = json.dumps(data, indent=2, ensure_ascii=False) + '\n'
payloads['README.md'] = f'''# SILO repository cards
Generated by [Repository Cards](https://github.com/pgsty/silo/actions/workflows/repository-cards.yml)
at 00:00 UTC daily (08:00 Asia/Shanghai). GitHub may queue scheduled runs.
Snapshot: {day}. {metadata['stargazers_count']:,} stars; {len(people)} community contributors.
- `contributors-light.svg` / `contributors-dark.svg`: human issue and PR authors across the SILO project scope, plus reviewed acknowledgements. Bots are excluded. Gold rings follow the reviewed companion-site roster; new authors are collected automatically.
- `star-history-light.svg` / `star-history-dark.svg`: initial history reconstructed from the then-current stargazers; later points are daily observed totals, including decreases. Missing days are not fabricated.
- `curated.json`: a cache of reviewed contributor credit from `pgsty/silo.pgsty.com/data/home/contributors.yaml`. The approved initial preview may be newer than the published site; a newer reviewed snapshot is retained until the site catches up.
- `contributors.json`: generated contributor data and embedded raster avatars. Failed avatar refreshes use the previous image, or an initial when no image is available.
- `history.json`: persistent daily totals. Keep this file when regenerating images.
The SVGs are self-contained. Source and instructions live on the default branch;
this branch contains generated assets only. Do not merge it into `main`.
'''
# Collect and validate everything before touching the publication checkout.
output.mkdir(parents=True, exist_ok=True)
for filename, text in payloads.items():
(output / filename).write_text(text)
print(f'{day}: {len(people)} contributors; {metadata["stargazers_count"]:,} stars; {len(history["points"])} history points')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output', type=Path, required=True)
args = parser.parse_args()
configured = os.environ.get('GITHUB_REPOSITORY', REPOSITORY)
if configured.lower() != REPOSITORY:
raise SystemExit('This workflow is scoped to pgsty/silo')
refresh(args.output, GitHub(os.environ.get('GH_TOKEN', '')), Path(__file__).resolve().parents[2])
if __name__ == '__main__':
main()
+1 -1
View File
@@ -113,7 +113,7 @@ helm_run template my-release "${new_chart}" \
go run ./buildscripts/helm-migration-guard "${old_render}" "${new_render}"
helm_run package "${new_chart}" --destination "${output_dir}" >/dev/null
test -s "${work_dir}/silo-7.0.2.tgz"
test -s "${work_dir}/silo-7.0.3.tgz"
if find "${work_dir}" -maxdepth 1 -type f -name 'minio-*.tgz' | grep -q .; then
echo "Helm packaging emitted a legacy MinIO chart name" >&2
exit 1
+1
View File
@@ -163,6 +163,7 @@ func registerAdminRouter(router *mux.Router, enableConfigOps bool) {
// StorageInfo operations
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/storageinfo").HandlerFunc(adminMiddleware(adminAPI.StorageInfoHandler, traceAllFlag))
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/multipart-preflight").HandlerFunc(adminMiddleware(adminAPI.MultipartPreflightHandler, traceAllFlag))
// DataUsageInfo operations
adminRouter.Methods(http.MethodGet).Path(adminVersion + "/datausageinfo").HandlerFunc(adminMiddleware(adminAPI.DataUsageInfoHandler, traceAllFlag))
// Metrics operation
+25 -1
View File
@@ -450,6 +450,9 @@ const (
ErrAdminNoSecretKey
ErrIAMNotInitialized
ErrMultipartListingLegacy
ErrMultipartListingIdentity
ErrSlowDown
apiErrCodeEnd // This is used only for the testing code
)
@@ -547,7 +550,7 @@ var errorCodes = errorCodeMap{
},
ErrBadDigest: {
Code: "BadDigest",
Description: "The Content-Md5 you specified did not match what we received.",
Description: "The Content-MD5 you specified did not match what we received.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrEntityTooSmall: {
@@ -1336,6 +1339,21 @@ var errorCodes = errorCodeMap{
Description: "IAM sub-system not initialized yet, please try again.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMultipartListingLegacy: {
Code: "MultipartListingNotReady",
Description: "Legacy multipart uploads prevent a complete listing. Upgrade all writers, drain old uploads and run the multipart preflight check.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMultipartListingIdentity: {
Code: "MultipartListingMetadataInvalid",
Description: "Multipart upload metadata is inconsistent. Run the multipart preflight check to locate the affected storage set.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrSlowDown: {
Code: "SlowDown",
Description: "Please reduce your request rate",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrBucketMetadataNotInitialized: {
Code: "XMinioBucketMetadataNotInitialized",
Description: "Bucket metadata not initialized yet, please try again.",
@@ -2173,6 +2191,10 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
err = unwrapAll(err)
switch err {
case errMultipartListingLegacy:
apiErr = ErrMultipartListingLegacy
case errMultipartListingIdentity:
apiErr = ErrMultipartListingIdentity
case errCompleteMultipartChecksumMismatch, errCompleteMultipartChecksumTypeMismatch:
apiErr = ErrBadDigest
case errMissingPartChecksum:
@@ -2303,6 +2325,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
}
switch err.(type) {
case SlowDown:
apiErr = ErrSlowDown
case StorageFull:
apiErr = ErrStorageFull
case hash.BadDigest:
+1 -1
View File
@@ -41,7 +41,7 @@ import (
const (
maxObjectList = 1000 // Limit number of objects in a listObjectsResponse/listObjectsVersionsResponse.
maxDeleteList = 1000 // Limit number of objects deleted in a delete call.
maxUploadsList = 10000 // Limit number of uploads in a listUploadsResponse.
maxUploadsList = 1000 // Limit number of uploads in a listUploadsResponse.
maxPartsList = 10000 // Limit number of parts in a listPartsResponse.
)
File diff suppressed because one or more lines are too long
-45
View File
@@ -28,13 +28,10 @@ import (
"mime"
"net/http"
"net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
"github.com/minio/minio/internal/etag"
"github.com/minio/minio/internal/hash"
xhttp "github.com/minio/minio/internal/http"
@@ -733,48 +730,6 @@ func setAuthMiddleware(h http.Handler) http.Handler {
})
}
func isPutRetentionAllowed(bucketName, objectName string, retDays int, retDate time.Time, retMode objectlock.RetMode, byPassSet bool, r *http.Request, cred auth.Credentials, owner bool) (s3Err APIErrorCode) {
var retSet bool
if cred.AccessKey == "" {
return ErrAccessDenied
}
conditions := getConditionValues(r, "", cred)
conditions["object-lock-mode"] = []string{string(retMode)}
conditions["object-lock-retain-until-date"] = []string{retDate.UTC().Format(time.RFC3339)}
if retDays > 0 {
conditions["object-lock-remaining-retention-days"] = []string{strconv.Itoa(retDays)}
}
if retMode == objectlock.RetGovernance && byPassSet {
byPassSet = globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.BypassGovernanceRetentionAction,
BucketName: bucketName,
ObjectName: objectName,
ConditionValues: conditions,
IsOwner: owner,
Claims: cred.Claims,
})
}
if globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.PutObjectRetentionAction,
BucketName: bucketName,
ConditionValues: conditions,
ObjectName: objectName,
IsOwner: owner,
Claims: cred.Claims,
}) {
retSet = true
}
if byPassSet || retSet {
return ErrNone
}
return ErrAccessDenied
}
// isPutActionAllowed - check if PUT operation is allowed on the resource, this
// call verifies bucket policies and IAM policies, supports multi user
// checks etc.
+5 -1
View File
@@ -583,7 +583,11 @@ func toObjectInfo(bucket, object string, objInfo minio.ObjectInfo) ObjectInfo {
func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLayer, remoteClnt *minio.Client, entries []ObjectInfo, prefix string) error {
input := make(chan minio.SnowballObject, 1)
opts := minio.SnowballOptions{
Opts: minio.PutObjectOptions{},
Opts: minio.PutObjectOptions{
Internal: minio.AdvancedPutOptions{
ReplicationRequest: true,
},
},
InMemory: *r.Source.Snowball.InMemory,
Compress: *r.Source.Snowball.Compress,
SkipErrs: *r.Source.Snowball.SkipErrs,
+6 -8
View File
@@ -277,14 +277,6 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
return
}
if keyMarker != "" {
// Marker not common with prefix is not implemented.
if !HasPrefix(keyMarker, prefix) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrNotImplemented), r.URL)
return
}
}
listMultipartsInfo, err := objectAPI.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
@@ -1281,6 +1273,12 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
return
}
if isDirObject(object) {
if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, false); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
}
opts.WantChecksum = checksum
fanOutOpts := fanOutOptions{Checksum: checksum}
+3 -3
View File
@@ -425,7 +425,7 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
shouldPass: true,
},
// Test case - 4.
// Setting Invalid prefix and marker combination.
// A key marker outside the prefix is valid and produces an empty page.
{
bucket: bucketName,
prefix: "asia",
@@ -435,8 +435,8 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
maxUploads: "0",
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
expectedRespStatus: http.StatusNotImplemented,
shouldPass: false,
expectedRespStatus: http.StatusOK,
shouldPass: true,
},
// Test case - 5.
// Invalid upload id and marker combination.
+122 -61
View File
@@ -20,8 +20,8 @@ package cmd
import (
"context"
"errors"
"math"
"net/http"
"strconv"
"strings"
"time"
@@ -164,76 +164,129 @@ func enforceRetentionBypassForDelete(ctx context.Context, r *http.Request, bucke
return nil
}
// enforceRetentionBypassForPut enforces whether an existing object under governance can be overwritten
// with governance bypass headers set in the request.
// Objects under site wide WORM cannot be overwritten.
// For objects in "Governance" mode, overwrite is allowed if a) object retention date is past OR
// governance bypass headers are set and user has governance bypass permissions.
// Objects in compliance mode can be overwritten only if retention date is being extended. No mode change is permitted.
// isRetentionActionAllowed evaluates action for a PutObjectRetention request
// with the object-lock condition keys derived from the requested retention.
// The keys come from the request, not from the stored version, so the
// handler can decide s3:PutObjectRetention before reading any object state.
func isRetentionActionAllowed(action policy.Action, bucket, object string, objRetention *objectlock.ObjectRetention, r *http.Request, cred auth.Credentials, owner bool) bool {
if cred.AccessKey == "" {
return false
}
t, err := objectlock.UTCNowNTP()
if err != nil {
return false
}
conditions := getConditionValues(r, "", cred)
conditions["object-lock-mode"] = []string{string(objRetention.Mode)}
conditions["object-lock-retain-until-date"] = []string{objRetention.RetainUntilDate.UTC().Format(time.RFC3339)}
// Relative days from now, for "object-lock-remaining-retention-days".
// Clearing retention (<Retention/>) has a zero RetainUntilDate, which must
// evaluate as 0 remaining days rather than a large positive math.Abs delta
// from year 0001.
days := 0
if !objRetention.RetainUntilDate.IsZero() && objRetention.RetainUntilDate.After(t) {
// Time.Sub saturates after about 292 years. Count seconds instead so
// maximum-retention policies also apply to distant RFC3339 dates.
seconds := objRetention.RetainUntilDate.Unix() - t.Unix()
days = int(seconds / 86400)
if seconds%86400 != 0 || objRetention.RetainUntilDate.Nanosecond() > t.Nanosecond() {
days++
}
}
conditions["object-lock-remaining-retention-days"] = []string{strconv.Itoa(days)}
return globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: action,
BucketName: bucket,
ConditionValues: conditions,
ObjectName: object,
IsOwner: owner,
Claims: cred.Claims,
})
}
// enforceRetentionBypassForPut enforces the stored retention of oi against a
// PutObjectRetention request. The caller must already hold
// s3:PutObjectRetention; this only decides changes that depend on the stored
// state. Expired or absent retention may be changed freely. Unexpired
// GOVERNANCE retention may be shortened, removed or have its mode changed only
// with the bypass header and s3:BypassGovernanceRetention. Unexpired
// COMPLIANCE retention may only be extended.
func enforceRetentionBypassForPut(ctx context.Context, r *http.Request, oi ObjectInfo, objRetention *objectlock.ObjectRetention, cred auth.Credentials, owner bool) error {
byPassSet := objectlock.IsObjectLockGovernanceBypassSet(r.Header)
ret := objectlock.GetObjectRetentionMeta(oi.UserDefined)
if !ret.Mode.Valid() {
return nil
}
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
if ret.RetainUntilDate.Before(t) {
return nil
}
// Pass in relative days from current time, to additionally
// to verify "object-lock-remaining-retention-days" policy if any.
days := int(math.Ceil(math.Abs(objRetention.RetainUntilDate.Sub(t).Hours()) / 24))
ret := objectlock.GetObjectRetentionMeta(oi.UserDefined)
if ret.Mode.Valid() {
// Retention has expired you may change whatever you like.
if ret.RetainUntilDate.Before(t) {
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time,
objRetention.Mode, byPassSet, r, cred,
owner)
if apiErr == ErrAccessDenied {
return errAuthentication
}
return nil
weakens := objRetention.Mode != ret.Mode || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time)
if !weakens {
return nil
}
switch ret.Mode {
case objectlock.RetGovernance:
// https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-bypass-governance
if !objectlock.IsObjectLockGovernanceBypassSet(r.Header) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
switch ret.Mode {
case objectlock.RetGovernance:
govPerm := isPutRetentionAllowed(oi.Bucket, oi.Name, days,
objRetention.RetainUntilDate.Time, objRetention.Mode,
byPassSet, r, cred, owner)
// Governance mode retention period cannot be shortened, if x-amz-bypass-governance is not set.
if !byPassSet {
if objRetention.Mode != objectlock.RetGovernance || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
}
if govPerm == ErrAccessDenied {
return errAuthentication
}
return nil
case objectlock.RetCompliance:
// Compliance retention mode cannot be changed or shortened.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html#object-lock-retention-modes
if objRetention.Mode != objectlock.RetCompliance || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time, objRetention.Mode,
false, r, cred, owner)
if apiErr == ErrAccessDenied {
return errAuthentication
}
return nil
if !isRetentionActionAllowed(policy.BypassGovernanceRetentionAction, oi.Bucket, oi.Name, objRetention, r, cred, owner) {
return errAuthentication
}
return nil
} // No pre-existing retention metadata present.
default:
// Compliance retention mode cannot be changed or shortened.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html#object-lock-retention-modes
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
}
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time,
objRetention.Mode, byPassSet, r, cred, owner)
if apiErr == ErrAccessDenied {
return errAuthentication
// checkObjectLockedForOverwrite re-checks under the object write lock whether an
// in-place write to the null version (such as a directory object) would
// overwrite a locked version in an Object Lock bucket.
func checkObjectLockedForOverwrite(ctx context.Context, bucket, object string, getObjectInfoFn GetObjectInfoFn) error {
if isMinioMetaBucketName(bucket) {
return nil
}
retentionCfg, err := globalBucketObjectLockSys.Get(bucket)
if err != nil {
return err
}
if !retentionCfg.LockEnabled {
return nil
}
objInfo, err := getObjectInfoFn(ctx, bucket, object, ObjectOptions{
VersionID: nullVersionID,
NoLock: true,
NoAuditLog: true,
})
if err != nil {
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || errors.As(err, &MethodNotAllowed{}) {
return nil
}
return err
}
if objectlock.GetObjectLegalHoldMeta(objInfo.UserDefined).Status == objectlock.LegalHoldOn {
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined)
if r.Mode == objectlock.RetCompliance || r.Mode == objectlock.RetGovernance {
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
if r.RetainUntilDate.After(t) {
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
}
return nil
}
@@ -274,16 +327,24 @@ func checkPutObjectLockAllowed(ctx context.Context, rq *http.Request, bucket, ob
if err != nil {
return mode, retainDate, legalHold, toAPIErrorCode(ctx, err)
}
if isDirObject(object) && opts.VersionID == "" {
opts.VersionID = nullVersionID
}
if opts.VersionID != "" && !replicaTrusted {
if objInfo, err := getObjectInfoFn(ctx, bucket, object, opts); err == nil {
objInfo, err := getObjectInfoFn(ctx, bucket, object, opts)
if err != nil {
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) && !errors.As(err, &MethodNotAllowed{}) {
return mode, retainDate, legalHold, toAPIErrorCode(ctx, err)
}
} else {
r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined)
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return mode, retainDate, legalHold, ErrObjectLocked
}
if r.Mode == objectlock.RetCompliance && r.RetainUntilDate.After(t) {
if (r.Mode == objectlock.RetCompliance || r.Mode == objectlock.RetGovernance) && r.RetainUntilDate.After(t) {
return mode, retainDate, legalHold, ErrObjectLocked
}
mode = r.Mode
+3
View File
@@ -418,6 +418,9 @@ func getReplicationState(rinfos replicatedInfos, prevState ReplicationState, vID
for _, rinfo := range rinfos.Targets {
if rinfo.ResyncTimestamp != "" {
if rs.ResetStatusesMap == nil {
rs.ResetStatusesMap = make(map[string]string)
}
rs.ResetStatusesMap[targetResetHeader(rinfo.Arn)] = rinfo.ResyncTimestamp
}
}
+175 -63
View File
@@ -425,6 +425,7 @@ func checkReplicateDelete(ctx context.Context, bucket string, dobj ObjectToDelet
// the mere presence or absence of the target version.
func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, objectAPI ObjectLayer) replicatedInfos {
var replicationStatus replication.StatusType
isPurge := dobj.isVersionPurge()
bucket := dobj.Bucket
versionID := dobj.DeleteMarkerVersionID
if versionID == "" {
@@ -484,6 +485,7 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
lk := objectAPI.NewNSLock(bucket, "/[replicate]/"+dobj.ObjectName)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
dobj.RetryCount++
globalReplicationPool.Get().queueMRFSave(dobj.ToMRFEntry())
sendEvent(eventArgs{
BucketName: bucket,
@@ -502,6 +504,35 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
if !isPurge && dobj.DeleteMarkerVersionID != "" {
// A creation task carries the marker as it looked when it was queued
// by the DELETE handler, a GET/HEAD/LIST heal, the scanner or MRF.
// While it waited for this lock, another frontend may have purged the
// marker and replicated that purge. The targets no longer hold the
// marker, so the queued creation would recreate it from the stale
// snapshot. Confirm the source version under the lock first.
switch deleteMarkerCreationState(ctx, objectAPI, dobj) {
case creationStale:
return replicatedInfos{}
case creationUnverified:
dobj.RetryCount++
globalReplicationPool.Get().queueMRFSave(dobj.ToMRFEntry())
sendEvent(eventArgs{
BucketName: bucket,
Object: ObjectInfo{
Bucket: bucket,
Name: dobj.ObjectName,
VersionID: versionID,
DeleteMarker: dobj.DeleteMarker,
},
UserAgent: "Internal: [Replication]",
Host: globalLocalNodeName,
EventName: event.ObjectReplicationNotTracked,
})
return replicatedInfos{}
}
}
rinfos := replicatedInfos{Targets: make([]replicatedTargetInfo, 0, len(dsc.targetsMap))}
var wg sync.WaitGroup
var mu sync.Mutex
@@ -546,25 +577,38 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
replicationStatus = rinfos.ReplicationStatus()
prevStatus := dobj.DeleteMarkerReplicationStatus()
if dobj.VersionID != "" {
prevStatus = replication.StatusType(dobj.VersionPurgeStatus())
replicationStatus = replication.StatusType(rinfos.VersionPurgeStatus())
if isPurge {
prevStatus = purgeReplicationStatus(dobj.VersionPurgeStatus())
replicationStatus = purgeReplicationStatus(rinfos.VersionPurgeStatus())
}
// to decrement pending count later.
for _, rinfo := range rinfos.Targets {
if rinfo.ReplicationStatus != rinfo.PrevReplicationStatus {
globalReplicationStats.Load().Update(dobj.Bucket, rinfo, replicationStatus,
prevStatus)
status, previous := rinfo.ReplicationStatus, rinfo.PrevReplicationStatus
if isPurge {
status = purgeReplicationStatus(rinfo.VersionPurgeStatus)
previous = purgeReplicationStatus(dobj.ReplicationState.PurgeTargets[rinfo.Arn])
}
if status != previous {
globalReplicationStats.Load().Update(dobj.Bucket, rinfo, status, previous)
}
}
eventName := event.ObjectReplicationComplete
if replicationStatus == replication.Failed {
eventName = event.ObjectReplicationFailed
dobj.RetryCount++
globalReplicationPool.Get().queueMRFSave(dobj.ToMRFEntry())
}
drs := getReplicationState(rinfos, dobj.ReplicationState, dobj.VersionID)
if isPurge {
// A purge must not rewrite the marker's creation/replica metadata.
// Multiple serialized empty target statuses can parse as nonempty,
// so explicitly send an empty creation update to the metadata writer.
drs.ReplicationStatusInternal = ""
drs.Targets = nil
drs.ReplicaStatus = ""
}
if replicationStatus != prevStatus {
drs.ReplicationTimeStamp = UTCNow()
}
@@ -606,6 +650,7 @@ func replicateDelete(ctx context.Context, dobj DeletedObjectReplicationInfo, obj
}
func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationInfo, tgt *TargetClient) (rinfo replicatedTargetInfo) {
isPurge := dobj.isVersionPurge()
versionID := dobj.DeleteMarkerVersionID
if versionID == "" {
versionID = dobj.VersionID
@@ -615,42 +660,50 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
rinfo.OpType = dobj.OpType
rinfo.endpoint = tgt.EndpointURL().Host
rinfo.secure = tgt.EndpointURL().Scheme == "https"
// A purge leaves ReplicationStatus empty: the metadata writer interprets
// that as preserving the entire creation-status block, including targets
// outside this fan-out. Only VersionPurgeStatus records the purge outcome.
defer func() {
if rinfo.ReplicationStatus == replication.Completed && tgt.ResetID != "" && dobj.OpType == replication.ExistingObjectReplicationType {
completed := rinfo.ReplicationStatus == replication.Completed
if isPurge {
completed = rinfo.VersionPurgeStatus == replication.VersionPurgeComplete
}
if completed && tgt.ResetID != "" && dobj.OpType == replication.ExistingObjectReplicationType {
rinfo.ResyncTimestamp = fmt.Sprintf("%s;%s", UTCNow().Format(http.TimeFormat), tgt.ResetID)
}
}()
if dobj.VersionID == "" && rinfo.PrevReplicationStatus == replication.Completed && dobj.OpType != replication.ExistingObjectReplicationType {
if !isPurge && rinfo.PrevReplicationStatus == replication.Completed && dobj.OpType != replication.ExistingObjectReplicationType {
rinfo.ReplicationStatus = rinfo.PrevReplicationStatus
return rinfo
}
if dobj.VersionID != "" && rinfo.VersionPurgeStatus == replication.VersionPurgeComplete {
if isPurge && rinfo.VersionPurgeStatus == replication.VersionPurgeComplete {
return rinfo
}
if globalBucketTargetSys.isOffline(tgt.EndpointURL()) {
replLogOnceIf(ctx, fmt.Errorf("remote target is offline for bucket:%s arn:%s", dobj.Bucket, tgt.ARN), "replication-target-offline-delete-"+tgt.ARN)
rinfo.Err = fmt.Errorf("remote target is offline for bucket:%s arn:%s", dobj.Bucket, tgt.ARN)
replLogOnceIf(ctx, rinfo.Err, "replication-target-offline-delete-"+tgt.ARN)
sendEvent(eventArgs{
BucketName: dobj.Bucket,
Object: ObjectInfo{
Bucket: dobj.Bucket,
Name: dobj.ObjectName,
VersionID: dobj.VersionID,
VersionID: versionID,
DeleteMarker: dobj.DeleteMarker,
},
UserAgent: "Internal: [Replication]",
Host: globalLocalNodeName,
EventName: event.ObjectReplicationNotTracked,
})
if dobj.VersionID == "" {
rinfo.ReplicationStatus = replication.Failed
} else {
if isPurge {
rinfo.VersionPurgeStatus = replication.VersionPurgeFailed
} else {
rinfo.ReplicationStatus = replication.Failed
}
return rinfo
}
// early return if already replicated delete marker for existing object replication/ healing delete markers
if dobj.DeleteMarkerVersionID != "" {
if !isPurge && dobj.DeleteMarkerVersionID != "" {
toi, err := tgt.StatObject(ctx, tgt.Bucket, dobj.ObjectName, minio.StatObjectOptions{
VersionID: versionID,
Internal: minio.AdvancedGetOptions{
@@ -662,16 +715,10 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
switch {
case isErrMethodNotAllowed(serr):
// delete marker already replicated
if dobj.VersionID == "" && rinfo.VersionPurgeStatus.Empty() {
rinfo.ReplicationStatus = replication.Completed
return rinfo
}
rinfo.ReplicationStatus = replication.Completed
return rinfo
case isErrObjectNotFound(serr), isErrVersionNotFound(serr):
// version being purged is already not found on target.
if !rinfo.VersionPurgeStatus.Empty() {
rinfo.VersionPurgeStatus = replication.VersionPurgeComplete
return rinfo
}
// The marker still needs to be created on the target.
case isErrReadQuorum(serr), isErrWriteQuorum(serr):
// destination has some quorum issues, perform removeObject() anyways
// to complete the operation.
@@ -691,7 +738,7 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
rmErr := tgt.RemoveObject(ctx, tgt.Bucket, dobj.ObjectName, minio.RemoveObjectOptions{
VersionID: versionID,
Internal: minio.AdvancedRemoveOptions{
ReplicationDeleteMarker: dobj.DeleteMarkerVersionID != "",
ReplicationDeleteMarker: !isPurge && dobj.DeleteMarkerVersionID != "",
ReplicationMTime: dobj.DeleteMarkerMTime.Time,
ReplicationStatus: minio.ReplicationStatusReplica,
ReplicationRequest: true, // always set this to distinguish between `mc mirror` replication and serverside
@@ -699,20 +746,20 @@ func replicateDeleteToTarget(ctx context.Context, dobj DeletedObjectReplicationI
})
if rmErr != nil {
rinfo.Err = rmErr
if dobj.VersionID == "" {
rinfo.ReplicationStatus = replication.Failed
} else {
if isPurge {
rinfo.VersionPurgeStatus = replication.VersionPurgeFailed
} else {
rinfo.ReplicationStatus = replication.Failed
}
replLogIf(ctx, fmt.Errorf("unable to replicate delete marker to %s: %s/%s(%s): %w", tgt.EndpointURL(), tgt.Bucket, dobj.ObjectName, versionID, rmErr))
if rmErr != nil && minio.IsNetworkOrHostDown(rmErr, true) && !globalBucketTargetSys.isOffline(tgt.EndpointURL()) {
globalBucketTargetSys.markOffline(tgt.EndpointURL())
}
} else {
if dobj.VersionID == "" {
rinfo.ReplicationStatus = replication.Completed
} else {
if isPurge {
rinfo.VersionPurgeStatus = replication.VersionPurgeComplete
} else {
rinfo.ReplicationStatus = replication.Completed
}
}
return rinfo
@@ -781,6 +828,18 @@ func (m caseInsensitiveMap) Lookup(key string) (string, bool) {
return "", false
}
// replicationTaggingTimestamp carries a recorded removal even when tags are
// empty. Only legacy nonempty tags use ModTime; absence is not a tombstone.
func replicationTaggingTimestamp(objInfo ObjectInfo) (time.Time, error) {
if stamp, ok := caseInsensitiveMap(objInfo.UserDefined).Lookup(ReservedMetadataPrefixLower + TaggingTimestamp); ok {
return time.Parse(time.RFC3339Nano, stamp)
}
if objInfo.UserTags != "" {
return objInfo.ModTime, nil
}
return time.Time{}, nil
}
func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo) (putOpts minio.PutObjectOptions, isMP bool, err error) {
meta := make(map[string]string)
isSSEC := crypto.SSEC.IsEncrypted(objInfo.UserDefined)
@@ -850,17 +909,12 @@ func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo) (put
tag, _ := tags.ParseObjectTags(objInfo.UserTags)
if tag != nil {
putOpts.UserTags = tag.ToMap()
// set tag timestamp in opts
tagTimestamp := objInfo.ModTime
if tagTmstampStr, ok := objInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]; ok {
tagTimestamp, err = time.Parse(time.RFC3339Nano, tagTmstampStr)
if err != nil {
return putOpts, false, err
}
}
putOpts.Internal.TaggingTimestamp = tagTimestamp
}
}
putOpts.Internal.TaggingTimestamp, err = replicationTaggingTimestamp(objInfo)
if err != nil {
return putOpts, false, err
}
lkMap := caseInsensitiveMap(objInfo.UserDefined)
if lang, ok := lkMap.Lookup(xhttp.ContentLanguage); ok {
@@ -1003,6 +1057,12 @@ func getReplicationAction(oi1 ObjectInfo, oi2 minio.ObjectInfo, opType replicati
if (oi2.UserTagCount > 0 && !reflect.DeepEqual(oi2Map, t.ToMap())) || (oi2.UserTagCount != len(t.ToMap())) {
return replicateMetadata
}
// HEAD does not report the tag revision. Equal values can hide a newer
// deletion or re-addition, so scheduled metadata/heal work must deliver it.
// Completed objects are still excluded by the existing scanner gates.
if _, ok := caseInsensitiveMap(oi1.UserDefined).Lookup(ReservedMetadataPrefixLower + TaggingTimestamp); ok {
return replicateMetadata
}
// Compare only necessary headers
compareKeys := []string{
@@ -1269,9 +1329,6 @@ func replicateObject(ctx context.Context, ri ReplicateObjectInfo, objectAPI Obje
oi.UserDefined[targetResetHeader(rinfo.Arn)] = rinfo.ResyncTimestamp
}
}
if ri.UserTags != "" {
oi.UserDefined[xhttp.AmzObjectTagging] = ri.UserTags
}
return dsc, nil
},
}
@@ -1689,14 +1746,11 @@ applyAction:
if _, ok := lkMap.Lookup(xhttp.AmzObjectLockRetainUntilDate); ok {
dstOpts.Internal.RetentionTimestamp = objInfo.ModTime
}
if objInfo.UserTags != "" {
dstOpts.Internal.TaggingTimestamp = objInfo.ModTime
}
if tagTmStr, ok := lkMap.Lookup(ReservedMetadataPrefixLower + TaggingTimestamp); ok {
ondiskTimestamp, err := time.Parse(time.RFC3339, tagTmStr)
if err == nil {
dstOpts.Internal.TaggingTimestamp = ondiskTimestamp
}
dstOpts.Internal.TaggingTimestamp, rinfo.Err = replicationTaggingTimestamp(objInfo)
if rinfo.Err != nil {
rinfo.ReplicationStatus = replication.Failed
replLogIf(ctx, fmt.Errorf("invalid tagging timestamp for object %s/%s(%s): %w", bucket, object, objInfo.VersionID, rinfo.Err))
return rinfo
}
if retTmStr, ok := lkMap.Lookup(ReservedMetadataPrefixLower + ObjectLockRetentionTimestamp); ok {
ondiskTimestamp, err := time.Parse(time.RFC3339, retTmStr)
@@ -1916,11 +1970,61 @@ func filterReplicationStatusMetadata(metadata map[string]string) map[string]stri
// DeletedObjectReplicationInfo has info on deleted object
type DeletedObjectReplicationInfo struct {
DeletedObject
Bucket string
EventType string
OpType replication.Type
ResetID string
TargetArn string
Bucket string
EventType string
OpType replication.Type
ResetID string
TargetArn string
RetryCount int
}
// isVersionPurge also recognizes the old marker-shaped purge task. Use the
// operation's state, rather than one target's possibly missing purge entry.
func (di DeletedObjectReplicationInfo) isVersionPurge() bool {
return di.VersionID != "" || di.DeleteMarkerVersionID != "" && !di.VersionPurgeStatus().Empty()
}
// creationState is the outcome of re-reading a queued delete-marker creation
// against the source.
type creationState int
const (
creationCurrent creationState = iota
creationStale
creationUnverified
)
// deleteMarkerCreationState re-reads the marker a queued creation task refers
// to. The version being absent, no longer a marker, or under a version purge
// makes the creation stale. A failed read is not absence: the caller retries
// later instead of guessing.
func deleteMarkerCreationState(ctx context.Context, objectAPI ObjectLayer, dobj DeletedObjectReplicationInfo) creationState {
oi, err := objectAPI.GetObjectInfo(ctx, dobj.Bucket, dobj.ObjectName, ObjectOptions{
VersionID: dobj.DeleteMarkerVersionID,
Versioned: globalBucketVersioningSys.PrefixEnabled(dobj.Bucket, dobj.ObjectName),
VersionSuspended: globalBucketVersioningSys.Suspended(dobj.Bucket),
})
switch {
case isErrObjectNotFound(err), isErrVersionNotFound(err):
return creationStale
case err != nil && !isErrMethodNotAllowed(err):
return creationUnverified
}
if !oi.DeleteMarker || oi.VersionID != dobj.DeleteMarkerVersionID {
return creationStale
}
if !oi.VersionPurgeStatus.Empty() || oi.VersionPurgeStatusInternal != "" {
return creationStale
}
return creationCurrent
}
// Purge metadata uses COMPLETE; operation statistics and audit use COMPLETED.
func purgeReplicationStatus(status VersionPurgeStatusType) replication.StatusType {
if replication.StatusType(status) == replication.CompletedLegacy {
return replication.Completed
}
return replication.StatusType(status)
}
// ToMRFEntry returns the relevant info needed by MRF
@@ -1930,9 +2034,10 @@ func (di DeletedObjectReplicationInfo) ToMRFEntry() MRFReplicateEntry {
versionID = di.VersionID
}
return MRFReplicateEntry{
Bucket: di.Bucket,
Object: di.ObjectName,
versionID: versionID,
Bucket: di.Bucket,
Object: di.ObjectName,
versionID: versionID,
RetryCount: di.RetryCount,
}
}
@@ -2424,6 +2529,7 @@ func (p *ReplicationPool) queueReplicaDeleteTask(doi DeletedObjectReplicationInf
case <-p.ctx.Done():
case ch <- doi:
default:
doi.RetryCount++
p.queueMRFSave(doi.ToMRFEntry())
p.mu.RLock()
prio := p.priority
@@ -3787,9 +3893,10 @@ func queueReplicationHeal(ctx context.Context, bucket string, oi ObjectInfo, rcf
DeleteMarkerMTime: DeleteMarkerMTime{roi.ModTime},
DeleteMarker: roi.DeleteMarker,
},
Bucket: roi.Bucket,
OpType: replication.HealReplicationType,
EventType: ReplicateHealDelete,
Bucket: roi.Bucket,
OpType: replication.HealReplicationType,
EventType: ReplicateHealDelete,
RetryCount: retryCount,
}
// heal delete marker replication failure or versioned delete replication failure
if roi.ReplicationStatus == replication.Pending ||
@@ -4072,7 +4179,12 @@ func (p *ReplicationPool) queueMRFHeal() error {
VersionID: vID,
})
cancel()
if err != nil {
// A versioned marker lookup returns its metadata with a 405. Only
// accept that error with a real, matching marker identity.
validMarker := isErrMethodNotAllowed(err) && oi.DeleteMarker &&
vID != "" && oi.VersionID == vID && !oi.ModTime.IsZero() &&
oi.Bucket == e.Bucket && oi.Name != "" && oi.Name == decodeDirObject(e.Object)
if err != nil && !validMarker || oi.Name == "" {
continue
}
+1
View File
@@ -445,6 +445,7 @@ func buildServerCtxt(ctx *cli.Context, ctxt *serverCtxt) (err error) {
ctxt.SendBufSize = ctx.Int("send-buf-size")
ctxt.RecvBufSize = ctx.Int("recv-buf-size")
ctxt.IdleTimeout = ctx.Duration("idle-timeout")
ctxt.ReadHeaderTimeout = ctx.Duration("read-header-timeout")
ctxt.UserTimeout = ctx.Duration("conn-user-timeout")
if conf := ctx.String("config"); len(conf) > 0 {
@@ -0,0 +1,109 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"net/http"
"net/http/httptest"
"testing"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
)
// An exact-version DELETE without a replication decision physically purges
// the version. Its response must describe the stored version: a data version
// is not a delete marker even while pending-purge metadata makes the lookup
// expose it as deleted, and a stored marker stays a marker.
func TestDeleteMarkerPurgeResponseIdentity(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, endpoints: []string{"DeleteObject"}, objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
defer replicationTestCapacity(obj)()
ctx := t.Context()
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
// Seed the purge state the DELETE handler would record for a target.
arn := "arn:minio:replication::" + mustGetUUID() + ":bucket"
pendingPurge := ReplicationState{
VersionPurgeStatusInternal: arn + "=PENDING;",
PurgeTargets: map[string]VersionPurgeStatusType{arn: replication.VersionPurgePending},
}
for _, tc := range []struct {
name string
marker, purged bool
}{
{name: "data"},
{name: "data-pending-purge", purged: true},
{name: "marker", marker: true},
{name: "marker-pending-purge", marker: true, purged: true},
} {
name := "purge-response-" + mustGetUUID()
oi, err := obj.PutObject(ctx, bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
version := oi.VersionID
if tc.marker {
opts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: UTCNow(), ReplicationRequest: true}
opts.SetReplicaStatus(replication.Replica)
if _, err := obj.DeleteObject(ctx, bucket, name, opts); err != nil {
t.Fatal(err)
}
version = opts.VersionID
}
if tc.purged {
if _, err := obj.DeleteObject(ctx, bucket, name, ObjectOptions{Versioned: true, VersionID: version, DeleteReplication: pendingPurge}); err != nil {
t.Fatal(err)
}
}
before, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{Versioned: true, VersionID: version})
if tc.marker || tc.purged {
if !isErrMethodNotAllowed(err) || !before.DeleteMarker {
t.Fatalf("%s: seeded version not exposed as deleted: %+v %v", tc.name, before, err)
}
} else if err != nil || before.DeleteMarker {
t.Fatalf("%s: seeded data version: %+v %v", tc.name, before, err)
}
if tc.purged && before.VersionPurgeStatus != replication.VersionPurgePending {
t.Fatalf("%s: purge state not seeded: %+v", tc.name, before)
}
req, err := newTestSignedRequestV4(http.MethodDelete, "/"+bucket+"/"+name+"?versionId="+version, 0, nil, creds.AccessKey, creds.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusNoContent {
t.Fatalf("%s: status=%d: %s", tc.name, rec.Code, rec.Body.String())
}
// The handler sets these headers by map key, not canonical name.
if got := rec.Header()[xhttp.AmzVersionID]; len(got) != 1 || got[0] != version {
t.Fatalf("%s: response version %q, want %q", tc.name, got, version)
}
if got := len(rec.Header()[xhttp.AmzDeleteMarker]) == 1 && rec.Header()[xhttp.AmzDeleteMarker][0] == "true"; got != tc.marker {
t.Fatalf("%s: x-amz-delete-marker=%v for a stored marker=%v", tc.name, got, tc.marker)
}
if _, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{Versioned: true, VersionID: version}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("%s: version not purged: %v", tc.name, err)
}
t.Logf("%s %s: purged with x-amz-delete-marker=%q", backend, tc.name, rec.Header()[xhttp.AmzDeleteMarker])
}
}})
}
+524
View File
@@ -0,0 +1,524 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"errors"
"fmt"
"maps"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/once"
)
func TestDeleteMarkerPurgeIntent(t *testing.T) {
version := mustGetUUID()
for _, tc := range []struct {
name string
edit func(*ObjectOptions)
want bool
}{
{"ordinary", func(*ObjectOptions) {}, true},
{"receiver", func(o *ObjectOptions) { o.ReplicationRequest = true; o.SetReplicaStatus(replication.Replica) }, true},
{"untrusted-replica", func(o *ObjectOptions) { o.SetReplicaStatus(replication.Replica) }, false},
{"complete-purge", func(o *ObjectOptions) {
o.DeleteReplication.VersionPurgeStatusInternal = string(replication.VersionPurgeComplete)
}, true},
{"pending-purge", func(o *ObjectOptions) { o.DeleteReplication.VersionPurgeStatusInternal = "PENDING" }, false},
{"failed-purge", func(o *ObjectOptions) { o.DeleteReplication.VersionPurgeStatusInternal = "FAILED" }, false},
{"unknown-purge", func(o *ObjectOptions) { o.DeleteReplication.VersionPurgeStatusInternal = "future-state" }, false},
{"pending-creation", func(o *ObjectOptions) { o.DeleteReplication.ReplicationStatusInternal = "PENDING" }, false},
{"failed-creation", func(o *ObjectOptions) { o.DeleteReplication.ReplicationStatusInternal = "FAILED" }, false},
{"completed-creation", func(o *ObjectOptions) { o.DeleteReplication.ReplicationStatusInternal = "COMPLETED" }, false},
{"create-marker", func(o *ObjectOptions) { o.DeleteMarker = true }, false},
{"empty", func(o *ObjectOptions) { o.VersionID = "" }, false},
{"null", func(o *ObjectOptions) { o.VersionID = nullVersionID }, false},
{"invalid", func(o *ObjectOptions) { o.VersionID = "invalid" }, false},
{"zero-uuid", func(o *ObjectOptions) { o.VersionID = emptyUUID }, false},
{"movement", func(o *ObjectOptions) { o.DataMovement = true }, false},
{"free-version", func(o *ObjectOptions) { o.InclFreeVersions = true }, false},
{"expiration", func(o *ObjectOptions) { o.Expiration.Expire = true }, false},
{"transition", func(o *ObjectOptions) { o.Transition.Status = "complete" }, false},
{"restored-expiration", func(o *ObjectOptions) { o.Transition.ExpireRestored = true }, false},
} {
t.Run(tc.name, func(t *testing.T) {
o := ObjectOptions{VersionID: version, Versioned: true}
tc.edit(&o)
if got := o.isVersionPurge(); got != tc.want {
t.Fatalf("purge=%v want=%v", got, tc.want)
}
})
}
}
// Every other StorageAPI method panics through the nil embedding: a proof may
// read metadata, but must never delete, heal, or ask a different storage API.
type absenceProofDisk struct {
StorageAPI
t *testing.T
err error
reads *atomic.Int32
deletes *atomic.Int32
}
func (d absenceProofDisk) ReadVersion(_ context.Context, _, _, _, _ string, opts ReadOptions) (FileInfo, error) {
if opts.ReadData || opts.Healing {
d.t.Error("absence proof requested data/healing")
}
d.reads.Add(1)
return FileInfo{}, d.err
}
func (d absenceProofDisk) DeleteVersion(_ context.Context, _, _ string, _ FileInfo, _ bool, _ DeleteOptions) error {
if d.deletes == nil {
d.t.Error("read-only absence proof attempted a deletion")
} else {
d.deletes.Add(1)
}
return d.err
}
func TestVersionPurgeAbsenceProof(t *testing.T) {
for _, tc := range []struct {
name string
errs []error
quorum, all bool
}{
{"all-absent", []error{errFileNotFound, errFileVersionNotFound, errFileNotFound, errFileVersionNotFound}, true, true},
{"majority-absent", []error{errFileNotFound, errFileVersionNotFound, errFileNotFound, nil}, true, false},
{"half-absent", []error{errFileNotFound, errFileVersionNotFound, errDiskNotFound, errDiskNotFound}, false, false},
{"corrupt", []error{errFileNotFound, errFileVersionNotFound, errFileCorrupt, errFileCorrupt}, false, false},
{"permission", []error{errFileNotFound, errFileVersionNotFound, errDiskAccessDenied, errVolumeAccessDenied}, false, false},
{"volume-missing", []error{errFileNotFound, errFileVersionNotFound, errVolumeNotFound, errVolumeNotFound}, false, false},
} {
t.Run(tc.name, func(t *testing.T) {
var reads atomic.Int32
disks := make([]StorageAPI, len(tc.errs))
for i, err := range tc.errs {
disks[i] = absenceProofDisk{t: t, err: err, reads: &reads}
}
er := erasureObjects{getDisks: func() []StorageAPI { return disks }}
oldQueue := globalMRFState.opCh
globalMRFState.opCh = make(chan PartialOperation, 1)
defer func() { globalMRFState.opCh = oldQueue }()
all, err := er.confirmVersionAbsent(t.Context(), "bucket", "object", mustGetUUID())
if (err == nil) != tc.quorum || all != tc.all || reads.Load() != int32(len(disks)) || len(globalMRFState.opCh) != 0 {
t.Fatalf("proof all=%v error=%v reads=%d queued=%d", all, err, reads.Load(), len(globalMRFState.opCh))
}
})
}
}
func TestVersionPurgeAggregation(t *testing.T) {
for _, pure := range []bool{false, true} {
for _, fault := range []error{nil, errFileCorrupt, errDiskAccessDenied} {
t.Run(fmt.Sprintf("purge_%v_fault_%v", pure, fault), func(t *testing.T) {
var calls atomic.Int32
disks := make([]StorageAPI, 4)
for i, err := range []error{nil, errFileNotFound, errFileVersionNotFound, fault} {
disks[i] = absenceProofDisk{t: t, err: err, deletes: &calls}
}
er := erasureObjects{getDisks: func() []StorageAPI { return disks }}
err := er.deleteObjectVersion(t.Context(), "bucket", "object", FileInfo{VersionID: mustGetUUID()}, false, pure)
if (err == nil) != pure || calls.Load() != 4 {
t.Fatalf("aggregation error=%v calls=%d", err, calls.Load())
}
})
}
}
}
func TestDeleteMarkerPurgeOmittedPool(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
const name = "partially-absent"
er := z.serverPools[1].getHashedSet(name)
_, opts := seedPurgeMarker(t, er, bucket, name, false)
opts.ReplicationRequest = false
opts.DeleteReplication = ReplicationState{}
// Pool 0 appears absent at read quorum, but half of it cannot be read.
missing := z.serverPools[0].getHashedSet(name)
original := missing.getDisks
disks := append([]StorageAPI(nil), original()...)
clear(disks[len(disks)/2:])
missing.getDisks = func() []StorageAPI { return disks }
defer func() { missing.getDisks = original }()
_, err := z.DeleteObject(t.Context(), bucket, name, opts)
if !isErrWriteQuorum(err) {
t.Errorf("omitted pool acknowledged deletion: %T %v", err, err)
}
if countPurgeMarkers(t, er.getDisks(), bucket, name, opts.VersionID) != 16 {
t.Fatal("mutated known copies before checking omitted pool")
}
}
func TestDeleteMarkerPurgeReceivingPool(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
for _, duplicate := range []bool{false, true} {
t.Run(fmt.Sprintf("duplicate_%v", duplicate), func(t *testing.T) {
name := "receiver-" + mustGetUUID()
_, opts := seedPurgeMarker(t, z.serverPools[0].getHashedSet(name), bucket, name, false)
if duplicate {
create := opts
create.DeleteMarker = true
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, name, create); err != nil {
t.Fatal(err)
}
} else {
// Latest-key routing selects pool 1, but the addressed marker is
// in pool 0. A replica purge addresses a version, not the latest.
putConsistencyObject(t, z, bucket, name, 1, "newer-data", ObjectOptions{Versioned: true})
}
if _, err := z.DeleteObject(t.Context(), bucket, name, opts); err != nil {
t.Errorf("replica purge did not find its addressed version: %v", err)
}
for i, pool := range z.serverPools {
if got := countPurgeMarkers(t, pool.getHashedSet(name).getDisks(), bucket, name, opts.VersionID); got != 0 {
t.Errorf("replica purge left %d marker copies in pool %d", got, i)
}
}
})
}
}
func TestDeleteMarkerPurgeCallbackIntent(t *testing.T) {
for _, loseCopies := range []bool{false, true} {
t.Run(fmt.Sprintf("missing_after_callback_%v", loseCopies), func(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
const name = "callback-pending"
er := z.serverPools[0].getHashedSet(name)
_, opts := seedPurgeMarker(t, er, bucket, name, false)
original := er.getDisks
all := append([]StorageAPI(nil), original()...)
defer func() { er.getDisks = original }()
opts.ReplicationRequest = false
opts.DeleteReplication = ReplicationState{}
metadata, retention := 0, 0
opts.EvalRetentionBypassFn = func(oi ObjectInfo, err error) error {
retention++
if !oi.DeleteMarker || !isErrMethodNotAllowed(err) {
t.Fatalf("retention callback lost marker: %+v %v", oi, err)
}
return nil
}
opts.EvalMetadataFn = func(*ObjectInfo, error) (ReplicateDecision, error) {
metadata++
if loseCopies {
// Inject a disk-state change after preflight, before the set's
// metadata-update read. It returns NotFound at read quorum.
for _, disk := range all[:8] {
if err := disk.DeleteVersion(t.Context(), bucket, name, FileInfo{VersionID: opts.VersionID}, false, DeleteOptions{}); err != nil {
t.Fatal(err)
}
}
partial := make([]StorageAPI, len(all))
copy(partial, all[:8])
er.getDisks = func() []StorageAPI { return partial }
}
decision := ReplicateDecision{}
decision.Set(newReplicateTargetDecision("arn1", true, false))
return decision, nil
}
_, err := z.DeleteObject(t.Context(), bucket, name, opts)
if loseCopies {
if !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Errorf("pending update misclassified as physical purge: %T %v", err, err)
}
} else if err != nil {
t.Errorf("pending update failed: %v", err)
}
want := 16
if loseCopies {
want = 8
}
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != want || metadata != 1 || retention != 1 {
t.Errorf("pending update: copies=%d want=%d metadata=%d retention=%d", got, want, metadata, retention)
}
if !loseCopies {
fi, err := all[0].ReadVersion(t.Context(), "", bucket, name, opts.VersionID, ReadOptions{})
if err != nil || fi.VersionPurgeStatus() != replication.VersionPurgePending {
t.Errorf("pending state not persisted: %+v %v", fi.ReplicationState, err)
}
}
})
}
}
func TestDeleteMarkerReceivingMetadataUpdate(t *testing.T) {
z, bucket := consistencyPools(t)
defer replicationTestCapacity(z)()
for _, state := range []VersionPurgeStatusType{replication.VersionPurgePending, replication.VersionPurgeFailed} {
t.Run(string(state), func(t *testing.T) {
name := "update-" + mustGetUUID()
_, opts := seedPurgeMarker(t, z.serverPools[0].getHashedSet(name), bucket, name, false)
create := opts
create.DeleteMarker = true
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, name, create); err != nil {
t.Fatal(err)
}
opts.DeleteReplication = ReplicationState{VersionPurgeStatusInternal: string(state)}
if _, err := z.DeleteObject(t.Context(), bucket, name, opts); err != nil {
t.Fatal(err)
}
for i, pool := range z.serverPools {
if got := countPurgeMarkers(t, pool.getHashedSet(name).getDisks(), bucket, name, opts.VersionID); got != 16 {
t.Errorf("metadata update removed marker copies in pool %d: %d", i, got)
}
}
})
}
}
func TestVersionPurgeRetentionGate(t *testing.T) {
z, er, all, bucket := markerPurgeFixture(t, 4)
data, _ := seedPurgeMarker(t, er, bucket, "protected-data", true)
denied := errors.New("retention denied")
opts := ObjectOptions{Versioned: true, VersionID: data, EvalRetentionBypassFn: func(oi ObjectInfo, err error) error {
if err != nil || oi.VersionID != data || oi.DeleteMarker {
t.Errorf("wrong retained version: %+v error=%v", oi, err)
}
return denied
}}
_, err := z.DeleteObject(t.Context(), bucket, "protected-data", opts)
if !errors.Is(err, denied) {
t.Fatalf("retention gate: %v", err)
}
for _, disk := range all {
if _, err := disk.ReadVersion(t.Context(), "", bucket, "protected-data", data, ReadOptions{}); err != nil {
t.Errorf("protected version changed: %v", err)
}
}
}
func markerPurgeFixture(t *testing.T, n int) (*erasureServerPools, *erasureObjects, []StorageAPI, string) {
t.Helper()
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure(ctx, n)
if err != nil {
cancel()
t.Fatal(err)
}
restore := replicationTestCapacity(obj)
t.Cleanup(func() {
cancel()
restore()
obj.Shutdown(context.Background())
removeRoots(dirs)
})
bucket := getRandomBucketName()
if err := obj.MakeBucket(ctx, bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
er := z.serverPools[0].sets[0]
original := er.getDisks
t.Cleanup(func() { er.getDisks = original })
return z, er, append([]StorageAPI(nil), original()...), bucket
}
func seedPurgeMarker(t *testing.T, er *erasureObjects, bucket, name string, withData bool) (string, ObjectOptions) {
t.Helper()
dataVersion := ""
if withData {
oi, err := er.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
dataVersion = oi.VersionID
}
opts := ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, ReplicationRequest: true, MTime: UTCNow().Add(-time.Hour), NoAuditLog: true}
opts.SetReplicaStatus(replication.Replica)
if _, err := er.DeleteObject(t.Context(), bucket, name, opts); err != nil {
t.Fatal(err)
}
opts.DeleteMarker = false
return dataVersion, opts
}
func countPurgeMarkers(t *testing.T, disks []StorageAPI, bucket, name, version string) int {
t.Helper()
n := 0
for _, disk := range disks {
fi, err := disk.ReadVersion(t.Context(), "", bucket, name, version, ReadOptions{})
if err == nil && fi.Deleted {
n++
} else if err != errFileNotFound && err != errFileVersionNotFound {
t.Fatalf("unexpected disk version: deleted=%v error=%v", fi.Deleted, err)
}
}
return n
}
func TestDeleteMarkerPurgeQuorum(t *testing.T) {
for _, tc := range []struct{ disks, online int }{{4, 2}, {16, 7}, {16, 8}, {16, 9}} {
t.Run(fmt.Sprintf("%d_disks_%d_online", tc.disks, tc.online), func(t *testing.T) {
z, er, all, bucket := markerPurgeFixture(t, tc.disks)
const name = "marker"
dataVersion, opts := seedPurgeMarker(t, er, bucket, name, true)
partial := make([]StorageAPI, len(all))
copy(partial, all[:tc.online])
er.getDisks = func() []StorageAPI { return partial }
_, first := z.DeleteObject(t.Context(), bucket, name, opts)
_, retry := z.DeleteObject(t.Context(), bucket, name, opts)
if tc.online <= tc.disks/2 {
if !isErrWriteQuorum(first) || !isErrWriteQuorum(retry) {
t.Errorf("below write quorum: first=%T %v, retry=%T %v", first, first, retry, retry)
}
} else if first != nil || (retry != nil && !isErrVersionNotFound(retry) && !isErrObjectNotFound(retry)) {
t.Errorf("write quorum available: first=%v retry=%v", first, retry)
}
want := tc.disks - tc.online
if tc.online < tc.disks/2 {
want = tc.disks
}
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != want {
t.Errorf("partial purge markers=%d want=%d", got, want)
}
er.getDisks = func() []StorageAPI { return all }
_, err := z.DeleteObject(t.Context(), bucket, name, opts)
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Errorf("full-online retry: %v", err)
}
if tc.online <= tc.disks/2 {
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != 0 {
t.Errorf("full-online retry recreated/retained %d marker copies", got)
}
}
// A quorum-confirmed missing retry can leave minority residue. Exercise
// the real dangling-version healer after every disk has returned.
_, _ = er.HealObject(t.Context(), bucket, name, opts.VersionID, madmin.HealOpts{Remove: true})
if got := countPurgeMarkers(t, all, bucket, name, opts.VersionID); got != 0 {
t.Errorf("marker copies after retry and heal=%d", got)
}
oi, err := z.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{Versioned: true})
if err != nil || oi.DeleteMarker || oi.VersionID != dataVersion {
t.Errorf("underlying version not visible: %+v error=%v", oi, err)
}
})
}
}
func TestDeleteMarkerPurgeMissingKey(t *testing.T) {
for _, pooled := range []bool{false, true} {
t.Run(fmt.Sprintf("pooled_%v", pooled), func(t *testing.T) {
z, er, all, bucket := markerPurgeFixture(t, 4)
_, opts := seedPurgeMarker(t, er, bucket, "marker-only", false)
er.getDisks = func() []StorageAPI { return []StorageAPI{all[0], all[1], nil, nil} }
deleteFn := er.DeleteObject
if pooled {
deleteFn = z.DeleteObject
}
_, _ = deleteFn(t.Context(), bucket, "marker-only", opts)
_, err := deleteFn(t.Context(), bucket, "marker-only", opts)
if !isErrWriteQuorum(err) {
t.Errorf("missing retry with only 2/4 absence votes returned %T %v", err, err)
}
})
}
}
func TestDeleteMarkerHealReplicationIdentity(t *testing.T) {
obj, er, all, bucket := markerPurgeFixture(t, 4)
const name = "healed-marker"
_, opts := seedPurgeMarker(t, er, bucket, name, true)
original, err := all[3].ReadVersion(t.Context(), "", bucket, name, opts.VersionID, ReadOptions{})
if err != nil {
t.Fatal(err)
}
for _, disk := range all[:2] {
if err := disk.DeleteVersion(t.Context(), bucket, name, FileInfo{VersionID: opts.VersionID}, false, DeleteOptions{}); err != nil {
t.Fatal(err)
}
}
if _, err := er.HealObject(t.Context(), bucket, name, opts.VersionID, madmin.HealOpts{}); err != nil {
t.Fatal(err)
}
for i, disk := range all {
fi, err := disk.ReadVersion(t.Context(), "", bucket, name, opts.VersionID, ReadOptions{})
if err != nil || !maps.Equal(fi.Metadata, original.Metadata) {
t.Errorf("disk %d lost marker metadata: before=%v after=%v error=%v", i, original.Metadata, fi.Metadata, err)
}
}
// Only the healed half remains readable. Read actual disk state before
// invoking scanner replication, rather than constructing an ObjectInfo.
er.getDisks = func() []StorageAPI { return []StorageAPI{all[0], all[1], nil, nil} }
oi, err := er.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{VersionID: opts.VersionID, Versioned: true})
if !oi.DeleteMarker || !isErrMethodNotAllowed(err) {
t.Fatalf("healed marker unreadable: %+v %v", oi, err)
}
er.getDisks = func() []StorageAPI { return all }
var outbound atomic.Int32
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodDelete {
outbound.Add(1)
if r.Header.Get(xhttp.MinIOSourceDeleteMarker) != "true" || r.URL.Query().Get("versionId") != opts.VersionID {
t.Errorf("unexpected outbound request: %s %s", r.Method, r.URL)
}
w.WriteHeader(http.StatusNoContent)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer remote.Close()
client, err := minio.New(strings.TrimPrefix(remote.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
arn := "arn:minio:replication::" + mustGetUUID() + ":bucket"
target := &TargetClient{Client: client, ARN: arn, Bucket: "target"}
oldTargets := globalBucketTargetSys
globalBucketTargetSys = &BucketTargetSys{arnRemotesMap: map[string]arnTarget{arn: {Client: target, lastRefresh: UTCNow()}}, targetsMap: map[string][]madmin.BucketTarget{bucket: {{Arn: arn, TargetBucket: "target"}}}, hc: map[string]epHealth{client.EndpointURL().Host: {Online: true}}}
defer func() { globalBucketTargetSys = oldTargets }()
cfg := configs[0]
cfg.RoleArn = arn
meta, err := globalBucketMetadataSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
meta.replicationConfig = &cfg
globalBucketMetadataSys.Set(bucket, meta)
worker := make(chan ReplicationWorkerOperation, 1)
oldPool := globalReplicationPool
globalReplicationPool = once.NewSingleton[ReplicationPool]()
globalReplicationPool.Set(&ReplicationPool{ctx: t.Context(), objLayer: obj, workers: []chan ReplicationWorkerOperation{worker}, stats: globalReplicationStats.Load(), mrfSaveCh: make(chan MRFReplicateEntry, 1)})
defer func() { globalReplicationPool = oldPool }()
roi := queueReplicationHeal(t.Context(), bucket, oi, replicationConfig{Config: &cfg, remotes: &madmin.BucketTargets{Targets: []madmin.BucketTarget{{Arn: arn, TargetBucket: "target"}}}}, 0)
select {
case op := <-worker:
d := op.(DeletedObjectReplicationInfo)
result := replicateDeleteToTarget(t.Context(), d, target)
t.Errorf("healed replica scheduled for creation: version=%s existing=%v result=%+v", d.DeleteMarkerVersionID, roi.ExistingObjResync.mustResync(), result)
default:
}
if outbound.Load() != 0 {
t.Errorf("healed replica sent %d new marker creations", outbound.Load())
}
}
+2 -2
View File
@@ -253,7 +253,7 @@ func testAPICompleteMultipartFullObjectChecksumMismatch(obj ObjectLayer, instanc
if apiErr.Code != "BadDigest" {
t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code)
}
if want := "The CRC32 checksum you specified did not match the calculated checksum."; apiErr.Message != want {
if want := "The CRC32 you specified did not match the calculated checksum."; apiErr.Message != want {
t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message)
}
@@ -361,7 +361,7 @@ func testAPICompleteMultipartCompositeChecksumMismatch(obj ObjectLayer, instance
if apiErr.Code != "BadDigest" {
t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code)
}
if want := "The CRC32 checksum you specified did not match the calculated checksum."; apiErr.Message != want {
if want := "The CRC32 you specified did not match the calculated checksum."; apiErr.Message != want {
t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message)
}
if _, err := obj.GetObjectInfo(t.Context(), bucketName, objectName, ObjectOptions{}); err == nil {
+385
View File
@@ -0,0 +1,385 @@
// Copyright (c) 2026 mr javad seydi and Ruohang Feng
//
// This file is part of Silo Object Storage stack.
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/pgsty/silo-pkg/v3/policy"
)
const multipartScanEntryLimit = 100_000
var (
multipartScanSlots = make(chan struct{}, 2)
errMultipartListingLegacy = errors.New("legacy multipart uploads require a coordinated upgrade and drain")
errMultipartListingIdentity = errors.New("multipart upload identity is invalid")
)
// One budget and admission slot cover the entire request, including all pools.
// A slot is released only after the scan workers have actually stopped.
type multipartScan struct {
ctx context.Context
cancel context.CancelFunc
remaining atomic.Int64
metadataSlots chan struct{}
preflight bool
sets []multipartScanSet
}
type multipartScanSet struct {
Pool int `json:"pool"`
Set int `json:"set"`
Drives int `json:"drives"`
ScannedDrives int `json:"scannedDrives"`
UncoveredDrives []int `json:"uncoveredDrives,omitempty"`
Candidates int `json:"candidates"`
LegacyUploads int `json:"legacyUploads"`
OldestLegacy time.Time `json:"oldestLegacy,omitempty"`
Error string `json:"error,omitempty"`
}
type multipartPreflightReport struct {
Ready bool `json:"ready"`
Complete bool `json:"complete"`
Mode string `json:"mode"`
ScannedEntries int64 `json:"scannedEntries"`
LegacyUploads int `json:"legacyUploads"`
Sets []multipartScanSet `json:"sets"`
}
func startMultipartScan(ctx context.Context, preflight bool) (*multipartScan, error) {
select {
case multipartScanSlots <- struct{}{}:
case <-ctx.Done():
return nil, ctx.Err()
default:
return nil, SlowDown{}
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
s := &multipartScan{ctx: ctx, cancel: cancel, preflight: preflight, metadataSlots: make(chan struct{}, multipartMetadataScanConcurrency)}
s.remaining.Store(multipartScanEntryLimit)
return s, nil
}
func (s *multipartScan) close() {
s.cancel()
<-multipartScanSlots
}
func (s *multipartScan) listDir(disk StorageAPI, bucket, dir string) ([]string, error) {
if err := s.ctx.Err(); err != nil {
return nil, SlowDown{}
}
remaining := s.remaining.Load()
if remaining <= 0 {
return nil, SlowDown{}
}
// Request one extra entry to detect overflow, never a silently partial page.
entries, err := disk.ListDir(s.ctx, bucket, minioMetaMultipartBucket, dir, int(remaining)+1)
if errors.Is(err, errFileNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
if s.remaining.Add(-int64(len(entries))) < 0 {
return nil, SlowDown{}
}
return entries, nil
}
func (s *multipartScan) listUploadDirs(disk StorageAPI, bucket string) ([]string, error) {
hashDirs, err := s.listDir(disk, bucket, "")
if err != nil {
return nil, err
}
var candidates []string
for _, hashDir := range hashDirs {
if !strings.HasSuffix(hashDir, SlashSeparator) {
continue
}
hashDir = strings.TrimSuffix(hashDir, SlashSeparator)
uploadDirs, err := s.listDir(disk, bucket, hashDir)
if err != nil {
return nil, err
}
for _, uploadDir := range uploadDirs {
if strings.HasSuffix(uploadDir, SlashSeparator) {
candidates = append(candidates, pathJoin(hashDir, strings.TrimSuffix(uploadDir, SlashSeparator)))
}
}
}
return candidates, nil
}
// Identity is immutable at hash/uploadUUID. Check the hash before using even
// a single source drive to exclude another bucket. Missing/bad identities must
// fall back to the full metadata read; they cannot prove absence.
func (er erasureObjects) multipartIdentity(fi FileInfo, shaDir string) (string, string, bool) {
bucket, object := fi.Metadata[multipartMetaBucket], fi.Metadata[multipartMetaObject]
return bucket, object, bucket != "" && object != "" && IsValidBucketName(bucket) &&
IsValidObjectPrefix(object) && er.getMultipartSHADir(bucket, object) == shaDir
}
func (er erasureObjects) readMultipartUploadCandidate(s *multipartScan, bucket, candidate string, source StorageAPI) (MultipartInfo, bool, bool, error) {
if s.ctx.Err() != nil {
return MultipartInfo{}, false, false, SlowDown{}
}
shaDir, uploadUUID, ok := strings.Cut(candidate, SlashSeparator)
if !ok || shaDir == "" || uploadUUID == "" || strings.Contains(uploadUUID, SlashSeparator) {
return MultipartInfo{}, false, false, errMultipartListingIdentity
}
if !s.preflight && bucket != "" && source != nil {
fi, err := source.ReadVersion(s.ctx, bucket, minioMetaMultipartBucket, candidate, "", ReadOptions{})
if err == nil {
storedBucket, _, valid := er.multipartIdentity(fi, shaDir)
if valid && storedBucket != bucket {
return MultipartInfo{}, false, false, nil
}
}
}
if s.ctx.Err() != nil {
return MultipartInfo{}, false, false, SlowDown{}
}
select {
case s.metadataSlots <- struct{}{}:
defer func() { <-s.metadataSlots }()
case <-s.ctx.Done():
return MultipartInfo{}, false, false, SlowDown{}
}
disks := er.getDisks()
metadata, errs := readAllFileInfo(s.ctx, disks, bucket, minioMetaMultipartBucket, candidate, "", false, false)
if s.preflight {
// Readiness is stronger than listing liveness: even one readable legacy
// copy must be drained, and an unreadable copy cannot certify readiness.
var oldest MultipartInfo
legacy := false
_, nativeID := multipartUploadTime(uploadUUID)
for i, err := range errs {
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
continue
}
if err != nil {
return MultipartInfo{}, false, false, err
}
fi := metadata[i]
storedBucket, storedObject, valid := er.multipartIdentity(fi, shaDir)
if storedBucket != "" && storedObject != "" && !valid {
return MultipartInfo{}, false, false, errMultipartListingIdentity
}
if !valid || !nativeID {
info := multipartUploadInfo(storedBucket, storedObject, uploadUUID, fi.ModTime)
if !legacy || info.Initiated.Before(oldest.Initiated) {
oldest = info
}
legacy = true
}
}
if legacy {
return oldest, false, true, nil
}
}
readQuorum, _, err := objectQuorumFromMeta(s.ctx, metadata, errs, er.defaultParityCount)
if err != nil {
return MultipartInfo{}, false, false, err
}
_, modTime, etag := listOnlineDisks(disks, metadata, errs, readQuorum)
if err := reduceReadQuorumErrs(s.ctx, errs, objectOpIgnoredErrs, readQuorum); err != nil {
return MultipartInfo{}, false, false, err
}
fi, err := pickValidFileInfo(s.ctx, metadata, modTime, etag, readQuorum)
if err != nil {
return MultipartInfo{}, false, false, err
}
storedBucket, storedObject, valid := er.multipartIdentity(fi, shaDir)
info := multipartUploadInfo(storedBucket, storedObject, uploadUUID, fi.ModTime)
if storedBucket == "" || storedObject == "" {
return info, false, true, nil
}
if !valid {
return MultipartInfo{}, false, false, fmt.Errorf("%w: %s", errMultipartListingIdentity, candidate)
}
if _, ok := multipartUploadTime(uploadUUID); !ok {
return info, false, true, nil
}
if bucket != "" && bucket != storedBucket {
return MultipartInfo{}, false, false, nil
}
return info, true, false, nil
}
func (er erasureObjects) scanMultipartUploads(s *multipartScan, bucket string, poolIdx, setIdx int) ([]MultipartInfo, bool, error) {
disks := er.getDisks()
report := multipartScanSet{Pool: poolIdx, Set: setIdx, Drives: er.setDriveCount}
var resultErr error
defer func() {
if resultErr != nil {
report.Error = resultErr.Error()
}
s.sets = append(s.sets, report)
}()
var candidateMu sync.Mutex
candidates := make(map[string]StorageAPI)
errs := make([]error, len(disks))
var wg sync.WaitGroup
for i, disk := range disks {
wg.Add(1)
go func() {
defer wg.Done()
if disk == nil || !disk.IsOnline() {
errs[i] = errDiskNotFound
return
}
paths, err := s.listUploadDirs(disk, bucket)
errs[i] = err
if err != nil {
return
}
candidateMu.Lock()
defer candidateMu.Unlock()
for _, p := range paths {
candidates[p] = disk
}
}()
}
wg.Wait()
for i, err := range errs {
if err == nil {
report.ScannedDrives++
} else {
report.UncoveredDrives = append(report.UncoveredDrives, i)
if _, limited := err.(SlowDown); limited {
resultErr = err
}
}
}
report.Candidates = len(candidates)
// A successful scan must intersect every metadata read quorum, including
// records left with R copies by a partially failed cancellation.
if report.ScannedDrives < er.setDriveCount/2+1 && resultErr == nil {
resultErr = toObjectErr(errErasureReadQuorum, bucket)
}
if resultErr != nil {
return nil, false, resultErr
}
type candidate struct {
path string
source StorageAPI
}
jobs := make(chan candidate)
var mu sync.Mutex
var uploads []MultipartInfo
for range min(16, len(candidates)) {
wg.Add(1)
go func() {
defer wg.Done()
for c := range jobs {
mu.Lock()
stopped := resultErr != nil
mu.Unlock()
if stopped || s.ctx.Err() != nil {
continue
}
upload, found, legacy, err := er.readMultipartUploadCandidate(s, bucket, c.path, c.source)
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
continue
}
mu.Lock()
switch {
case err != nil && resultErr == nil:
resultErr = err
case legacy:
report.LegacyUploads++
if report.OldestLegacy.IsZero() || upload.Initiated.Before(report.OldestLegacy) {
report.OldestLegacy = upload.Initiated
}
case found && !s.preflight:
uploads = append(uploads, upload)
}
mu.Unlock()
}
}()
}
dispatch:
for p, source := range candidates {
select {
case jobs <- candidate{p, source}:
case <-s.ctx.Done():
break dispatch
}
}
close(jobs)
wg.Wait()
if s.ctx.Err() != nil {
resultErr = SlowDown{}
}
return uploads, report.LegacyUploads != 0, resultErr
}
// multipartPreflight scans all pools, sets and drives, independent of caches.
// A majority suffices for normal listing; upgrade readiness requires every
// drive to have been inspected. The operator must also upgrade all writers.
func (z *erasureServerPools) multipartPreflight(ctx context.Context) (multipartPreflightReport, error) {
s, err := startMultipartScan(ctx, true)
if err != nil {
return multipartPreflightReport{}, err
}
defer s.close()
report := multipartPreflightReport{Complete: true, Mode: "strict"}
if globalAPIConfig.getMultipartListingLegacy() {
report.Mode = "legacy"
}
for p, pool := range z.serverPools {
for i, set := range pool.sets {
_, _, _ = set.scanMultipartUploads(s, "", p, i)
}
}
report.Sets = s.sets
for _, set := range s.sets {
report.LegacyUploads += set.LegacyUploads
if set.Error != "" || set.ScannedDrives != set.Drives {
report.Complete = false
}
}
report.ScannedEntries = multipartScanEntryLimit - s.remaining.Load()
report.Ready = report.Complete && report.LegacyUploads == 0
return report, nil
}
// MultipartPreflightHandler is a read-only storage-admin diagnostic. It never
// accepts an arbitrary deletion path or changes upload lifetime settings.
func (a adminAPIHandlers) MultipartPreflightHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
obj, _ := validateAdminReq(ctx, w, r, policy.StorageInfoAdminAction)
if obj == nil {
return
}
z, ok := obj.(*erasureServerPools)
if !ok {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrNotImplemented), r.URL)
return
}
report, err := z.multipartPreflight(ctx)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
b, err := json.Marshal(report)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
writeSuccessResponseJSON(w, b)
}
+819
View File
@@ -0,0 +1,819 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/minio/minio/internal/config/storageclass"
)
// Check the real handler and storage path, including a successful abort between
// pages. Choose IDs increasing in both initiation time and lexical order so the
// failure does not depend on differing interpretations of S3 marker ordering.
func TestMultipartListingAbortBetweenHTTPPages(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads", "AbortMultipart"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
var firstID, secondID string
for attempt := 0; attempt < 32; attempt++ {
one, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
two, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if one.UploadID < two.UploadID {
firstID, secondID = one.UploadID, two.UploadID
break
}
for _, id := range []string{one.UploadID, two.UploadID} {
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", id, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
}
if firstID == "" {
t.Fatal("could not construct increasing upload IDs")
}
if _, err := z.NewMultipartUpload(t.Context(), bucket, "b", ObjectOptions{}); err != nil {
t.Fatal(err)
}
request := func(method, u string) *httptest.ResponseRecorder {
t.Helper()
req, err := newTestSignedRequestV4(method, u, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
return rec
}
list := func(keyMarker, uploadMarker string, limit int) ListMultipartUploadsResponse {
t.Helper()
rec := request(http.MethodGet, getListMultipartUploadsURLWithParams("", bucket, "", keyMarker, uploadMarker, "", strconv.Itoa(limit)))
if rec.Code != http.StatusOK {
t.Fatalf("list HTTP %d: %s", rec.Code, rec.Body.String())
}
var result ListMultipartUploadsResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &result); err != nil {
t.Fatal(err)
}
return result
}
first := list("", "", 1)
if len(first.Uploads) != 1 || first.Uploads[0].UploadID != firstID || !first.IsTruncated {
t.Fatalf("unexpected first page: %+v", first)
}
rec := request(http.MethodDelete, getAbortMultipartUploadURL("", bucket, "a", firstID))
if rec.Code != http.StatusNoContent {
t.Fatalf("abort HTTP %d: %s", rec.Code, rec.Body.String())
}
rest := list(first.NextKeyMarker, first.NextUploadIDMarker, 10)
var keys []string
for _, upload := range rest.Uploads {
keys = append(keys, upload.Key)
}
t.Logf("GET first page=200; DELETE marker=204; GET next page=200 keys=%v truncated=%v", keys, rest.IsTruncated)
if _, err := z.GetMultipartInfo(t.Context(), bucket, "a", secondID, ObjectOptions{}); err != nil {
t.Fatalf("remaining upload is not valid: %v", err)
}
if len(rest.Uploads) != 2 || rest.Uploads[0].UploadID != secondID {
t.Fatalf("valid remaining upload for key a is missing from continuation: %+v", rest.Uploads)
}
}
type multipartListingFaultDisk struct {
StorageAPI
read func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error)
delete func(context.Context, string, string, DeleteOptions) error
}
func (d multipartListingFaultDisk) ReadVersion(ctx context.Context, original, volume, object, version string, opts ReadOptions) (FileInfo, error) {
if d.read != nil {
return d.read(ctx, original, volume, object, version, opts)
}
return d.StorageAPI.ReadVersion(ctx, original, volume, object, version, opts)
}
func (d multipartListingFaultDisk) Delete(ctx context.Context, volume, object string, opts DeleteOptions) error {
if d.delete != nil {
return d.delete(ctx, volume, object, opts)
}
return d.StorageAPI.Delete(ctx, volume, object, opts)
}
func TestMultipartListingLegacyPreflight(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
const other = "multipart-legacy-other"
if err := z.MakeBucket(t.Context(), other, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
old, err := z.NewMultipartUpload(t.Context(), other, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
fi, metadata, err := set.checkUploadIDExists(t.Context(), other, "old", old.UploadID, true)
if err != nil {
t.Fatal(err)
}
for i := range metadata {
delete(metadata[i].Metadata, multipartMetaBucket)
delete(metadata[i].Metadata, multipartMetaObject)
}
if _, err = writeAllMetadata(t.Context(), set.getDisks(), other, minioMetaMultipartBucket,
set.getUploadIDDir(other, "old", old.UploadID), metadata, fi.WriteQuorum(set.defaultWQuorum())); err != nil {
t.Fatal(err)
}
if _, err = z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
z.mpCache.Clear()
t.Run("legacy-upgrade", func(t *testing.T) {
setMultipartListingTestMode(t, true)
exact, err := z.ListMultipartUploads(t.Context(), other, "old", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, exact, "old")
const empty = "multipart-legacy-empty"
if err := z.MakeBucket(t.Context(), empty, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
got, err := z.ListMultipartUploads(t.Context(), empty, "", "", "", "", 10)
if err != nil || len(got.Uploads) != 0 {
t.Fatalf("old upload in another bucket broke legacy listing: %+v %v", got, err)
}
})
_, err = z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if !errors.Is(err, errMultipartListingLegacy) {
t.Fatalf("old upload in another bucket: %v", err)
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || !report.Complete || report.LegacyUploads != 1 {
t.Fatalf("legacy preflight: %+v %v", report, err)
}
if err = z.AbortMultipartUpload(t.Context(), other, "old", old.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
report, err = z.multipartPreflight(t.Context())
if err != nil || !report.Ready || report.LegacyUploads != 0 {
t.Fatalf("drained preflight: %+v %v", report, err)
}
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
}
func TestMultipartListingIdentityFallback(t *testing.T) {
for _, kind := range []string{"old", "corrupt", "read-failure", "wrong-bucket"} {
t.Run(kind, func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
if _, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
var first atomic.Bool
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(ctx context.Context, b, v, p, version string, opts ReadOptions) (FileInfo, error) {
fi, err := d.ReadVersion(ctx, b, v, p, version, opts)
if v == minioMetaMultipartBucket && first.CompareAndSwap(false, true) {
if kind == "read-failure" {
return FileInfo{}, errDiskNotFound
}
if kind == "corrupt" {
return FileInfo{}, errFileCorrupt
}
fi.Metadata = cloneMSS(fi.Metadata)
if kind == "old" {
delete(fi.Metadata, multipartMetaBucket)
} else {
fi.Metadata[multipartMetaBucket] = "another-bucket"
}
}
return fi, err
}}
}
return disks
}
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
})
}
}
func TestMultipartAbortPoolsAndRetry(t *testing.T) {
z, bucket := consistencyPools(t)
setMultipartListingTestMode(t, false)
mp, err := z.serverPools[1].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
emptySet := z.serverPools[0].getHashedSet("a")
original := emptySet.getDisks
t.Cleanup(func() { emptySet.getDisks = original })
emptySet.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(context.Context, string, string, string, string, ReadOptions) (FileInfo, error) {
return FileInfo{}, errDiskNotFound
}}
}
return disks
}
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("unknown pool must not acknowledge cancellation: %v", err)
}
emptySet.getDisks = original
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if _, ok := err.(InvalidUploadID); !ok {
t.Fatalf("retry after all pools confirm absence: %v", err)
}
mp, err = z.serverPools[1].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if _, err = z.serverPools[1].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err == nil {
t.Fatal("empty first pool hid the actual upload")
}
}
func TestMultipartAbortRetryBelowReadQuorum(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
disks[3] = nil
d := disks[2]
disks[2] = multipartListingFaultDisk{StorageAPI: d, delete: func(ctx context.Context, v, p string, opts DeleteOptions) error {
if err := d.Delete(ctx, v, p, opts); err != nil {
return err
}
return context.DeadlineExceeded // operation finished, acknowledgement lost
}}
return disks
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err == nil {
t.Fatal("lost acknowledgement must fail")
}
set.getDisks = original
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("one remaining metadata copy prevented retry: %v", err)
}
}
func TestMultipartListingMarkerHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
for _, tc := range []struct {
key, marker string
status int
}{
{"", "not-base64=", 200},
{"a", "not-base64=", 404},
{"a", base64.RawURLEncoding.EncodeToString([]byte("not-native")), 400},
{"a", multipartListingTestID(time.Unix(100, 0), 1), 200},
} {
u := getListMultipartUploadsURLWithParams("", bucket, "", tc.key, tc.marker, "", "10")
req, err := newTestSignedRequestV4(http.MethodGet, u, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != tc.status {
t.Fatalf("key=%q marker=%q: %d %s", tc.key, tc.marker, rec.Code, rec.Body.String())
}
}
}
func TestMultipartPreflightAdminHTTP(t *testing.T) {
bed, err := prepareAdminErasureTestBed(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { bed.done(); bed.objLayer.Shutdown(context.Background()); removeRoots(bed.erasureDirs) })
const target = "/minio/admin/v3/multipart-preflight"
rec := httptest.NewRecorder()
bed.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, target, nil))
if rec.Code != 403 {
t.Fatalf("anonymous preflight: %d", rec.Code)
}
req, err := newTestSignedRequestV4(http.MethodGet, target, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec = httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != 200 {
t.Fatalf("admin preflight: %d %s", rec.Code, rec.Body.String())
}
var report multipartPreflightReport
if err = json.Unmarshal(rec.Body.Bytes(), &report); err != nil || !report.Ready || !report.Complete {
t.Fatalf("preflight: %+v %v", report, err)
}
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
rec = httptest.NewRecorder()
bed.router.ServeHTTP(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("busy admin preflight: %d %s", rec.Code, rec.Body.String())
}
}
type multipartLateCreateDisk struct {
StorageAPI
late chan<- func() error
}
func (d multipartLateCreateDisk) WriteMetadata(_ context.Context, original, volume, object string, fi FileInfo) error {
if volume == minioMetaMultipartBucket {
d.late <- func() error { return d.StorageAPI.WriteMetadata(context.Background(), original, volume, object, fi) }
return context.DeadlineExceeded
}
return d.StorageAPI.WriteMetadata(context.Background(), original, volume, object, fi)
}
// This characterization is deliberately NOT an assertion of terminal abort
// correctness. It preserves an executable example of the separately scoped
// late-creation-write limitation. Change the expectation when fencing is added.
func TestMultipartAbortLateCreateBoundary(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), 16)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
setMultipartListingTestMode(t, false)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
saved := globalStorageClass
globalStorageClass.Update(storageclass.Config{Standard: storageclass.StorageClass{Parity: 8}})
t.Cleanup(func() { globalStorageClass.Update(saved) })
const bucket = "multipart-late-create"
if err = z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
set := z.serverPools[0].getHashedSet("a")
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
late := make(chan func() error, 16)
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 9; i < 16; i++ {
disks[i] = multipartLateCreateDisk{disks[i], late}
}
return disks
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if len(late) != 7 {
t.Fatalf("expected seven timed-out creation writes, got %d", len(late))
}
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 2; i < 9; i++ {
disks[i] = nil
}
return disks
}
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
for range 7 {
if err = (<-late)(); err != nil {
t.Fatal(err)
}
}
set.getDisks = original
_, metadata, err := set.checkUploadIDExists(t.Context(), bucket, "a", mp.UploadID, true)
if err != nil {
t.Fatalf("late creation boundary changed; review and remove the documented limitation: %v", err)
}
count := 0
for _, fi := range metadata {
if fi.IsValid() {
count++
}
}
if count != 14 {
t.Fatalf("expected fourteen resurrected copies, got %d", count)
}
t.Log("KNOWN UNRESOLVED BOUNDARY: seven delayed creation writes plus seven offline copies restore a writable upload after acknowledged cancellation")
}
type multipartListingCountingDisk struct {
StorageAPI
directoryCalls *atomic.Int64
metadataCalls *atomic.Int64
}
func (d multipartListingCountingDisk) ListDir(ctx context.Context, original, volume, dir string, count int) ([]string, error) {
if volume == minioMetaMultipartBucket {
d.directoryCalls.Add(1)
}
return d.StorageAPI.ListDir(ctx, original, volume, dir, count)
}
func (d multipartListingCountingDisk) ReadVersion(ctx context.Context, original, volume, object, version string, opts ReadOptions) (FileInfo, error) {
if volume == minioMetaMultipartBucket {
d.metadataCalls.Add(1)
}
return d.StorageAPI.ReadVersion(ctx, original, volume, object, version, opts)
}
// Counts storage API calls; this is not a deployment throughput benchmark.
func TestMultipartListingScanCosts(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), 4)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
setMultipartListingTestMode(t, false)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket, otherBucket = "r9-scan-target", "r9-scan-unrelated"
for _, name := range []string{bucket, otherBucket} {
if err := z.MakeBucket(t.Context(), name, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
}
if _, err := z.NewMultipartUpload(t.Context(), bucket, "dir/target", ObjectOptions{}); err != nil {
t.Fatal(err)
}
var directoryCalls, metadataCalls atomic.Int64
for _, pool := range z.serverPools {
for _, set := range pool.sets {
original := set.getDisks
set.getDisks = func() []StorageAPI {
disks := original()
wrapped := make([]StorageAPI, len(disks))
for i, disk := range disks {
if disk != nil {
wrapped[i] = multipartListingCountingDisk{disk, &directoryCalls, &metadataCalls}
}
}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
}
}
measure := func(label string) (int64, int64) {
t.Helper()
directoryCalls.Store(0)
metadataCalls.Store(0)
result, err := z.ListMultipartUploads(t.Context(), bucket, "dir/", "", "", "", 1)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, result, "dir/target")
d, m := directoryCalls.Load(), metadataCalls.Load()
t.Logf("%s: max-uploads=1 returned=%d ListDir=%d ReadVersion=%d", label, len(result.Uploads), d, m)
return d, m
}
_, baseline := measure("no unrelated uploads")
for n := 0; n < 32; n++ {
if _, err := z.NewMultipartUpload(t.Context(), otherBucket, fmt.Sprintf("other/%03d", n), ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
_, loaded := measure("32 uploads in another bucket")
_, repeated := measure("same query repeated")
if loaded != baseline+32 || repeated != loaded {
t.Fatalf("unexpected scan accounting: baseline=%d loaded=%d repeated=%d", baseline, loaded, repeated)
}
}
func multipartListingTestID(created time.Time, n int) string {
return base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("00000000-0000-4000-8000-000000000000.00000000-0000-4000-8000-%012xx%d", n, created.UnixNano())))
}
func multipartListingFixture(t *testing.T) (*erasureServerPools, *erasureObjects, string) {
t.Helper()
obj, dirs, err := prepareErasure(t.Context(), 4)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
const bucket = "multipart-listing-test"
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
return z, z.serverPools[0].getHashedSet("a"), bucket
}
func TestMultipartListingMissingMarkers(t *testing.T) {
base := time.Unix(100, 0)
sameTime := []string{multipartListingTestID(base.Add(time.Second), 1), multipartListingTestID(base.Add(time.Second), 2), multipartListingTestID(base.Add(time.Second), 3)}
slices.Sort(sameTime)
uploads := []MultipartInfo{
{Bucket: "bucket", Object: "a", UploadID: multipartListingTestID(base, 1), Initiated: base},
{Bucket: "bucket", Object: "a", UploadID: sameTime[1], Initiated: base.Add(time.Second)},
{Bucket: "bucket", Object: "b", UploadID: multipartListingTestID(base, 4), Initiated: base},
}
for _, tc := range []struct {
name string
created time.Time
id int
want []string
}{
{"before", base.Add(-time.Second), 0, []string{"a", "a", "b"}},
{"between", base.Add(time.Second / 2), 2, []string{"a", "b"}},
{"same-time-before", base.Add(time.Second), 2, []string{"a", "b"}},
{"same-time-after", base.Add(time.Second), 4, []string{"b"}},
{"after", base.Add(2 * time.Second), 5, []string{"b"}},
} {
t.Run(tc.name, func(t *testing.T) {
marker := multipartListingTestID(tc.created, tc.id)
if tc.name == "same-time-before" {
marker = sameTime[0]
}
if tc.name == "same-time-after" {
marker = sameTime[2]
}
if err := checkListMultipartArgs(t.Context(), "bucket", "", "a", marker, ""); err != nil {
t.Fatal(err)
}
got := paginateMultipartUploads(uploads, "", "a", marker, "", 10)
if !slices.Equal(multipartUploadKeys(got.Uploads), tc.want) {
t.Fatalf("got %v want %v", multipartUploadKeys(got.Uploads), tc.want)
}
})
}
}
func TestMultipartListingAbortRecovery(t *testing.T) {
for _, offline := range []int{1, 2} {
t.Run(fmt.Sprint(offline), func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i := 0; i < offline; i++ {
disks[i] = nil
}
return disks
}
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if offline == 1 && err != nil {
t.Fatal(err)
}
if offline == 2 && (err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503) {
t.Fatalf("two offline drives must not acknowledge cancellation: %v", err)
}
set.getDisks = original
if offline == 2 {
// Retry a partially completed deletion after the original disks return.
if err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 100)
if err != nil || len(got.Uploads) != 0 {
t.Fatalf("canceled upload reappeared: %+v %v", got, err)
}
})
}
}
func TestMultipartListingCoverage(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
disks := original()
// Leave a readable two-copy record, simulating a partially failed abort.
for _, d := range disks[:2] {
if err = d.Delete(t.Context(), minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
set.getDisks = func() []StorageAPI { return []StorageAPI{disks[0], disks[1], nil, nil} }
_, err = z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("incomplete discovery returned success: %v", err)
}
set.getDisks = func() []StorageAPI { return []StorageAPI{nil, disks[1], disks[2], disks[3]} }
got, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 10)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, got, "a")
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || report.Complete || len(report.Sets[0].UncoveredDrives) != 1 {
t.Fatalf("offline upgrade preflight: %+v %v", report, err)
}
}
func TestMultipartListingBudgetAndAdmission(t *testing.T) {
_, set, bucket := multipartListingFixture(t)
if _, err := set.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{}); err != nil {
t.Fatal(err)
}
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
scan.remaining.Store(1)
_, _, err = set.scanMultipartUploads(scan, bucket, 0, 0)
var limited SlowDown
if !errors.As(err, &limited) {
t.Fatalf("budget: %v", err)
}
if apiErr := toAPIError(t.Context(), err); apiErr.HTTPStatusCode != http.StatusServiceUnavailable || apiErr.Code != "SlowDown" {
t.Fatalf("budget error mapping: %+v", apiErr)
}
second, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer second.close()
if third, err := startMultipartScan(t.Context(), false); err == nil {
third.close()
t.Fatal("third scan admitted")
}
}
func TestMultipartListingAdmissionHTTP(t *testing.T) {
z, _, _ := multipartListingFixture(t)
bucket, router, err := initAPIHandlerTest(t.Context(), z, []string{"ListMultipartUploads"}, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
setMultipartListingTestMode(t, false)
for range cap(multipartScanSlots) {
scan, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer scan.close()
}
req, err := newTestSignedRequestV4(http.MethodGet,
getListMultipartUploadsURLWithParams("", bucket, "", "", "", "", "1"),
0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
var response APIErrorResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &response); err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusServiceUnavailable || response.Code != "SlowDown" {
t.Fatalf("admission returned %d %s", rec.Code, rec.Body.String())
}
}
func TestMultipartListingPreflightMinorityLegacy(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
path := set.getUploadIDDir(bucket, "old", mp.UploadID)
disks := set.getDisks()
fi, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, path, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
if err = disks[0].WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, path, fi); err != nil {
t.Fatal(err)
}
for _, d := range disks[1:] {
if err = d.Delete(t.Context(), minioMetaMultipartBucket, path, DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || !report.Complete || report.LegacyUploads != 1 {
t.Fatalf("minority legacy copy must prevent readiness: %+v %v", report, err)
}
}
func TestMultipartListingCancellationRetainsAdmission(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
for i := range 40 {
if _, err := z.NewMultipartUpload(t.Context(), bucket, fmt.Sprintf("key-%02d", i), ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
original := set.getDisks
t.Cleanup(func() { set.getDisks = original })
entered := make(chan struct{}, 40)
release := make(chan struct{})
var once sync.Once
defer once.Do(func() { close(release) })
var reads atomic.Int32
set.getDisks = func() []StorageAPI {
disks := append([]StorageAPI(nil), original()...)
for i, d := range disks {
disks[i] = multipartListingFaultDisk{StorageAPI: d, read: func(ctx context.Context, b, v, p, version string, opts ReadOptions) (FileInfo, error) {
if v != minioMetaMultipartBucket {
return d.ReadVersion(ctx, b, v, p, version, opts)
}
reads.Add(1)
entered <- struct{}{}
// Model an RPC which does not return immediately on cancellation.
<-release
return FileInfo{}, ctx.Err()
}}
}
return disks
}
second, err := startMultipartScan(t.Context(), false)
if err != nil {
t.Fatal(err)
}
defer second.close()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() {
_, err := z.ListMultipartUploads(ctx, bucket, "", "", "", "", 10)
done <- err
}()
for range 16 {
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("identity workers did not start")
}
}
cancel()
if extra, err := startMultipartScan(t.Context(), false); err == nil {
extra.close()
t.Fatal("canceled request released admission while RPCs were still running")
}
start := time.Now()
once.Do(func() { close(release) })
select {
case err := <-done:
if err == nil {
t.Fatal("canceled scan returned a successful partial list")
}
case <-time.After(2 * time.Second):
t.Fatal("scan did not stop after blocked RPCs returned")
}
if got := reads.Load(); got != 16 {
t.Fatalf("scheduled more identity/metadata reads after cancellation: %d", got)
}
t.Logf("16 identity RPCs bounded; admission held until return; cancellation settled in %s", time.Since(start))
}
+331
View File
@@ -0,0 +1,331 @@
// Copyright (c) 2026 Ruohang Feng
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"context"
"encoding/base64"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
"github.com/minio/minio/internal/config/api"
"github.com/minio/minio/internal/config/storageclass"
"github.com/minio/minio/internal/dsync"
"github.com/minio/minio/internal/grid"
xnet "github.com/pgsty/silo-pkg/v3/net"
)
func setMultipartListingTestMode(t *testing.T, legacy bool) {
t.Helper()
globalAPIConfig.mu.Lock()
previous := globalAPIConfig.multipartListingStrict
globalAPIConfig.multipartListingStrict = !legacy
globalAPIConfig.mu.Unlock()
t.Cleanup(func() {
globalAPIConfig.mu.Lock()
globalAPIConfig.multipartListingStrict = previous
globalAPIConfig.mu.Unlock()
})
}
func TestMultipartListingDefaultMode(t *testing.T) {
var uninitialized apiConfig
if !uninitialized.getMultipartListingLegacy() {
t.Fatal("runtime zero value enabled strict mode before config initialization")
}
for _, mode := range []string{"", "legacy", "strict"} {
var local apiConfig
local.init(api.Config{RequestsMax: 1, MultipartListing: mode}, []int{4}, false)
if local.getMultipartListingLegacy() != (mode != "strict") {
t.Fatalf("unexpected mode for %q", mode)
}
}
}
func TestMultipartAbortLegacyAvailability(t *testing.T) {
for _, drives := range []int{4, 16} {
for _, offline := range []int{0, 1, drives / 2} {
t.Run(fmt.Sprintf("drives=%d/offline=%d", drives, offline), func(t *testing.T) {
obj, dirs, err := prepareErasure(t.Context(), drives)
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { z.Shutdown(context.Background()); removeRoots(dirs) })
setMultipartListingTestMode(t, true)
previous := globalStorageClass
globalStorageClass.Update(storageclass.Config{Standard: storageclass.StorageClass{Parity: drives / 2}})
t.Cleanup(func() { globalStorageClass.Update(previous) })
const bucket, key = "multipart-legacy-availability", "keep-available"
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
set := z.serverPools[0].getHashedSet(key)
original := set.getDisks
disks := original()
set.getDisks = func() []StorageAPI {
visible := append([]StorageAPI(nil), disks...)
clear(visible[:offline])
return visible
}
t.Cleanup(func() { set.getDisks = original })
if err := z.AbortMultipartUpload(t.Context(), bucket, key, mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("released read-quorum availability regressed: %v", err)
}
for _, disk := range disks[offline:] {
_, err := disk.ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, key, mp.UploadID), "", ReadOptions{})
if !errors.Is(err, errFileNotFound) {
t.Fatalf("online replica not cleaned: %v", err)
}
}
})
}
}
}
func TestMultipartAbortLegacyPoolOrder(t *testing.T) {
for _, owner := range []int{0, 1} {
t.Run(fmt.Sprintf("owner=%d", owner), func(t *testing.T) {
z, bucket := consistencyPools(t)
setMultipartListingTestMode(t, true)
mp, err := z.serverPools[owner].NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
emptySet := z.serverPools[1-owner].getHashedSet("a")
original := emptySet.getDisks
t.Cleanup(func() { emptySet.getDisks = original })
emptySet.getDisks = func() []StorageAPI { return make([]StorageAPI, emptySet.setDriveCount) }
err = z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{})
if owner == 0 && err != nil {
t.Fatalf("unrelated later pool blocked legacy cancellation: %v", err)
}
if owner == 1 {
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("unknown earlier pool must retain released error behavior: %v", err)
}
if _, err := z.serverPools[owner].GetMultipartInfo(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("later pool visited despite earlier error: %v", err)
}
emptySet.getDisks = original
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
})
}
}
func TestMultipartAbortMinorityLegacyCleanup(t *testing.T) {
for _, failDelete := range []bool{false, true} {
t.Run(fmt.Sprintf("delete-fails=%v", failDelete), func(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "old", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
path := set.getUploadIDDir(bucket, "old", mp.UploadID)
original := set.getDisks
disks := original()
fi, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, path, "", ReadOptions{})
if err != nil {
t.Fatal(err)
}
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
if err := disks[0].WriteMetadata(t.Context(), bucket, minioMetaMultipartBucket, path, fi); err != nil {
t.Fatal(err)
}
for _, d := range disks[1:] {
if err := d.Delete(t.Context(), minioMetaMultipartBucket, path, DeleteOptions{Recursive: true}); err != nil {
t.Fatal(err)
}
}
report, err := z.multipartPreflight(t.Context())
if err != nil || report.Ready || report.LegacyUploads != 1 {
t.Fatalf("invalid legacy fixture: %+v %v", report, err)
}
if failDelete {
set.getDisks = func() []StorageAPI {
wrapped := append([]StorageAPI(nil), disks...)
wrapped[0] = multipartListingFaultDisk{StorageAPI: disks[0], delete: func(context.Context, string, string, DeleteOptions) error { return errFaultyDisk }}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
err := z.AbortMultipartUpload(t.Context(), bucket, "old", mp.UploadID, ObjectOptions{})
if err == nil || toAPIError(t.Context(), err).HTTPStatusCode != 503 {
t.Fatalf("empty drives masked failed deletion of the observed replica: %v", err)
}
if _, present := z.mpCache.Load(mp.UploadID); !present {
t.Fatal("failed cancellation evicted upload from cache")
}
set.getDisks = original
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "old", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
report, err = z.multipartPreflight(t.Context())
if err != nil || !report.Ready || report.LegacyUploads != 0 {
t.Fatalf("acknowledged cleanup left online legacy remnants: %+v %v", report, err)
}
if _, present := z.mpCache.Load(mp.UploadID); present {
t.Fatal("successful cancellation retained cache entry")
}
})
}
}
func TestMultipartAbortWrongTargetKeepsCache(t *testing.T) {
for _, legacy := range []bool{true, false} {
t.Run(fmt.Sprintf("legacy=%v", legacy), func(t *testing.T) {
z, _, bucket := multipartListingFixture(t)
setMultipartListingTestMode(t, legacy)
const other = "multipart-other-bucket"
if err := z.MakeBucket(t.Context(), other, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
mp, err := z.NewMultipartUpload(t.Context(), bucket, "valid-key", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, target := range [][2]string{{bucket, "wrong-key"}, {other, "valid-key"}} {
err := z.AbortMultipartUpload(t.Context(), target[0], target[1], mp.UploadID, ObjectOptions{})
var invalid InvalidUploadID
if !errors.As(err, &invalid) {
t.Fatalf("wrong target result: %v", err)
}
if _, present := z.mpCache.Load(mp.UploadID); !present {
t.Fatal("wrong-target abort evicted another upload")
}
if _, err := z.GetMultipartInfo(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("wrong-target abort harmed valid upload: %v", err)
}
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid-key", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("repeat abort: %v", err)
}
})
}
}
func TestMultipartAbortRejectsUnsafeID(t *testing.T) {
for _, legacy := range []bool{true, false} {
t.Run(fmt.Sprintf("legacy=%v", legacy), func(t *testing.T) {
z, _, bucket := multipartListingFixture(t)
setMultipartListingTestMode(t, legacy)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "keep", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, suffix := range []string{".", "..", "../other", "a/b", "a\\b", ""} {
id := base64.RawURLEncoding.EncodeToString([]byte("deployment." + suffix))
err := z.AbortMultipartUpload(t.Context(), bucket, "keep", id, ObjectOptions{})
var invalid InvalidUploadID
if !errors.As(err, &invalid) {
t.Fatalf("unsafe ID accepted: suffix=%q err=%v", suffix, err)
}
if _, err := z.GetMultipartInfo(t.Context(), bucket, "keep", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("valid upload harmed by rejected ID: %v", err)
}
}
})
}
}
func TestMultipartAbortPeerNotification(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
tg, err := grid.SetupTestGrid(2)
if err != nil {
t.Fatal(err)
}
t.Cleanup(tg.Cleanup)
var notifications atomic.Int32
if err := cleanupUploadIDCacheMetaRPC.Register(tg.Managers[1], func(*grid.MSS) (grid.NoPayload, *grid.RemoteErr) {
notifications.Add(1)
return grid.NoPayload{}, nil
}); err != nil {
t.Fatal(err)
}
host, err := xnet.ParseHost(strings.TrimPrefix(tg.Hosts[1], "http://"))
if err != nil {
t.Fatal(err)
}
previous := globalNotificationSys
globalNotificationSys = &NotificationSys{peerClients: []*peerRESTClient{{
host: host,
gridConn: func() *grid.Connection { return tg.Managers[0].Connection(tg.Hosts[1]) },
}}}
t.Cleanup(func() { globalNotificationSys = previous })
// Use the real distributed lock implementation: Unlock cancels its derived
// context. Local locks do not, and would hide a broken notification context.
oldMutex, oldLockers := set.nsMutex, set.getLockers
lockers := []dsync.NetLocker{newLocker(), newLocker()}
set.nsMutex = newNSLock(true)
set.getLockers = func() ([]dsync.NetLocker, string) { return lockers, "multipart-test" }
t.Cleanup(func() { set.nsMutex, set.getLockers = oldMutex, oldLockers })
for _, legacy := range []bool{true, false} {
t.Run(fmt.Sprintf("legacy=%v", legacy), func(t *testing.T) {
setMultipartListingTestMode(t, legacy)
for range 4 {
mp, err := z.NewMultipartUpload(t.Context(), bucket, "valid", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
before := notifications.Load()
var invalid InvalidUploadID
if err := z.AbortMultipartUpload(t.Context(), bucket, "wrong", mp.UploadID, ObjectOptions{}); !errors.As(err, &invalid) {
t.Fatalf("wrong target: %v", err)
}
if notifications.Load() != before {
t.Fatal("wrong-target abort sent a destructive peer notification")
}
if err := z.AbortMultipartUpload(t.Context(), bucket, "valid", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
if notifications.Load() != before+1 {
t.Fatal("successful abort lost peer notification after unlocking")
}
}
})
}
}
func TestMultipartAbortStrictMajority(t *testing.T) {
z, set, bucket := multipartListingFixture(t)
mp, err := z.NewMultipartUpload(t.Context(), bucket, "a", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original := set.getDisks
disks := original()
set.getDisks = func() []StorageAPI {
wrapped := append([]StorageAPI(nil), disks...)
wrapped[0] = multipartListingFaultDisk{StorageAPI: disks[0], delete: func(context.Context, string, string, DeleteOptions) error { return errFaultyDisk }}
return wrapped
}
t.Cleanup(func() { set.getDisks = original })
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatalf("ordinary majority cancellation was tightened: %v", err)
}
// One failed deletion is tolerated in the ordinary majority path. Retrying
// after recovery must also clean the now-minority remnant.
if _, err := disks[0].ReadVersion(t.Context(), bucket, minioMetaMultipartBucket, set.getUploadIDDir(bucket, "a", mp.UploadID), "", ReadOptions{}); err != nil {
t.Fatalf("fault fixture did not leave a remnant: %v", err)
}
set.getDisks = original
if err := z.AbortMultipartUpload(t.Context(), bucket, "a", mp.UploadID, ObjectOptions{}); err != nil {
t.Fatal(err)
}
}
+297 -35
View File
@@ -31,6 +31,7 @@ import (
"sync"
"time"
"github.com/google/uuid"
"github.com/klauspost/readahead"
"github.com/minio/minio-go/v7/pkg/set"
"github.com/minio/minio/internal/config/storageclass"
@@ -44,6 +45,15 @@ import (
"github.com/pgsty/silo-pkg/v3/sync/errgroup"
)
const (
multipartMetaBucket = ReservedMetadataPrefixLower + "multipart-v1-bucket"
multipartMetaObject = ReservedMetadataPrefixLower + "multipart-v1-object"
// ponytail: keep scan concurrency fixed until the multipart-list benchmark
// establishes a better adaptive limit.
multipartMetadataScanConcurrency = 4
)
func (er erasureObjects) getUploadIDDir(bucket, object, uploadID string) string {
uploadUUID := uploadID
uploadBytes, err := base64.RawURLEncoding.DecodeString(uploadID)
@@ -251,14 +261,152 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
})
}
// ListMultipartUploads - lists all the pending multipart
// uploads for a particular object in a bucket.
//
// Implements minimal S3 compatible ListMultipartUploads API. We do
// not support prefix based listing, this is a deliberate attempt
// towards simplification of multipart APIs.
// The resulting ListMultipartsInfo structure is unmarshalled directly as XML.
func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
func multipartUploadInfo(bucket, object, uploadUUID string, fallback time.Time) MultipartInfo {
initiated := fallback
if parsed, ok := multipartUploadTime(uploadUUID); ok {
initiated = parsed
}
return MultipartInfo{
Bucket: bucket,
Object: object,
UploadID: base64.RawURLEncoding.EncodeToString(fmt.Appendf(nil, "%s.%s", globalDeploymentID(), uploadUUID)),
Initiated: initiated,
}
}
// multipartUploadTime is shared by stored records and continuation markers.
// The time is part of the immutable upload ID, so a removed marker still
// identifies the same ordering boundary.
func multipartUploadTime(uploadUUID string) (time.Time, bool) {
if len(uploadUUID) < 38 || uploadUUID[36] != 'x' {
return time.Time{}, false
}
if _, err := uuid.Parse(uploadUUID[:36]); err != nil {
return time.Time{}, false
}
ns, err := strconv.ParseInt(uploadUUID[37:], 10, 64)
if err != nil || ns <= 0 || strconv.FormatInt(ns, 10) != uploadUUID[37:] {
return time.Time{}, false
}
return time.Unix(0, ns), true
}
func multipartMarkerTime(uploadID string) (time.Time, bool) {
b, err := base64.RawURLEncoding.DecodeString(uploadID)
if err != nil {
return time.Time{}, false
}
_, uploadUUID, ok := strings.Cut(string(b), ".")
if !ok {
return time.Time{}, false
}
return multipartUploadTime(uploadUUID)
}
type multipartListEntry struct {
upload *MultipartInfo
commonPrefix string
}
// paginateMultipartUploads applies the S3 ordering, prefix, delimiter, marker,
// and page rules exactly once after all pools and sets have been merged.
func paginateMultipartUploads(uploads []MultipartInfo, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) ListMultipartsInfo {
if maxUploads > maxUploadsList {
maxUploads = maxUploadsList
}
result := ListMultipartsInfo{
MaxUploads: maxUploads,
KeyMarker: keyMarker,
UploadIDMarker: uploadIDMarker,
Prefix: prefix,
Delimiter: delimiter,
}
deduplicated := make([]MultipartInfo, 0, len(uploads))
seenUploads := make(map[string]struct{}, len(uploads))
for _, upload := range uploads {
identity := upload.Bucket + "\x00" + upload.Object + "\x00" + upload.UploadID
if _, ok := seenUploads[identity]; ok {
continue
}
seenUploads[identity] = struct{}{}
deduplicated = append(deduplicated, upload)
}
sort.Slice(deduplicated, func(i, j int) bool {
if deduplicated[i].Object != deduplicated[j].Object {
return deduplicated[i].Object < deduplicated[j].Object
}
if !deduplicated[i].Initiated.Equal(deduplicated[j].Initiated) {
return deduplicated[i].Initiated.Before(deduplicated[j].Initiated)
}
return deduplicated[i].UploadID < deduplicated[j].UploadID
})
markerTime, _ := multipartMarkerTime(uploadIDMarker)
seenPrefixes := make(map[string]struct{})
entries := make([]multipartListEntry, 0, len(deduplicated))
for i := range deduplicated {
upload := &deduplicated[i]
if !strings.HasPrefix(upload.Object, prefix) {
continue
}
if keyMarker != "" {
switch strings.Compare(upload.Object, keyMarker) {
case -1:
continue
case 0:
if uploadIDMarker == "" || upload.Initiated.Before(markerTime) ||
(upload.Initiated.Equal(markerTime) && upload.UploadID <= uploadIDMarker) {
continue
}
}
}
if delimiter != "" {
remainder := strings.TrimPrefix(upload.Object, prefix)
if i := strings.Index(remainder, delimiter); i >= 0 {
commonPrefix := prefix + remainder[:i+len(delimiter)]
if keyMarker != "" && commonPrefix <= keyMarker {
continue
}
if _, ok := seenPrefixes[commonPrefix]; ok {
continue
}
seenPrefixes[commonPrefix] = struct{}{}
entries = append(entries, multipartListEntry{commonPrefix: commonPrefix})
continue
}
}
entries = append(entries, multipartListEntry{upload: upload})
}
if maxUploads <= 0 {
return result
}
pageSize := min(maxUploads, len(entries))
for _, entry := range entries[:pageSize] {
if entry.upload != nil {
result.Uploads = append(result.Uploads, *entry.upload)
continue
}
result.CommonPrefixes = append(result.CommonPrefixes, entry.commonPrefix)
}
result.IsTruncated = pageSize < len(entries)
if result.IsTruncated && pageSize > 0 {
last := entries[pageSize-1]
if last.upload != nil {
result.NextKeyMarker = last.upload.Object
result.NextUploadIDMarker = last.upload.UploadID
} else {
result.NextKeyMarker = last.commonPrefix
}
}
return result
}
// listMultipartUploadsExact preserves the hashed exact-object lookup used by
// multipart write placement and by rolling-upgrade legacy mode.
func (er erasureObjects) listMultipartUploadsExact(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
auditObjectErasureSet(ctx, "ListMultipartUploads", object, &er)
result.MaxUploads = maxUploads
@@ -311,20 +459,16 @@ func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, objec
if populatedUploadIDs.Contains(uploadID) {
continue
}
// If present, use time stored in ID.
startTime := time.Now()
if split := strings.Split(uploadID, "x"); len(split) == 2 {
t, err := strconv.ParseInt(split[1], 10, 64)
if err == nil {
startTime = time.Unix(0, t)
var fallback time.Time
if _, ok := multipartUploadTime(uploadID); !ok {
fi, err := disk.ReadVersion(ctx, bucket, minioMetaMultipartBucket,
pathJoin(er.getMultipartSHADir(bucket, object), uploadID), "", ReadOptions{})
if err != nil {
return result, toObjectErr(err, bucket, object)
}
fallback = fi.ModTime
}
uploads = append(uploads, MultipartInfo{
Bucket: bucket,
Object: object,
UploadID: base64.RawURLEncoding.EncodeToString(fmt.Appendf(nil, "%s.%s", globalDeploymentID(), uploadID)),
Initiated: startTime,
})
uploads = append(uploads, multipartUploadInfo(bucket, object, uploadID, fallback))
populatedUploadIDs.Add(uploadID)
}
@@ -365,6 +509,25 @@ func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, objec
return result, nil
}
func (er erasureObjects) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
uploads, legacy, err := er.scanMultipartUploads(scan, bucket, 0, 0)
if err != nil {
return ListMultipartsInfo{}, err
}
if legacy {
return ListMultipartsInfo{}, errMultipartListingLegacy
}
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
// newMultipartUpload - wrapper for initializing a new multipart
// request; returns a unique upload id.
//
@@ -402,6 +565,8 @@ func (er erasureObjects) newMultipartUpload(ctx context.Context, bucket string,
}
userDefined := cloneMSS(opts.UserDefined)
userDefined[multipartMetaBucket] = bucket
userDefined[multipartMetaObject] = object
if opts.PreserveETag != "" {
userDefined["etag"] = opts.PreserveETag
}
@@ -1459,6 +1624,8 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
// Remove superfluous internal headers.
delete(fi.Metadata, hash.MinIOMultipartChecksum)
delete(fi.Metadata, hash.MinIOMultipartChecksumType)
delete(fi.Metadata, multipartMetaBucket)
delete(fi.Metadata, multipartMetaObject)
// Save the final object size and modtime.
fi.Size = objectSize
@@ -1529,6 +1696,32 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
defer lk.Unlock(lkctx)
}
if !opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement {
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
if fi.VersionID == "" || fi.VersionID == nullVersionID {
if err := checkObjectLockedForOverwrite(ctx, bucket, object, getObjectInfo); err != nil {
return oi, toObjectErr(err, bucket, object)
}
} else {
_, gerr := getObjectInfo(ctx, bucket, object, ObjectOptions{
VersionID: fi.VersionID,
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoLock: true,
})
switch {
case gerr == nil, errors.As(gerr, &MethodNotAllowed{}):
return oi, toObjectErr(errAuthentication, bucket, object)
case isErrVersionNotFound(gerr) || isErrObjectNotFound(gerr):
default:
return oi, toObjectErr(gerr, bucket, object)
}
}
}
er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename().
defer func() {
@@ -1586,22 +1779,91 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
return fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
}
// AbortMultipartUpload - aborts an ongoing multipart operation
// signified by the input uploadID. This is an atomic operation
// doesn't require clients to initiate multiple such requests.
//
// All parts are purged from all disks and reference to the uploadID
// would be removed from the system, rollback is not possible on this
// operation.
func (er erasureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (err error) {
// abortMultipartUpload retains read-quorum validation and best-effort cleanup
// in legacy mode. Strict mode requires majority deletion acknowledgements and
// permits retrying remnants below read quorum. Neither mode fences creation
// writes still executing after a storage timeout.
func (er erasureObjects) abortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions, legacy bool) (bool, error) {
if !opts.NoAuditLog {
auditObjectErasureSet(ctx, "AbortMultipartUpload", object, &er)
}
// Cleanup all uploaded parts.
defer er.deleteAll(ctx, minioMetaMultipartBucket, er.getUploadIDDir(bucket, object, uploadID))
// Validates if upload ID exists.
_, _, err = er.checkUploadIDExists(ctx, bucket, object, uploadID, false)
return toObjectErr(err, bucket, object, uploadID)
b, err := base64.RawURLEncoding.DecodeString(uploadID)
if err != nil {
return false, MalformedUploadID{UploadID: uploadID}
}
_, internalID, ok := strings.Cut(string(b), ".")
if !ok || internalID == "" || internalID == "." || internalID == ".." || strings.ContainsAny(internalID, "/\\") {
return false, InvalidUploadID{Bucket: bucket, Object: object, UploadID: uploadID}
}
if legacy {
// Keep the released read-quorum and best-effort cleanup behavior.
// The upload ID safety check above applies to both modes.
defer er.deleteAll(ctx, minioMetaMultipartBucket, er.getUploadIDDir(bucket, object, uploadID))
_, _, err := er.checkUploadIDExists(ctx, bucket, object, uploadID, false)
err = toObjectErr(err, bucket, object, uploadID)
if _, absent := err.(InvalidUploadID); absent {
return false, nil
}
return err == nil, err
}
disks := er.getDisks()
uploadPath := er.getUploadIDDir(bucket, object, uploadID)
_, errs := readAllFileInfo(ctx, disks, bucket, minioMetaMultipartBucket, uploadPath, "", false, false)
quorum := er.setDriveCount/2 + 1
found, absent := false, 0
observed := make([]bool, len(disks))
for i, err := range errs {
switch {
case err == nil, errors.Is(err, errFileCorrupt):
found = true
observed[i] = true
case errors.Is(err, errFileNotFound), errors.Is(err, errFileVersionNotFound):
absent++
}
}
if absent >= quorum && !found {
return false, nil
}
if !found {
return false, toObjectErr(errErasureReadQuorum, bucket, object, uploadID)
}
g := errgroup.WithNErrs(len(disks))
for i, disk := range disks {
g.Go(func() error {
if disk == nil {
return errDiskNotFound
}
err := disk.Delete(ctx, minioMetaMultipartBucket, uploadPath, DeleteOptions{Recursive: true, Immediate: false})
if errors.Is(err, errFileNotFound) || errors.Is(err, errFileVersionNotFound) {
return nil
}
return err
}, i)
}
deleteErrs := g.Wait()
if err := reduceWriteQuorumErrs(ctx, deleteErrs, nil, quorum); err != nil {
return true, toObjectErr(err, bucket, object, uploadID)
}
if absent >= quorum {
// Missing disks must not mask a failed cleanup of known remnants.
for i, found := range observed {
if found && deleteErrs[i] != nil {
return true, toObjectErr(errErasureWriteQuorum, bucket, object, uploadID)
}
}
}
return true, nil
}
// AbortMultipartUpload cancels an upload using the configured mode. Offline
// part data may still need stale-upload cleanup after its drives return.
func (er erasureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string, opts ObjectOptions) (err error) {
found, err := er.abortMultipartUpload(ctx, bucket, object, uploadID, opts, globalAPIConfig.getMultipartListingLegacy())
if err != nil {
return err
}
if !found {
return InvalidUploadID{Bucket: bucket, Object: object, UploadID: uploadID}
}
return nil
}
+58 -6
View File
@@ -150,6 +150,17 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
}
}
if !dstOpts.ReplicationRequest && !dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement &&
(versionID == "" || versionID == nullVersionID) {
getObjectInfo := er.getObjectInfo
if dstOpts.replicaObjectInfo != nil {
getObjectInfo = dstOpts.replicaObjectInfo
}
if err := checkObjectLockedForOverwrite(ctx, dstBucket, dstObject, getObjectInfo); err != nil {
return oi, toObjectErr(err, dstBucket, dstObject)
}
}
modTime = UTCNow() // We only preserve modTime if dstOpts.MTime is true.
// in all other cases mtime is latest.
@@ -1594,6 +1605,17 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
defer lk.Unlock(lkctx)
}
if !opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement &&
(fi.VersionID == "" || fi.VersionID == nullVersionID) {
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
if err := checkObjectLockedForOverwrite(ctx, bucket, object, getObjectInfo); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
}
// Rename the successfully written temporary object to final location.
onlineDisks, versions, oldDataDir, err := renameData(ctx, onlineDisks, minioMetaTmpBucket, tempObj, partsMetadata, bucket, object, writeQuorum)
if err != nil {
@@ -1657,7 +1679,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
return fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
}
func (er erasureObjects) deleteObjectVersion(ctx context.Context, bucket, object string, fi FileInfo, forceDelMarker bool) error {
func (er erasureObjects) deleteObjectVersion(ctx context.Context, bucket, object string, fi FileInfo, forceDelMarker, purge bool) error {
disks := er.getDisks()
// Assume (N/2 + 1) quorum for Delete()
// this is a theoretical assumption such that
@@ -1673,7 +1695,13 @@ func (er erasureObjects) deleteObjectVersion(ctx context.Context, bucket, object
if disks[index] == nil {
return errDiskNotFound
}
return disks[index].DeleteVersion(ctx, bucket, object, fi, forceDelMarker, DeleteOptions{})
err := disks[index].DeleteVersion(ctx, bucket, object, fi, forceDelMarker, DeleteOptions{})
// Physical removal and reliable already-absent replies are the same
// outcome. Creation and metadata updates must not use this quorum.
if purge && (err == errFileNotFound || err == errFileVersionNotFound) {
return nil
}
return err
}, index)
}
// return errors if any during deletion
@@ -2023,6 +2051,11 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
if opts.DeleteMarker {
versionFound = false
} else if !tryDel {
if opts.isVersionPurge() && (isErrObjectNotFound(gerr) || isErrVersionNotFound(gerr)) {
if err := er.checkPurgeAbsent(ctx, bucket, object, opts.VersionID); err != nil {
return objInfo, err
}
}
return objInfo, gerr
}
}
@@ -2105,6 +2138,13 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
}
}
purge := opts.isVersionPurge()
if purge {
// A delete marker describes the version being removed; it is not an
// instruction to create that marker on disks which already lack it.
markDelete, deleteMarker = false, false
}
modTime := opts.MTime
if opts.MTime.IsZero() {
modTime = UTCNow()
@@ -2151,7 +2191,7 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
// delete marker. Add delete marker, since we don't have
// any version specified explicitly. Or if a particular
// version id needs to be replicated.
if err = er.deleteObjectVersion(ctx, bucket, object, fi, opts.DeleteMarker); err != nil {
if err = er.deleteObjectVersion(ctx, bucket, object, fi, opts.DeleteMarker, false); err != nil {
return objInfo, toObjectErr(err, bucket, object)
}
oi := fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended)
@@ -2174,11 +2214,19 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
if opts.SkipFreeVersion {
dfi.SetSkipTierFreeVersion()
}
if err = er.deleteObjectVersion(ctx, bucket, object, dfi, opts.DeleteMarker); err != nil {
if err = er.deleteObjectVersion(ctx, bucket, object, dfi, opts.DeleteMarker, purge); err != nil {
return objInfo, toObjectErr(err, bucket, object)
}
return dfi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended), nil
oi := dfi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended)
if purge {
// Preserve the DELETE response's identity without reusing it as a disk
// instruction to create a marker. The lookup also exposes a data
// version pending purge as deleted for visibility; only a stored
// marker, which carries no erasure layout, is reported as one.
oi.DeleteMarker = goi.DeleteMarker && goi.DataBlocks == 0
}
return oi, nil
}
// Send the successful but partial upload/delete, however ignore
@@ -2327,7 +2375,11 @@ func (er erasureObjects) PutObjectTags(ctx context.Context, bucket, object strin
fi.Metadata[xhttp.AmzObjectTagging] = tags
fi.ReplicationState = opts.PutReplicationState()
stamp := monotonicTaggingTimestamp(opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
maps.Copy(fi.Metadata, opts.UserDefined)
if stamp != "" {
fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
}
if err = er.updateObjectMeta(ctx, bucket, object, fi, onlineDisks); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
@@ -2463,7 +2515,7 @@ func (er erasureObjects) TransitionObject(ctx context.Context, bucket, object st
storageDisks := er.getDisks()
if err = er.deleteObjectVersion(ctx, bucket, object, fi, false); err != nil {
if err = er.deleteObjectVersion(ctx, bucket, object, fi, false, false); err != nil {
eventName = event.ObjectTransitionFailed
}
+35
View File
@@ -242,6 +242,21 @@ func reconcileStoredObjectTags(metadata map[string]string, storedTags, storedTim
}
}
// Local tagging mutations must advance the revision they overwrite, even when
// a request's clock or lock acquisition order is behind the stored revision.
// Replica writes use reconcileStoredObjectTags instead of minting a revision.
func monotonicTaggingTimestamp(incoming, stored string) string {
requested, err := time.Parse(time.RFC3339Nano, incoming)
if err != nil {
return incoming
}
current, err := time.Parse(time.RFC3339Nano, stored)
if err != nil || requested.After(current) {
return incoming
}
return current.Add(time.Nanosecond).UTC().Format(time.RFC3339Nano)
}
// A restored version still owns its tier reference even while IsRemote is
// false. Only the last copy of a reference may schedule its contents for GC.
func sharesTierObject(oi ObjectInfo, copies []PoolObjInfo) bool {
@@ -301,6 +316,11 @@ func (z *erasureServerPools) retireReplicaCopies(ctx context.Context, bucket, ob
func (z *erasureServerPools) deleteObjectReconciled(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
if err != nil {
if opts.isVersionPurge() && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
if quorumErr := z.checkPurgeAbsent(ctx, bucket, object, opts.VersionID); quorumErr != nil {
return ObjectInfo{}, quorumErr
}
}
return ObjectInfo{}, err
}
primary := copies[0]
@@ -350,6 +370,21 @@ func (z *erasureServerPools) deleteObjectReconciled(ctx context.Context, bucket,
opts.EvalMetadataFn = nil
}
}
if opts.isVersionPurge() {
// A missing pool may have only read-quorum absence. Verify every pool
// omitted by lookup before mutating the known copies.
for i, pool := range z.serverPools {
found := false
for _, copy := range copies {
found = found || copy.Index == i
}
if !found {
if err := pool.getHashedSet(object).checkPurgeAbsent(ctx, bucket, object, opts.VersionID); err != nil {
return ObjectInfo{}, err
}
}
}
}
if opts.VersionID == "" && (opts.Versioned || opts.VersionSuspended) {
// A single new delete marker hides the current version. Older versions
// remain history and must not be removed by an unqualified DELETE.
+2 -2
View File
@@ -618,7 +618,7 @@ func (z *erasureServerPools) decommissionObject(ctx context.Context, idx int, bu
if objInfo.isMultipart() {
res, err := z.NewMultipartUpload(ctx, bucket, objInfo.Name, ObjectOptions{
VersionID: objInfo.VersionID,
UserDefined: objInfo.UserDefined,
UserDefined: migrationObjectMetadata(objInfo),
NoAuditLog: true,
SrcPoolIdx: idx,
DataMovement: true,
@@ -681,7 +681,7 @@ func (z *erasureServerPools) decommissionObject(ctx context.Context, idx int, bu
SrcPoolIdx: idx,
VersionID: objInfo.VersionID,
MTime: objInfo.ModTime,
UserDefined: objInfo.UserDefined,
UserDefined: migrationObjectMetadata(objInfo),
PreserveETag: objInfo.ETag, // Preserve original ETag to ensure same metadata.
IndexCB: func() []byte {
return objInfo.Parts[0].Index // Preserve part Index to ensure decompression works.
+33
View File
@@ -0,0 +1,33 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import xhttp "github.com/minio/minio/internal/http"
// migrationObjectMetadata restores tags removed from the read representation.
// Keep the original revision, including ordered empty states, for the existing
// locked reconciliation at PUT or multipart completion. Moving is not a new
// tagging mutation, and the write must not modify the reader's metadata map.
func migrationObjectMetadata(oi ObjectInfo) map[string]string {
metadata := cloneMSS(oi.UserDefined)
delete(metadata, xhttp.AmzObjectTagging)
if oi.UserTags != "" {
metadata[xhttp.AmzObjectTagging] = oi.UserTags
}
return metadata
}
+592
View File
@@ -0,0 +1,592 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"maps"
"strings"
"sync"
"testing"
"time"
xhttp "github.com/minio/minio/internal/http"
)
const migrationTagRevision = ReservedMetadataPrefixLower + TaggingTimestamp
var migrationTagStates = []struct {
name, tags, revision string
hasRevision bool
}{
{name: "initial", tags: "team=storage&path=a%2Fb"},
{name: "empty-revision", tags: "team=storage", hasRevision: true},
{name: "invalid-revision", tags: "team=storage", revision: "invalid", hasRevision: true},
{name: "ordered", tags: "team=storage", revision: "2026-09-16T09:00:00Z", hasRevision: true},
{name: "cleared", revision: "2026-09-16T09:00:00Z", hasRevision: true},
{name: "never-tagged"},
{name: "empty-empty-revision", hasRevision: true},
{name: "empty-invalid-revision", revision: "invalid", hasRevision: true},
}
func TestMigrationObjectMetadata(t *testing.T) {
for _, state := range migrationTagStates {
for _, residual := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/residual=%t", state.name, residual), func(t *testing.T) {
raw := map[string]string{
xhttp.AmzObjectTagging: state.tags,
"x-amz-meta-team": "storage",
"x-amz-meta-empty": "",
ReservedMetadataPrefix + "compression": "opaque-compression-state",
ReservedMetadataPrefix + "sealed-key": "opaque-key-state",
"x-amz-object-lock-retain-until-date": "2030-01-01T00:00:00Z",
"x-amz-object-lock-legal-hold": "ON",
"x-amz-tagging": "keep-noncanonical-metadata",
ReservedMetadataPrefix + "actual-size": "42",
ReservedMetadataPrefix + "replica-status": "opaque-replication-state",
}
if state.hasRevision {
raw[migrationTagRevision] = state.revision
}
oi := (FileInfo{Metadata: raw}).ToObjectInfo("bucket", "object", false)
if residual {
oi.UserDefined[xhttp.AmzObjectTagging] = "stale=raw"
}
before := maps.Clone(oi.UserDefined)
got := migrationObjectMetadata(oi)
if got == nil || !maps.Equal(before, oi.UserDefined) {
t.Fatal("helper must return a new non-nil map without changing the read snapshot")
}
value, exists := got[xhttp.AmzObjectTagging]
if value != state.tags || exists != (state.tags != "") {
t.Fatalf("raw tags=(%q,%t), want (%q,%t)", value, exists, state.tags, state.tags != "")
}
stamp, present := got[migrationTagRevision]
if stamp != state.revision || present != state.hasRevision {
t.Fatalf("revision=(%q,%t), want (%q,%t)", stamp, present, state.revision, state.hasRevision)
}
for key, value := range before {
if stored, exists := got[key]; key != xhttp.AmzObjectTagging && (!exists || stored != value) {
t.Errorf("lost metadata %q", key)
}
}
got["x-amz-meta-team"] = "changed"
if !maps.Equal(before, oi.UserDefined) {
t.Fatal("output aliases the read snapshot")
}
})
}
}
for _, metadata := range []map[string]string{nil, {}} {
got := migrationObjectMetadata(ObjectInfo{UserDefined: metadata})
if got == nil || len(got) != 0 {
t.Fatalf("empty input must yield a writable empty map: %#v", got)
}
got["new"] = "value"
if len(metadata) != 0 {
t.Fatal("empty map input was aliased")
}
}
}
// Allocation ignores the host's used-space percentage, as in the existing tag
// fixtures. All object data and metadata still use real erasure-storage disks.
func migrationTestPools(t *testing.T) (*erasureServerPools, string) {
t.Helper()
z, bucket := consistencyPools(t)
for _, pool := range z.serverPools {
for _, set := range pool.sets {
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
for i := range disks {
disks[i] = tagTestCapacityDisk{StorageAPI: disks[i]}
}
set.getDisks = func() []StorageAPI { return disks }
t.Cleanup(func() { set.getDisks = original })
}
}
return z, bucket
}
func migrationTestMover(t *testing.T, z *erasureServerPools, source int, kind string) func(context.Context, int, string, *GetObjectReader) error {
t.Helper()
if kind == "rebalance" {
z.rebalMu.Lock()
z.rebalMeta = &rebalanceMeta{PoolStats: []*rebalanceStats{{}, {}}}
z.rebalMeta.PoolStats[source] = &rebalanceStats{Participating: true, Info: rebalanceInfo{Status: rebalStarted}}
z.rebalMu.Unlock()
t.Cleanup(func() { z.rebalMu.Lock(); z.rebalMeta = nil; z.rebalMu.Unlock() })
return z.rebalanceObject
}
z.poolMetaMutex.Lock()
z.poolMeta.Pools[source].Decommission = &PoolDecommissionInfo{}
z.poolMetaMutex.Unlock()
t.Cleanup(func() { z.poolMetaMutex.Lock(); z.poolMeta.Pools[source].Decommission = nil; z.poolMetaMutex.Unlock() })
return z.decommissionObject
}
func migrationTestObject(t *testing.T, z *erasureServerPools, bucket, object string, source int, multipart bool, parts []string, opts ObjectOptions) ObjectInfo {
t.Helper()
if !multipart {
oi := putConsistencyObject(t, z, bucket, object, source, strings.Join(parts, ""), opts)
return migrationTestPersistedInfo(t, z, source, oi)
}
mp, err := z.serverPools[source].NewMultipartUpload(t.Context(), bucket, object, opts)
if err != nil {
t.Fatal(err)
}
completed := make([]CompletePart, len(parts))
for i, body := range parts {
part, err := z.serverPools[source].PutObjectPart(t.Context(), bucket, object, mp.UploadID, i+1,
mustGetPutObjReader(t, strings.NewReader(body), int64(len(body)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
completed[i] = CompletePart{PartNumber: i + 1, ETag: part.ETag}
}
oi, err := z.serverPools[source].CompleteMultipartUpload(t.Context(), bucket, object, mp.UploadID, completed, opts)
if err != nil {
t.Fatal(err)
}
if !oi.isMultipart() {
t.Fatal("fixture did not create a multipart object")
}
return migrationTestPersistedInfo(t, z, source, oi)
}
func migrationTestPersistedInfo(t *testing.T, z *erasureServerPools, pool int, oi ObjectInfo) ObjectInfo {
t.Helper()
// PUT can return transient fields such as tier-free-versionID that are not
// stored. Take the same persisted read representation used by the movers.
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), oi.Bucket, oi.Name, ObjectOptions{VersionID: migrationTestVersion(oi)})
if err != nil {
t.Fatal(err)
}
return got
}
func migrationTestVersion(oi ObjectInfo) string {
if oi.VersionID == "" {
return nullVersionID
}
return oi.VersionID
}
func migrationTestReader(t *testing.T, z *erasureServerPools, source int, oi ObjectInfo) *GetObjectReader {
t.Helper()
gr, err := z.serverPools[source].GetObjectNInfo(t.Context(), oi.Bucket, oi.Name, nil, nil,
ObjectOptions{VersionID: migrationTestVersion(oi), NoLock: true, NoDecryption: true})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { gr.Close() })
return gr
}
func migrationTestStored(t *testing.T, z *erasureServerPools, pool int, original ObjectInfo, tags, revision string, hasRevision bool, body string) {
t.Helper()
version := migrationTestVersion(original)
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), original.Bucket, original.Name, ObjectOptions{VersionID: version})
if err != nil {
t.Fatal(err)
}
if got.UserTags != tags || got.ETag != original.ETag || !got.ModTime.Equal(original.ModTime) || got.VersionID != original.VersionID || got.Size != original.Size {
t.Errorf("pool %d changed tags/object identity: tags=%q want=%q version=%q/%q etag=%q/%q size=%d/%d mtime=%v/%v", pool, got.UserTags, tags, got.VersionID, original.VersionID, got.ETag, original.ETag, got.Size, original.Size, got.ModTime, original.ModTime)
}
infos, errs := readAllFileInfo(t.Context(), z.serverPools[pool].getHashedSet(original.Name).getDisks(), "", original.Bucket, original.Name, version, false, false)
for disk, fi := range infos {
if errs[disk] != nil {
t.Fatalf("pool %d disk %d: %v", pool, disk, errs[disk])
}
stamp, exists := fi.Metadata[migrationTagRevision]
if fi.Metadata[xhttp.AmzObjectTagging] != tags || stamp != revision || exists != hasRevision {
t.Errorf("pool %d disk %d: tags=%q revision=(%q,%t), want %q (%q,%t)", pool, disk, fi.Metadata[xhttp.AmzObjectTagging], stamp, exists, tags, revision, hasRevision)
}
for key, value := range original.UserDefined {
if key != migrationTagRevision && fi.Metadata[key] != value {
t.Errorf("pool %d disk %d lost metadata %q", pool, disk, key)
}
}
}
gr, err := z.serverPools[pool].GetObjectNInfo(t.Context(), original.Bucket, original.Name, nil, nil, ObjectOptions{VersionID: version})
if err != nil {
t.Fatal(err)
}
data, err := io.ReadAll(gr)
gr.Close()
if err != nil || !bytes.Equal(data, []byte(body)) {
t.Fatalf("pool %d content changed: bytes=%d err=%v", pool, len(data), err)
}
}
func TestPoolsMigrationPreservesTagState(t *testing.T) {
z, bucket := migrationTestPools(t)
for _, kind := range []string{"rebalance", "decommission"} {
for _, method := range []string{"put", "multipart"} {
for source := range 2 {
for _, state := range migrationTagStates {
t.Run(fmt.Sprintf("%s/%s/source=%d/%s", kind, method, source, state.name), func(t *testing.T) {
move := migrationTestMover(t, z, source, kind)
meta := map[string]string{xhttp.AmzObjectTagging: state.tags, "x-amz-meta-owner": "retained"}
if state.hasRevision {
meta[migrationTagRevision] = state.revision
}
oi := migrationTestObject(t, z, bucket, t.Name(), source, method == "multipart", []string{"payload"}, ObjectOptions{Versioned: true, UserDefined: meta})
migrationTestStored(t, z, source, oi, state.tags, state.revision, state.hasRevision, "payload")
gr := migrationTestReader(t, z, source, oi)
before := maps.Clone(gr.ObjInfo.UserDefined)
if err := move(t.Context(), source, bucket, gr); err != nil {
t.Fatal(err)
}
if !maps.Equal(before, gr.ObjInfo.UserDefined) {
t.Error("migration modified its source snapshot")
}
migrationTestStored(t, z, 1-source, oi, state.tags, state.revision, state.hasRevision, "payload")
})
}
}
}
}
}
type migrationTestGate struct {
entered, resume chan struct{}
once, release sync.Once
}
func newMigrationTestGate() *migrationTestGate {
return &migrationTestGate{entered: make(chan struct{}), resume: make(chan struct{})}
}
func (g *migrationTestGate) wait(ctx context.Context) error {
g.once.Do(func() { close(g.entered) })
select {
case <-g.resume:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func (g *migrationTestGate) unblock() { g.release.Do(func() { close(g.resume) }) }
type migrationTestGateReader struct {
io.Reader
ctx context.Context
gate *migrationTestGate
}
func (r migrationTestGateReader) Read(p []byte) (int, error) {
if err := r.gate.wait(r.ctx); err != nil {
return 0, err
}
return r.Reader.Read(p)
}
func TestPoolsMigrationRechecksTags(t *testing.T) {
z, bucket := migrationTestPools(t)
const recent = "2026-09-16T10:00:00Z"
for _, kind := range []string{"rebalance", "decommission"} {
for _, method := range []string{"put", "multipart"} {
for source := range 2 {
for _, tags := range []string{"state=after", ""} {
t.Run(fmt.Sprintf("%s/%s/source=%d/clear=%t", kind, method, source, tags == ""), func(t *testing.T) {
move := migrationTestMover(t, z, source, kind)
oi := migrationTestObject(t, z, bucket, t.Name(), source, method == "multipart", []string{"payload"}, ObjectOptions{
Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "state=before"},
})
gr := migrationTestReader(t, z, source, oi)
before := maps.Clone(gr.ObjInfo.UserDefined)
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
gate := newMigrationTestGate()
done, finished := make(chan error, 1), make(chan struct{})
if method == "multipart" {
// The first part read is after persisted upload initialization,
// but before completion takes the object lock and reconciles.
gr.Reader = migrationTestGateReader{Reader: gr.Reader, ctx: ctx, gate: gate}
go func() { defer close(finished); done <- move(ctx, source, bucket, gr) }()
defer func() { gate.unblock(); cancel(); <-finished }()
select {
case <-gate.entered:
case err := <-done:
t.Fatalf("migration finished before the part barrier: %v", err)
case <-ctx.Done():
t.Fatal(ctx.Err())
}
uploads, err := z.serverPools[1-source].listMultipartUploadsExact(ctx, bucket, oi.Name)
if err != nil || len(uploads.Uploads) != 1 {
t.Fatalf("upload not persisted at barrier: %+v, %v", uploads, err)
}
info, err := z.serverPools[1-source].GetMultipartInfo(ctx, bucket, oi.Name, uploads.Uploads[0].UploadID, ObjectOptions{})
if err != nil || info.UserDefined[xhttp.AmzObjectTagging] != "state=before" {
t.Fatalf("upload lost snapshot tags: %v, %v", info.UserDefined, err)
}
}
// For ordinary PUT this must finish before calling the mover:
// blocking its data Read would already hold the object lock.
opts := ObjectOptions{VersionID: migrationTestVersion(oi), UserDefined: map[string]string{migrationTagRevision: recent}}
var err error
if tags == "" {
_, err = z.DeleteObjectTags(ctx, bucket, oi.Name, opts)
} else {
_, err = z.PutObjectTags(ctx, bucket, oi.Name, tags, opts)
}
if err != nil {
t.Fatal(err)
}
if gr.ObjInfo.UserTags != "state=before" || !maps.Equal(before, gr.ObjInfo.UserDefined) {
t.Fatal("test must retain the old source snapshot")
}
migrationTestStored(t, z, source, oi, tags, recent, true, "payload")
if method == "multipart" {
gate.unblock()
err = <-done
} else {
err = move(ctx, source, bucket, gr)
}
if err != nil {
t.Fatal(err)
}
if !maps.Equal(before, gr.ObjInfo.UserDefined) {
t.Error("reconciliation changed the reader's map")
}
migrationTestStored(t, z, 1-source, oi, tags, recent, true, "payload")
})
}
}
}
}
}
type migrationTestMetadataGateDisk struct {
StorageAPI
bucket, object string
gate *migrationTestGate
}
func (d migrationTestMetadataGateDisk) UpdateMetadata(ctx context.Context, volume, path string, fi FileInfo, opts UpdateMetadataOpts) error {
if volume == d.bucket && path == d.object {
if err := d.gate.wait(ctx); err != nil {
return err
}
}
return d.StorageAPI.UpdateMetadata(ctx, volume, path, fi, opts)
}
func TestPoolsMigrationTagsDuringCleanup(t *testing.T) {
z, bucket := migrationTestPools(t)
const recent = "2026-09-16T10:00:00Z"
for _, kind := range []string{"rebalance", "decommission"} {
for source := range 2 {
for _, tags := range []string{"state=after", ""} {
for _, interrupt := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/source=%d/clear=%t/interrupt=%t", kind, source, tags == "", interrupt), func(t *testing.T) {
move := migrationTestMover(t, z, source, kind)
oi := migrationTestObject(t, z, bucket, t.Name(), source, false, []string{"payload"}, ObjectOptions{
Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "state=before"},
})
if err := move(t.Context(), source, bucket, migrationTestReader(t, z, source, oi)); err != nil {
t.Fatal(err)
}
migrationTestStored(t, z, 1-source, oi, "state=before", "", false, "payload")
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
mutate := func() (ObjectInfo, error) {
opts := ObjectOptions{VersionID: migrationTestVersion(oi), UserDefined: map[string]string{migrationTagRevision: recent}}
if tags == "" {
return z.DeleteObjectTags(ctx, bucket, oi.Name, opts)
}
return z.PutObjectTags(ctx, bucket, oi.Name, tags, opts)
}
set := z.serverPools[source].getHashedSet(oi.Name)
cleanup := func() {
// Use the same source prefix cleanup as both outer movers.
if _, err := set.DeleteObject(ctx, bucket, oi.Name, ObjectOptions{DeletePrefix: true, DeletePrefixObject: true}); err != nil {
t.Fatal(err)
}
}
var updated ObjectInfo
if !interrupt {
var err error
updated, err = mutate()
if err != nil {
t.Fatal(err)
}
cleanup()
} else {
gate := newMigrationTestGate()
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
for i := range disks {
disks[i] = migrationTestMetadataGateDisk{StorageAPI: disks[i], bucket: bucket, object: oi.Name, gate: gate}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
done, finished := make(chan error, 1), make(chan struct{})
go func() { defer close(finished); _, err := mutate(); done <- err }()
defer func() { gate.unblock(); cancel(); <-finished }()
select {
case <-gate.entered:
case err := <-done:
t.Fatalf("metadata update missed the barrier: %v", err)
case <-ctx.Done():
t.Fatal(ctx.Err())
}
cleanup()
gate.unblock()
if err := <-done; err == nil {
t.Fatal("update reported success after its source copy was removed")
}
<-finished
set.getDisks = original
var err error
updated, err = mutate()
if err != nil {
t.Fatalf("metadata retry did not converge: %v", err)
}
}
if _, err := z.serverPools[source].GetObjectInfo(ctx, bucket, oi.Name, ObjectOptions{VersionID: oi.VersionID}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("source cleanup left a copy: %v", err)
}
migrationTestStored(t, z, 1-source, oi, tags, updated.UserDefined[migrationTagRevision], true, "payload")
})
}
}
}
}
}
func TestPoolsMigrationVersionHistory(t *testing.T) {
z, bucket := migrationTestPools(t)
for _, kind := range []string{"rebalance", "decommission"} {
for _, method := range []string{"put", "multipart"} {
for _, versioning := range []string{"unversioned", "null", "history"} {
t.Run(kind+"/"+method+"/"+versioning, func(t *testing.T) {
move := migrationTestMover(t, z, 0, kind)
opts := ObjectOptions{
Versioned: versioning == "history", VersionSuspended: versioning == "null",
MTime: time.Date(2026, 9, 16, 8, 0, 0, 0, time.UTC),
UserDefined: map[string]string{xhttp.AmzObjectTagging: "version=old"},
}
versions := []ObjectInfo{migrationTestObject(t, z, bucket, t.Name(), 0, method == "multipart", []string{"old"}, opts)}
if versioning == "history" {
opts.MTime = opts.MTime.Add(time.Minute)
versions = append(versions, migrationTestObject(t, z, bucket, t.Name(), 0, method == "multipart", []string{"old"}, opts))
}
var newest ObjectInfo
if versioning != "unversioned" {
newest = migrationTestObject(t, z, bucket, t.Name(), 0, false, []string{"latest"}, ObjectOptions{
Versioned: true, MTime: opts.MTime.Add(time.Minute),
UserDefined: map[string]string{xhttp.AmzObjectTagging: "version=new"},
})
}
for _, oi := range versions {
if err := move(t.Context(), 0, bucket, migrationTestReader(t, z, 0, oi)); err != nil {
t.Fatal(err)
}
migrationTestStored(t, z, 1, oi, "version=old", "", false, "old")
}
if versioning != "unversioned" {
migrationTestStored(t, z, 0, newest, "version=new", "", false, "latest")
if _, err := z.serverPools[1].GetObjectInfo(t.Context(), bucket, t.Name(), ObjectOptions{VersionID: newest.VersionID}); !isErrVersionNotFound(err) {
t.Fatalf("moving an addressed version affected the newer version: %v", err)
}
}
})
}
}
}
}
func TestPoolsMigrationMultipartParts(t *testing.T) {
z, bucket := migrationTestPools(t)
parts := []string{strings.Repeat("a", 5<<20), "tail-with-tags"}
body := strings.Join(parts, "")
for _, kind := range []string{"rebalance", "decommission"} {
for source := range 2 {
t.Run(fmt.Sprintf("%s/source=%d", kind, source), func(t *testing.T) {
move := migrationTestMover(t, z, source, kind)
oi := migrationTestObject(t, z, bucket, t.Name(), source, true, parts, ObjectOptions{
Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "multipart=kept"},
})
if err := move(t.Context(), source, bucket, migrationTestReader(t, z, source, oi)); err != nil {
t.Fatal(err)
}
migrationTestStored(t, z, 1-source, oi, "multipart=kept", "", false, body)
got, err := z.serverPools[1-source].GetObjectInfo(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: oi.VersionID})
if err != nil || len(got.Parts) != len(oi.Parts) {
t.Fatalf("parts changed: %+v, %v", got.Parts, err)
}
for i, part := range got.Parts {
old := oi.Parts[i]
if part.Number != old.Number || part.Size != old.Size || part.ActualSize != old.ActualSize || part.ETag != old.ETag || !bytes.Equal(part.Index, old.Index) {
t.Errorf("part %d changed: %+v / %+v", i, part, old)
}
}
start := int64(len(parts[0]) - 3)
gr, err := z.serverPools[1-source].GetObjectNInfo(t.Context(), bucket, oi.Name, &HTTPRangeSpec{Start: start, End: start + 7}, nil, ObjectOptions{VersionID: oi.VersionID})
if err != nil {
t.Fatal(err)
}
data, err := io.ReadAll(gr)
gr.Close()
if err != nil || string(data) != body[start:start+8] {
t.Fatalf("cross-part range changed: %q, %v", data, err)
}
})
}
}
}
func TestPoolsMigrationUnreadablePool(t *testing.T) {
z, bucket := migrationTestPools(t)
for _, kind := range []string{"rebalance", "decommission"} {
for _, method := range []string{"put", "multipart"} {
t.Run(kind+"/"+method, func(t *testing.T) {
move := migrationTestMover(t, z, 0, kind)
oi := migrationTestObject(t, z, bucket, t.Name(), 0, method == "multipart", []string{"payload"}, ObjectOptions{
Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "keep=source"},
})
gr := migrationTestReader(t, z, 0, oi)
set := z.serverPools[0].getHashedSet(oi.Name)
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
for i := range disks {
disks[i] = consistencyReadFaultDisk{StorageAPI: disks[i], bucket: bucket, object: oi.Name}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
err := move(t.Context(), 0, bucket, gr)
var quorum InsufficientReadQuorum
if !errors.As(err, &quorum) {
t.Fatalf("unreadable pool must fail migration with read quorum error: %v", err)
}
set.getDisks = original
migrationTestStored(t, z, 0, oi, "keep=source", "", false, "payload")
if _, err := z.serverPools[1].GetObjectInfo(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: oi.VersionID}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("failed migration committed a target version: %v", err)
}
})
}
}
}
@@ -265,6 +265,7 @@ func TestPoolsMultipartConditionalLatestVersionAndCallbackOnce(t *testing.T) {
if explicit {
current = putConsistencyObject(t, z, bucket, object, 1, "latest-other-version", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
}
opts.VersionID = ""
id, parts := multipartConditionUpload(t, z, bucket, object, 1, opts)
called := 0
opts.CheckPrecondFn = func(oi ObjectInfo) bool { called++; return oi.ETag != current.ETag }
@@ -0,0 +1,716 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"fmt"
"maps"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/kms"
)
// Change allocation capacity only; metadata and data use real fixture disks.
// Restoring the adapters also lets encrypted fixtures move the write target
// between requests without changing the production allocation policy.
type conditionalPutCapacityDisk struct {
StorageAPI
full bool
}
func (d conditionalPutCapacityDisk) DiskInfo(ctx context.Context, opts DiskInfoOptions) (DiskInfo, error) {
info, err := d.StorageAPI.DiskInfo(ctx, opts)
info.Total, info.Used = 1<<40, 0
if d.full {
info.Used = info.Total - (1 << 20)
}
info.Free = info.Total - info.Used
return info, err
}
// Keep getDisks immutable while background IAM and storage readers use it.
// GetDisks takes this same mutex when it copies the backing disk list.
func conditionalPutSwapDisks(pool *erasureSets, object string, wrap func(StorageAPI) StorageAPI) func() {
setIndex := pool.getHashedSet(object).setIndex
pool.erasureDisksMu.Lock()
previous := pool.erasureDisks[setIndex]
disks := append([]StorageAPI(nil), previous...)
for i, disk := range disks {
disks[i] = wrap(disk)
}
pool.erasureDisks[setIndex] = disks
pool.erasureDisksMu.Unlock()
return func() {
pool.erasureDisksMu.Lock()
pool.erasureDisks[setIndex] = previous
pool.erasureDisksMu.Unlock()
}
}
func conditionalPutPool(t *testing.T, z *erasureServerPools, object string, target int) func() {
t.Helper()
var restore []func()
for i, pool := range z.serverPools {
restore = append(restore, conditionalPutSwapDisks(pool, object, func(disk StorageAPI) StorageAPI {
return conditionalPutCapacityDisk{StorageAPI: disk, full: i != target}
}))
}
return func() {
for _, fn := range restore {
fn()
}
}
}
func conditionalPutBucket(t *testing.T, z *erasureServerPools, mode string) (string, http.Handler) {
t.Helper()
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: mode != "unversioned"})
if err != nil {
t.Fatal(err)
}
if mode == "suspended" {
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig,
[]byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
}
return bucket, router
}
func TestPoolsConditionalPutHTTP(t *testing.T) {
for _, mode := range []string{"unversioned", "versioned", "suspended"} {
t.Run(mode, func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, mode)
for target := range 2 {
for _, tc := range []struct {
name string
oldCopy, localCurrent bool
condition string
status int
}{
{"stale-etag-accepted", true, false, "old", http.StatusPreconditionFailed},
{"current-etag-rejected", true, false, "current", http.StatusOK},
{"create-only-overwrites-other-pool", false, false, "none", http.StatusPreconditionFailed},
{"current-etag-missing-in-write-pool", false, false, "current", http.StatusOK},
{"control-current-in-write-pool", true, true, "current", http.StatusOK},
{"control-stale-etag-rejected", true, true, "old", http.StatusPreconditionFailed},
{"none-match-current-etag", true, false, "none-current", http.StatusPreconditionFailed},
{"none-match-old-etag", true, false, "none-old", http.StatusOK},
} {
t.Run(fmt.Sprintf("target=%d/%s", target, tc.name), func(t *testing.T) {
object := fmt.Sprintf("%d-%s", target, tc.name)
currentPool := 1 - target
if tc.localCurrent {
currentPool = target
}
opts := ObjectOptions{Versioned: mode == "versioned", VersionSuspended: mode == "suspended", MTime: UTCNow().Add(-time.Hour)}
oldETag := "absent-old"
if tc.oldCopy {
oldETag = putConsistencyObject(t, z, bucket, object, 1-currentPool, "old", opts).ETag
}
opts.MTime = UTCNow().Add(-time.Minute)
current := putConsistencyObject(t, z, bucket, object, currentPool, "current", opts)
defer conditionalPutPool(t, z, object, target)()
idx, err := z.getWritePoolIdx(t.Context(), bucket, object, 11, false)
if err != nil || idx != target {
t.Fatalf("allocation target=%d: idx=%d err=%v", target, idx, err)
}
headers := map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", current.ETag)}
if tc.condition == "old" {
headers[xhttp.IfMatch] = fmt.Sprintf("%q", oldETag)
}
if tc.condition == "none" {
headers = map[string]string{xhttp.IfNoneMatch: "*"}
}
if tc.condition == "none-current" {
headers = map[string]string{xhttp.IfNoneMatch: fmt.Sprintf("%q", current.ETag)}
}
if tc.condition == "none-old" {
headers = map[string]string{xhttp.IfNoneMatch: fmt.Sprintf("%q", oldETag)}
}
url := getPutObjectURL("", bucket, object)
before := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if before.Code != http.StatusOK || before.Body.String() != "current" || multipartConditionResponseETag(before) != current.ETag {
t.Fatalf("invalid current object: %d %q %v", before.Code, before.Body.String(), before.Header())
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != tc.status {
t.Errorf("PUT status=%d want=%d: %s", put.Code, tc.status, put.Body.String())
}
if put.Code == http.StatusPreconditionFailed {
multipartConditionError(t, put, "PreconditionFailed")
if multipartConditionResponseETag(put) != current.ETag || put.Header().Get(xhttp.LastModified) != before.Header().Get(xhttp.LastModified) {
t.Errorf("412 headers do not describe current object: %v", put.Header())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
wantBody, wantETag := "current", current.ETag
if tc.status == http.StatusOK {
wantBody, wantETag = "replacement", fmt.Sprintf("%x", md5.Sum([]byte("replacement")))
}
t.Logf("PUT %d; GET %d bytes=%q ETag=%s", put.Code, get.Code, get.Body.String(), multipartConditionResponseETag(get))
if get.Code != http.StatusOK || get.Body.String() != wantBody || multipartConditionResponseETag(get) != wantETag {
t.Errorf("GET=%d bytes=%q ETag=%s; want %q %s", get.Code, get.Body.String(), multipartConditionResponseETag(get), wantBody, wantETag)
}
})
}
}
})
}
}
func TestPoolsConditionalPutHTTPAbsence(t *testing.T) {
for _, state := range []string{"missing", "uuid-marker", "null-marker"} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/match=%t", state, match), func(t *testing.T) {
z, _ := consistencyPools(t)
mode := "versioned"
if state == "null-marker" {
mode = "suspended"
}
bucket, router := conditionalPutBucket(t, z, mode)
object := "absent-key"
if state != "missing" {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
vid := mustGetUUID()
if state == "null-marker" {
vid = nullVersionID
}
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: vid, DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
defer conditionalPutPool(t, z, object, 0)()
headers := map[string]string{xhttp.IfNoneMatch: "*"}
want := http.StatusOK
if match {
headers = map[string]string{xhttp.IfMatch: "*"}
want = http.StatusNotFound
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url, "new", headers)
if put.Code != want {
t.Fatalf("PUT %d want %d: %s", put.Code, want, put.Body.String())
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if match {
multipartConditionError(t, put, "NoSuchKey")
if get.Code != http.StatusNotFound {
t.Fatalf("failed PUT exposed data: %d %s", get.Code, get.Body.String())
}
} else if get.Code != http.StatusOK || get.Body.String() != "new" || multipartConditionResponseETag(get) != multipartConditionResponseETag(put) {
t.Fatalf("successful create GET: %d %q %v", get.Code, get.Body.String(), get.Header())
}
})
}
}
}
func TestPoolsConditionalPutUnreadable(t *testing.T) {
for faultPool := range 2 {
for _, present := range []bool{false, true} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("fault-pool=%d/present=%t/match=%t", faultPool, present, match), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
object := "unreadable-key"
if present {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
}
defer conditionalPutPool(t, z, object, 0)()
restoreFault := conditionalPutSwapDisks(z.serverPools[faultPool], object, func(disk StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
})
defer restoreFault()
headers := map[string]string{xhttp.IfNoneMatch: "*"}
if match {
headers = map[string]string{xhttp.IfMatch: "*"}
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != http.StatusServiceUnavailable {
t.Errorf("unverified PUT must fail: %d %s", put.Code, put.Body.String())
}
called := 0
_, err := z.PutObject(t.Context(), bucket, object, mustGetPutObjReader(t, strings.NewReader("replacement"), 11, "", ""), ObjectOptions{HasIfMatch: match, CheckPrecondFn: func(ObjectInfo) bool { called++; return false }})
if !isErrReadQuorum(err) || called != 0 {
t.Errorf("lookup error=%v callback calls=%d", err, called)
}
restoreFault()
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if present {
if get.Code != http.StatusOK || get.Body.String() != "current" || multipartConditionResponseETag(get) != fmt.Sprintf("%x", md5.Sum([]byte("current"))) {
t.Fatalf("failed PUT changed object: %d %q", get.Code, get.Body.String())
}
} else if get.Code != http.StatusNotFound {
t.Fatalf("failed PUT created object: %d %q", get.Code, get.Body.String())
}
})
}
}
}
}
func TestPoolsConditionalPutEncryptedETag(t *testing.T) {
for _, kind := range []string{"SSE-C", "SSE-S3", "SSE-KMS"} {
t.Run(kind, func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
oldKMS, oldTLS := GlobalKMS, globalIsTLS
GlobalKMS, globalIsTLS = kms.NewStub("conditional-put-key"), true
defer func() { GlobalKMS, globalIsTLS = oldKMS, oldTLS }()
headers := map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionAES}
readHeaders := map[string]string{}
if kind == "SSE-C" {
key := bytes.Repeat([]byte{0x42}, 32)
digest := md5.Sum(key)
headers = map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(digest[:]),
}
readHeaders = maps.Clone(headers)
}
if kind == "SSE-KMS" {
headers[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionKMS
headers[xhttp.AmzServerSideEncryptionKmsID] = "conditional-put-key"
}
object := "encrypted-key"
url := getPutObjectURL("", bucket, object)
restore := conditionalPutPool(t, z, object, 0)
old := multipartConditionRequest(t, router, http.MethodPut, url, "old", headers)
restore()
restore = conditionalPutPool(t, z, object, 1)
current := multipartConditionRequest(t, router, http.MethodPut, url, "current", headers)
restore()
if old.Code != http.StatusOK || current.Code != http.StatusOK {
t.Fatalf("encrypted setup: %d %s / %d %s", old.Code, old.Body.String(), current.Code, current.Body.String())
}
defer conditionalPutPool(t, z, object, 0)()
for _, stale := range []bool{true, false} {
h := maps.Clone(headers)
h[xhttp.IfMatch] = fmt.Sprintf("%q", multipartConditionResponseETag(current))
wantStatus, wantBody, wantETag := http.StatusOK, "replacement", ""
if stale {
h[xhttp.IfMatch] = fmt.Sprintf("%q", multipartConditionResponseETag(old))
wantStatus, wantBody, wantETag = http.StatusPreconditionFailed, "current", multipartConditionResponseETag(current)
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", h)
if put.Code != wantStatus {
t.Fatalf("encrypted condition: %d want %d: %s", put.Code, wantStatus, put.Body.String())
}
if !stale {
wantETag = multipartConditionResponseETag(put)
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", readHeaders)
if get.Code != http.StatusOK || get.Body.String() != wantBody || multipartConditionResponseETag(get) != wantETag {
t.Fatalf("encrypted GET: %d %q %v", get.Code, get.Body.String(), get.Header())
}
}
})
}
}
func TestPoolsConditionalPutVersionSelection(t *testing.T) {
for _, kind := range []string{"public-version", "replica", "replica-preserve-etag", "movement", "no-lock", "tie", "draining"} {
t.Run(kind, func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "version-selection"
addressed := putConsistencyObject(t, z, bucket, object, 1, "addressed", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
opts := ObjectOptions{Versioned: true, VersionID: addressed.VersionID, HasIfMatch: true}
want := current
switch kind {
case "replica":
opts.ReplicaLockReconcile, opts.ReplicationRequest = true, true
want = addressed
case "replica-preserve-etag":
opts.PreserveETag = addressed.ETag
opts.ReplicaLockReconcile, opts.ReplicationRequest = true, true
want = addressed
case "movement":
opts.DataMovement, opts.SrcPoolIdx = true, 1
want = addressed
case "tie":
want = putConsistencyObject(t, z, bucket, object, 0, "tie-winner", ObjectOptions{Versioned: true, MTime: current.ModTime})
case "draining":
z.poolMetaMutex.Lock()
z.poolMeta.Pools[1].Decommission = &PoolDecommissionInfo{}
z.poolMetaMutex.Unlock()
}
defer conditionalPutPool(t, z, object, 0)()
ctx := t.Context()
if kind == "no-lock" {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
t.Fatal(err)
}
defer lk.Unlock(lkctx)
ctx, opts.NoLock = lkctx.Context(), true
}
called := 0
opts.UserDefined = make(map[string]string)
opts.CheckPrecondFn = func(oi ObjectInfo) bool {
called++
if oi.ETag != want.ETag || oi.VersionID != want.VersionID {
t.Errorf("comparison ETag/version=%s/%s want %s/%s", oi.ETag, oi.VersionID, want.ETag, want.VersionID)
}
return oi.ETag != want.ETag
}
oi, err := z.PutObject(ctx, bucket, object, mustGetPutObjReader(t, strings.NewReader("replacement"), 11, "", ""), opts)
if err != nil || called != 1 {
t.Fatalf("PUT err=%v callback calls=%d", err, called)
}
if oi.VersionID != addressed.VersionID {
t.Fatalf("destination version changed: %s", oi.VersionID)
}
if opts.PreserveETag != "" && oi.ETag != opts.PreserveETag {
t.Fatalf("PreserveETag changed: %s", oi.ETag)
}
})
}
}
func TestPoolsConditionalPutReplicaDuplicateHTTP(t *testing.T) {
for _, null := range []bool{false, true} {
t.Run(fmt.Sprintf("null=%t", null), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "replica-duplicate"
vid := mustGetUUID()
if null {
vid = nullVersionID
}
addressed := putConsistencyObject(t, z, bucket, object, 1, "addressed", ObjectOptions{Versioned: true, VersionID: vid, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
headers := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA",
xhttp.MinIOSourceETag: addressed.ETag,
xhttp.MinIOSourceMTime: addressed.ModTime.Format(time.RFC3339Nano),
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url+"?versionId="+vid, "addressed", headers)
if put.Code != http.StatusPreconditionFailed {
t.Fatalf("replica duplicate: %d %s", put.Code, put.Body.String())
}
multipartConditionError(t, put, "PreconditionFailed")
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "current" || multipartConditionResponseETag(get) != current.ETag {
t.Fatalf("duplicate changed current object: %d %q", get.Code, get.Body.String())
}
get = multipartConditionRequest(t, router, http.MethodGet, url+"?versionId="+vid, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "addressed" || multipartConditionResponseETag(get) != addressed.ETag {
t.Fatalf("duplicate changed addressed version: %d %q", get.Code, get.Body.String())
}
})
}
}
func TestPoolsConditionalPutConcurrentHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
for _, match := range []bool{false, true} {
for iteration := range 5 {
t.Run(fmt.Sprintf("match=%t/iteration=%d", match, iteration), func(t *testing.T) {
object := fmt.Sprintf("concurrent-%t-%d", match, iteration)
headers := map[string]string{xhttp.IfNoneMatch: "*"}
if match {
oi := putConsistencyObject(t, z, bucket, object, 1, "old", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
headers = map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", oi.ETag)}
}
defer conditionalPutPool(t, z, object, 0)()
start := make(chan struct{})
results := make(chan *httptest.ResponseRecorder, 2)
url := getPutObjectURL("", bucket, object)
for i := range 2 {
body := fmt.Sprintf("writer-%d", i)
req, err := newTestSignedRequestV4(http.MethodPut, url, int64(len(body)), strings.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
go func() {
<-start
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
results <- rec
}()
}
close(start)
success, failed, winnerETag := 0, 0, ""
for range 2 {
result := <-results
switch result.Code {
case http.StatusOK:
success++
winnerETag = multipartConditionResponseETag(result)
case http.StatusPreconditionFailed:
failed++
multipartConditionError(t, result, "PreconditionFailed")
default:
t.Errorf("unexpected PUT %d: %s", result.Code, result.Body.String())
}
}
if success != 1 || failed != 1 {
t.Fatalf("success=%d precondition failures=%d", success, failed)
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || multipartConditionResponseETag(get) != winnerETag || fmt.Sprintf("%x", md5.Sum(get.Body.Bytes())) != winnerETag {
t.Fatalf("winner lost: %d %q %v", get.Code, get.Body.String(), get.Header())
}
})
}
}
}
func TestPoolsConditionalPutSerializesMutation(t *testing.T) {
for _, deletion := range []bool{false, true} {
t.Run(fmt.Sprintf("delete=%t", deletion), func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "conditional-mutation"
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
defer cancel()
gate := &consistencyGateReader{Reader: strings.NewReader("replacement"), entered: make(chan struct{}), resume: make(chan struct{})}
release := func() { gate.release.Do(func() { close(gate.resume) }) }
defer release()
reader := mustGetPutObjReader(t, gate, 11, "", "")
written := make(chan error, 1)
go func() {
_, err := z.PutObject(ctx, bucket, object, reader, ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { return oi.ETag != current.ETag }})
written <- err
}()
select {
case <-gate.entered:
case err := <-written:
t.Fatalf("PUT failed before body read: %v", err)
case <-ctx.Done():
t.Fatal(ctx.Err())
}
mutated := make(chan error, 1)
go func() {
var err error
if deletion {
_, err = z.DeleteObject(ctx, bucket, object, ObjectOptions{})
} else {
_, err = z.PutObjectMetadata(ctx, bucket, object, ObjectOptions{EvalMetadataFn: func(oi *ObjectInfo, _ error) (ReplicateDecision, error) {
oi.UserDefined["x-amz-meta-after-put"] = "present"
return ReplicateDecision{}, nil
}})
}
mutated <- err
}()
select {
case err := <-mutated:
t.Fatalf("mutation escaped PUT lock: %v", err)
case <-time.After(100 * time.Millisecond):
}
release()
if err := <-written; err != nil {
t.Fatal(err)
}
if err := <-mutated; err != nil {
t.Fatal(err)
}
oi, err := z.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if deletion {
if !isErrObjectNotFound(err) {
t.Fatalf("delete lost: %v", err)
}
} else if err != nil || oi.UserDefined["x-amz-meta-after-put"] != "present" || oi.ETag != fmt.Sprintf("%x", md5.Sum([]byte("replacement"))) {
t.Fatalf("metadata/PUT lost: %+v %v", oi, err)
}
})
}
}
func TestSinglePoolConditionalPutHTTP(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure16(ctx)
if err != nil {
cancel()
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { cancel(); z.Shutdown(context.Background()); removeRoots(dirs) })
if !z.SinglePool() {
t.Fatal("fixture is not a single pool")
}
bucket, router := conditionalPutBucket(t, z, "unversioned")
object := "single-pool-condition"
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object)
for _, tc := range []struct {
body, match, none string
status int
}{
{"missing", "*", "", http.StatusNotFound},
{"first", "", "*", http.StatusOK},
{"blocked", "", "*", http.StatusPreconditionFailed},
{"blocked", "stale", "", http.StatusPreconditionFailed},
{"second", fmt.Sprintf("%x", md5.Sum([]byte("first"))), "", http.StatusOK},
} {
rec := multipartConditionRequest(t, router, http.MethodPut, url, tc.body, map[string]string{xhttp.IfMatch: tc.match, xhttp.IfNoneMatch: tc.none})
if rec.Code != tc.status {
t.Fatalf("PUT %d want %d: %s", rec.Code, tc.status, rec.Body.String())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "second" {
t.Fatalf("single-pool GET: %d %q", get.Code, get.Body.String())
}
}
// An internal replica callback without an addressed version is not a public
// condition. Preserve its availability when another pool is unreadable. An
// addressed replica already requires all pools for lock/tag reconciliation.
func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
for _, addressed := range []bool{false, true} {
t.Run(fmt.Sprintf("addressed=%t", addressed), func(t *testing.T) {
z, _ := consistencyPools(t)
mode := "unversioned"
if addressed {
mode = "versioned"
}
bucket, router := conditionalPutBucket(t, z, mode)
object := "replica-availability"
oi := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: addressed, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
restoreFault := conditionalPutSwapDisks(z.serverPools[1], object, func(disk StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
})
defer restoreFault()
headers := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA",
xhttp.MinIOSourceETag: fmt.Sprintf("%x", md5.Sum([]byte("replacement"))),
}
url := getPutObjectURL("", bucket, object)
wantStatus, wantBody := http.StatusOK, "replacement"
if addressed {
url += "?versionId=" + oi.VersionID
wantStatus, wantBody = http.StatusServiceUnavailable, "old"
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != wantStatus {
t.Fatalf("replica PUT: %d want %d: %s", put.Code, wantStatus, put.Body.String())
}
restoreFault()
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != wantBody {
t.Fatalf("replica GET: %d %q", get.Code, get.Body.String())
}
})
}
}
// A public client cannot address a destination version, with or without a
// precondition; only trusted replication may (SN-2026-016).
func TestPoolsConditionalPutDestinationVersionHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "client-version"
old := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object) + "?versionId=" + old.VersionID
for _, tag := range []string{old.ETag, current.ETag} {
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", tag)})
if put.Code != http.StatusForbidden {
t.Fatalf("version-addressed public PUT: %d want 403: %s", put.Code, put.Body.String())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "old" {
t.Fatalf("addressed GET: %d %q", get.Code, get.Body.String())
}
}
func TestPoolsConditionalPutDeleteMarkerTie(t *testing.T) {
for markerPool := range 2 {
t.Run(fmt.Sprintf("marker-pool=%d", markerPool), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "marker-tie"
mtime := UTCNow().Add(-time.Minute)
putConsistencyObject(t, z, bucket, object, 1-markerPool, "live", ObjectOptions{Versioned: true, MTime: mtime})
if _, err := z.serverPools[markerPool].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: mtime}); err != nil {
t.Fatal(err)
}
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object)
before := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
want := http.StatusPreconditionFailed
if markerPool == 0 {
want = http.StatusOK
if before.Code != http.StatusNotFound {
t.Fatalf("GET tie: %d", before.Code)
}
} else if before.Code != http.StatusOK {
t.Fatalf("GET tie: %d", before.Code)
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfNoneMatch: "*"})
if put.Code != want {
t.Fatalf("PUT tie: %d want %d: %s", put.Code, want, put.Body.String())
}
})
}
}
// Overlap fixture changes with the real IAM Walk reader instead of relying on
// its periodic refresh timer to expose an unsynchronized disk-adapter swap.
func TestPoolsConditionalPutFixtureConcurrentIAM(t *testing.T) {
z, _ := consistencyPools(t)
conditionalPutBucket(t, z, "unversioned")
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
started, finished := make(chan struct{}), make(chan error, 1)
iam := globalIAMSys
go func() {
close(started)
for range 20 {
if err := iam.Load(ctx, false); err != nil {
finished <- err
return
}
}
finished <- nil
}()
<-started
for i := range 5000 {
restore := conditionalPutPool(t, z, "fixture-concurrent-iam", i%2)
restore()
}
if err := <-finished; err != nil {
t.Fatal(err)
}
}
+2 -2
View File
@@ -865,7 +865,7 @@ func (z *erasureServerPools) rebalanceObject(ctx context.Context, poolIdx int, b
if oi.isMultipart() {
res, err := z.NewMultipartUpload(ctx, bucket, oi.Name, ObjectOptions{
VersionID: oi.VersionID,
UserDefined: oi.UserDefined,
UserDefined: migrationObjectMetadata(oi),
NoAuditLog: true,
DataMovement: true,
SrcPoolIdx: poolIdx,
@@ -924,7 +924,7 @@ func (z *erasureServerPools) rebalanceObject(ctx context.Context, poolIdx int, b
DataMovement: true,
VersionID: oi.VersionID,
MTime: oi.ModTime,
UserDefined: oi.UserDefined,
UserDefined: migrationObjectMetadata(oi),
PreserveETag: oi.ETag, // Preserve original ETag to ensure same metadata.
IndexCB: func() []byte {
return oi.Parts[0].Index // Preserve part Index to ensure decompression works.
+131 -21
View File
@@ -1149,6 +1149,40 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
}
opts.NoLock = true
// Public write conditions compare the logical current object while the
// pools-layer write lock is held. The destination selected by capacity may
// be empty or stale, and draining pools can still hold the current object.
// Replica callbacks retain their existing addressed-version semantics and
// metadata reconciliation at the set layer.
if opts.CheckPrecondFn != nil && !opts.ReplicationRequest &&
!opts.ReplicaLockReconcile && !opts.DataMovement {
copies, lerr := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{
VersionID: "", // Compare the current object, not the write's version.
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoAuditLog: true,
})
var latest ObjectInfo
if lerr == nil {
latest = copies[0].ObjInfo
if latest.DeleteMarker {
lerr = toObjectErr(errFileNotFound, bucket, object)
}
}
// An unreadable pool may hold the newest object; it is not absence.
if lerr != nil && !isErrObjectNotFound(lerr) && !isErrVersionNotFound(lerr) {
return ObjectInfo{}, lerr
}
if lerr == nil && opts.CheckPrecondFn(latest) {
return ObjectInfo{}, PreConditionFailed{}
}
if lerr != nil && opts.HasIfMatch {
return ObjectInfo{}, lerr
}
// Do not repeat an accepted condition against the destination's copy.
opts.CheckPrecondFn = nil
}
idx, err := z.getWritePoolIdx(ctx, bucket, object, data.Size(), true)
if err != nil {
return ObjectInfo{}, err
@@ -1165,6 +1199,10 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
if opts.ReplicaLockReconcile || opts.DataMovement {
opts.ReplicaLockReconcile = true
}
// Public overwrites need all pools for Object Lock. Unaddressed replica
// callbacks retain their set-local reads unless reconciliation is needed.
if opts.ReplicaLockReconcile || !opts.ReplicationRequest {
opts.replicaObjectInfo = z.replicaObjectInfo
}
oi, err := z.serverPools[idx].PutObject(ctx, bucket, object, data, opts)
@@ -1205,9 +1243,12 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
return ObjectInfo{}, z.deletePrefix(ctx, bucket, object)
}
// Reconcile ordinary addressed-version deletes independently of pool movement.
// Resolve a physical purge by its addressed version even on a replica
// receiver: latest-key routing can select a different pool or miss copies.
// Marker creation and specialized movement/scanner operations retain their
// existing routing.
reconcileVersion := opts.VersionID != "" && !opts.DataMovement &&
!opts.ReplicationRequest && !opts.Expiration.Expire && !opts.InclFreeVersions
(!opts.ReplicationRequest || opts.isVersionPurge()) && !opts.Expiration.Expire && !opts.InclFreeVersions
if !z.SinglePool() && (opts.CheckPrecondFn != nil || reconcileVersion) {
return z.deleteObjectReconciled(ctx, bucket, object, opts)
}
@@ -1220,6 +1261,13 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
if _, ok := err.(InsufficientReadQuorum); ok {
return objInfo, InsufficientWriteQuorum{}
}
// Lookup can return before the set's purge confirmation. Check here,
// before any callback can change the request into a metadata update.
if opts.isVersionPurge() && (isErrObjectNotFound(err) || isErrVersionNotFound(err)) {
if quorumErr := z.checkPurgeAbsent(ctx, bucket, object, opts.VersionID); quorumErr != nil {
return objInfo, quorumErr
}
}
// A conditional (If-Match) delete addressing a specific version treats an
// absent key as an absent version. getPoolInfoExistingWithOpts strips
// VersionID, so a missing key surfaces ObjectNotFound here even for a
@@ -1250,6 +1298,11 @@ func (z *erasureServerPools) DeleteObject(ctx context.Context, bucket string, ob
if verr != nil && (!isErrMethodNotAllowed(verr) || !vi.DeleteMarker) {
// Genuine read failure for the addressed version: a missing
// version -> VersionNotFound (NoSuchVersion), read-quorum loss, etc.
if opts.isVersionPurge() && (isErrObjectNotFound(verr) || isErrVersionNotFound(verr)) {
if quorumErr := z.checkPurgeAbsent(ctx, bucket, object, opts.VersionID); quorumErr != nil {
return objInfo, quorumErr
}
}
return objInfo, verr
}
// verr is nil for a live version, or MethodNotAllowed with a populated
@@ -1461,9 +1514,9 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
return objInfo, err
}
if !z.SinglePool() && dstOpts.ReplicaLockReconcile {
if !z.SinglePool() && (dstOpts.ReplicaLockReconcile || !dstOpts.ReplicationRequest) {
dstOpts.replicaObjectInfo = z.replicaObjectInfo
if !dstOpts.DataMovement {
if dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement {
defer func() {
if err == nil {
err = z.retireReplicaCopies(ctx, dstBucket, dstObject, poolIdx, objInfo)
@@ -1500,13 +1553,16 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
ServerSideEncryption: dstOpts.ServerSideEncryption,
UserDefined: srcInfo.UserDefined,
Versioned: dstOpts.Versioned,
VersionSuspended: dstOpts.VersionSuspended,
VersionID: dstOpts.VersionID,
MTime: dstOpts.MTime,
NoLock: true,
EncryptFn: dstOpts.EncryptFn,
WantChecksum: dstOpts.WantChecksum,
WantServerSideChecksumType: dstOpts.WantServerSideChecksumType,
ReplicationRequest: dstOpts.ReplicationRequest,
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
DataMovement: dstOpts.DataMovement,
replicaObjectInfo: dstOpts.replicaObjectInfo,
}
@@ -1857,7 +1913,41 @@ func (z *erasureServerPools) ListMultipartUploads(ctx context.Context, bucket, p
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
return ListMultipartsInfo{}, toObjectErr(err, bucket)
}
if globalAPIConfig.getMultipartListingLegacy() {
return z.listMultipartUploadsLegacy(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
var uploads []MultipartInfo
var keyless bool
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
poolUploads, poolKeyless, err := pool.scanMultipartUploads(scan, bucket, idx)
if err != nil {
return ListMultipartsInfo{}, err
}
uploads = append(uploads, poolUploads...)
keyless = keyless || poolKeyless
}
// The old format cannot be enumerated authoritatively. Migration mode is
// explicit: another bucket must never silently change this API's semantics.
if keyless {
return ListMultipartsInfo{}, errMultipartListingLegacy
}
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
func (z *erasureServerPools) listMultipartUploadsLegacy(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
poolResult := ListMultipartsInfo{}
poolResult.MaxUploads = maxUploads
poolResult.KeyMarker = keyMarker
@@ -1883,15 +1973,14 @@ func (z *erasureServerPools) ListMultipartUploads(ctx context.Context, bucket, p
}
if z.SinglePool() {
return z.serverPools[0].ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
return z.serverPools[0].getHashedSet(prefix).listMultipartUploadsExact(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
}
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
result, err := pool.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker,
delimiter, maxUploads)
result, err := pool.getHashedSet(prefix).listMultipartUploadsExact(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
return result, err
}
@@ -1927,7 +2016,7 @@ func (z *erasureServerPools) NewMultipartUpload(ctx context.Context, bucket, obj
continue
}
result, err := pool.ListMultipartUploads(ctx, bucket, object, "", "", "", maxUploadsList)
result, err := pool.listMultipartUploadsExact(ctx, bucket, object)
if err != nil {
return nil, err
}
@@ -2089,13 +2178,18 @@ func (z *erasureServerPools) AbortMultipartUpload(ctx context.Context, bucket, o
if err := checkAbortMultipartArgs(ctx, bucket, object, uploadID); err != nil {
return err
}
if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
return toObjectErr(err, bucket)
}
defer func() {
// Unlock cancels the derived lock context before this notification runs.
// Keep the request context so successful cancellation reaches peer caches.
defer func(ctx context.Context) {
if err == nil {
z.mpCache.Delete(uploadID)
globalNotificationSys.DeleteUploadID(ctx, uploadID)
}
}()
}(ctx)
lk := z.NewNSLock(bucket, pathJoin(object, uploadID))
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
@@ -2105,23 +2199,28 @@ func (z *erasureServerPools) AbortMultipartUpload(ctx context.Context, bucket, o
ctx = lkctx.Context()
defer lk.Unlock(lkctx)
if z.SinglePool() {
return z.serverPools[0].AbortMultipartUpload(ctx, bucket, object, uploadID, opts)
}
legacy := globalAPIConfig.getMultipartListingLegacy()
found := false
var firstErr error
for idx, pool := range z.serverPools {
if z.IsSuspended(idx) {
continue
}
err := pool.AbortMultipartUpload(ctx, bucket, object, uploadID, opts)
if err == nil {
return nil
poolFound, err := pool.getHashedSet(object).abortMultipartUpload(ctx, bucket, object, uploadID, opts, legacy)
if legacy && (poolFound || err != nil) {
// Match the released first-matching-pool behavior.
return err
}
if _, ok := err.(InvalidUploadID); ok {
// upload id not found move to next pool
continue
found = found || poolFound
if err != nil && firstErr == nil {
firstErr = err
}
return err
}
if firstErr != nil {
return firstErr
}
if found {
return nil
}
return InvalidUploadID{
Bucket: bucket,
@@ -2213,6 +2312,8 @@ func (z *erasureServerPools) CompleteMultipartUpload(ctx context.Context, bucket
if opts.ReplicaLockReconcile || opts.DataMovement {
opts.ReplicaLockReconcile = true
}
if opts.ReplicaLockReconcile || !opts.ReplicationRequest {
opts.replicaObjectInfo = z.replicaObjectInfo
}
@@ -3051,6 +3152,15 @@ func (z *erasureServerPools) PutObjectTags(ctx context.Context, bucket, object s
if err != nil {
return ObjectInfo{}, err
}
// Ordinary reads and replication can return any owning pool. Persist one
// revision beyond all copies, so the returned value and every pool agree.
if stamp := opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]; stamp != "" {
for _, copy := range copies {
stamp = monotonicTaggingTimestamp(stamp, copy.ObjInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
}
opts.UserDefined = cloneMSS(opts.UserDefined)
opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
}
opts.NoLock = true
opts.VersionID = copies[0].ObjInfo.VersionID
if opts.VersionID == "" {
+40 -4
View File
@@ -869,21 +869,57 @@ func (s *erasureSets) CopyObject(ctx context.Context, srcBucket, srcObject, dstB
ServerSideEncryption: dstOpts.ServerSideEncryption,
UserDefined: srcInfo.UserDefined,
Versioned: dstOpts.Versioned,
VersionSuspended: dstOpts.VersionSuspended,
VersionID: dstOpts.VersionID,
MTime: dstOpts.MTime,
NoLock: dstOpts.NoLock,
EncryptFn: dstOpts.EncryptFn,
WantChecksum: dstOpts.WantChecksum,
WantServerSideChecksumType: dstOpts.WantServerSideChecksumType,
ReplicationRequest: dstOpts.ReplicationRequest,
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
DataMovement: dstOpts.DataMovement,
replicaObjectInfo: dstOpts.replicaObjectInfo,
}
return dstSet.putObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, putOpts)
}
func (s *erasureSets) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
// In list multipart uploads we are going to treat input prefix as the object,
// this means that we are not supporting directory navigation.
set := s.getHashedSet(prefix)
return set.ListMultipartUploads(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err := checkListMultipartArgs(ctx, bucket, prefix, keyMarker, uploadIDMarker, delimiter); err != nil {
return ListMultipartsInfo{}, err
}
scan, err := startMultipartScan(ctx, false)
if err != nil {
return ListMultipartsInfo{}, err
}
defer scan.close()
uploads, legacy, err := s.scanMultipartUploads(scan, bucket, 0)
if err != nil {
return ListMultipartsInfo{}, err
}
if legacy {
return ListMultipartsInfo{}, errMultipartListingLegacy
}
return paginateMultipartUploads(uploads, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads), nil
}
func (s *erasureSets) scanMultipartUploads(scan *multipartScan, bucket string, poolIdx int) ([]MultipartInfo, bool, error) {
var uploads []MultipartInfo
var keyless bool
for i, set := range s.sets {
setUploads, setKeyless, err := set.scanMultipartUploads(scan, bucket, poolIdx, i)
if err != nil {
return nil, false, err
}
uploads = append(uploads, setUploads...)
keyless = keyless || setKeyless
}
return uploads, keyless, nil
}
func (s *erasureSets) listMultipartUploadsExact(ctx context.Context, bucket, object string) (ListMultipartsInfo, error) {
return s.getHashedSet(object).listMultipartUploadsExact(ctx, bucket, object, "", "", "", maxUploadsList)
}
// Initiate a new multipart upload on a hashedSet based on object name.
+87
View File
@@ -0,0 +1,87 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"github.com/google/uuid"
"github.com/minio/minio/internal/bucket/replication"
)
// isVersionPurge distinguishes physical removal from marker creation and
// replication-state updates. Call it again after metadata callbacks, which
// can change an ordinary deletion into a pending purge update.
func (o ObjectOptions) isVersionPurge() bool {
if o.VersionID == "" || o.VersionID == nullVersionID || o.DeleteMarker ||
o.DeletePrefix || o.DataMovement || o.InclFreeVersions || o.Expiration.Expire ||
o.Transition != (TransitionOptions{}) {
return false
}
id, err := uuid.Parse(o.VersionID)
if err != nil || id == uuid.Nil {
return false
}
purge := o.VersionPurgeStatus()
if purge == replication.VersionPurgeComplete {
return true
}
if !purge.Empty() || o.DeleteReplication.VersionPurgeStatusInternal != "" {
return false
}
status := o.DeleteMarkerReplicationStatus()
return (status.Empty() && o.DeleteReplication.ReplicationStatusInternal == "") ||
(status == replication.Replica && o.ReplicationRequest)
}
// confirmVersionAbsent is a read-only proof for an already-missing purge.
// Read quorum (or a synthesized NotFound) cannot acknowledge a write. Do not
// delete here: unreadable minority copies may carry retention we cannot check.
// The caller holds the object lock. This function never heals or enqueues work.
func (er erasureObjects) confirmVersionAbsent(ctx context.Context, bucket, object, versionID string) (allAbsent bool, err error) {
disks := er.getDisks()
_, errs := readAllFileInfo(ctx, disks, "", bucket, object, versionID, false, false)
absent := 0
for _, err := range errs {
if err == errFileNotFound || err == errFileVersionNotFound {
absent++
}
}
if absent < len(disks)/2+1 {
return false, InsufficientWriteQuorum{}
}
return absent == len(disks), nil
}
// checkPurgeAbsent schedules recovery explicitly, outside the read-only proof.
func (er erasureObjects) checkPurgeAbsent(ctx context.Context, bucket, object, versionID string) error {
allAbsent, err := er.confirmVersionAbsent(ctx, bucket, object, versionID)
if !allAbsent {
er.addPartial(bucket, object, versionID)
}
return err
}
func (z *erasureServerPools) checkPurgeAbsent(ctx context.Context, bucket, object, versionID string) error {
for _, pool := range z.serverPools {
if err := pool.getHashedSet(object).checkPurgeAbsent(ctx, bucket, object, versionID); err != nil {
return err
}
}
return nil
}
+8
View File
@@ -50,6 +50,7 @@ type apiConfig struct {
transitionWorkers int
staleUploadsExpiry time.Duration
multipartListingStrict bool
staleUploadsCleanupInterval time.Duration
deleteCleanupInterval time.Duration
enableODirect bool
@@ -181,6 +182,7 @@ func (t *apiConfig) init(cfg api.Config, setDriveCounts []int, legacy bool) {
t.transitionWorkers = cfg.TransitionWorkers
t.staleUploadsExpiry = cfg.StaleUploadsExpiry
t.multipartListingStrict = cfg.MultipartListing == "strict"
t.deleteCleanupInterval = cfg.DeleteCleanupInterval
t.enableODirect = cfg.EnableODirect
t.gzipObjects = cfg.GzipObjects
@@ -206,6 +208,12 @@ func (t *apiConfig) odirectEnabled() bool {
return t.enableODirect
}
func (t *apiConfig) getMultipartListingLegacy() bool {
t.mu.RLock()
defer t.mu.RUnlock()
return !t.multipartListingStrict
}
func (t *apiConfig) shouldGzipObjects() bool {
t.mu.RLock()
defer t.mu.RUnlock()
+33 -23
View File
@@ -246,16 +246,6 @@ func extractMetadata(ctx context.Context, mimesHeader ...textproto.MIMEHeader) (
// extractMetadata extracts metadata from map values.
func extractMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
return extractMetadataFromMimeWithReplication(ctx, v, m, false)
}
// extractReplicationMetadataFromMime restores replication-only metadata after the
// caller has validated that the request is a trusted replication write.
func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
return extractMetadataFromMimeWithReplication(ctx, v, m, true)
}
func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIMEHeader, m map[string]string, allowReplication bool) error {
if v == nil {
bugLogIf(ctx, errInvalidArgument)
return errInvalidArgument
@@ -267,18 +257,14 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
nv[http.CanonicalHeaderKey(k)] = kv
}
// Save all supported headers.
// Save ordinary object metadata. Replication-only headers are restored only
// after the request has been validated as a trusted replication write.
for _, supportedHeader := range supportedHeaders {
value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]
if ok {
if v, ok := replicationToInternalHeaders[supportedHeader]; ok {
if !allowReplication {
continue
}
m[v] = strings.Join(value, ",")
} else {
m[supportedHeader] = strings.Join(value, ",")
}
if _, ok := replicationToInternalHeaders[supportedHeader]; ok {
continue
}
if value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]; ok {
m[supportedHeader] = strings.Join(value, ",")
}
}
@@ -287,8 +273,7 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
if !stringsHasPrefixFold(key, prefix) {
continue
}
value, ok := nv[http.CanonicalHeaderKey(key)]
if ok {
if value, ok := nv[http.CanonicalHeaderKey(key)]; ok {
m[key] = strings.Join(value, ",")
break
}
@@ -297,6 +282,31 @@ func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIM
return nil
}
// extractReplicationMetadataFromMime restores replication-only metadata after the
// caller has validated that the request is a trusted replication write.
func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
if v == nil {
bugLogIf(ctx, errInvalidArgument)
return errInvalidArgument
}
nv := make(textproto.MIMEHeader, len(v))
for k, kv := range v {
// Canonicalize all headers, to remove any duplicates.
nv[http.CanonicalHeaderKey(k)] = kv
}
// Ordinary object metadata belongs to the caller. Re-extracting it would
// undo normalization (such as removing aws-chunked) or copy an outer
// Snowball archive's metadata onto its individual entries.
for header, internalHeader := range replicationToInternalHeaders {
if value, ok := nv[http.CanonicalHeaderKey(header)]; ok {
m[internalHeader] = strings.Join(value, ",")
}
}
return nil
}
// Returns access credentials in the request Authorization header.
func getReqAccessCred(r *http.Request, region string) (cred auth.Credentials) {
cred, _, _ = getReqAccessKeyV4(r, region, serviceS3)
+12 -1
View File
@@ -254,6 +254,9 @@ func TestExtractMetadataFromRequestKeepsQueryCompatibility(t *testing.T) {
func TestExtractReplicationMetadataHeaders(t *testing.T) {
header := http.Header{
"Content-Type": []string{"application/wasm"},
"Content-Encoding": []string{"aws-chunked"},
"X-Amz-Meta-Source": []string{"client"},
"X-Minio-Replication-Server-Side-Encryption-Sealed-Key": []string{"sealed-key"},
"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
"X-Minio-Replication-Server-Side-Encryption-Iv": []string{"iv"},
@@ -262,12 +265,17 @@ func TestExtractReplicationMetadataHeaders(t *testing.T) {
ReplicationSsecChecksumHeader: []string{"checksum"},
}
metadata := make(map[string]string)
metadata := map[string]string{
"content-type": "application/wasm",
"x-amz-meta-source": "client",
}
if err := extractReplicationMetadataFromMime(t.Context(), textproto.MIMEHeader(header), metadata); err != nil {
t.Fatalf("failed to extract replication metadata: %v", err)
}
expected := map[string]string{
"content-type": "application/wasm",
"x-amz-meta-source": "client",
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key": "sealed-key",
"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "DAREv2-HMAC-SHA256",
"X-Minio-Internal-Server-Side-Encryption-Iv": "iv",
@@ -279,6 +287,9 @@ func TestExtractReplicationMetadataHeaders(t *testing.T) {
if !reflect.DeepEqual(metadata, expected) {
t.Fatalf("unexpected replication metadata: expected %#v, got %#v", expected, metadata)
}
if _, ok := metadata["content-encoding"]; ok {
t.Fatalf("replication metadata restored transport content-encoding: %#v", metadata)
}
}
func TestGetCopyObjectMetadataFromHeaderReplication(t *testing.T) {
+292
View File
@@ -0,0 +1,292 @@
// Copyright (c) 2026 mr javad seydi
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"errors"
"fmt"
"slices"
"testing"
"time"
)
func multipartUploadKeys(uploads []MultipartInfo) []string {
keys := make([]string, len(uploads))
for i := range uploads {
keys[i] = uploads[i].Object
}
return keys
}
func requireMultipartUploadKeys(t *testing.T, got ListMultipartsInfo, want ...string) {
t.Helper()
if keys := multipartUploadKeys(got.Uploads); !slices.Equal(keys, want) {
t.Fatalf("uploads = %v, want %v", keys, want)
}
}
func TestListMultipartUploadsS3Compatibility(t *testing.T) {
obj, dirs, err := prepareErasureSets32(t.Context())
if err != nil {
t.Fatal(err)
}
z := obj.(*erasureServerPools)
setMultipartListingTestMode(t, false)
t.Cleanup(func() {
z.Shutdown(t.Context())
removeRoots(dirs)
})
const bucket = "multipart-list-compat"
if err = z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
objects := []string{"t/a_b/p1", "t/a_b/p2", "t/c_d/p1", "u/x"}
sets := z.serverPools[0]
firstSet := sets.getHashedSetIndex(objects[0])
if !slices.ContainsFunc(objects[1:], func(object string) bool {
return sets.getHashedSetIndex(object) != firstSet
}) {
for n := 0; ; n++ {
object := fmt.Sprintf("v/cross-set-%d", n)
if sets.getHashedSetIndex(object) != firstSet {
objects = append(objects, object)
break
}
}
}
uploadIDs := make(map[string]string, len(objects))
for _, object := range objects {
mp, err := z.NewMultipartUpload(t.Context(), bucket, object, ObjectOptions{})
if err != nil {
t.Fatalf("NewMultipartUpload(%q): %v", object, err)
}
uploadIDs[object] = mp.UploadID
}
// Durable multipart metadata, rather than this node-local cache, must be
// authoritative after a restart or when another node handles the request.
z.mpCache.Range(func(uploadID string, _ MultipartInfo) bool {
z.mpCache.Delete(uploadID)
return true
})
all, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, all, objects...)
if all.IsTruncated || all.NextKeyMarker != "" || all.NextUploadIDMarker != "" {
t.Fatalf("complete listing has truncation state: %+v", all)
}
first, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 1)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, first, objects[0])
if !first.IsTruncated || first.NextKeyMarker != objects[0] || first.NextUploadIDMarker != uploadIDs[objects[0]] {
t.Fatalf("first page markers = (%q, %q, %t), want (%q, %q, true)",
first.NextKeyMarker, first.NextUploadIDMarker, first.IsTruncated,
objects[0], uploadIDs[objects[0]])
}
rest, err := z.ListMultipartUploads(t.Context(), bucket, "", first.NextKeyMarker, first.NextUploadIDMarker, "", 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, rest, objects[1:]...)
afterKey, err := z.ListMultipartUploads(t.Context(), bucket, "", objects[1], "", "", 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, afterKey, objects[2:]...)
prefixed, err := z.ListMultipartUploads(t.Context(), bucket, "t/", "", "", "", 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, prefixed, objects[:3]...)
nested, err := z.ListMultipartUploads(t.Context(), bucket, "t/a_b/", "", "", "", 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, nested, objects[:2]...)
grouped, err := z.ListMultipartUploads(t.Context(), bucket, "t/", "", "", SlashSeparator, 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, grouped)
if want := []string{"t/a_b/", "t/c_d/"}; !slices.Equal(grouped.CommonPrefixes, want) {
t.Fatalf("common prefixes = %v, want %v", grouped.CommonPrefixes, want)
}
groupPage, err := z.ListMultipartUploads(t.Context(), bucket, "t/", "", "", SlashSeparator, 1)
if err != nil {
t.Fatal(err)
}
if want := []string{"t/a_b/"}; !slices.Equal(groupPage.CommonPrefixes, want) {
t.Fatalf("first common-prefix page = %v, want %v", groupPage.CommonPrefixes, want)
}
if !groupPage.IsTruncated || groupPage.NextKeyMarker != "t/a_b/" || groupPage.NextUploadIDMarker != "" {
t.Fatalf("common-prefix page markers = (%q, %q, %t)",
groupPage.NextKeyMarker, groupPage.NextUploadIDMarker, groupPage.IsTruncated)
}
groupRest, err := z.ListMultipartUploads(t.Context(), bucket, "t/", groupPage.NextKeyMarker, "", SlashSeparator, 1)
if err != nil {
t.Fatal(err)
}
if want := []string{"t/c_d/"}; !slices.Equal(groupRest.CommonPrefixes, want) {
t.Fatalf("second common-prefix page = %v, want %v", groupRest.CommonPrefixes, want)
}
if groupRest.IsTruncated {
t.Fatalf("last common-prefix page is truncated: %+v", groupRest)
}
part, err := z.PutObjectPart(t.Context(), bucket, objects[0], uploadIDs[objects[0]], 1,
mustGetPutObjReader(t, bytes.NewBufferString("part"), 4, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
completed, err := z.CompleteMultipartUpload(t.Context(), bucket, objects[0], uploadIDs[objects[0]],
[]CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
for _, key := range []string{multipartMetaBucket, multipartMetaObject} {
if _, ok := completed.UserDefined[key]; ok {
t.Errorf("completed object retained upload-only metadata %q", key)
}
}
// Simulate an upload written by a pre-upgrade server. Its key cannot be
// recovered by scanning the hashed namespace. Strict listing must fail;
// the old path is available only through an explicit migration setting.
legacyObject := objects[1]
er := sets.getHashedSet(legacyObject)
fi, metadata, err := er.checkUploadIDExists(t.Context(), bucket, legacyObject, uploadIDs[legacyObject], true)
if err != nil {
t.Fatal(err)
}
for i := range metadata {
delete(metadata[i].Metadata, multipartMetaBucket)
delete(metadata[i].Metadata, multipartMetaObject)
}
if _, err = writeAllMetadata(t.Context(), er.getDisks(), bucket, minioMetaMultipartBucket,
er.getUploadIDDir(bucket, legacyObject, uploadIDs[legacyObject]), metadata, fi.WriteQuorum(er.defaultWQuorum())); err != nil {
t.Fatal(err)
}
_, err = z.ListMultipartUploads(t.Context(), bucket, legacyObject, "", "", "", 100)
if !errors.Is(err, errMultipartListingLegacy) {
t.Fatalf("legacy strict listing: %v", err)
}
setMultipartListingTestMode(t, true)
legacy, err := z.ListMultipartUploads(t.Context(), bucket, legacyObject, "", "", "", 100)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, legacy, legacyObject)
}
func TestPaginateMultipartUploads(t *testing.T) {
base := time.Unix(100, 0)
id1, id2, id3 := multipartListingTestID(base, 1), multipartListingTestID(base.Add(time.Second), 2), multipartListingTestID(base, 3)
uploads := []MultipartInfo{
{Bucket: "bucket", Object: "b", UploadID: id3, Initiated: base},
{Bucket: "bucket", Object: "a", UploadID: id2, Initiated: base.Add(time.Second)},
{Bucket: "bucket", Object: "a", UploadID: id1, Initiated: base},
{Bucket: "bucket", Object: "a", UploadID: id1, Initiated: base}, // duplicate discovery
}
first := paginateMultipartUploads(uploads, "", "", "", "", 1)
requireMultipartUploadKeys(t, first, "a")
if !first.IsTruncated || first.NextKeyMarker != "a" || first.NextUploadIDMarker != id1 {
t.Fatalf("first page = %+v", first)
}
second := paginateMultipartUploads(uploads, "", first.NextKeyMarker, first.NextUploadIDMarker, "", 1)
if len(second.Uploads) != 1 || second.Uploads[0].Object != "a" || second.Uploads[0].UploadID != id2 {
t.Fatalf("second page uploads = %+v", second.Uploads)
}
if !second.IsTruncated || second.NextKeyMarker != "a" || second.NextUploadIDMarker != id2 {
t.Fatalf("second page = %+v", second)
}
last := paginateMultipartUploads(uploads, "", second.NextKeyMarker, second.NextUploadIDMarker, "", 1)
requireMultipartUploadKeys(t, last, "b")
if last.IsTruncated || last.NextKeyMarker != "" || last.NextUploadIDMarker != "" {
t.Fatalf("last page = %+v", last)
}
missingUploadMarker := paginateMultipartUploads(uploads, "", "a", multipartListingTestID(base.Add(2*time.Second), 4), "", 10)
requireMultipartUploadKeys(t, missingUploadMarker, "b")
if err := checkListMultipartArgs(t.Context(), "bucket", "", "", "not-base64=", ""); err != nil {
t.Fatalf("upload-id-marker without key-marker must be ignored: %v", err)
}
overLimit := make([]MultipartInfo, maxUploadsList+1)
for i := range overLimit {
overLimit[i] = MultipartInfo{Bucket: "bucket", Object: fmt.Sprintf("%04d", i), UploadID: fmt.Sprint(i)}
}
capped := paginateMultipartUploads(overLimit, "", "", "", "", maxUploadsList+1)
if capped.MaxUploads != maxUploadsList || len(capped.Uploads) != maxUploadsList || !capped.IsTruncated {
t.Fatalf("over-limit page = MaxUploads %d, uploads %d, truncated %t",
capped.MaxUploads, len(capped.Uploads), capped.IsTruncated)
}
}
func TestListMultipartUploadsGlobalPageAcrossPools(t *testing.T) {
z, bucket := consistencyPools(t)
setMultipartListingTestMode(t, false)
objects := []string{"a/one", "b/two", "c/three", "d/four"}
for i, object := range objects {
if _, err := z.serverPools[i%len(z.serverPools)].NewMultipartUpload(t.Context(), bucket, object, ObjectOptions{}); err != nil {
t.Fatalf("NewMultipartUpload(%q): %v", object, err)
}
}
z.mpCache.Range(func(uploadID string, _ MultipartInfo) bool {
z.mpCache.Delete(uploadID)
return true
})
page, err := z.ListMultipartUploads(t.Context(), bucket, "", "", "", "", 2)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, page, objects[:2]...)
if !page.IsTruncated || page.NextKeyMarker != objects[1] {
t.Fatalf("first global page = %+v", page)
}
rest, err := z.ListMultipartUploads(t.Context(), bucket, "", page.NextKeyMarker, page.NextUploadIDMarker, "", 2)
if err != nil {
t.Fatal(err)
}
requireMultipartUploadKeys(t, rest, objects[2:]...)
if rest.IsTruncated {
t.Fatalf("last global page is truncated: %+v", rest)
}
}
+433
View File
@@ -0,0 +1,433 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"sync"
"testing"
"time"
madmin "github.com/minio/madmin-go/v3"
)
// Hold nine completed disk walks while a PUT overwrites one name. The other
// seven walks start after that PUT completes. Both generations come from real
// disk walkers and valid PUT metadata; only the reader schedule is controlled.
type nullQuorumWalkSchedule struct {
bucket string
oldReady chan struct{}
resume chan struct{}
mu sync.Mutex
snapshots [16][]byte
}
type nullQuorumWalkDisk struct {
StorageAPI
schedule *nullQuorumWalkSchedule
index int
}
func (d *nullQuorumWalkDisk) WalkDir(ctx context.Context, opts WalkDirOptions, out io.Writer) error {
if opts.Bucket != d.schedule.bucket {
return d.StorageAPI.WalkDir(ctx, opts, out)
}
var stream bytes.Buffer
if d.index < 9 {
if err := d.StorageAPI.WalkDir(ctx, opts, &stream); err != nil {
return err
}
select {
case d.schedule.oldReady <- struct{}{}:
case <-ctx.Done():
return ctx.Err()
}
}
select {
case <-d.schedule.resume:
case <-ctx.Done():
return ctx.Err()
}
if d.index >= 9 {
if err := d.StorageAPI.WalkDir(ctx, opts, &stream); err != nil {
return err
}
}
d.schedule.mu.Lock()
d.schedule.snapshots[d.index] = bytes.Clone(stream.Bytes())
d.schedule.mu.Unlock()
_, err := out.Write(stream.Bytes())
return err
}
func nullQuorumBackend(t *testing.T) (*erasureServerPools, string, http.Handler) {
t.Helper()
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure16(ctx)
if err != nil {
cancel()
t.Fatal(err)
}
z := obj.(*erasureServerPools)
previous := newObjectLayerFn()
setObjectLayer(z)
t.Cleanup(func() {
cancel()
z.Shutdown(context.Background())
removeRoots(dirs)
setObjectLayer(previous)
})
bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
return z, bucket, router
}
func nullQuorumRequest(t *testing.T, router http.Handler, method, target string) *httptest.ResponseRecorder {
t.Helper()
req, err := newTestSignedRequestV4(method, target, 0, nil, globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req.WithContext(t.Context()))
return rec
}
func TestListObjectsSingleNullQuorumHTTP(t *testing.T) {
z, bucket, router := nullQuorumBackend(t)
oldBody := bytes.Repeat([]byte{'a'}, 8192)
newBody := bytes.Repeat([]byte{'b'}, 8192)
oldTime := time.Now().UTC().Add(-time.Hour)
newTime := oldTime.Add(time.Minute)
var want []string
for i := range 16 {
name := fmt.Sprintf("fixed/%02d", i)
want = append(want, name)
_, err := z.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, bytes.NewReader(oldBody), int64(len(oldBody)), "", ""), ObjectOptions{MTime: oldTime})
if err != nil {
t.Fatal(err)
}
}
const overwritten = "fixed/10"
schedule := &nullQuorumWalkSchedule{bucket: bucket, oldReady: make(chan struct{}, 16), resume: make(chan struct{})}
set := z.serverPools[0].sets[0]
disks := set.getDisks()
wrapped := make([]StorageAPI, len(disks))
for i, disk := range disks {
wrapped[i] = &nullQuorumWalkDisk{StorageAPI: disk, schedule: schedule, index: i}
}
set.getDisks = func() []StorageAPI { return wrapped }
if globalAPIConfig.getListQuorum() != "strict" {
t.Fatal("test requires strict listing across all sixteen disks")
}
writeDone := make(chan error, 1)
putReader := mustGetPutObjReader(t, bytes.NewReader(newBody), int64(len(newBody)), "", "")
go func() {
defer close(schedule.resume)
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
for range 9 {
select {
case <-schedule.oldReady:
case <-ctx.Done():
writeDone <- ctx.Err()
return
}
}
_, err := z.PutObject(ctx, bucket, overwritten, putReader, ObjectOptions{MTime: newTime})
writeDone <- err
}()
rec := nullQuorumRequest(t, router, http.MethodGet, getListObjectsV2URL("", bucket, "fixed/", "1000", "", "", ""))
if err := <-writeDone; err != nil {
t.Fatal(err)
}
var list ListObjectsV2Response
if rec.Code != http.StatusOK || xml.Unmarshal(rec.Body.Bytes(), &list) != nil {
t.Fatalf("LIST: %d %s", rec.Code, rec.Body.String())
}
var got []string
for _, object := range list.Contents {
got = append(got, object.Key)
}
t.Logf("LIST HTTP=%d KeyCount=%d IsTruncated=%v keys=%v", rec.Code, list.KeyCount, list.IsTruncated, got)
if list.KeyCount != 16 || list.IsTruncated || !slices.Equal(got, want) {
t.Errorf("LIST omitted a name during overwrite: got %v, want %v", got, want)
}
// Verify the actual reader inputs, including shape and EC, instead of
// assuming the scheduling barrier produced the intended resolver case.
schedule.mu.Lock()
snapshots := schedule.snapshots
schedule.mu.Unlock()
for i, snapshot := range snapshots {
reader := newMetacacheReader(bytes.NewReader(snapshot))
var found bool
for {
entry, err := reader.next()
if err == io.EOF {
break
}
if err != nil {
t.Fatal(err)
}
if entry.name != overwritten {
continue
}
xl, err := entry.xlmeta()
if err != nil || len(xl.versions) != 1 {
t.Fatalf("disk %d: invalid input shape: %v", i, err)
}
header := xl.versions[0].header
wantTime := oldTime
if i >= 9 {
wantTime = newTime
}
if header.ModTime != wantTime.UnixNano() || header.VersionID != [16]byte{} || header.Type != ObjectType || header.FreeVersion() {
t.Fatalf("disk %d: unexpected header %v", i, header)
}
t.Logf("disk=%d versions=1 header=%v", i, header)
found = true
}
reader.Close()
if !found {
t.Fatalf("disk %d did not emit the overwritten name", i)
}
}
get := nullQuorumRequest(t, router, http.MethodGet, getGetObjectURL("", bucket, overwritten))
head := nullQuorumRequest(t, router, http.MethodHead, getGetObjectURL("", bucket, overwritten))
t.Logf("completed PUT readback: GET=%d bytes=%d HEAD=%d length=%s", get.Code, get.Body.Len(), head.Code, head.Header().Get("Content-Length"))
if get.Code != http.StatusOK || !bytes.Equal(get.Body.Bytes(), newBody) || head.Code != http.StatusOK || head.Header().Get("Content-Length") != "8192" {
t.Fatalf("completed PUT was not readable: GET=%d HEAD=%d", get.Code, head.Code)
}
}
func TestSingleNullQuorumReadAndConsumers(t *testing.T) {
z, bucket, router := nullQuorumBackend(t)
set := z.serverPools[0].sets[0]
disks := set.getDisks()
const object = "object"
oldBody, newBody := bytes.Repeat([]byte{'a'}, 8192), bytes.Repeat([]byte{'b'}, 8192)
oldTime := time.Now().UTC().Add(-time.Hour)
put := func(body []byte, modTime time.Time) {
t.Helper()
_, err := z.PutObject(t.Context(), bucket, object, mustGetPutObjReader(t, bytes.NewReader(body), int64(len(body)), "", ""), ObjectOptions{MTime: modTime})
if err != nil {
t.Fatal(err)
}
}
put(oldBody, oldTime)
oldMeta := make([][]byte, len(disks))
for i, disk := range disks {
var err error
oldMeta[i], err = disk.ReadAll(t.Context(), bucket, object+"/"+xlStorageFormatFile)
if err != nil {
t.Fatal(err)
}
}
put(newBody, oldTime.Add(time.Minute))
newMinorityMeta := mustReadNullQuorumMeta(t, disks[14], bucket, object)
// Model an incomplete overwrite using real inline shards: fifteen old
// copies retain a read quorum; the final disk contains the newer minority.
for i := range 15 {
if err := disks[i].WriteAll(t.Context(), bucket, object+"/"+xlStorageFormatFile, oldMeta[i]); err != nil {
t.Fatal(err)
}
}
checkRead := func(body []byte, modTime time.Time) {
t.Helper()
get := nullQuorumRequest(t, router, http.MethodGet, getGetObjectURL("", bucket, object))
head := nullQuorumRequest(t, router, http.MethodHead, getGetObjectURL("", bucket, object))
if get.Code != 200 || !bytes.Equal(get.Body.Bytes(), body) || head.Code != 200 || head.Header().Get("Content-Length") != "8192" {
t.Fatalf("GET/HEAD lost readable quorum: GET=%d HEAD=%d body=%q", get.Code, head.Code, get.Body.String())
}
oi, err := z.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
if err != nil || !oi.ModTime.Equal(modTime) {
t.Fatalf("wrong readable generation: %v, %v", oi.ModTime, err)
}
}
checkRead(oldBody, oldTime)
// Reading must not reinterpret the minority as absent or delete it.
minority, err := disks[15].ReadVersion(t.Context(), "", bucket, object, "", ReadOptions{})
if err != nil || !minority.ModTime.Equal(oldTime.Add(time.Minute)) {
t.Fatalf("read mutated the minority: %+v, %v", minority, err)
}
for _, kind := range []string{"rebalance", "decommission"} {
t.Run(kind, func(t *testing.T) {
var names []string
consume := func(entry metaCacheEntry) {
versions, err := entry.fileInfoVersions(bucket)
if err != nil || len(versions.Versions) != 1 || !versions.Versions[0].ModTime.Equal(oldTime) {
t.Errorf("invalid migration metadata: %+v, %v", versions, err)
return
}
// Migration reopens the selected name/version on all drives;
// it must not treat the selected listing metadata as disk agreement.
reader, err := set.GetObjectNInfo(t.Context(), bucket, entry.name, nil, nil, ObjectOptions{VersionID: nullVersionID, NoLock: true, NoDecryption: true})
if err != nil {
t.Error(err)
return
}
body, err := io.ReadAll(reader)
reader.Close()
if err != nil || !bytes.Equal(body, oldBody) {
t.Errorf("migration could not read selected object data: %v", err)
}
names = append(names, entry.name)
}
var err error
if kind == "rebalance" {
err = set.listObjectsToRebalance(t.Context(), bucket, consume)
} else {
err = set.listObjectsToDecommission(t.Context(), decomBucketInfo{Name: bucket}, consume)
}
if err != nil || !slices.Equal(names, []string{object}) {
t.Fatalf("migration listing dropped the quorum: %v, %v", names, err)
}
})
}
for _, latestOnly := range []bool{false, true} {
results := make(chan itemOrErr[ObjectInfo], 16)
if err := z.Walk(t.Context(), bucket, "", results, WalkOptions{AskDisks: "strict", LatestOnly: latestOnly}); err != nil {
t.Fatal(err)
}
var names []string
for result := range results {
if result.Err != nil || !result.Item.ModTime.Equal(oldTime) {
t.Fatalf("Walk returned invalid metadata: %+v", result)
}
names = append(names, result.Item.Name)
}
if !slices.Equal(names, []string{object}) {
t.Fatalf("Walk dropped the quorum: %v", names)
}
}
// Exercise the scanner's actual abandoned-child path. Make the scan drive
// miss the object, while keeping an older quorum and a newer minority on
// the remaining disks. Its queued heal must read all disks and repair both.
if err := disks[14].WriteAll(t.Context(), bucket, object+"/"+xlStorageFormatFile, newMinorityMeta); err != nil {
t.Fatal(err)
}
if err := os.RemoveAll(filepath.Join(disks[15].Endpoint().Path, bucket, object)); err != nil {
t.Fatal(err)
}
scanNullQuorumAbandoned(t, z, bucket, object, oldTime)
for i, disk := range disks {
fi, err := disk.ReadVersion(t.Context(), "", bucket, object, "", ReadOptions{Healing: true})
if err != nil || !fi.ModTime.Equal(oldTime) {
t.Fatalf("scanner heal did not reconcile disk %d: %v, %v", i, fi.ModTime, err)
}
}
checkRead(oldBody, oldTime)
put(newBody, oldTime.Add(2*time.Minute))
checkRead(newBody, oldTime.Add(2*time.Minute))
}
func mustReadNullQuorumMeta(t *testing.T, disk StorageAPI, bucket, object string) []byte {
t.Helper()
data, err := disk.ReadAll(t.Context(), bucket, object+"/"+xlStorageFormatFile)
if err != nil {
t.Fatal(err)
}
return data
}
func scanNullQuorumAbandoned(t *testing.T, z *erasureServerPools, bucket, object string, oldTime time.Time) {
t.Helper()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
disks := z.serverPools[0].sets[0].getDisks()
seq := newBgHealSequence()
defer seq.cancelCtx()
previousState, previousRoutine := globalBackgroundHealState, globalBackgroundHealRoutine
globalBackgroundHealState = &allHealState{healSeqMap: map[string]*healSequence{"test": seq}}
routine := &healRoutine{tasks: make(chan healTask)}
globalBackgroundHealRoutine = routine
defer func() { globalBackgroundHealState, globalBackgroundHealRoutine = previousState, previousRoutine }()
workerDone := make(chan struct{})
var healed []string
go func() {
defer close(workerDone)
for {
select {
case <-ctx.Done():
return
case task := <-routine.tasks:
var result madmin.HealResultItem
var err error
if task.object == "" {
result, err = z.HealBucket(ctx, task.bucket, task.opts)
} else {
healed = append(healed, task.object+"/"+task.versionID)
result, err = z.HealObject(ctx, task.bucket, task.object, task.versionID, task.opts)
}
task.respCh <- healResult{result: result, err: err}
}
}
}()
cache := dataUsageCache{Info: dataUsageCacheInfo{Name: bucket}}
cache.replace(bucket, "", dataUsageEntry{})
cache.replace(bucket+"/"+object, bucket, dataUsageEntry{Size: 8192, Objects: 1, Versions: 1})
scanner := folderScanner{
root: disks[15].Endpoint().Path, oldCache: cache,
newCache: dataUsageCache{Info: cache.Info}, updateCache: dataUsageCache{Info: cache.Info},
disks: disks, disksQuorum: 8, healObjectSelect: 1,
weSleep: func() bool { return false }, shouldHeal: func() bool { return true }, updateCurrentPath: func(string) {},
getSize: func(item scannerItem) (sizeSummary, error) {
if filepath.Base(item.Path) != xlStorageFormatFile {
return sizeSummary{}, errSkipFile
}
data, err := os.ReadFile(item.Path)
if err != nil {
return sizeSummary{}, err
}
var xl xlMetaV2
if err := xl.Load(data); err != nil {
return sizeSummary{}, err
}
fi, err := xl.ToFileInfo(bucket, object, "", false, false)
if err != nil || !fi.ModTime.Equal(oldTime) {
return sizeSummary{}, fmt.Errorf("scanner re-read wrong generation: %v, %v", fi.ModTime, err)
}
return sizeSummary{totalSize: fi.Size, versions: 1}, nil
},
}
var usage dataUsageEntry
err := scanner.scanFolder(ctx, cachedFolder{name: bucket, objectHealProbDiv: 1}, &usage)
cancel()
<-workerDone
if err != nil || len(healed) != 1 || healed[0] != object+"/"+nullVersionID && healed[0] != object+"/" {
t.Fatalf("scanner did not queue a name-based heal: %v, %v", healed, err)
}
flat := scanner.newCache.sizeRecursive(bucket)
if flat == nil || flat.Objects != 1 || flat.Size != 8192 {
t.Fatalf("scanner lost the healed object from usage: %+v", flat)
}
t.Logf("scanner queued %v; healed old quorum, repaired minority/missing copies, and retained one 8192-byte object", healed)
}
+2
View File
@@ -69,6 +69,8 @@ func loadCPUMetrics(ctx context.Context, m MetricValues, c *metricsCache) error
// metrics-resource.go runs a job to collect resource metrics including their Avg values and
// stores them in resourceMetricsMap. We can use it to get the Avg values of CPU idle and IOWait.
resourceMetricsMapMu.RLock()
defer resourceMetricsMapMu.RUnlock()
cpuResourceMetrics, found := resourceMetricsMap[cpuSubsystem]
if found {
if cpuIdleMetric, ok := cpuResourceMetrics[getResourceKey(cpuIdle, nil)]; ok {
+185
View File
@@ -0,0 +1,185 @@
// Copyright (c) 2026 Ruohang Feng
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"fmt"
"runtime"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/cachevalue"
"github.com/prometheus/client_golang/prometheus"
dto "github.com/prometheus/client_model/go"
cpustats "github.com/shirou/gopsutil/v3/cpu"
"github.com/shirou/gopsutil/v3/load"
)
// These tests replace global resource metrics and must not run in parallel.
// Concurrent writers must finish before the fixture is restored.
func setCPUResourceMetricsForTest(t *testing.T, value map[MetricSubsystem]ResourceMetrics) {
t.Helper()
resourceMetricsMapMu.Lock()
saved := resourceMetricsMap
resourceMetricsMap = value
resourceMetricsMapMu.Unlock()
t.Cleanup(func() {
resourceMetricsMapMu.Lock()
resourceMetricsMap = saved
resourceMetricsMapMu.Unlock()
})
}
func newCPUMetricsTestRegistry() *prometheus.Registry {
c := &metricsCache{cpuMetrics: cachevalue.NewFromFunc(time.Hour, cachevalue.Opts{},
func(context.Context) (madmin.CPUMetrics, error) {
return madmin.CPUMetrics{
CPUCount: 4,
LoadStat: &load.AvgStat{Load1: 2},
TimesStat: &cpustats.TimesStat{
User: 10, System: 20, Idle: 60, Iowait: 5, Nice: 3, Steal: 2,
},
}, nil
})}
_, _ = c.cpuMetrics.Get() // Warm the cache: the cache does not protect the resource map.
g := NewMetricsGroup(systemCPUCollectorPath, []MetricDescriptor{
sysCPUAvgIdleMD, sysCPUAvgIOWaitMD, sysCPULoadMD, sysCPULoadPercMD,
sysCPUNiceMD, sysCPUStealMD, sysCPUSystemMD, sysCPUUserMD,
}, loadCPUMetrics)
g.SetCache(c)
r := prometheus.NewPedanticRegistry()
r.MustRegister(g)
return r
}
func TestLoadCPUMetricsValues(t *testing.T) {
for _, tc := range []struct {
name string
data map[MetricSubsystem]ResourceMetrics
want map[string]float64
}{
{name: "nil-map"},
{name: "empty-map", data: map[MetricSubsystem]ResourceMetrics{}},
{name: "no-cpu", data: map[MetricSubsystem]ResourceMetrics{memSubsystem: {}}},
{name: "nil-cpu", data: map[MetricSubsystem]ResourceMetrics{cpuSubsystem: nil}},
{name: "empty-cpu", data: map[MetricSubsystem]ResourceMetrics{cpuSubsystem: {}}},
{name: "idle-only", data: map[MetricSubsystem]ResourceMetrics{cpuSubsystem: {
getResourceKey(cpuIdle, nil): {Avg: 87.654},
}}, want: map[string]float64{"minio_system_cpu_avg_idle": 87.65}},
{name: "iowait-only", data: map[MetricSubsystem]ResourceMetrics{cpuSubsystem: {
getResourceKey(cpuIOWait, nil): {Avg: 1.236},
}}, want: map[string]float64{"minio_system_cpu_avg_iowait": 1.24}},
{name: "both-rounded", data: map[MetricSubsystem]ResourceMetrics{cpuSubsystem: {
getResourceKey(cpuIdle, nil): {Avg: 87.654},
getResourceKey(cpuIOWait, nil): {Avg: 1.236},
}}, want: map[string]float64{"minio_system_cpu_avg_idle": 87.65, "minio_system_cpu_avg_iowait": 1.24}},
{name: "zero-keeps-existing-omission", data: map[MetricSubsystem]ResourceMetrics{cpuSubsystem: {
getResourceKey(cpuIdle, nil): {Avg: 0},
getResourceKey(cpuIOWait, nil): {Avg: 0},
}}},
} {
t.Run(tc.name, func(t *testing.T) {
setCPUResourceMetricsForTest(t, tc.data)
families, err := newCPUMetricsTestRegistry().Gather()
if err != nil {
t.Fatal(err)
}
want := map[string]float64{
"minio_system_cpu_load": 2, "minio_system_cpu_load_perc": 50,
"minio_system_cpu_nice": 3, "minio_system_cpu_steal": 2,
"minio_system_cpu_system": 20, "minio_system_cpu_user": 10,
}
for k, v := range tc.want {
want[k] = v
}
if len(families) != len(want) {
t.Fatalf("got %d families, want %d", len(families), len(want))
}
for _, family := range families {
v, ok := want[family.GetName()]
if !ok || family.GetType() != dto.MetricType_GAUGE || len(family.Metric) != 1 || family.Metric[0].GetGauge().GetValue() != v {
t.Errorf("unexpected family: %v (want %v)", family, want)
}
}
})
}
}
func TestLoadCPUMetricsConcurrentUpdate(t *testing.T) {
for _, subsystem := range []MetricSubsystem{cpuSubsystem, memSubsystem} {
for _, readers := range []int{1, 4} {
t.Run(fmt.Sprintf("writer-%s/readers-%d", subsystem, readers), func(t *testing.T) {
setCPUResourceMetricsForTest(t, map[MetricSubsystem]ResourceMetrics{})
updateResourceMetrics(cpuSubsystem, cpuIdle, 80, nil, false)
updateResourceMetrics(cpuSubsystem, cpuIOWait, 5, nil, false)
r := newCPUMetricsTestRegistry()
stop := make(chan struct{})
done := make(chan struct{})
ready := make(chan struct{})
var updates atomic.Uint64
go func() {
defer close(done)
if subsystem == cpuSubsystem {
updateResourceMetrics(subsystem, cpuIdle, 80, nil, false)
} else {
updateResourceMetrics(subsystem, memUsed, 1024, nil, false)
}
close(ready)
for {
select {
case <-stop:
return
default:
}
if subsystem == cpuSubsystem {
updateResourceMetrics(subsystem, cpuIdle, 80, nil, false)
updateResourceMetrics(subsystem, cpuIOWait, 5, nil, false)
} else {
updateResourceMetrics(subsystem, memUsed, 1024, nil, false)
}
updates.Add(1)
runtime.Gosched()
}
}()
<-ready
defer func() { close(stop); <-done }()
var wg sync.WaitGroup
for range readers {
wg.Add(1)
go func() {
defer wg.Done()
for range 1000 {
families, err := r.Gather()
if err != nil || len(families) != 8 {
t.Errorf("Gather: families=%d, error=%v", len(families), err)
return
}
}
}()
}
wg.Wait()
if updates.Load() == 0 {
t.Fatal("writer made no progress")
}
t.Logf("completed %d gathers with %d concurrent update iterations", readers*1000, updates.Load())
})
}
}
}
+1 -1
View File
@@ -668,7 +668,7 @@ var (
func completeMultipartChecksumMismatch(algorithm string) error {
description := "The checksum you specified did not match the calculated checksum."
if algorithm != "" {
description = fmt.Sprintf("The %s checksum you specified did not match the calculated checksum.", algorithm)
description = fmt.Sprintf("The %s you specified did not match the calculated checksum.", algorithm)
}
return fmt.Errorf("%w: %s", errCompleteMultipartChecksumMismatch, description)
}
+6 -1
View File
@@ -20,6 +20,7 @@ package cmd
import (
"context"
"encoding/base64"
"errors"
"runtime"
"strings"
@@ -83,7 +84,8 @@ func checkListMultipartArgs(ctx context.Context, bucket, prefix, keyMarker, uplo
if err := checkListObjsArgs(ctx, bucket, prefix, keyMarker); err != nil {
return err
}
if uploadIDMarker != "" {
// S3 ignores upload-id-marker when key-marker is absent.
if uploadIDMarker != "" && keyMarker != "" {
if HasSuffix(keyMarker, SlashSeparator) {
return InvalidUploadIDKeyCombination{
UploadIDMarker: uploadIDMarker,
@@ -96,6 +98,9 @@ func checkListMultipartArgs(ctx context.Context, bucket, prefix, keyMarker, uplo
UploadID: uploadIDMarker,
}
}
if _, ok := multipartMarkerTime(uploadIDMarker); !ok {
return InvalidArgument{Bucket: bucket, Object: keyMarker, Err: errors.New("upload-id-marker must contain a native multipart upload ID")}
}
}
return nil
}
+113
View File
@@ -0,0 +1,113 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"encoding/base64"
"net/http"
"reflect"
"strconv"
"strings"
"testing"
"time"
xhttp "github.com/minio/minio/internal/http"
)
func TestPutOptsFromHeadersReplicationTimestamps(t *testing.T) {
stamp := time.Date(2026, 9, 15, 1, 2, 3, 123456789, time.UTC)
context := base64.StdEncoding.EncodeToString([]byte(`{"purpose":"tag-replication"}`))
for _, encryption := range []struct {
name string
headers map[string]string
}{
{name: "none"},
{name: "SSE-S3", headers: map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionAES}},
{name: "SSE-KMS", headers: map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionKMS}},
{name: "SSE-KMS-context", headers: map[string]string{
xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionKMS, xhttp.AmzServerSideEncryptionKmsID: "tag-replication-key",
xhttp.AmzServerSideEncryptionKmsContext: context,
}},
{name: "SSE-C", headers: ssecKeyHeaders([]byte("01234567890123456789012345678901"), false)},
} {
t.Run(encryption.name, func(t *testing.T) {
for _, trusted := range []bool{false, true} {
t.Run("trusted="+strconv.FormatBool(trusted), func(t *testing.T) {
for _, tagging := range []struct {
name, header string
want time.Time
invalid bool
}{
{name: "absent"},
{name: "nanoseconds", header: stamp.Format(time.RFC3339Nano), want: stamp},
{name: "offset-whitespace", header: " " + stamp.In(time.FixedZone("UTC+8", 8*60*60)).Format(time.RFC3339Nano) + " ", want: stamp},
{name: "invalid", header: "not-a-timestamp", invalid: true},
} {
t.Run(tagging.name, func(t *testing.T) {
for _, metadata := range []map[string]string{nil, {"x-amz-meta-test": "kept"}} {
hdr := make(http.Header)
wantEncryption := make(http.Header)
for key, value := range encryption.headers {
hdr.Set(key, value)
wantEncryption.Set(key, value)
}
hdr.Set(xhttp.MinIOSourceTaggingTimestamp, tagging.header)
hdr.Set(xhttp.MinIOSourceMTime, stamp.Add(-time.Hour).Format(time.RFC3339Nano))
hdr.Set(xhttp.MinIOSourceObjectRetentionTimestamp, stamp.Add(-time.Minute).Format(time.RFC3339Nano))
hdr.Set(xhttp.MinIOSourceObjectLegalHoldTimestamp, stamp.Add(-time.Second).Format(time.RFC3339Nano))
hdr.Set(xhttp.MinIOSourceETag, "source-etag")
opts, err := putOptsFromHeaders(t.Context(), hdr, metadata, trusted)
if trusted && tagging.invalid {
if err == nil || !strings.Contains(err.Error(), xhttp.MinIOSourceTaggingTimestamp) {
t.Fatalf("malformed trusted timestamp: got %v", err)
}
continue
}
if err != nil {
t.Fatal(err)
}
wantTag, wantMTime, wantRetention, wantLegalhold, wantETag := time.Time{}, time.Time{}, time.Time{}, time.Time{}, ""
if trusted {
wantTag, wantMTime = tagging.want, stamp.Add(-time.Hour)
wantRetention, wantLegalhold, wantETag = stamp.Add(-time.Minute), stamp.Add(-time.Second), "source-etag"
}
if !opts.ReplicationSourceTaggingTimestamp.Equal(wantTag) {
t.Errorf("tag timestamp=%s, want %s", opts.ReplicationSourceTaggingTimestamp, wantTag)
}
if !opts.MTime.Equal(wantMTime) || !opts.ReplicationSourceRetentionTimestamp.Equal(wantRetention) ||
!opts.ReplicationSourceLegalholdTimestamp.Equal(wantLegalhold) || opts.PreserveETag != wantETag || opts.ReplicationRequest != trusted {
t.Error("other source fields did not preserve the replication trust boundary")
}
if opts.UserDefined == nil || (metadata != nil && !reflect.DeepEqual(opts.UserDefined, metadata)) {
t.Errorf("metadata=%v, want nonnil map preserving %v", opts.UserDefined, metadata)
}
gotEncryption := make(http.Header)
if opts.ServerSideEncryption != nil {
opts.ServerSideEncryption.Marshal(gotEncryption)
}
if !reflect.DeepEqual(gotEncryption, wantEncryption) {
t.Errorf("SSE headers=%v, want %v", gotEncryption, wantEncryption)
}
}
})
}
})
}
})
}
}
+8 -3
View File
@@ -347,6 +347,11 @@ func putOpts(ctx context.Context, bucket, object, vid string, hdrs http.Header,
versionSuspended := globalBucketVersioningSys.PrefixSuspended(bucket, object)
vid = strings.TrimSpace(vid)
// A caller-chosen version ID writes that version in place. Only trusted
// replication may recreate a source version under its original ID.
if vid != "" && !trustedReplication {
return opts, errAuthentication
}
if vid != "" && vid != nullVersionID {
_, err := uuid.Parse(vid)
if err != nil {
@@ -452,11 +457,11 @@ func putOptsFromHeaders(ctx context.Context, hdr http.Header, metadata map[strin
MTime: mtime,
PreserveETag: etag,
ReplicationRequest: trustedReplication,
// The Object Lock timestamps order replicated retention and legal
// hold updates. Dropping them here would leave every update on an
// SSE-KMS destination unordered.
// These timestamps order replicated retention, legal hold and tagging
// updates on an SSE-KMS destination.
ReplicationSourceLegalholdTimestamp: lholdtimestmp,
ReplicationSourceRetentionTimestamp: retaintimestmp,
ReplicationSourceTaggingTimestamp: taggingtimestmp,
}
return op, nil
}
+143
View File
@@ -0,0 +1,143 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/crypto"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/kms"
)
// TestAPICopyObjectReplicaTaggingTimestampUnderKMS covers signed replica COPY
// requests through encryption, metadata replacement and disk persistence. Both
// the single-disk and 16-disk fixtures are single-pool backends.
func TestAPICopyObjectReplicaTaggingTimestampUnderKMS(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testAPICopyObjectReplicaTaggingTimestampUnderKMS})
}
func testAPICopyObjectReplicaTaggingTimestampUnderKMS(obj ObjectLayer, instance, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
// Ignore the host free-space percentage while retaining real disk I/O.
for _, pool := range obj.(*erasureServerPools).serverPools {
for _, set := range pool.sets {
original := set.getDisks
disks := append([]StorageAPI(nil), original()...)
for i := range disks {
disks[i] = tagTestCapacityDisk{StorageAPI: disks[i]}
}
set.getDisks = func() []StorageAPI { return disks }
defer func() { set.getDisks = original }()
}
}
oldKMS, oldAuto := GlobalKMS, globalAutoEncryption
GlobalKMS = kms.NewStub("replica-tags-key")
globalAutoEncryption = false
defer func() { GlobalKMS, globalAutoEncryption = oldKMS, oldAuto }()
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
const tsKey = ReservedMetadataPrefixLower + TaggingTimestamp
stamp := time.Date(2026, 9, 15, 1, 0, 0, 123456789, time.UTC)
for _, mode := range []string{"none", "explicit-sse-s3", "explicit-kms", "auto-kms", "bucket-kms"} {
t.Run(instance+"/"+mode, func(t *testing.T) {
globalAutoEncryption = mode == "auto-kms"
if mode == "bucket-kms" {
sseXML := []byte(`<ServerSideEncryptionConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>aws:kms</SSEAlgorithm><KMSMasterKeyID>replica-tags-key</KMSMasterKeyID></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>`)
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketSSEConfig, sseXML); err != nil {
t.Fatal(err)
}
}
const data = "encrypted replica copy remains readable"
oi, err := obj.PutObject(t.Context(), bucket, mode, mustGetPutObjReader(t, bytes.NewReader([]byte(data)), int64(len(data)), "", ""), ObjectOptions{
Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "key=old", tsKey: stamp.Format(time.RFC3339Nano)},
})
if err != nil {
t.Fatal(err)
}
for _, event := range []struct {
name, tags, wantTags string
delta, wantDelta time.Duration
missingTimestamp bool
}{
{"newer", "key=new", "key=new", 2, 2, false},
{"stale", "key=stale", "key=new", 1, 2, false},
{"duplicate", "key=new", "key=new", 2, 2, false},
{"newer-again", "key=latest", "key=latest", 3, 3, false},
{"missing-timestamp", "key=unordered", "key=latest", 0, 3, true},
} {
headers := map[string]string{
xhttp.AmzCopySource: "/" + bucket + "/" + mode + "?versionId=" + oi.VersionID,
xhttp.AmzMetadataDirective: "REPLACE", xhttp.AmzTagDirective: "REPLACE",
xhttp.AmzObjectTagging: event.tags, xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA", xhttp.MinIOSourceTaggingTimestamp: stamp.Add(event.delta).Format(time.RFC3339Nano),
xhttp.MinIOSourceMTime: oi.ModTime.Format(time.RFC3339Nano), xhttp.MinIOSourceETag: oi.ETag,
}
if event.missingTimestamp {
delete(headers, xhttp.MinIOSourceTaggingTimestamp)
}
if mode == "explicit-sse-s3" {
headers[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionAES
}
if mode == "explicit-kms" {
headers[xhttp.AmzServerSideEncryption] = "aws:kms"
headers[xhttp.AmzServerSideEncryptionKmsID] = "replica-tags-key"
}
req, err := newTestSignedRequestV4(http.MethodPut, "/"+bucket+"/"+mode+"?versionId="+oi.VersionID, 0, nil, creds.AccessKey, creds.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("%s: COPY %d %s", event.name, w.Code, w.Body.String())
}
got, err := obj.GetObjectInfo(t.Context(), bucket, mode, ObjectOptions{VersionID: oi.VersionID})
if err != nil {
t.Fatal(err)
}
t.Logf("%s: tags=%q timestamp=%q kms=%v", event.name, got.UserTags, got.UserDefined[tsKey], crypto.S3KMS.IsEncrypted(got.UserDefined))
if got.UserTags != event.wantTags || got.UserDefined[tsKey] != stamp.Add(event.wantDelta).Format(time.RFC3339Nano) {
t.Errorf("%s: incorrect persisted tags/timestamp", event.name)
}
wantKMS := mode == "explicit-kms" || mode == "auto-kms" || mode == "bucket-kms"
if crypto.S3KMS.IsEncrypted(got.UserDefined) != wantKMS || crypto.S3.IsEncrypted(got.UserDefined) != (mode == "explicit-sse-s3") {
t.Errorf("%s: unexpected destination encryption", event.name)
}
if got.VersionID != oi.VersionID {
t.Errorf("%s: version=%q, want %q", event.name, got.VersionID, oi.VersionID)
}
req, err = newTestSignedRequestV4(http.MethodGet, "/"+bucket+"/"+mode+"?versionId="+oi.VersionID, 0, nil, creds.AccessKey, creds.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
w = httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK || w.Body.String() != data {
t.Fatalf("%s: GET %d %q", event.name, w.Code, w.Body.String())
}
}
})
}
}
+4 -1
View File
@@ -240,7 +240,10 @@ func checkPreconditionsPUT(ctx context.Context, w http.ResponseWriter, r *http.R
// updated. The predicate is the incoming request's restored SSE-C metadata,
// not what the destination happens to hold.
ssecReplica := isReplicaTrusted(r.Context()) && crypto.SSEC.IsEncrypted(opts.UserDefined)
if etagMatch && vidMatch && !ssecReplica {
// Matching content does not imply that its tag revision was delivered.
// Keep client preconditions above; relax only the internal duplicate check.
newerTags := isReplicaTrusted(r.Context()) && olderThan(objInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], opts.ReplicationSourceTaggingTimestamp)
if etagMatch && vidMatch && !ssecReplica && !newerTags {
writeHeaders()
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrPreconditionFailed), r.URL)
return true
+48 -30
View File
@@ -1797,6 +1797,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
// source timestamp is newer than the stored one, and a stale update must
// leave the stored state in place instead of erasing it.
storedLock := storedObjectLockState(srcInfo.UserDefined)
storedTagTimestamp := srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
srcInfo.UserDefined, err = getCpObjMetadataFromHeader(ctx, r, srcInfo.UserDefined, allowReplicationMetadata)
if err != nil {
@@ -1814,23 +1815,29 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
}
}
if objTags != "" {
lastTaggingTimestamp := srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
if dstOpts.ReplicationRequest {
srcTimestamp := dstOpts.ReplicationSourceTaggingTimestamp
if !srcTimestamp.IsZero() {
ondiskTimestamp, err := time.Parse(time.RFC3339Nano, lastTaggingTimestamp)
// update tagging metadata only if replica timestamp is newer than what's on disk
if err != nil || (err == nil && !ondiskTimestamp.After(srcTimestamp)) {
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = srcTimestamp.UTC().Format(time.RFC3339Nano)
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
}
}
} else {
if dstOpts.ReplicationRequest {
srcTimestamp := dstOpts.ReplicationSourceTaggingTimestamp
if !srcTimestamp.IsZero() {
// An empty value with a timestamp is an ordered deletion. Recheck
// the captured state even if metadata REPLACE rebuilt the map.
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = srcTimestamp.UTC().Format(time.RFC3339Nano)
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = UTCNow().Format(time.RFC3339Nano)
reconcileStoredObjectTags(srcInfo.UserDefined, srcInfo.UserTags, storedTagTimestamp)
} else {
srcInfo.UserDefined[xhttp.AmzObjectTagging] = srcInfo.UserTags
if storedTagTimestamp != "" {
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = storedTagTimestamp
} else {
delete(srcInfo.UserDefined, ReservedMetadataPrefixLower+TaggingTimestamp)
}
}
} else {
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = UTCNow().Format(time.RFC3339Nano)
}
// SSE-C rotation snapshots reserved metadata before the tag decision. Its
// later merge must not put the old timestamp back over the accepted state.
delete(encMetadata, ReservedMetadataPrefixLower+TaggingTimestamp)
srcInfo.UserDefined = filterReplicationStatusMetadata(srcInfo.UserDefined)
srcInfo.UserDefined = objectlock.FilterObjectLockMetadata(srcInfo.UserDefined, true, true)
@@ -2313,6 +2320,9 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if opts.ReplicationRequest && !opts.ReplicationSourceTaggingTimestamp.IsZero() {
metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = opts.ReplicationSourceTaggingTimestamp.UTC().Format(time.RFC3339Nano)
}
actualSize := size
var idxCb func() []byte
@@ -3425,27 +3435,16 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
return
}
// Check permissions to perform this object retention operation
if s3Error := authenticateRequest(ctx, r, policy.PutObjectRetentionAction); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
return
}
if _, err := objectAPI.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if !validateLengthAndChecksum(r) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL)
return
}
if rcfg, _ := globalBucketObjectLockSys.Get(bucket); !rcfg.LockEnabled {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidBucketObjectLockConfiguration), r.URL)
return
}
objRetention, err := objectlock.ParseObjectRetention(r.Body)
if err != nil {
apiErr := errorCodes.ToAPIErr(ErrMalformedXML)
@@ -3457,6 +3456,24 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
reqInfo := logger.GetReqInfo(ctx)
reqInfo.SetTags("retention", objRetention.String())
// Check permissions to perform this object retention operation. The
// object-lock condition keys come from the request body, so this is
// decided before any bucket or object state is read.
if !isRetentionActionAllowed(policy.PutObjectRetentionAction, bucket, object, objRetention, r, reqInfo.Cred, reqInfo.Owner) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if _, err := objectAPI.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if rcfg, _ := globalBucketObjectLockSys.Get(bucket); !rcfg.LockEnabled {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidBucketObjectLockConfiguration), r.URL)
return
}
opts, err := getOpts(ctx, r, bucket, object)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
@@ -3760,11 +3777,11 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
}
dsc := mustReplicate(ctx, bucket, object, getMustReplicateOptions(objInfo.UserDefined, tagsStr, objInfo.ReplicationStatus, replication.MetadataReplicationType, opts))
stamp := UTCNow().Format(time.RFC3339Nano)
opts.UserDefined = map[string]string{ReservedMetadataPrefixLower + TaggingTimestamp: stamp}
if dsc.ReplicateAny() {
opts.UserDefined = make(map[string]string)
opts.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] = UTCNow().Format(time.RFC3339Nano)
opts.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] = stamp
opts.UserDefined[ReservedMetadataPrefixLower+ReplicationStatus] = dsc.PendingStatus()
opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = UTCNow().Format(time.RFC3339Nano)
}
// Put object tags
@@ -3863,9 +3880,10 @@ func (api objectAPIHandlers) DeleteObjectTaggingHandler(w http.ResponseWriter, r
}
dsc := mustReplicate(ctx, bucket, object, oi.getMustReplicateOptions(replication.MetadataReplicationType, opts))
stamp := UTCNow().Format(time.RFC3339Nano)
opts.UserDefined = map[string]string{ReservedMetadataPrefixLower + TaggingTimestamp: stamp}
if dsc.ReplicateAny() {
opts.UserDefined = make(map[string]string)
opts.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] = UTCNow().Format(time.RFC3339Nano)
opts.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] = stamp
opts.UserDefined[ReservedMetadataPrefixLower+ReplicationStatus] = dsc.PendingStatus()
}
+5 -1
View File
@@ -312,6 +312,10 @@ func (api objectAPIHandlers) NewMultipartUploadHandler(w http.ResponseWriter, r
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
// Completion orders the upload's persisted tag state under the object lock.
if opts.ReplicationRequest && !opts.ReplicationSourceTaggingTimestamp.IsZero() {
metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = opts.ReplicationSourceTaggingTimestamp.UTC().Format(time.RFC3339Nano)
}
if r.Header.Get(xhttp.IfMatch) != "" {
opts.HasIfMatch = true
@@ -1162,7 +1166,7 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
return
}
if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, false); s3Err != ErrNone {
if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, trustedReplication); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
+679
View File
@@ -0,0 +1,679 @@
// Copyright (c) 2015-2026 MinIO, Inc.
// Copyright (c) 2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
package cmd
import (
"archive/tar"
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/amztime"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
xhttp "github.com/minio/minio/internal/http"
"github.com/pgsty/silo-pkg/v3/policy"
)
func retentionAuthzRequest(t *testing.T, router http.Handler, creds auth.Credentials, method, target string, body []byte, headers map[string]string) *httptest.ResponseRecorder {
t.Helper()
var rd io.ReadSeeker
if body != nil {
rd = bytes.NewReader(body)
}
req, err := newTestSignedRequestV4(method, target, int64(len(body)), rd, creds.AccessKey, creds.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
return rec
}
func retentionBody(mode string, until time.Time) []byte {
if mode == "" {
return []byte(`<Retention xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></Retention>`)
}
return fmt.Appendf(nil, `<Retention xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Mode>%s</Mode><RetainUntilDate>%s</RetainUntilDate></Retention>`,
mode, until.UTC().Format(time.RFC3339))
}
func putRetention(t *testing.T, router http.Handler, cred auth.Credentials, bucket, key, vid, mode string, until time.Time, bypass bool) int {
t.Helper()
body := retentionBody(mode, until)
sum := md5.Sum(body)
headers := map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])}
if bypass {
headers[objectlock.AmzObjectLockBypassRetGovernance] = "true"
}
rec := retentionAuthzRequest(t, router, cred, http.MethodPut,
makeTestTargetURL("", bucket, key, url.Values{"retention": {""}, "versionId": {vid}}), body, headers)
return rec.Code
}
func newRetentionPolicyUser(t *testing.T, instanceType, policyJSON string) auth.Credentials {
t.Helper()
ctx := t.Context()
accessKey, secretKey, err := auth.GenerateCredentials()
if err != nil {
t.Fatalf("%s: generate credentials: %v", instanceType, err)
}
creds := auth.Credentials{AccessKey: accessKey, SecretKey: secretKey}
if _, err = globalIAMSys.CreateUser(ctx, creds.AccessKey, madmin.AddOrUpdateUserReq{
SecretKey: creds.SecretKey,
Status: madmin.AccountEnabled,
}); err != nil {
t.Fatalf("%s: create user: %v", instanceType, err)
}
parsed, err := policy.ParseConfig(strings.NewReader(policyJSON))
if err != nil {
t.Fatalf("%s: parse policy: %v", instanceType, err)
}
policyName := "retention-authz-" + mustGetUUID()
if _, err = globalIAMSys.SetPolicy(ctx, policyName, *parsed); err != nil {
t.Fatalf("%s: install policy: %v", instanceType, err)
}
if _, err = globalIAMSys.PolicyDBSet(ctx, creds.AccessKey, policyName, regUser, false); err != nil {
t.Fatalf("%s: attach policy: %v", instanceType, err)
}
return creds
}
// TestAPIPutObjectRetentionAuthorization covers SN-2026-015: the governance
// bypass header is intent only; s3:PutObjectRetention is always required and
// weakening unexpired GOVERNANCE also requires s3:BypassGovernanceRetention.
func TestAPIPutObjectRetentionAuthorization(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPIPutObjectRetentionAuthorization,
})
}
func testAPIPutObjectRetentionAuthorization(obj ObjectLayer, instanceType, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
enableBucketObjectLock(t, bucket)
noRet := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:GetObject"`)
retOnly := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:PutObjectRetention"`)
retBypass := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:PutObjectRetention","s3:BypassGovernanceRetention"`)
bypassOnly := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:BypassGovernanceRetention"`)
condDenyShort := newRetentionPolicyUser(t, instanceType, `{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObjectRetention", "s3:BypassGovernanceRetention"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"]
},
{
"Effect": "Deny",
"Action": ["s3:PutObjectRetention"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"],
"Condition": {"NumericLessThan": {"s3:object-lock-remaining-retention-days": "60"}}
}
]
}`)
explicitDenyBypass := newRetentionPolicyUser(t, instanceType, `{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"]
},
{
"Effect": "Deny",
"Action": ["s3:BypassGovernanceRetention"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"]
}
]
}`)
far := UTCNow().Add(365 * 24 * time.Hour).Truncate(time.Second)
nearer := UTCNow().Add(30 * 24 * time.Hour).Truncate(time.Second)
farther := far.Add(24 * time.Hour)
put := func(t *testing.T, object, mode string) string {
t.Helper()
payload := []byte("retention authorization")
opts := ObjectOptions{Versioned: true, UserDefined: map[string]string{}}
if mode != "" {
opts.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] = mode
opts.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(far)
}
info, err := obj.PutObject(t.Context(), bucket, object,
mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""), opts)
if err != nil {
t.Fatal(err)
}
return info.VersionID
}
stored := func(t *testing.T, object, vid string) objectlock.ObjectRetention {
t.Helper()
info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: vid})
if err != nil {
t.Fatal(err)
}
return objectlock.GetObjectRetentionMeta(info.UserDefined)
}
for _, test := range []struct {
name string
creds auth.Credentials
existing string // stored retention mode on the version, until far
mode string // requested mode, "" removes retention
until time.Time
bypass bool
want int
changed bool
}{
{name: "no-permission removal", creds: noRet, existing: "GOVERNANCE", bypass: true, want: http.StatusForbidden},
{name: "no-permission mode change", creds: noRet, existing: "GOVERNANCE", mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "no-permission new compliance", creds: noRet, mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "retention-only shorten with header", creds: retOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "retention-only removal with header", creds: retOnly, existing: "GOVERNANCE", bypass: true, want: http.StatusForbidden},
{name: "retention-only mode change with header", creds: retOnly, existing: "GOVERNANCE", mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "retention-only shorten without header", creds: retOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, want: http.StatusBadRequest},
{name: "retention-only extend with header", creds: retOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: farther, bypass: true, want: http.StatusOK, changed: true},
{name: "retention-only new retention", creds: retOnly, mode: "GOVERNANCE", until: nearer, want: http.StatusOK, changed: true},
{name: "bypass shorten with header", creds: retBypass, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusOK, changed: true},
{name: "bypass removal with header", creds: retBypass, existing: "GOVERNANCE", bypass: true, want: http.StatusOK, changed: true},
{name: "bypass shorten without header", creds: retBypass, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, want: http.StatusBadRequest},
{name: "bypass shorten compliance", creds: retBypass, existing: "COMPLIANCE", mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusBadRequest},
{name: "bypass-only extend", creds: bypassOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: farther, bypass: true, want: http.StatusForbidden},
{name: "conditional-deny-put-retention with bypass header", creds: condDenyShort, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "conditional-allow-put-retention satisfied", creds: condDenyShort, existing: "GOVERNANCE", mode: "GOVERNANCE", until: farther, bypass: true, want: http.StatusOK, changed: true},
{name: "explicit-deny-bypass shorten with header", creds: explicitDenyBypass, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "owner shorten with header", creds: root, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusOK, changed: true},
} {
t.Run(instanceType+"/"+test.name, func(t *testing.T) {
object := "retention-authz/" + strings.ReplaceAll(test.name, " ", "-")
vid := put(t, object, test.existing)
before := stored(t, object, vid)
body := retentionBody(test.mode, test.until)
sum := md5.Sum(body)
headers := map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])}
if test.bypass {
headers[objectlock.AmzObjectLockBypassRetGovernance] = "true"
}
target := makeTestTargetURL("", bucket, object, url.Values{"retention": {""}, "versionId": {vid}})
rec := retentionAuthzRequest(t, router, test.creds, http.MethodPut, target, body, headers)
if rec.Code != test.want {
t.Fatalf("status = %d, want %d: %s", rec.Code, test.want, rec.Body.String())
}
after := stored(t, object, vid)
if changed := after.Mode != before.Mode || !after.RetainUntilDate.Equal(before.RetainUntilDate.Time); changed != test.changed {
t.Fatalf("retention changed = %v, want %v (before %s, after %s)", changed, test.changed, before.String(), after.String())
}
})
}
t.Run(instanceType+"/no-permission missing bucket", func(t *testing.T) {
body := retentionBody("", time.Time{})
sum := md5.Sum(body)
target := makeTestTargetURL("", "retention-authz-missing", "object", url.Values{"retention": {""}})
rec := retentionAuthzRequest(t, router, noRet, http.MethodPut, target, body,
map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403 before bucket lookup: %s", rec.Code, rec.Body.String())
}
})
}
// TestAPIVersionAddressedWriteRequiresReplicationTrust covers SN-2026-016: a
// caller-chosen destination versionId is honored only for trusted replication,
// and write paths cannot overwrite locked versions via directory keys or
// version-addressed multipart completions.
func TestAPIVersionAddressedWriteRequiresReplicationTrust(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPIVersionAddressedWriteRequiresReplicationTrust,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
})
}
func testAPIVersionAddressedWriteRequiresReplicationTrust(obj ObjectLayer, instanceType, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:GetObject","s3:PutObject"`)
original := []byte("original version content")
info, err := obj.PutObject(t.Context(), bucket, "addressed",
mustGetPutObjReader(t, bytes.NewReader(original), int64(len(original)), "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
vid := info.VersionID
unchanged := func(t *testing.T) {
t.Helper()
got, err := obj.GetObjectInfo(t.Context(), bucket, "addressed", ObjectOptions{VersionID: vid})
if err != nil {
t.Fatal(err)
}
if got.Size != int64(len(original)) || got.ETag != info.ETag {
t.Fatalf("version %s was rewritten: size %d etag %s", vid, got.Size, got.ETag)
}
}
for _, creds := range []struct {
name string
creds auth.Credentials
}{{"writer", writer}, {"owner", root}} {
t.Run(instanceType+"/"+creds.name+"/put", func(t *testing.T) {
body := []byte("replacement")
target := makeTestTargetURL("", bucket, "addressed", url.Values{"versionId": {vid}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPut, target, body, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
})
t.Run(instanceType+"/"+creds.name+"/null-version", func(t *testing.T) {
body := []byte("replacement")
target := makeTestTargetURL("", bucket, "addressed", url.Values{"versionId": {nullVersionID}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPut, target, body, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
})
t.Run(instanceType+"/"+creds.name+"/copy", func(t *testing.T) {
target := makeTestTargetURL("", bucket, "addressed", url.Values{"versionId": {vid}})
req, err := newTestSignedRequestV4(http.MethodPut, target, 0, nil, creds.creds.AccessKey, creds.creds.SecretKey,
map[string]string{xhttp.AmzCopySource: "/" + bucket + "/addressed?versionId=" + vid, xhttp.AmzMetadataDirective: "REPLACE"})
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
})
t.Run(instanceType+"/"+creds.name+"/multipart", func(t *testing.T) {
target := makeTestTargetURL("", bucket, "addressed", url.Values{"uploads": {""}, "versionId": {vid}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPost, target, nil, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
})
t.Run(instanceType+"/"+creds.name+"/complete-multipart-existing-version", func(t *testing.T) {
// Simulate a pending upload persisted with an existing version ID
// (e.g. initiated pre-upgrade or by a replication peer).
up, err := obj.NewMultipartUpload(t.Context(), bucket, "addressed", ObjectOptions{
Versioned: true,
VersionID: vid,
})
if err != nil {
t.Fatal(err)
}
defer obj.AbortMultipartUpload(t.Context(), bucket, "addressed", up.UploadID, ObjectOptions{})
partPayload := []byte("replacement-part")
pi, err := obj.PutObjectPart(t.Context(), bucket, "addressed", up.UploadID, 1,
mustGetPutObjReader(t, bytes.NewReader(partPayload), int64(len(partPayload)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
completeBody := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"%s"</ETag></Part></CompleteMultipartUpload>`, pi.ETag)
target := makeTestTargetURL("", bucket, "addressed", url.Values{"uploadId": {up.UploadID}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPost, target, completeBody, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
})
}
t.Run(instanceType+"/locked-directory-object", func(t *testing.T) {
far := UTCNow().Add(365 * 24 * time.Hour).Truncate(time.Second)
for _, lockCase := range []struct {
name string
meta map[string]string
}{
{
name: "governance",
meta: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "GOVERNANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(far),
},
},
{
name: "compliance",
meta: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "COMPLIANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(far),
},
},
{
name: "legal-hold",
meta: map[string]string{
strings.ToLower(xhttp.AmzObjectLockLegalHold): "ON",
},
},
} {
t.Run(lockCase.name, func(t *testing.T) {
dirKey := "locked-dir-" + lockCase.name + "/"
if _, err := obj.PutObject(t.Context(), bucket, dirKey,
mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""),
ObjectOptions{Versioned: true, VersionID: nullVersionID, UserDefined: lockCase.meta}); err != nil {
t.Fatal(err)
}
target := makeTestTargetURL("", bucket, dirKey, nil)
rec := retentionAuthzRequest(t, router, writer, http.MethodPut, target, nil, nil)
if rec.Code != http.StatusBadRequest {
t.Fatalf("PUT locked dir status = %d, want 400: %s", rec.Code, rec.Body.String())
}
got, err := obj.GetObjectInfo(t.Context(), bucket, dirKey, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
for k, wantVal := range lockCase.meta {
if got.UserDefined[k] != wantVal {
t.Fatalf("metadata %s = %q, want %q", k, got.UserDefined[k], wantVal)
}
}
})
}
})
t.Run(instanceType+"/snowball-archive-version-id", func(t *testing.T) {
var tarBuf bytes.Buffer
tw := tar.NewWriter(&tarBuf)
entryData := []byte("snowball-overwrite")
if err := tw.WriteHeader(&tar.Header{
Typeflag: tar.TypeReg,
Name: "addressed",
Size: int64(len(entryData)),
Format: tar.FormatPAX,
PAXRecords: map[string]string{"minio.versionId": vid},
}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(entryData); err != nil {
t.Fatal(err)
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
// Untrusted writer cannot overwrite an existing version via PAX minio.versionId.
target := makeTestTargetURL("", bucket, "archive.tar", nil)
rec := retentionAuthzRequest(t, router, writer, http.MethodPut, target, tarBuf.Bytes(),
map[string]string{"x-amz-meta-snowball-auto-extract": "true"})
if rec.Code != http.StatusForbidden {
t.Fatalf("untrusted snowball status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
// Trusted batch replication (with outer MinIOSourceReplicationRequest) preserves PAX minio.versionId.
replicaVID := mustGetUUID()
var repBuf bytes.Buffer
repTW := tar.NewWriter(&repBuf)
repData := []byte("replicated-via-snowball")
if err := repTW.WriteHeader(&tar.Header{
Typeflag: tar.TypeReg,
Name: "snowball-replicated",
Size: int64(len(repData)),
Format: tar.FormatPAX,
PAXRecords: map[string]string{"minio.versionId": replicaVID},
}); err != nil {
t.Fatal(err)
}
if _, err := repTW.Write(repData); err != nil {
t.Fatal(err)
}
if err := repTW.Close(); err != nil {
t.Fatal(err)
}
repRec := retentionAuthzRequest(t, router, root, http.MethodPut, target, repBuf.Bytes(),
map[string]string{
"x-amz-meta-snowball-auto-extract": "true",
xhttp.MinIOSourceReplicationRequest: "true",
})
if repRec.Code != http.StatusOK {
t.Fatalf("trusted snowball status = %d, want 200: %s", repRec.Code, repRec.Body.String())
}
if _, err := obj.GetObjectInfo(t.Context(), bucket, "snowball-replicated", ObjectOptions{VersionID: replicaVID}); err != nil {
t.Fatalf("trusted snowball replication did not preserve version ID: %v", err)
}
})
t.Run(instanceType+"/trusted replication", func(t *testing.T) {
replicaVID := mustGetUUID()
body := []byte("replicated version")
target := makeTestTargetURL("", bucket, "replicated", url.Values{"versionId": {replicaVID}})
rec := retentionAuthzRequest(t, router, root, http.MethodPut, target, body,
map[string]string{xhttp.MinIOSourceReplicationRequest: "true"})
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200: %s", rec.Code, rec.Body.String())
}
if _, err := obj.GetObjectInfo(t.Context(), bucket, "replicated", ObjectOptions{VersionID: replicaVID}); err != nil {
t.Fatalf("trusted replication did not keep its version ID: %v", err)
}
})
}
func TestPutObjectRetentionClearMinimumDaysPolicy(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
for _, conditionalAllow := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/conditional-allow-%v", instance, conditionalAllow), func(t *testing.T) {
statement := `{"Effect":"Allow","Action":["s3:PutObjectRetention","s3:BypassGovernanceRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"]}, {"Effect":"Deny","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"],"Condition":{"NumericLessThan":{"s3:object-lock-remaining-retention-days":"60"}}}`
if conditionalAllow {
statement = `{"Effect":"Allow","Action":["s3:BypassGovernanceRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"]}, {"Effect":"Allow","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"],"Condition":{"NumericGreaterThanEquals":{"s3:object-lock-remaining-retention-days":"60"}}}`
}
cred := newRetentionPolicyUser(t, instance, `{"Version":"2012-10-17","Statement":[`+statement+`]}`)
key := fmt.Sprintf("review-minimum-%v", conditionalAllow)
far := UTCNow().Add(365 * 24 * time.Hour).Truncate(time.Second)
info, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""), ObjectOptions{
Versioned: true,
UserDefined: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "GOVERNANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(far),
},
})
if err != nil {
t.Fatal(err)
}
if status := putRetention(t, router, cred, bucket, key, info.VersionID, "GOVERNANCE", UTCNow().Add(30*24*time.Hour), true); status != http.StatusForbidden {
t.Fatalf("shortening to 30 days status = %d, want 403", status)
}
status := putRetention(t, router, cred, bucket, key, info.VersionID, "", time.Time{}, true)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: info.VersionID})
if err != nil {
t.Fatal(err)
}
ret := objectlock.GetObjectRetentionMeta(after.UserDefined)
if status != http.StatusForbidden || ret.Mode != objectlock.RetGovernance {
t.Fatalf("clearing retention status = %d, stored retention = %s; want 403 and original GOVERNANCE", status, ret.String())
}
})
}
},
})
}
type hookBeforeWrite struct {
ObjectLayer
key string
before func()
}
func (p *hookBeforeWrite) fire(key string) {
if key == p.key && p.before != nil {
f := p.before
p.before = nil
f()
}
}
func (p *hookBeforeWrite) PutObject(ctx context.Context, bucket, key string, r *PutObjReader, opts ObjectOptions) (ObjectInfo, error) {
p.fire(key)
return p.ObjectLayer.PutObject(ctx, bucket, key, r, opts)
}
func (p *hookBeforeWrite) CopyObject(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject string, srcInfo ObjectInfo, srcOpts, dstOpts ObjectOptions) (ObjectInfo, error) {
p.fire(dstObject)
return p.ObjectLayer.CopyObject(ctx, srcBucket, srcObject, dstBucket, dstObject, srcInfo, srcOpts, dstOpts)
}
func (p *hookBeforeWrite) CompleteMultipartUpload(ctx context.Context, bucket, key, uploadID string, parts []CompletePart, opts ObjectOptions) (ObjectInfo, error) {
p.fire(key)
return p.ObjectLayer.CompleteMultipartUpload(ctx, bucket, key, uploadID, parts, opts)
}
func TestDirectoryObjectLockAfterPreflight(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:GetObject","s3:PutObject"`)
srcDir := "other-dir/"
if rec := retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, srcDir, nil), nil, nil); rec.Code != http.StatusOK {
t.Fatalf("srcDir PUT: %d %s", rec.Code, rec.Body.String())
}
for _, mode := range []string{"GOVERNANCE", "COMPLIANCE", "LEGAL-HOLD"} {
for _, op := range []string{"PutObject", "CopyObject", "SelfCopy"} {
t.Run(instance+"/"+mode+"/"+op, func(t *testing.T) {
key := fmt.Sprintf("review-race-%s-%s/", mode, op)
if rec := retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, nil); rec.Code != http.StatusOK {
t.Fatalf("initial PUT: %d %s", rec.Code, rec.Body.String())
}
var locked ObjectInfo
hook := &hookBeforeWrite{ObjectLayer: obj, key: key}
hook.before = func() {
if mode == "LEGAL-HOLD" {
body := []byte(`<LegalHold><Status>ON</Status></LegalHold>`)
sum := md5.Sum(body)
rec := retentionAuthzRequest(t, router, root, http.MethodPut, makeTestTargetURL("", bucket, key, url.Values{"legal-hold": {""}, "versionId": {nullVersionID}}), body, map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
if rec.Code != http.StatusOK {
t.Fatalf("set legal hold: %d %s", rec.Code, rec.Body.String())
}
} else if status := putRetention(t, router, root, bucket, key, nullVersionID, mode, UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("set retention: %d", status)
}
var err error
locked, err = obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
}
setObjectLayer(hook)
defer setObjectLayer(obj)
var rec *httptest.ResponseRecorder
switch op {
case "PutObject":
rec = retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, nil)
case "CopyObject":
rec = retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, map[string]string{
xhttp.AmzCopySource: pathJoin(bucket, srcDir),
})
case "SelfCopy":
rec = retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, map[string]string{
xhttp.AmzCopySource: pathJoin(bucket, key), xhttp.AmzMetadataDirective: "REPLACE", "x-amz-meta-updated": "true",
})
}
if hook.before != nil {
t.Fatalf("%s never reached backend: %d %s", op, rec.Code, rec.Body.String())
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusBadRequest {
t.Fatalf("locked null write via %s: %d %s", op, rec.Code, rec.Body.String())
}
multipartConditionError(t, rec, "InvalidRequest")
retentionOverwriteUnchanged(t, locked, after)
})
}
}
},
})
}
func TestMultipartVersioningTransitions(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
t.Run(instance+"/unversioned-to-enabled", func(t *testing.T) {
key := "review-upload-before-versioning"
initRec := retentionAuthzRequest(t, router, root, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploads": {""}}), nil, nil)
if initRec.Code != http.StatusOK {
t.Fatalf("init: %d %s", initRec.Code, initRec.Body.String())
}
var mp InitiateMultipartUploadResponse
if err := xml.Unmarshal(initRec.Body.Bytes(), &mp); err != nil {
t.Fatal(err)
}
part := retentionAuthzRequest(t, router, root, http.MethodPut, getPutObjectPartURL("", bucket, key, mp.UploadID, "1"), []byte("ordinary pending upload"), nil)
if part.Code != http.StatusOK {
t.Fatalf("part: %d %s", part.Code, part.Body.String())
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, []byte(`<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
body := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>%s</ETag></Part></CompleteMultipartUpload>`, multipartConditionResponseETag(part))
rec := retentionAuthzRequest(t, router, root, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {mp.UploadID}}), body, nil)
if rec.Code != http.StatusOK {
t.Fatalf("ordinary upload initiated before versioning fails completion: %d %s", rec.Code, rec.Body.String())
}
})
t.Run(instance+"/addressed-upload-after-suspension", func(t *testing.T) {
key := "review-addressed-before-suspension"
original := []byte("original")
info, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(original), int64(len(original)), "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
up, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{Versioned: true, VersionID: info.VersionID})
if err != nil {
t.Fatal(err)
}
part := []byte("replaced existing version")
pi, err := obj.PutObjectPart(t.Context(), bucket, key, up.UploadID, 1, mustGetPutObjReader(t, bytes.NewReader(part), int64(len(part)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, []byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
body := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"%s"</ETag></Part></CompleteMultipartUpload>`, pi.ETag)
rec := retentionAuthzRequest(t, router, writer, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}}), body, nil)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: info.VersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusForbidden || after.ETag != info.ETag {
t.Fatalf("suspended-bucket existing UUID overwritten: status=%d old-etag=%s new-etag=%s body=%s", rec.Code, info.ETag, after.ETag, rec.Body.String())
}
})
},
})
}
+546
View File
@@ -0,0 +1,546 @@
// Copyright (c) 2026 Feng Ruohang
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This file is part of Silo Object Storage stack.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/amztime"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
xhttp "github.com/minio/minio/internal/http"
)
func retentionOverwriteUnchanged(t *testing.T, before, after ObjectInfo) {
t.Helper()
if after.ETag != before.ETag || after.Size != before.Size || after.VersionID != before.VersionID || !after.ModTime.Equal(before.ModTime) {
t.Error("rejected write changed the stored object")
}
if objectlock.GetObjectRetentionMeta(after.UserDefined).String() != objectlock.GetObjectRetentionMeta(before.UserDefined).String() || objectlock.GetObjectLegalHoldMeta(after.UserDefined).Status != objectlock.GetObjectLegalHoldMeta(before.UserDefined).Status {
t.Error("rejected write changed the stored lock")
}
}
func TestObjectLockOverwriteConfigUnavailable(t *testing.T) {
previous := newObjectLayerFn()
setObjectLayer(nil)
defer setObjectLayer(previous)
err := checkObjectLockedForOverwrite(t.Context(), "lock-config-unavailable", "object", func(context.Context, string, string, ObjectOptions) (ObjectInfo, error) {
t.Fatal("object must not be read without a bucket configuration")
return ObjectInfo{}, nil
})
if !errors.Is(err, errServerNotInitialized) {
t.Fatalf("unavailable lock configuration must fail closed, got %v", err)
}
}
func TestPutObjectRetentionMaximumDaysPolicy(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, _ auth.Credentials, t *testing.T) {
for _, conditionalAllow := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/conditional-allow=%t", instance, conditionalAllow), func(t *testing.T) {
statement := fmt.Sprintf(`{"Effect":"Allow","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::%s/*"],"Condition":{"NumericLessThanEquals":{"s3:object-lock-remaining-retention-days":"200000"}}}`, bucket)
if !conditionalAllow {
statement = fmt.Sprintf(`{"Effect":"Allow","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::%s/*"]},{"Effect":"Deny","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::%s/*"],"Condition":{"NumericGreaterThan":{"s3:object-lock-remaining-retention-days":"200000"}}}`, bucket, bucket)
}
cred := newRetentionPolicyUser(t, instance, `{"Version":"2012-10-17","Statement":[`+statement+`]}`)
key := fmt.Sprintf("maximum-retention-%t", conditionalAllow)
initial, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
// Both dates exceed time.Duration's range (~292 years). A saturated
// Time.Sub would report 106752 days and incorrectly allow 1000 years.
if status := putRetention(t, router, cred, bucket, key, initial.VersionID, "GOVERNANCE", UTCNow().AddDate(1000, 0, 0), false); status != http.StatusForbidden {
t.Fatalf("1000-year retention: want 403, got %d", status)
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
retentionOverwriteUnchanged(t, initial, after)
if status := putRetention(t, router, cred, bucket, key, initial.VersionID, "GOVERNANCE", UTCNow().AddDate(500, 0, 0), false); status != http.StatusOK {
t.Fatalf("500-year retention: want 200, got %d", status)
}
})
}
},
})
}
func TestAPICopyObjectPreservesVersionAfterSuspension(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
t.Run(instance, func(t *testing.T) {
key := "self-copy-after-suspension"
original, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("payload"), 7, "", ""), ObjectOptions{
Versioned: true, UserDefined: map[string]string{"x-amz-meta-state": "original"},
})
if err != nil {
t.Fatal(err)
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig,
[]byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
rec := retentionAuthzRequest(t, router, root, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, map[string]string{
xhttp.AmzCopySource: "/" + bucket + "/" + key, xhttp.AmzMetadataDirective: "REPLACE", "x-amz-meta-state": "replacement",
})
if rec.Code != http.StatusOK {
t.Fatalf("copy: %d %s", rec.Code, rec.Body.String())
}
old, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: original.VersionID})
if err != nil {
t.Fatal(err)
}
if old.UserDefined["x-amz-meta-state"] != "original" {
t.Errorf("copy changed metadata of existing UUID %s: %v", original.VersionID, old.UserDefined)
}
current, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatalf("copy must create the null version: %v", err)
}
if current.UserDefined["X-Amz-Meta-State"] != "replacement" {
t.Errorf("new null version metadata: %v", current.UserDefined)
}
})
},
})
}
// A persisted null upload can survive an upgrade or a versioning transition.
// The completion must see locks committed after the handler's preflight.
func TestMultipartNullObjectLockAfterPreflight(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
for _, mode := range []string{"GOVERNANCE", "COMPLIANCE", "LEGAL-HOLD"} {
t.Run(instance+"/"+mode, func(t *testing.T) {
ctx := t.Context()
key := "multipart-null-" + mode
opts := ObjectOptions{Versioned: true, VersionID: nullVersionID, ReplicationRequest: true}
if _, err := obj.PutObject(ctx, bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), opts); err != nil {
t.Fatal(err)
}
mp, err := obj.NewMultipartUpload(ctx, bucket, key, opts)
if err != nil {
t.Fatal(err)
}
part, err := obj.PutObjectPart(ctx, bucket, key, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
var locked ObjectInfo
hook := &hookBeforeWrite{ObjectLayer: obj, key: key, before: func() {
if mode == "LEGAL-HOLD" {
body := []byte(`<LegalHold><Status>ON</Status></LegalHold>`)
sum := md5.Sum(body)
rec := retentionAuthzRequest(t, router, root, http.MethodPut, makeTestTargetURL("", bucket, key, url.Values{"legal-hold": {""}, "versionId": {nullVersionID}}), body, map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
if rec.Code != http.StatusOK {
t.Fatalf("set legal hold: %d %s", rec.Code, rec.Body.String())
}
} else if status := putRetention(t, router, root, bucket, key, nullVersionID, mode, UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("set retention: %d", status)
}
var err error
locked, err = obj.GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
}}
setObjectLayer(hook)
defer setObjectLayer(obj)
body := []byte(multipartConditionCompleteBody([]CompletePart{{PartNumber: 1, ETag: part.ETag}}))
rec := retentionAuthzRequest(t, router, writer, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {mp.UploadID}}), body, nil)
if hook.before != nil || rec.Code != http.StatusBadRequest {
t.Fatalf("completion after locking: %d %s", rec.Code, rec.Body.String())
}
multipartConditionError(t, rec, "InvalidRequest")
after, err := obj.GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
retentionOverwriteUnchanged(t, locked, after)
parts, err := obj.ListObjectParts(ctx, bucket, key, mp.UploadID, 0, 100, ObjectOptions{})
if err != nil || len(parts.Parts) != 1 || parts.Parts[0].ETag != part.ETag {
t.Fatalf("rejected completion removed upload parts: %v %+v", err, parts.Parts)
}
})
}
},
})
}
// The newer, unlocked copy is in the write pool; the lock exists only in the
// other pool. A set-local check would miss it even while holding a write lock.
func TestPoolsObjectLockOverwrite(t *testing.T) {
z, _ := consistencyPools(t)
bucket, _, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: true, LockEnabled: true})
if err != nil {
t.Fatal(err)
}
for _, state := range []string{"GOVERNANCE", "COMPLIANCE", "LEGAL-HOLD", "expired", "unlocked", "unreadable"} {
for _, op := range []string{"put", "copy", "self-copy", "multipart"} {
t.Run(state+"/"+op, func(t *testing.T) {
ctx := t.Context()
key := encodeDirObject("null-" + state + "-" + op + "/")
meta := map[string]string{}
switch state {
case "GOVERNANCE", "COMPLIANCE":
meta[strings.ToLower(xhttp.AmzObjectLockMode)] = state
meta[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(UTCNow().Add(365 * 24 * time.Hour))
case "LEGAL-HOLD":
meta[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = "ON"
case "expired":
meta[strings.ToLower(xhttp.AmzObjectLockMode)] = "GOVERNANCE"
meta[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(UTCNow().Add(-24 * time.Hour))
}
seed := ObjectOptions{Versioned: true, VersionID: nullVersionID, ReplicationRequest: true, MTime: UTCNow().Add(-time.Hour), UserDefined: meta}
putConsistencyObject(t, z, bucket, key, 1, "original", seed)
seed.UserDefined = map[string]string{}
seed.MTime = UTCNow()
putConsistencyObject(t, z, bucket, key, 0, "original", seed)
before := make([]ObjectInfo, len(z.serverPools))
for i, pool := range z.serverPools {
var err error
before[i], err = pool.GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
}
defer conditionalPutPool(t, z, key, 0)()
var restore func()
if state == "unreadable" {
restore = conditionalPutSwapDisks(z.serverPools[1], key, func(d StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: d, bucket: bucket, object: key}
})
defer restore()
}
opts := ObjectOptions{Versioned: true, VersionID: nullVersionID, UserDefined: map[string]string{"x-amz-meta-updated": "true"}}
var writeErr error
switch op {
case "put":
_, writeErr = z.PutObject(ctx, bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), opts)
case "copy", "self-copy":
source := key
if op == "copy" {
source = "source-" + state
putConsistencyObject(t, z, bucket, source, 0, "replacement", ObjectOptions{Versioned: true})
}
gr, gerr := z.GetObjectNInfo(ctx, bucket, source, nil, nil, ObjectOptions{})
if gerr != nil {
t.Fatal(gerr)
}
defer gr.Close()
info := gr.ObjInfo
info.UserDefined = opts.UserDefined
info.metadataOnly = op == "self-copy"
info.PutObjReader = mustGetPutObjReader(t, gr, info.Size, "", "")
info.Reader = info.PutObjReader.Reader
_, writeErr = z.CopyObject(ctx, bucket, source, bucket, key, info, ObjectOptions{VersionID: nullVersionID}, opts)
case "multipart":
id, parts := multipartConditionUpload(t, z, bucket, key, 0, opts)
_, writeErr = z.CompleteMultipartUpload(ctx, bucket, key, id, parts, ObjectOptions{Versioned: true})
}
allowed := state == "expired" || state == "unlocked"
if allowed {
if writeErr != nil {
t.Fatalf("unlocked write: %v", writeErr)
}
return
}
if state == "unreadable" {
if !isErrReadQuorum(writeErr) {
t.Fatalf("unreadable pool: want quorum error, got %v", writeErr)
}
restore()
} else if !errors.As(writeErr, &ObjectLocked{}) {
t.Fatalf("lock in other pool: want ObjectLocked, got %v", writeErr)
}
for i, original := range before {
after, gerr := z.serverPools[i].GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if gerr != nil {
t.Fatal(gerr)
}
retentionOverwriteUnchanged(t, original, after)
}
})
}
}
}
type planReviewGatedBody struct {
io.ReadCloser
first sync.Once
ready chan struct{}
release chan struct{}
}
func (g *planReviewGatedBody) Read(p []byte) (int, error) {
g.first.Do(func() { close(g.ready); <-g.release })
return g.ReadCloser.Read(p)
}
func TestPlanReviewSlowDirectoryPut(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
key := "slow-body-directory/"
before, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), ObjectOptions{Versioned: true, VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
payload := []byte("replacement-body")
req, err := newTestSignedRequestV4(http.MethodPut, makeTestTargetURL("", bucket, key, nil), int64(len(payload)), bytes.NewReader(payload), writer.AccessKey, writer.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
gate := &planReviewGatedBody{ReadCloser: req.Body, ready: make(chan struct{}), release: make(chan struct{})}
req.Body = gate
var once sync.Once
release := func() { once.Do(func() { close(gate.release) }) }
defer release()
rec := httptest.NewRecorder()
done := make(chan struct{})
go func() { router.ServeHTTP(rec, req); close(done) }()
select {
case <-gate.ready:
case <-done:
t.Fatalf("PUT finished before body: %d %s", rec.Code, rec.Body.String())
case <-time.After(5 * time.Second):
t.Fatal("PUT did not reach body read")
}
locked := make(chan int, 1)
go func() {
locked <- putRetention(t, router, root, bucket, key, nullVersionID, "COMPLIANCE", UTCNow().Add(365*24*time.Hour), false)
}()
select {
case status := <-locked:
if status != http.StatusOK {
t.Fatalf("set lock while body paused: %d", status)
}
case <-time.After(5 * time.Second):
t.Fatal("retention update blocked behind paused body")
}
release()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("PUT did not finish")
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusBadRequest || after.ETag != before.ETag || objectlock.GetObjectRetentionMeta(after.UserDefined).Mode != objectlock.RetCompliance {
t.Fatal("paused ordinary PUT overwrote the COMPLIANCE null version after the lock was acknowledged")
}
})
},
})
}
type planReviewReadFault struct {
ObjectLayer
key string
fired bool
putCalled bool
}
func (p *planReviewReadFault) GetObjectInfo(ctx context.Context, bucket, key string, opts ObjectOptions) (ObjectInfo, error) {
if key == p.key && opts.VersionID == nullVersionID && !p.fired {
p.fired = true
return ObjectInfo{}, InsufficientReadQuorum{}
}
return p.ObjectLayer.GetObjectInfo(ctx, bucket, key, opts)
}
func (p *planReviewReadFault) PutObject(ctx context.Context, bucket, key string, data *PutObjReader, opts ObjectOptions) (ObjectInfo, error) {
if key == p.key {
p.putCalled = true
}
return p.ObjectLayer.PutObject(ctx, bucket, key, data, opts)
}
func TestPlanReviewPreflightReadFailure(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, _ auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
key := "locked-directory-read-fault/"
_, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), ObjectOptions{
Versioned: true, VersionID: nullVersionID, ReplicationRequest: true,
UserDefined: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "COMPLIANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(UTCNow().Add(365 * 24 * time.Hour)),
},
})
if err != nil {
t.Fatal(err)
}
initial, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
fault := &planReviewReadFault{ObjectLayer: obj, key: key}
setObjectLayer(fault)
defer setObjectLayer(obj)
rec := retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), []byte("replacement"), nil)
if !fault.fired {
t.Fatal("preflight fault was not injected")
}
if fault.putCalled {
t.Fatal("preflight read error allowed PutObject to proceed to storage layer")
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code < 400 {
t.Errorf("metadata read quorum error was treated as absence: status=%d", rec.Code)
}
retentionOverwriteUnchanged(t, initial, after)
})
},
})
}
func TestPlanReviewLegacyMPUCompliance(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
key := "legacy-mpu-compliance"
initial, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
up, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{Versioned: true, VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
payload := []byte("replacement-part")
part, err := obj.PutObjectPart(t.Context(), bucket, key, up.UploadID, 1, mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if status := putRetention(t, router, root, bucket, key, initial.VersionID, "COMPLIANCE", UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("set compliance: %d", status)
}
before, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
body := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"%s"</ETag></Part></CompleteMultipartUpload>`, part.ETag)
rec := retentionAuthzRequest(t, router, writer, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}}), body, nil)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusForbidden {
t.Errorf("legacy upload replaced locked UUID: status=%d", rec.Code)
}
retentionOverwriteUnchanged(t, before, after)
})
},
})
}
// A recovered upload retains an authentic server-generated version ID while a
// surviving replica reintroduces its already-completed version and lock.
// Only the fixture uses replication privileges; completion uses PutObject only.
func TestRecoveredMultipartUploadAfterReplica(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
z := obj.(*erasureServerPools)
key := "recovered-generated-upload"
up, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
er := z.serverPools[0].getHashedSet(key)
fi, _, err := er.checkUploadIDExists(t.Context(), bucket, key, up.UploadID, true)
if err != nil {
t.Fatal(err)
}
_, err = obj.PutObject(t.Context(), bucket, key,
mustGetPutObjReader(t, bytes.NewBufferString("protected-original"), 18, "", ""),
ObjectOptions{Versioned: true, VersionID: fi.VersionID, ReplicationRequest: true})
if err != nil {
t.Fatal(err)
}
if status := putRetention(t, router, root, bucket, key, fi.VersionID, "COMPLIANCE", UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("setting compliance lock: %d", status)
}
before, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: fi.VersionID})
if err != nil {
t.Fatal(err)
}
partRec := retentionAuthzRequest(t, router, writer, http.MethodPut,
makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}, "partNumber": {"1"}}),
[]byte("attacker-replacement"), nil)
if partRec.Code != http.StatusOK {
t.Fatalf("upload part: %d %s", partRec.Code, partRec.Body.String())
}
etag := multipartConditionResponseETag(partRec)
body := []byte(multipartConditionCompleteBody([]CompletePart{{PartNumber: 1, ETag: etag}}))
rec := retentionAuthzRequest(t, router, writer, http.MethodPost,
makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}}), body, nil)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: fi.VersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusForbidden {
t.Errorf("recovered upload bypassed existing UUID guard: status=%d", rec.Code)
}
retentionOverwriteUnchanged(t, before, after)
})
},
})
}
+276
View File
@@ -0,0 +1,276 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"archive/tar"
"bytes"
"compress/gzip"
"encoding/xml"
"maps"
"net/http"
"net/http/httptest"
"net/url"
"reflect"
"strings"
"testing"
"github.com/minio/minio/internal/auth"
xhttp "github.com/minio/minio/internal/http"
)
// Exercise authenticated handlers and actual disk metadata, including the
// response headers consumers see after replication has completed.
func TestAPIReplicaContentEncoding(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testAPIReplicaContentEncoding})
}
func testAPIReplicaContentEncoding(obj ObjectLayer, instance, bucket string, router http.Handler, owner auth.Credentials, t *testing.T) {
ordinary := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject","s3:GetObject"`)
replicator := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject","s3:GetObject","s3:ReplicateObject"`)
for _, mode := range []string{"ordinary", "untrusted-marker", "replica"} {
for _, tc := range []struct{ name, wire, want string }{
{"bare", "aws-chunked", ""}, {"mixed", "aws-chunked,gzip", "gzip"}, {"gzip", "gzip", "gzip"},
} {
for _, operation := range []string{"put", "copy-replace", "multipart"} {
t.Run(instance+"/"+mode+"/"+tc.name+"/"+operation, func(t *testing.T) {
object := mode + "/" + tc.name + "/" + operation
payload := replicaEncodingPayload(t, tc.want)
creds := ordinary
headers := map[string]string{xhttp.ContentEncoding: tc.wire, xhttp.ContentType: "application/octet-stream", "X-Amz-Meta-Source": "encoding-test"}
if mode != "ordinary" {
headers[xhttp.MinIOSourceReplicationRequest] = "true"
}
if mode == "replica" {
creds = replicator
headers[xhttp.AmzBucketReplicationStatus] = "REPLICA"
}
send := func(method, target string, data []byte, hdrs map[string]string) *httptest.ResponseRecorder {
t.Helper()
req, err := newTestSignedRequestV4(method, target, int64(len(data)), bytes.NewReader(data), creds.AccessKey, creds.SecretKey, hdrs)
if err != nil {
t.Fatal(err)
}
return replicaEncodingServe(t, router, req, http.StatusOK)
}
switch operation {
case "put":
if strings.Contains(tc.wire, "aws-chunked") {
req := replicaEncodingStream(t, getPutObjectURL("", bucket, object), payload, creds, headers)
replicaEncodingServe(t, router, req, http.StatusOK)
} else {
send(http.MethodPut, getPutObjectURL("", bucket, object), payload, headers)
}
case "copy-replace":
source := object + "-source"
if _, err := obj.PutObject(t.Context(), bucket, source, mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""), ObjectOptions{}); err != nil {
t.Fatal(err)
}
headers[xhttp.AmzCopySource] = url.QueryEscape("/" + bucket + "/" + source)
headers[xhttp.AmzMetadataDirective] = replaceDirective
send(http.MethodPut, getCopyObjectURL("", bucket, object), nil, headers)
case "multipart":
rec := send(http.MethodPost, getNewMultipartURL("", bucket, object), nil, headers)
var init InitiateMultipartUploadResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &init); err != nil {
t.Fatal(err)
}
// Part/completion metadata must not replace the encoding saved at initiation.
partHeaders := map[string]string{xhttp.ContentEncoding: "br"}
if mode == "replica" {
partHeaders[xhttp.MinIOSourceReplicationRequest] = "true"
partHeaders[xhttp.AmzBucketReplicationStatus] = "REPLICA"
}
part := send(http.MethodPut, getPutObjectPartURL("", bucket, object, init.UploadID, "1"), payload, partHeaders)
partETags := part.Header()[xhttp.ETag]
if len(partETags) != 1 {
t.Fatalf("missing part ETag: %#v", part.Header())
}
complete, err := xml.Marshal(CompleteMultipartUpload{Parts: []CompletePart{{PartNumber: 1, ETag: canonicalizeETag(partETags[0])}}})
if err != nil {
t.Fatal(err)
}
send(http.MethodPost, getCompleteMultipartUploadURL("", bucket, object, init.UploadID), complete, partHeaders)
}
assertReplicaEncodingObject(t, obj, router, owner, bucket, object, tc.want, payload)
info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if got := info.UserDefined[xhttp.AmzBucketReplicationStatus]; (got == "REPLICA") != (mode == "replica") {
t.Errorf("replica status %q for mode %s", got, mode)
}
if info.ContentType != "application/octet-stream" {
t.Errorf("content-type=%q", info.ContentType)
}
if value, ok := caseInsensitiveMap(info.UserDefined).Lookup("x-amz-meta-source"); !ok || value != "encoding-test" {
t.Errorf("user metadata lost: %#v", info.UserDefined)
}
})
}
}
}
t.Run(instance+"/unauthorized-replica", func(t *testing.T) {
object := "denied-replica"
req := replicaEncodingStream(t, getPutObjectURL("", bucket, object), []byte("denied"), ordinary, map[string]string{xhttp.ContentEncoding: "aws-chunked", xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA"})
rec := replicaEncodingServe(t, router, req, http.StatusForbidden)
var response APIErrorResponse
if err := xml.Unmarshal(rec.Body.Bytes(), &response); err != nil {
t.Fatal(err)
}
if response.Code != "AccessDenied" {
t.Fatalf("expected permission denial, got %s", response.Code)
}
if _, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{}); err == nil {
t.Error("denied replica created an object")
}
})
}
func replicaEncodingPayload(t *testing.T, encoding string) []byte {
t.Helper()
data := bytes.Repeat([]byte("replica encoding payload\n"), 128)
if encoding != "gzip" {
return data
}
var b bytes.Buffer
w := gzip.NewWriter(&b)
if _, err := w.Write(data); err != nil {
t.Fatal(err)
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
func replicaEncodingStream(t *testing.T, target string, data []byte, creds auth.Credentials, headers map[string]string) *http.Request {
t.Helper()
const chunkSize = 64
body := bytes.NewReader(data)
req, err := newTestStreamingRequest(http.MethodPut, target, int64(len(data)), chunkSize, body)
if err != nil {
t.Fatal(err)
}
for k, v := range headers {
req.Header.Set(k, v)
}
now := UTCNow()
signature, err := signStreamingRequest(req, creds.AccessKey, creds.SecretKey, now)
if err != nil {
t.Fatal(err)
}
req, err = assembleStreamingChunks(req, body, chunkSize, creds.SecretKey, signature, now)
if err != nil {
t.Fatal(err)
}
return req
}
func replicaEncodingServe(t *testing.T, router http.Handler, req *http.Request, want int) *httptest.ResponseRecorder {
t.Helper()
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != want {
t.Fatalf("%s %s: status=%d want=%d body=%s", req.Method, req.URL, rec.Code, want, rec.Body.String())
}
return rec
}
func assertReplicaEncodingObject(t *testing.T, obj ObjectLayer, router http.Handler, creds auth.Credentials, bucket, object, encoding string, data []byte) {
t.Helper()
info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if info.ContentEncoding != encoding {
t.Errorf("persisted content-encoding=%q want=%q", info.ContentEncoding, encoding)
}
if encoding == "" {
if _, present := info.UserDefined["content-encoding"]; present {
t.Error("transport-only content-encoding key persisted")
}
}
for _, method := range []string{http.MethodGet, http.MethodHead} {
req, err := newTestSignedRequestV4(method, getPutObjectURL("", bucket, object), 0, nil, creds.AccessKey, creds.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
rec := replicaEncodingServe(t, router, req, http.StatusOK)
if got := rec.Header().Get(xhttp.ContentEncoding); got != encoding {
t.Errorf("%s content-encoding=%q want=%q", method, got, encoding)
}
if encoding == "" {
if _, present := rec.Header()[xhttp.ContentEncoding]; present {
t.Errorf("%s sent an empty/transport encoding header", method)
}
}
if method == http.MethodGet && !bytes.Equal(rec.Body.Bytes(), data) {
t.Errorf("GET body differs: got %d bytes want %d", rec.Body.Len(), len(data))
}
}
}
func TestAPISnowballReplicaContentEncoding(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
for _, tc := range []struct {
name string
pax map[string]string
want string
}{
{name: "no-pax"},
{name: "pax-without-encoding", pax: map[string]string{"minio.metadata.Content-Type": "application/octet-stream"}},
{name: "pax-bare", pax: map[string]string{"minio.metadata.Content-Encoding": "aws-chunked"}},
{name: "pax-mixed", pax: map[string]string{"minio.metadata.Content-Encoding": "aws-chunked,gzip"}, want: "gzip"},
} {
t.Run(instance+"/"+tc.name, func(t *testing.T) {
object := "snowball/" + tc.name
data := replicaEncodingPayload(t, tc.want)
var archive bytes.Buffer
tw := tar.NewWriter(&archive)
if err := tw.WriteHeader(&tar.Header{Name: object, Mode: 0o600, Size: int64(len(data)), PAXRecords: tc.pax}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(data); err != nil {
t.Fatal(err)
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
var ordinaryMetadata map[string]string
// An unauthorized entry in a REPLICA request is rejected. Compare the
// same archive across ordinary and authorized replica requests instead.
for _, replica := range []bool{false, true} {
headers := map[string]string{
xhttp.ContentEncoding: "aws-chunked", xhttp.AmzSnowballExtract: "true",
xhttp.ContentType: "application/x-tar", xhttp.CacheControl: "max-age=123",
"X-Amz-Meta-Archive": "outer-request",
}
if replica {
headers[xhttp.MinIOSourceReplicationRequest] = "true"
headers[xhttp.AmzBucketReplicationStatus] = "REPLICA"
}
req := replicaEncodingStream(t, getPutObjectURL("", bucket, "archive.tar"), archive.Bytes(), creds, headers)
replicaEncodingServe(t, router, req, http.StatusOK)
assertReplicaEncodingObject(t, obj, router, creds, bucket, object, tc.want, data)
info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
metadata := maps.Clone(info.UserDefined)
for _, key := range []string{xhttp.AmzBucketReplicationStatus, ReservedMetadataPrefixLower + ReplicaStatus, ReservedMetadataPrefixLower + ReplicaTimestamp, "etag"} {
delete(metadata, key)
}
if !replica {
ordinaryMetadata = metadata
} else if !reflect.DeepEqual(metadata, ordinaryMetadata) {
t.Errorf("replica inherited ordinary archive metadata: got %#v want %#v", metadata, ordinaryMetadata)
}
}
})
}
}})
}
+5 -19
View File
@@ -106,6 +106,7 @@ func TestReplicateDeleteMarkerTargetSemantics(t *testing.T) {
func testReplicateDeleteMarkerPurge(obj ObjectLayer, instanceType, bucket string, router http.Handler, creds auth.Credentials, t *testing.T, legacy bool) {
ctx := t.Context()
defer replicationTestCapacity(obj)()
const arn = "arn:minio:replication::af470089-d354-4473-934c-9e1f52f6da89:bucket"
const name = "marker"
version := mustGetUUID()
@@ -208,27 +209,12 @@ func testReplicateDeleteMarkerPurge(obj ObjectLayer, instanceType, bucket string
t.Errorf("purge scheduled as marker creation: version=%q marker=%q", deletion.VersionID, deletion.DeleteMarkerVersionID)
}
if legacy {
// Reproduce the old producer's state and let the existing scanner/heal
// path recover it. Upgrades must also finish purges already left pending.
// Old task shapes must complete directly; scanner/MRF recovery is
// covered separately by TestReplicationMRFMarkerRecovery.
deletion.VersionID, deletion.DeleteMarkerVersionID = "", version
replicateDelete(ctx, deletion, obj)
oi, _ := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true})
if oi.VersionPurgeStatus != replication.VersionPurgePending {
t.Fatalf("legacy source purge = %s, want PENDING", oi.VersionPurgeStatus)
}
targets, err := globalBucketTargetSys.ListBucketTargets(ctx, bucket)
if err != nil {
t.Fatal(err)
}
queueReplicationHeal(ctx, bucket, oi, replicationConfig{Config: &cfg, remotes: targets}, 0)
select {
case op := <-worker:
deletion = op.(DeletedObjectReplicationInfo)
case <-time.After(time.Second):
t.Fatal("legacy pending purge was not scheduled for healing")
}
}
result := replicateDelete(context.Background(), deletion, obj)
result := replicateDelete(context.Background(), deletion, markerPurgeUpdateLayer{ObjectLayer: obj, t: t})
if result.VersionPurgeStatus() != replication.VersionPurgeComplete {
t.Errorf("remote purge result = %s, want COMPLETE", result.VersionPurgeStatus())
}
+665
View File
@@ -0,0 +1,665 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/logger"
loghttp "github.com/minio/minio/internal/logger/target/http"
"github.com/minio/minio/internal/once"
xnet "github.com/pgsty/silo-pkg/v3/net"
)
type markerRecoveryCase struct {
name string
legacy, creation, lostReply, partial, exhaust, lockFirst, invalidMRF, replicaSource, offline, unrecordedPurge bool
targets int
}
func TestReplicationMRFMarkerRecovery(t *testing.T) {
for _, tc := range []markerRecoveryCase{
{name: "canonical", targets: 1},
{name: "lock-failure", lockFirst: true, targets: 1},
{name: "invalid-MRF-metadata", invalidMRF: true, targets: 1},
{name: "legacy", legacy: true, targets: 1},
{name: "unrecorded-purge", unrecordedPurge: true, targets: 2},
{name: "two-targets", legacy: true, targets: 2},
{name: "two-targets-offline", offline: true, targets: 2},
{name: "replica-source", replicaSource: true, targets: 1},
{name: "lost-reply-canonical", lostReply: true, targets: 1},
{name: "lost-reply-legacy", legacy: true, lostReply: true, targets: 1},
{name: "creation", creation: true, targets: 1},
{name: "partial-creation-block", partial: true, targets: 2},
{name: "retry-budget-and-scanner", exhaust: true, targets: 1},
} {
t.Run(tc.name, func(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, endpoints: []string{"DeleteObject"}, objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
testReplicationMRFMarkerRecovery(t, obj, backend, bucket, router, creds, tc)
}})
})
}
}
type markerRecoveryTarget struct {
arn, bucket string
client *minio.Client
reject, loseReply atomic.Bool
deletes atomic.Int32
}
// Only adapt host capacity accounting, using the existing real-disk adapter.
// Reads, object metadata, MRF files and writes all still use the fixture disks.
func replicationTestCapacity(obj ObjectLayer) func() {
var restore []func()
for _, pool := range obj.(*erasureServerPools).serverPools {
pool.erasureDisksMu.Lock()
for _, set := range pool.sets {
original := pool.erasureDisks[set.setIndex]
disks := append([]StorageAPI(nil), original...)
for i, disk := range disks {
if disk != nil {
disks[i] = tagTestCapacityDisk{StorageAPI: disk}
}
}
// GetDisks copies this list under the same mutex. Keep its function
// stable while background IAM scans use the fixture, both when
// installing the adapter and when restoring the original disks.
pool.erasureDisks[set.setIndex] = disks
restore = append(restore, func() {
pool.erasureDisksMu.Lock()
pool.erasureDisks[set.setIndex] = original
pool.erasureDisksMu.Unlock()
})
}
pool.erasureDisksMu.Unlock()
}
return func() {
for _, fn := range restore {
fn()
}
}
}
func TestReplicationCapacityConcurrentIAM(t *testing.T) {
z, _ := consistencyPools(t)
if _, _, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
iam := globalIAMSys
started, finished := make(chan struct{}), make(chan error, 1)
go func() {
close(started)
for range 20 {
if err := iam.Load(ctx, false); err != nil {
finished <- err
return
}
}
finished <- nil
}()
<-started
for range 5000 {
replicationTestCapacity(z)()
}
if err := <-finished; err != nil {
t.Fatal(err)
}
}
func testReplicationMRFMarkerRecovery(t *testing.T, obj ObjectLayer, backend, bucket string, router http.Handler, creds auth.Credentials, tc markerRecoveryCase) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
defer replicationTestCapacity(obj)()
stats := NewReplicationStats(ctx, nil)
oldStats := globalReplicationStats.Swap(stats)
defer globalReplicationStats.Store(oldStats)
oldPool := globalReplicationPool
defer func() { globalReplicationPool = oldPool }()
const name = "marker"
version := mustGetUUID()
creationTime := UTCNow().Add(-time.Hour).Truncate(time.Second)
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
if _, err := obj.PutObject(ctx, bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true}); err != nil {
t.Fatal(err)
}
var targets []*markerRecoveryTarget
cfg := replication.Config{}
creationStates := make(map[string]replication.StatusType)
var creationInternal string
for i := 0; i < tc.targets; i++ {
target := &markerRecoveryTarget{arn: "arn:minio:replication::" + mustGetUUID() + ":bucket", bucket: getRandomBucketName()}
if err := obj.MakeBucket(ctx, target.bucket, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
if _, err := obj.PutObject(ctx, target.bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true}); err != nil {
t.Fatal(err)
}
if !tc.creation {
opts := ObjectOptions{VersionID: version, Versioned: true, DeleteMarker: true, ReplicationRequest: true, MTime: creationTime}
opts.SetReplicaStatus(replication.Replica)
if _, err := obj.DeleteObject(ctx, target.bucket, name, opts); err != nil {
t.Fatal(err)
}
}
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
opts := ObjectOptions{VersionID: r.URL.Query().Get("versionId"), Versioned: true}
switch r.Method {
case http.MethodHead:
oi, err := obj.GetObjectInfo(r.Context(), target.bucket, name, opts)
if oi.DeleteMarker {
w.Header().Set(xhttp.AmzDeleteMarker, "true")
w.Header().Set(xhttp.AmzVersionID, oi.VersionID)
}
if err != nil {
writeErrorResponseHeadersOnly(w, toAPIError(r.Context(), err))
return
}
w.WriteHeader(http.StatusOK)
case http.MethodDelete:
target.deletes.Add(1)
if got := r.Header.Get(xhttp.MinIOSourceDeleteMarker) == "true"; got != tc.creation {
t.Errorf("purge/creation wire flag=%v creation=%v", got, tc.creation)
}
if opts.VersionID == "" {
t.Error("missing remote versionId")
}
if target.reject.Load() {
w.WriteHeader(http.StatusForbidden)
fmt.Fprint(w, `<Error><Code>AccessDenied</Code></Error>`)
return
}
opts.DeleteMarker = r.Header.Get(xhttp.MinIOSourceDeleteMarker) == "true"
opts.ReplicationRequest = true
opts.SetReplicaStatus(replication.Replica)
_, err := obj.DeleteObject(r.Context(), target.bucket, name, opts)
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
writeErrorResponse(r.Context(), w, toAPIError(r.Context(), err), r.URL)
return
}
if target.loseReply.Swap(false) {
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Error(err)
return
}
conn.Close()
return
}
w.WriteHeader(http.StatusNoContent)
default:
t.Errorf("unexpected remote method %s", r.Method)
}
}))
defer remote.Close()
client, err := minio.New(strings.TrimPrefix(remote.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
target.client = client
globalBucketTargetSys.Lock()
globalBucketTargetSys.arnRemotesMap[target.arn] = arnTarget{Client: &TargetClient{Client: client, ARN: target.arn, Bucket: target.bucket}, lastRefresh: UTCNow()}
globalBucketTargetSys.targetsMap[bucket] = append(globalBucketTargetSys.targetsMap[bucket], madmin.BucketTarget{Arn: target.arn, TargetBucket: target.bucket})
globalBucketTargetSys.Unlock()
globalBucketTargetSys.hMutex.Lock()
globalBucketTargetSys.hc[client.EndpointURL().Host] = epHealth{Online: true}
globalBucketTargetSys.hMutex.Unlock()
rule := configs[0].Rules[0]
rule.Priority = i + 1
rule.Destination = replication.Destination{ARN: target.arn, Bucket: target.bucket}
cfg.Rules = append(cfg.Rules, rule)
creationStates[target.arn] = replication.Completed
creationInternal += target.arn + "=COMPLETED;"
targets = append(targets, target)
}
if tc.targets == 1 {
cfg.RoleArn = targets[0].arn
}
if !tc.creation {
opts := ObjectOptions{VersionID: version, Versioned: true, DeleteMarker: true, MTime: creationTime, DeleteReplication: ReplicationState{Targets: creationStates, ReplicationStatusInternal: creationInternal, ReplicationTimeStamp: creationTime}}
if tc.replicaSource {
opts.DeleteReplication = ReplicationState{}
opts.SetReplicaStatus(replication.Replica)
}
if _, err := obj.DeleteObject(ctx, bucket, name, opts); err != nil {
t.Fatal(err)
}
}
meta, err := globalBucketMetadataSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
meta.replicationConfig = &cfg
globalBucketMetadataSys.Set(bucket, meta)
newPool := func() *ReplicationPool {
p := &ReplicationPool{ctx: ctx, objLayer: obj, workers: []chan ReplicationWorkerOperation{make(chan ReplicationWorkerOperation, 8)}, stats: stats, mrfSaveCh: make(chan MRFReplicateEntry, 8)}
globalReplicationPool = once.NewSingleton[ReplicationPool]()
globalReplicationPool.Set(p)
return p
}
p := newPool()
receive := func() DeletedObjectReplicationInfo {
select {
case op := <-p.workers[0]:
d, ok := op.(DeletedObjectReplicationInfo)
if !ok {
t.Fatalf("wrong queued operation %T", op)
}
return d
case <-time.After(3 * time.Second):
t.Fatal("no marker task from actual MRF/handler/scanner queue")
return DeletedObjectReplicationInfo{}
}
}
uri := "/" + bucket + "/" + name
if !tc.creation {
uri += "?versionId=" + version
}
req, err := newTestSignedRequestV4(http.MethodDelete, uri, 0, nil, creds.AccessKey, creds.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusNoContent {
t.Fatalf("source DELETE status %d: %s", w.Code, w.Body)
}
deletion := receive()
if tc.creation {
version = deletion.DeleteMarkerVersionID
} else if deletion.VersionID != version || deletion.DeleteMarkerVersionID != "" {
t.Fatalf("current producer emitted noncanonical purge: %+v", deletion)
}
if tc.unrecordedPurge {
// Exercise a task carrying purge state while the disk marker still has
// only creation metadata. Recreate only this isolated fixture marker.
if _, err := obj.DeleteObject(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true}); err != nil {
t.Fatal(err)
}
opts := ObjectOptions{VersionID: version, Versioned: true, DeleteMarker: true, MTime: creationTime, DeleteReplication: ReplicationState{Targets: creationStates, ReplicationStatusInternal: creationInternal, ReplicationTimeStamp: creationTime}}
if _, err := obj.DeleteObject(ctx, bucket, name, opts); err != nil {
t.Fatal(err)
}
}
if tc.legacy {
deletion.VersionID, deletion.DeleteMarkerVersionID = "", version
}
if tc.partial {
deletion.TargetArn = targets[len(targets)-1].arn
}
if tc.invalidMRF {
testReplicationMRFInvalidLookups(ctx, t, obj, p, newPool, deletion)
return
}
var auditStatuses <-chan string
if tc.name == "canonical" {
var stopAudit func()
auditStatuses, stopAudit = replicationTestAudit(ctx, t, bucket)
defer stopAudit()
}
assertAudit := func(want string) {
if auditStatuses == nil {
return
}
select {
case got := <-auditStatuses:
if got != want {
t.Fatalf("replication audit status=%q want %q", got, want)
}
case <-time.After(3 * time.Second):
t.Fatal("missing replication audit event")
}
}
deletion.OpType = replication.HealReplicationType // Exercise operation-status statistics too.
failing := targets[len(targets)-1]
failing.reject.Store(!tc.lostReply)
failing.loseReply.Store(tc.lostReply)
if tc.offline {
globalBucketTargetSys.hMutex.Lock()
globalBucketTargetSys.hc[failing.client.EndpointURL().Host] = epHealth{Online: false}
globalBucketTargetSys.hMutex.Unlock()
}
before, _ := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version})
beforeCreation := before.ReplicationStatusInternal
beforeStamp := before.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp]
beforeReplica := before.UserDefined[ReservedMetadataPrefixLower+ReplicaStatus]
beforeReplicaStamp := before.UserDefined[ReservedMetadataPrefixLower+ReplicaTimestamp]
if !tc.creation && !tc.replicaSource && (len(replicationStatusesMap(beforeCreation)) != tc.targets || beforeStamp == "") {
t.Fatalf("missing seeded creation block: %+v", before)
}
if tc.replicaSource && (beforeReplica != "REPLICA" || beforeReplicaStamp == "") {
t.Fatalf("missing replica block: %+v", before)
}
updateObj := obj
if !tc.creation {
updateObj = markerPurgeUpdateLayer{ObjectLayer: obj, t: t}
}
rounds := 2
if tc.lostReply {
rounds = 1
}
if tc.exhaust {
rounds = mrfRetryLimit + 1
}
for round := 1; round <= rounds; round++ {
callObj := updateObj
if tc.lockFirst && round == 1 {
callObj = markerLockFailureLayer{ObjectLayer: updateObj}
}
result := replicateDelete(ctx, deletion, callObj)
switch {
case tc.lockFirst && round == 1:
if len(result.Targets) != 0 {
t.Fatal("lock failure attempted a target")
}
case tc.creation:
if result.ReplicationStatus() != replication.Failed {
t.Fatalf("creation result: %+v", result)
}
case result.VersionPurgeStatus() != replication.VersionPurgeFailed:
t.Fatalf("purge failure result: %+v", result)
}
assertAudit("FAILED")
if round == 1 && !tc.lockFirst && !tc.creation {
stats.RLock()
failed := stats.Cache[bucket].Stats[failing.arn].FailStats.SinceUptime
stats.RUnlock()
if failed.Count != 1 || failed.Bytes != 0 {
t.Fatalf("purge failure stats=%+v, want count 1 and zero bytes", failed)
}
}
oi, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version})
if !isErrMethodNotAllowed(err) || !oi.DeleteMarker {
t.Fatalf("source marker metadata/405 missing: %+v %v", oi, err)
}
if !tc.creation && (oi.ReplicationStatusInternal != beforeCreation || oi.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] != beforeStamp) {
t.Fatalf("purge rewrote creation block: before=%q/%q after=%q/%q", beforeCreation, beforeStamp, oi.ReplicationStatusInternal, oi.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp])
}
if !tc.creation && (oi.UserDefined[ReservedMetadataPrefixLower+ReplicaStatus] != beforeReplica || oi.UserDefined[ReservedMetadataPrefixLower+ReplicaTimestamp] != beforeReplicaStamp) {
t.Fatal("purge rewrote replica block")
}
if tc.targets == 2 && !tc.partial {
purges := versionPurgeStatusesMap(oi.VersionPurgeStatusInternal)
if purges[targets[0].arn] != replication.VersionPurgeComplete || purges[failing.arn] != replication.VersionPurgeFailed {
t.Fatalf("incorrect persisted target states: %v", purges)
}
if targets[0].deletes.Load() != 1 {
t.Fatalf("successful target resent %d times", targets[0].deletes.Load())
}
}
if tc.partial {
select {
case <-p.mrfSaveCh:
default:
t.Fatal("partial failure missing MRF")
}
dsc := deletion.ReplicationState.ReplicateDecisionStr
deletion.ReplicationState = oi.ReplicationState()
deletion.ReplicationState.ReplicateDecisionStr = dsc
continue
}
if tc.exhaust && round > mrfRetryLimit {
if len(p.mrfSaveCh) != 0 || atomic.LoadUint64(&stats.mrfStats.TotalDroppedCount) != 1 {
t.Fatalf("retry budget not applied: queued=%d drops=%d", len(p.mrfSaveCh), stats.mrfStats.TotalDroppedCount)
}
break
}
var entry MRFReplicateEntry
select {
case entry = <-p.mrfSaveCh:
default:
t.Fatal("failure did not enter MRF")
}
if entry.RetryCount != round || entry.versionID != version {
t.Fatalf("MRF entry lost identity/budget: %+v, round=%d", entry, round)
}
p.saveMRFEntries(ctx, map[string]MRFReplicateEntry{entry.versionID: entry})
record, err := p.loadMRF()
if err != nil || len(record.Entries) != 1 {
t.Fatalf("disk MRF missing: %+v %v", record, err)
}
if got := record.Entries[version]; got.RetryCount != round || got.Object != name || got.Bucket != bucket {
t.Fatalf("disk MRF mismatch: %+v", got)
}
p.saveMRFEntries(ctx, record.Entries) // loadMRF consumes the file; each replay uses a fresh disk record.
p = newPool() // no in-memory entries carried to the replacement pool.
if err := p.queueMRFHeal(); err != nil {
t.Fatal(err)
}
deletion = receive()
if deletion.RetryCount != round {
t.Fatalf("MRF retry count=%d want %d", deletion.RetryCount, round)
}
if !tc.creation && (deletion.VersionID != version || deletion.DeleteMarkerVersionID != "") {
t.Fatalf("MRF emitted wrong purge: %+v", deletion)
}
}
if tc.partial {
return
}
failing.reject.Store(false)
globalBucketTargetSys.hMutex.Lock()
globalBucketTargetSys.hc[failing.client.EndpointURL().Host] = epHealth{Online: true}
globalBucketTargetSys.hMutex.Unlock()
if tc.exhaust {
oi, _ := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version})
QueueReplicationHeal(ctx, bucket, oi, 0) // Separately prove the existing scanner fallback after MRF exhaustion.
deletion = receive()
if deletion.RetryCount != 0 {
t.Fatal("scanner did not start a fresh retry budget")
}
}
result := replicateDelete(ctx, deletion, updateObj)
assertAudit("COMPLETED")
if tc.creation {
if result.ReplicationStatus() != replication.Completed {
t.Fatalf("creation recovery failed: %+v", result)
}
} else if result.VersionPurgeStatus() != replication.VersionPurgeComplete {
t.Fatalf("purge recovery failed: %+v", result)
}
for _, b := range append([]string{bucket}, func() []string {
var b []string
for _, target := range targets {
b = append(b, target.bucket)
}
return b
}()...) {
oi, err := obj.GetObjectInfo(ctx, b, name, ObjectOptions{VersionID: version})
if tc.creation {
if !oi.DeleteMarker || !isErrMethodNotAllowed(err) {
t.Fatalf("creation missing in %s: %+v %v", b, oi, err)
}
} else if !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("purge left marker in %s: %+v %v", b, oi, err)
}
}
if !tc.creation {
// Re-deliver an ambiguous purge directly. An absent marker must stay absent.
duplicate := deletion
duplicate.VersionID, duplicate.DeleteMarkerVersionID = "", version
duplicate.ReplicationState.PurgeTargets = map[string]VersionPurgeStatusType{failing.arn: replication.VersionPurgePending}
duplicate.ReplicationState.VersionPurgeStatusInternal = ""
got := replicateDeleteToTarget(ctx, duplicate, &TargetClient{Client: failing.client, ARN: failing.arn, Bucket: failing.bucket})
if got.VersionPurgeStatus != replication.VersionPurgeComplete {
t.Fatalf("duplicate purge: %+v", got)
}
if oi, err := obj.GetObjectInfo(ctx, failing.bucket, name, ObjectOptions{VersionID: version}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("duplicate recreated marker: %+v %v", oi, err)
}
}
if len(p.mrfSaveCh) != 0 || len(p.workers[0]) != 0 {
t.Fatalf("completion left queued work: mrf=%d worker=%d", len(p.mrfSaveCh), len(p.workers[0]))
}
if !tc.creation && !tc.exhaust && !tc.lostReply {
stats.RLock()
got := stats.Cache[bucket].Stats[failing.arn].ReplicatedCount
size := stats.Cache[bucket].Stats[failing.arn].ReplicatedSize
stats.RUnlock()
if got != 1 || size != 0 {
t.Fatalf("purge completion statistic=%d want 1", got)
}
}
t.Logf("%s: %s recovered; %d target(s), persisted MRF, source/target metadata checked", backend, tc.name, tc.targets)
}
func TestReplicationDeleteQueueFullRetryBudget(t *testing.T) {
stats := NewReplicationStats(t.Context(), nil)
p := &ReplicationPool{ctx: t.Context(), objLayer: &replicationMRFTestObjectLayer{}, workers: []chan ReplicationWorkerOperation{make(chan ReplicationWorkerOperation)}, stats: stats, mrfSaveCh: make(chan MRFReplicateEntry, 1), priority: "slow"}
d := DeletedObjectReplicationInfo{Bucket: "bucket", DeletedObject: DeletedObject{ObjectName: "marker", VersionID: mustGetUUID()}, RetryCount: 2}
p.queueReplicaDeleteTask(d)
if got := <-p.mrfSaveCh; got.RetryCount != 3 {
t.Fatalf("queue-full retry count=%d", got.RetryCount)
}
d.RetryCount = mrfRetryLimit
p.queueReplicaDeleteTask(d)
if len(p.mrfSaveCh) != 0 || atomic.LoadUint64(&stats.mrfStats.TotalDroppedCount) != 1 {
t.Fatal("queue-full retry exceeded budget without a visible drop")
}
}
type markerLockFailureLayer struct{ ObjectLayer }
func (markerLockFailureLayer) NewNSLock(string, ...string) RWLocker { return markerFailedLock{} }
type markerFailedLock struct{ RWLocker }
func (markerFailedLock) GetLock(context.Context, *dynamicTimeout) (LockContext, error) {
return LockContext{}, context.DeadlineExceeded
}
type markerLookupLayer struct {
ObjectLayer
mutate func(ObjectInfo, error) (ObjectInfo, error)
lookedUp chan struct{}
}
func (l markerLookupLayer) GetObjectInfo(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
oi, err := l.ObjectLayer.GetObjectInfo(ctx, bucket, object, opts)
defer close(l.lookedUp)
return l.mutate(oi, err)
}
func testReplicationMRFInvalidLookups(ctx context.Context, t *testing.T, obj ObjectLayer, p *ReplicationPool, newPool func() *ReplicationPool, d DeletedObjectReplicationInfo) {
for _, tc := range []struct {
name string
mutate func(ObjectInfo, error) (ObjectInfo, error)
}{
{"empty-info", func(_ ObjectInfo, e error) (ObjectInfo, error) { return ObjectInfo{}, e }},
{"not-a-marker", func(o ObjectInfo, e error) (ObjectInfo, error) { o.DeleteMarker = false; return o, e }},
{"wrong-version", func(o ObjectInfo, e error) (ObjectInfo, error) { o.VersionID = mustGetUUID(); return o, e }},
{"wrong-bucket", func(o ObjectInfo, e error) (ObjectInfo, error) { o.Bucket = "different-bucket"; return o, e }},
{"wrong-object", func(o ObjectInfo, e error) (ObjectInfo, error) { o.Name = "different-object"; return o, e }},
{"zero-modtime", func(o ObjectInfo, e error) (ObjectInfo, error) { o.ModTime = time.Time{}; return o, e }},
{"missing", func(o ObjectInfo, _ error) (ObjectInfo, error) {
return o, ObjectNotFound{Bucket: o.Bucket, Object: o.Name}
}},
{"read-error", func(o ObjectInfo, _ error) (ObjectInfo, error) { return o, InsufficientReadQuorum{} }},
} {
t.Run(tc.name, func(t *testing.T) {
entry := d.ToMRFEntry()
p.saveMRFEntries(ctx, map[string]MRFReplicateEntry{entry.versionID: entry})
fresh := newPool()
looked := make(chan struct{})
fresh.objLayer = markerLookupLayer{ObjectLayer: obj, mutate: tc.mutate, lookedUp: looked}
if err := fresh.queueMRFHeal(); err != nil {
t.Fatal(err)
}
select {
case <-looked:
case <-time.After(3 * time.Second):
t.Fatal("disk MRF entry was not read")
}
select {
case op := <-fresh.workers[0]:
t.Fatalf("invalid lookup scheduled %T", op)
case <-time.After(100 * time.Millisecond):
}
})
}
}
// Capture the actual internal audit event through the supported webhook sink.
func replicationTestAudit(ctx context.Context, t *testing.T, bucket string) (<-chan string, func()) {
t.Helper()
if len(logger.AuditTargets()) != 0 {
t.Fatal("unexpected pre-existing test audit targets")
}
statuses := make(chan string, 16)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var entry struct {
API struct{ Name, Bucket, Status string }
}
if err := json.NewDecoder(r.Body).Decode(&entry); err != nil {
t.Error(err)
} else if entry.API.Name == ReplicateDeleteAPI && entry.API.Bucket == bucket {
statuses <- entry.API.Status
}
w.WriteHeader(http.StatusOK)
}))
endpoint, err := xnet.ParseHTTPURL(server.URL)
if err != nil {
t.Fatal(err)
}
if errs := logger.UpdateAuditWebhooks(ctx, map[string]loghttp.Config{"r6": {Enabled: true, Name: "r6", Endpoint: endpoint, BatchSize: 1, QueueSize: 128, MaxRetry: 1, RetryIntvl: time.Millisecond, HTTPTimeout: time.Second}}); len(errs) > 0 {
t.Fatal(errs)
}
targets := logger.AuditTargets()
return statuses, func() {
logger.UpdateAuditWebhooks(ctx, nil)
for _, target := range targets {
target.Cancel()
}
server.Close()
}
}
type markerPurgeUpdateLayer struct {
ObjectLayer
t *testing.T
}
func (l markerPurgeUpdateLayer) DeleteObject(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
l.t.Helper()
rs := opts.DeleteReplication
if rs.ReplicationStatusInternal != "" || rs.Targets != nil || rs.ReplicaStatus != "" || !rs.CompositeReplicationStatus().Empty() {
l.t.Fatalf("purge has a nonempty creation update: %+v", rs)
}
if rs.CompositeVersionPurgeStatus().Empty() {
l.t.Fatal("purge update lost its purge status")
}
return l.ObjectLayer.DeleteObject(ctx, bucket, object, opts)
}
+215
View File
@@ -0,0 +1,215 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"fmt"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
)
// Purge uses the version-delete wire form regardless of the in-memory shape.
// Creation status, purge status and successful resync markers are independent.
func TestReplicateDeleteOperationExits(t *testing.T) {
for _, shape := range []string{"marker-creation", "legacy-marker-purge", "marker-purge", "object-purge"} {
purge := shape != "marker-creation"
for _, tc := range []struct {
name string
creation replication.StatusType
priorPurge VersionPurgeStatusType
headCode, deleteCode int
headError string
offline, resync bool
}{
{name: "pending-success", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 404, deleteCode: 204},
{name: "completed-creation", creation: replication.Completed, priorPurge: replication.VersionPurgePending, headCode: 405, deleteCode: 204},
{name: "existing-marker", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 405, deleteCode: 204},
// Use the quorum S3 code without 503, which is classified as backend-down first.
{name: "head-read-quorum", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 400, headError: "SlowDownRead", deleteCode: 204},
{name: "replica-creation-status", creation: replication.Replica, priorPurge: replication.VersionPurgeFailed, headCode: 405, deleteCode: 204},
{name: "head-unavailable", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 503, deleteCode: 204},
{name: "head-forbidden", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 403, deleteCode: 204},
{name: "delete-forbidden", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 404, deleteCode: 403},
{name: "delete-method-rejected", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 404, deleteCode: 405},
{name: "delete-unavailable", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 404, deleteCode: 503},
{name: "offline", creation: replication.Pending, priorPurge: replication.VersionPurgePending, headCode: 404, deleteCode: 204, offline: true},
{name: "retry-failed", creation: replication.Completed, priorPurge: replication.VersionPurgeFailed, headCode: 404, deleteCode: 204},
{name: "purge-complete", creation: replication.Pending, priorPurge: replication.VersionPurgeComplete, headCode: 404, deleteCode: 403},
{name: "resync-success", creation: replication.Completed, priorPurge: replication.VersionPurgePending, headCode: 405, deleteCode: 204, resync: true},
{name: "resync-already-purged", creation: replication.Pending, priorPurge: replication.VersionPurgeComplete, headCode: 404, deleteCode: 403, resync: true},
{name: "resync-failure", creation: replication.Completed, priorPurge: replication.VersionPurgePending, headCode: 404, deleteCode: 403, resync: true},
} {
t.Run(shape+"/"+tc.name, func(t *testing.T) {
version := mustGetUUID()
var heads, deletes atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("versionId") != version {
t.Errorf("wrong request version: %s", r.URL)
}
switch r.Method {
case http.MethodHead:
heads.Add(1)
w.Header().Set(xhttp.AmzVersionID, version)
w.Header().Set(xhttp.LastModified, time.Now().UTC().Format(http.TimeFormat))
if tc.headCode == 405 {
w.Header().Set(xhttp.AmzDeleteMarker, "true")
}
if tc.headError != "" {
w.Header().Set("x-minio-error-code", tc.headError)
}
w.WriteHeader(tc.headCode)
case http.MethodDelete:
deletes.Add(1)
if got := r.Header.Get(xhttp.MinIOSourceDeleteMarker) == "true"; got == purge {
t.Errorf("source delete-marker header=%v, purge=%v", got, purge)
}
w.WriteHeader(tc.deleteCode)
if tc.deleteCode != 204 {
fmt.Fprintf(w, `<Error><Code>%s</Code><Message>injected rejection</Message></Error>`, map[int]string{403: "AccessDenied", 405: "MethodNotAllowed", 503: "ServiceUnavailable"}[tc.deleteCode])
}
default:
t.Errorf("unexpected method %s", r.Method)
}
}))
defer server.Close()
client, err := minio.New(strings.TrimPrefix(server.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
old := globalBucketTargetSys
globalBucketTargetSys = &BucketTargetSys{hc: map[string]epHealth{client.EndpointURL().Host: {Online: !tc.offline}}}
defer func() { globalBucketTargetSys = old }()
d := DeletedObjectReplicationInfo{Bucket: "source", DeletedObject: DeletedObject{ObjectName: "marker", DeleteMarker: shape != "object-purge"}}
if shape == "marker-creation" || shape == "legacy-marker-purge" {
d.DeleteMarkerVersionID = version
} else {
d.VersionID = version
}
d.ReplicationState.Targets = map[string]replication.StatusType{"arn1": tc.creation}
d.ReplicationState.ResetStatusesMap = map[string]string{"arn1": "previous-reset"}
if purge {
d.ReplicationState.PurgeTargets = map[string]VersionPurgeStatusType{"arn1": tc.priorPurge}
}
if tc.resync {
d.OpType = replication.ExistingObjectReplicationType
}
got := replicateDeleteToTarget(t.Context(), d, &TargetClient{Client: client, ARN: "arn1", Bucket: "target", ResetID: "current-reset"})
var wantCreation replication.StatusType
var wantPurge VersionPurgeStatusType
var wantHeads, wantDeletes int32
success := false
if purge {
wantCreation = ""
wantPurge = replication.VersionPurgeComplete
switch {
case tc.priorPurge == replication.VersionPurgeComplete:
success = true
case tc.offline:
wantPurge = replication.VersionPurgeFailed
default:
wantDeletes = 1
success = tc.deleteCode == 204
if !success {
wantPurge = replication.VersionPurgeFailed
}
}
} else {
switch {
case tc.creation == replication.Completed && !tc.resync:
success = true
case tc.offline:
default:
wantHeads = 1
switch tc.headCode {
case 405:
success = true
case 403, 503:
default:
wantDeletes = 1
success = tc.deleteCode == 204
}
}
if success {
wantCreation = replication.Completed
} else {
wantCreation = replication.Failed
}
}
if got.PrevReplicationStatus != tc.creation {
t.Error("previous creation state changed")
}
if got.ReplicationStatus != wantCreation || got.VersionPurgeStatus != wantPurge {
t.Errorf("status=%+v, want creation=%s purge=%s", got, wantCreation, wantPurge)
}
if heads.Load() != wantHeads || deletes.Load() != wantDeletes {
t.Errorf("HEAD/DELETE=%d/%d, want %d/%d", heads.Load(), deletes.Load(), wantHeads, wantDeletes)
}
if (got.Err != nil) == success {
t.Errorf("success=%v, error=%v", success, got.Err)
}
if tc.resync && success {
if !strings.HasSuffix(got.ResyncTimestamp, ";current-reset") {
t.Errorf("successful resync missing reset: %q", got.ResyncTimestamp)
}
} else if got.ResyncTimestamp != "previous-reset" {
t.Errorf("unexpected reset: %q", got.ResyncTimestamp)
}
})
}
}
}
func TestReplicateDeletePurgeMissingTargetState(t *testing.T) {
// Classification belongs to the operation, even if this target has no
// previous purge entry (another target supplies the tracked purge state).
var deletes atomic.Int32
version := mustGetUUID()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodDelete || r.URL.Query().Get("versionId") != version || r.Header.Get(xhttp.MinIOSourceDeleteMarker) == "true" {
t.Errorf("wrong purge request: %s %s %v", r.Method, r.URL, r.Header)
}
deletes.Add(1)
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
client, err := minio.New(strings.TrimPrefix(server.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
old := globalBucketTargetSys
globalBucketTargetSys = &BucketTargetSys{hc: map[string]epHealth{client.EndpointURL().Host: {Online: true}}}
defer func() { globalBucketTargetSys = old }()
d := DeletedObjectReplicationInfo{Bucket: "source", DeletedObject: DeletedObject{ObjectName: "marker", DeleteMarker: true, DeleteMarkerVersionID: version, ReplicationState: ReplicationState{Targets: map[string]replication.StatusType{"arn1": replication.Completed}, PurgeTargets: map[string]VersionPurgeStatusType{"arn2": replication.VersionPurgePending}}}}
got := replicateDeleteToTarget(t.Context(), d, &TargetClient{Client: client, ARN: "arn1", Bucket: "target", ResetID: "reset"})
if deletes.Load() != 1 || got.VersionPurgeStatus != replication.VersionPurgeComplete || got.ReplicationStatus != "" {
t.Fatalf("operation misclassified: %+v, deletes=%d", got, deletes.Load())
}
got.ResyncTimestamp = "resync;reset"
state := getReplicationState(replicatedInfos{Targets: []replicatedTargetInfo{got}}, ReplicationState{}, "")
if state.ResetStatusesMap[targetResetHeader("arn1")] != got.ResyncTimestamp {
t.Fatal("resync timestamp not preserved with nil reset map")
}
}
+264
View File
@@ -0,0 +1,264 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/once"
)
type staleCreationCase struct {
name string
purged, pendingPurge, unreadable bool
}
// A queued delete-marker creation must be checked against the source marker
// under the replication lock before it is sent. Between queueing (DELETE
// handler, GET/HEAD/LIST heal, scanner, MRF) and sending, a user purge of the
// same marker can complete and reach the targets; the stale creation would
// then recreate the marker there.
func TestReplicationDeleteMarkerCreationRevalidated(t *testing.T) {
for _, tc := range []staleCreationCase{
{name: "current"},
{name: "purged", purged: true},
{name: "purge-pending", pendingPurge: true},
{name: "unreadable", unreadable: true},
} {
t.Run(tc.name, func(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, endpoints: []string{"DeleteObject"}, objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, creds auth.Credentials, t *testing.T) {
testReplicationDeleteMarkerCreationRevalidated(t, obj, backend, bucket, router, creds, tc)
}})
})
}
}
type staleCreationTarget struct {
arn, bucket string
creations atomic.Int32
purges atomic.Int32
}
func testReplicationDeleteMarkerCreationRevalidated(t *testing.T, obj ObjectLayer, backend, bucket string, router http.Handler, creds auth.Credentials, tc staleCreationCase) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
defer replicationTestCapacity(obj)()
stats := NewReplicationStats(ctx, nil)
oldStats := globalReplicationStats.Swap(stats)
defer globalReplicationStats.Store(oldStats)
oldPool := globalReplicationPool
defer func() { globalReplicationPool = oldPool }()
const name = "marker"
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
if _, err := obj.PutObject(ctx, bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true}); err != nil {
t.Fatal(err)
}
target := &staleCreationTarget{arn: "arn:minio:replication::" + mustGetUUID() + ":bucket", bucket: getRandomBucketName()}
if err := obj.MakeBucket(ctx, target.bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
t.Fatal(err)
}
if _, err := obj.PutObject(ctx, target.bucket, name, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{Versioned: true}); err != nil {
t.Fatal(err)
}
// The remote behaves like a real receiver: a marker lookup and a
// replicated DELETE applied to the target bucket on the same fixture.
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
opts := ObjectOptions{VersionID: r.URL.Query().Get("versionId"), Versioned: true}
switch r.Method {
case http.MethodHead:
oi, err := obj.GetObjectInfo(r.Context(), target.bucket, name, opts)
if oi.DeleteMarker {
w.Header().Set(xhttp.AmzDeleteMarker, "true")
w.Header().Set(xhttp.AmzVersionID, oi.VersionID)
}
if err != nil {
writeErrorResponseHeadersOnly(w, toAPIError(r.Context(), err))
return
}
w.WriteHeader(http.StatusOK)
case http.MethodDelete:
opts.DeleteMarker = r.Header.Get(xhttp.MinIOSourceDeleteMarker) == "true"
if opts.DeleteMarker {
target.creations.Add(1)
} else {
target.purges.Add(1)
}
opts.ReplicationRequest = true
opts.SetReplicaStatus(replication.Replica)
if _, err := obj.DeleteObject(r.Context(), target.bucket, name, opts); err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
writeErrorResponse(r.Context(), w, toAPIError(r.Context(), err), r.URL)
return
}
w.WriteHeader(http.StatusNoContent)
default:
t.Errorf("unexpected remote method %s", r.Method)
}
}))
defer remote.Close()
client, err := minio.New(strings.TrimPrefix(remote.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
globalBucketTargetSys.Lock()
globalBucketTargetSys.arnRemotesMap[target.arn] = arnTarget{Client: &TargetClient{Client: client, ARN: target.arn, Bucket: target.bucket}, lastRefresh: UTCNow()}
globalBucketTargetSys.targetsMap[bucket] = append(globalBucketTargetSys.targetsMap[bucket], madmin.BucketTarget{Arn: target.arn, TargetBucket: target.bucket})
globalBucketTargetSys.Unlock()
globalBucketTargetSys.hMutex.Lock()
globalBucketTargetSys.hc[client.EndpointURL().Host] = epHealth{Online: true}
globalBucketTargetSys.hMutex.Unlock()
rule := configs[0].Rules[0]
rule.Destination = replication.Destination{ARN: target.arn, Bucket: target.bucket}
cfg := replication.Config{RoleArn: target.arn, Rules: []replication.Rule{rule}}
meta, err := globalBucketMetadataSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
meta.replicationConfig = &cfg
globalBucketMetadataSys.Set(bucket, meta)
p := &ReplicationPool{ctx: ctx, objLayer: obj, workers: []chan ReplicationWorkerOperation{make(chan ReplicationWorkerOperation, 8)}, stats: stats, mrfSaveCh: make(chan MRFReplicateEntry, 8)}
globalReplicationPool = once.NewSingleton[ReplicationPool]()
globalReplicationPool.Set(p)
// The real DELETE handler creates the pending marker and queues the
// creation task that a worker would later run.
req, err := newTestSignedRequestV4(http.MethodDelete, "/"+bucket+"/"+name, 0, nil, creds.AccessKey, creds.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusNoContent {
t.Fatalf("source DELETE status %d: %s", w.Code, w.Body)
}
var creation DeletedObjectReplicationInfo
select {
case op := <-p.workers[0]:
creation = op.(DeletedObjectReplicationInfo)
case <-time.After(3 * time.Second):
t.Fatal("handler queued no creation task")
}
version := creation.DeleteMarkerVersionID
if version == "" || creation.VersionID != "" || !creation.DeleteMarker {
t.Fatalf("handler queued a non-creation task: %+v", creation)
}
before, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true})
if !isErrMethodNotAllowed(err) || !before.DeleteMarker || before.ReplicationStatus != replication.Pending {
t.Fatalf("source marker not pending: %+v %v", before, err)
}
// Meanwhile the user purges the marker through another path. The purge is
// either complete (version gone) or still pending on the source.
switch {
case tc.purged:
if _, err := obj.DeleteObject(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true}); err != nil {
t.Fatal(err)
}
if _, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true}); !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
t.Fatalf("source purge left the marker: %v", err)
}
case tc.pendingPurge:
opts := ObjectOptions{VersionID: version, Versioned: true, DeleteReplication: ReplicationState{
ReplicateDecisionStr: creation.ReplicationState.ReplicateDecisionStr,
VersionPurgeStatusInternal: target.arn + "=PENDING;",
PurgeTargets: map[string]VersionPurgeStatusType{target.arn: replication.VersionPurgePending},
}}
if _, err := obj.DeleteObject(ctx, bucket, name, opts); err != nil {
t.Fatal(err)
}
oi, err := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true})
if !isErrMethodNotAllowed(err) || !oi.DeleteMarker || oi.VersionPurgeStatus != replication.VersionPurgePending {
t.Fatalf("source marker not pending purge: %+v %v", oi, err)
}
}
source := obj
if tc.unreadable {
source = staleLookupLayer{ObjectLayer: obj}
}
pending, _ := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true})
result := replicateDelete(ctx, creation, source)
after, aerr := obj.GetObjectInfo(ctx, bucket, name, ObjectOptions{VersionID: version, Versioned: true})
_, terr := obj.GetObjectInfo(ctx, target.bucket, name, ObjectOptions{VersionID: version, Versioned: true})
targetHasMarker := isErrMethodNotAllowed(terr)
switch {
case tc.purged, tc.pendingPurge, tc.unreadable:
if len(result.Targets) != 0 || target.creations.Load() != 0 || target.purges.Load() != 0 {
t.Fatalf("%s: stale creation was sent: result=%+v creations=%d purges=%d", tc.name, result, target.creations.Load(), target.purges.Load())
}
if targetHasMarker {
t.Fatalf("%s: target marker recreated from the stale creation", tc.name)
}
if tc.purged {
if !isErrVersionNotFound(aerr) && !isErrObjectNotFound(aerr) {
t.Fatalf("stale creation resurrected the source marker: %+v %v", after, aerr)
}
} else if !isErrMethodNotAllowed(aerr) || !after.DeleteMarker ||
after.ReplicationStatusInternal != pending.ReplicationStatusInternal ||
after.VersionPurgeStatusInternal != pending.VersionPurgeStatusInternal ||
after.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] != pending.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] {
t.Fatalf("skipped creation rewrote the source marker: before=%+v after=%+v %v", pending, after, aerr)
}
if tc.unreadable {
select {
case entry := <-p.mrfSaveCh:
if entry.versionID != version || entry.RetryCount != 1 || entry.Bucket != bucket || entry.Object != name {
t.Fatalf("unverified creation queued wrong MRF entry: %+v", entry)
}
default:
t.Fatal("unverified source read did not queue a retry")
}
}
if len(p.mrfSaveCh) != 0 {
t.Fatalf("%s: unexpected MRF entries queued: %d", tc.name, len(p.mrfSaveCh))
}
default:
if result.ReplicationStatus() != replication.Completed || target.creations.Load() != 1 || !targetHasMarker {
t.Fatalf("current creation not replicated: result=%+v creations=%d targetMarker=%v", result, target.creations.Load(), targetHasMarker)
}
if !isErrMethodNotAllowed(aerr) || !after.DeleteMarker || replicationStatusesMap(after.ReplicationStatusInternal)[target.arn] != replication.Completed {
t.Fatalf("source creation state not completed: %+v %v", after, aerr)
}
if len(p.mrfSaveCh) != 0 {
t.Fatal("completed creation queued MRF work")
}
}
t.Logf("%s: %s checked; creations=%d purges=%d", backend, tc.name, target.creations.Load(), target.purges.Load())
}
// staleLookupLayer cannot confirm the source marker: the read fails without
// saying whether the version is present.
type staleLookupLayer struct{ ObjectLayer }
func (staleLookupLayer) GetObjectInfo(context.Context, string, string, ObjectOptions) (ObjectInfo, error) {
return ObjectInfo{}, InsufficientReadQuorum{}
}
+121
View File
@@ -0,0 +1,121 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"maps"
"net/http"
"net/textproto"
"reflect"
"strings"
"testing"
xhttp "github.com/minio/minio/internal/http"
)
func TestExtractReplicationMetadataPreservesNormalizedMetadata(t *testing.T) {
for _, tc := range []struct {
name string
wire []string
want string
}{
{name: "absent"},
{name: "transport-only", wire: []string{"aws-chunked"}},
{name: "mixed", wire: []string{"aws-chunked,gzip"}, want: "gzip"},
{name: "gzip", wire: []string{"gzip"}, want: "gzip"},
{name: "transport-last", wire: []string{"gzip,aws-chunked"}, want: "gzip"},
{name: "multiple-values", wire: []string{"aws-chunked", "gzip"}, want: "gzip"},
// Preserve the existing exact-token grammar; whitespace is not normalized here.
{name: "space-before-gzip", wire: []string{"aws-chunked, gzip"}, want: " gzip"},
{name: "space-before-transport", wire: []string{"gzip, aws-chunked"}, want: "gzip, aws-chunked"},
} {
for _, lowercase := range []bool{false, true} {
name := tc.name + "/canonical"
if lowercase {
name = tc.name + "/lowercase"
}
t.Run(name, func(t *testing.T) {
header := http.Header{
"Content-Type": []string{"application/octet-stream"},
"X-Amz-Meta-Source": []string{"raw"},
"X-Minio-Replication-Server-Side-Encryption-Sealed-Key": []string{"sealed-key"},
"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
"X-Minio-Replication-Server-Side-Encryption-Iv": []string{"iv"},
"X-Minio-Replication-Encrypted-Multipart": []string{""},
"X-Minio-Replication-Actual-Object-Size": []string{"1"},
ReplicationSsecChecksumHeader: []string{"checksum"},
xhttp.AmzMetaUnencryptedContentLength: []string{"injected-length"},
xhttp.AmzMetaUnencryptedContentMD5: []string{"injected-md5"},
}
if tc.wire != nil {
header[xhttp.ContentEncoding] = tc.wire
}
if lowercase {
h := make(http.Header, len(header))
for k, v := range header {
h[strings.ToLower(k)] = v
}
header = h
}
metadata, err := extractMetadata(t.Context(), textproto.MIMEHeader(header))
if err != nil {
t.Fatal(err)
}
if metadata["content-encoding"] != tc.want {
t.Fatalf("ordinary encoding=%q want=%q", metadata["content-encoding"], tc.want)
}
for _, internal := range replicationToInternalHeaders {
if _, ok := metadata[internal]; ok {
t.Fatalf("ordinary request accepted internal field %s", internal)
}
}
// Callers own ordinary metadata and may transform it after extraction.
metadata["content-type"] = "application/wasm"
for k := range metadata {
if strings.EqualFold(k, "x-amz-meta-source") {
metadata[k] = "caller"
}
}
want := maps.Clone(metadata)
maps.Copy(want, map[string]string{
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key": "sealed-key",
"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "DAREv2-HMAC-SHA256",
"X-Minio-Internal-Server-Side-Encryption-Iv": "iv",
"X-Minio-Internal-Encrypted-Multipart": "",
"X-Minio-Internal-Actual-Object-Size": "1",
ReplicationSsecChecksumHeader: "checksum",
})
for range 2 {
if err := extractReplicationMetadataFromMime(t.Context(), textproto.MIMEHeader(header), metadata); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(metadata, want) {
t.Errorf("restoration changed normalized metadata: got %#v want %#v", metadata, want)
}
}
if tc.want == "" {
if _, present := metadata["content-encoding"]; present {
t.Error("transport-only content-encoding key restored")
}
}
for _, key := range []string{xhttp.AmzMetaUnencryptedContentLength, xhttp.AmzMetaUnencryptedContentMD5} {
if _, present := caseInsensitiveMap(metadata).Lookup(key); present {
t.Errorf("redacted metadata restored: %s", key)
}
}
})
}
}
}
func TestExtractReplicationMetadataNilHeader(t *testing.T) {
metadata := map[string]string{"content-type": "application/wasm"}
want := maps.Clone(metadata)
if err := extractReplicationMetadataFromMime(t.Context(), nil, metadata); err != errInvalidArgument {
t.Fatalf("nil header: got %v want %v", err, errInvalidArgument)
}
if !reflect.DeepEqual(metadata, want) {
t.Fatalf("nil input changed metadata: %#v", metadata)
}
}
+635
View File
@@ -0,0 +1,635 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"encoding/xml"
"fmt"
"maps"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/kms"
)
const r5TagStamp = ReservedMetadataPrefixLower + TaggingTimestamp
func r5Capacity(z *erasureServerPools) func() {
var restores []func()
for _, pool := range z.serverPools {
pool.erasureDisksMu.Lock()
for _, set := range pool.sets {
old := pool.erasureDisks[set.setIndex]
disks := append([]StorageAPI(nil), old...)
for i := range disks {
disks[i] = tagTestCapacityDisk{StorageAPI: disks[i]}
}
// GetDisks copies this list under the same mutex. Keep its function
// stable while background IAM scans are using the fixture.
pool.erasureDisks[set.setIndex] = disks
restores = append(restores, func() {
pool.erasureDisksMu.Lock()
pool.erasureDisks[set.setIndex] = old
pool.erasureDisksMu.Unlock()
})
}
pool.erasureDisksMu.Unlock()
}
return func() {
for _, restore := range restores {
restore()
}
}
}
func TestAPITaggingCapacityConcurrentIAM(t *testing.T) {
z, _ := consistencyPools(t)
if _, _, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
iam := globalIAMSys
started, finished := make(chan struct{}), make(chan error, 1)
go func() {
close(started)
for range 20 {
if err := iam.Load(ctx, false); err != nil {
finished <- err
return
}
}
finished <- nil
}()
<-started
for range 5000 {
r5Capacity(z)()
}
if err := <-finished; err != nil {
t.Fatal(err)
}
}
func r5Request(t *testing.T, router http.Handler, cred auth.Credentials, method, path, body string, headers map[string]string) *httptest.ResponseRecorder {
t.Helper()
r, err := newTestSignedRequestV4(method, path, int64(len(body)), strings.NewReader(body), cred.AccessKey, cred.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
router.ServeHTTP(w, r)
return w
}
func r5Stored(t *testing.T, obj interface {
GetObjectInfo(context.Context, string, string, ObjectOptions) (ObjectInfo, error)
}, bucket, name, vid, wantTags, wantStamp string,
) ObjectInfo {
t.Helper()
oi, err := obj.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{VersionID: vid})
if err != nil {
t.Fatal(err)
}
if oi.UserTags != wantTags || oi.UserDefined[r5TagStamp] != wantStamp {
t.Fatalf("%s(%s): tags=%q stamp=%q, want %q %q", name, vid, oi.UserTags, oi.UserDefined[r5TagStamp], wantTags, wantStamp)
}
return oi
}
// The request bytes are signed and enter the real API and storage implementation.
// Equal/stale retransmits may retain the existing 412 duplicate response.
func r5Receive(t *testing.T, obj ObjectLayer, router http.Handler, cred auth.Credentials, bucket, operation string, source ObjectInfo, stamp string, afterInit func()) {
t.Helper()
opts, _, err := putReplicationOpts(t.Context(), "", source)
if err != nil {
t.Fatal(err)
}
if operation == "multipart" {
opts.Internal.SourceMTime = time.Time{}
}
headers := make(map[string]string)
for k, vs := range opts.Header() {
if len(vs) > 0 {
headers[k] = vs[0]
}
}
if stamp == "" {
delete(headers, http.CanonicalHeaderKey(xhttp.MinIOSourceTaggingTimestamp))
delete(headers, xhttp.MinIOSourceTaggingTimestamp)
} else {
headers[http.CanonicalHeaderKey(xhttp.MinIOSourceTaggingTimestamp)] = stamp
}
path := "/" + bucket + "/" + source.Name + "?versionId=" + source.VersionID
var w *httptest.ResponseRecorder
switch operation {
case "copy", "copy-default":
maps.Copy(headers, getCopyObjMetadata(source, ""))
headers[xhttp.AmzCopySource] = "/" + bucket + "/" + source.Name + "?versionId=" + source.VersionID
if operation == "copy" {
headers[xhttp.AmzMetadataDirective] = "REPLACE"
}
w = r5Request(t, router, cred, http.MethodPut, path, "", headers)
case "put":
w = r5Request(t, router, cred, http.MethodPut, path, "data", headers)
case "multipart":
w = r5Request(t, router, cred, http.MethodPost, path+"&uploads", "", headers)
if w.Code == http.StatusPreconditionFailed {
return
}
if w.Code != http.StatusOK {
t.Fatalf("init: %d %s", w.Code, w.Body.String())
}
var init struct {
UploadID string `xml:"UploadId"`
}
if err := xml.Unmarshal(w.Body.Bytes(), &init); err != nil || init.UploadID == "" {
t.Fatalf("init XML: %v %s", err, w.Body.String())
}
mi, err := obj.GetMultipartInfo(t.Context(), bucket, source.Name, init.UploadID, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if stamp != "" && mi.UserDefined[r5TagStamp] != stamp {
t.Fatalf("upload persisted stamp=%q, want %q", mi.UserDefined[r5TagStamp], stamp)
}
partPath := "/" + bucket + "/" + source.Name + "?uploadId=" + url.QueryEscape(init.UploadID)
ph := map[string]string{xhttp.MinIOSourceReplicationRequest: "true"}
part := r5Request(t, router, cred, http.MethodPut, partPath+"&partNumber=1", "data", ph)
if part.Code != http.StatusOK {
t.Fatalf("part: %d %s", part.Code, part.Body.String())
}
if afterInit != nil {
afterInit()
}
body := "<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>" + canonicalizeETag(part.Header()[xhttp.ETag][0]) + "</ETag></Part></CompleteMultipartUpload>"
ph[xhttp.MinIOSourceMTime] = source.ModTime.Format(time.RFC3339Nano)
ph[xhttp.MinIOSourceETag] = source.ETag
w = r5Request(t, router, cred, http.MethodPost, partPath, body, ph)
}
if w.Code != http.StatusOK && w.Code != http.StatusPreconditionFailed {
t.Fatalf("%s: %d %s", operation, w.Code, w.Body.String())
}
}
func TestAPITaggingReplicationOrdering(t *testing.T) { r5APIOrdering(t, false) }
func TestAPITaggingReplicationOrderingKMS(t *testing.T) { r5APIOrdering(t, true) }
func r5APIOrdering(t *testing.T, encrypted bool) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
defer r5Capacity(obj.(*erasureServerPools))()
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
if encrypted {
prev := GlobalKMS
GlobalKMS = kms.NewStub("r5-tag-order")
defer func() { GlobalKMS = prev }()
sse := []byte(`<ServerSideEncryptionConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>aws:kms</SSEAlgorithm><KMSMasterKeyID>r5-tag-order</KMSMasterKeyID></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>`)
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketSSEConfig, sse); err != nil {
t.Fatal(err)
}
}
base := time.Now().UTC().Add(-5 * time.Hour)
for _, op := range []string{"copy", "copy-default", "put", "multipart"} {
for _, version := range []string{"uuid", "null"} {
t.Run(instance+"/"+op+"/"+version, func(t *testing.T) {
name := op + "-" + version
vid := mustGetUUID()
if version == "null" {
vid = nullVersionID
}
original, err := obj.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{Versioned: true, VersionID: vid, UserDefined: map[string]string{xhttp.AmzObjectTagging: "key=original", r5TagStamp: base.Format(time.RFC3339Nano)}})
if err != nil {
t.Fatal(err)
}
if op == "multipart" {
// The production sender uses multipart only for multipart
// sources. Preserve a real multipart ETag and part layout.
metadata := maps.Clone(original.UserDefined)
metadata[xhttp.AmzObjectTagging] = original.UserTags
mp, err := obj.NewMultipartUpload(t.Context(), bucket, name, ObjectOptions{Versioned: true, VersionID: vid, UserDefined: metadata})
if err != nil {
t.Fatal(err)
}
part, err := obj.PutObjectPart(t.Context(), bucket, name, mp.UploadID, 1, mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
original, err = obj.CompleteMultipartUpload(t.Context(), bucket, name, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{Versioned: true, ReplicationRequest: true})
if err != nil {
t.Fatal(err)
}
}
// A later unrelated version must not contribute tags to an explicitly addressed UUID/null version.
latest, err := obj.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, strings.NewReader("other"), 5, "", ""), ObjectOptions{Versioned: true, UserDefined: map[string]string{xhttp.AmzObjectTagging: "key=latest", r5TagStamp: base.Add(10 * time.Hour).Format(time.RFC3339Nano)}})
if err != nil {
t.Fatal(err)
}
for _, event := range []struct {
name, tags string
hours int
wantTags string
wantHours int
}{
{"delete", "", 3, "", 3}, {"stale", "key=stale", 2, "", 3}, {"equal-conflict", "key=conflict", 3, "", 3}, {"newer", "key=new", 4, "key=new", 4},
} {
t.Run(event.name, func(t *testing.T) {
source := original
source.VersionID = vid
source.UserDefined = maps.Clone(original.UserDefined)
source.UserTags = event.tags
stamp := base.Add(time.Duration(event.hours) * time.Hour).Format(time.RFC3339Nano)
source.UserDefined[r5TagStamp] = stamp
r5Receive(t, obj, router, cred, bucket, op, source, stamp, nil)
r5Stored(t, obj, bucket, name, vid, event.wantTags, base.Add(time.Duration(event.wantHours)*time.Hour).Format(time.RFC3339Nano))
r5Stored(t, obj, bucket, name, latest.VersionID, "key=latest", base.Add(10*time.Hour).Format(time.RFC3339Nano))
})
}
source := original
source.VersionID = vid
source.UserTags = "key=unversioned-event"
r5Receive(t, obj, router, cred, bucket, op, source, "", nil)
r5Stored(t, obj, bucket, name, vid, "key=new", base.Add(4*time.Hour).Format(time.RFC3339Nano))
get := r5Request(t, router, cred, http.MethodGet, "/"+bucket+"/"+name+"?versionId="+vid, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "data" {
t.Fatalf("plaintext GET: %d %q", get.Code, get.Body.String())
}
})
}
}
}})
}
func TestAPITaggingMultipartCommitRechecksRevision(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
defer r5Capacity(obj.(*erasureServerPools))()
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
base := time.Now().UTC().Add(-time.Hour)
source := ObjectInfo{Name: "commit-recheck", VersionID: mustGetUUID(), ModTime: base, UserTags: "key=incoming", UserDefined: map[string]string{r5TagStamp: base.Format(time.RFC3339Nano)}}
later := base.Add(time.Minute).Format(time.RFC3339Nano)
r5Receive(t, obj, router, cred, bucket, "multipart", source, base.Format(time.RFC3339Nano), func() {
_, err := obj.PutObject(t.Context(), bucket, source.Name, mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{Versioned: true, VersionID: source.VersionID, MTime: base, UserDefined: map[string]string{r5TagStamp: later}})
if err != nil {
t.Fatal(err)
}
})
r5Stored(t, obj, bucket, source.Name, source.VersionID, "", later)
t.Logf("%s: deletion committed between initiation and completion survived", instance)
}})
}
func TestAPILocalTaggingAlwaysAdvancesRevision(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
defer r5Capacity(obj.(*erasureServerPools))()
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
oi, err := obj.PutObject(t.Context(), bucket, "local-tags", mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
path := "/" + bucket + "/local-tags?tagging&versionId=" + oi.VersionID
for n, method := range []string{http.MethodPut, http.MethodDelete, http.MethodDelete, http.MethodPut} {
body := ""
want := ""
status := http.StatusNoContent
if method == http.MethodPut {
status = http.StatusOK
body = "<Tagging><TagSet/></Tagging>"
if n == 0 {
want = "key=local"
body = "<Tagging><TagSet><Tag><Key>key</Key><Value>local</Value></Tag></TagSet></Tagging>"
}
}
before := time.Now().UTC()
w := r5Request(t, router, cred, method, path, body, nil)
if w.Code != status {
t.Fatalf("%s: %d %s", method, w.Code, w.Body.String())
}
now, err := obj.GetObjectInfo(t.Context(), bucket, "local-tags", ObjectOptions{VersionID: oi.VersionID})
if err != nil {
t.Fatal(err)
}
stamp, err := time.Parse(time.RFC3339Nano, now.UserDefined[r5TagStamp])
if err != nil || stamp.Before(before) || now.UserTags != want {
t.Fatalf("local %s: tags=%q timestamp=%q err=%v", method, now.UserTags, now.UserDefined[r5TagStamp], err)
}
if !now.ModTime.Equal(oi.ModTime) {
t.Fatal("tagging changed the object's data modification time")
}
t.Logf("%s mutation %d persisted tags=%q timestamp=%s", instance, n, now.UserTags, stamp)
}
// Ordinary COPY with empty REPLACE has the same local-deletion semantics.
before := time.Now().UTC()
headers := map[string]string{xhttp.AmzCopySource: "/" + bucket + "/local-tags?versionId=" + oi.VersionID, xhttp.AmzMetadataDirective: "REPLACE", xhttp.AmzTagDirective: "REPLACE"}
w := r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/copied-empty", "", headers)
if w.Code != http.StatusOK {
t.Fatalf("copy: %d %s", w.Code, w.Body.String())
}
copied, err := obj.GetObjectInfo(t.Context(), bucket, "copied-empty", ObjectOptions{})
if err != nil {
t.Fatal(err)
}
stamp, err := time.Parse(time.RFC3339Nano, copied.UserDefined[r5TagStamp])
if err != nil || stamp.Before(before) || copied.UserTags != "" {
t.Fatalf("local COPY: %v %+v", err, copied)
}
}})
}
func TestTaggingTimestampWire(t *testing.T) {
now := time.Now().UTC()
for _, tag := range []string{"", "key=value"} {
for _, stamp := range []string{"", now.Format(time.RFC3339Nano), "invalid"} {
t.Run(fmt.Sprintf("%s/%s", tag, stamp), func(t *testing.T) {
source := ObjectInfo{ModTime: now.Add(-time.Hour), UserTags: tag, UserDefined: map[string]string{}}
if stamp != "" {
source.UserDefined[r5TagStamp] = stamp
}
opts, _, err := putReplicationOpts(t.Context(), "", source)
if stamp == "invalid" {
if err == nil {
t.Fatal("invalid timestamp accepted")
}
return
}
if err != nil {
t.Fatal(err)
}
want := stamp
if want == "" && tag != "" {
want = source.ModTime.Format(time.RFC3339Nano)
}
if got := opts.Header().Get(xhttp.MinIOSourceTaggingTimestamp); got != want {
t.Fatalf("wire timestamp=%q want %q", got, want)
}
})
}
}
}
func TestAPIPoolsTaggingReplicaDeletion(t *testing.T) {
z, bucket := consistencyPools(t)
defer r5Capacity(z)()
if err := newTestConfig(globalMinioDefaultRegion, z); err != nil {
t.Fatal(err)
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
router := initTestAPIEndPoints(z, nil)
base := time.Now().UTC().Add(-5 * time.Hour)
for _, op := range []string{"copy", "copy-default", "put", "multipart"} {
for _, kind := range []string{"uuid", "null"} {
t.Run(op+"/"+kind, func(t *testing.T) {
vid := mustGetUUID()
if kind == "null" {
vid = nullVersionID
}
name := "pool-tags-" + op + "-" + kind
var source ObjectInfo
for pool := range 2 {
tag := "key=stale-pool"
ts := base
if pool == 1 {
tag = ""
ts = base.Add(3 * time.Hour)
}
source = putConsistencyObject(t, z, bucket, name, pool, "data", ObjectOptions{Versioned: true, VersionID: vid, MTime: base, UserDefined: map[string]string{xhttp.AmzObjectTagging: tag, r5TagStamp: ts.Format(time.RFC3339Nano)}})
}
// Clear all copies through the signed local handler, then replay a stale incoming state.
req := r5Request(t, router, globalActiveCred, http.MethodDelete, "/"+bucket+"/"+name+"?tagging&versionId="+vid, "", nil)
if req.Code != http.StatusNoContent {
t.Fatalf("DELETE: %d %s", req.Code, req.Body.String())
}
current, err := z.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{VersionID: vid})
if err != nil {
t.Fatal(err)
}
deletedAt := current.UserDefined[r5TagStamp]
for pool := range 2 {
r5Stored(t, z.serverPools[pool], bucket, name, vid, "", deletedAt)
}
source.VersionID = vid
source.UserTags = "key=delayed"
source.UserDefined = map[string]string{r5TagStamp: base.Add(time.Hour).Format(time.RFC3339Nano)}
r5Receive(t, z, router, globalActiveCred, bucket, op, source, source.UserDefined[r5TagStamp], nil)
// The addressed version must remain readable through normal routing.
r5Stored(t, z, bucket, name, vid, "", deletedAt)
// Existing duplicate suppression may leave both identical tombstones; any retained copy must be correct.
for pool := range 2 {
got, err := z.serverPools[pool].GetObjectInfo(t.Context(), bucket, name, ObjectOptions{VersionID: vid})
if isErrVersionNotFound(err) {
continue
}
if err != nil {
t.Fatal(err)
}
if got.UserTags != "" || got.UserDefined[r5TagStamp] != deletedAt {
t.Fatalf("pool %d: tags=%q stamp=%q want deletion %q", pool, got.UserTags, got.UserDefined[r5TagStamp], deletedAt)
}
}
})
}
}
}
func TestAPITaggingSSECRotationPreservesDeletionRevision(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
defer r5Capacity(obj.(*erasureServerPools))()
oldTLS := globalIsTLS
globalIsTLS = true
defer func() { globalIsTLS = oldTLS }()
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
keys := [][]byte{[]byte(strings.Repeat("a", 32)), []byte(strings.Repeat("b", 32)), []byte(strings.Repeat("c", 32)), []byte(strings.Repeat("d", 32))}
const name = "tag-rotation"
putCopyChecksumSource(t, router, cred, bucket, name, []byte("data"), ssecKeyHeaders(keys[0], false))
oi, err := obj.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
base := time.Now().UTC().Add(-time.Hour)
for i, event := range []struct {
tags string
delta int
wantTags string
wantDelta int
}{
{"key=live", 1, "key=live", 1}, {"", 3, "", 3}, {"key=stale", 2, "", 3},
} {
h := ssecKeyHeaders(keys[i], true)
maps.Copy(h, ssecKeyHeaders(keys[i+1], false))
h[xhttp.AmzObjectTagging] = event.tags
h[xhttp.AmzTagDirective] = "REPLACE"
h[xhttp.MinIOSourceTaggingTimestamp] = base.Add(time.Duration(event.delta) * time.Minute).Format(time.RFC3339Nano)
sendReplicaLockCopy(t, router, cred, bucket, name, oi.VersionID, h)
r5Stored(t, obj, bucket, name, oi.VersionID, event.wantTags, base.Add(time.Duration(event.wantDelta)*time.Minute).Format(time.RFC3339Nano))
}
get := r5Request(t, router, cred, http.MethodGet, "/"+bucket+"/"+name+"?versionId="+oi.VersionID, "", ssecKeyHeaders(keys[3], false))
if get.Code != http.StatusOK || get.Body.String() != "data" {
t.Fatalf("%s GET after rotations: %d %q", instance, get.Code, get.Body.String())
}
}})
}
func TestTaggingRepeatedValueNeedsRevisionDelivery(t *testing.T) {
for _, tag := range []string{"", "key=same"} {
now := time.Now().UTC()
source := ObjectInfo{ModTime: now, UserTags: tag, UserDefined: map[string]string{r5TagStamp: now.Add(time.Hour).Format(time.RFC3339Nano)}}
target := minio.ObjectInfo{LastModified: now}
if tag != "" {
target.UserTags = map[string]string{"key": "same"}
target.UserTagCount = 1
}
if got := getReplicationAction(source, target, replication.MetadataReplicationType); got != replicateMetadata {
t.Errorf("equal tags %q suppress a newer revision: got %s; an intervening delayed deletion can win", tag, got)
}
}
}
func TestTaggingProductionCopyWireShape(t *testing.T) {
got := make(chan http.Header, 1)
peer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got <- r.Header.Clone()
w.Header().Set(xhttp.ContentType, "application/xml")
w.Write([]byte(`<CopyObjectResult><LastModified>2026-09-15T01:00:00Z</LastModified><ETag>"abc"</ETag></CopyObjectResult>`))
}))
defer peer.Close()
c, err := minio.New(strings.TrimPrefix(peer.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
oi := ObjectInfo{Name: "object", ModTime: now, ETag: "abc", VersionID: mustGetUUID()}
core := minio.Core{Client: c}
_, err = core.CopyObject(t.Context(), "bucket", "object", "bucket", "object", getCopyObjMetadata(oi, ""), minio.CopySrcOptions{VersionID: oi.VersionID}, minio.PutObjectOptions{Internal: minio.AdvancedPutOptions{SourceVersionID: oi.VersionID, ReplicationRequest: true, TaggingTimestamp: now}})
if err != nil {
t.Fatal(err)
}
h := <-got
t.Logf("SDK wire: metadata-directive=%q tagging-directive=%q tagging=%q time=%q", h.Get(xhttp.AmzMetadataDirective), h.Get(xhttp.AmzTagDirective), h.Get(xhttp.AmzObjectTagging), h.Get(xhttp.MinIOSourceTaggingTimestamp))
if h.Get(xhttp.AmzMetadataDirective) != "" || h.Get(xhttp.AmzTagDirective) != "REPLACE" || h.Get(xhttp.MinIOSourceTaggingTimestamp) != now.Format(time.RFC3339Nano) {
t.Fatalf("unexpected SDK wire: %v", h)
}
}
func TestLocalTaggingCommitCannotRegressRevision(t *testing.T) {
z, bucket := consistencyPools(t)
incoming := "2026-09-15T01:00:00Z"
newer := "2026-09-15T02:00:00Z"
newest := "2026-09-15T03:00:00Z"
vid := mustGetUUID()
name := "inverted-local-tags"
for pool := range 2 {
putConsistencyObject(t, z, bucket, name, pool, "data", ObjectOptions{Versioned: true, VersionID: vid, UserDefined: map[string]string{r5TagStamp: []string{newer, newest}[pool]}})
}
oi, err := z.PutObjectTags(t.Context(), bucket, name, "key=after-delete", ObjectOptions{VersionID: vid, UserDefined: map[string]string{r5TagStamp: incoming}})
if err != nil {
t.Fatal(err)
}
want := "2026-09-15T03:00:00.000000001Z"
for pool := range 2 {
r5Stored(t, z.serverPools[pool], bucket, name, vid, "key=after-delete", want)
}
if oi.UserDefined[r5TagStamp] != want {
t.Fatalf("response stamp=%q want %q", oi.UserDefined[r5TagStamp], want)
}
// The single-set guard also applies when the pools dispatcher is bypassed.
_, err = z.serverPools[0].PutObjectTags(t.Context(), bucket, name, "", ObjectOptions{VersionID: vid, UserDefined: map[string]string{r5TagStamp: incoming}})
if err != nil {
t.Fatal(err)
}
r5Stored(t, z.serverPools[0], bucket, name, vid, "", "2026-09-15T03:00:00.000000002Z")
}
func TestTaggingReplicaContentDuplicateGuard(t *testing.T) {
stamp := time.Date(2026, 9, 15, 1, 0, 0, 0, time.UTC)
for _, tc := range []struct {
name string
delta int
stored string
trusted, replica bool
ifMatch, ifNone string
wantSkip bool
}{
{name: "newer", delta: 1, trusted: true, replica: true},
{name: "equal", trusted: true, replica: true, wantSkip: true},
{name: "older", delta: -1, trusted: true, replica: true, wantSkip: true},
{name: "invalid-stored", stored: "invalid", delta: 1, trusted: true, replica: true},
{name: "untrusted", delta: 1, wantSkip: true},
{name: "marker-only", delta: 1, trusted: true, wantSkip: true},
{name: "if-match-fails", delta: 1, trusted: true, replica: true, ifMatch: "other", wantSkip: true},
{name: "if-none-match-fails", delta: 1, trusted: true, replica: true, ifNone: "etag", wantSkip: true},
{name: "if-match-passes", delta: 1, trusted: true, replica: true, ifMatch: "etag"},
} {
t.Run(tc.name, func(t *testing.T) {
ctx := withReplicationTrust(t.Context(), tc.trusted, tc.replica)
r := httptest.NewRequest(http.MethodPut, "/bucket/object", nil).WithContext(ctx)
if tc.ifMatch != "" {
r.Header.Set(xhttp.IfMatch, tc.ifMatch)
}
if tc.ifNone != "" {
r.Header.Set(xhttp.IfNoneMatch, tc.ifNone)
}
stored := tc.stored
if stored == "" {
stored = stamp.Format(time.RFC3339Nano)
}
oi := ObjectInfo{ModTime: stamp, VersionID: mustGetUUID(), ETag: "etag", UserDefined: map[string]string{r5TagStamp: stored}}
opts := ObjectOptions{VersionID: oi.VersionID, PreserveETag: oi.ETag, ReplicationRequest: tc.trusted, ReplicationSourceTaggingTimestamp: stamp.Add(time.Duration(tc.delta) * time.Second)}
if skip := checkPreconditionsPUT(ctx, httptest.NewRecorder(), r, oi, opts); skip != tc.wantSkip {
t.Fatalf("skip=%v, want %v", skip, tc.wantSkip)
}
})
}
}
func TestAPITaggingUnqualifiedCopyOrdering(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
defer r5Capacity(obj.(*erasureServerPools))()
stamp := time.Now().UTC().Add(-time.Hour)
oi, err := obj.PutObject(t.Context(), bucket, "unqualified-tags", mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{UserDefined: map[string]string{xhttp.AmzObjectTagging: "key=stored", r5TagStamp: stamp.Format(time.RFC3339Nano)}})
if err != nil {
t.Fatal(err)
}
source := oi
source.UserTags = ""
source.UserDefined = maps.Clone(oi.UserDefined)
r5Receive(t, obj, router, cred, bucket, "copy", source, stamp.Format(time.RFC3339Nano), nil)
r5Stored(t, obj, bucket, oi.Name, "", "key=stored", stamp.Format(time.RFC3339Nano))
later := stamp.Add(time.Minute).Format(time.RFC3339Nano)
r5Receive(t, obj, router, cred, bucket, "copy-default", source, later, nil)
r5Stored(t, obj, bucket, oi.Name, "", "", later)
source.UserTags = "key=delayed"
r5Receive(t, obj, router, cred, bucket, "copy-default", source, stamp.Format(time.RFC3339Nano), nil)
r5Stored(t, obj, bucket, oi.Name, "", "", later)
t.Logf("%s: unqualified COPY keeps stored ties and ordered deletion", instance)
}})
}
+154
View File
@@ -0,0 +1,154 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/bucket/replication"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/once"
)
func r5ReplicationFixture(t *testing.T, obj ObjectLayer, bucket string, client *minio.Client) (chan ReplicationWorkerOperation, func()) {
t.Helper()
const arn = "arn:minio:replication::af470089-d354-4473-934c-9e1f52f6da89:bucket"
target := &TargetClient{Client: client, ARN: arn, Bucket: bucket}
globalBucketTargetSys.arnRemotesMap[arn] = arnTarget{Client: target, lastRefresh: UTCNow()}
globalBucketTargetSys.targetsMap[bucket] = []madmin.BucketTarget{{Arn: arn, TargetBucket: bucket}}
meta, err := globalBucketMetadataSys.Get(bucket)
if err != nil {
t.Fatal(err)
}
cfg := configs[0]
cfg.RoleArn = arn
meta.replicationConfig = &cfg
globalBucketMetadataSys.Set(bucket, meta)
worker := make(chan ReplicationWorkerOperation, 10)
previous := globalReplicationPool
globalReplicationPool = once.NewSingleton[ReplicationPool]()
globalReplicationPool.Set(&ReplicationPool{ctx: t.Context(), objLayer: obj, workers: []chan ReplicationWorkerOperation{worker}, stats: globalReplicationStats.Load(), mrfSaveCh: make(chan MRFReplicateEntry, 10)})
return worker, func() { globalReplicationPool = previous }
}
func TestTaggingReplicationSenderRetryAndAcknowledgment(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
defer r5Capacity(obj.(*erasureServerPools))()
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, enabledBucketVersioningConfig); err != nil {
t.Fatal(err)
}
const name = "tagging-old-queue"
oi, err := obj.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
requests := make(chan http.Header, 4)
var attempts atomic.Int32
peer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set(xhttp.AmzVersionID, oi.VersionID)
w.Header().Set(xhttp.ETag, "\""+oi.ETag+"\"")
w.Header().Set(xhttp.LastModified, oi.ModTime.Format(http.TimeFormat))
w.Header().Set(xhttp.ContentType, oi.ContentType)
if r.Method == http.MethodHead {
w.Header().Set(xhttp.ContentLength, "4")
w.WriteHeader(http.StatusOK)
return
}
requests <- r.Header.Clone()
w.Header().Set(xhttp.ContentType, "application/xml")
if attempts.Add(1) == 1 {
w.WriteHeader(http.StatusServiceUnavailable)
w.Write([]byte(`<Error><Code>SlowDown</Code><Message>retry fixture</Message></Error>`))
return
}
w.Write([]byte("<CopyObjectResult><LastModified>" + oi.ModTime.Format(time.RFC3339Nano) + "</LastModified><ETag>\"" + oi.ETag + "\"</ETag></CopyObjectResult>"))
}))
defer peer.Close()
client, err := minio.New(strings.TrimPrefix(peer.URL, "http://"), &minio.Options{Region: "us-east-1", MaxRetries: 1})
if err != nil {
t.Fatal(err)
}
worker, cleanup := r5ReplicationFixture(t, obj, bucket, client)
defer cleanup()
body := `<Tagging><TagSet><Tag><Key>key</Key><Value>queued</Value></Tag></TagSet></Tagging>`
w := r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/"+name+"?tagging&versionId="+oi.VersionID, body, nil)
if w.Code != http.StatusOK || len(worker) != 1 {
t.Fatalf("tagging PUT: %d %s queued=%d", w.Code, w.Body.String(), len(worker))
}
old := (<-worker).(ReplicateObjectInfo)
// Delete through the actual handler before processing the old task.
w = r5Request(t, router, cred, http.MethodDelete, "/"+bucket+"/"+name+"?tagging&versionId="+oi.VersionID, "", nil)
if w.Code != http.StatusNoContent || len(worker) != 1 {
t.Fatalf("tagging DELETE: %d %s queued=%d", w.Code, w.Body.String(), len(worker))
}
deleted, err := obj.GetObjectInfo(t.Context(), bucket, name, ObjectOptions{VersionID: oi.VersionID})
if err != nil {
t.Fatal(err)
}
stamp := deleted.UserDefined[r5TagStamp]
for attempt := 0; attempt < 2; attempt++ {
result := replicateObject(t.Context(), old, obj)
want := replication.Failed
if attempt == 1 {
want = replication.Completed
}
if result.ReplicationStatus() != want {
t.Fatalf("attempt %d result=%+v want %s", attempt, result, want)
}
if len(result.Targets) != 1 || result.Targets[0].ReplicationAction != replicateMetadata || (result.Targets[0].Err != nil) != (attempt == 0) {
t.Fatalf("attempt %d reported wrong action/error: %+v", attempt, result)
}
r5Stored(t, obj, bucket, name, oi.VersionID, "", stamp)
select {
case h := <-requests:
if h.Get(xhttp.MinIOSourceTaggingTimestamp) != stamp || h.Get(xhttp.AmzObjectTagging) != "" || h.Get(xhttp.AmzTagDirective) != "REPLACE" || h.Get(xhttp.AmzMetadataDirective) != "" {
t.Fatalf("sender did not carry current deletion: %v", h)
}
t.Logf("%s attempt %d sent tags=%q timestamp=%s status=%s", instance, attempt, h.Get(xhttp.AmzObjectTagging), stamp, want)
default:
t.Fatal("no metadata COPY sent for same-empty target")
}
}
// With a real outgoing rule enabled, a signed incoming replica COPY
// must not queue another outgoing event and create a feedback loop.
before := len(worker)
r5Receive(t, obj, router, cred, bucket, "copy", deleted, stamp, nil)
if len(worker) != before {
t.Fatal("incoming replica COPY scheduled another outgoing event")
}
// The metadata sender must fail malformed stored revisions before COPY,
// just as the full retransmission option builder does.
_, err = obj.PutObjectTags(t.Context(), bucket, name, "", ObjectOptions{VersionID: oi.VersionID, UserDefined: map[string]string{r5TagStamp: "invalid"}})
if err != nil {
t.Fatal(err)
}
target := globalBucketTargetSys.GetRemoteTargetClient(bucket, globalBucketTargetSys.targetsMap[bucket][0].Arn)
invalid := old.replicateAll(t.Context(), obj, target)
if invalid.ReplicationStatus != replication.Failed || invalid.Err == nil || len(requests) != 0 {
t.Fatalf("invalid timestamp was not rejected before send: %+v", invalid)
}
}})
}
+2
View File
@@ -901,6 +901,8 @@ func serverMain(ctx *cli.Context) {
close(globalGridStart)
close(globalLockGridStart)
// The HTTP/1 listener preserves absolute header deadlines and renews the
// body read/write idle limits, so transfers may outlast IdleTimeout.
httpServer := xhttp.NewServer(getServerListenAddrs()).
UseHandler(setCriticalErrorHandler(corsHandler(handler))).
UseTLSConfig(newTLSConfig(getCert)).
+97
View File
@@ -0,0 +1,97 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"os"
"testing"
"time"
"github.com/minio/cli"
xhttp "github.com/minio/minio/internal/http"
)
func TestServerReadHeaderTimeoutConfig(t *testing.T) {
for _, tc := range []struct {
name, env, idle string
args []string
want time.Duration
fmtgen bool
}{
{name: "default", want: xhttp.DefaultReadHeaderTimeout},
{name: "flag", args: []string{"--read-header-timeout=100ms"}, want: 100 * time.Millisecond},
{name: "environment", env: "170ms", want: 170 * time.Millisecond},
{name: "flag-over-environment", env: "170ms", args: []string{"--read-header-timeout=80ms"}, want: 80 * time.Millisecond},
{name: "yaml-retains-flag", args: []string{"--config=testdata/config/1.yaml", "--read-header-timeout=100ms"}, want: 100 * time.Millisecond},
{name: "zero-fallback", args: []string{"--read-header-timeout=0s"}},
{name: "zero-idle-default-header", idle: "0s", want: xhttp.DefaultReadHeaderTimeout},
{name: "negative-idle-default-header", idle: "-1s", want: xhttp.DefaultReadHeaderTimeout},
{name: "fmt-gen-unregistered-duration", env: "100ms", fmtgen: true},
{name: "negative-disabled", args: []string{"--read-header-timeout=-1s"}, want: -time.Second},
} {
t.Run(tc.name, func(t *testing.T) {
for _, key := range []string{"MINIO_ARGS", "MINIO_VOLUMES", "MINIO_ENDPOINTS", "MINIO_CONFIG", "MINIO_ERASURE_SET_DRIVE_COUNT"} {
t.Setenv(key, "")
}
t.Setenv("MINIO_READ_HEADER_TIMEOUT", tc.env)
if tc.env == "" {
if err := os.Unsetenv("MINIO_READ_HEADER_TIMEOUT"); err != nil {
t.Fatal(err)
}
}
idle := tc.idle
if idle == "" {
idle = "2s"
}
t.Setenv("MINIO_IDLE_TIMEOUT", idle)
idleWant, err := time.ParseDuration(idle)
if err != nil {
t.Fatal(err)
}
commandName, flags := "server", serverCmd.Flags
if tc.fmtgen {
commandName, flags = "fmt-gen", fmtGenFlags
idleWant = 0
}
var got serverCtxt
called := false
app := cli.NewApp()
app.Commands = []cli.Command{{Name: commandName, Flags: flags, Action: func(ctx *cli.Context) error {
called = true
if parsed := ctx.Duration("read-header-timeout"); parsed != tc.want {
t.Errorf("CLI read-header-timeout=%s, expected=%s", parsed, tc.want)
}
return buildServerCtxt(ctx, &got)
}}}
args := append([]string{"silo", commandName}, tc.args...)
args = append(args, t.TempDir())
if err := app.Run(args); err != nil {
t.Fatal(err)
}
if !called {
t.Fatal("server action did not run")
}
if got.ReadHeaderTimeout != tc.want {
t.Errorf("parsed ReadHeaderTimeout = %s, want %s", got.ReadHeaderTimeout, tc.want)
}
if got.IdleTimeout != idleWant {
t.Errorf("parsed IdleTimeout = %s, want %s", got.IdleTimeout, idleWant)
}
})
}
}
+9
View File
@@ -109,6 +109,15 @@ func testStorageAPIListDir(t *testing.T, storage StorageAPI) {
}
}
}
for _, name := range []string{"one", "two", "three"} {
if err := storage.AppendFile(t.Context(), "foo", "bounded/"+name, []byte("x")); err != nil {
t.Fatal(err)
}
}
entries, err := storage.ListDir(t.Context(), "", "foo", "bounded", 2)
if err != nil || len(entries) != 2 {
t.Fatalf("ListDir count was lost in storage/RPC path: %v %v", entries, err)
}
}
func testStorageAPIReadAll(t *testing.T, storage StorageAPI) {
+101
View File
@@ -0,0 +1,101 @@
// Copyright (c) 2026 Feng Ruohang
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"archive/tar"
"bytes"
"fmt"
"io"
"os"
"sync"
"testing"
"testing/iotest"
"github.com/pierrec/lz4/v4"
)
func TestUntarLZ4(t *testing.T) {
for _, empty := range []bool{false, true} {
t.Run(fmt.Sprintf("empty=%t", empty), func(t *testing.T) {
want := map[string][]byte{}
if !empty {
want["small.txt"] = bytes.Repeat([]byte("small object\n"), 10)
want["large.txt"] = bytes.Repeat([]byte("large object\n"), 32768)
}
var compressed bytes.Buffer
compressor := lz4.NewWriter(&compressed)
archive := tar.NewWriter(compressor)
for name, data := range want {
if err := archive.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(data))}); err != nil {
t.Fatal(err)
}
if _, err := archive.Write(data); err != nil {
t.Fatal(err)
}
}
if err := archive.Close(); err != nil {
t.Fatal(err)
}
if err := compressor.Close(); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
got := map[string][]byte{}
err := untar(t.Context(), iotest.HalfReader(bytes.NewReader(compressed.Bytes())), func(r io.Reader, info os.FileInfo, name string) error {
// Exercise a short first read followed by streaming the remaining content.
var output bytes.Buffer
prefix := make([]byte, 7)
if _, err := io.ReadFull(r, prefix); err != nil {
return err
}
output.Write(prefix)
if _, err := io.Copy(&output, r); err != nil {
return err
}
if int64(output.Len()) != info.Size() {
return fmt.Errorf("size mismatch for %s", name)
}
mu.Lock()
got[name] = output.Bytes()
mu.Unlock()
return nil
}, untarOptions{})
if err != nil {
t.Fatal(err)
}
if len(got) != len(want) {
t.Fatalf("objects=%d, want %d", len(got), len(want))
}
for name, data := range want {
if !bytes.Equal(got[name], data) {
t.Errorf("content mismatch for %s", name)
}
}
t.Run("corrupt-header", func(t *testing.T) {
damaged := bytes.Clone(compressed.Bytes())
damaged[6] ^= 0xff
err := untar(t.Context(), bytes.NewReader(damaged), func(io.Reader, os.FileInfo, string) error {
t.Error("corrupt header must be rejected before uploading objects")
return nil
}, untarOptions{})
if err == nil {
t.Fatal("corrupt LZ4 header accepted")
}
})
})
}
}
@@ -0,0 +1,148 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"io"
"maps"
"net/http"
"testing"
"time"
"github.com/minio/minio/internal/bucket/replication"
)
func TestDeleteMarkerReadDataControls(t *testing.T) {
obj, er, disks, bucket := markerPurgeFixture(t, 4)
for _, body := range []string{"", "inline-data-control"} {
name := "data-" + mustGetUUID()
oi, err := er.PutObject(t.Context(), bucket, name, mustGetPutObjReader(t, bytes.NewBufferString(body), int64(len(body)), "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
for _, disk := range disks {
fi, err := disk.ReadVersion(t.Context(), "", bucket, name, oi.VersionID, ReadOptions{ReadData: true})
if err != nil || fi.Deleted || fi.Size != int64(len(body)) || !fi.InlineData() || (body != "" && len(fi.Data) == 0) {
t.Fatalf("data read: size=%d inline=%v bytes=%d error=%v", fi.Size, fi.InlineData(), len(fi.Data), err)
}
// Legacy inline data without its annotation still gets the original
// ReadData behavior; the new marker guard must not affect it.
delete(fi.Metadata, ReservedMetadataPrefixLower+"inline-data")
if err := disk.WriteMetadata(t.Context(), "", bucket, name, fi); err != nil {
t.Fatal(err)
}
fi, err = disk.ReadVersion(t.Context(), "", bucket, name, oi.VersionID, ReadOptions{ReadData: true})
if err != nil || !fi.InlineData() {
t.Fatalf("legacy inline read: %+v error=%v", fi, err)
}
}
reader, err := obj.GetObjectNInfo(t.Context(), bucket, name, nil, http.Header{}, ObjectOptions{VersionID: oi.VersionID})
if err != nil {
t.Fatal(err)
}
got, err := io.ReadAll(reader)
reader.Close()
if err != nil || string(got) != body {
t.Fatalf("payload changed: %q error=%v", got, err)
}
}
_, opts := seedPurgeMarker(t, er, bucket, "stored-marker-key", false)
for _, disk := range disks {
fi, err := disk.ReadVersion(t.Context(), "", bucket, "stored-marker-key", opts.VersionID, ReadOptions{})
if err != nil {
t.Fatal(err)
}
// Preserve even an existing unusual marker key; suppress only the
// manufacture of a new inline annotation by the data-read branch.
fi.Metadata[ReservedMetadataPrefixLower+"inline-data"] = "original"
if err := disk.WriteMetadata(t.Context(), "", bucket, "stored-marker-key", fi); err != nil {
t.Fatal(err)
}
fi, err = disk.ReadVersion(t.Context(), "", bucket, "stored-marker-key", opts.VersionID, ReadOptions{ReadData: true})
if err != nil || fi.Metadata[ReservedMetadataPrefixLower+"inline-data"] != "original" {
t.Errorf("existing marker key changed: metadata=%v error=%v", fi.Metadata, err)
}
}
}
func TestDeleteMarkerMetadataRoundTrip(t *testing.T) {
stamp := time.Date(2026, 9, 1, 12, 1, 2, 345, time.UTC)
for _, free := range []bool{false, true} {
t.Run(map[bool]string{false: "replication", true: "free-version"}[free], func(t *testing.T) {
metadata := map[string]string{
ReservedMetadataPrefixLower + ReplicaStatus: "REPLICA",
ReservedMetadataPrefixLower + ReplicaTimestamp: stamp.Format(time.RFC3339Nano),
ReservedMetadataPrefixLower + ReplicationStatus: "arn1=COMPLETED;arn2=FAILED;",
ReservedMetadataPrefixLower + ReplicationTimestamp: stamp.Add(-time.Hour).Format(time.RFC3339Nano),
VersionPurgeStatusKey: "arn1=PENDING;arn2=FAILED;",
targetResetHeader("arn1"): "original;reset1",
targetResetHeader("arn2"): "original;reset2",
ReservedMetadataPrefixLower + "unknown": "",
}
if free {
metadata[ReservedMetadataPrefixLower+freeVersion] = ""
metadata[metaTierName] = "tier"
metadata[metaTierObjName] = "remote-object"
metadata[metaTierVersionID] = "remote-version"
}
fi := FileInfo{VersionID: mustGetUUID(), Deleted: true, ModTime: stamp, Metadata: maps.Clone(metadata)}
// Deliberately conflicting parsed state must never replace stored
// values or add a key missing from the raw metadata.
fi.ReplicationState = ReplicationState{ReplicaStatus: replication.Failed, ReplicaTimeStamp: stamp.Add(time.Hour), ResetStatusesMap: map[string]string{"new-arn": "invented"}}
fi.SetHealing()
fi.SetDataMov()
fi.SetTierFreeVersionID(mustGetUUID())
fi.SetTierFreeVersion()
fi.SetSkipTierFreeVersion()
xl := xlMetaV2{}
if err := xl.AddVersion(fi); err != nil {
t.Fatal(err)
}
stored, err := xl.getIdx(0)
if err != nil {
t.Fatal(err)
}
got := make(map[string]string)
for k, v := range stored.DeleteMarker.MetaSys {
got[k] = string(v)
}
if !maps.Equal(got, metadata) {
t.Errorf("stored metadata=%v want=%v", got, metadata)
}
decoded, err := xl.ToFileInfo("bucket", "marker", fi.VersionID, true, true)
if err != nil || !decoded.ModTime.Equal(fi.ModTime) || decoded.TierFreeVersion() != free {
t.Errorf("round trip identity: %+v error=%v", decoded, err)
}
if decoded.ReplicationState.ReplicaStatus != replication.Replica || decoded.ReplicationState.PurgeTargets["arn2"] != replication.VersionPurgeFailed || decoded.ReplicationState.Targets["arn1"] != replication.Completed {
t.Errorf("lost parsed replication state: %+v", decoded.ReplicationState)
}
})
}
}
func TestDeleteMarkerCreationMetadata(t *testing.T) {
_, er, disks, bucket := markerPurgeFixture(t, 4)
_, opts := seedPurgeMarker(t, er, bucket, "empty-key", false)
for i, disk := range disks {
fi, err := disk.ReadVersion(t.Context(), "", bucket, "empty-key", opts.VersionID, ReadOptions{})
if err != nil || fi.Metadata[ReservedMetadataPrefixLower+ReplicaStatus] != "REPLICA" || fi.Metadata[ReservedMetadataPrefixLower+ReplicaTimestamp] != opts.DeleteReplication.ReplicaTimeStamp.Format(time.RFC3339Nano) {
t.Errorf("disk %d lost new marker replica identity: metadata=%v error=%v", i, fi.Metadata, err)
}
}
}
+67
View File
@@ -0,0 +1,67 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"strings"
"time"
)
// deleteMarkerMetadata preserves the complete stored marker state during
// healing. The parsed ReplicationState is lossy; even absent raw keys must not
// be reconstructed from it. Only creation writers without stored metadata
// use typed state, and never invent timestamps.
func deleteMarkerMetadata(fi FileInfo) map[string][]byte {
meta := make(map[string][]byte, len(fi.Metadata))
for k, v := range fi.Metadata {
switch k {
case xMinIOHealing, xMinIODataMov,
ReservedMetadataPrefixLower + tierFVID,
ReservedMetadataPrefixLower + tierFVMarker,
ReservedMetadataPrefixLower + tierSkipFVID:
continue
}
meta[k] = []byte(v)
}
if len(meta) != 0 || fi.Healing() || fi.DataMov() {
return meta
}
rs := fi.ReplicationState
if !rs.ReplicaStatus.Empty() {
meta[ReservedMetadataPrefixLower+ReplicaStatus] = []byte(rs.ReplicaStatus)
if !rs.ReplicaTimeStamp.IsZero() {
meta[ReservedMetadataPrefixLower+ReplicaTimestamp] = []byte(rs.ReplicaTimeStamp.UTC().Format(time.RFC3339Nano))
}
}
if rs.ReplicationStatusInternal != "" {
meta[ReservedMetadataPrefixLower+ReplicationStatus] = []byte(rs.ReplicationStatusInternal)
if !rs.ReplicationTimeStamp.IsZero() {
meta[ReservedMetadataPrefixLower+ReplicationTimestamp] = []byte(rs.ReplicationTimeStamp.UTC().Format(time.RFC3339Nano))
}
}
if rs.VersionPurgeStatusInternal != "" {
meta[VersionPurgeStatusKey] = []byte(rs.VersionPurgeStatusInternal)
}
for k, v := range rs.ResetStatusesMap {
if !strings.HasPrefix(k, ReservedMetadataPrefixLower+ReplicationReset) {
k = targetResetHeader(k)
}
meta[k] = []byte(v)
}
return meta
}
+274
View File
@@ -0,0 +1,274 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"fmt"
"math/rand"
"reflect"
"slices"
"testing"
"time"
)
// Use complete metadata so the selected version can also be decoded by LIST
// and the object read path, rather than testing only synthetic shallow headers.
func quorumNullVersion(t testing.TB, generation int64) xlMetaV2ShallowVersion {
t.Helper()
fi := newFileInfo("fixed/object", 12, 4)
fi.Erasure.Index = 1
fi.DataDir = "11111111-1111-1111-1111-111111111111"
fi.ModTime = time.Unix(generation, 0).UTC()
fi.Size = 8192
fi.Parts = []ObjectPartInfo{{Number: 1, Size: fi.Size, ActualSize: fi.Size}}
fi.Metadata = map[string]string{"etag": fmt.Sprintf("%032d", generation)}
var xl xlMetaV2
if err := xl.AddVersion(fi); err != nil {
t.Fatal(err)
}
return xl.versions[0]
}
// Start with forward, reverse and interleaved orders, then fixed-seed shuffles.
func quorumVersionOrders(input [][]xlMetaV2ShallowVersion) [][][]xlMetaV2ShallowVersion {
orders := [][][]xlMetaV2ShallowVersion{slices.Clone(input), slices.Clone(input)}
slices.Reverse(orders[1])
interleaved := make([][]xlMetaV2ShallowVersion, 0, len(input))
for i, j := 0, len(input)-1; i <= j; i, j = i+1, j-1 {
interleaved = append(interleaved, input[i])
if i != j {
interleaved = append(interleaved, input[j])
}
}
orders = append(orders, interleaved)
for seed := range int64(32) {
order := slices.Clone(input)
rand.New(rand.NewSource(seed)).Shuffle(len(order), func(i, j int) {
order[i], order[j] = order[j], order[i]
})
orders = append(orders, order)
}
return orders
}
func TestMergeXLV2SingleNullQuorum(t *testing.T) {
v := []xlMetaV2ShallowVersion{quorumNullVersion(t, 100), quorumNullVersion(t, 200), quorumNullVersion(t, 300)}
for _, tt := range []struct {
name string
counts [3]int
empty int
quorum int
want int
}{
{"consistent", [3]int{16}, 0, 8, 0},
{"old9-new7", [3]int{9, 7}, 0, 8, 0},
{"old15-new1", [3]int{15, 1}, 0, 8, 0},
{"old3-new1", [3]int{3, 1}, 0, 3, 0},
{"two-quorums", [3]int{8, 8}, 0, 8, 1},
{"empty-streams", [3]int{9, 1}, 6, 8, 0},
{"two-subquorums", [3]int{7, 7}, 2, 8, -1},
{"three-subquorums", [3]int{6, 5, 5}, 0, 8, -1},
} {
t.Run(tt.name, func(t *testing.T) {
var input [][]xlMetaV2ShallowVersion
for g, n := range tt.counts {
for range n {
input = append(input, []xlMetaV2ShallowVersion{v[g]})
}
}
input = append(input, make([][]xlMetaV2ShallowVersion, tt.empty)...)
want := []xlMetaV2ShallowVersion{}
if tt.want >= 0 {
want = append(want, v[tt.want])
}
for order, versions := range quorumVersionOrders(input) {
for _, strict := range []bool{false, true} {
for _, requested := range []int{0, 1} {
got := mergeXLV2Versions(tt.quorum, strict, requested, versions...)
if !reflect.DeepEqual(got, want) {
t.Fatalf("order=%d strict=%v requested=%d: got %#v, want %#v", order, strict, requested, got, want)
}
}
}
}
})
}
}
func TestMergeXLV2SingleNullHeaderGroups(t *testing.T) {
old := quorumNullVersion(t, 100)
newer := quorumNullVersion(t, 200)
differentSig := old
differentSig.header.Signature[0]++
differentFlags := old
differentFlags.header.Flags ^= xlFlagInlineData
for _, tt := range []struct {
name string
input [][]xlMetaV2ShallowVersion
strict bool
want []xlMetaV2ShallowVersion
}{
{"signature-nonstrict", [][]xlMetaV2ShallowVersion{{old}, {old}, {differentSig}, {newer}}, false, []xlMetaV2ShallowVersion{differentSig}},
{"signature-strict", [][]xlMetaV2ShallowVersion{{old}, {old}, {differentSig}, {newer}}, true, []xlMetaV2ShallowVersion{}},
{"flags-nonstrict", [][]xlMetaV2ShallowVersion{{old}, {old}, {differentFlags}, {newer}}, false, []xlMetaV2ShallowVersion{}},
{"flags-strict", [][]xlMetaV2ShallowVersion{{old}, {old}, {differentFlags}, {newer}}, true, []xlMetaV2ShallowVersion{}},
} {
t.Run(tt.name, func(t *testing.T) {
got := mergeXLV2Versions(3, tt.strict, 0, tt.input...)
if !reflect.DeepEqual(got, tt.want) {
t.Fatalf("got %#v, want %#v", got, tt.want)
}
})
}
}
func TestResolveSingleNullQuorum(t *testing.T) {
old, newer := quorumNullVersion(t, 100), quorumNullVersion(t, 200)
input := make([][]xlMetaV2ShallowVersion, 16)
for i := range input {
input[i] = []xlMetaV2ShallowVersion{old}
if i >= 9 {
input[i] = []xlMetaV2ShallowVersion{newer}
}
}
for order, versions := range quorumVersionOrders(input) {
entries := make(metaCacheEntries, len(versions))
for i := range versions {
xl := &xlMetaV2{versions: versions[i]}
metadata, err := xl.AppendTo(nil)
if err != nil {
t.Fatal(err)
}
entries[i] = metaCacheEntry{name: "fixed/object", metadata: metadata}
}
selected, ok := entries.resolve(&metadataResolutionParams{objQuorum: 8, dirQuorum: 8, requestedVersions: 1})
if !ok {
t.Fatalf("order %d: a complete null object version has quorum but was omitted", order)
}
xl, err := selected.xlmeta()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(xl.versions, []xlMetaV2ShallowVersion{old}) {
t.Fatalf("order %d: incorrect complete selected version: %#v", order, xl.versions)
}
fi, err := xl.ToFileInfo("bucket", "fixed/object", "", false, false)
if err != nil || !fi.IsValid() || fi.Size != 8192 || !fi.ModTime.Equal(time.Unix(100, 0)) {
t.Fatalf("order %d: selected metadata cannot be read: %+v, %v", order, fi, err)
}
}
}
// These are compatibility assertions for inputs outside the narrow repair.
// In particular, the mixed-version outputs below are not a new correctness
// contract for version enumeration; that behavior requires a separate change.
func TestMergeXLV2NullHistoriesUnchanged(t *testing.T) {
old, newer := quorumNullVersion(t, 100), quorumNullVersion(t, 300)
middle := quorumNullVersion(t, 200)
middle.header.VersionID = [16]byte{1}
highest := quorumNullVersion(t, 400)
highest.header.VersionID = [16]byte{2}
for _, tt := range []struct {
name string
input [][]xlMetaV2ShallowVersion
want []xlMetaV2ShallowVersion
}{
{"mixed-forward", [][]xlMetaV2ShallowVersion{{old}, {old}, {old}, {newer, middle}, {middle}, {middle}}, []xlMetaV2ShallowVersion{middle}},
{"mixed-reverse", [][]xlMetaV2ShallowVersion{{middle}, {middle}, {newer, middle}, {old}, {old}, {old}}, []xlMetaV2ShallowVersion{old}},
{"history-pruned-first", [][]xlMetaV2ShallowVersion{{highest, old}, {highest, old}, {highest, old}, {highest, newer}}, []xlMetaV2ShallowVersion{highest}},
} {
t.Run(tt.name, func(t *testing.T) {
got := mergeXLV2Versions(3, false, 0, tt.input...)
if !reflect.DeepEqual(got, tt.want) {
t.Fatalf("excluded history changed: got %#v, want %#v", got, tt.want)
}
})
}
for _, kind := range []string{"delete", "free", "legacy", "mixed-ec", "legacy-modern-ec", "nonzero-strict"} {
t.Run(kind, func(t *testing.T) {
a, b := old, newer
switch kind {
case "delete":
b.header.Type = DeleteType
case "free":
b.header.Flags |= xlFlagFreeVersion
case "legacy":
b.header.Type = LegacyType
case "mixed-ec":
b.header.EcN, b.header.EcM = 8, 8
case "legacy-modern-ec":
b.header.EcN, b.header.EcM = 0, 0
case "nonzero-strict":
a.header.VersionID, b.header.VersionID = [16]byte{1}, [16]byte{1}
}
for _, requested := range []int{0, 1} {
got := mergeXLV2Versions(3, true, requested, []xlMetaV2ShallowVersion{a}, []xlMetaV2ShallowVersion{a}, []xlMetaV2ShallowVersion{a}, []xlMetaV2ShallowVersion{b})
if len(got) != 0 {
t.Fatalf("excluded input changed: requested=%d got %#v", requested, got)
}
}
})
}
// All-legacy EC headers are eligible, unlike mixing legacy and modern EC.
old.header.EcN, old.header.EcM = 0, 0
newer.header.EcN, newer.header.EcM = 0, 0
got := mergeXLV2Versions(3, false, 0, []xlMetaV2ShallowVersion{old}, []xlMetaV2ShallowVersion{old}, []xlMetaV2ShallowVersion{old}, []xlMetaV2ShallowVersion{newer})
if !reflect.DeepEqual(got, []xlMetaV2ShallowVersion{old}) {
t.Fatalf("all-legacy EC lost the old quorum: %#v", got)
}
}
func BenchmarkResolveSingleNullQuorumPage(b *testing.B) {
old, newer := quorumNullVersion(b, 100), quorumNullVersion(b, 200)
for _, kind := range []string{"consistent", "old-first", "new-first"} {
b.Run(kind, func(b *testing.B) {
var templates [16]metaCacheEntry
for i := range templates {
v := old
if kind == "old-first" && i >= 9 || kind == "new-first" && i < 7 {
v = newer
}
xl := &xlMetaV2{versions: []xlMetaV2ShallowVersion{v}}
data, err := xl.AppendTo(nil)
if err != nil {
b.Fatal(err)
}
templates[i] = metaCacheEntry{metadata: data}
}
var names [1000]string
for i := range names {
names[i] = fmt.Sprintf("fixed/%04d", i)
}
resolver := metadataResolutionParams{objQuorum: 8, dirQuorum: 8, requestedVersions: 1}
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
for _, name := range names {
entries := templates
for i := range entries {
entries[i].name = name
}
selected, ok := metaCacheEntries(entries[:]).resolve(&resolver)
if ok && selected.reusable {
metaDataPoolPut(selected.metadata)
}
}
}
})
}
}
+83 -36
View File
@@ -1630,7 +1630,7 @@ func (x *xlMetaV2) AddVersion(fi FileInfo) error {
ventry.DeleteMarker = &xlMetaV2DeleteMarker{
VersionID: uv,
ModTime: fi.ModTime.UnixNano(),
MetaSys: make(map[string][]byte),
MetaSys: deleteMarkerMetadata(fi),
}
} else {
ventry.Type = ObjectType
@@ -1941,6 +1941,10 @@ func mergeXLV2Versions(quorum int, strict bool, requestedVersions int, versions
// No need for non-strict checks if quorum is 1.
strict = true
}
// Keep the original stream shapes: pruning must not make a versioned object
// eligible for the single null-version recount below.
originalVersions := versions
var checkedSingleNull, singleNull bool
// Shallow copy input
versions = append(make([][]xlMetaV2ShallowVersion, 0, len(versions)), versions...)
@@ -2015,44 +2019,23 @@ func mergeXLV2Versions(quorum int, strict bool, requestedVersions int, versions
// Version IDs match, but otherwise unable to resolve.
// We are either strict, or don't have enough information to match.
// Switch to a pure counting algo.
x := make(map[xlMetaV2VersionHeader]int, len(tops))
for _, a := range tops {
if a.header.VersionID != ver.header.VersionID {
continue
}
if !strict {
// we must match EC, when we are not strict.
if !a.header.matchesEC(ver.header) {
continue
}
a.header.Signature = [4]byte{}
}
x[a.header]++
}
latestCount = 0
for k, v := range x {
if v < latestCount {
continue
}
if v == latestCount && latest.header.sortsBefore(k) {
// Tiebreak, use sort.
continue
}
for _, a := range tops {
hdr := a.header
if !strict {
hdr.Signature = [4]byte{}
}
if hdr == k {
latest = a
}
}
latestCount = v
}
latest, latestCount = countXLV2Versions(tops, ver.header, latest, strict)
break
}
}
if latestCount < quorum {
if !checkedSingleNull {
singleNull = singleNullVersionStreams(originalVersions)
checkedSingleNull = true
}
if singleNull {
// A newer minority at the end can hide an older quorum from
// the selection loop. Recount before discarding the null ID.
if candidate, count := countXLV2Versions(tops, latest.header, latest, strict); count >= quorum {
latest, latestCount = candidate, count
}
}
}
if latestCount >= quorum {
merged = append(merged, latest)
@@ -2113,6 +2096,70 @@ func mergeXLV2Versions(quorum int, strict bool, requestedVersions int, versions
return merged
}
// singleNullVersionStreams excludes histories and other version types from the
// additional recount. Check the original inputs, before any stream is pruned.
func singleNullVersionStreams(versions [][]xlMetaV2ShallowVersion) bool {
var ec xlMetaV2VersionHeader
var haveEC bool
for _, stream := range versions {
if len(stream) == 0 {
continue
}
if len(stream) != 1 {
return false
}
h := stream[0].header
if h.VersionID != [16]byte{} || h.Type != ObjectType || h.FreeVersion() {
return false
}
if haveEC && (h.EcN != ec.EcN || h.EcM != ec.EcM) {
return false
}
ec, haveEC = h, true
}
return haveEC
}
// countXLV2Versions selects the most frequent compatible header for reference's
// VersionID, retaining the existing sort tiebreak and last matching entry.
func countXLV2Versions(tops []xlMetaV2ShallowVersion, reference xlMetaV2VersionHeader, latest xlMetaV2ShallowVersion, strict bool) (xlMetaV2ShallowVersion, int) {
x := make(map[xlMetaV2VersionHeader]int, len(tops))
for _, a := range tops {
if a.header.VersionID != reference.VersionID {
continue
}
if !strict {
// we must match EC, when we are not strict.
if !a.header.matchesEC(reference) {
continue
}
a.header.Signature = [4]byte{}
}
x[a.header]++
}
var latestCount int
for k, v := range x {
if v < latestCount {
continue
}
if v == latestCount && latest.header.sortsBefore(k) {
// Tiebreak, use sort.
continue
}
for _, a := range tops {
hdr := a.header
if !strict {
hdr.Signature = [4]byte{}
}
if hdr == k {
latest = a
}
}
latestCount = v
}
return latest, latestCount
}
type xlMetaBuf []byte
// ToFileInfo converts xlMetaV2 into a common FileInfo datastructure
+3 -1
View File
@@ -1719,7 +1719,9 @@ func (s *xlStorage) ReadVersion(ctx context.Context, origvolume, volume, path, v
defer metaDataPoolPut(buf)
}
if readData {
// Delete markers have no payload. In particular, do not manufacture an
// inline-data metadata key when healing their zero-size FileInfo.
if readData && !fi.Deleted {
if len(fi.Data) > 0 || fi.Size == 0 {
if fi.InlineData() {
// If written with header we are fine.
@@ -2,7 +2,7 @@
PR #60 introduced an opt-in scheduler that moved objects between local server pools according to GET frequency. It and its feature-specific fixes have been removed. This does not remove ordinary lifecycle expiration, transitions to remote tiers, rebalance, decommission, or the general multi-pool correctness fixes from PR #178.
The [introduction and rollback record](../../investigations/access-tiering-revert.md) documents the commit history, scope decision, review corrections and unresolved validation findings.
The [introduction and rollback record](https://silo.pgsty.com/compatibility/access-tiering-removal/) documents the commit history, scope decision, review corrections and unresolved validation findings.
The published Server 20260903 predates this feature. These instructions concern main/snapshot deployments that included #60; upgrading from the published version does not require access-tier configuration cleanup.
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,8 +58,8 @@ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/
silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
./mc ready sitea
./mc ready siteb
@@ -83,7 +83,7 @@ done
echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket
remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket)
sleep 1
@@ -137,7 +137,7 @@ fi
echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket-version/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket-version
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket-version
./mc mb sitea/bucket-version/directory/
@@ -37,7 +37,7 @@ set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -70,9 +70,9 @@ silo server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/
silo server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo12345@127.0.0.1:9006
./mc ready sitea
./mc ready siteb
@@ -93,73 +93,73 @@ export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
echo "adding replication rule for a -> b : ${remote_arn}"
sleep 1
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for b -> a : ${remote_arn}"
./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for a -> c : ${remote_arn}"
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for c -> a : ${remote_arn}"
./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for b -> c : ${remote_arn}"
./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for c -> b : ${remote_arn}"
./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for olockbucket a -> b : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for olockbucket b -> a : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for olockbucket a -> c : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for olockbucket c -> a : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for olockbucket b -> c : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for olockbucket c -> b : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
@@ -204,33 +204,33 @@ head -c 221227088 </dev/urandom >200M
sleep 10
echo "Verifying ETag for all objects"
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
# additional tests for encryption object alignment
go install -v github.com/minio/multipart-debug@latest
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo12345 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt
dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt
sudo apt install jq -y
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo12345 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo12345 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo12345 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
sleep 10
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -70,10 +70,10 @@ silo server --address 127.0.0.1:9008 "http://127.0.0.1:9007/tmp/multisited/data/
# Wait to make sure all Silo instances are up
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008
export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo12345@127.0.0.1:9006
export MC_HOST_sited=http://minio:silo12345@127.0.0.1:9008
./mc ready sitea
./mc ready siteb
@@ -89,7 +89,7 @@ export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008
sleep 10s
## Add warm tier
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo123 --bucket bucket
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo12345 --bucket bucket
## Add ILM rules
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2"
+2
View File
@@ -1,5 +1,7 @@
# Compression Guide
For SILO's current SSE-C compression and historical-object boundary, see the [maintained encryption guide](https://silo.pgsty.com/administration/server-side-encryption/server-side-encryption-sse-c/) and [SSE-C replica design](https://silo.pgsty.com/blog/design/ssec-replica-integrity/). Check the documented release boundary before applying main-branch behavior to an older binary.
Silo server allows streaming compression to ensure efficient disk space usage.
Compression happens inflight, i.e objects are compressed before being written to disk(s).
Silo uses [`klauspost/compress/s2`](https://github.com/klauspost/compress/tree/master/s2)
+3 -3
View File
@@ -10,13 +10,13 @@ Silo in distributed mode can help you setup a highly-available storage system wi
Distributed Silo provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/silo/blob/main/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed Silo is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed Silo.
If one or more drives are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data protection bits added automatically to provide additional safety for these objects.
With the default `availability` storage-class optimization, Silo can increase parity for new objects when drives are offline, up to half the drives in the erasure set. Writes must still meet the quorum for the resulting shard layout. This does not guarantee unchanged availability or keep a two-drive `EC:1` set writable after one drive fails. See the [storage-class reference](https://silo.pgsty.com/reference/minio-server/settings/storage-class/) for details.
### High availability
A stand-alone Silo server would go down if the server hosting the drives goes offline. In contrast, a distributed Silo setup with _m_ servers and _n_ drives will have your data safe as long as _m/2_ servers or _m*n_/2 or more drives are online.
A stand-alone Silo server becomes unavailable if its host goes offline. In a distributed deployment, determine node-failure tolerance from how many drives each failed node removes from every erasure set and from the parity recorded for the affected objects. Aggregate online node or drive counts alone do not establish read or write quorum.
For example, an 16-server distributed setup with 200 drives per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 drives down Silo would continue to read and write objects.
For example, if a 16-node pool has 16-drive erasure sets with one drive per node in each set, a fixed `EC:4` layout has a read and write quorum of 12. Losing four nodes leaves 12 drives in every set. This result depends on that drive placement and parity; it cannot be generalized to arbitrary failures of the same total number of drives. In a two-node, two-drive `EC:1` set, losing one node leaves only one drive, so existing intact objects can remain readable but writes cannot continue.
Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/pgsty/silo/blob/main/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class).
+6 -6
View File
@@ -1,18 +1,18 @@
# Silo Erasure Code Quickstart Guide
Silo protects data against hardware failures and silent data corruption using erasure code and checksums. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data.
Silo protects data against hardware failures and silent data corruption using erasure coding and checksums. Recovery depends on the healthy shards and metadata remaining in each object's erasure set. With maximum parity, an object can tolerate the loss of up to `floor(N/2)` shards in its `N`-drive set; the deployment's total online drive count is not sufficient to determine recoverability.
## What is Erasure Code?
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Silo uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.
Erasure coding uses mathematical algorithms to reconstruct missing or corrupted data. Silo uses Reed-Solomon codes to split objects into data and parity shards. For a 12-drive erasure set, parity can range from `EC:0` (12 data shards and no parity) to `EC:6` (six data shards and six parity shards).
By default, Silo shards the objects across N/2 data and N/2 parity drives. Though, you can use [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class) to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.
The default parity depends on the erasure set size: `EC:0` for 1 drive, `EC:1` for 2–3 drives, `EC:2` for 4–5 drives, `EC:3` for 6–7 drives, and `EC:4` for 8–16 drives. See the [Silo storage-class reference](https://silo.pgsty.com/reference/minio-server/settings/storage-class/) for configuration details.
In 12 drive example above, with Silo server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.
In the 12-drive example, the default `EC:4` layout uses eight data shards and four parity shards. An existing object can tolerate four unavailable shards if the remaining shards and metadata are intact. Explicit `EC:6` instead uses six data and six parity shards, with a read quorum of six and a write quorum of seven.
## Why is Erasure Code useful?
Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in Silo erasure code you can lose as many as half of drives and still the data remains safe. Further, Silo's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As Silo encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. Silo's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.
Erasure coding protects objects against drive failures within their erasure set, up to the parity recorded for each object. Silo encodes objects individually and can heal them incrementally when enough healthy shards and consistent metadata remain. Failed drives still need repair or replacement to restore redundancy; erasure coding does not eliminate that maintenance.
![Erasure](https://github.com/pgsty/silo/blob/main/docs/screenshots/erasure-code.jpg?raw=true)
@@ -64,4 +64,4 @@ podman run \
### 3. Test your setup
You may unplug drives randomly and continue to perform I/O on the system.
In a test deployment, take drives offline and verify reads and writes against the quorum required by each erasure set and object layout. Restore the drives after each test; continued I/O depends on the remaining healthy shards and metadata.
+10 -18
View File
@@ -38,31 +38,23 @@ You can calculate _approximate_ storage usage ratio using the formula - total dr
### Allowed values for STANDARD storage class
`STANDARD` storage class implies more parity than `REDUCED_REDUNDANCY` class. So, `STANDARD` parity drives should be
`STANDARD` supports `EC:0` without erasure-code redundancy and nonzero parity values up to `floor(N/2)`, where `N` is the number of drives in the erasure set. When both `STANDARD` and `REDUCED_REDUNDANCY` parity are nonzero, `STANDARD` must be greater than or equal to `REDUCED_REDUNDANCY`; equal parity is allowed.
- Greater than or equal to 2, if `REDUCED_REDUNDANCY` parity is not set.
- Greater than `REDUCED_REDUNDANCY` parity, if it is set.
The default `STANDARD` parity is:
Parity blocks can not be higher than data blocks, so `STANDARD` storage class parity can not be higher than N/2. (N being total number of drives)
The default value for the `STANDARD` storage class depends on the number of volumes in the erasure set:
| Erasure Set Size | Default Parity (EC:N) |
|------------------|-----------------------|
| 5 or fewer | EC:2 |
| 6-7 | EC:3 |
| 8 or more | EC:4 |
| Erasure Set Size | Default Parity (EC:M) |
| --- | --- |
| 1 | EC:0 |
| 2–3 | EC:1 |
| 4–5 | EC:2 |
| 6–7 | EC:3 |
| 8–16 | EC:4 |
For more complete documentation on Erasure Set sizing, see the [Silo Documentation on Erasure Sets](https://silo.pgsty.com/operations/concepts/erasure-coding/#minio-ec-erasure-set).
### Allowed values for REDUCED_REDUNDANCY storage class
`REDUCED_REDUNDANCY` implies lesser parity than `STANDARD` class. So,`REDUCED_REDUNDANCY` parity drives should be
- Less than N/2, if `STANDARD` parity is not set.
- Less than `STANDARD` Parity, if it is set.
Default value for `REDUCED_REDUNDANCY` storage class is `1`.
`REDUCED_REDUNDANCY` parity can be zero or up to `floor(N/2)`. When both storage classes have nonzero parity, its parity must be less than or equal to `STANDARD` parity. The default is `EC:1` for multi-drive sets and `EC:0` for single-drive deployments. See the [Silo storage-class reference](https://silo.pgsty.com/reference/minio-server/settings/storage-class/) for the maintained configuration documentation.
## Get started with Storage Class
+2
View File
@@ -1,5 +1,7 @@
# Federation Quickstart Guide *Federation feature is deprecated and should be avoided for future deployments*
The maintained [federated CopyObject design](https://silo.pgsty.com/blog/design/federated-copy-object/) records the destination encryption, checksum, Object Lock and committed-response contract, including its Server release boundary.
This document explains how to configure Silo with `Bucket lookup from DNS` style federation.
## Cross-deployment copy behavior
+2 -2
View File
@@ -38,7 +38,7 @@ pid=$!
mc ready mysilo
mc admin user add mysilo/ silo123 silo123
mc admin user add mysilo/ silo123 silo12345
mc admin policy create mysilo/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json
mc admin policy create mysilo/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json
@@ -50,7 +50,7 @@ mc admin policy attach mysilo consoleAdmin --user silo123
mc mb -l mysilo/test-bucket
mc mb -l mysilo/multi-key-poc
export MC_HOST_mysilo1="http://silo123:silo123@localhost:9000/"
export MC_HOST_mysilo1="http://silo123:silo12345@localhost:9000/"
mc cp /etc/issue mysilo1/test-bucket
ret=$?

Some files were not shown because too many files have changed in this diff Show More