Compare commits

...

43 Commits

Author SHA1 Message Date
Feng Ruohang b7f52ca433 Merge pull request #34 from pinginfo/fix-api-listenbucketnotification
fix: implement Flush on trackingResponseWriter
2026-07-29 14:48:23 +08:00
Feng Ruohang 4dfc27ce32 fix: upgrade security-sensitive Go dependencies
Require Go 1.26.5, gRPC 1.82.1, and x/text 0.39.0 while retaining the dependency graph's existing pins wherever MVS permits. Restore the blocking govulncheck job and correct the documented security advisories.
2026-07-29 10:34:55 +08:00
Feng Ruohang ce01ccbdc1 helm: default to the pgsty/minio image
Use the maintained Silo image for both the server and post-install mc jobs. The image bundles mcli with an mc compatibility link.

Bump the chart major version because changing the default registry can affect admission and pull policies. Keep Helm repository packaging and indexes outside this source-chart change.
2026-07-29 10:34:55 +08:00
Wesley Schwengle d495d30d57 docs: point documentation links to silo.pigsty.io (#41)
Replace upstream docs.min.io references with the SILO documentation domain and remove the upstream community path prefix.

Co-authored-by: waterkip <6317502+waterkip@users.noreply.github.com>
2026-07-21 12:14:10 +08:00
Feng Ruohang 3e61b1d3a5 chore: update Go module dependencies 2026-06-18 16:14:50 +08:00
Feng Ruohang df627ff896 fix: bump Go toolchain to 1.26.4
Update the module Go directive and release Docker build images from Go 1.26.2 to Go 1.26.4 so local, CI, hotfix, and release builds use the same patched toolchain.

Keep ordinary Go module requirements and replacements unchanged; this intentionally avoids a third-party dependency refresh while allowing container system packages to refresh through the newer golang Alpine base image and existing apk resolution.

Update the security advisory index to record the Go 1.26.4 toolchain bump alongside the earlier Go 1.26.2 security update.

Verified with go build ./..., go vet ./cmd/, and focused cmd tests. go mod tidy -diff was attempted as a read-only dependency drift check but could not complete because proxy.golang.org timed out while fetching uncached transitive test modules.

Co-authored-by: Codex <codex@openai.com>
2026-06-12 21:24:23 +08:00
Feng Ruohang 73ac524724 fix: CVE-2026-42600 remove ReadMultiple storage-REST API
The internode storage-REST ReadMultiple endpoint (/rmpl) joined
attacker-controlled Bucket/Prefix/Files into a filesystem path with no
validation, letting a peer with internode credentials read files outside
the drive root (GHSA-xh8f-g2qw-gcm7).

ReadMultiple has had no production caller since upstream #20390 removed
the last one (listParts) in Sep 2024; multipart now uses ReadParts (/rps).
Following the upstream fix, remove the whole API instead of validating
paths: route constant and registration, server handler, REST client
wrapper, the StorageAPI/xlStorage/xlStorageDiskIDCheck methods, the
storageMetricReadMultiple metric, and the ReadMultipleReq/Resp datatypes.
Regenerated the msgp and stringer outputs; storageRESTVersion stays at v63.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-06-12 21:08:59 +08:00
Feng Ruohang fd69c89d05 fix: complete CVE-2026-39414 S3 Select record limit enforcement
Route JSON Lines through the bounded PReader path so oversized records are rejected consistently instead of bypassing the limit on SIMD-capable CPUs.

Preserve S3 Select error codes in stream error events, wrap JSON parser errors as JSONParsingError, and flush completed records before returning a terminal error event. Add regression coverage for oversized JSON Lines input and error code preservation.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-06-12 21:08:59 +08:00
Feng Ruohang 5e40665acd fix: harden LDAP STS rate-limit source bucketing
Remove the per-username LDAP STS throttle bucket and keep the limiter keyed only by source IP. A username bucket is shared across all clients and lets one source keep a known account's bucket drained with bad-password attempts, locking the legitimate user out before LDAP bind.

For trusted proxies, stop trusting the left-most forwarded address. Resolve X-Forwarded-For right-to-left, skip trusted proxy hops, reject catch-all trusted-proxy CIDRs, and intentionally ignore RFC 7239 Forwarded for this security-sensitive bucket. Document that X-Real-IP is trusted verbatim and must be overwritten by the trusted proxy, not passed through from clients.

Update focused limiter, source-IP, trusted-proxy, and LDAP config tests to cover source-only buckets, spoofed appended XFF, multi-hop trusted proxies, Forwarded fallback, catch-all rejection, and the X-Real-IP deployment contract.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-06-12 21:08:59 +08:00
ping 65795ee1f4 fix: implement Flush on trackingResponseWriter
Allows the trackingResponseWriter to properly proxy flush
requests to the underlying http.Flusher. This ensures
compatibility with streaming responses.
2026-05-02 09:53:11 +02:00
Feng Ruohang f48dbe777d docs: refresh security docs and fork references
Update the STS, security, select, and Docker documentation to reflect the recent hardening work, including LDAP STS throttling details, OIDC JWT verification changes, and the new pgsty-specific security policy and advisory index.

Rewrite repository and raw-document links that still pointed at minio/minio so the docs consistently reference pgsty/minio instead.

The core idea is to keep the documentation aligned with the fork's actual security behavior, ownership, and upgrade guidance without mixing in unrelated code changes.
2026-04-17 15:06:42 +08:00
Feng Ruohang f44110890b fix: tighten LDAP STS rate-limit accounting
Prevent LDAP STS reservation cancel paths from over-crediting rate-limit buckets by capping refill and refund capacity against in-flight reservations.

Add an explicit trusted-proxy allowlist for LDAP STS source bucketing, prefer clean X-Real-IP values on trusted peers, and extend tests/docs for the new behavior.
2026-04-16 23:22:13 +08:00
Feng Ruohang 9e10f6d9a0 fix: harden LDAP STS rate-limit source IP
Use the socket peer address for LDAP STS per-IP rate limiting instead of the generic forwarded-header-aware helper. This keeps the security-sensitive rate-limit key from trusting spoofable X-Forwarded-For, X-Real-IP, and Forwarded headers while leaving the rest of the source-IP behavior unchanged.

Add focused regression coverage for RemoteAddr parsing, header spoofing, and peer-address bucket selection.
2026-04-16 21:21:53 +08:00
Feng Ruohang 18b712d49a fix: preserve LDAP STS rate limits without penalizing success 2026-04-16 17:56:51 +08:00
Feng Ruohang db4c0fd5e3 fix: bump Go to 1.26.2 for upstream security fixes
Update go.mod and all golang build images from 1.26.1 to 1.26.2 to pick up the upstream 2026-04-07 security release.

This includes fixes for CVE-2026-32280 / CVE-2026-32281 in crypto/x509, CVE-2026-32283 in crypto/tls, and the related toolchain and standard library security fixes shipped in go1.26.2, without changing any unrelated dependencies.
2026-04-16 15:08:42 +08:00
Feng Ruohang efb6e5b00b fix: fake CVE-2026-40028 harden snowball unsigned-trailer auth
Track issue #28 / GHSA-9c4q-hq6p-c237 as fake CVE-2026-40028. Close the Snowball auto-extract auth gap in PutObjectExtractHandler by treating authTypeStreamingUnsignedTrailer the same as ordinary PUTs: honor X-Amz-Decoded-Content-Length, initialize newUnsignedV4ChunkedReader(), and verify the SigV4 request before any tar bytes reach untar(). This removes the forged-signature write primitive that let a single request fan out into arbitrary extracted object creation.

Add regression coverage for forged-signature Snowball unsigned-trailer writes, anonymous Snowball requests against non-public buckets, and legitimate signed Snowball extraction with trailing CRC32 trailers. Validate the new tests against the vulnerable parent and patched tree, and confirm with containerized before/after smoke runs that the exploit succeeds pre-fix, fails post-fix, and normal signed Snowball uploads still extract correctly.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-04-16 14:46:45 +08:00
Feng Ruohang f444b6f37e fix: fake CVE-2026-40027 block unsigned-trailer query auth bypass
Track issue #27 / GHSA-hv4r-mvr4-25vw as fake CVE-2026-40027. Close the unsigned-trailer trust flaw that let query-string credentials skip signature verification in PutObject and PutObjectPart by moving presigned rejection and SigV4 verification into newUnsignedV4ChunkedReader(), so authTypeStreamingUnsignedTrailer can no longer silently downgrade query auth into an anonymous body read.

Add focused regression coverage for forged query-string-only unsigned-trailer PUTs and multipart uploads, mixed header/query auth rejection, and anonymous unsigned-trailer writes that remain allowed only when bucket policy explicitly permits them. Validate the new tests against the vulnerable parent and confirm with before/after live-server runs that presigned unsigned-trailer attacks are rejected while legitimate header-authenticated and policy-driven flows still work.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-04-16 14:46:44 +08:00
Feng Ruohang 3252d5b7f3 fix: CVE-2026-39414 harden S3 Select oversized record handling
Enforce the 1 MiB maxCharsPerRecord limit while splitting CSV and line-delimited JSON input so oversized records are rejected before they can be buffered and parsed.

Return OverMaxRecordSize for these failures instead of collapsing them into InternalError, and preserve splitter errors in the JSON worker so oversized-record failures are not lost after successful partial decode.
2026-04-15 22:55:52 +08:00
Feng Ruohang 56fa63bfd1 fix: CVE-2026-34204 block replication metadata injection
Close the replication-header trust flaw that allowed ordinary PutObject and CopyObject requests to smuggle X-Minio-Replication-* headers into X-Minio-Internal-* SSE metadata and write objects into an unreadable state. Stop accepting replication-only metadata in the default extraction path, restore it only after a trusted replication write has passed ReplicateObjectAction, and tighten CopyObject by sanitizing replication-only request headers before metadata, precondition, and SSE-C source handling consume them. Also gate replica status writes on the same trusted replication path and restore replication SSE metadata in multipart and snowball upload flows so legitimate replication continues to work.

Add focused regression coverage for untrusted PUT and COPY header poisoning at the handler layer, plus helper tests for trusted vs untrusted metadata extraction and CopyObject header sanitization. Validate the new tests against both the patched tree and the vulnerable HEAD baseline, and confirm with live server before/after runs that malicious PUT/COPY requests no longer turn objects unreadable.

Co-authored-by: Codex <codex@openai.com>
Co-Authored-By: Claude <noreply@anthropic.com>
2026-04-15 18:36:49 +08:00
Feng Ruohang 3b950f8fa8 fix: CVE-2026-33419 harden LDAP STS auth
Prevent username enumeration in AssumeRoleWithLDAPIdentity by returning the same external STS error for unknown users and invalid passwords, while preserving LDAP infrastructure failures as upstream errors so they continue to surface as 500s and remain visible in server logs.

Add a small in-memory rate limiter for LDAP STS login attempts, keyed by source IP and normalized username, and add regression coverage for auth failure classification, throttling, and Docker-backed LDAP end-to-end flows.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-04-15 14:02:41 +08:00
Feng Ruohang d24f449e08 fix: CVE-2026-33322 harden OIDC JWT verification
Close the OIDC JWT algorithm confusion flaw in AssumeRoleWithWebIdentity by restoring a JWKS-only verification path. Stop injecting the client secret into the verifier keyring and restrict accepted signing methods to the asymmetric algorithms already supported by the existing JWKS flow.

Add regression coverage to verify HS256 tokens are rejected, RS256 tokens remain valid, and JWKS refresh and retry logic cannot bypass the method allowlist.

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
2026-04-15 14:02:41 +08:00
Feng Ruohang e4fa063942 Merge pull request #19 from mfredenhagen/mario/main/CVE-2026-39883-fix
Bump go.opentelemetry.io version to address CVE-2026-39883
2026-04-13 00:12:10 +08:00
Feng Ruohang ff58df9499 Merge pull request #18 from ZouhairCharef/master
fix: upgrade go-jose to v4.1.4 to patch CVE-2026-34986
2026-04-13 00:11:43 +08:00
Mario Fredenhagen 1869bd30b8 Bump go.opentelemetry.io version to address CVE-2026-39883 2026-04-10 14:33:39 +02:00
Zouhair EC-charef 68e0ba9971 fix: upgrade go-jose to v4.1.4 to patch CVE-2026-34986
Updates github.com/go-jose/go-jose/v4 from v4.1.3 to v4.1.4 to fix a high-severity denial of service vulnerability (CVE-2026-34986).
2026-04-06 20:34:51 +01:00
Feng Ruohang ce1c537eb1 fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
Replace minio/pkg/v3 with pgsty/minio-pkg/v3 v3.6.3 to fix LDAP TLS
regression where DialURL() was not passing TLS config for ldaps://
connections, causing InsecureSkipVerify and RootCAs to be silently
ignored (x509: certificate signed by unknown authority).

Pin four dependencies to avoid breaking changes introduced in 5abd9a80f:
- go-ldap/ldap/v3 v3.4.12: v3.4.13 rewrote GetLDAPError() internals
- IBM/sarama v1.45.1: v1.46.0 changed Kafka protocol version negotiation
- lib/pq v1.10.9: v1.11.0 treats nil []byte as NULL and drops PG <14
- etcd v3.6.8: stay on intermediate version per policy

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 12:24:00 +08:00
Feng Ruohang ee55e5391a remove upstream CI/CD workflows inherited from minio/minio
We maintain our own release pipeline (release.yml, test-release.yml)
and have no use for the upstream test/lint/integration workflows.
They reference infrastructure and secrets we don't have, and the
PR-triggered jobs never fire since we don't take external pull requests.

Closes #14

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 09:41:26 +08:00
Feng Ruohang f2f9a40dce add mcli/mc from pgsty/mc to Docker image
Rework Dockerfile.goreleaser to download the latest mcli binary from
pgsty/mc GitHub releases, verify its SHA-256 checksum, and install both
mcli and mc (symlink) into the final image alongside minio and curl.
Also add download-static-curl.sh to goreleaser extra_files and enable
workflow_dispatch for the release workflow.
2026-03-24 09:15:52 +08:00
Feng Ruohang 377fc616d9 fix: satisfy stricter Go 1.26.1 linter checks
Go 1.26.1 tightens a few toolchain checks that older builds tolerated.\n\nCast aliased replication status values back to their defining type before calling the generated msgp helpers, and replace Sprintf+WriteString pairs with direct Fprintf calls where needed.\n\nThese are compatibility-only source changes to keep the cmd package building cleanly under the newer linker/toolchain.
2026-03-21 13:49:36 +08:00
Feng Ruohang 5abd9a80f6 bump golang to 1.26.1 and update deps 2026-03-21 13:41:04 +08:00
Feng Ruohang 00f3cf74fc RELEASE.2026-03-14T12-00-00Z with go 1.26.0
Switch to community-maintained console fork (georgmangold/console v1.9.1)
and update dependencies accordingly. Fix go vet format directive in
grid_test.go and adapt test status code for Go 1.26 HTTP semantics.
2026-03-14 17:39:57 +08:00
Feng Ruohang 68521b37f2 add github ci/cd pipeline 2026-02-18 10:00:20 +08:00
Feng Ruohang 8630937e7d Restore embedded console and update README for community fork
- Revert console dependency from stripped v1.7.7-pre to v1.7.6,
  restoring the full embedded management console
- Rewrite README disclaimer with proper trademark attribution and
  nominative fair use language for AGPL compliance
- Update documentation links and Go module paths to this repository
- Restore docs removed upstream (hotfixes.md, metrics/v3.md)
- Restore feature request issue template
- Update Go version to 1.26.0

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 11:47:30 +08:00
Feng Ruohang d4cd4b4337 RELEASE.2025-12-03T12-00-00Z with go 1.25.5 2025-12-15 16:39:39 +08:00
Harshavardhana 27742d4694 update README.md maintenance mode 2025-12-03 00:13:11 -08:00
Krutika Dhananjay 58659f26f4 Drop v3 metrics from community docs (#21678) 2025-11-06 02:38:16 -08:00
dorman 3a0cc6c86e fix doc 404 (#21670) 2025-10-26 19:47:37 -07:00
yangw 10b0a234d2 fix: update metric descriptions to specify current MinIO server instance (#21638)
Signed-off-by: yangw <wuyangmuc@gmail.com>
2025-10-23 21:06:31 -07:00
Raul-Mircea Crivineanu 18f97e70b1 Updates for conditional put read quorum issue (#21653) 2025-10-23 21:05:31 -07:00
Menno Finlay-Smits 52eee5a2f1 fix(api): Don't send multiple responses for one request (#21651)
fix(api): Don't send responses twice.

In some cases multiple responses are being sent for one request, causing
the API server to incorrectly drop connections.

This change introduces a ResponseWriter which tracks whether a
response has already been sent. This is used to prevent a response being
sent if something already has (e.g. by a preconditions check function).

Fixes #21633.

Co-authored-by: Menno Finlay-Smits <hello@menno.io>
2025-10-23 21:05:19 -07:00
Rishabh Agrahari c6d3aac5c4 Fix typo in entrypoint script path in README (#21657) 2025-10-23 08:10:39 -07:00
M Alvee fa18589d1c fix: Tagging in PostPolicy upload does not enforce policy tags (#21656) 2025-10-23 08:10:12 -07:00
Harshavardhana 05e569960a update scripts pointing to internal registry for community releases 2025-10-19 01:22:05 -07:00
173 changed files with 5168 additions and 4168 deletions
+1 -9
View File
@@ -1,20 +1,12 @@
---
name: Bug report
about: Report a bug in MinIO (community edition is source-only)
about: Create a report to help us improve
title: ''
labels: community, triage
assignees: ''
---
## IMPORTANT NOTES
**Community Edition**: MinIO community edition is now source-only. Install via `go install github.com/minio/minio@latest`
**Feature Requests**: We are no longer accepting feature requests for the community edition. For feature requests and enterprise support, please subscribe to [MinIO Enterprise Support](https://min.io/pricing).
**Urgent Issues**: If this case is urgent or affects production, please subscribe to [SUBNET](https://min.io/pricing) for 24/7 enterprise support.
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
+20
View File
@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: community, triage
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+108
View File
@@ -0,0 +1,108 @@
version: 2
env:
- CGO_ENABLED=0
builds:
- id: minio
main: .
binary: minio
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64
goamd64:
- v1
flags:
- -tags=kqueue
- -trimpath
ldflags:
- "{{ .Env.LDFLAGS }}"
archives:
- id: minio
ids:
- minio
name_template: "minio_{{ .Env.PKG_VERSION }}_{{ .Os }}_{{ .Arch }}"
dockers:
- id: minio-amd64
ids:
- minio
goos: linux
goarch: amd64
dockerfile: Dockerfile.goreleaser
use: buildx
image_templates:
- "pgsty/minio:{{ .Tag }}-amd64"
- "pgsty/minio:latest-amd64"
build_flag_templates:
- "--platform=linux/amd64"
- "--label=org.opencontainers.image.version={{ .Tag }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- dockerscripts/docker-entrypoint.sh
- dockerscripts/download-static-curl.sh
- LICENSE
- CREDITS
- id: minio-arm64
ids:
- minio
goos: linux
goarch: arm64
dockerfile: Dockerfile.goreleaser
use: buildx
image_templates:
- "pgsty/minio:{{ .Tag }}-arm64"
- "pgsty/minio:latest-arm64"
build_flag_templates:
- "--platform=linux/arm64"
- "--label=org.opencontainers.image.version={{ .Tag }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- dockerscripts/docker-entrypoint.sh
- dockerscripts/download-static-curl.sh
- LICENSE
- CREDITS
docker_manifests:
- name_template: "pgsty/minio:{{ .Tag }}"
image_templates:
- "pgsty/minio:{{ .Tag }}-amd64"
- "pgsty/minio:{{ .Tag }}-arm64"
- name_template: "pgsty/minio:latest"
image_templates:
- "pgsty/minio:latest-amd64"
- "pgsty/minio:latest-arm64"
checksum:
name_template: "minio_{{ .Env.PKG_VERSION }}_checksums.txt"
algorithm: sha256
release:
github:
owner: pgsty
name: minio
draft: false
prerelease: false
mode: replace
replace_existing_artifacts: true
name_template: "{{ .Tag }}"
changelog:
sort: asc
filters:
exclude:
- "^docs:"
- "^test:"
- "Merge pull request"
- "Merge branch"
announce:
skip: true
+6 -1
View File
@@ -1 +1,6 @@
<svg data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 162.612 24.465"><path d="M52.751.414h9.108v23.63h-9.108zM41.711.74l-18.488 9.92a.919.919 0 0 1-.856 0L3.879.74A2.808 2.808 0 0 0 2.558.414h-.023A2.4 2.4 0 0 0 0 2.641v21.376h9.1V13.842a.918.918 0 0 1 1.385-.682l10.361 5.568a3.634 3.634 0 0 0 3.336.028l10.933-5.634a.917.917 0 0 1 1.371.69v10.205h9.1V2.641A2.4 2.4 0 0 0 43.055.414h-.023a2.808 2.808 0 0 0-1.321.326zm65.564-.326h-9.237v10.755a.913.913 0 0 1-1.338.706L72.762.675a2.824 2.824 0 0 0-1.191-.261h-.016a2.4 2.4 0 0 0-2.535 2.227v21.377h9.163V13.275a.914.914 0 0 1 1.337-.707l24.032 11.2a2.813 2.813 0 0 0 1.188.26 2.4 2.4 0 0 0 2.535-2.227zm7.161 23.63V.414h4.191v23.63zm28.856.421c-11.274 0-19.272-4.7-19.272-12.232C124.02 4.741 132.066 0 143.292 0s19.32 4.7 19.32 12.233-7.902 12.232-19.32 12.232zm0-21.333c-8.383 0-14.84 3.217-14.84 9.1 0 5.926 6.457 9.1 14.84 9.1s14.887-3.174 14.887-9.1c0-5.883-6.504-9.1-14.887-9.1z" fill="#c72c48"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="255 293 539 140" fill="none">
<path d="M255.222 385.861 304.199 383.375c1.061 6.446 3.22 11.358 6.478 14.734 5.303 5.465 12.879 8.197 22.728 8.197 7.348 0 13.011-1.397 16.988-4.19 3.977-2.794 5.966-6.032 5.966-9.716 0-3.5-1.894-6.631-5.682-9.394-3.788-2.763-12.576-5.372-26.364-7.828-22.576-4.113-38.674-9.577-48.295-16.392-9.697-6.815-14.546-15.502-14.546-26.062 0-6.938 2.481-13.491 7.443-19.662 4.962-6.17 12.424-11.02 22.386-14.55 9.962-3.53 23.617-5.295 40.966-5.295 21.288 0 37.519 3.208 48.693 9.624 11.174 6.416 17.822 16.622 19.943 30.621l-48.523 2.302c-1.288-6.078-3.996-10.498-8.125-13.261-4.129-2.763-9.83-4.144-17.103-4.144-5.984 0-10.492 1.028-13.522 3.085-3.03 2.056-4.545 4.558-4.545 7.505 0 2.149 1.25 4.083 3.75 5.802 2.424 1.78 8.182 3.438 17.273 4.973 22.5 3.93 38.617 7.905 48.352 11.926 9.735 4.022 16.818 9.01 21.25 14.965 4.432 5.956 6.648 12.617 6.648 19.984 0 8.657-2.954 16.638-8.863 23.944-5.909 7.306-14.167 12.847-24.773 16.623-10.606 3.776-23.977 5.664-40.113 5.664-28.333 0-47.954-4.42-58.863-13.261-10.909-8.841-17.083-20.076-18.522-33.706Z" fill="#287CAB"/>
<path d="M434.578 295.519h43.304v135.007h-43.304z" fill="#287CAB"/>
<path d="M517.881 295.519h43.209v101.762h67.436v33.245H517.881Z" fill="#287CAB"/>
<path d="M639.4 363.115c0-22.041 6.359-39.201 19.077-51.48 12.718-12.28 30.427-18.419 53.129-18.419 23.274 0 41.206 6.032 53.796 18.096 12.591 12.064 18.886 28.963 18.886 50.697 0 15.778-2.75 28.717-8.251 38.817-5.5 10.099-13.449 17.958-23.846 23.575-10.397 5.618-23.353 8.427-38.869 8.427-15.77 0-28.822-2.425-39.155-7.275-10.333-4.85-18.711-12.525-25.133-23.023-6.423-10.499-9.634-23.637-9.634-39.416Zm43.209.184c0 13.63 2.623 23.422 7.869 29.377 5.246 5.956 12.384 8.934 21.413 8.934 9.284 0 16.47-2.916 21.557-8.749 5.087-5.833 7.631-16.301 7.631-31.403 0-12.709-2.655-21.995-7.965-27.858-5.31-5.863-12.511-8.795-21.604-8.795-8.712 0-15.707 2.978-20.985 8.933-5.278 5.956-7.916 15.81-7.916 29.561Z" fill="#287CAB"/>
</svg>

Before

Width:  |  Height:  |  Size: 978 B

After

Width:  |  Height:  |  Size: 2.0 KiB

-14
View File
@@ -1,14 +0,0 @@
name: 'Dependency Review'
on: [pull_request]
permissions:
contents: read
jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- name: 'Checkout Repository'
uses: actions/checkout@v4
- name: 'Dependency Review'
uses: actions/dependency-review-action@v4
-39
View File
@@ -1,39 +0,0 @@
name: Crosscompile
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Build Tests with Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Build on ${{ matrix.os }}
if: matrix.os == 'ubuntu-latest'
env:
CGO_ENABLED: 0
GO111MODULE: on
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make crosscompile
-44
View File
@@ -1,44 +0,0 @@
name: Healing Functional Tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Build on ${{ matrix.os }}
if: matrix.os == 'ubuntu-latest'
env:
CGO_ENABLED: 0
GO111MODULE: on
MINIO_KMS_SECRET_KEY: "my-minio-key:oyArl7zlPECEduNbB1KXgdzDn2Bdpvvw0l8VO51HQnY="
MINIO_KMS_AUTO_ENCRYPTION: on
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make verify-healing
make verify-healing-inconsistent-versions
make verify-healing-with-root-disks
make verify-healing-with-rewrite
-42
View File
@@ -1,42 +0,0 @@
name: Linters and Tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Build on ${{ matrix.os }}
if: matrix.os == 'ubuntu-latest'
env:
CGO_ENABLED: 0
GO111MODULE: on
run: |
sudo apt install jq -y
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make
make test
make test-race
-39
View File
@@ -1,39 +0,0 @@
name: Resiliency Functional Tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Build on ${{ matrix.os }}
if: matrix.os == 'ubuntu-latest'
env:
CGO_ENABLED: 0
GO111MODULE: on
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-resiliency
-42
View File
@@ -1,42 +0,0 @@
name: Functional Tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }} - healing
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Build on ${{ matrix.os }}
if: matrix.os == 'ubuntu-latest'
env:
CGO_ENABLED: 0
GO111MODULE: on
MINIO_KMS_SECRET_KEY: "my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
MINIO_KMS_AUTO_ENCRYPTION: on
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make verify
make test-timeout
-30
View File
@@ -1,30 +0,0 @@
name: Helm Chart linting
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Helm
uses: azure/setup-helm@v4
- name: Run helm lint
run: |
cd helm/minio
helm lint .
-161
View File
@@ -1,161 +0,0 @@
name: IAM integration
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
iam-matrix-test:
name: "[Go=${{ matrix.go-version }}|ldap=${{ matrix.ldap }}|etcd=${{ matrix.etcd }}|openid=${{ matrix.openid }}]"
runs-on: ubuntu-latest
services:
openldap:
image: quay.io/minio/openldap
ports:
- "389:389"
- "636:636"
env:
LDAP_ORGANIZATION: "MinIO Inc"
LDAP_DOMAIN: "min.io"
LDAP_ADMIN_PASSWORD: "admin"
etcd:
image: "quay.io/coreos/etcd:v3.5.1"
env:
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS: "http://0.0.0.0:2379"
ports:
- "2379:2379"
options: >-
--health-cmd "etcdctl endpoint health"
--health-interval 10s
--health-timeout 5s
--health-retries 5
openid:
image: quay.io/minio/dex
ports:
- "5556:5556"
env:
DEX_LDAP_SERVER: "openldap:389"
openid2:
image: quay.io/minio/dex
ports:
- "5557:5557"
env:
DEX_LDAP_SERVER: "openldap:389"
DEX_ISSUER: "http://127.0.0.1:5557/dex"
DEX_WEB_HTTP: "0.0.0.0:5557"
strategy:
# When ldap, etcd or openid vars are empty below, those external servers
# are turned off - i.e. if ldap="", then ldap server is not enabled for
# the tests.
matrix:
go-version: [1.24.x]
ldap: ["", "localhost:389"]
etcd: ["", "http://localhost:2379"]
openid: ["", "http://127.0.0.1:5556/dex"]
exclude:
# exclude combos where all are empty.
- ldap: ""
etcd: ""
openid: ""
# exclude combos where both ldap and openid IDPs are specified.
- ldap: "localhost:389"
openid: "http://127.0.0.1:5556/dex"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Test LDAP/OpenID/Etcd combo
env:
_MINIO_LDAP_TEST_SERVER: ${{ matrix.ldap }}
_MINIO_ETCD_TEST_SERVER: ${{ matrix.etcd }}
_MINIO_OPENID_TEST_SERVER: ${{ matrix.openid }}
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-iam
- name: Test with multiple OpenID providers
if: matrix.openid == 'http://127.0.0.1:5556/dex'
env:
_MINIO_LDAP_TEST_SERVER: ${{ matrix.ldap }}
_MINIO_ETCD_TEST_SERVER: ${{ matrix.etcd }}
_MINIO_OPENID_TEST_SERVER: ${{ matrix.openid }}
_MINIO_OPENID_TEST_SERVER_2: "http://127.0.0.1:5557/dex"
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-iam
- name: Test with Access Management Plugin enabled
env:
_MINIO_LDAP_TEST_SERVER: ${{ matrix.ldap }}
_MINIO_ETCD_TEST_SERVER: ${{ matrix.etcd }}
_MINIO_OPENID_TEST_SERVER: ${{ matrix.openid }}
_MINIO_POLICY_PLUGIN_TEST_ENDPOINT: "http://127.0.0.1:8080"
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
go run docs/iam/access-manager-plugin.go &
make test-iam
- name: Test MinIO Old Version data to IAM import current version
if: matrix.ldap == 'ldaphost:389'
env:
_MINIO_LDAP_TEST_SERVER: ${{ matrix.ldap }}
run: |
make test-iam-ldap-upgrade-import
- name: Test LDAP for automatic site replication
if: matrix.ldap == 'localhost:389'
run: |
make test-site-replication-ldap
- name: Test OIDC for automatic site replication
if: matrix.openid == 'http://127.0.0.1:5556/dex'
run: |
make test-site-replication-oidc
iam-import-with-missing-entities:
name: Test IAM import in new cluster with missing entities
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Checkout minio-iam-testing
uses: actions/checkout@v4
with:
repository: minio/minio-iam-testing
path: minio-iam-testing
- name: Test import of IAM artifacts when in fresh cluster there are missing groups etc
run: |
make test-iam-import-with-missing-entities
iam-import-with-openid:
name: Test IAM import in new cluster with opendid configurations
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Checkout minio-iam-testing
uses: actions/checkout@v4
with:
repository: minio/minio-iam-testing
path: minio-iam-testing
- name: Test import of IAM artifacts when in fresh cluster with openid configurations
run: |
make test-iam-import-with-openid
-18
View File
@@ -1,18 +0,0 @@
# @format
name: Issue Workflow
on:
issues:
types:
- opened
jobs:
add-to-project:
name: Add issue to project
runs-on: ubuntu-latest
steps:
- uses: actions/add-to-project@v0.5.0
with:
project-url: https://github.com/orgs/miniohq/projects/2
github-token: ${{ secrets.BOT_PAT }}
-24
View File
@@ -1,24 +0,0 @@
name: 'Lock Threads'
on:
schedule:
- cron: '0 0 * * *'
workflow_dispatch:
permissions:
issues: write
concurrency:
group: lock
jobs:
action:
runs-on: ubuntu-latest
steps:
- uses: dessant/lock-threads@v3
with:
github-token: ${{ github.token }}
issue-inactive-days: '365'
exclude-any-issue-labels: 'do-not-close'
issue-lock-reason: 'resolved'
log-output: true
-81
View File
@@ -1,81 +0,0 @@
name: Mint Tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
mint-test:
runs-on: mint
timeout-minutes: 120
steps:
- name: cleanup #https://github.com/actions/checkout/issues/273
run: |
sudo -S rm -rf ${GITHUB_WORKSPACE}
mkdir ${GITHUB_WORKSPACE}
- name: checkout-step
uses: actions/checkout@v4
- name: setup-go-step
uses: actions/setup-go@v5
with:
go-version: 1.24.x
- name: github sha short
id: vars
run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
- name: build-minio
run: |
TAG="quay.io/minio/minio:${{ steps.vars.outputs.sha_short }}" make docker
- name: multipart uploads test
run: |
${GITHUB_WORKSPACE}/.github/workflows/multipart/migrate.sh "${{ steps.vars.outputs.sha_short }}"
- name: compress and encrypt
run: |
${GITHUB_WORKSPACE}/.github/workflows/run-mint.sh "compress-encrypt" "minio" "minio123" "${{ steps.vars.outputs.sha_short }}"
- name: multiple pools
run: |
${GITHUB_WORKSPACE}/.github/workflows/run-mint.sh "pools" "minio" "minio123" "${{ steps.vars.outputs.sha_short }}"
- name: standalone erasure
run: |
${GITHUB_WORKSPACE}/.github/workflows/run-mint.sh "erasure" "minio" "minio123" "${{ steps.vars.outputs.sha_short }}"
# FIXME: renable this back when we have a valid way to add deadlines for PUT()s (internode CreateFile)
# - name: resiliency
# run: |
# ${GITHUB_WORKSPACE}/.github/workflows/run-mint.sh "resiliency" "minio" "minio123" "${{ steps.vars.outputs.sha_short }}"
- name: The job must cleanup
if: ${{ always() }}
run: |
export JOB_NAME=${{ steps.vars.outputs.sha_short }}
for mode in $(echo compress-encrypt pools erasure); do
docker-compose -f ${GITHUB_WORKSPACE}/.github/workflows/mint/minio-${mode}.yaml down || true
docker-compose -f ${GITHUB_WORKSPACE}/.github/workflows/mint/minio-${mode}.yaml rm || true
done
docker-compose -f ${GITHUB_WORKSPACE}/.github/workflows/multipart/docker-compose-site1.yaml rm -s -f || true
docker-compose -f ${GITHUB_WORKSPACE}/.github/workflows/multipart/docker-compose-site2.yaml rm -s -f || true
for volume in $(docker volume ls -q | grep minio); do
docker volume rm ${volume} || true
done
docker rmi -f quay.io/minio/minio:${{ steps.vars.outputs.sha_short }}
docker system prune -f || true
docker volume prune -f || true
docker volume rm $(docker volume ls -q -f dangling=true) || true
@@ -1,80 +0,0 @@
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:${JOB_NAME}
command: server --console-address ":9001" http://minio{1...4}/cdata{1...2}
expose:
- "9000"
- "9001"
environment:
MINIO_CI_CD: "on"
MINIO_ROOT_USER: "minio"
MINIO_ROOT_PASSWORD: "minio123"
MINIO_COMPRESSION_ENABLE: "on"
MINIO_COMPRESSION_MIME_TYPES: "*"
MINIO_COMPRESSION_ALLOW_ENCRYPTION: "on"
MINIO_KMS_SECRET_KEY: "my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
minio1:
<<: *minio-common
hostname: minio1
volumes:
- cdata1-1:/cdata1
- cdata1-2:/cdata2
minio2:
<<: *minio-common
hostname: minio2
volumes:
- cdata2-1:/cdata1
- cdata2-2:/cdata2
minio3:
<<: *minio-common
hostname: minio3
volumes:
- cdata3-1:/cdata1
- cdata3-2:/cdata2
minio4:
<<: *minio-common
hostname: minio4
volumes:
- cdata4-1:/cdata1
- cdata4-2:/cdata2
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
volumes:
- ./nginx-4-node.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
depends_on:
- minio1
- minio2
- minio3
- minio4
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
cdata1-1:
cdata1-2:
cdata2-1:
cdata2-2:
cdata3-1:
cdata3-2:
cdata4-1:
cdata4-2:
-51
View File
@@ -1,51 +0,0 @@
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:${JOB_NAME}
command: server --console-address ":9001" edata{1...4}
expose:
- "9000"
- "9001"
environment:
MINIO_CI_CD: "on"
MINIO_ROOT_USER: "minio"
MINIO_ROOT_PASSWORD: "minio123"
MINIO_KMS_SECRET_KEY: "my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
minio1:
<<: *minio-common
hostname: minio1
volumes:
- edata1-1:/edata1
- edata1-2:/edata2
- edata1-3:/edata3
- edata1-4:/edata4
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
volumes:
- ./nginx-1-node.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
depends_on:
- minio1
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
edata1-1:
edata1-2:
edata1-3:
edata1-4:
-117
View File
@@ -1,117 +0,0 @@
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:${JOB_NAME}
command: server --console-address ":9001" http://minio{1...4}/pdata{1...2} http://minio{5...8}/pdata{1...2}
expose:
- "9000"
- "9001"
environment:
MINIO_CI_CD: "on"
MINIO_ROOT_USER: "minio"
MINIO_ROOT_PASSWORD: "minio123"
MINIO_KMS_SECRET_KEY: "my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
minio1:
<<: *minio-common
hostname: minio1
volumes:
- pdata1-1:/pdata1
- pdata1-2:/pdata2
minio2:
<<: *minio-common
hostname: minio2
volumes:
- pdata2-1:/pdata1
- pdata2-2:/pdata2
minio3:
<<: *minio-common
hostname: minio3
volumes:
- pdata3-1:/pdata1
- pdata3-2:/pdata2
minio4:
<<: *minio-common
hostname: minio4
volumes:
- pdata4-1:/pdata1
- pdata4-2:/pdata2
minio5:
<<: *minio-common
hostname: minio5
volumes:
- pdata5-1:/pdata1
- pdata5-2:/pdata2
minio6:
<<: *minio-common
hostname: minio6
volumes:
- pdata6-1:/pdata1
- pdata6-2:/pdata2
minio7:
<<: *minio-common
hostname: minio7
volumes:
- pdata7-1:/pdata1
- pdata7-2:/pdata2
minio8:
<<: *minio-common
hostname: minio8
volumes:
- pdata8-1:/pdata1
- pdata8-2:/pdata2
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
volumes:
- ./nginx-8-node.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
depends_on:
- minio1
- minio2
- minio3
- minio4
- minio5
- minio6
- minio7
- minio8
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
pdata1-1:
pdata1-2:
pdata2-1:
pdata2-2:
pdata3-1:
pdata3-2:
pdata4-1:
pdata4-2:
pdata5-1:
pdata5-2:
pdata6-1:
pdata6-2:
pdata7-1:
pdata7-2:
pdata8-1:
pdata8-2:
@@ -1,78 +0,0 @@
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:${JOB_NAME}
command: server --console-address ":9001" http://minio{1...4}/rdata{1...2}
expose:
- "9000"
- "9001"
environment:
MINIO_CI_CD: "on"
MINIO_ROOT_USER: "minio"
MINIO_ROOT_PASSWORD: "minio123"
MINIO_KMS_SECRET_KEY: "my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
MINIO_DRIVE_MAX_TIMEOUT: "5s"
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
minio1:
<<: *minio-common
hostname: minio1
volumes:
- rdata1-1:/rdata1
- rdata1-2:/rdata2
minio2:
<<: *minio-common
hostname: minio2
volumes:
- rdata2-1:/rdata1
- rdata2-2:/rdata2
minio3:
<<: *minio-common
hostname: minio3
volumes:
- rdata3-1:/rdata1
- rdata3-2:/rdata2
minio4:
<<: *minio-common
hostname: minio4
volumes:
- rdata4-1:/rdata1
- rdata4-2:/rdata2
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
volumes:
- ./nginx-4-node.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
depends_on:
- minio1
- minio2
- minio3
- minio4
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
rdata1-1:
rdata1-2:
rdata2-1:
rdata2-2:
rdata3-1:
rdata3-2:
rdata4-1:
rdata4-2:
-100
View File
@@ -1,100 +0,0 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server minio1:9000;
}
upstream console {
ip_hash;
server minio1:9001;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9001;
listen [::]:9001;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}
-105
View File
@@ -1,105 +0,0 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server minio1:9000 max_fails=1 fail_timeout=10s;
server minio2:9000 max_fails=1 fail_timeout=10s;
server minio3:9000 max_fails=1 fail_timeout=10s;
}
upstream console {
ip_hash;
server minio1:9001;
server minio2:9001;
server minio3:9001;
server minio4:9001;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9001;
listen [::]:9001;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}
-114
View File
@@ -1,114 +0,0 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server minio1:9000 max_fails=1 fail_timeout=10s;
server minio2:9000 max_fails=1 fail_timeout=10s;
server minio3:9000 max_fails=1 fail_timeout=10s;
server minio4:9000 max_fails=1 fail_timeout=10s;
server minio5:9000 max_fails=1 fail_timeout=10s;
server minio6:9000 max_fails=1 fail_timeout=10s;
server minio7:9000 max_fails=1 fail_timeout=10s;
server minio8:9000 max_fails=1 fail_timeout=10s;
}
upstream console {
ip_hash;
server minio1:9001;
server minio2:9001;
server minio3:9001;
server minio4:9001;
server minio5:9001;
server minio6:9001;
server minio7:9001;
server minio8:9001;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9001;
listen [::]:9001;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}
-106
View File
@@ -1,106 +0,0 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server minio1:9000 max_fails=1 fail_timeout=10s;
server minio2:9000 max_fails=1 fail_timeout=10s;
server minio3:9000 max_fails=1 fail_timeout=10s;
server minio4:9000 max_fails=1 fail_timeout=10s;
}
upstream console {
ip_hash;
server minio1:9001;
server minio2:9001;
server minio3:9001;
server minio4:9001;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9001;
listen [::]:9001;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}
@@ -1,66 +0,0 @@
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:${RELEASE}
command: server http://site1-minio{1...4}/data{1...2}
environment:
- MINIO_PROMETHEUS_AUTH_TYPE=public
- CI=true
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
site1-minio1:
<<: *minio-common
hostname: site1-minio1
volumes:
- site1-data1-1:/data1
- site1-data1-2:/data2
site1-minio2:
<<: *minio-common
hostname: site1-minio2
volumes:
- site1-data2-1:/data1
- site1-data2-2:/data2
site1-minio3:
<<: *minio-common
hostname: site1-minio3
volumes:
- site1-data3-1:/data1
- site1-data3-2:/data2
site1-minio4:
<<: *minio-common
hostname: site1-minio4
volumes:
- site1-data4-1:/data1
- site1-data4-2:/data2
site1-nginx:
image: nginx:1.19.2-alpine
hostname: site1-nginx
volumes:
- ./nginx-site1.conf:/etc/nginx/nginx.conf:ro
ports:
- "9001:9001"
depends_on:
- site1-minio1
- site1-minio2
- site1-minio3
- site1-minio4
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
site1-data1-1:
site1-data1-2:
site1-data2-1:
site1-data2-2:
site1-data3-1:
site1-data3-2:
site1-data4-1:
site1-data4-2:
@@ -1,66 +0,0 @@
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:${RELEASE}
command: server http://site2-minio{1...4}/data{1...2}
environment:
- MINIO_PROMETHEUS_AUTH_TYPE=public
- CI=true
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
site2-minio1:
<<: *minio-common
hostname: site2-minio1
volumes:
- site2-data1-1:/data1
- site2-data1-2:/data2
site2-minio2:
<<: *minio-common
hostname: site2-minio2
volumes:
- site2-data2-1:/data1
- site2-data2-2:/data2
site2-minio3:
<<: *minio-common
hostname: site2-minio3
volumes:
- site2-data3-1:/data1
- site2-data3-2:/data2
site2-minio4:
<<: *minio-common
hostname: site2-minio4
volumes:
- site2-data4-1:/data1
- site2-data4-2:/data2
site2-nginx:
image: nginx:1.19.2-alpine
hostname: site2-nginx
volumes:
- ./nginx-site2.conf:/etc/nginx/nginx.conf:ro
ports:
- "9002:9002"
depends_on:
- site2-minio1
- site2-minio2
- site2-minio3
- site2-minio4
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
site2-data1-1:
site2-data1-2:
site2-data2-1:
site2-data2-2:
site2-data3-1:
site2-data3-2:
site2-data4-1:
site2-data4-2:
-147
View File
@@ -1,147 +0,0 @@
#!/bin/bash
set -x
## change working directory
cd .github/workflows/multipart/
function cleanup() {
docker-compose -f docker-compose-site1.yaml rm -s -f || true
docker-compose -f docker-compose-site2.yaml rm -s -f || true
for volume in $(docker volume ls -q | grep minio); do
docker volume rm ${volume} || true
done
docker system prune -f || true
docker volume prune -f || true
docker volume rm $(docker volume ls -q -f dangling=true) || true
}
cleanup
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
export RELEASE=RELEASE.2023-08-29T23-07-35Z
docker-compose -f docker-compose-site1.yaml up -d
docker-compose -f docker-compose-site2.yaml up -d
sleep 30s
./mc alias set site1 http://site1-nginx:9001 minioadmin minioadmin --api s3v4
./mc alias set site2 http://site2-nginx:9002 minioadmin minioadmin --api s3v4
./mc ready site1/
./mc ready site2/
./mc admin replicate add site1 site2
./mc mb site1/testbucket/
./mc cp -r --quiet /usr/bin site1/testbucket/
sleep 5
./s3-check-md5 -h
failed_count_site1=$(./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://site1-nginx:9001 -bucket testbucket 2>&1 | grep FAILED | wc -l)
failed_count_site2=$(./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://site2-nginx:9002 -bucket testbucket 2>&1 | grep FAILED | wc -l)
if [ $failed_count_site1 -ne 0 ]; then
echo "failed with multipart on site1 uploads"
exit 1
fi
if [ $failed_count_site2 -ne 0 ]; then
echo "failed with multipart on site2 uploads"
exit 1
fi
./mc cp -r --quiet /usr/bin site1/testbucket/
sleep 5
failed_count_site1=$(./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://site1-nginx:9001 -bucket testbucket 2>&1 | grep FAILED | wc -l)
failed_count_site2=$(./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://site2-nginx:9002 -bucket testbucket 2>&1 | grep FAILED | wc -l)
## we do not need to fail here, since we are going to test
## upgrading to master, healing and being able to recover
## the last version.
if [ $failed_count_site1 -ne 0 ]; then
echo "failed with multipart on site1 uploads ${failed_count_site1}"
fi
if [ $failed_count_site2 -ne 0 ]; then
echo "failed with multipart on site2 uploads ${failed_count_site2}"
fi
export RELEASE=${1}
docker-compose -f docker-compose-site1.yaml up -d
docker-compose -f docker-compose-site2.yaml up -d
./mc ready site1/
./mc ready site2/
for i in $(seq 1 10); do
# mc admin heal -r --remove when used against a LB endpoint
# behaves flaky, let this run 10 times before giving up
./mc admin heal -r --remove --json site1/ 2>&1 >/dev/null
./mc admin heal -r --remove --json site2/ 2>&1 >/dev/null
done
failed_count_site1=$(./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://site1-nginx:9001 -bucket testbucket 2>&1 | grep FAILED | wc -l)
failed_count_site2=$(./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://site2-nginx:9002 -bucket testbucket 2>&1 | grep FAILED | wc -l)
if [ $failed_count_site1 -ne 0 ]; then
echo "failed with multipart on site1 uploads"
exit 1
fi
if [ $failed_count_site2 -ne 0 ]; then
echo "failed with multipart on site2 uploads"
exit 1
fi
# Add user group test
./mc admin user add site1 site-replication-issue-user site-replication-issue-password
./mc admin group add site1 site-replication-issue-group site-replication-issue-user
max_wait_attempts=30
wait_interval=5
attempt=1
while true; do
diff <(./mc admin group info site1 site-replication-issue-group) <(./mc admin group info site2 site-replication-issue-group)
if [[ $? -eq 0 ]]; then
echo "Outputs are consistent."
break
fi
remaining_attempts=$((max_wait_attempts - attempt))
if ((attempt >= max_wait_attempts)); then
echo "Outputs remain inconsistent after $max_wait_attempts attempts. Exiting with error."
exit 1
else
echo "Outputs are inconsistent. Waiting for $wait_interval seconds (attempt $attempt/$max_wait_attempts)."
sleep $wait_interval
fi
((attempt++))
done
status=$(./mc admin group info site1 site-replication-issue-group --json | jq .groupStatus | tr -d '"')
if [[ $status == "enabled" ]]; then
echo "Success"
else
echo "Expected status: enabled, actual status: $status"
exit 1
fi
cleanup
## change working directory
cd ../../../
@@ -1,61 +0,0 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server site1-minio1:9000;
server site1-minio2:9000;
server site1-minio3:9000;
server site1-minio4:9000;
}
server {
listen 9001;
listen [::]:9001;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
}
@@ -1,61 +0,0 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server site2-minio1:9000;
server site2-minio2:9000;
server site2-minio3:9000;
server site2-minio4:9000;
}
server {
listen 9002;
listen [::]:9002;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
}
+141
View File
@@ -0,0 +1,141 @@
name: Release
on:
push:
tags:
- "RELEASE.*"
workflow_dispatch:
inputs:
tag:
description: "Release tag (e.g. RELEASE.2026-03-24T12-00-00Z)"
required: true
permissions:
contents: write
packages: write
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
- name: Compute release variables
run: |
set -euo pipefail
TAG="${{ github.event.inputs.tag || github.ref_name }}"
VERSION_HYPHEN="${TAG#RELEASE.}"
PKG_VERSION="$(echo "${VERSION_HYPHEN}" | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/\1\2\3\4\5\6.0.0/')"
if [ "${PKG_VERSION}" = "${VERSION_HYPHEN}" ]; then
echo "Invalid release tag format: ${TAG}"
exit 1
fi
VERSION_COLON="$(echo "${VERSION_HYPHEN}" | sed -E 's/T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/T\1:\2:\3Z/')"
LDFLAGS="$(MINIO_RELEASE=RELEASE go run buildscripts/gen-ldflags.go "${VERSION_COLON}")"
{
echo "RELEASE_TAG=${TAG}"
echo "PKG_VERSION=${PKG_VERSION}"
echo "LDFLAGS=${LDFLAGS}"
} >> "${GITHUB_ENV}"
echo "Release tag: ${TAG}"
echo "Package version: ${PKG_VERSION}"
echo "LDFLAGS: ${LDFLAGS}"
- name: Validate Docker Hub credentials
run: |
set -euo pipefail
if [ -z "${{ secrets.DOCKERHUB_USERNAME }}" ] || [ -z "${{ secrets.DOCKERHUB_TOKEN }}" ]; then
echo "Missing Docker Hub credentials. Set DOCKERHUB_USERNAME and DOCKERHUB_TOKEN repository secrets."
exit 1
fi
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
with:
platforms: arm64
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and publish with GoReleaser
uses: goreleaser/goreleaser-action@v6
with:
version: "~> v2"
args: release --clean --skip=validate --config .github/goreleaser.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LDFLAGS: ${{ env.LDFLAGS }}
PKG_VERSION: ${{ env.PKG_VERSION }}
- name: Install pkger
run: |
go install github.com/minio/pkger/v2@v2.6.18
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
- name: Prepare package layout
run: |
set -euo pipefail
copy_binary() {
local arch="$1"
local pattern="$2"
local src
src="$(find dist -maxdepth 2 -type f -path "dist/${pattern}/minio" | head -n1 || true)"
if [ -z "${src}" ]; then
echo "Missing GoReleaser binary for ${arch} (${pattern})"
exit 1
fi
mkdir -p "dist/linux-${arch}"
cp "${src}" "dist/linux-${arch}/minio.${RELEASE_TAG}"
}
copy_binary amd64 "minio_linux_amd64*"
copy_binary arm64 "minio_linux_arm64*"
- name: Build standard pkger packages
run: |
set -euo pipefail
pkger -r "${RELEASE_TAG}" --appName minio --releaseDir dist --ignore
# Keep only full package files; drop convenience symlinks (minio.rpm/minio.deb/minio.apk)
find dist/linux-* -maxdepth 1 -type l \
\( -name 'minio.rpm' -o -name 'minio.deb' -o -name 'minio.apk' \) -delete
find dist -maxdepth 2 -type f \
\( -name '*.rpm' -o -name '*.deb' -o -name '*.apk' -o -name '*.sha256sum' -o -name 'downloads-minio.json' \) | sort
- name: Upload pkger artifacts to GitHub release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
mapfile -t files < <(find dist -maxdepth 2 -type f \
\( -name '*.rpm' -o -name '*.deb' -o -name '*.apk' -o -name '*.sha256sum' -o -name 'downloads-minio.json' \) | sort)
if [ "${#files[@]}" -eq 0 ]; then
echo "No packages were generated."
exit 1
fi
gh release upload "${RELEASE_TAG}" "${files[@]}" --clobber
- name: Upload dist artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: dist
path: dist/
-79
View File
@@ -1,79 +0,0 @@
name: MinIO advanced tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
replication-test:
name: Advanced Tests with Go ${{ matrix.go-version }}
runs-on: ubuntu-latest
strategy:
matrix:
go-version: [1.24.x]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Test Decom
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-decom
- name: Test ILM
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-ilm
make test-ilm-transition
- name: Test PBAC
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-pbac
- name: Test Config File
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-configfile
- name: Test Replication
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-replication
- name: Test MinIO IDP for automatic site replication
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-site-replication-minio
- name: Test Versioning
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-versioning
- name: Test Multipart upload with failures
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-multipart
-34
View File
@@ -1,34 +0,0 @@
name: Root lockdown tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Start root lockdown tests
run: |
make test-root-disable
-9
View File
@@ -1,9 +0,0 @@
-----BEGIN CERTIFICATE-----
MIIBKDCB26ADAgECAhB6vebGMUfKnmBKyqoApRSOMAUGAytlcDAbMRkwFwYDVQQD
DBByb290QHBsYXkubWluLmlvMB4XDTIwMDQzMDE1MjIyNVoXDTI1MDQyOTE1MjIy
NVowGzEZMBcGA1UEAwwQcm9vdEBwbGF5Lm1pbi5pbzAqMAUGAytlcAMhALzn735W
fmSH/ghKs+4iPWziZMmWdiWr/sqvqeW+WwSxozUwMzAOBgNVHQ8BAf8EBAMCB4Aw
EwYDVR0lBAwwCgYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAFBgMrZXADQQDZOrGK
b2ATkDlu2pTcP3LyhSBDpYh7V4TvjRkBTRgjkacCzwFLm+mh+7US8V4dBpIDsJ4u
uWoF0y6vbLVGIlkG
-----END CERTIFICATE-----
-3
View File
@@ -1,3 +0,0 @@
-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEID9E7FSYWrMD+VjhI6q545cYT9YOyFxZb7UnjEepYDRc
-----END PRIVATE KEY-----
-64
View File
@@ -1,64 +0,0 @@
#!/bin/bash
set -ex
export MODE="$1"
export ACCESS_KEY="$2"
export SECRET_KEY="$3"
export JOB_NAME="$4"
export MINT_MODE="full"
docker system prune -f || true
docker volume prune -f || true
docker volume rm $(docker volume ls -f dangling=true) || true
## change working directory
cd .github/workflows/mint
## always pull latest
docker pull docker.io/minio/mint:edge
docker-compose -f minio-${MODE}.yaml up -d
sleep 1m
docker system prune -f || true
docker volume prune -f || true
docker volume rm $(docker volume ls -q -f dangling=true) || true
# Stop two nodes, one of each pool, to check that all S3 calls work while quorum is still there
[ "${MODE}" == "pools" ] && docker-compose -f minio-${MODE}.yaml stop minio2
[ "${MODE}" == "pools" ] && docker-compose -f minio-${MODE}.yaml stop minio6
# Pause one node, to check that all S3 calls work while one node goes wrong
[ "${MODE}" == "resiliency" ] && docker-compose -f minio-${MODE}.yaml pause minio4
docker run --rm --net=mint_default \
--name="mint-${MODE}-${JOB_NAME}" \
-e SERVER_ENDPOINT="nginx:9000" \
-e ACCESS_KEY="${ACCESS_KEY}" \
-e SECRET_KEY="${SECRET_KEY}" \
-e ENABLE_HTTPS=0 \
-e MINT_MODE="${MINT_MODE}" \
docker.io/minio/mint:edge
# FIXME: enable this after fixing aws-sdk-java-v2 tests
# # unpause the node, to check that all S3 calls work while one node goes wrong
# [ "${MODE}" == "resiliency" ] && docker-compose -f minio-${MODE}.yaml unpause minio4
# [ "${MODE}" == "resiliency" ] && docker run --rm --net=mint_default \
# --name="mint-${MODE}-${JOB_NAME}" \
# -e SERVER_ENDPOINT="nginx:9000" \
# -e ACCESS_KEY="${ACCESS_KEY}" \
# -e SECRET_KEY="${SECRET_KEY}" \
# -e ENABLE_HTTPS=0 \
# -e MINT_MODE="${MINT_MODE}" \
# docker.io/minio/mint:edge
docker-compose -f minio-${MODE}.yaml down || true
sleep 10s
docker system prune -f || true
docker volume prune -f || true
docker volume rm $(docker volume ls -q -f dangling=true) || true
## change working directory
cd ../../../
-22
View File
@@ -1,22 +0,0 @@
name: Shell formatting checks
on:
pull_request:
branches:
- master
permissions:
contents: read
jobs:
build:
name: runner / shfmt
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: luizm/action-sh-checker@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SHFMT_OPTS: "-s"
with:
sh_checker_shellcheck_disable: true # disable for now
+89
View File
@@ -0,0 +1,89 @@
name: Test Release Pipeline
on:
workflow_dispatch:
pull_request:
paths:
- ".github/goreleaser.yml"
- "Dockerfile.goreleaser"
- "minio.service"
- ".github/workflows/release.yml"
- ".github/workflows/test-release.yml"
permissions:
contents: read
jobs:
validate:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
- name: Compute test variables
run: |
set -euo pipefail
RELEASE_TAG="RELEASE.2026-02-14T12-00-00Z"
VERSION_COLON="2026-02-14T12:00:00Z"
PKG_VERSION="20260214120000.0.0"
LDFLAGS="$(MINIO_RELEASE=RELEASE go run buildscripts/gen-ldflags.go "${VERSION_COLON}")"
echo "RELEASE_TAG=${RELEASE_TAG}" >> "${GITHUB_ENV}"
echo "PKG_VERSION=${PKG_VERSION}" >> "${GITHUB_ENV}"
echo "LDFLAGS=${LDFLAGS}" >> "${GITHUB_ENV}"
echo "PKG_VERSION: ${PKG_VERSION}"
echo "LDFLAGS: ${LDFLAGS}"
- name: GoReleaser config check
uses: goreleaser/goreleaser-action@v6
with:
version: "~> v2"
args: check --config .github/goreleaser.yml
- name: Build snapshot artifacts
uses: goreleaser/goreleaser-action@v6
with:
version: "~> v2"
args: release --snapshot --clean --skip=publish,docker --config .github/goreleaser.yml
env:
LDFLAGS: ${{ env.LDFLAGS }}
PKG_VERSION: ${{ env.PKG_VERSION }}
- name: Install pkger
run: |
go install github.com/minio/pkger/v2@v2.6.18
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
- name: Package snapshot binaries with pkger
run: |
set -euo pipefail
copy_binary() {
local arch="$1"
local pattern="$2"
local src
src="$(find dist -maxdepth 2 -type f -path "dist/${pattern}/minio" | head -n1 || true)"
if [ -z "${src}" ]; then
echo "Missing GoReleaser binary for ${arch} (${pattern})"
exit 1
fi
mkdir -p "dist/linux-${arch}"
cp "${src}" "dist/linux-${arch}/minio.${RELEASE_TAG}"
}
copy_binary amd64 "minio_linux_amd64*"
copy_binary arm64 "minio_linux_arm64*"
pkger -r "${RELEASE_TAG}" --appName minio --releaseDir dist --ignore
# Keep only full package files; drop convenience symlinks (minio.rpm/minio.deb/minio.apk)
find dist/linux-* -maxdepth 1 -type l \
\( -name 'minio.rpm' -o -name 'minio.deb' -o -name 'minio.apk' \) -delete
find dist -maxdepth 2 -type f \
\( -name '*.rpm' -o -name '*.deb' -o -name '*.apk' -o -name '*.sha256sum' -o -name 'downloads-minio.json' \) | sort
-15
View File
@@ -1,15 +0,0 @@
---
name: Spelling
on: [pull_request]
jobs:
run:
name: Spell Check with Typos
runs-on: ubuntu-latest
steps:
- name: Checkout Actions Repository
uses: actions/checkout@v4
- name: Check spelling of repo
uses: crate-ci/typos@master
-34
View File
@@ -1,34 +0,0 @@
name: Upgrade old version tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.24.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Start upgrade tests
run: |
make test-upgrade
+14 -9
View File
@@ -1,31 +1,36 @@
name: VulnCheck
on:
pull_request:
branches:
- master
push:
branches:
- master
workflow_dispatch:
permissions:
contents: read # to fetch code (actions/checkout)
contents: read
jobs:
vulncheck:
name: Analysis
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Check out code into the Go module directory
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: 1.24.x
cached: false
- name: Get official govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest
shell: bash
go-version-file: go.mod
cache: true
- name: Install govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@v1.6.0
echo "$(go env GOPATH)/bin" >> "${GITHUB_PATH}"
- name: Run govulncheck
run: govulncheck -show verbose ./...
shell: bash
+11
View File
@@ -53,3 +53,14 @@ s3-check-md5
s3-verify
xattr
xl-meta
.gitignore
.goreleaser.yml
dist/
.claude/
.codex/
_bmad/
_bmad-output/
+7 -1
View File
@@ -1,8 +1,14 @@
FROM minio/minio:latest
ARG TARGETARCH
ARG RELEASE
RUN chmod -R 777 /usr/bin
COPY ./minio /usr/bin/minio
COPY ./minio-${TARGETARCH}.${RELEASE} /usr/bin/minio
COPY ./minio-${TARGETARCH}.${RELEASE}.minisig /usr/bin/minio.minisig
COPY ./minio-${TARGETARCH}.${RELEASE}.sha256sum /usr/bin/minio.sha256sum
COPY dockerscripts/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
+96
View File
@@ -0,0 +1,96 @@
FROM golang:1.26.5-alpine AS build
ARG TARGETARCH
ENV GOPATH=/go
ENV CGO_ENABLED=0
ARG MC_REPO=pgsty/mc
ARG MC_VERSION=latest
RUN apk add -U --no-cache \
ca-certificates \
bash \
curl \
jq && \
case "${TARGETARCH}" in \
amd64) MC_ARCH=amd64 ;; \
arm64) MC_ARCH=arm64 ;; \
*) echo "Unsupported TARGETARCH=${TARGETARCH}"; exit 1 ;; \
esac && \
if [ "${MC_VERSION}" = "latest" ]; then \
MC_RELEASE_URL="https://api.github.com/repos/${MC_REPO}/releases/latest"; \
else \
MC_RELEASE_URL="https://api.github.com/repos/${MC_REPO}/releases/tags/${MC_VERSION}"; \
fi && \
curl -fsSL "${MC_RELEASE_URL}" -o /tmp/mc-release.json && \
MC_ARCHIVE_URL=$(jq -r --arg arch "${MC_ARCH}" \
'.assets[] | select(.name | endswith("_linux_" + $arch + ".tar.gz")) | .browser_download_url' \
/tmp/mc-release.json | head -n 1) && \
MC_CHECKSUM_URL=$(jq -r \
'.assets[] | select(.name | endswith("_checksums.txt")) | .browser_download_url' \
/tmp/mc-release.json | head -n 1) && \
[ -n "${MC_ARCHIVE_URL}" ] || { echo "Cannot find mcli archive for linux/${MC_ARCH}"; exit 1; } && \
[ -n "${MC_CHECKSUM_URL}" ] || { echo "Cannot find mcli checksums file"; exit 1; } && \
ARCHIVE_NAME=$(basename "${MC_ARCHIVE_URL}") && \
echo "Downloading ${ARCHIVE_NAME} ..." && \
curl -fsSL "${MC_ARCHIVE_URL}" -o /tmp/mcli.tar.gz && \
curl -fsSL "${MC_CHECKSUM_URL}" -o /tmp/mcli_checksums.txt && \
EXPECTED=$(grep " ${ARCHIVE_NAME}$" /tmp/mcli_checksums.txt | awk '{print $1}') && \
ACTUAL=$(sha256sum /tmp/mcli.tar.gz | awk '{print $1}') && \
[ -n "${EXPECTED}" ] || { echo "Checksum entry not found for ${ARCHIVE_NAME}"; exit 1; } && \
[ "${EXPECTED}" = "${ACTUAL}" ] || { echo "Checksum mismatch: expected ${EXPECTED}, got ${ACTUAL}"; exit 1; } && \
echo "Checksum OK: ${ACTUAL}" && \
mkdir -p /tmp/mcli-extract && \
tar -xzf /tmp/mcli.tar.gz -C /tmp/mcli-extract/ && \
if [ -f /tmp/mcli-extract/mcli ]; then \
cp /tmp/mcli-extract/mcli /go/bin/mcli; \
elif [ -f /tmp/mcli-extract/mc ]; then \
cp /tmp/mcli-extract/mc /go/bin/mcli; \
else \
echo "No mc or mcli binary found in archive:"; ls -la /tmp/mcli-extract/; exit 1; \
fi && \
chmod +x /go/bin/mcli && \
ln -sf mcli /go/bin/mc
COPY dockerscripts/download-static-curl.sh /build/download-static-curl
RUN chmod +x /build/download-static-curl && \
/build/download-static-curl
FROM registry.access.redhat.com/ubi9/ubi:latest AS certs
RUN dnf -y install ca-certificates && \
update-ca-trust && \
cp /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /tmp/ca-certificates.crt && \
dnf clean all && \
rm -rf /var/cache/dnf
FROM registry.access.redhat.com/ubi9/ubi-micro:latest
LABEL maintainer="pgsty <https://github.com/pgsty/minio>" \
description="MinIO community fork, build by pgsty"
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key \
MINIO_ROOT_USER_FILE=access_key \
MINIO_ROOT_PASSWORD_FILE=secret_key \
MINIO_KMS_SECRET_KEY_FILE=kms_master_key \
MINIO_UPDATE_MINISIGN_PUBKEY="RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav" \
MINIO_CONFIG_ENV_FILE=config.env \
MC_CONFIG_DIR=/tmp/.mc
COPY --from=certs /tmp/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY minio /usr/bin/minio
COPY --from=build /go/bin/mcli /usr/bin/mcli
COPY --from=build /go/bin/curl* /usr/bin/
COPY dockerscripts/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
COPY LICENSE /licenses/LICENSE
COPY CREDITS /licenses/CREDITS
RUN chmod +x /usr/bin/minio /usr/bin/mcli /usr/bin/docker-entrypoint.sh && \
ln -sf mcli /usr/bin/mc
EXPOSE 9000
VOLUME ["/data"]
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
CMD ["minio"]
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine as build
FROM golang:1.26.5-alpine as build
ARG TARGETARCH
ARG RELEASE
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine AS build
FROM golang:1.26.5-alpine AS build
ARG TARGETARCH
ARG RELEASE
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine AS build
FROM golang:1.26.5-alpine AS build
ARG TARGETARCH
ARG RELEASE
+1 -1
View File
@@ -32,7 +32,7 @@ verifiers: lint check-gen
check-gen: ## check for updated autogenerated files
@go generate ./... >/dev/null
@go mod tidy -compat=1.21
@go mod tidy -compat=1.26
@(! git diff --name-only | grep '_gen.go$$') || (echo "Non-committed changes in auto-generated code is detected, please commit them to proceed." && false)
@(! git diff --name-only | grep 'go.sum') || (echo "Non-committed changes in auto-generated go.sum is detected, please commit them to proceed." && false)
+99 -119
View File
@@ -1,160 +1,140 @@
# MinIO Quickstart Guide
[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/) [![license](https://img.shields.io/badge/license-AGPL%20V3-blue)](https://github.com/minio/minio/blob/master/LICENSE)
<h1 align="center">Silo</h1>
[![MinIO](https://raw.githubusercontent.com/minio/minio/master/.github/logo.svg?sanitize=true)](https://min.io)
<p align="center">
<strong>A conservatively maintained MinIO fork</strong><br>
Security maintenance, versioned release artifacts, and operational continuity for existing deployments.
</p>
MinIO is a high-performance, S3-compatible object storage solution released under the GNU AGPL v3.0 license.
Designed for speed and scalability, it powers AI/ML, analytics, and data-intensive workloads with industry-leading performance.
<p align="center">
<a href="README_ZH.md">中文</a> ·
<a href="https://silo.pigsty.io">Documentation</a> ·
<a href="https://github.com/pgsty/minio/releases">Releases</a> ·
<a href="https://hub.docker.com/r/pgsty/minio">Container Images</a> ·
<a href="SECURITY.md">Security</a>
</p>
- S3 API Compatible Seamless integration with existing S3 tools
- Built for AI & Analytics Optimized for large-scale data pipelines
- High Performance Ideal for demanding storage workloads.
<p align="center">
<a href="https://github.com/pgsty/minio/releases"><img alt="GitHub Release" src="https://img.shields.io/github/v/release/pgsty/minio?include_prereleases&label=release&logo=github"></a>
<a href="https://hub.docker.com/r/pgsty/minio"><img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/pgsty/minio?logo=docker"></a>
<a href="go.mod"><img alt="Go Version" src="https://img.shields.io/github/go-mod/go-version/pgsty/minio?logo=go"></a>
<a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-AGPLv3-blue"></a>
</p>
This README provides instructions for building MinIO from source and deploying onto baremetal hardware.
Use the [MinIO Documentation](https://github.com/minio/docs) project to build and host a local copy of the documentation.
> [!IMPORTANT]
> Silo is an independent, community-maintained fork of the open-source MinIO server, published by [Pigsty](https://pigsty.io) from [`pgsty/minio`](https://github.com/pgsty/minio). It is not affiliated with, endorsed by, or sponsored by MinIO, Inc. “MinIO” is used only to identify the upstream project and compatibility lineage.
## MinIO is Open Source Software
## Overview
We designed MinIO as Open Source software for the Open Source software community. We encourage the community to remix, redesign, and reshare MinIO under the terms of the AGPLv3 license.
Silo maintains one downstream release line based on MinIO [`RELEASE.2025-12-03T12-00-00Z`](https://github.com/minio/minio/releases/tag/RELEASE.2025-12-03T12-00-00Z). It provides maintained builds and release artifacts for existing MinIO-compatible deployments after upstream community distribution ended.
All usage of MinIO in your application stack requires validation against AGPLv3 obligations, which include but are not limited to the release of modified code to the community from which you have benefited. Any commercial/proprietary usage of the AGPLv3 software, including repackaging or reselling services/features, is done at your own risk.
Pigsty uses this fork for object storage, including PostgreSQL backups.
The AGPLv3 provides no obligation by any party to support, maintain, or warranty the original or any modified work.
All support is provided on a best-effort basis through Github and our [Slack](https//slack.min.io) channel, and any member of the community is welcome to contribute and assist others in their usage of the software.
## Maintenance Policy
MinIO [AIStor](https://www.min.io/product/aistor) includes enterprise-grade support and licensing for workloads which require commercial or proprietary usage and production-level SLA/SLO-backed support. For more information, [reach out for a quote](https://min.io/pricing).
The active release line covers:
## Source-Only Distribution
- build and dependency maintenance;
- applicable security fixes and advisories;
- focused fixes for reproducible defects;
- versioned binaries, packages, checksums, and multi-architecture images;
- the web console, client, documentation, and Pigsty integration.
**Important:** The MinIO community edition is now distributed as source code only. We will no longer provide pre-compiled binary releases for the community version.
Changes are kept narrow and tested where practical. Maintenance is best effort; no response, remediation, or release schedule is guaranteed.
### Installing Latest MinIO Community Edition
### Out of scope
To use MinIO community edition, you have two options:
- a separate product roadmap, new storage engine, or speculative S3 features;
- broad rewrites or changes that materially expand the downstream delta;
- historical releases or multiple support branches;
- commercial support, SLAs, 24×7 coverage, or SUBNET access;
- deployment design, access control, monitoring, backup, or recovery.
1. **Install from source** using `go install github.com/minio/minio@latest` (recommended)
2. **Build a Docker image** from the provided Dockerfile
## Compatibility
See the sections below for detailed instructions on each method.
Silo aims to preserve:
### Legacy Binary Releases
- the `minio` executable and `github.com/minio/minio` module path;
- MinIO-compatible S3 APIs, configuration, environment variables, and CLI conventions;
- `RELEASE.YYYY-MM-DDTHH-MM-SSZ` tags, container entrypoints, and common deployment workflows.
Historical pre-compiled binary releases remain available for reference but are no longer maintained:
- GitHub Releases: https://github.com/minio/minio/releases
- Direct downloads: https://dl.min.io/server/minio/release/
Compatibility is a goal, not a guarantee. Security fixes may change behavior. Treat each release as a downstream upgrade: pin versions, review [release notes](https://github.com/pgsty/minio/releases) and [security advisories](docs/security/advisories.md), keep a rollback path, and test before production use.
**These legacy binaries will not receive updates.** We strongly recommend using source builds for access to the latest features, bug fixes, and security updates.
## Release Artifacts
## Install from Source
| Artifact | Location |
| :-- | :-- |
| Source | [`github.com/pgsty/minio`](https://github.com/pgsty/minio) |
| Container image | [`pgsty/minio`](https://hub.docker.com/r/pgsty/minio), multi-arch for `linux/amd64` and `linux/arm64` |
| Server binaries | [GitHub Releases](https://github.com/pgsty/minio/releases) for Linux, macOS, and Windows on `amd64` and `arm64` |
| Linux packages | RPM, DEB, and APK artifacts, also distributed through the [Pigsty repository](https://pigsty.io/docs/repo/) |
| Client | [`pgsty/mc`](https://github.com/pgsty/mc), bundled in the container as `mcli` with an `mc` compatibility alias |
| Console | Maintained [`georgmangold/console`](https://github.com/georgmangold/console) fork, embedded in the server build |
| Documentation | [English](https://silo.pigsty.io), [Chinese](https://silo.pigsty.cc), and [`pgsty/minio-docs`](https://github.com/pgsty/minio-docs) |
| Security record | [`SECURITY.md`](SECURITY.md), [`VULNERABILITY_REPORT.md`](VULNERABILITY_REPORT.md), and [fork advisories](docs/security/advisories.md) |
Use the following commands to compile and run a standalone MinIO server from source.
If you do not have a working Golang environment, please follow [How to install Golang](https://golang.org/doc/install). Minimum version required is [go1.24](https://golang.org/dl/#stable)
## Quick Start
```sh
go install github.com/minio/minio@latest
For local evaluation:
```bash
mkdir -p data
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=change-me-long-password
docker run -d --name silo \
-p 9000:9000 \
-p 9001:9001 \
-e MINIO_ROOT_USER \
-e MINIO_ROOT_PASSWORD \
-v "$PWD/data:/data" \
pgsty/minio:latest server /data --console-address ":9001"
```
You can alternatively run `go build` and use the `GOOS` and `GOARCH` environment variables to control the OS and architecture target.
For example:
Open the console at <http://localhost:9001>; the S3 API listens on <http://localhost:9000>.
```
env GOOS=linux GOARCh=arm64 go build
The image includes the compatible client as `mcli`:
```bash
docker exec silo mcli alias set local http://127.0.0.1:9000 \
"$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
docker exec silo mcli mb local/demo
docker exec silo mcli ls local
```
Start MinIO by running `minio server PATH` where `PATH` is any empty folder on your local filesystem.
> [!WARNING]
> For production, pin a release, use unique credentials and TLS, monitor the service, keep independent backups, and test recovery.
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`.
You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server.
Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials.
You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
Build the server from source:
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool:
```sh
mc alias set local http://localhost:9000 minioadmin minioadmin
mc admin info local
```bash
go build -o minio .
./minio --version
```
See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool.
For application developers, see <https://docs.min.io/community/minio-object-store/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
Deployment: [Silo documentation](https://silo.pigsty.io) · [Pigsty MinIO module](https://pigsty.io/docs/minio/)
> [!NOTE]
> Production environments using compiled-from-source MinIO binaries do so at their own risk.
> The AGPLv3 license provides no warranties nor liabilites for any such usage.
## Security
## Build Docker Image
Security fixes target the active `master` branch and are recorded in the [advisory log](docs/security/advisories.md). Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md) and [`VULNERABILITY_REPORT.md`](VULNERABILITY_REPORT.md). Report issues that also affect upstream MinIO there as well.
You can use the `docker build .` command to build a Docker image on your local host machine.
You must first [build MinIO](#install-from-source) and ensure the `minio` binary exists in the project root.
## Contributing
The following command builds the Docker image using the default `Dockerfile` in the root project directory with the repository and image tag `myminio:minio`
Useful contributions include security and dependency updates, reproducible bug fixes, tests, release automation, packaging, and documentation.
```sh
docker build -t myminio:minio .
```
Issues and pull requests should include the affected version, reproduction steps, impact, expected behavior, tests, and compatibility notes. Discuss large changes in an issue first.
Use `docker image ls` to confirm the image exists in your local repository.
You can run the server using standard Docker invocation:
## Background
```sh
docker run -p 9000:9000 -p 9001:9001 myminio:minio server /tmp/minio --console-address :9001
```
This project was created in response to changes in the upstream community distribution and maintenance model. The maintainers analysis, alternatives considered, and early maintenance record are documented below:
Complete documentation for building Docker containers, managing custom images, or loading images into orchestration platforms is out of scope for this documentation.
You can modify the `Dockerfile` and `dockerscripts/socker-entrypoint.sh` as-needed to reflect your specific image requirements.
| Essay | Subject |
| :-- | :-- |
| [MinIO Is Dead](https://blog.vonng.com/en/db/minio-is-dead/) | Changes to the upstream project and distribution model |
| [MinIO Is Dead, Long Live MinIO](https://blog.vonng.com/en/db/minio-resurrect/) | Establishing the fork and its release pipeline |
| [Two months into maintaining a MinIO fork](https://blog.vonng.com/en/db/minio-promise-kept/) | Initial security and maintenance work |
See the [MinIO Container](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-as-a-container.html#deploy-minio-container) documentation for more guidance on running MinIO within a Container image.
## License and Trademark
## Install using Helm Charts
There are two paths for installing MinIO onto Kubernetes infrastructure:
- Use the [MinIO Operator](https://github.com/minio/operator)
- Use the community-maintained [Helm charts](https://github.com/minio/minio/tree/master/helm/minio)
See the [MinIO Documentation](https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html) for guidance on deploying using the Operator.
The Community Helm chart has instructions in the folder-level README.
## Test MinIO Connectivity
### Test using MinIO Console
MinIO Server comes with an embedded web based object browser.
Point your web browser to <http://127.0.0.1:9000> to ensure your server has started successfully.
> [!NOTE]
> MinIO runs console on random port by default, if you wish to choose a specific port use `--console-address` to pick a specific interface and port.
### Test using MinIO Client `mc`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services.
The following commands set a local alias, validate the server information, create a bucket, copy data to that bucket, and list the contents of the bucket.
```sh
mc alias set local http://localhost:9000 minioadmin minioadmin
mc admin info
mc mb data
mc cp ~/Downloads/mydata data/
mc ls data/
```
Follow the MinIO Client [Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart) for further instructions.
## Explore Further
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)
- [MinIO Erasure Code Overview](https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html)
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
## Contribute to MinIO Project
Please follow MinIO [Contributor's Guide](https://github.com/minio/minio/blob/master/CONTRIBUTING.md) for guidance on making new contributions to the repository.
## License
- MinIO source is licensed under the [GNU AGPLv3](https://github.com/minio/minio/blob/master/LICENSE).
- MinIO [documentation](https://github.com/minio/minio/tree/master/docs) is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
- [License Compliance](https://github.com/minio/minio/blob/master/COMPLIANCE.md)
The server remains licensed under the [GNU Affero General Public License v3.0](LICENSE). See [`CREDITS`](CREDITS) for upstream authorship and attribution. MinIO is a trademark of MinIO, Inc. Silo and `pgsty/minio` are independent community efforts and are not affiliated with or endorsed by MinIO, Inc.
+146
View File
@@ -0,0 +1,146 @@
<p align="center">
<img src=".github/logo.svg" alt="Silo" width="260">
</p>
<h1 align="center">Silo</h1>
<p align="center">
<strong>审慎维护的 MinIO 社区分支</strong><br>
为现有部署提供安全维护、带版本的发行产物与持续运维支持。
</p>
<p align="center">
<a href="README.md">English</a> ·
<a href="https://silo.pigsty.cc">文档</a> ·
<a href="https://github.com/pgsty/minio/releases">版本发布</a> ·
<a href="https://hub.docker.com/r/pgsty/minio">容器镜像</a> ·
<a href="SECURITY.md">安全策略</a>
</p>
<p align="center">
<a href="https://github.com/pgsty/minio/releases"><img alt="GitHub Release" src="https://img.shields.io/github/v/release/pgsty/minio?include_prereleases&label=release&logo=github"></a>
<a href="https://hub.docker.com/r/pgsty/minio"><img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/pgsty/minio?logo=docker"></a>
<a href="go.mod"><img alt="Go Version" src="https://img.shields.io/github/go-mod/go-version/pgsty/minio?logo=go"></a>
<a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-AGPLv3-blue"></a>
</p>
> [!IMPORTANT]
> Silo 是由 [Pigsty](https://pigsty.cc) 独立维护、从 [`pgsty/minio`](https://github.com/pgsty/minio) 发布的开源 MinIO 社区分支。本项目与 MinIO, Inc. 不存在隶属、背书或赞助关系;文中使用 “MinIO” 仅用于说明上游项目及兼容谱系。
## 概述
Silo 维护一条基于 MinIO [`RELEASE.2025-12-03T12-00-00Z`](https://github.com/minio/minio/releases/tag/RELEASE.2025-12-03T12-00-00Z) 的下游版本线,为上游停止社区发行后仍在运行 MinIO 兼容部署的用户提供持续构建与发行产物。
Pigsty 使用本分支提供对象存储,包括 PostgreSQL 备份存储。
## 维护政策
活跃版本线的维护范围包括:
- 构建与依赖项维护;
- 适用的安全修复与公告;
- 针对可复现缺陷的范围明确的修复;
- 带版本的二进制、软件包、校验和与多架构镜像;
- Web Console、客户端、文档与 Pigsty 集成。
改动保持克制,并在可行时提供测试。所有维护均为尽力而为,不承诺固定的响应、修复或发布时间。
### 范围之外
- 独立产品路线图、新存储引擎或假设性的 S3 新特性;
- 大规模重写或显著扩大下游差异的改动;
- 历史版本或多条支持分支;
- 商业支持、SLA、7×24 服务或 SUBNET 服务;
- 部署设计、访问控制、监控、备份与恢复。
## 兼容策略
Silo 尽量保留:
- `minio` 可执行文件与 `github.com/minio/minio` module path
- MinIO 兼容的 S3 API、配置、环境变量与命令行约定;
- `RELEASE.YYYY-MM-DDTHH-MM-SSZ` 标签、容器入口与常见部署方式。
兼容是目标,而非保证。安全修复可能改变行为。每个版本都应视为下游升级:锁定版本,阅读[版本说明](https://github.com/pgsty/minio/releases)与[安全公告](docs/security/advisories.md),保留回滚路径,并在生产使用前完成测试。
## 发行产物
| 产物 | 位置 |
| :-- | :-- |
| 源码 | [`github.com/pgsty/minio`](https://github.com/pgsty/minio) |
| 容器镜像 | [`pgsty/minio`](https://hub.docker.com/r/pgsty/minio),支持 `linux/amd64``linux/arm64` 多架构清单 |
| 服务端二进制 | [GitHub Releases](https://github.com/pgsty/minio/releases),覆盖 Linux、macOS、Windows 的 `amd64``arm64` |
| Linux 软件包 | RPM、DEB、APK,并通过 [Pigsty 软件仓库](https://pigsty.cc/docs/repo/) 分发 |
| 客户端 | [`pgsty/mc`](https://github.com/pgsty/mc),容器内以 `mcli` 提供,并保留 `mc` 兼容别名 |
| 管理控制台 | 社区维护的 [`georgmangold/console`](https://github.com/georgmangold/console),嵌入服务端构建 |
| 文档 | [中文](https://silo.pigsty.cc)、[英文](https://silo.pigsty.io)与 [`pgsty/minio-docs`](https://github.com/pgsty/minio-docs) |
| 安全记录 | [`SECURITY.md`](SECURITY.md)、[`VULNERABILITY_REPORT.md`](VULNERABILITY_REPORT.md) 与[本分支安全公告](docs/security/advisories.md) |
## 快速开始
本地体验:
```bash
mkdir -p data
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=change-me-long-password
docker run -d --name silo \
-p 9000:9000 \
-p 9001:9001 \
-e MINIO_ROOT_USER \
-e MINIO_ROOT_PASSWORD \
-v "$PWD/data:/data" \
pgsty/minio:latest server /data --console-address ":9001"
```
管理控制台位于 <http://localhost:9001>S3 API 位于 <http://localhost:9000>。
镜像内置兼容客户端 `mcli`
```bash
docker exec silo mcli alias set local http://127.0.0.1:9000 \
"$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
docker exec silo mcli mb local/demo
docker exec silo mcli ls local
```
> [!WARNING]
> 生产环境应锁定版本,使用独立凭据与 TLS,配置监控,保留独立备份,并验证恢复流程。
从源码构建服务端:
```bash
go build -o minio .
./minio --version
```
部署:[Silo 中文文档](https://silo.pigsty.cc) · [Pigsty MinIO 模块](https://pigsty.cc/docs/minio/)
## 安全
安全修复面向活跃的 `master` 分支,并记录在[安全公告](docs/security/advisories.md)中。请按照 [`SECURITY.md`](SECURITY.md) 与 [`VULNERABILITY_REPORT.md`](VULNERABILITY_REPORT.md) 私密报告漏洞;同时影响上游 MinIO 的问题也应向上游报告。
## 参与贡献
欢迎安全与依赖项更新、可复现缺陷修复、测试、发布自动化、打包与文档改进。
Issue 与 Pull Request 应说明受影响版本、复现步骤、影响、预期行为、测试与兼容性说明。大型改动请先提交 Issue 讨论。
## 背景
本项目源于上游社区发行与维护模式的变化。维护者对相关变化的分析、替代方案评估与早期维护记录见以下文章:
| 文章 | 主题 |
| :-- | :-- |
| [MinIO已死](https://vonng.com/db/minio-is-dead/) | 上游项目与发行模式的变化 |
| [MinIO已死,谁能接盘?](https://vonng.com/db/minio-alternative/) | 可选替代方案评估 |
| [MinIO 已死,MinIO 复生](https://vonng.com/db/minio-resurrect/) | 建立分支及其发行流水线 |
| [续命 MinIO:承诺兑现](https://vonng.com/db/minio-promise-kept/) | 初期安全与维护工作 |
## 许可证与商标
服务端继续采用 [GNU Affero General Public License v3.0](LICENSE) 发布。上游作者与署名信息见 [`CREDITS`](CREDITS)。
MinIO 是 MinIO, Inc. 的商标。Silo、Pigsty 与 `pgsty/minio` 均为独立社区项目,与 MinIO, Inc. 不存在隶属或背书关系。
+10 -32
View File
@@ -1,42 +1,20 @@
# Security Policy
This repository is the `pgsty/minio` community fork of `minio/minio`. Upstream MinIO security contacts do not handle fork-specific fixes or release notes for this repository.
## Supported Versions
We always provide security updates for the [latest release](https://github.com/minio/minio/releases/latest).
Whenever there is a security update you just need to upgrade to the latest version.
Security fixes are tracked on the active `master` branch and summarized in [docs/security/advisories.md](docs/security/advisories.md).
## Reporting a Vulnerability
All security bugs in [minio/minio](https://github,com/minio/minio) (or other minio/* repositories)
should be reported by email to security@min.io. Your email will be acknowledged within 48 hours,
and you'll receive a more detailed response to your email within 72 hours indicating the next steps
in handling your report.
For vulnerabilities in this fork:
Please, provide a detailed explanation of the issue. In particular, outline the type of the security
issue (DoS, authentication bypass, information disclose, ...) and the assumptions you're making (e.g. do
you need access credentials for a successful exploit).
1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
2. Prefer the `pgsty/minio` repository's GitHub security reporting workflow when it is available.
3. If private reporting is not available, contact the maintainers through the `pgsty/minio` repository before publishing detailed exploit information.
4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.
If you have not received a reply to your email within 48 hours or you have not heard from the security team
for the past five days please contact the security team directly:
## Disclosure Process
- Primary security coordinator: aead@min.io
- Secondary coordinator: harsha@min.io
- If you receive no response: dev@min.io
### Disclosure Process
MinIO uses the following disclosure process:
1. Once the security report is received one member of the security team tries to verify and reproduce
the issue and determines the impact it has.
2. A member of the security team will respond and either confirm or reject the security report.
If the report is rejected the response explains why.
3. Code is audited to find any potential similar problems.
4. Fixes are prepared for the latest release.
5. On the date that the fixes are applied a security advisory will be published on <https://blog.min.io>.
Please inform us in your report email whether MinIO should mention your contribution w.r.t. fixing
the security issue. By default MinIO will **not** publish this information to protect your privacy.
This process can take some time, especially when coordination is required with maintainers of other projects.
Every effort will be made to handle the bug in as timely a manner as possible, however it's important that we
follow the process described above to ensure that disclosures are handled consistently.
Fork-specific fixes and user-visible upgrade notes are published in [docs/security/advisories.md](docs/security/advisories.md). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
+19 -20
View File
@@ -1,38 +1,37 @@
# Vulnerability Management Policy
This document formally describes the process of addressing and managing a
reported vulnerability that has been found in the MinIO server code base,
any directly connected ecosystem component or a direct / indirect dependency
of the code base.
This document describes how the `pgsty/minio` maintainers investigate,
assess, and remediate reported vulnerabilities affecting this fork, any
directly shipped component, or a direct / indirect dependency used by this
repository.
## Scope
The vulnerability management policy described in this document covers the
process of investigating, assessing and resolving a vulnerability report
opened by a MinIO employee or an external third party.
This policy covers vulnerability reports opened by repository maintainers or
external third parties against `pgsty/minio` itself, its release artifacts, or
dependencies that materially affect this fork.
Therefore, it lists pre-conditions and actions that should be performed to
resolve and fix a reported vulnerability.
It defines the information needed for triage and the expected remediation
workflow for supported fixes.
## Vulnerability Management Process
The vulnerability management process requires that the vulnerability report
contains the following information:
A useful vulnerability report should contain the following information:
- The project / component that contains the reported vulnerability.
- A description of the vulnerability. In particular, the type of the
reported vulnerability and how it might be exploited. Alternatively,
a well-established vulnerability identifier, e.g. CVE number, can be
used instead.
reported vulnerability and how it might be exploited. Alternatively,
a well-established vulnerability identifier, such as a CVE or GHSA ID, can
be used instead.
Based on the description mentioned above, a MinIO engineer or security team
member investigates:
Based on the report, the `pgsty/minio` maintainers investigate:
- Whether the reported vulnerability exists.
- The conditions that are required such that the vulnerability can be exploited.
- Which releases, branches, or deployment paths are affected.
- The steps required to fix the vulnerability.
In general, if the vulnerability exists in one of the MinIO code bases
itself - not in a code dependency - then MinIO will, if possible, fix
the vulnerability or implement reasonable countermeasures such that the
vulnerability cannot be exploited anymore.
If the vulnerability exists in this fork itself, the maintainers will, when
feasible, fix the issue or implement reasonable countermeasures such that the
vulnerability can no longer be exploited. Fork-specific upgrade notes and
security advisories are published in `docs/security/advisories.md`.
+1 -1
View File
@@ -267,7 +267,7 @@ func (a adminAPIHandlers) AddServiceAccountLDAP(w http.ResponseWriter, r *http.R
lookupResult, targetGroups, err = globalIAMSys.LDAPConfig.LookupUserDN(targetUser)
if err != nil {
// if not found, check if DN
if strings.Contains(err.Error(), "User DN not found for:") {
if strings.Contains(strings.ToLower(err.Error()), "user dn not found for:") {
if isDN {
// warn user that DNs are not allowed
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErrWithErr(ErrAdminLDAPExpectedLoginName, err), r.URL)
+54
View File
@@ -889,6 +889,12 @@ func generateMultiDeleteResponse(quiet bool, deletedObjects []DeletedObject, err
}
func writeResponse(w http.ResponseWriter, statusCode int, response []byte, mType mimeType) {
// Don't write a response if one has already been written.
// Fixes https://github.com/minio/minio/issues/21633
if headersAlreadyWritten(w) {
return
}
if statusCode == 0 {
statusCode = 200
}
@@ -1015,3 +1021,51 @@ func writeCustomErrorResponseJSON(ctx context.Context, w http.ResponseWriter, er
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeJSON)
}
type unwrapper interface {
Unwrap() http.ResponseWriter
}
// headersAlreadyWritten returns true if the headers have already been written
// to this response writer. It will unwrap the ResponseWriter if possible to try
// and find a trackingResponseWriter.
func headersAlreadyWritten(w http.ResponseWriter) bool {
for {
if trw, ok := w.(*trackingResponseWriter); ok {
return trw.headerWritten
} else if uw, ok := w.(unwrapper); ok {
w = uw.Unwrap()
} else {
return false
}
}
}
// trackingResponseWriter wraps a ResponseWriter and notes when WriterHeader has
// been called. This allows high level request handlers to check if something
// has already sent the header.
type trackingResponseWriter struct {
http.ResponseWriter
headerWritten bool
}
func (w *trackingResponseWriter) WriteHeader(statusCode int) {
if !w.headerWritten {
w.headerWritten = true
w.ResponseWriter.WriteHeader(statusCode)
}
}
func (w *trackingResponseWriter) Write(b []byte) (int, error) {
return w.ResponseWriter.Write(b)
}
func (w *trackingResponseWriter) Flush() {
if f, ok := w.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
func (w *trackingResponseWriter) Unwrap() http.ResponseWriter {
return w.ResponseWriter
}
+114
View File
@@ -18,8 +18,13 @@
package cmd
import (
"io"
"net/http"
"net/http/httptest"
"testing"
"github.com/klauspost/compress/gzhttp"
xhttp "github.com/minio/minio/internal/http"
)
// Tests object location.
@@ -122,3 +127,112 @@ func TestGetURLScheme(t *testing.T) {
t.Errorf("Expected %s, got %s", httpsScheme, gotScheme)
}
}
func TestTrackingResponseWriter(t *testing.T) {
rw := httptest.NewRecorder()
trw := &trackingResponseWriter{ResponseWriter: rw}
trw.WriteHeader(299)
if !trw.headerWritten {
t.Fatal("headerWritten was not set by WriteHeader call")
}
_, err := trw.Write([]byte("hello"))
if err != nil {
t.Fatalf("Write unexpectedly failed: %v", err)
}
// Check that WriteHeader and Write were called on the underlying response writer
resp := rw.Result()
if resp.StatusCode != 299 {
t.Fatalf("unexpected status: %v", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("reading response body failed: %v", err)
}
if string(body) != "hello" {
t.Fatalf("response body incorrect: %v", string(body))
}
// Check that Unwrap works
if trw.Unwrap() != rw {
t.Fatalf("Unwrap returned wrong result: %v", trw.Unwrap())
}
}
func TestTrackingResponseWriterFlush(t *testing.T) {
rw := httptest.NewRecorder()
trw := &trackingResponseWriter{ResponseWriter: rw}
trw.Flush()
if trw.headerWritten {
t.Fatal("Flush() should not set headerWritten")
}
// Simulate the ListenNotificationHandler flow: WriteHeader, Write, Flush
trw.WriteHeader(http.StatusOK)
_, err := trw.Write([]byte("event data"))
if err != nil {
t.Fatalf("Write failed: %v", err)
}
xhttp.Flush(trw)
if !rw.Flushed {
t.Fatalf("xhttp.Flush should have flushed the underlying ResponseRecorder via trackingResponseWriter.Flush()")
}
}
func TestHeadersAlreadyWritten(t *testing.T) {
rw := httptest.NewRecorder()
trw := &trackingResponseWriter{ResponseWriter: rw}
if headersAlreadyWritten(trw) {
t.Fatal("headers have not been written yet")
}
trw.WriteHeader(123)
if !headersAlreadyWritten(trw) {
t.Fatal("headers were written")
}
}
func TestHeadersAlreadyWrittenWrapped(t *testing.T) {
rw := httptest.NewRecorder()
trw := &trackingResponseWriter{ResponseWriter: rw}
wrap1 := &gzhttp.NoGzipResponseWriter{ResponseWriter: trw}
wrap2 := &gzhttp.NoGzipResponseWriter{ResponseWriter: wrap1}
if headersAlreadyWritten(wrap2) {
t.Fatal("headers have not been written yet")
}
wrap2.WriteHeader(123)
if !headersAlreadyWritten(wrap2) {
t.Fatal("headers were written")
}
}
func TestWriteResponseHeadersNotWritten(t *testing.T) {
rw := httptest.NewRecorder()
trw := &trackingResponseWriter{ResponseWriter: rw}
writeResponse(trw, 299, []byte("hello"), "application/foo")
resp := rw.Result()
if resp.StatusCode != 299 {
t.Fatal("response wasn't written")
}
}
func TestWriteResponseHeadersWritten(t *testing.T) {
rw := httptest.NewRecorder()
rw.Code = -1
trw := &trackingResponseWriter{ResponseWriter: rw, headerWritten: true}
writeResponse(trw, 200, []byte("hello"), "application/foo")
if rw.Code != -1 {
t.Fatalf("response was written when it shouldn't have been (Code=%v)", rw.Code)
}
}
+2
View File
@@ -218,6 +218,8 @@ func s3APIMiddleware(f http.HandlerFunc, flags ...s3HFlag) http.HandlerFunc {
handlerName := getHandlerName(f, "objectAPIHandlers")
var handler http.HandlerFunc = func(w http.ResponseWriter, r *http.Request) {
w = &trackingResponseWriter{ResponseWriter: w}
// Wrap the actual handler with the appropriate tracing middleware.
var tracedHandler http.HandlerFunc
if handlerFlags.has(traceHdrsS3HFlag) {
+1 -1
View File
@@ -37,7 +37,7 @@ import (
"github.com/minio/pkg/v3/wildcard"
"github.com/minio/pkg/v3/workers"
"github.com/minio/pkg/v3/xtime"
"gopkg.in/yaml.v3"
"go.yaml.in/yaml/v3"
)
// expire: # Expire objects that match a condition
+1 -1
View File
@@ -21,7 +21,7 @@ import (
"slices"
"testing"
"gopkg.in/yaml.v3"
"go.yaml.in/yaml/v3"
)
func TestParseBatchJobExpire(t *testing.T) {
+1 -1
View File
@@ -52,7 +52,7 @@ import (
"github.com/minio/pkg/v3/env"
"github.com/minio/pkg/v3/policy"
"github.com/minio/pkg/v3/workers"
"gopkg.in/yaml.v3"
"go.yaml.in/yaml/v3"
)
var globalBatchConfig batch.Config
+1 -1
View File
@@ -21,7 +21,7 @@ import (
"slices"
"testing"
"gopkg.in/yaml.v3"
"go.yaml.in/yaml/v3"
)
func TestBatchJobPrefix_UnmarshalYAML(t *testing.T) {
+1 -1
View File
@@ -24,7 +24,7 @@ import (
"github.com/dustin/go-humanize"
"github.com/minio/pkg/v3/wildcard"
"gopkg.in/yaml.v3"
"go.yaml.in/yaml/v3"
)
//go:generate msgp -file $GOFILE
+1 -1
View File
@@ -21,7 +21,7 @@ import (
"slices"
"testing"
"gopkg.in/yaml.v3"
"go.yaml.in/yaml/v3"
)
func TestParseBatchJobReplicate(t *testing.T) {
+5 -5
View File
@@ -903,7 +903,7 @@ func (z *ReplicationState) DecodeMsg(dc *msgp.Reader) (err error) {
return
}
var za0004 VersionPurgeStatusType
err = za0004.DecodeMsg(dc)
err = (*replication.VersionPurgeStatusType)(&za0004).DecodeMsg(dc)
if err != nil {
err = msgp.WrapError(err, "PurgeTargets", za0003)
return
@@ -1060,7 +1060,7 @@ func (z *ReplicationState) EncodeMsg(en *msgp.Writer) (err error) {
err = msgp.WrapError(err, "PurgeTargets")
return
}
err = za0004.EncodeMsg(en)
err = (*replication.VersionPurgeStatusType)(&za0004).EncodeMsg(en)
if err != nil {
err = msgp.WrapError(err, "PurgeTargets", za0003)
return
@@ -1136,7 +1136,7 @@ func (z *ReplicationState) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.AppendMapHeader(o, uint32(len(z.PurgeTargets)))
for za0003, za0004 := range z.PurgeTargets {
o = msgp.AppendString(o, za0003)
o, err = za0004.MarshalMsg(o)
o, err = (*replication.VersionPurgeStatusType)(&za0004).MarshalMsg(o)
if err != nil {
err = msgp.WrapError(err, "PurgeTargets", za0003)
return
@@ -1261,7 +1261,7 @@ func (z *ReplicationState) UnmarshalMsg(bts []byte) (o []byte, err error) {
err = msgp.WrapError(err, "PurgeTargets")
return
}
bts, err = za0004.UnmarshalMsg(bts)
bts, err = (*replication.VersionPurgeStatusType)(&za0004).UnmarshalMsg(bts)
if err != nil {
err = msgp.WrapError(err, "PurgeTargets", za0003)
return
@@ -1321,7 +1321,7 @@ func (z *ReplicationState) Msgsize() (s int) {
if z.PurgeTargets != nil {
for za0003, za0004 := range z.PurgeTargets {
_ = za0004
s += msgp.StringPrefixSize + len(za0003) + za0004.Msgsize()
s += msgp.StringPrefixSize + len(za0003) + (*replication.VersionPurgeStatusType)(&za0004).Msgsize()
}
}
s += 17 + msgp.MapHeaderSize
+225
View File
@@ -0,0 +1,225 @@
// Copyright (c) 2015-2025 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"testing"
"github.com/dustin/go-humanize"
xhttp "github.com/minio/minio/internal/http"
)
// TestNewMultipartUploadConditionalWithReadQuorumFailure tests that conditional
// multipart uploads (with if-match/if-none-match) behave correctly when read quorum
// cannot be reached.
//
// Related to: https://github.com/minio/minio/issues/21603
//
// Should return an error when read quorum cannot
// be reached, as we cannot reliably determine if the precondition is met.
func TestNewMultipartUploadConditionalWithReadQuorumFailure(t *testing.T) {
ctx := context.Background()
obj, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatal(err)
}
defer obj.Shutdown(context.Background())
defer removeRoots(fsDirs)
z := obj.(*erasureServerPools)
xl := z.serverPools[0].sets[0]
bucket := "test-bucket"
object := "test-object"
err = obj.MakeBucket(ctx, bucket, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
// Put an initial object so it exists
_, err = obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("initial-value")),
int64(len("initial-value")), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
// Get object info to capture the ETag
objInfo, err := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
existingETag := objInfo.ETag
// Simulate read quorum failure by taking enough disks offline
// With 16 disks (EC 8+8), read quorum is 9. Taking 8 disks offline leaves only 8,
// which is below read quorum.
erasureDisks := xl.getDisks()
z.serverPools[0].erasureDisksMu.Lock()
xl.getDisks = func() []StorageAPI {
for i := range erasureDisks[:8] {
erasureDisks[i] = nil
}
return erasureDisks
}
z.serverPools[0].erasureDisksMu.Unlock()
t.Run("if-none-match with read quorum failure", func(t *testing.T) {
// Test Case 1: if-none-match (create only if doesn't exist)
// With if-none-match: *, this should only succeed if object doesn't exist.
// Since read quorum fails, we can't determine if object exists.
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfNoneMatch: "*",
},
CheckPrecondFn: func(oi ObjectInfo) bool {
// Precondition fails if object exists (ETag is not empty)
return oi.ETag != ""
},
}
_, err := obj.NewMultipartUpload(ctx, bucket, object, opts)
if !isErrReadQuorum(err) {
t.Errorf("Expected read quorum error when if-none-match is used with quorum failure, got: %v", err)
}
})
t.Run("if-match with wrong ETag and read quorum failure", func(t *testing.T) {
// Test Case 2: if-match with WRONG ETag
// This should fail even without quorum issues, but with quorum failure
// we can't verify the ETag at all.
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfMatch: "wrong-etag-12345",
},
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
// Precondition fails if ETags don't match
return oi.ETag != "wrong-etag-12345"
},
}
_, err := obj.NewMultipartUpload(ctx, bucket, object, opts)
if !isErrReadQuorum(err) {
t.Logf("Got error (as expected): %v", err)
t.Logf("But expected read quorum error, not object-not-found error")
}
})
t.Run("if-match with correct ETag and read quorum failure", func(t *testing.T) {
// Test Case 3: if-match with CORRECT ETag but read quorum failure
// Even with the correct ETag, we shouldn't proceed if we can't verify it.
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfMatch: existingETag,
},
HasIfMatch: true,
CheckPrecondFn: func(oi ObjectInfo) bool {
// Precondition fails if ETags don't match
return oi.ETag != existingETag
},
}
_, err := obj.NewMultipartUpload(ctx, bucket, object, opts)
if !isErrReadQuorum(err) {
t.Errorf("Expected read quorum error when if-match is used with quorum failure, got: %v", err)
}
})
}
// TestCompleteMultipartUploadConditionalWithReadQuorumFailure tests that conditional
// complete multipart upload operations behave correctly when read quorum cannot be reached.
func TestCompleteMultipartUploadConditionalWithReadQuorumFailure(t *testing.T) {
ctx := context.Background()
obj, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatal(err)
}
defer obj.Shutdown(context.Background())
defer removeRoots(fsDirs)
z := obj.(*erasureServerPools)
xl := z.serverPools[0].sets[0]
bucket := "test-bucket"
object := "test-object"
err = obj.MakeBucket(ctx, bucket, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
// Put an initial object
_, err = obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("initial-value")),
int64(len("initial-value")), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
// Start a multipart upload WITHOUT conditional checks (this should work)
res, err := obj.NewMultipartUpload(ctx, bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
// Upload a part
partData := bytes.Repeat([]byte("a"), 5*humanize.MiByte)
md5Hex := getMD5Hash(partData)
_, err = obj.PutObjectPart(ctx, bucket, object, res.UploadID, 1,
mustGetPutObjReader(t, bytes.NewReader(partData), int64(len(partData)), md5Hex, ""),
ObjectOptions{})
if err != nil {
t.Fatal(err)
}
// Now simulate read quorum failure
erasureDisks := xl.getDisks()
z.serverPools[0].erasureDisksMu.Lock()
xl.getDisks = func() []StorageAPI {
for i := range erasureDisks[:8] {
erasureDisks[i] = nil
}
return erasureDisks
}
z.serverPools[0].erasureDisksMu.Unlock()
t.Run("complete multipart with if-none-match and read quorum failure", func(t *testing.T) {
// Try to complete the multipart upload with if-none-match
// This should fail because we can't verify the condition due to read quorum failure
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfNoneMatch: "*",
},
CheckPrecondFn: func(oi ObjectInfo) bool {
return oi.ETag != ""
},
}
parts := []CompletePart{{PartNumber: 1, ETag: md5Hex}}
_, err := obj.CompleteMultipartUpload(ctx, bucket, object, res.UploadID, parts, opts)
if !isErrReadQuorum(err) {
t.Errorf("Expected read quorum error, got: %v", err)
}
})
}
+2 -2
View File
@@ -390,7 +390,7 @@ func (er erasureObjects) newMultipartUpload(ctx context.Context, bucket string,
if err == nil && opts.CheckPrecondFn(obj) {
return nil, PreConditionFailed{}
}
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) && !isErrReadQuorum(err) {
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
return nil, err
}
@@ -1114,7 +1114,7 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
if err == nil && opts.CheckPrecondFn(obj) {
return ObjectInfo{}, PreConditionFailed{}
}
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) && !isErrReadQuorum(err) {
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
return ObjectInfo{}, err
}
+150
View File
@@ -0,0 +1,150 @@
// Copyright (c) 2015-2025 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"testing"
xhttp "github.com/minio/minio/internal/http"
)
// TestPutObjectConditionalWithReadQuorumFailure tests that conditional
// PutObject operations (with if-match/if-none-match) behave correctly when read quorum
// cannot be reached.
//
// Related to: https://github.com/minio/minio/issues/21603
//
// Should return an error when read quorum cannot
// be reached, as we cannot reliably determine if the precondition is met.
func TestPutObjectConditionalWithReadQuorumFailure(t *testing.T) {
ctx := context.Background()
obj, fsDirs, err := prepareErasure16(ctx)
if err != nil {
t.Fatal(err)
}
defer obj.Shutdown(context.Background())
defer removeRoots(fsDirs)
z := obj.(*erasureServerPools)
xl := z.serverPools[0].sets[0]
bucket := "test-bucket"
object := "test-object"
err = obj.MakeBucket(ctx, bucket, MakeBucketOptions{})
if err != nil {
t.Fatal(err)
}
// Put an initial object so it exists
_, err = obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("initial-value")),
int64(len("initial-value")), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
// Get object info to capture the ETag
objInfo, err := obj.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if err != nil {
t.Fatal(err)
}
existingETag := objInfo.ETag
// Simulate read quorum failure by taking enough disks offline
// With 16 disks (EC 8+8), read quorum is 9. Taking 8 disks offline leaves only 8,
// which is below read quorum.
erasureDisks := xl.getDisks()
z.serverPools[0].erasureDisksMu.Lock()
xl.getDisks = func() []StorageAPI {
for i := range erasureDisks[:8] {
erasureDisks[i] = nil
}
return erasureDisks
}
z.serverPools[0].erasureDisksMu.Unlock()
t.Run("if-none-match with read quorum failure", func(t *testing.T) {
// Test Case 1: if-none-match (create only if doesn't exist)
// With if-none-match: *, this should only succeed if object doesn't exist.
// Since read quorum fails, we can't determine if object exists.
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfNoneMatch: "*",
},
CheckPrecondFn: func(oi ObjectInfo) bool {
// Precondition fails if object exists (ETag is not empty)
return oi.ETag != ""
},
}
_, err := obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("new-value")),
int64(len("new-value")), "", ""), opts)
if !isErrReadQuorum(err) {
t.Errorf("Expected read quorum error when if-none-match is used with quorum failure, got: %v", err)
}
})
t.Run("if-match with read quorum failure", func(t *testing.T) {
// Test Case 2: if-match (update only if ETag matches)
// With if-match: <etag>, this should only succeed if object exists with matching ETag.
// Since read quorum fails, we can't determine if object exists or ETag matches.
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfMatch: existingETag,
},
CheckPrecondFn: func(oi ObjectInfo) bool {
// Precondition fails if ETag doesn't match
return oi.ETag != existingETag
},
}
_, err := obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("updated-value")),
int64(len("updated-value")), "", ""), opts)
if !isErrReadQuorum(err) {
t.Errorf("Expected read quorum error when if-match is used with quorum failure, got: %v", err)
}
})
t.Run("if-match wrong etag with read quorum failure", func(t *testing.T) {
// Test Case 3: if-match with wrong ETag
// Even if the ETag doesn't match, we should still get read quorum error
// because we can't read the object to check the condition.
opts := ObjectOptions{
UserDefined: map[string]string{
xhttp.IfMatch: "wrong-etag",
},
CheckPrecondFn: func(oi ObjectInfo) bool {
// Precondition fails if ETag doesn't match
return oi.ETag != "wrong-etag"
},
}
_, err := obj.PutObject(ctx, bucket, object,
mustGetPutObjReader(t, bytes.NewReader([]byte("should-fail")),
int64(len("should-fail")), "", ""), opts)
if !isErrReadQuorum(err) {
t.Errorf("Expected read quorum error when if-match is used with quorum failure (even with wrong ETag), got: %v", err)
}
})
}
+1 -1
View File
@@ -1274,7 +1274,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
if err == nil && opts.CheckPrecondFn(obj) {
return objInfo, PreConditionFailed{}
}
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) && !isErrReadQuorum(err) {
if err != nil && !isErrVersionNotFound(err) && !isErrObjectNotFound(err) {
return objInfo, err
}
+1 -1
View File
@@ -177,7 +177,7 @@ func formatGetBackendErasureVersion(b []byte) (string, error) {
return "", fmt.Errorf(`format.Version expected: %s, got: %s`, formatMetaVersionV1, meta.Version)
}
if meta.Format != formatBackendErasure && meta.Format != formatBackendErasureSingle {
return "", fmt.Errorf(`found backend type %s, expected %s or %s - to migrate to a supported backend visit https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-migrate-fs-gateway.html`, meta.Format, formatBackendErasure, formatBackendErasureSingle)
return "", fmt.Errorf(`found backend type %s, expected %s or %s - to migrate to a supported backend visit https://silo.pigsty.io/operations/deployments/baremetal-migrate-fs-gateway.html`, meta.Format, formatBackendErasure, formatBackendErasureSingle)
}
// Erasure backend found, proceed to detect version.
format := &formatErasureVersionDetect{}
+13
View File
@@ -192,6 +192,16 @@ func extractMetadata(ctx context.Context, mimesHeader ...textproto.MIMEHeader) (
// extractMetadata extracts metadata from map values.
func extractMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
return extractMetadataFromMimeWithReplication(ctx, v, m, false)
}
// extractReplicationMetadataFromMime restores replication-only metadata after the
// caller has validated that the request is a trusted replication write.
func extractReplicationMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[string]string) error {
return extractMetadataFromMimeWithReplication(ctx, v, m, true)
}
func extractMetadataFromMimeWithReplication(ctx context.Context, v textproto.MIMEHeader, m map[string]string, allowReplication bool) error {
if v == nil {
bugLogIf(ctx, errInvalidArgument)
return errInvalidArgument
@@ -208,6 +218,9 @@ func extractMetadataFromMime(ctx context.Context, v textproto.MIMEHeader, m map[
value, ok := nv[http.CanonicalHeaderKey(supportedHeader)]
if ok {
if v, ok := replicationToInternalHeaders[supportedHeader]; ok {
if !allowReplication {
continue
}
m[v] = strings.Join(value, ",")
} else {
m[supportedHeader] = strings.Join(value, ",")
+116
View File
@@ -24,11 +24,13 @@ import (
"io"
"net/http"
"net/textproto"
"net/url"
"os"
"reflect"
"testing"
"github.com/minio/minio/internal/config"
xhttp "github.com/minio/minio/internal/http"
)
// Tests validate bucket LocationConstraint.
@@ -152,6 +154,22 @@ func TestExtractMetadataHeaders(t *testing.T) {
},
shouldFail: false,
},
// Replication-only headers must not be accepted on ordinary requests.
{
header: http.Header{
"Content-Type": []string{"image/png"},
"X-Minio-Replication-Server-Side-Encryption-Sealed-Key": []string{"sealed-key"},
"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
"X-Minio-Replication-Server-Side-Encryption-Iv": []string{"iv"},
"X-Minio-Replication-Encrypted-Multipart": []string{""},
"X-Minio-Replication-Actual-Object-Size": []string{"1"},
ReplicationSsecChecksumHeader: []string{"checksum"},
},
metadata: map[string]string{
"content-type": "image/png",
},
shouldFail: false,
},
// Empty header input returns empty metadata.
{
header: nil,
@@ -176,6 +194,104 @@ func TestExtractMetadataHeaders(t *testing.T) {
}
}
func TestExtractReplicationMetadataHeaders(t *testing.T) {
header := http.Header{
"X-Minio-Replication-Server-Side-Encryption-Sealed-Key": []string{"sealed-key"},
"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": []string{"DAREv2-HMAC-SHA256"},
"X-Minio-Replication-Server-Side-Encryption-Iv": []string{"iv"},
"X-Minio-Replication-Encrypted-Multipart": []string{""},
"X-Minio-Replication-Actual-Object-Size": []string{"1"},
ReplicationSsecChecksumHeader: []string{"checksum"},
}
metadata := make(map[string]string)
if err := extractReplicationMetadataFromMime(t.Context(), textproto.MIMEHeader(header), metadata); err != nil {
t.Fatalf("failed to extract replication metadata: %v", err)
}
expected := map[string]string{
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key": "sealed-key",
"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "DAREv2-HMAC-SHA256",
"X-Minio-Internal-Server-Side-Encryption-Iv": "iv",
"X-Minio-Internal-Encrypted-Multipart": "",
"X-Minio-Internal-Actual-Object-Size": "1",
ReplicationSsecChecksumHeader: "checksum",
}
if !reflect.DeepEqual(metadata, expected) {
t.Fatalf("unexpected replication metadata: expected %#v, got %#v", expected, metadata)
}
}
func TestGetCopyObjectMetadataFromHeaderReplication(t *testing.T) {
req, err := http.NewRequest(http.MethodPut, "http://localhost/test", nil)
if err != nil {
t.Fatal(err)
}
req.Form = make(url.Values)
req.Header.Set("X-Amz-Metadata-Directive", replaceDirective)
req.Header.Set("X-Minio-Replication-Server-Side-Encryption-Sealed-Key", "sealed-key")
metadata, err := getCpObjMetadataFromHeader(t.Context(), req, nil, false)
if err != nil {
t.Fatalf("copy metadata extraction failed: %v", err)
}
if _, ok := metadata["X-Minio-Internal-Server-Side-Encryption-Sealed-Key"]; ok {
t.Fatalf("unexpected replication metadata without validation: %#v", metadata)
}
metadata, err = getCpObjMetadataFromHeader(t.Context(), req, nil, true)
if err != nil {
t.Fatalf("copy metadata extraction with replication failed: %v", err)
}
if got := metadata["X-Minio-Internal-Server-Side-Encryption-Sealed-Key"]; got != "sealed-key" {
t.Fatalf("expected restored replication metadata, got %#v", metadata)
}
}
func TestCloneRequestWithoutCopyReplicationHeaders(t *testing.T) {
req, err := http.NewRequest(http.MethodPut, "http://localhost/test", nil)
if err != nil {
t.Fatal(err)
}
req.Header.Set(xhttp.MinIOSourceReplicationRequest, "true")
req.Header.Set(xhttp.MinIOSourceETag, "etag")
req.Header.Set(xhttp.MinIOSourceMTime, "2026-04-15T10:00:00Z")
req.Header.Set(xhttp.MinIOSourceTaggingTimestamp, "2026-04-15T10:00:00Z")
req.Header.Set(xhttp.MinIOSourceObjectRetentionTimestamp, "2026-04-15T10:00:00Z")
req.Header.Set(xhttp.MinIOSourceObjectLegalHoldTimestamp, "2026-04-15T10:00:00Z")
req.Header.Set(xhttp.MinIOReplicationActualObjectSize, "123")
req.Header.Set(ReplicationSsecChecksumHeader, "checksum")
req.Header.Set("Content-Type", "application/octet-stream")
clone := cloneRequestWithoutCopyReplicationHeaders(req)
if clone == req {
t.Fatal("expected cloned request")
}
for _, header := range []string{
xhttp.MinIOSourceReplicationRequest,
xhttp.MinIOSourceETag,
xhttp.MinIOSourceMTime,
xhttp.MinIOSourceTaggingTimestamp,
xhttp.MinIOSourceObjectRetentionTimestamp,
xhttp.MinIOSourceObjectLegalHoldTimestamp,
xhttp.MinIOReplicationActualObjectSize,
ReplicationSsecChecksumHeader,
} {
if got := clone.Header.Get(header); got != "" {
t.Fatalf("expected %s to be stripped, got %q", header, got)
}
if got := req.Header.Get(header); got == "" {
t.Fatalf("expected original request to preserve %s", header)
}
}
if got := clone.Header.Get("Content-Type"); got != "application/octet-stream" {
t.Fatalf("expected non-replication headers to be preserved, got %q", got)
}
}
// Test getResource()
func TestGetResource(t *testing.T) {
testCases := []struct {
+1 -1
View File
@@ -576,7 +576,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
if took := time.Since(listStartTime); took > maxIAMLoadOpTime {
var s strings.Builder
for k, v := range listedConfigItems {
s.WriteString(fmt.Sprintf(" %s: %d items\n", k, len(v)))
fmt.Fprintf(&s, " %s: %d items\n", k, len(v))
}
logger.Info("listAllIAMConfigItems took %.2fs with contents:\n%s", took.Seconds(), s.String())
}
+4 -4
View File
@@ -386,7 +386,7 @@ func storageMetricsPrometheus(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(
prometheus.BuildFQName(minioNamespace, "capacity_raw", "total"),
"Total capacity online in the cluster",
"Total capacity online in current MinIO server instance",
nil, nil),
prometheus.GaugeValue,
float64(GetTotalCapacity(server.Disks)),
@@ -396,7 +396,7 @@ func storageMetricsPrometheus(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(
prometheus.BuildFQName(minioNamespace, "capacity_raw_free", "total"),
"Total free capacity online in the cluster",
"Total free capacity online in current MinIO server instance",
nil, nil),
prometheus.GaugeValue,
float64(GetTotalCapacityFree(server.Disks)),
@@ -408,7 +408,7 @@ func storageMetricsPrometheus(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(
prometheus.BuildFQName(minioNamespace, "capacity_usable", "total"),
"Total usable capacity online in the cluster",
"Total usable capacity online in current MinIO server instance",
nil, nil),
prometheus.GaugeValue,
float64(GetTotalUsableCapacity(server.Disks, sinfo)),
@@ -418,7 +418,7 @@ func storageMetricsPrometheus(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(
prometheus.BuildFQName(minioNamespace, "capacity_usable_free", "total"),
"Total free usable capacity online in the cluster",
"Total free usable capacity online in current MinIO server instance",
nil, nil),
prometheus.GaugeValue,
float64(GetTotalUsableCapacityFree(server.Disks, sinfo)),
-8
View File
@@ -324,14 +324,6 @@ func (d *naughtyDisk) StatInfoFile(ctx context.Context, volume, path string, glo
return d.disk.StatInfoFile(ctx, volume, path, glob)
}
func (d *naughtyDisk) ReadMultiple(ctx context.Context, req ReadMultipleReq, resp chan<- ReadMultipleResp) error {
if err := d.calcError(); err != nil {
close(resp)
return err
}
return d.disk.ReadMultiple(ctx, req, resp)
}
func (d *naughtyDisk) CleanAbandonedData(ctx context.Context, volume string, path string) error {
if err := d.calcError(); err != nil {
return err
+75 -11
View File
@@ -1036,7 +1036,7 @@ func (api objectAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.Re
// Extract metadata relevant for an CopyObject operation based on conditional
// header values specified in X-Amz-Metadata-Directive.
func getCpObjMetadataFromHeader(ctx context.Context, r *http.Request, userMeta map[string]string) (map[string]string, error) {
func getCpObjMetadataFromHeader(ctx context.Context, r *http.Request, userMeta map[string]string, allowReplication bool) (map[string]string, error) {
// Make a copy of the supplied metadata to avoid
// to change the original one.
defaultMeta := make(map[string]string, len(userMeta))
@@ -1067,6 +1067,11 @@ func getCpObjMetadataFromHeader(ctx context.Context, r *http.Request, userMeta m
if err != nil {
return nil, err
}
if allowReplication {
if err = extractReplicationMetadataFromMime(ctx, textproto.MIMEHeader(r.Header), emetadata); err != nil {
return nil, err
}
}
if sc != "" {
emetadata[xhttp.AmzStorageClass] = sc
}
@@ -1087,6 +1092,31 @@ func getCpObjMetadataFromHeader(ctx context.Context, r *http.Request, userMeta m
return defaultMeta, nil
}
func cloneRequestWithoutCopyReplicationHeaders(r *http.Request) *http.Request {
if r == nil {
return nil
}
clone := new(http.Request)
*clone = *r
clone.Header = r.Header.Clone()
for _, header := range []string{
xhttp.MinIOSourceReplicationRequest,
xhttp.MinIOSourceETag,
xhttp.MinIOSourceMTime,
xhttp.MinIOSourceTaggingTimestamp,
xhttp.MinIOSourceObjectRetentionTimestamp,
xhttp.MinIOSourceObjectLegalHoldTimestamp,
xhttp.MinIOReplicationActualObjectSize,
ReplicationSsecChecksumHeader,
} {
clone.Header.Del(header)
}
return clone
}
// getRemoteInstanceTransport contains a roundtripper for external (not peers) servers
var remoteInstanceTransport atomic.Value
@@ -1232,6 +1262,19 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidStorageClass), r.URL)
return
}
allowReplicationMetadata := false
if r.Header.Get(xhttp.AmzBucketReplicationStatus) == replication.Replica.String() {
if s3Error := checkRequestAuthType(ctx, r, policy.ReplicateObjectAction, dstBucket, dstObject); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
return
}
allowReplicationMetadata = true
}
trustedReplicationRequest := allowReplicationMetadata && r.Header.Get(xhttp.MinIOSourceReplicationRequest) == "true"
optsReq := r
if !trustedReplicationRequest {
optsReq = cloneRequestWithoutCopyReplicationHeaders(r)
}
// Check if bucket encryption is enabled
sseConfig, _ := globalBucketSSEConfigSys.Get(dstBucket)
@@ -1240,7 +1283,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
})
var srcOpts, dstOpts ObjectOptions
srcOpts, err = copySrcOpts(ctx, r, srcBucket, srcObject)
srcOpts, err = copySrcOpts(ctx, optsReq, srcBucket, srcObject)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -1252,14 +1295,14 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
VersionID: srcOpts.VersionID,
Versioned: srcOpts.Versioned,
VersionSuspended: srcOpts.VersionSuspended,
ReplicationRequest: r.Header.Get(xhttp.MinIOSourceReplicationRequest) == "true",
ReplicationRequest: trustedReplicationRequest,
}
getSSE := encrypt.SSE(srcOpts.ServerSideEncryption)
if getSSE != srcOpts.ServerSideEncryption {
getOpts.ServerSideEncryption = getSSE
}
dstOpts, err = copyDstOpts(ctx, r, dstBucket, dstObject, nil)
dstOpts, err = copyDstOpts(ctx, optsReq, dstBucket, dstObject, nil)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -1269,7 +1312,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
getObjectNInfo := objectAPI.GetObjectNInfo
checkCopyPrecondFn := func(o ObjectInfo) bool {
if _, err := DecryptObjectInfo(&o, r); err != nil {
if _, err := DecryptObjectInfo(&o, optsReq); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return true
}
@@ -1380,7 +1423,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
return
}
// Encryption parameters not present for this object.
if crypto.SSEC.IsEncrypted(srcInfo.UserDefined) && !crypto.SSECopy.IsRequested(r.Header) && r.Header.Get(xhttp.MinIOSourceReplicationRequest) != "true" {
if crypto.SSEC.IsEncrypted(srcInfo.UserDefined) && !crypto.SSECopy.IsRequested(r.Header) && !trustedReplicationRequest {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidSSECustomerAlgorithm), r.URL)
return
}
@@ -1546,7 +1589,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
srcInfo.PutObjReader = pReader
srcInfo.UserDefined, err = getCpObjMetadataFromHeader(ctx, r, srcInfo.UserDefined)
srcInfo.UserDefined, err = getCpObjMetadataFromHeader(ctx, r, srcInfo.UserDefined, allowReplicationMetadata)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
@@ -1628,10 +1671,10 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
if rs := r.Header.Get(xhttp.AmzBucketReplicationStatus); rs != "" {
if allowReplicationMetadata {
srcInfo.UserDefined[ReservedMetadataPrefixLower+ReplicaStatus] = replication.Replica.String()
srcInfo.UserDefined[ReservedMetadataPrefixLower+ReplicaTimestamp] = UTCNow().Format(time.RFC3339Nano)
srcInfo.UserDefined[xhttp.AmzBucketReplicationStatus] = rs
srcInfo.UserDefined[xhttp.AmzBucketReplicationStatus] = replication.Replica.String()
}
op := replication.ObjectReplicationType
@@ -1896,7 +1939,7 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
}
case authTypeStreamingUnsignedTrailer:
// Initialize stream chunked reader with optional trailers.
rd, s3Err = newUnsignedV4ChunkedReader(r, true, r.Header.Get(xhttp.Authorization) != "")
rd, s3Err = newUnsignedV4ChunkedReader(r, true)
if s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
@@ -1933,6 +1976,10 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
if err = extractReplicationMetadataFromMime(ctx, textproto.MIMEHeader(r.Header), metadata); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
metadata[ReservedMetadataPrefixLower+ReplicaStatus] = replication.Replica.String()
metadata[ReservedMetadataPrefixLower+ReplicaTimestamp] = UTCNow().Format(time.RFC3339Nano)
defer globalReplicationStats.Load().UpdateReplicaStat(bucket, size)
@@ -2242,7 +2289,7 @@ func (api objectAPIHandlers) PutObjectExtractHandler(w http.ResponseWriter, r *h
// if Content-Length is unknown/missing, deny the request
size := r.ContentLength
rAuthType := getRequestAuthType(r)
if rAuthType == authTypeStreamingSigned || rAuthType == authTypeStreamingSignedTrailer {
if rAuthType == authTypeStreamingSigned || rAuthType == authTypeStreamingSignedTrailer || rAuthType == authTypeStreamingUnsignedTrailer {
if sizeStr, ok := r.Header[xhttp.AmzDecodedContentLength]; ok {
if sizeStr[0] == "" {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentLength), r.URL)
@@ -2296,6 +2343,13 @@ func (api objectAPIHandlers) PutObjectExtractHandler(w http.ResponseWriter, r *h
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
case authTypeStreamingUnsignedTrailer:
// Initialize stream chunked reader with optional trailers.
reader, s3Err = newUnsignedV4ChunkedReader(r, true)
if s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
case authTypeSignedV2, authTypePresignedV2:
s3Err = isReqAuthenticatedV2(r)
if s3Err != ErrNone {
@@ -2388,10 +2442,15 @@ func (api objectAPIHandlers) PutObjectExtractHandler(w http.ResponseWriter, r *h
rawReader := hashReader
pReader := NewPutObjReader(rawReader)
allowReplicationMetadata := false
if r.Header.Get(xhttp.AmzBucketReplicationStatus) == replication.Replica.String() {
if s3Err = isPutActionAllowed(ctx, getRequestAuthType(r), bucket, object, r, policy.ReplicateObjectAction); s3Err != ErrNone {
return errors.New(errorCodes.ToAPIErr(s3Err).Code)
}
allowReplicationMetadata = true
if err = extractReplicationMetadataFromMime(ctx, textproto.MIMEHeader(r.Header), metadata); err != nil {
return err
}
metadata[ReservedMetadataPrefixLower+ReplicaStatus] = replication.Replica.String()
metadata[ReservedMetadataPrefixLower+ReplicaTimestamp] = UTCNow().Format(time.RFC3339Nano)
}
@@ -2417,6 +2476,11 @@ func (api objectAPIHandlers) PutObjectExtractHandler(w http.ResponseWriter, r *h
if err != nil {
return err
}
if allowReplicationMetadata {
if err = extractReplicationMetadataFromMime(ctx, textproto.MIMEHeader(hdrs), m); err != nil {
return err
}
}
maps.Copy(metadata, m)
} else {
versionID = r.Form.Get(xhttp.VersionID)
+157
View File
@@ -42,6 +42,7 @@ import (
"github.com/dustin/go-humanize"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/crypto"
"github.com/minio/minio/internal/hash/sha256"
xhttp "github.com/minio/minio/internal/http"
ioutilx "github.com/minio/minio/internal/ioutil"
@@ -60,6 +61,47 @@ const (
MissingUploadID
)
func replicationSSEPoisonHeaders() map[string]string {
return map[string]string{
"X-Minio-Replication-Server-Side-Encryption-Sealed-Key": base64.StdEncoding.EncodeToString(make([]byte, 64)),
"X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm": crypto.SealAlgorithm,
"X-Minio-Replication-Server-Side-Encryption-Iv": base64.StdEncoding.EncodeToString(make([]byte, 32)),
}
}
func assertObjectMetadataKeysAbsent(t *testing.T, metadata map[string]string, keys ...string) {
t.Helper()
for _, key := range keys {
if got, ok := metadata[key]; ok {
t.Fatalf("expected metadata %q to be absent, got %q", key, got)
}
}
}
func assertObjectMetadataValueNotEqual(t *testing.T, metadata map[string]string, key, unexpected string) {
t.Helper()
if got := metadata[key]; got == unexpected {
t.Fatalf("expected metadata %q to differ from %q", key, unexpected)
}
}
func assertObjectContents(t *testing.T, obj ObjectLayer, bucketName, objectName string, expected []byte) {
t.Helper()
reader, err := obj.GetObjectNInfo(context.Background(), bucketName, objectName, nil, nil, ObjectOptions{})
if err != nil {
t.Fatalf("failed to fetch object %s/%s: %v", bucketName, objectName, err)
}
defer reader.Close()
got, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("failed to read object %s/%s: %v", bucketName, objectName, err)
}
if !bytes.Equal(got, expected) {
t.Fatalf("unexpected object contents: got %d bytes, expected %d bytes", len(got), len(expected))
}
}
// Wrapper for calling HeadObject API handler tests for both Erasure multiple disks and FS single drive setup.
func TestAPIHeadObjectHandler(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: testAPIHeadObjectHandler, endpoints: []string{"HeadObject"}})
@@ -1819,6 +1861,53 @@ func testAPICopyObjectPartHandlerSanity(obj ObjectLayer, instanceType, bucketNam
}
}
func TestAPIPutObjectReplicationHeaderPoisoning(t *testing.T) {
defer DetectTestLeak(t)()
ExecExtendedObjectLayerAPITest(t, testAPIPutObjectReplicationHeaderPoisoning, []string{"PutObject"})
}
func testAPIPutObjectReplicationHeaderPoisoning(obj ObjectLayer, instanceType, bucketName string, apiRouter http.Handler,
credentials auth.Credentials, t *testing.T,
) {
objectName := "replication-header-poison-put"
payload := []byte("replication-header-poison-put-payload")
headers := replicationSSEPoisonHeaders()
req, err := newTestSignedRequestV4(
http.MethodPut,
getPutObjectURL("", bucketName, objectName),
int64(len(payload)),
bytes.NewReader(payload),
credentials.AccessKey,
credentials.SecretKey,
headers,
)
if err != nil {
t.Fatalf("%s: failed to create signed put request: %v", instanceType, err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("%s: expected put to succeed, got %d", instanceType, rec.Code)
}
objInfo, err := obj.GetObjectInfo(context.Background(), bucketName, objectName, ObjectOptions{})
if err != nil {
t.Fatalf("%s: failed to fetch object info: %v", instanceType, err)
}
assertObjectMetadataValueNotEqual(t, objInfo.UserDefined,
crypto.MetaSealedKeySSEC,
headers["X-Minio-Replication-Server-Side-Encryption-Sealed-Key"],
)
assertObjectMetadataValueNotEqual(t, objInfo.UserDefined,
crypto.MetaIV,
headers["X-Minio-Replication-Server-Side-Encryption-Iv"],
)
assertObjectContents(t, obj, bucketName, objectName, payload)
}
// Wrapper for calling Copy Object Part API handler tests for both Erasure multiple disks and single node setup.
func TestAPICopyObjectPartHandler(t *testing.T) {
defer DetectTestLeak(t)()
@@ -2860,6 +2949,74 @@ func testAPINewMultipartHandlerParallel(obj ObjectLayer, instanceType, bucketNam
}
}
func TestAPICopyObjectReplicationHeaderPoisoning(t *testing.T) {
defer DetectTestLeak(t)()
ExecExtendedObjectLayerAPITest(t, testAPICopyObjectReplicationHeaderPoisoning, []string{"CopyObject", "PutObject"})
}
func testAPICopyObjectReplicationHeaderPoisoning(obj ObjectLayer, instanceType, bucketName string, apiRouter http.Handler,
credentials auth.Credentials, t *testing.T,
) {
srcObject := "replication-header-poison-copy-src"
dstObject := "replication-header-poison-copy-dst"
payload := []byte("replication-header-poison-copy-payload")
if _, err := obj.PutObject(
context.Background(),
bucketName,
srcObject,
mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""),
ObjectOptions{},
); err != nil {
t.Fatalf("%s: failed to create source object: %v", instanceType, err)
}
headers := replicationSSEPoisonHeaders()
headers[xhttp.AmzCopySource] = url.QueryEscape(SlashSeparator + bucketName + SlashSeparator + srcObject)
headers[xhttp.AmzMetadataDirective] = replaceDirective
headers[xhttp.AmzBucketReplicationStatus] = "PENDING"
headers["Content-Type"] = "application/octet-stream"
req, err := newTestSignedRequestV4(
http.MethodPut,
getCopyObjectURL("", bucketName, dstObject),
0,
nil,
credentials.AccessKey,
credentials.SecretKey,
headers,
)
if err != nil {
t.Fatalf("%s: failed to create signed copy request: %v", instanceType, err)
}
rec := httptest.NewRecorder()
apiRouter.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("%s: expected copy to succeed, got %d", instanceType, rec.Code)
}
objInfo, err := obj.GetObjectInfo(context.Background(), bucketName, dstObject, ObjectOptions{})
if err != nil {
t.Fatalf("%s: failed to fetch copied object info: %v", instanceType, err)
}
assertObjectMetadataValueNotEqual(t, objInfo.UserDefined,
crypto.MetaSealedKeySSEC,
headers["X-Minio-Replication-Server-Side-Encryption-Sealed-Key"],
)
assertObjectMetadataValueNotEqual(t, objInfo.UserDefined,
crypto.MetaIV,
headers["X-Minio-Replication-Server-Side-Encryption-Iv"],
)
assertObjectMetadataKeysAbsent(t, objInfo.UserDefined,
xhttp.AmzBucketReplicationStatus,
ReservedMetadataPrefixLower+ReplicaStatus,
ReservedMetadataPrefixLower+ReplicaTimestamp,
)
assertObjectContents(t, obj, bucketName, dstObject, payload)
}
// The UploadID from the response body is parsed and its existence is asserted with an attempt to ListParts using it.
func TestAPICompleteMultipartHandler(t *testing.T) {
defer DetectTestLeak(t)()
+11 -2
View File
@@ -24,6 +24,7 @@ import (
"io"
"maps"
"net/http"
"net/textproto"
"net/url"
"sort"
"strconv"
@@ -157,6 +158,14 @@ func (api objectAPIHandlers) NewMultipartUploadHandler(w http.ResponseWriter, r
metadata[xhttp.AmzObjectTagging] = objTags
}
if r.Header.Get(xhttp.AmzBucketReplicationStatus) == replication.Replica.String() {
if s3Err := isPutActionAllowed(ctx, getRequestAuthType(r), bucket, object, r, policy.ReplicateObjectAction); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
if err = extractReplicationMetadataFromMime(ctx, textproto.MIMEHeader(r.Header), metadata); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
metadata[ReservedMetadataPrefixLower+ReplicaStatus] = replication.Replica.String()
metadata[ReservedMetadataPrefixLower+ReplicaTimestamp] = UTCNow().Format(time.RFC3339Nano)
}
@@ -685,8 +694,8 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
return
}
case authTypeStreamingUnsignedTrailer:
// Initialize stream signature verifier.
reader, s3Error = newUnsignedV4ChunkedReader(r, true, r.Header.Get(xhttp.Authorization) != "")
// Initialize stream chunked reader with optional trailers.
reader, s3Error = newUnsignedV4ChunkedReader(r, true)
if s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
return
+1
View File
@@ -51,6 +51,7 @@ var startsWithConds = map[string]bool{
"$x-amz-algorithm": false,
"$x-amz-credential": false,
"$x-amz-date": false,
"$tagging": false,
}
// Add policy conditionals.
+2 -2
View File
@@ -144,7 +144,7 @@ func printServerCommonMsg(apiEndpoints []string) {
// Prints startup message for Object API access, prints link to our SDK documentation.
func printObjectAPIMsg() {
logger.Startup(color.Blue("\nDocs: ") + "https://docs.min.io")
logger.Startup(color.Blue("\nDocs: ") + "https://silo.pigsty.io")
}
func printLambdaTargets() {
@@ -184,7 +184,7 @@ func printCLIAccessMsg(endPoint string, alias string) {
// Get saved credentials.
cred := globalActiveCred
const mcQuickStartGuide = "https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart"
const mcQuickStartGuide = "https://silo.pigsty.io/reference/minio-mc.html#quickstart"
// Configure 'mc', following block prints platform specific information for minio client.
if color.IsTerminal() && (!globalServerCtxt.Anonymous && globalAPIConfig.permitRootAccess()) {
+384 -19
View File
@@ -18,10 +18,15 @@
package cmd
import (
"archive/tar"
"bytes"
"context"
"encoding/base64"
"encoding/binary"
"encoding/json"
"encoding/xml"
"fmt"
"hash/crc32"
"io"
"math/rand"
"net/http"
@@ -128,6 +133,13 @@ func runAllTests(suite *TestSuiteCommon, c *check) {
suite.TestBucketSQSNotificationWebHook(c)
suite.TestBucketSQSNotificationAMQP(c)
suite.TestUnsignedCVE(c)
suite.TestUnsignedQueryStringCVE(c)
suite.TestUnsignedQueryStringCVEMultipart(c)
suite.TestUnsignedTrailerRejectsMultipleAuthSources(c)
suite.TestUnsignedTrailerSnowballAnonymousDenied(c)
suite.TestUnsignedTrailerSnowballRequiresSignature(c)
suite.TestUnsignedTrailerSnowballExtract(c)
suite.TestAnonymousUnsignedTrailer(c)
suite.TearDownSuite(c)
}
@@ -374,24 +386,13 @@ func (s *TestSuiteCommon) TestUnsignedCVE(c *check) {
// assert the http response status code.
c.Assert(response.StatusCode, http.StatusOK)
req, err := http.NewRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, "test-cve-object.txt"), nil)
now := UTCNow()
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, "test-cve-object.txt"), []byte("foobar!\n"), now)
c.Assert(err, nil)
req.Body = io.NopCloser(bytes.NewReader([]byte("foobar!\n")))
req.Trailer = http.Header{}
req.Trailer.Set("x-amz-checksum-crc32", "rK0DXg==")
now := UTCNow()
req = signer.StreamingUnsignedV4(req, "", 8, now)
maliciousHeaders := http.Header{
"Authorization": []string{fmt.Sprintf("AWS4-HMAC-SHA256 Credential=%s/%s/us-east-1/s3/aws4_request, SignedHeaders=invalidheader, Signature=deadbeefdeadbeefdeadbeeddeadbeeddeadbeefdeadbeefdeadbeefdeadbeef", s.accessKey, now.Format(yyyymmdd))},
"User-Agent": []string{"A malicious request"},
"X-Amz-Decoded-Content-Length": []string{"8"},
"Content-Encoding": []string{"aws-chunked"},
"X-Amz-Trailer": []string{"x-amz-checksum-crc32"},
"x-amz-content-sha256": []string{unsignedPayloadTrailer},
"Authorization": []string{fmt.Sprintf("AWS4-HMAC-SHA256 Credential=%s/%s/us-east-1/s3/aws4_request, SignedHeaders=invalidheader, Signature=deadbeefdeadbeefdeadbeeddeadbeeddeadbeefdeadbeefdeadbeefdeadbeef", s.accessKey, now.Format(yyyymmdd))},
"User-Agent": []string{"A malicious request"},
}
for k, v := range maliciousHeaders {
@@ -409,6 +410,370 @@ func (s *TestSuiteCommon) TestUnsignedCVE(c *check) {
c.Assert(response.StatusCode, http.StatusBadRequest)
}
func (s *TestSuiteCommon) TestUnsignedQueryStringCVE(c *check) {
c.Helper()
// generate a random bucket Name.
bucketName := getRandomBucketName()
// HTTP request to create the bucket.
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
// execute the request.
response, err := s.client.Do(request)
c.Assert(err, nil)
// assert the http response status code.
c.Assert(response.StatusCode, http.StatusOK)
now := UTCNow()
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, "test-cve-presigned-object.txt"), []byte("foobar!\n"), now)
c.Assert(err, nil)
err = presignStreamingUnsignedTrailerRequest(req, s.accessKey, s.secretKey, 60, now)
c.Assert(err, nil)
// execute the request.
response, err = s.client.Do(req)
c.Assert(err, nil)
verifyError(c, response, "InvalidRequest", "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.", http.StatusBadRequest)
}
func (s *TestSuiteCommon) TestUnsignedQueryStringCVEMultipart(c *check) {
c.Helper()
bucketName := getRandomBucketName()
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
uploadID := s.mustStartMultipartUpload(c, bucketName, "test-cve-presigned-multipart.txt")
now := UTCNow()
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPartUploadURL(s.endPoint, bucketName, "test-cve-presigned-multipart.txt", uploadID, "1"), []byte("foobar!\n"), now)
c.Assert(err, nil)
err = presignStreamingUnsignedTrailerRequest(req, s.accessKey, s.secretKey, 60, now)
c.Assert(err, nil)
response, err = s.client.Do(req)
c.Assert(err, nil)
verifyError(c, response, "InvalidRequest", "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.", http.StatusBadRequest)
}
func (s *TestSuiteCommon) TestUnsignedTrailerRejectsMultipleAuthSources(c *check) {
c.Helper()
bucketName := getRandomBucketName()
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
now := UTCNow()
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, "test-cve-mixed-auth.txt"), []byte("foobar!\n"), now)
c.Assert(err, nil)
err = presignStreamingUnsignedTrailerRequest(req, s.accessKey, s.secretKey, 60, now)
c.Assert(err, nil)
err = signRequestV4(req, s.accessKey, s.secretKey)
c.Assert(err, nil)
response, err = s.client.Do(req)
c.Assert(err, nil)
verifyError(c, response, "InvalidRequest", "Invalid Request (request has multiple authentication types, please use one)", http.StatusBadRequest)
}
func (s *TestSuiteCommon) TestUnsignedTrailerSnowballRequiresSignature(c *check) {
c.Helper()
bucketName := getRandomBucketName()
objectName := "snowball-upload.tar"
extractedObject := "payload.txt"
extractedData := []byte("snowball object\n")
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
tarData, err := newTestTarArchive(extractedObject, extractedData)
c.Assert(err, nil)
// Deliberately send raw tar bytes: before the fix this path never
// initialized the unsigned-trailer reader, so Snowball accepted the
// body without chunked decoding or signature verification.
req, err := http.NewRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, objectName), bytes.NewReader(tarData))
c.Assert(err, nil)
req.ContentLength = int64(len(tarData))
req.Header.Set(xhttp.AmzSnowballExtract, "true")
req.Header.Set(xhttp.AmzContentSha256, unsignedPayloadTrailer)
err = signRequestV4(req, s.accessKey, s.secretKey)
c.Assert(err, nil)
req.Header.Set("Authorization",
regexp.MustCompile(`Signature=[0-9a-f]+`).ReplaceAllString(req.Header.Get("Authorization"), "Signature="+strings.Repeat("0", 64)))
response, err = s.client.Do(req)
c.Assert(err, nil)
verifyError(c, response, "SignatureDoesNotMatch", "The request signature we calculated does not match the signature you provided. Check your key and signing method.", http.StatusForbidden)
}
func (s *TestSuiteCommon) TestUnsignedTrailerSnowballAnonymousDenied(c *check) {
c.Helper()
bucketName := getRandomBucketName()
objectName := "snowball-upload.tar"
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
tarData, err := newTestTarArchive("payload.txt", []byte("snowball object\n"))
c.Assert(err, nil)
req, err := http.NewRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, objectName), bytes.NewReader(tarData))
c.Assert(err, nil)
req.ContentLength = int64(len(tarData))
req.Header.Set(xhttp.AmzSnowballExtract, "true")
req.Header.Set(xhttp.AmzContentSha256, unsignedPayloadTrailer)
response, err = s.client.Do(req)
c.Assert(err, nil)
verifyError(c, response, "AccessDenied", "Access Denied.", http.StatusForbidden)
}
func (s *TestSuiteCommon) TestUnsignedTrailerSnowballExtract(c *check) {
c.Helper()
bucketName := getRandomBucketName()
objectName := "snowball-upload.tar"
extractedObject := "payload.txt"
extractedData := []byte("snowball object\n")
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
tarData, err := newTestTarArchive(extractedObject, extractedData)
c.Assert(err, nil)
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, objectName), tarData, UTCNow())
c.Assert(err, nil)
req.Header.Set(xhttp.AmzSnowballExtract, "true")
err = signRequestV4(req, s.accessKey, s.secretKey)
c.Assert(err, nil)
response, err = s.client.Do(req)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
request, err = newTestSignedRequest(http.MethodGet, getGetObjectURL(s.endPoint, bucketName, extractedObject),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err = s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
got, err := io.ReadAll(response.Body)
c.Assert(err, nil)
c.Assert(bytes.Equal(got, extractedData), true)
}
func (s *TestSuiteCommon) TestAnonymousUnsignedTrailer(c *check) {
c.Helper()
bucketName := getRandomBucketName()
objectName := "test-anonymous-unsigned-trailer.txt"
objectData := []byte("foobar!\n")
request, err := newTestSignedRequest(http.MethodPut, getMakeBucketURL(s.endPoint, bucketName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
s.mustPutBucketPolicy(c, bucketName, getAnonWriteOnlyObjectPolicy(bucketName, objectName))
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPutObjectURL(s.endPoint, bucketName, objectName), objectData, UTCNow())
c.Assert(err, nil)
response, err = s.client.Do(req)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
request, err = newTestSignedRequest(http.MethodGet, getGetObjectURL(s.endPoint, bucketName, objectName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err = s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
got, err := io.ReadAll(response.Body)
c.Assert(err, nil)
c.Assert(bytes.Equal(got, objectData), true)
}
func newTestTarArchive(objectName string, objectData []byte) ([]byte, error) {
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
err := tw.WriteHeader(&tar.Header{
Name: objectName,
Mode: 0o600,
Size: int64(len(objectData)),
})
if err != nil {
return nil, err
}
if _, err = tw.Write(objectData); err != nil {
return nil, err
}
if err = tw.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func newStreamingUnsignedTrailerRequest(method, targetURL string, data []byte, reqTime time.Time) (*http.Request, error) {
req, err := http.NewRequest(method, targetURL, nil)
if err != nil {
return nil, err
}
req.Body = io.NopCloser(bytes.NewReader(data))
req.Trailer = http.Header{}
req.Trailer.Set("x-amz-checksum-crc32", crc32Base64(data))
req = signer.StreamingUnsignedV4(req, "", int64(len(data)), reqTime)
req.Header.Set("X-Amz-Decoded-Content-Length", fmt.Sprintf("%d", len(data)))
req.Header.Set("Content-Encoding", "aws-chunked")
req.Header.Set("X-Amz-Trailer", "x-amz-checksum-crc32")
req.Header.Set(xhttp.AmzContentSha256, unsignedPayloadTrailer)
return req, nil
}
func crc32Base64(data []byte) string {
var crc [4]byte
binary.BigEndian.PutUint32(crc[:], crc32.ChecksumIEEE(data))
return base64.StdEncoding.EncodeToString(crc[:])
}
func presignStreamingUnsignedTrailerRequest(req *http.Request, accessKeyID, secretAccessKey string, expires int64, reqTime time.Time) error {
if accessKeyID == "" || secretAccessKey == "" {
return fmt.Errorf("presign cannot be generated without access and secret keys")
}
signedHeaders := []string{
"content-encoding",
"host",
"x-amz-content-sha256",
"x-amz-decoded-content-length",
"x-amz-trailer",
}
region := globalSite.Region()
scope := getScope(reqTime, region)
credential := fmt.Sprintf("%s/%s", accessKeyID, scope)
query := req.URL.Query()
query.Set(xhttp.AmzAlgorithm, signV4Algorithm)
query.Set(xhttp.AmzDate, reqTime.Format(iso8601Format))
query.Set(xhttp.AmzExpires, fmt.Sprintf("%d", expires))
query.Set(xhttp.AmzSignedHeaders, strings.Join(signedHeaders, ";"))
query.Set(xhttp.AmzCredential, credential)
query.Set(xhttp.AmzContentSha256, unsignedPayloadTrailer)
req.Form = query
extractedSignedHeaders, errCode := extractSignedHeaders(signedHeaders, req)
if errCode != ErrNone {
return fmt.Errorf("extractSignedHeaders failed: %v", errCode)
}
queryStr := strings.ReplaceAll(query.Encode(), "+", "%20")
canonicalRequest := getCanonicalRequest(extractedSignedHeaders, unsignedPayloadTrailer, queryStr, req.URL.Path, req.Method)
stringToSign := getStringToSign(canonicalRequest, reqTime, scope)
signingKey := getSigningKey(secretAccessKey, reqTime, region, serviceS3)
signature := getSignature(signingKey, stringToSign)
query.Set(xhttp.AmzSignature, signature)
req.URL.RawQuery = query.Encode()
req.Form = req.URL.Query()
return nil
}
func (s *TestSuiteCommon) mustPutBucketPolicy(c *check, bucketName string, bucketPolicy *policy.BucketPolicy) {
c.Helper()
policyBytes, err := json.Marshal(bucketPolicy)
c.Assert(err, nil)
request, err := newTestSignedRequest(http.MethodPut, getPutPolicyURL(s.endPoint, bucketName),
int64(len(policyBytes)), bytes.NewReader(policyBytes), s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusNoContent)
}
func (s *TestSuiteCommon) mustStartMultipartUpload(c *check, bucketName, objectName string) string {
c.Helper()
request, err := newTestSignedRequest(http.MethodPost, getNewMultipartURL(s.endPoint, bucketName, objectName),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
response, err := s.client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
decoder := xml.NewDecoder(response.Body)
newResponse := &InitiateMultipartUploadResponse{}
err = decoder.Decode(newResponse)
c.Assert(err, nil)
c.Assert(len(newResponse.UploadID) > 0, true)
return newResponse.UploadID
}
func (s *TestSuiteCommon) TestBucketSQSNotificationAMQP(c *check) {
// Sample bucket notification.
bucketNotificationBuf := `<NotificationConfiguration><QueueConfiguration><Event>s3:ObjectCreated:Put</Event><Filter><S3Key><FilterRule><Name>prefix</Name><Value>images/</Value></FilterRule></S3Key></Filter><Id>1</Id><Queue>arn:minio:sqs:us-east-1:444455556666:amqp</Queue></QueueConfiguration></NotificationConfiguration>`
@@ -2128,7 +2493,7 @@ func (s *TestSuiteCommon) TestGetObjectLarge10MiB(c *check) {
1234567890,1234567890,1234567890,1234567890,1234567890,123"`
// Create 10MiB content where each line contains 1024 characters.
for i := range 10 * 1024 {
buffer.WriteString(fmt.Sprintf("[%05d] %s\n", i, line))
fmt.Fprintf(&buffer, "[%05d] %s\n", i, line)
}
putContent := buffer.String()
@@ -2190,7 +2555,7 @@ func (s *TestSuiteCommon) TestGetObjectLarge11MiB(c *check) {
1234567890,1234567890,1234567890,123`
// Create 11MiB content where each line contains 1024 characters.
for i := range 11 * 1024 {
buffer.WriteString(fmt.Sprintf("[%05d] %s\n", i, line))
fmt.Fprintf(&buffer, "[%05d] %s\n", i, line)
}
putMD5 := getMD5Hash(buffer.Bytes())
@@ -2341,7 +2706,7 @@ func (s *TestSuiteCommon) TestGetPartialObjectLarge11MiB(c *check) {
// Create 11MiB content where each line contains 1024
// characters.
for i := range 11 * 1024 {
buffer.WriteString(fmt.Sprintf("[%05d] %s\n", i, line))
fmt.Fprintf(&buffer, "[%05d] %s\n", i, line)
}
putContent := buffer.String()
@@ -2407,7 +2772,7 @@ func (s *TestSuiteCommon) TestGetPartialObjectLarge10MiB(c *check) {
1234567890,1234567890,1234567890,123`
// Create 10MiB content where each line contains 1024 characters.
for i := range 10 * 1024 {
buffer.WriteString(fmt.Sprintf("[%05d] %s\n", i, line))
fmt.Fprintf(&buffer, "[%05d] %s\n", i, line)
}
putContent := buffer.String()
-22
View File
@@ -396,28 +396,6 @@ func newFileInfo(object string, dataBlocks, parityBlocks int) (fi FileInfo) {
return fi
}
// ReadMultipleReq contains information of multiple files to read from disk.
type ReadMultipleReq struct {
Bucket string `msg:"bk"` // Bucket. Can be empty if multiple buckets.
Prefix string `msg:"pr,omitempty"` // Shared prefix of all files. Can be empty. Will be joined to filename without modification.
Files []string `msg:"fl"` // Individual files to read.
MaxSize int64 `msg:"ms"` // Return error if size is exceed.
MetadataOnly bool `msg:"mo"` // Read as XL meta and truncate data.
AbortOn404 bool `msg:"ab"` // Stop reading after first file not found.
MaxResults int `msg:"mr"` // Stop after this many successful results. <= 0 means all.
}
// ReadMultipleResp contains a single response from a ReadMultipleReq.
type ReadMultipleResp struct {
Bucket string `msg:"bk"` // Bucket as given by request.
Prefix string `msg:"pr,omitempty"` // Prefix as given by request.
File string `msg:"fl"` // File name as given in request.
Exists bool `msg:"ex"` // Returns whether the file existed on disk.
Error string `msg:"er,omitempty"` // Returns any error when reading.
Data []byte `msg:"d"` // Contains all data of file.
Modtime time.Time `msg:"m"` // Modtime of file on disk.
}
// DeleteVersionHandlerParams are parameters for DeleteVersionHandler
type DeleteVersionHandlerParams struct {
DiskID string `msg:"id"`
-666
View File
@@ -4234,672 +4234,6 @@ func (z ReadAllHandlerParams) Msgsize() (s int) {
return
}
// DecodeMsg implements msgp.Decodable
func (z *ReadMultipleReq) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zb0001 uint32
zb0001, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err)
return
}
var zb0001Mask uint8 /* 1 bits */
_ = zb0001Mask
for zb0001 > 0 {
zb0001--
field, err = dc.ReadMapKeyPtr()
if err != nil {
err = msgp.WrapError(err)
return
}
switch msgp.UnsafeString(field) {
case "bk":
z.Bucket, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
case "pr":
z.Prefix, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
zb0001Mask |= 0x1
case "fl":
var zb0002 uint32
zb0002, err = dc.ReadArrayHeader()
if err != nil {
err = msgp.WrapError(err, "Files")
return
}
if cap(z.Files) >= int(zb0002) {
z.Files = (z.Files)[:zb0002]
} else {
z.Files = make([]string, zb0002)
}
for za0001 := range z.Files {
z.Files[za0001], err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "Files", za0001)
return
}
}
case "ms":
z.MaxSize, err = dc.ReadInt64()
if err != nil {
err = msgp.WrapError(err, "MaxSize")
return
}
case "mo":
z.MetadataOnly, err = dc.ReadBool()
if err != nil {
err = msgp.WrapError(err, "MetadataOnly")
return
}
case "ab":
z.AbortOn404, err = dc.ReadBool()
if err != nil {
err = msgp.WrapError(err, "AbortOn404")
return
}
case "mr":
z.MaxResults, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "MaxResults")
return
}
default:
err = dc.Skip()
if err != nil {
err = msgp.WrapError(err)
return
}
}
}
// Clear omitted fields.
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
return
}
// EncodeMsg implements msgp.Encodable
func (z *ReadMultipleReq) EncodeMsg(en *msgp.Writer) (err error) {
// check for omitted fields
zb0001Len := uint32(7)
var zb0001Mask uint8 /* 7 bits */
_ = zb0001Mask
if z.Prefix == "" {
zb0001Len--
zb0001Mask |= 0x2
}
// variable map header, size zb0001Len
err = en.Append(0x80 | uint8(zb0001Len))
if err != nil {
return
}
// skip if no fields are to be emitted
if zb0001Len != 0 {
// write "bk"
err = en.Append(0xa2, 0x62, 0x6b)
if err != nil {
return
}
err = en.WriteString(z.Bucket)
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
if (zb0001Mask & 0x2) == 0 { // if not omitted
// write "pr"
err = en.Append(0xa2, 0x70, 0x72)
if err != nil {
return
}
err = en.WriteString(z.Prefix)
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
}
// write "fl"
err = en.Append(0xa2, 0x66, 0x6c)
if err != nil {
return
}
err = en.WriteArrayHeader(uint32(len(z.Files)))
if err != nil {
err = msgp.WrapError(err, "Files")
return
}
for za0001 := range z.Files {
err = en.WriteString(z.Files[za0001])
if err != nil {
err = msgp.WrapError(err, "Files", za0001)
return
}
}
// write "ms"
err = en.Append(0xa2, 0x6d, 0x73)
if err != nil {
return
}
err = en.WriteInt64(z.MaxSize)
if err != nil {
err = msgp.WrapError(err, "MaxSize")
return
}
// write "mo"
err = en.Append(0xa2, 0x6d, 0x6f)
if err != nil {
return
}
err = en.WriteBool(z.MetadataOnly)
if err != nil {
err = msgp.WrapError(err, "MetadataOnly")
return
}
// write "ab"
err = en.Append(0xa2, 0x61, 0x62)
if err != nil {
return
}
err = en.WriteBool(z.AbortOn404)
if err != nil {
err = msgp.WrapError(err, "AbortOn404")
return
}
// write "mr"
err = en.Append(0xa2, 0x6d, 0x72)
if err != nil {
return
}
err = en.WriteInt(z.MaxResults)
if err != nil {
err = msgp.WrapError(err, "MaxResults")
return
}
}
return
}
// MarshalMsg implements msgp.Marshaler
func (z *ReadMultipleReq) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// check for omitted fields
zb0001Len := uint32(7)
var zb0001Mask uint8 /* 7 bits */
_ = zb0001Mask
if z.Prefix == "" {
zb0001Len--
zb0001Mask |= 0x2
}
// variable map header, size zb0001Len
o = append(o, 0x80|uint8(zb0001Len))
// skip if no fields are to be emitted
if zb0001Len != 0 {
// string "bk"
o = append(o, 0xa2, 0x62, 0x6b)
o = msgp.AppendString(o, z.Bucket)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// string "pr"
o = append(o, 0xa2, 0x70, 0x72)
o = msgp.AppendString(o, z.Prefix)
}
// string "fl"
o = append(o, 0xa2, 0x66, 0x6c)
o = msgp.AppendArrayHeader(o, uint32(len(z.Files)))
for za0001 := range z.Files {
o = msgp.AppendString(o, z.Files[za0001])
}
// string "ms"
o = append(o, 0xa2, 0x6d, 0x73)
o = msgp.AppendInt64(o, z.MaxSize)
// string "mo"
o = append(o, 0xa2, 0x6d, 0x6f)
o = msgp.AppendBool(o, z.MetadataOnly)
// string "ab"
o = append(o, 0xa2, 0x61, 0x62)
o = msgp.AppendBool(o, z.AbortOn404)
// string "mr"
o = append(o, 0xa2, 0x6d, 0x72)
o = msgp.AppendInt(o, z.MaxResults)
}
return
}
// UnmarshalMsg implements msgp.Unmarshaler
func (z *ReadMultipleReq) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zb0001 uint32
zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err)
return
}
var zb0001Mask uint8 /* 1 bits */
_ = zb0001Mask
for zb0001 > 0 {
zb0001--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
err = msgp.WrapError(err)
return
}
switch msgp.UnsafeString(field) {
case "bk":
z.Bucket, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
case "pr":
z.Prefix, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
zb0001Mask |= 0x1
case "fl":
var zb0002 uint32
zb0002, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Files")
return
}
if cap(z.Files) >= int(zb0002) {
z.Files = (z.Files)[:zb0002]
} else {
z.Files = make([]string, zb0002)
}
for za0001 := range z.Files {
z.Files[za0001], bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Files", za0001)
return
}
}
case "ms":
z.MaxSize, bts, err = msgp.ReadInt64Bytes(bts)
if err != nil {
err = msgp.WrapError(err, "MaxSize")
return
}
case "mo":
z.MetadataOnly, bts, err = msgp.ReadBoolBytes(bts)
if err != nil {
err = msgp.WrapError(err, "MetadataOnly")
return
}
case "ab":
z.AbortOn404, bts, err = msgp.ReadBoolBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AbortOn404")
return
}
case "mr":
z.MaxResults, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "MaxResults")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
err = msgp.WrapError(err)
return
}
}
}
// Clear omitted fields.
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
o = bts
return
}
// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *ReadMultipleReq) Msgsize() (s int) {
s = 1 + 3 + msgp.StringPrefixSize + len(z.Bucket) + 3 + msgp.StringPrefixSize + len(z.Prefix) + 3 + msgp.ArrayHeaderSize
for za0001 := range z.Files {
s += msgp.StringPrefixSize + len(z.Files[za0001])
}
s += 3 + msgp.Int64Size + 3 + msgp.BoolSize + 3 + msgp.BoolSize + 3 + msgp.IntSize
return
}
// DecodeMsg implements msgp.Decodable
func (z *ReadMultipleResp) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zb0001 uint32
zb0001, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err)
return
}
var zb0001Mask uint8 /* 2 bits */
_ = zb0001Mask
for zb0001 > 0 {
zb0001--
field, err = dc.ReadMapKeyPtr()
if err != nil {
err = msgp.WrapError(err)
return
}
switch msgp.UnsafeString(field) {
case "bk":
z.Bucket, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
case "pr":
z.Prefix, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
zb0001Mask |= 0x1
case "fl":
z.File, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "File")
return
}
case "ex":
z.Exists, err = dc.ReadBool()
if err != nil {
err = msgp.WrapError(err, "Exists")
return
}
case "er":
z.Error, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "Error")
return
}
zb0001Mask |= 0x2
case "d":
z.Data, err = dc.ReadBytes(z.Data)
if err != nil {
err = msgp.WrapError(err, "Data")
return
}
case "m":
z.Modtime, err = dc.ReadTime()
if err != nil {
err = msgp.WrapError(err, "Modtime")
return
}
default:
err = dc.Skip()
if err != nil {
err = msgp.WrapError(err)
return
}
}
}
// Clear omitted fields.
if zb0001Mask != 0x3 {
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
if (zb0001Mask & 0x2) == 0 {
z.Error = ""
}
}
return
}
// EncodeMsg implements msgp.Encodable
func (z *ReadMultipleResp) EncodeMsg(en *msgp.Writer) (err error) {
// check for omitted fields
zb0001Len := uint32(7)
var zb0001Mask uint8 /* 7 bits */
_ = zb0001Mask
if z.Prefix == "" {
zb0001Len--
zb0001Mask |= 0x2
}
if z.Error == "" {
zb0001Len--
zb0001Mask |= 0x10
}
// variable map header, size zb0001Len
err = en.Append(0x80 | uint8(zb0001Len))
if err != nil {
return
}
// skip if no fields are to be emitted
if zb0001Len != 0 {
// write "bk"
err = en.Append(0xa2, 0x62, 0x6b)
if err != nil {
return
}
err = en.WriteString(z.Bucket)
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
if (zb0001Mask & 0x2) == 0 { // if not omitted
// write "pr"
err = en.Append(0xa2, 0x70, 0x72)
if err != nil {
return
}
err = en.WriteString(z.Prefix)
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
}
// write "fl"
err = en.Append(0xa2, 0x66, 0x6c)
if err != nil {
return
}
err = en.WriteString(z.File)
if err != nil {
err = msgp.WrapError(err, "File")
return
}
// write "ex"
err = en.Append(0xa2, 0x65, 0x78)
if err != nil {
return
}
err = en.WriteBool(z.Exists)
if err != nil {
err = msgp.WrapError(err, "Exists")
return
}
if (zb0001Mask & 0x10) == 0 { // if not omitted
// write "er"
err = en.Append(0xa2, 0x65, 0x72)
if err != nil {
return
}
err = en.WriteString(z.Error)
if err != nil {
err = msgp.WrapError(err, "Error")
return
}
}
// write "d"
err = en.Append(0xa1, 0x64)
if err != nil {
return
}
err = en.WriteBytes(z.Data)
if err != nil {
err = msgp.WrapError(err, "Data")
return
}
// write "m"
err = en.Append(0xa1, 0x6d)
if err != nil {
return
}
err = en.WriteTime(z.Modtime)
if err != nil {
err = msgp.WrapError(err, "Modtime")
return
}
}
return
}
// MarshalMsg implements msgp.Marshaler
func (z *ReadMultipleResp) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// check for omitted fields
zb0001Len := uint32(7)
var zb0001Mask uint8 /* 7 bits */
_ = zb0001Mask
if z.Prefix == "" {
zb0001Len--
zb0001Mask |= 0x2
}
if z.Error == "" {
zb0001Len--
zb0001Mask |= 0x10
}
// variable map header, size zb0001Len
o = append(o, 0x80|uint8(zb0001Len))
// skip if no fields are to be emitted
if zb0001Len != 0 {
// string "bk"
o = append(o, 0xa2, 0x62, 0x6b)
o = msgp.AppendString(o, z.Bucket)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// string "pr"
o = append(o, 0xa2, 0x70, 0x72)
o = msgp.AppendString(o, z.Prefix)
}
// string "fl"
o = append(o, 0xa2, 0x66, 0x6c)
o = msgp.AppendString(o, z.File)
// string "ex"
o = append(o, 0xa2, 0x65, 0x78)
o = msgp.AppendBool(o, z.Exists)
if (zb0001Mask & 0x10) == 0 { // if not omitted
// string "er"
o = append(o, 0xa2, 0x65, 0x72)
o = msgp.AppendString(o, z.Error)
}
// string "d"
o = append(o, 0xa1, 0x64)
o = msgp.AppendBytes(o, z.Data)
// string "m"
o = append(o, 0xa1, 0x6d)
o = msgp.AppendTime(o, z.Modtime)
}
return
}
// UnmarshalMsg implements msgp.Unmarshaler
func (z *ReadMultipleResp) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zb0001 uint32
zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err)
return
}
var zb0001Mask uint8 /* 2 bits */
_ = zb0001Mask
for zb0001 > 0 {
zb0001--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
err = msgp.WrapError(err)
return
}
switch msgp.UnsafeString(field) {
case "bk":
z.Bucket, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
case "pr":
z.Prefix, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
zb0001Mask |= 0x1
case "fl":
z.File, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "File")
return
}
case "ex":
z.Exists, bts, err = msgp.ReadBoolBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Exists")
return
}
case "er":
z.Error, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Error")
return
}
zb0001Mask |= 0x2
case "d":
z.Data, bts, err = msgp.ReadBytesBytes(bts, z.Data)
if err != nil {
err = msgp.WrapError(err, "Data")
return
}
case "m":
z.Modtime, bts, err = msgp.ReadTimeBytes(bts)
if err != nil {
err = msgp.WrapError(err, "Modtime")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
err = msgp.WrapError(err)
return
}
}
}
// Clear omitted fields.
if zb0001Mask != 0x3 {
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
if (zb0001Mask & 0x2) == 0 {
z.Error = ""
}
}
o = bts
return
}
// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *ReadMultipleResp) Msgsize() (s int) {
s = 1 + 3 + msgp.StringPrefixSize + len(z.Bucket) + 3 + msgp.StringPrefixSize + len(z.Prefix) + 3 + msgp.StringPrefixSize + len(z.File) + 3 + msgp.BoolSize + 3 + msgp.StringPrefixSize + len(z.Error) + 2 + msgp.BytesPrefixSize + len(z.Data) + 2 + msgp.TimeSize
return
}
// DecodeMsg implements msgp.Decodable
func (z *ReadPartsReq) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
-226
View File
@@ -2156,232 +2156,6 @@ func BenchmarkDecodeReadAllHandlerParams(b *testing.B) {
}
}
func TestMarshalUnmarshalReadMultipleReq(t *testing.T) {
v := ReadMultipleReq{}
bts, err := v.MarshalMsg(nil)
if err != nil {
t.Fatal(err)
}
left, err := v.UnmarshalMsg(bts)
if err != nil {
t.Fatal(err)
}
if len(left) > 0 {
t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
}
left, err = msgp.Skip(bts)
if err != nil {
t.Fatal(err)
}
if len(left) > 0 {
t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
}
}
func BenchmarkMarshalMsgReadMultipleReq(b *testing.B) {
v := ReadMultipleReq{}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
v.MarshalMsg(nil)
}
}
func BenchmarkAppendMsgReadMultipleReq(b *testing.B) {
v := ReadMultipleReq{}
bts := make([]byte, 0, v.Msgsize())
bts, _ = v.MarshalMsg(bts[0:0])
b.SetBytes(int64(len(bts)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
bts, _ = v.MarshalMsg(bts[0:0])
}
}
func BenchmarkUnmarshalReadMultipleReq(b *testing.B) {
v := ReadMultipleReq{}
bts, _ := v.MarshalMsg(nil)
b.ReportAllocs()
b.SetBytes(int64(len(bts)))
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := v.UnmarshalMsg(bts)
if err != nil {
b.Fatal(err)
}
}
}
func TestEncodeDecodeReadMultipleReq(t *testing.T) {
v := ReadMultipleReq{}
var buf bytes.Buffer
msgp.Encode(&buf, &v)
m := v.Msgsize()
if buf.Len() > m {
t.Log("WARNING: TestEncodeDecodeReadMultipleReq Msgsize() is inaccurate")
}
vn := ReadMultipleReq{}
err := msgp.Decode(&buf, &vn)
if err != nil {
t.Error(err)
}
buf.Reset()
msgp.Encode(&buf, &v)
err = msgp.NewReader(&buf).Skip()
if err != nil {
t.Error(err)
}
}
func BenchmarkEncodeReadMultipleReq(b *testing.B) {
v := ReadMultipleReq{}
var buf bytes.Buffer
msgp.Encode(&buf, &v)
b.SetBytes(int64(buf.Len()))
en := msgp.NewWriter(msgp.Nowhere)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
v.EncodeMsg(en)
}
en.Flush()
}
func BenchmarkDecodeReadMultipleReq(b *testing.B) {
v := ReadMultipleReq{}
var buf bytes.Buffer
msgp.Encode(&buf, &v)
b.SetBytes(int64(buf.Len()))
rd := msgp.NewEndlessReader(buf.Bytes(), b)
dc := msgp.NewReader(rd)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
err := v.DecodeMsg(dc)
if err != nil {
b.Fatal(err)
}
}
}
func TestMarshalUnmarshalReadMultipleResp(t *testing.T) {
v := ReadMultipleResp{}
bts, err := v.MarshalMsg(nil)
if err != nil {
t.Fatal(err)
}
left, err := v.UnmarshalMsg(bts)
if err != nil {
t.Fatal(err)
}
if len(left) > 0 {
t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
}
left, err = msgp.Skip(bts)
if err != nil {
t.Fatal(err)
}
if len(left) > 0 {
t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
}
}
func BenchmarkMarshalMsgReadMultipleResp(b *testing.B) {
v := ReadMultipleResp{}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
v.MarshalMsg(nil)
}
}
func BenchmarkAppendMsgReadMultipleResp(b *testing.B) {
v := ReadMultipleResp{}
bts := make([]byte, 0, v.Msgsize())
bts, _ = v.MarshalMsg(bts[0:0])
b.SetBytes(int64(len(bts)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
bts, _ = v.MarshalMsg(bts[0:0])
}
}
func BenchmarkUnmarshalReadMultipleResp(b *testing.B) {
v := ReadMultipleResp{}
bts, _ := v.MarshalMsg(nil)
b.ReportAllocs()
b.SetBytes(int64(len(bts)))
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := v.UnmarshalMsg(bts)
if err != nil {
b.Fatal(err)
}
}
}
func TestEncodeDecodeReadMultipleResp(t *testing.T) {
v := ReadMultipleResp{}
var buf bytes.Buffer
msgp.Encode(&buf, &v)
m := v.Msgsize()
if buf.Len() > m {
t.Log("WARNING: TestEncodeDecodeReadMultipleResp Msgsize() is inaccurate")
}
vn := ReadMultipleResp{}
err := msgp.Decode(&buf, &vn)
if err != nil {
t.Error(err)
}
buf.Reset()
msgp.Encode(&buf, &v)
err = msgp.NewReader(&buf).Skip()
if err != nil {
t.Error(err)
}
}
func BenchmarkEncodeReadMultipleResp(b *testing.B) {
v := ReadMultipleResp{}
var buf bytes.Buffer
msgp.Encode(&buf, &v)
b.SetBytes(int64(buf.Len()))
en := msgp.NewWriter(msgp.Nowhere)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
v.EncodeMsg(en)
}
en.Flush()
}
func BenchmarkDecodeReadMultipleResp(b *testing.B) {
v := ReadMultipleResp{}
var buf bytes.Buffer
msgp.Encode(&buf, &v)
b.SetBytes(int64(buf.Len()))
rd := msgp.NewEndlessReader(buf.Bytes(), b)
dc := msgp.NewReader(rd)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
err := v.DecodeMsg(dc)
if err != nil {
b.Fatal(err)
}
}
}
func TestMarshalUnmarshalReadPartsReq(t *testing.T) {
v := ReadPartsReq{}
bts, err := v.MarshalMsg(nil)
-1
View File
@@ -101,7 +101,6 @@ type StorageAPI interface {
VerifyFile(ctx context.Context, volume, path string, fi FileInfo) (*CheckPartsResp, error)
StatInfoFile(ctx context.Context, volume, path string, glob bool) (stat []StatInfo, err error)
ReadParts(ctx context.Context, bucket string, partMetaPaths ...string) ([]*ObjectPartInfo, error)
ReadMultiple(ctx context.Context, req ReadMultipleReq, resp chan<- ReadMultipleResp) error
CleanAbandonedData(ctx context.Context, volume string, path string) error
// Write all data, syncs the data to disk.
-39
View File
@@ -912,45 +912,6 @@ func (client *storageRESTClient) StatInfoFile(ctx context.Context, volume, path
return stat, toStorageErr(err)
}
// ReadMultiple will read multiple files and send each back as response.
// Files are read and returned in the given order.
// The resp channel is closed before the call returns.
// Only a canceled context or network errors returns an error.
func (client *storageRESTClient) ReadMultiple(ctx context.Context, req ReadMultipleReq, resp chan<- ReadMultipleResp) error {
defer xioutil.SafeClose(resp)
body, err := req.MarshalMsg(nil)
if err != nil {
return err
}
respBody, err := client.call(ctx, storageRESTMethodReadMultiple, nil, bytes.NewReader(body), int64(len(body)))
if err != nil {
return err
}
defer xhttp.DrainBody(respBody)
pr, pw := io.Pipe()
go func() {
pw.CloseWithError(waitForHTTPStream(respBody, xioutil.NewDeadlineWriter(pw, globalDriveConfig.GetMaxTimeout())))
}()
mr := msgp.NewReader(pr)
defer readMsgpReaderPoolPut(mr)
for {
var file ReadMultipleResp
if err := file.DecodeMsg(mr); err != nil {
if errors.Is(err, io.EOF) {
err = nil
}
pr.CloseWithError(err)
return toStorageErr(err)
}
select {
case <-ctx.Done():
return ctx.Err()
case resp <- file:
}
}
}
// CleanAbandonedData will read metadata of the object on disk
// and delete any data directories and inline data that isn't referenced in metadata.
func (client *storageRESTClient) CleanAbandonedData(ctx context.Context, volume string, path string) error {
-1
View File
@@ -41,7 +41,6 @@ const (
storageRESTMethodRenameFile = "/rfile"
storageRESTMethodVerifyFile = "/vfile"
storageRESTMethodStatInfoFile = "/sfile"
storageRESTMethodReadMultiple = "/rmpl"
storageRESTMethodCleanAbandoned = "/cln"
storageRESTMethodDeleteBulk = "/dblk"
storageRESTMethodReadParts = "/rps"
-44
View File
@@ -28,7 +28,6 @@ import (
"net/http"
"os/user"
"path"
"runtime/debug"
"strconv"
"strings"
"sync"
@@ -1283,48 +1282,6 @@ func (s *storageRESTServer) DeleteBulkHandler(w http.ResponseWriter, r *http.Req
keepHTTPResponseAlive(w)(s.getStorage().DeleteBulk(r.Context(), volume, req.Paths...))
}
// ReadMultiple returns multiple files
func (s *storageRESTServer) ReadMultiple(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
return
}
rw := streamHTTPResponse(w)
defer func() {
if r := recover(); r != nil {
debug.PrintStack()
rw.CloseWithError(fmt.Errorf("panic: %v", r))
}
}()
var req ReadMultipleReq
mr := msgpNewReader(r.Body)
defer readMsgpReaderPoolPut(mr)
err := req.DecodeMsg(mr)
if err != nil {
rw.CloseWithError(err)
return
}
mw := msgp.NewWriter(rw)
responses := make(chan ReadMultipleResp, len(req.Files))
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for resp := range responses {
err := resp.EncodeMsg(mw)
if err != nil {
rw.CloseWithError(err)
return
}
mw.Flush()
}
}()
err = s.getStorage().ReadMultiple(r.Context(), req, responses)
wg.Wait()
rw.CloseWithError(err)
}
// globalLocalSetDrives is used for local drive as well as remote REST
// API caller for other nodes to talk to this node.
//
@@ -1363,7 +1320,6 @@ func registerStorageRESTHandlers(router *mux.Router, endpointServerPools Endpoin
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodDeleteVersions).HandlerFunc(h(server.DeleteVersionsHandler))
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodVerifyFile).HandlerFunc(h(server.VerifyFileHandler))
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodStatInfoFile).HandlerFunc(h(server.StatInfoFile))
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadMultiple).HandlerFunc(h(server.ReadMultiple))
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodCleanAbandoned).HandlerFunc(h(server.CleanAbandonedDataHandler))
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodDeleteBulk).HandlerFunc(h(server.DeleteBulkHandler))
subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadParts).HandlerFunc(h(server.ReadPartsHandler))
+8 -9
View File
@@ -33,18 +33,17 @@ func _() {
_ = x[storageMetricReadXL-22]
_ = x[storageMetricReadAll-23]
_ = x[storageMetricStatInfoFile-24]
_ = x[storageMetricReadMultiple-25]
_ = x[storageMetricDeleteAbandonedParts-26]
_ = x[storageMetricDiskInfo-27]
_ = x[storageMetricDeleteBulk-28]
_ = x[storageMetricRenamePart-29]
_ = x[storageMetricReadParts-30]
_ = x[storageMetricLast-31]
_ = x[storageMetricDeleteAbandonedParts-25]
_ = x[storageMetricDiskInfo-26]
_ = x[storageMetricDeleteBulk-27]
_ = x[storageMetricRenamePart-28]
_ = x[storageMetricReadParts-29]
_ = x[storageMetricLast-30]
}
const _storageMetric_name = "MakeVolBulkMakeVolListVolsStatVolDeleteVolWalkDirListDirReadFileAppendFileCreateFileReadFileStreamRenameFileRenameDataCheckPartsDeleteDeleteVersionsVerifyFileWriteAllDeleteVersionWriteMetadataUpdateMetadataReadVersionReadXLReadAllStatInfoFileReadMultipleDeleteAbandonedPartsDiskInfoDeleteBulkRenamePartReadPartsLast"
const _storageMetric_name = "MakeVolBulkMakeVolListVolsStatVolDeleteVolWalkDirListDirReadFileAppendFileCreateFileReadFileStreamRenameFileRenameDataCheckPartsDeleteDeleteVersionsVerifyFileWriteAllDeleteVersionWriteMetadataUpdateMetadataReadVersionReadXLReadAllStatInfoFileDeleteAbandonedPartsDiskInfoDeleteBulkRenamePartReadPartsLast"
var _storageMetric_index = [...]uint16{0, 11, 18, 26, 33, 42, 49, 56, 64, 74, 84, 98, 108, 118, 128, 134, 148, 158, 166, 179, 192, 206, 217, 223, 230, 242, 254, 274, 282, 292, 302, 311, 315}
var _storageMetric_index = [...]uint16{0, 11, 18, 26, 33, 42, 49, 56, 64, 74, 84, 98, 108, 118, 128, 134, 148, 158, 166, 179, 192, 206, 217, 223, 230, 242, 262, 270, 280, 290, 299, 303}
func (i storageMetric) String() string {
if i >= storageMetric(len(_storageMetric_index)-1) {
+17 -5
View File
@@ -26,12 +26,24 @@ import (
"strings"
)
// newUnsignedV4ChunkedReader returns a new s3UnsignedChunkedReader that translates the data read from r
// out of HTTP "chunked" format before returning it.
// newUnsignedV4ChunkedReader returns a new s3UnsignedChunkedReader that translates
// the data read from r out of HTTP "chunked" format before returning it.
//
// STREAMING-UNSIGNED-PAYLOAD-TRAILER requests cross the auth boundary here because
// the handler starts consuming the body immediately after this reader is created.
// Header-authenticated requests must therefore be validated before any body bytes
// are read, preserving the security boundary added in #21103, while presigned
// query-string auth is intentionally rejected for this streaming mode.
//
// The s3ChunkedReader returns io.EOF when the final 0-length chunk is read.
func newUnsignedV4ChunkedReader(req *http.Request, trailer bool, signature bool) (io.ReadCloser, APIErrorCode) {
if signature {
if errCode := doesSignatureMatch(unsignedPayloadTrailer, req, globalSite.Region(), serviceS3); errCode != ErrNone {
func newUnsignedV4ChunkedReader(req *http.Request, trailer bool) (io.ReadCloser, APIErrorCode) {
// Reject any request that claims presigned auth parameters are in use, even if
// the query is incomplete, to avoid silently downgrading it into the anonymous path.
if isRequestPresignedSignatureV4(req) {
return nil, ErrSignatureVersionNotSupported
}
if isRequestSignatureV4(req) {
if errCode := reqSignatureV4Verify(req, globalSite.Region(), serviceS3); errCode != ErrNone {
return nil, errCode
}
}
+332 -4
View File
@@ -25,14 +25,17 @@ import (
"encoding/json"
"errors"
"fmt"
"net"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
idldap "github.com/minio/minio/internal/config/identity/ldap"
"github.com/minio/minio/internal/config/identity/openid"
"github.com/minio/minio/internal/hash/sha256"
xhttp "github.com/minio/minio/internal/http"
@@ -91,6 +94,15 @@ const (
// maximum supported STS session policy size
maxSTSSessionPolicySize = 2048
stsLDAPLoginBurst = 10
stsLDAPLoginEntryTTL = 15 * time.Minute
stsLDAPLoginRetryAfterSec = int((time.Minute / stsLDAPLoginBurst) / time.Second)
)
var (
errLDAPAuthenticationFailed = errors.New("LDAP authentication failed")
globalSTSLDAPLoginRateLimiter = newSTSLDAPLoginRateLimiter(time.Minute/time.Duration(stsLDAPLoginBurst), stsLDAPLoginBurst, stsLDAPLoginEntryTTL)
)
type stsClaims map[string]any
@@ -254,6 +266,308 @@ func getTokenSigningKey() (string, error) {
return secret, nil
}
// stsLDAPLoginRateLimiter throttles LDAP STS logins per source IP. It does not
// bucket by username on purpose: a username-keyed bucket is shared across all
// sources, so a single client sending bad-password attempts for a known account
// could keep that account's bucket drained and lock the legitimate user out.
//
// Scope of protection: the per-source bucket caps the attempt rate from any one
// source, and the uniform auth-failure response (not this limiter) is what hides
// whether a username exists. It does not stop attackers who spread across many
// sources (botnets, IPv6 address rotation) or the residual bind-timing side
// channel; those are accepted limitations of an in-memory, per-source control.
type stsLDAPLoginRateLimiter struct {
source *stsLDAPLoginKeyLimiterSet
}
type stsLDAPLoginReservation struct {
source *stsLDAPLoginKeyReservation
}
type stsLDAPLoginKeyLimiterSet struct {
mu sync.Mutex
refillEvery time.Duration
burst int
ttl time.Duration
lastCleanup time.Time
entries map[string]*stsLDAPLoginKeyLimiter
}
type stsLDAPLoginKeyLimiter struct {
tokens float64
lastRefill time.Time
lastSeen time.Time
inFlight int
}
type stsLDAPLoginKeyReservation struct {
set *stsLDAPLoginKeyLimiterSet
entry *stsLDAPLoginKeyLimiter
finalized bool
}
func newSTSLDAPLoginRateLimiter(refillEvery time.Duration, burst int, ttl time.Duration) *stsLDAPLoginRateLimiter {
return &stsLDAPLoginRateLimiter{
source: newSTSLDAPLoginKeyLimiterSet(refillEvery, burst, ttl),
}
}
func newSTSLDAPLoginKeyLimiterSet(refillEvery time.Duration, burst int, ttl time.Duration) *stsLDAPLoginKeyLimiterSet {
return &stsLDAPLoginKeyLimiterSet{
refillEvery: refillEvery,
burst: burst,
ttl: ttl,
entries: make(map[string]*stsLDAPLoginKeyLimiter),
}
}
func (l *stsLDAPLoginRateLimiter) Allow(sourceIP string) bool {
reservation := l.Reserve(sourceIP)
if reservation == nil {
return false
}
reservation.Commit()
return true
}
func (l *stsLDAPLoginRateLimiter) Reserve(sourceIP string) *stsLDAPLoginReservation {
// An empty source IP means we could not identify the peer; do not throttle
// rather than collapse every such request into one shared bucket.
if sourceIP == "" {
return &stsLDAPLoginReservation{}
}
source := l.source.Reserve(UTCNow(), sourceIP)
if source == nil {
return nil
}
return &stsLDAPLoginReservation{source: source}
}
func (r *stsLDAPLoginReservation) Commit() {
if r == nil || r.source == nil {
return
}
r.source.CommitAt(UTCNow())
r.source = nil
}
func (r *stsLDAPLoginReservation) Cancel() {
if r == nil || r.source == nil {
return
}
r.source.CancelAt(UTCNow())
r.source = nil
}
func (l *stsLDAPLoginKeyLimiterSet) Allow(now time.Time, key string) bool {
reservation := l.Reserve(now, key)
if reservation == nil {
return false
}
reservation.CommitAt(now)
return true
}
func (l *stsLDAPLoginKeyLimiterSet) Reserve(now time.Time, key string) *stsLDAPLoginKeyReservation {
l.mu.Lock()
defer l.mu.Unlock()
if l.lastCleanup.IsZero() || now.Sub(l.lastCleanup) >= l.ttl {
l.cleanup(now)
l.lastCleanup = now
}
entry := l.getOrCreateLocked(now, key)
l.refillLocked(now, entry)
if entry.tokens < 1 {
entry.lastSeen = now
return nil
}
entry.tokens--
entry.inFlight++
entry.lastSeen = now
return &stsLDAPLoginKeyReservation{set: l, entry: entry}
}
func (l *stsLDAPLoginKeyLimiterSet) cleanup(now time.Time) {
for key, entry := range l.entries {
if entry.inFlight == 0 && now.Sub(entry.lastSeen) > l.ttl {
delete(l.entries, key)
}
}
}
func (l *stsLDAPLoginKeyLimiterSet) getOrCreateLocked(now time.Time, key string) *stsLDAPLoginKeyLimiter {
entry, ok := l.entries[key]
if !ok {
entry = &stsLDAPLoginKeyLimiter{
tokens: float64(l.burst),
lastRefill: now,
lastSeen: now,
}
l.entries[key] = entry
}
return entry
}
func (l *stsLDAPLoginKeyLimiterSet) refillLocked(now time.Time, entry *stsLDAPLoginKeyLimiter) {
if entry.lastRefill.IsZero() {
entry.lastRefill = now
}
lastRefill := entry.lastRefill
if now.Before(lastRefill) {
lastRefill = now
}
if l.refillEvery <= 0 {
entry.lastRefill = now
entry.tokens = l.maxTokensLocked(entry)
return
}
entry.tokens += float64(now.Sub(lastRefill)) / float64(l.refillEvery)
if maxTokens := l.maxTokensLocked(entry); entry.tokens > maxTokens {
entry.tokens = maxTokens
}
entry.lastRefill = now
}
func (l *stsLDAPLoginKeyLimiterSet) maxTokensLocked(entry *stsLDAPLoginKeyLimiter) float64 {
maxTokens := l.burst - entry.inFlight
if maxTokens < 0 {
return 0
}
return float64(maxTokens)
}
func (r *stsLDAPLoginKeyReservation) CommitAt(now time.Time) {
r.finalize(now, false)
}
func (r *stsLDAPLoginKeyReservation) CancelAt(now time.Time) {
r.finalize(now, true)
}
func (r *stsLDAPLoginKeyReservation) finalize(now time.Time, refund bool) {
if r == nil {
return
}
r.set.mu.Lock()
defer r.set.mu.Unlock()
if r.finalized {
return
}
r.set.refillLocked(now, r.entry)
r.entry.lastSeen = now
if r.entry.inFlight > 0 {
r.entry.inFlight--
}
if refund {
r.entry.tokens++
if maxTokens := r.set.maxTokensLocked(r.entry); r.entry.tokens > maxTokens {
r.entry.tokens = maxTokens
}
}
r.finalized = true
}
func getSTSLDAPLoginSourceIP(r *http.Request) string {
peerIP := getSTSLDAPLoginCanonicalIP(r.RemoteAddr)
sourceIP := peerIP
if sourceIP == "" {
sourceIP = getSTSLDAPLoginPeerAddr(r.RemoteAddr)
}
if peerIP != "" && globalIAMSys != nil && globalIAMSys.LDAPConfig.IsSTSTrustedProxy(peerIP) {
if forwardedIP := getSTSLDAPTrustedProxySourceIP(r); forwardedIP != "" {
return forwardedIP
}
}
return sourceIP
}
// getSTSLDAPTrustedProxySourceIP resolves the client IP for a request whose peer
// is an allow-listed trusted proxy. A single clean X-Real-IP is preferred; for
// X-Forwarded-For we walk the chain right-to-left and skip trusted-proxy hops,
// returning the first untrusted address. The XFF result ignores any client-
// supplied (left-most) value unless the entire chain to its right is trusted,
// which an external client cannot forge.
//
// X-Real-IP, unlike XFF, is a single value with no chain, so it cannot be
// validated against the allowlist: it is trusted verbatim. The deployment
// contract is therefore that the trusted proxy MUST overwrite (not pass through)
// any client-supplied X-Real-IP; otherwise an attacker can vary it per request
// to evade per-source throttling. This is the standard reverse-proxy real-IP
// contract; reordering to prefer XFF would not remove the dependency, only move
// it (an X-Real-IP-only proxy would then be evaded via an injected XFF header).
//
// The RFC 7239 Forwarded header is intentionally not honored here; such
// deployments fall back to the safe peer-address bucket.
func getSTSLDAPTrustedProxySourceIP(r *http.Request) string {
if realIP := getSTSLDAPLoginCanonicalIP(r.Header.Get("X-Real-IP")); realIP != "" {
return realIP
}
forwarded := strings.Split(r.Header.Get("X-Forwarded-For"), ",")
for i := len(forwarded) - 1; i >= 0; i-- {
ip := getSTSLDAPLoginCanonicalIP(forwarded[i])
if ip == "" || globalIAMSys.LDAPConfig.IsSTSTrustedProxy(ip) {
continue
}
return ip
}
return ""
}
func getSTSLDAPLoginPeerAddr(remoteAddr string) string {
sourceIP, _, err := net.SplitHostPort(remoteAddr)
if err == nil {
return sourceIP
}
return remoteAddr
}
func getSTSLDAPLoginCanonicalIP(addr string) string {
addr = strings.TrimSpace(getSTSLDAPLoginPeerAddr(addr))
addr = strings.TrimPrefix(addr, "[")
addr = strings.TrimSuffix(addr, "]")
if ip := net.ParseIP(addr); ip != nil {
return ip.String()
}
return ""
}
// reserveSTSLDAPLogin acquires an immediate token from the per-source limiter
// before contacting LDAP. Call Commit on auth failures and Cancel when the
// attempt should not count as an authentication failure.
func reserveSTSLDAPLogin(r *http.Request) *stsLDAPLoginReservation {
return globalSTSLDAPLoginRateLimiter.Reserve(getSTSLDAPLoginSourceIP(r))
}
func ldapBindErrorToSTS(err error) (STSErrorCode, error) {
if idldap.IsAuthError(err) {
return ErrSTSInvalidParameterValue, errLDAPAuthenticationFailed
}
return ErrSTSUpstreamError, nil
}
func writeSTSThrottledResponse(w http.ResponseWriter) {
stsErrorResponse := STSErrorResponse{}
stsErrorResponse.Error.Code = "ThrottlingException"
stsErrorResponse.Error.Message = "Request throttled, please retry later."
stsErrorResponse.RequestID = w.Header().Get(xhttp.AmzRequestID)
w.Header().Set("Retry-After", strconv.Itoa(stsLDAPLoginRetryAfterSec))
encodedErrorResponse := encodeResponse(stsErrorResponse)
writeResponse(w, http.StatusTooManyRequests, encodedErrorResponse, mimeXML)
}
// AssumeRole - implementation of AWS STS API AssumeRole to get temporary
// credentials for regular users on Minio.
// https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html
@@ -690,12 +1004,26 @@ func (sts *stsAPIHandlers) AssumeRoleWithLDAPIdentity(w http.ResponseWriter, r *
return
}
lookupResult, groupDistNames, err := globalIAMSys.LDAPConfig.Bind(ldapUsername, ldapPassword)
if err != nil {
err = fmt.Errorf("LDAP server error: %w", err)
writeSTSErrorResponse(ctx, w, ErrSTSInvalidParameterValue, err)
loginReservation := reserveSTSLDAPLogin(r)
if loginReservation == nil {
writeSTSThrottledResponse(w)
return
}
defer loginReservation.Cancel()
lookupResult, groupDistNames, err := globalIAMSys.LDAPConfig.Bind(ldapUsername, ldapPassword)
if err != nil {
errCode, errResp := ldapBindErrorToSTS(err)
if errCode == ErrSTSUpstreamError {
loginReservation.Cancel()
stsLogIf(ctx, err, logger.ErrorKind)
} else {
loginReservation.Commit()
}
writeSTSErrorResponse(ctx, w, errCode, errResp)
return
}
loginReservation.Cancel()
ldapUserDN := lookupResult.NormDN
ldapActualUserDN := lookupResult.ActualDN
+918
View File
@@ -20,12 +20,19 @@ package cmd
import (
"bytes"
"context"
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"os"
"reflect"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -977,6 +984,345 @@ func TestIAMWithLDAPServerSuite(t *testing.T) {
}
}
type ldapSTSErrorResult struct {
StatusCode int
RetryAfter string
Code string
Message string
Body string
}
type ldapSTSHTTPResult struct {
StatusCode int
RetryAfter string
Body string
}
func withGlobalSTSLDAPLoginRateLimiterForTest(limiter *stsLDAPLoginRateLimiter, fn func()) {
previous := globalSTSLDAPLoginRateLimiter
globalSTSLDAPLoginRateLimiter = limiter
defer func() {
globalSTSLDAPLoginRateLimiter = previous
}()
fn()
}
func withLDAPSTSTrustedProxiesForTest(t *testing.T, trustedProxies string, fn func()) {
t.Helper()
previousIAMSys := globalIAMSys
if globalIAMSys == nil {
globalIAMSys = &IAMSys{}
}
previous := globalIAMSys.LDAPConfig.Clone()
if err := globalIAMSys.LDAPConfig.SetSTSTrustedProxies(trustedProxies); err != nil {
t.Fatalf("unable to set LDAP STS trusted proxies for test: %v", err)
}
defer func() {
globalIAMSys.LDAPConfig = previous
globalIAMSys = previousIAMSys
}()
fn()
}
func singleHeader(key, value string) http.Header {
header := make(http.Header)
if key != "" {
header.Set(key, value)
}
return header
}
func (s *TestSuiteIAM) postLDAPSTSWithHeaders(c *check, username, password string, headers http.Header) ldapSTSHTTPResult {
c.Helper()
ctx, cancel := context.WithTimeout(context.Background(), testDefaultTimeout)
defer cancel()
form := url.Values{}
form.Set("Action", ldapIdentity)
form.Set("Version", stsAPIVersion)
form.Set(stsLDAPUsername, username)
form.Set(stsLDAPPassword, password)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.endPoint, strings.NewReader(form.Encode()))
if err != nil {
c.Fatalf("unexpected request creation error: %v", err)
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
for key, values := range headers {
for _, value := range values {
req.Header.Add(key, value)
}
}
resp, err := s.TestSuiteCommon.client.Do(req)
if err != nil {
c.Fatalf("unexpected LDAP STS request error: %v", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
c.Fatalf("unexpected LDAP STS response read error: %v", err)
}
return ldapSTSHTTPResult{
StatusCode: resp.StatusCode,
RetryAfter: resp.Header.Get("Retry-After"),
Body: string(body),
}
}
func (s *TestSuiteIAM) postLDAPSTS(c *check, username, password string) ldapSTSHTTPResult {
c.Helper()
return s.postLDAPSTSWithHeaders(c, username, password, nil)
}
func (s *TestSuiteIAM) postLDAPSTSForError(c *check, username, password string) ldapSTSErrorResult {
c.Helper()
result := s.postLDAPSTS(c, username, password)
if result.StatusCode == http.StatusOK {
c.Fatalf("expected LDAP STS request to fail, got success: %s", result.Body)
}
var stsErr STSErrorResponse
if err := xml.Unmarshal([]byte(result.Body), &stsErr); err != nil {
c.Fatalf("unexpected LDAP STS XML decode error: %v, body: %s", err, result.Body)
}
return ldapSTSErrorResult{
StatusCode: result.StatusCode,
RetryAfter: result.RetryAfter,
Code: stsErr.Error.Code,
Message: stsErr.Error.Message,
Body: result.Body,
}
}
func (s *TestSuiteIAM) TestLDAPSTSAuthFailureUniformResponse(c *check) {
withGlobalSTSLDAPLoginRateLimiterForTest(
newSTSLDAPLoginRateLimiter(time.Minute, stsLDAPLoginBurst, stsLDAPLoginEntryTTL),
func() {
missingUser := s.postLDAPSTSForError(c, "missing-user", "nottherightpassword")
wrongPassword := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
if missingUser.StatusCode != http.StatusBadRequest {
c.Fatalf("expected missing-user request status %d, got %d", http.StatusBadRequest, missingUser.StatusCode)
}
if wrongPassword.StatusCode != http.StatusBadRequest {
c.Fatalf("expected wrong-password request status %d, got %d", http.StatusBadRequest, wrongPassword.StatusCode)
}
if missingUser.Code != "InvalidParameterValue" || wrongPassword.Code != "InvalidParameterValue" {
c.Fatalf("expected InvalidParameterValue for both auth failures, got missing=%q wrong=%q", missingUser.Code, wrongPassword.Code)
}
if missingUser.Message != errLDAPAuthenticationFailed.Error() || wrongPassword.Message != errLDAPAuthenticationFailed.Error() {
c.Fatalf("expected uniform LDAP auth failure message, got missing=%q wrong=%q", missingUser.Message, wrongPassword.Message)
}
if strings.Contains(strings.ToLower(missingUser.Body), "unable to find user dn") {
c.Fatalf("missing-user response leaked lookup details: %s", missingUser.Body)
}
if strings.Contains(strings.ToLower(wrongPassword.Body), "ldap auth failed for dn") {
c.Fatalf("wrong-password response leaked bind details: %s", wrongPassword.Body)
}
},
)
}
func (s *TestSuiteIAM) TestLDAPSTSRateLimit(c *check) {
withGlobalSTSLDAPLoginRateLimiterForTest(
newSTSLDAPLoginRateLimiter(time.Hour, 2, stsLDAPLoginEntryTTL),
func() {
first := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
second := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
throttled := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
if first.StatusCode != http.StatusBadRequest || second.StatusCode != http.StatusBadRequest {
c.Fatalf("expected first two failed auth attempts to return %d, got %d and %d", http.StatusBadRequest, first.StatusCode, second.StatusCode)
}
if throttled.StatusCode != http.StatusTooManyRequests {
c.Fatalf("expected throttled request status %d, got %d", http.StatusTooManyRequests, throttled.StatusCode)
}
if throttled.Code != "ThrottlingException" {
c.Fatalf("expected throttled code %q, got %q", "ThrottlingException", throttled.Code)
}
if throttled.Message != "Request throttled, please retry later." {
c.Fatalf("expected throttled message %q, got %q", "Request throttled, please retry later.", throttled.Message)
}
if throttled.RetryAfter != fmt.Sprintf("%d", stsLDAPLoginRetryAfterSec) {
c.Fatalf("expected Retry-After %d, got %q", stsLDAPLoginRetryAfterSec, throttled.RetryAfter)
}
},
)
}
func (s *TestSuiteIAM) TestLDAPSTSSuccessDoesNotConsumeRateLimit(c *check) {
ctx, cancel := context.WithTimeout(context.Background(), testDefaultTimeout)
defer cancel()
userReq := madmin.PolicyAssociationReq{
Policies: []string{"consoleAdmin"},
User: "uid=dillon,ou=people,ou=swengg,dc=min,dc=io",
}
if _, err := s.adm.AttachPolicyLDAP(ctx, userReq); err != nil {
c.Fatalf("unable to attach LDAP policy for success rate-limit test: %v", err)
}
withGlobalSTSLDAPLoginRateLimiterForTest(
newSTSLDAPLoginRateLimiter(time.Hour, 2, stsLDAPLoginEntryTTL),
func() {
for attempt := 1; attempt <= 3; attempt++ {
success := s.postLDAPSTS(c, "dillon", "dillon")
if success.StatusCode != http.StatusOK {
c.Fatalf("expected successful LDAP STS login on attempt %d, got status %d body: %s", attempt, success.StatusCode, success.Body)
}
}
firstFailure := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
secondFailure := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
throttled := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
if firstFailure.StatusCode != http.StatusBadRequest || secondFailure.StatusCode != http.StatusBadRequest {
c.Fatalf("expected failed auth attempts after successful logins to return %d, got %d and %d", http.StatusBadRequest, firstFailure.StatusCode, secondFailure.StatusCode)
}
if throttled.StatusCode != http.StatusTooManyRequests {
c.Fatalf("expected third failed auth attempt after successful logins to be throttled with %d, got %d", http.StatusTooManyRequests, throttled.StatusCode)
}
},
)
}
func (s *TestSuiteIAM) TestLDAPSTSUpstreamFailure(c *check) {
original := globalIAMSys.LDAPConfig.Clone()
globalIAMSys.LDAPConfig.LDAP.ServerAddr = "127.0.0.1:1"
defer func() {
globalIAMSys.LDAPConfig = original
}()
withGlobalSTSLDAPLoginRateLimiterForTest(
newSTSLDAPLoginRateLimiter(time.Hour, 2, stsLDAPLoginEntryTTL),
func() {
for range 3 {
upstreamFailure := s.postLDAPSTSForError(c, "dillon", "dillon")
if upstreamFailure.StatusCode != http.StatusInternalServerError {
c.Fatalf("expected upstream failure status %d, got %d", http.StatusInternalServerError, upstreamFailure.StatusCode)
}
if upstreamFailure.Code != "InternalError" {
c.Fatalf("expected upstream failure code %q, got %q", "InternalError", upstreamFailure.Code)
}
if upstreamFailure.Message != stsErrCodes.ToSTSErr(ErrSTSUpstreamError).Description {
c.Fatalf("expected upstream failure message %q, got %q", stsErrCodes.ToSTSErr(ErrSTSUpstreamError).Description, upstreamFailure.Message)
}
if upstreamFailure.Message == errLDAPAuthenticationFailed.Error() {
c.Fatalf("expected upstream failure to stay distinct from auth failure, got %q", upstreamFailure.Message)
}
}
globalIAMSys.LDAPConfig = original
authFailure := s.postLDAPSTSForError(c, "dillon", "nottherightpassword")
if authFailure.StatusCode == http.StatusTooManyRequests {
c.Fatalf("expected upstream failures not to consume rate limit budget, got throttled response: %+v", authFailure)
}
if authFailure.StatusCode != http.StatusBadRequest {
c.Fatalf("expected auth failure after upstream recovery to return %d, got %d", http.StatusBadRequest, authFailure.StatusCode)
}
},
)
}
func (s *TestSuiteIAM) TestLDAPSTSTrustedProxyRateLimit(c *check) {
withLDAPSTSTrustedProxiesForTest(c.T, "127.0.0.0/8,::1/128", func() {
withGlobalSTSLDAPLoginRateLimiterForTest(
newSTSLDAPLoginRateLimiter(time.Hour, 1, stsLDAPLoginEntryTTL),
func() {
// These usernames intentionally do not exist. The test asserts that
// LDAP user-not-found stays classified as an auth error, so failed
// attempts still commit the reservation and hit the source bucket.
first := s.postLDAPSTSWithHeaders(c, "missing-user-a", "nottherightpassword", singleHeader("X-Real-IP", "203.0.113.10"))
second := s.postLDAPSTSWithHeaders(c, "missing-user-b", "nottherightpassword", singleHeader("X-Real-IP", "198.51.100.23"))
third := s.postLDAPSTSWithHeaders(c, "missing-user-c", "nottherightpassword", singleHeader("X-Real-IP", "203.0.113.10"))
if first.StatusCode != http.StatusBadRequest {
c.Fatalf("expected first trusted-proxy LDAP STS auth failure to return %d, got %d body: %s", http.StatusBadRequest, first.StatusCode, first.Body)
}
if second.StatusCode != http.StatusBadRequest {
c.Fatalf("expected a different forwarded client IP behind the same trusted proxy to avoid source throttling, got %d body: %s", second.StatusCode, second.Body)
}
if third.StatusCode != http.StatusTooManyRequests {
c.Fatalf("expected the same forwarded client IP behind the trusted proxy to be throttled with %d, got %d body: %s", http.StatusTooManyRequests, third.StatusCode, third.Body)
}
},
)
})
}
func TestIAMWithLDAPSecurityServerSuite(t *testing.T) {
tests := []struct {
name string
run func(*TestSuiteIAM, *check, string)
}{
{
name: "AuthFailureUniformResponse",
run: func(suite *TestSuiteIAM, c *check, ldapServer string) {
suite.TestLDAPSTSAuthFailureUniformResponse(c)
},
},
{
name: "RateLimit",
run: func(suite *TestSuiteIAM, c *check, ldapServer string) {
suite.TestLDAPSTSRateLimit(c)
},
},
{
name: "SuccessDoesNotConsumeRateLimit",
run: func(suite *TestSuiteIAM, c *check, ldapServer string) {
suite.TestLDAPSTSSuccessDoesNotConsumeRateLimit(c)
},
},
{
name: "UpstreamFailure",
run: func(suite *TestSuiteIAM, c *check, ldapServer string) {
suite.TestLDAPSTSUpstreamFailure(c)
},
},
{
name: "TrustedProxyRateLimit",
run: func(suite *TestSuiteIAM, c *check, ldapServer string) {
suite.TestLDAPSTSTrustedProxyRateLimit(c)
},
},
}
for i, testCase := range iamTestSuites {
t.Run(
fmt.Sprintf("Test: %d, ServerType: %s", i+1, testCase.ServerTypeDescription),
func(t *testing.T) {
ldapServer := os.Getenv(EnvTestLDAPServer)
if ldapServer == "" {
t.Skipf("Skipping LDAP security test as no LDAP server is provided via %s", EnvTestLDAPServer)
}
suite := testCase
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
c := &check{t, testCase.serverType}
suite.SetUpSuite(c)
suite.SetUpLDAP(c, ldapServer)
tc.run(suite, c, ldapServer)
suite.TearDownSuite(c)
})
}
},
)
}
}
// This test is for a fix added to handle non-normalized base DN values in the
// LDAP configuration. It runs the existing LDAP sub-tests with a non-normalized
// LDAP configuration.
@@ -1052,6 +1398,578 @@ func TestIAMExportImportWithLDAP(t *testing.T) {
}
}
type matchingAuthError struct{}
func (matchingAuthError) Error() string {
return "ldap auth failed"
}
func (matchingAuthError) Is(target error) bool {
return targetIsLDAPAuthFailure(target)
}
func targetIsLDAPAuthFailure(target error) bool {
return target != nil && target.Error() == "ldap authentication failed"
}
func TestSTSLDAPLoginRateLimiter(t *testing.T) {
limiter := newSTSLDAPLoginRateLimiter(time.Hour, 2, time.Minute)
if !limiter.Allow("192.0.2.10") {
t.Fatal("expected first attempt to be allowed")
}
if !limiter.Allow("192.0.2.10") {
t.Fatal("expected second attempt within burst to be allowed")
}
if limiter.Allow("192.0.2.10") {
t.Fatal("expected source IP bucket to be throttled after burst")
}
// A different source IP has its own independent bucket, so one client
// cannot exhaust another's budget (no per-username lockout dimension).
if !limiter.Allow("192.0.2.11") {
t.Fatal("expected a different source IP to be allowed")
}
// An empty source IP cannot be identified and must never be throttled,
// otherwise all such requests would collapse into one shared bucket.
if !limiter.Allow("") || !limiter.Allow("") {
t.Fatal("expected unidentified source to stay unthrottled")
}
}
func TestSTSLDAPLoginRateLimiterReserveCancel(t *testing.T) {
limiter := newSTSLDAPLoginRateLimiter(time.Hour, 1, time.Minute)
reservation := limiter.Reserve("192.0.2.10")
if reservation == nil {
t.Fatal("expected first reservation to succeed")
}
if limiter.Reserve("192.0.2.10") != nil {
t.Fatal("expected second reservation on the same source IP to be throttled before cancel")
}
reservation.Cancel()
reservation = limiter.Reserve("192.0.2.10")
if reservation == nil {
t.Fatal("expected canceled reservation to restore source-IP capacity")
}
reservation.Cancel()
reservation = limiter.Reserve("192.0.2.11")
if reservation == nil {
t.Fatal("expected a different source IP to have independent capacity")
}
reservation.Cancel()
}
func TestSTSLDAPLoginKeyLimiterCancelDoesNotOverCreditAfterRefill(t *testing.T) {
set := newSTSLDAPLoginKeyLimiterSet(10*time.Millisecond, 2, time.Minute)
start := time.Unix(0, 0)
first := set.Reserve(start, "192.0.2.10")
if first == nil {
t.Fatal("expected first reservation to succeed")
}
second := set.Reserve(start.Add(5*time.Millisecond), "192.0.2.10")
if second == nil {
t.Fatal("expected second reservation to succeed while one token remains available")
}
first.CancelAt(start.Add(10 * time.Millisecond))
third := set.Reserve(start.Add(10*time.Millisecond), "192.0.2.10")
if third == nil {
t.Fatal("expected canceled reservation to restore exactly one slot")
}
defer third.CancelAt(start.Add(10 * time.Millisecond))
if extra := set.Reserve(start.Add(10*time.Millisecond), "192.0.2.10"); extra != nil {
extra.CancelAt(start.Add(10 * time.Millisecond))
t.Fatal("expected only one slot to be restored after cancel; got over-credit from refill")
}
second.CancelAt(start.Add(10 * time.Millisecond))
}
func TestSTSLDAPLoginRateLimiterConcurrentReserveLifecycle(t *testing.T) {
limiter := newSTSLDAPLoginRateLimiter(time.Hour, 4, time.Minute)
const workers = 8
start := make(chan struct{})
finish := make(chan struct{})
var wg sync.WaitGroup
var reserveWG sync.WaitGroup
reservations := make([]*stsLDAPLoginReservation, workers)
var canceledReservations atomic.Int32
reserveWG.Add(workers)
for i := 0; i < workers; i++ {
wg.Add(1)
go func(worker int) {
defer wg.Done()
<-start
reservations[worker] = limiter.Reserve("192.0.2.10")
reserveWG.Done()
if reservations[worker] == nil {
return
}
<-finish
if worker%2 == 0 {
canceledReservations.Add(1)
reservations[worker].Cancel()
return
}
reservations[worker].Commit()
}(i)
}
close(start)
reserveWG.Wait()
successfulReservations := 0
for _, reservation := range reservations {
if reservation != nil {
successfulReservations++
}
}
if got := successfulReservations; got != 4 {
t.Fatalf("expected exactly 4 successful concurrent reservations, got %d", got)
}
close(finish)
wg.Wait()
remainingBudget := 0
for {
reservation := limiter.Reserve("192.0.2.10")
if reservation == nil {
break
}
remainingBudget++
reservation.Commit()
}
if want, got := int(canceledReservations.Load()), remainingBudget; got != want {
t.Fatalf("expected %d tokens to remain after concurrent commit/cancel mix, got %d", want, got)
}
}
func TestGetSTSLDAPLoginSourceIP(t *testing.T) {
tests := []struct {
name string
remoteAddr string
want string
}{
{name: "empty", remoteAddr: "", want: ""},
{name: "ipv4", remoteAddr: "192.0.2.10:9000", want: "192.0.2.10"},
{name: "ipv6", remoteAddr: "[2001:db8::10]:9000", want: "2001:db8::10"},
{name: "bare-host", remoteAddr: "192.0.2.10", want: "192.0.2.10"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := &http.Request{RemoteAddr: tt.remoteAddr}
if got := getSTSLDAPLoginSourceIP(req); got != tt.want {
t.Fatalf("expected %q, got %q", tt.want, got)
}
})
}
}
func TestGetSTSLDAPLoginSourceIPIgnoresSpoofedForwardingHeaders(t *testing.T) {
tests := []struct {
name string
headerKey string
headerValue string
}{
{
name: "x-forwarded-for",
headerKey: "X-Forwarded-For",
headerValue: "203.0.113.10, 198.51.100.24",
},
{
name: "x-real-ip",
headerKey: "X-Real-IP",
headerValue: "203.0.113.10",
},
{
name: "forwarded",
headerKey: "Forwarded",
headerValue: `for=203.0.113.10;proto=https`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := &http.Request{
Header: singleHeader(tt.headerKey, tt.headerValue),
RemoteAddr: "192.0.2.10:9000",
}
if got := getSTSLDAPLoginSourceIP(req); got != "192.0.2.10" {
t.Fatalf("expected helper to ignore spoofed %s and return peer address, got %q", tt.headerKey, got)
}
})
}
}
func TestGetSTSLDAPLoginSourceIPUsesForwardedHeadersForTrustedProxy(t *testing.T) {
tests := []struct {
name string
headerKey string
headerValue string
want string
}{
{
name: "x-forwarded-for",
headerKey: "X-Forwarded-For",
headerValue: "203.0.113.10",
want: "203.0.113.10",
},
{
name: "x-real-ip",
headerKey: "X-Real-IP",
headerValue: "203.0.113.10",
want: "203.0.113.10",
},
}
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := &http.Request{
Header: singleHeader(tt.headerKey, tt.headerValue),
RemoteAddr: "192.0.2.10:9000",
}
if got := getSTSLDAPLoginSourceIP(req); got != tt.want {
t.Fatalf("expected trusted proxy header %s to resolve %q, got %q", tt.headerKey, tt.want, got)
}
})
}
})
}
// A client behind a trusted, appending proxy can prepend a spoofed left-most
// X-Forwarded-For value. The right-to-left walk must skip only trusted hops and
// return the real (right-most untrusted) client, ignoring the spoofed value.
func TestGetSTSLDAPLoginSourceIPTrustedProxyStripsSpoofedForwardedFor(t *testing.T) {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req := &http.Request{
Header: singleHeader("X-Forwarded-For", "1.2.3.4, 198.51.100.50"),
RemoteAddr: "192.0.2.10:9000",
}
if got := getSTSLDAPLoginSourceIP(req); got != "198.51.100.50" {
t.Fatalf("expected spoofed left-most XFF entry to be ignored and real client returned, got %q", got)
}
})
}
// When several hops in the chain are trusted proxies, the walk skips all of
// them and resolves the left-most (real client) address.
func TestGetSTSLDAPLoginSourceIPTrustedProxyWalksMultipleTrustedHops(t *testing.T) {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req := &http.Request{
Header: singleHeader("X-Forwarded-For", "203.0.113.10, 192.0.2.20, 192.0.2.21"),
RemoteAddr: "192.0.2.10:9000",
}
if got := getSTSLDAPLoginSourceIP(req); got != "203.0.113.10" {
t.Fatalf("expected walk to skip trusted hops and return real client, got %q", got)
}
})
}
// The RFC 7239 Forwarded header is not honored for trusted-proxy bucketing; such
// requests fall back to the safe peer-address bucket.
func TestGetSTSLDAPLoginSourceIPTrustedProxyIgnoresForwardedHeader(t *testing.T) {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req := &http.Request{
Header: singleHeader("Forwarded", `for=203.0.113.10;proto=https`),
RemoteAddr: "192.0.2.10:9000",
}
if got := getSTSLDAPLoginSourceIP(req); got != "192.0.2.10" {
t.Fatalf("expected RFC 7239 Forwarded to be ignored and peer address used, got %q", got)
}
})
}
func TestGetSTSLDAPLoginSourceIPTrustedProxyPrefersXRealIPOverXForwardedFor(t *testing.T) {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req := &http.Request{
Header: make(http.Header),
RemoteAddr: "192.0.2.10:9000",
}
req.Header.Set("X-Forwarded-For", "198.51.100.99, 203.0.113.10")
req.Header.Set("X-Real-IP", "203.0.113.10")
if got := getSTSLDAPLoginSourceIP(req); got != "203.0.113.10" {
t.Fatalf("expected trusted proxy path to prefer X-Real-IP over appended X-Forwarded-For, got %q", got)
}
})
}
// X-Real-IP is trusted verbatim (it cannot be chain-validated like X-Forwarded-For),
// so a client-supplied X-Real-IP that the proxy fails to overwrite wins even over a
// correctly appended X-Forwarded-For chain. This locks the documented deployment
// contract: the trusted proxy MUST overwrite X-Real-IP, otherwise it is a spoofing
// vector. If this assertion ever changes, the change must be deliberate.
func TestGetSTSLDAPLoginSourceIPTrustedProxyTrustsXRealIPVerbatim(t *testing.T) {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req := &http.Request{
Header: make(http.Header),
RemoteAddr: "192.0.2.10:9000",
}
// Attacker spoofs X-Real-IP with a value that appears nowhere in the XFF
// chain; the trusted proxy still appends the real client (198.51.100.50).
// The spoofed X-Real-IP wins, so the result can only have come from it.
req.Header.Set("X-Real-IP", "10.10.10.10")
req.Header.Set("X-Forwarded-For", "1.1.1.1, 198.51.100.50")
if got := getSTSLDAPLoginSourceIP(req); got != "10.10.10.10" {
t.Fatalf("expected verbatim X-Real-IP trust (spoofable contract), got %q", got)
}
})
}
func TestGetSTSLDAPLoginSourceIPTrustedProxyFallsBackToPeerWithoutForwardingHeaders(t *testing.T) {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req := &http.Request{RemoteAddr: "192.0.2.10:9000"}
if got := getSTSLDAPLoginSourceIP(req); got != "192.0.2.10" {
t.Fatalf("expected trusted proxy path without forwarding headers to fall back to peer address, got %q", got)
}
})
}
func TestReserveSTSLDAPLoginUsesPeerAddressBuckets(t *testing.T) {
tests := []struct {
name string
headerKey string
firstValue string
secondValue string
}{
{
name: "x-forwarded-for",
headerKey: "X-Forwarded-For",
firstValue: "203.0.113.10",
secondValue: "198.51.100.23, 198.51.100.24",
},
{
name: "x-real-ip",
headerKey: "X-Real-IP",
firstValue: "203.0.113.10",
secondValue: "198.51.100.23",
},
{
name: "forwarded",
headerKey: "Forwarded",
firstValue: `for=203.0.113.10;proto=https`,
secondValue: `for=198.51.100.23;proto=https`,
},
}
for _, tt := range tests {
t.Run(tt.name+"-same-peer", func(t *testing.T) {
limiter := newSTSLDAPLoginRateLimiter(time.Hour, 2, time.Minute)
withGlobalSTSLDAPLoginRateLimiterForTest(limiter, func() {
req1 := &http.Request{
Header: singleHeader(tt.headerKey, tt.firstValue),
RemoteAddr: "192.0.2.10:9000",
Form: url.Values{stsLDAPUsername: []string{"alice"}},
}
req2 := &http.Request{
Header: singleHeader(tt.headerKey, tt.secondValue),
RemoteAddr: "192.0.2.10:9001",
Form: url.Values{stsLDAPUsername: []string{"bob"}},
}
reservation1 := reserveSTSLDAPLogin(req1)
if reservation1 == nil {
t.Fatal("expected first reservation to succeed")
}
defer reservation1.Cancel()
reservation2 := reserveSTSLDAPLogin(req2)
if reservation2 == nil {
t.Fatal("expected second reservation to succeed with shared source bucket capacity")
}
defer reservation2.Cancel()
if got := len(limiter.source.entries); got != 1 {
t.Fatalf("expected requests from the same peer address to share one source bucket, got %d", got)
}
if _, ok := limiter.source.entries["192.0.2.10"]; !ok {
t.Fatalf("expected source bucket for peer address %q, got keys %v", "192.0.2.10", limiter.source.entries)
}
})
})
t.Run(tt.name+"-different-peers", func(t *testing.T) {
limiter := newSTSLDAPLoginRateLimiter(time.Hour, 1, time.Minute)
withGlobalSTSLDAPLoginRateLimiterForTest(limiter, func() {
req1 := &http.Request{
Header: singleHeader(tt.headerKey, tt.firstValue),
RemoteAddr: "192.0.2.10:9000",
Form: url.Values{stsLDAPUsername: []string{"alice"}},
}
req2 := &http.Request{
Header: singleHeader(tt.headerKey, tt.firstValue),
RemoteAddr: "192.0.2.11:9000",
Form: url.Values{stsLDAPUsername: []string{"bob"}},
}
reservation1 := reserveSTSLDAPLogin(req1)
if reservation1 == nil {
t.Fatal("expected first reservation to succeed")
}
defer reservation1.Cancel()
reservation2 := reserveSTSLDAPLogin(req2)
if reservation2 == nil {
t.Fatal("expected second reservation from a different peer address to succeed")
}
defer reservation2.Cancel()
if got := len(limiter.source.entries); got != 2 {
t.Fatalf("expected requests from different peer addresses to use different source buckets, got %d", got)
}
if _, ok := limiter.source.entries["192.0.2.10"]; !ok {
t.Fatalf("expected source bucket for peer address %q, got keys %v", "192.0.2.10", limiter.source.entries)
}
if _, ok := limiter.source.entries["192.0.2.11"]; !ok {
t.Fatalf("expected source bucket for peer address %q, got keys %v", "192.0.2.11", limiter.source.entries)
}
})
})
}
}
func TestReserveSTSLDAPLoginUsesForwardedBucketsForTrustedProxy(t *testing.T) {
limiter := newSTSLDAPLoginRateLimiter(time.Hour, 1, time.Minute)
withGlobalSTSLDAPLoginRateLimiterForTest(limiter, func() {
withLDAPSTSTrustedProxiesForTest(t, "192.0.2.0/24", func() {
req1 := &http.Request{
Header: singleHeader("X-Forwarded-For", "203.0.113.10"),
RemoteAddr: "192.0.2.10:9000",
Form: url.Values{stsLDAPUsername: []string{"alice"}},
}
req2 := &http.Request{
Header: singleHeader("X-Forwarded-For", "198.51.100.23"),
RemoteAddr: "192.0.2.10:9001",
Form: url.Values{stsLDAPUsername: []string{"bob"}},
}
reservation1 := reserveSTSLDAPLogin(req1)
if reservation1 == nil {
t.Fatal("expected first reservation through trusted proxy to succeed")
}
defer reservation1.Cancel()
reservation2 := reserveSTSLDAPLogin(req2)
if reservation2 == nil {
t.Fatal("expected forwarded client IPs behind the same trusted proxy to use distinct source buckets")
}
defer reservation2.Cancel()
if got := len(limiter.source.entries); got != 2 {
t.Fatalf("expected distinct forwarded client IPs to use two source buckets, got %d", got)
}
if _, ok := limiter.source.entries["203.0.113.10"]; !ok {
t.Fatalf("expected source bucket for forwarded client IP %q, got keys %v", "203.0.113.10", limiter.source.entries)
}
if _, ok := limiter.source.entries["198.51.100.23"]; !ok {
t.Fatalf("expected source bucket for forwarded client IP %q, got keys %v", "198.51.100.23", limiter.source.entries)
}
})
})
}
func TestLDAPBindErrorToSTS(t *testing.T) {
tests := []struct {
name string
err error
code STSErrorCode
message string
}{
{
name: "auth failure",
err: matchingAuthError{},
code: ErrSTSInvalidParameterValue,
message: errLDAPAuthenticationFailed.Error(),
},
{
name: "upstream failure",
err: errors.New("ldap server unavailable"),
code: ErrSTSUpstreamError,
message: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
code, err := ldapBindErrorToSTS(tt.err)
if code != tt.code {
t.Fatalf("expected code %v, got %v", tt.code, code)
}
if tt.message == "" {
if err != nil {
t.Fatalf("expected nil response error, got %v", err)
}
return
}
if err == nil || err.Error() != tt.message {
t.Fatalf("expected %q, got %v", tt.message, err)
}
})
}
}
func TestSTSLDAPLoginRateLimiterCleanup(t *testing.T) {
set := newSTSLDAPLoginKeyLimiterSet(time.Hour, 1, time.Minute)
start := time.Unix(0, 0)
if !set.Allow(start, "old-key") {
t.Fatal("expected initial key to be allowed")
}
if len(set.entries) != 1 {
t.Fatalf("expected one entry, got %d", len(set.entries))
}
if !set.Allow(start.Add(2*time.Minute), "new-key") {
t.Fatal("expected new key to be allowed after ttl expiry")
}
if _, ok := set.entries["old-key"]; ok {
t.Fatal("expected expired key to be cleaned up")
}
}
func TestWriteSTSThrottledResponse(t *testing.T) {
req := httptest.NewRequest(http.MethodPost, "http://minio.test", strings.NewReader(""))
rr := httptest.NewRecorder()
req = req.WithContext(newContext(req, rr, "test-throttle"))
writeSTSThrottledResponse(rr)
if rr.Code != http.StatusTooManyRequests {
t.Fatalf("expected status %d, got %d", http.StatusTooManyRequests, rr.Code)
}
if got := rr.Header().Get("Retry-After"); got != "6" {
t.Fatalf("expected Retry-After header %q, got %q", "6", got)
}
body := rr.Body.String()
if !strings.Contains(body, "<Code>ThrottlingException</Code>") {
t.Fatalf("expected throttling code in response, got %s", body)
}
if !strings.Contains(body, "<Message>Request throttled, please retry later.</Message>") {
t.Fatalf("expected throttling message in response, got %s", body)
}
}
func TestIAMImportAssetWithLDAP(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), testDefaultTimeout)
defer cancel()
+2 -2
View File
@@ -450,10 +450,10 @@ func getLatestReleaseTime(u *url.URL, timeout time.Duration, mode string) (sha25
const (
// Kubernetes deployment doc link.
kubernetesDeploymentDoc = "https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html"
kubernetesDeploymentDoc = "https://silo.pigsty.io/operations/deployments/kubernetes.html"
// Mesos deployment doc link.
mesosDeploymentDoc = "https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html"
mesosDeploymentDoc = "https://silo.pigsty.io/operations/deployments/kubernetes.html"
)
func getDownloadURL(releaseTag string) (downloadURL string) {
-16
View File
@@ -68,7 +68,6 @@ const (
storageMetricReadXL
storageMetricReadAll
storageMetricStatInfoFile
storageMetricReadMultiple
storageMetricDeleteAbandonedParts
storageMetricDiskInfo
storageMetricDeleteBulk
@@ -723,21 +722,6 @@ func (p *xlStorageDiskIDCheck) ReadParts(ctx context.Context, volume string, par
return p.storage.ReadParts(ctx, volume, partMetaPaths...)
}
// ReadMultiple will read multiple files and send each files as response.
// Files are read and returned in the given order.
// The resp channel is closed before the call returns.
// Only a canceled context will return an error.
func (p *xlStorageDiskIDCheck) ReadMultiple(ctx context.Context, req ReadMultipleReq, resp chan<- ReadMultipleResp) (err error) {
ctx, done, err := p.TrackDiskHealth(ctx, storageMetricReadMultiple, req.Bucket, req.Prefix)
if err != nil {
xioutil.SafeClose(resp)
return err
}
defer done(0, &err)
return p.storage.ReadMultiple(ctx, req, resp)
}
// CleanAbandonedData will read metadata of the object on disk
// and delete any data directories and inline data that isn't referenced in metadata.
func (p *xlStorageDiskIDCheck) CleanAbandonedData(ctx context.Context, volume string, path string) (err error) {
-75
View File
@@ -3187,81 +3187,6 @@ func (s *xlStorage) ReadParts(ctx context.Context, volume string, partMetaPaths
return parts, nil
}
// ReadMultiple will read multiple files and send each back as response.
// Files are read and returned in the given order.
// The resp channel is closed before the call returns.
// Only a canceled context will return an error.
func (s *xlStorage) ReadMultiple(ctx context.Context, req ReadMultipleReq, resp chan<- ReadMultipleResp) error {
defer xioutil.SafeClose(resp)
volumeDir := pathJoin(s.drivePath, req.Bucket)
found := 0
for _, f := range req.Files {
if contextCanceled(ctx) {
return ctx.Err()
}
r := ReadMultipleResp{
Bucket: req.Bucket,
Prefix: req.Prefix,
File: f,
}
var data []byte
var mt time.Time
fullPath := pathJoin(volumeDir, req.Prefix, f)
w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
if err := w.Run(func() (err error) {
if req.MetadataOnly {
data, mt, err = s.readMetadataWithDMTime(ctx, fullPath)
} else {
data, mt, err = s.readAllDataWithDMTime(ctx, req.Bucket, volumeDir, fullPath)
}
return err
}); err != nil {
if !IsErr(err, errFileNotFound, errVolumeNotFound) {
r.Exists = true
r.Error = err.Error()
}
select {
case <-ctx.Done():
return ctx.Err()
case resp <- r:
}
if req.AbortOn404 && !r.Exists {
// We stop at first file not found.
// We have already reported the error, return nil.
return nil
}
continue
}
diskHealthCheckOK(ctx, nil)
if req.MaxSize > 0 && int64(len(data)) > req.MaxSize {
r.Exists = true
r.Error = fmt.Sprintf("max size (%d) exceeded: %d", req.MaxSize, len(data))
select {
case <-ctx.Done():
return ctx.Err()
case resp <- r:
continue
}
}
found++
r.Exists = true
r.Data = data
r.Modtime = mt
select {
case <-ctx.Done():
return ctx.Err()
case resp <- r:
}
if req.MaxResults > 0 && found >= req.MaxResults {
return nil
}
}
return nil
}
func (s *xlStorage) StatInfoFile(ctx context.Context, volume, path string, glob bool) (stat []StatInfo, err error) {
volumeDir, err := s.getVolDir(volume)
if err != nil {
+59 -27
View File
@@ -1,37 +1,69 @@
#!/bin/bash
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
set -ex
remote=$(git remote get-url upstream)
if test "$remote" != "git@github.com:minio/minio.git"; then
echo "Script requires that the 'upstream' remote is set to git@github.com:minio/minio.git"
exit 1
fi
function _init() {
## All binaries are static make sure to disable CGO.
export CGO_ENABLED=0
export CRED_DIR="/media/${USER}/minio"
git remote update upstream && git checkout master && git rebase upstream/master
## List of architectures and OS to test coss compilation.
SUPPORTED_OSARCH="linux/ppc64le linux/amd64 linux/arm64"
release=$(git describe --abbrev=0 --tags)
remote=$(git remote get-url upstream)
if test "$remote" != "git@github.com:minio/minio.git"; then
echo "Script requires that the 'upstream' remote is set to git@github.com:minio/minio.git"
exit 1
fi
docker buildx build --push --no-cache \
--build-arg RELEASE="${release}" \
-t "minio/minio:latest" \
-t "minio/minio:latest-cicd" \
-t "quay.io/minio/minio:latest" \
-t "quay.io/minio/minio:latest-cicd" \
-t "minio/minio:${release}" \
-t "quay.io/minio/minio:${release}" \
--platform=linux/arm64,linux/amd64,linux/ppc64le \
-f Dockerfile.release .
git remote update upstream && git checkout master && git rebase upstream/master
docker buildx prune -f
release=$(git describe --abbrev=0 --tags)
export release
}
docker buildx build --push --no-cache \
--build-arg RELEASE="${release}" \
-t "minio/minio:${release}-cpuv1" \
-t "quay.io/minio/minio:${release}-cpuv1" \
--platform=linux/arm64,linux/amd64,linux/ppc64le \
-f Dockerfile.release.old_cpu .
function _build() {
local osarch=$1
IFS=/ read -r -a arr <<<"$osarch"
os="${arr[0]}"
arch="${arr[1]}"
package=$(go list -f '{{.ImportPath}}')
printf -- "--> %15s:%s\n" "${osarch}" "${package}"
docker buildx prune -f
# go build -trimpath to build the binary.
export GOOS=$os
export GOARCH=$arch
export MINIO_RELEASE=RELEASE
LDFLAGS=$(go run buildscripts/gen-ldflags.go)
go build -tags kqueue -trimpath --ldflags "${LDFLAGS}" -o ./minio-${arch}.${release}
minisign -qQSm ./minio-${arch}.${release} -s "$CRED_DIR/minisign.key" <"$CRED_DIR/minisign-passphrase"
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sha256sum_str=$(sha256sum <./minio-${arch}.${release})
rc=$?
if [ "$rc" -ne 0 ]; then
abort "unable to generate sha256sum for ${1}"
fi
echo "${sha256sum_str// -/minio.${release}}" >./minio-${arch}.${release}.sha256sum
}
function main() {
echo "Testing builds for OS/Arch: ${SUPPORTED_OSARCH}"
for each_osarch in ${SUPPORTED_OSARCH}; do
_build "${each_osarch}"
done
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
docker buildx build --push --no-cache \
--build-arg RELEASE="${release}" \
-t "registry.min.dev/community/minio:latest" \
-t "registry.min.dev/community/minio:${release}" \
--platform=linux/arm64,linux/amd64,linux/ppc64le \
-f Dockerfile .
docker buildx prune -f
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
}
_init && main "$@"
+5 -1
View File
@@ -1,8 +1,12 @@
#!/bin/bash
# Pin to v8.11.0 the last release that includes curl-aarch64.
# v8.17.0 (current latest) dropped the aarch64 build.
STATIC_CURL_VERSION="v8.11.0"
function download_arch_specific_executable {
curl -f -L -s -q \
https://github.com/moparisthebest/static-curl/releases/latest/download/curl-$1 \
"https://github.com/moparisthebest/static-curl/releases/download/${STATIC_CURL_VERSION}/curl-$1" \
-o /go/bin/curl || exit 1
chmod +x /go/bin/curl
}
+13 -13
View File
@@ -2,7 +2,7 @@
## **1. Cloud-native Architecture**
![cloud-native](https://github.com/minio/minio/blob/master/docs/bigdata/images/image1.png?raw=true "cloud native architecture")
![cloud-native](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image1.png?raw=true "cloud native architecture")
Kubernetes manages stateless Spark and Hive containers elastically on the compute nodes. Spark has native scheduler integration with Kubernetes. Hive, for legacy reasons, uses YARN scheduler on top of Kubernetes.
@@ -16,24 +16,24 @@ MinIO also supports multi-cluster, multi-site federation similar to AWS regions
- [Setup Ambari](https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/set_up_the_ambari_server.html) which automatically sets up YARN
- [Installing Spark](https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/installing-spark/content/installing_spark.html)
- Install MinIO Distributed Server using one of the guides below.
- [Deployment based on Kubernetes](https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html)
- [Deployment based on Kubernetes](https://silo.pigsty.io/operations/deployments/kubernetes.html)
- [Deployment based on MinIO Helm Chart](https://github.com/helm/charts/tree/master/stable/minio)
## **3. Configure Hadoop, Spark, Hive to use MinIO**
After successful installation navigate to the Ambari UI `http://<ambari-server>:8080/` and login using the default credentials: [**_username: admin, password: admin_**]
![ambari-login](https://github.com/minio/minio/blob/master/docs/bigdata/images/image3.png?raw=true "ambari login")
![ambari-login](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image3.png?raw=true "ambari login")
### **3.1 Configure Hadoop**
Navigate to **Services** -> **HDFS** -> **CONFIGS** -> **ADVANCED** as shown below
![hdfs-configs](https://github.com/minio/minio/blob/master/docs/bigdata/images/image2.png?raw=true "hdfs advanced configs")
![hdfs-configs](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image2.png?raw=true "hdfs advanced configs")
Navigate to **Custom core-site** to configure MinIO parameters for `_s3a_` connector
![s3a-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image5.png?raw=true "custom core-site")
![s3a-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image5.png?raw=true "custom core-site")
```
sudo pip install yq
@@ -100,17 +100,17 @@ The rest of the other optimization options are discussed in the links below
Once the config changes are applied, proceed to restart **Hadoop** services.
![hdfs-services](https://github.com/minio/minio/blob/master/docs/bigdata/images/image7.png?raw=true "hdfs restart services")
![hdfs-services](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image7.png?raw=true "hdfs restart services")
### **3.2 Configure Spark2**
Navigate to **Services** -> **Spark2** -> **CONFIGS** as shown below
![spark-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image6.png?raw=true "spark config")
![spark-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image6.png?raw=true "spark config")
Navigate to “**Custom spark-defaults**” to configure MinIO parameters for `_s3a_` connector
![spark-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image9.png?raw=true "spark defaults")
![spark-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image9.png?raw=true "spark defaults")
Add the following optimal entries for _spark-defaults.conf_ to configure Spark with **MinIO**.
@@ -146,17 +146,17 @@ spark.hadoop.fs.s3a.threads.max 2048 # maximum number of threads for S3A
Once the config changes are applied, proceed to restart **Spark** services.
![spark-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image12.png?raw=true "spark restart services")
![spark-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image12.png?raw=true "spark restart services")
### **3.3 Configure Hive**
Navigate to **Services** -> **Hive** -> **CONFIGS**-> **ADVANCED** as shown below
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image10.png?raw=true "hive advanced config")
![hive-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image10.png?raw=true "hive advanced config")
Navigate to “**Custom hive-site**” to configure MinIO parameters for `_s3a_` connector
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image11.png?raw=true "hive advanced config")
![hive-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image11.png?raw=true "hive advanced config")
Add the following optimal entries for `hive-site.xml` to configure Hive with **MinIO**.
@@ -171,11 +171,11 @@ mapreduce.input.fileinputformat.list-status.num-threads=50
For more information about these options please visit [https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html](https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html)
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image13.png?raw=true "hive advanced custom config")
![hive-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image13.png?raw=true "hive advanced custom config")
Once the config changes are applied, proceed to restart all Hive services.
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image14.png?raw=true "restart hive services")
![hive-config](https://github.com/pgsty/minio/blob/master/docs/bigdata/images/image14.png?raw=true "restart hive services")
## **4. Run Sample Applications**
+2 -2
View File
@@ -1,6 +1,6 @@
# ILM Tiering Design [![slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/minio/minio/master/docs/bucket/lifecycle/README.md) allows tiering of content from MinIO object store to public clouds or other MinIO clusters.
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/pgsty/minio/blob/master/docs/bucket/lifecycle/README.md) allows tiering of content from MinIO object store to public clouds or other MinIO clusters.
Transition tiers can be added to MinIO using `mc admin tier add` command to associate a `gcs`, `s3` or `azure` bucket or prefix path on a bucket to the tier name.
Lifecycle transition rules can be applied to buckets (both versioned and un-versioned) by specifying the tier name defined above as the transition storage class for the lifecycle rule.
@@ -51,5 +51,5 @@ Tiering and lifecycle transition are applicable only to erasure/distributed MinI
## Explore Further
- [MinIO | Golang Client API Reference](https://docs.min.io/community/minio-object-store/developers/go/API.html#SetBucketLifecycle)
- [MinIO | Golang Client API Reference](https://silo.pigsty.io/developers/go/API.html#SetBucketLifecycle)
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
+4 -4
View File
@@ -4,8 +4,8 @@ Enable object lifecycle configuration on buckets to setup automatic deletion of
## 1. Prerequisites
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
- Install `mc` - [mc Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
- Install MinIO - [MinIO Quickstart Guide](https://silo.pigsty.io/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
- Install `mc` - [mc Quickstart Guide](https://silo.pigsty.io/reference/minio-mc.html#quickstart)
## 2. Enable bucket lifecycle configuration
@@ -59,7 +59,7 @@ TempUploads | temp/ | ✓ | ✓ | 7 day(s) | ✗
## 3. Activate ILM versioning features
This will only work with a versioned bucket, take a look at [Bucket Versioning Guide](https://docs.min.io/community/minio-object-store/administration/object-management/object-versioning.html) for more understanding.
This will only work with a versioned bucket, take a look at [Bucket Versioning Guide](https://silo.pigsty.io/administration/object-management/object-versioning.html) for more understanding.
### 3.1 Automatic removal of non current objects versions
@@ -228,5 +228,5 @@ Note that transition event notification is a MinIO extension.
## Explore Further
- [MinIO | Golang Client API Reference](https://docs.min.io/community/minio-object-store/developers/go/API.html)
- [MinIO | Golang Client API Reference](https://silo.pigsty.io/developers/go/API.html)
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
+3 -3
View File
@@ -30,7 +30,7 @@ Various event types supported by MinIO server are
| `s3:BucketCreated` |
| `s3:BucketRemoved` |
Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://docs.min.io/community/minio-object-store/reference/minio-mc/mc-event-add.html). MinIO SDK's [`BucketNotification` APIs](https://docs.min.io/community/minio-object-store/developers/go/API.html#setbucketnotification-ctx-context-context-bucketname-string-config-notification-configuration-error) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://silo.pigsty.io/reference/minio-mc/mc-event-add.html). MinIO SDK's [`BucketNotification` APIs](https://silo.pigsty.io/developers/go/API.html#setbucketnotification-ctx-context-context-bucketname-string-config-notification-configuration-error) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
Bucket events can be published to the following targets:
@@ -43,8 +43,8 @@ Bucket events can be published to the following targets:
## Prerequisites
- Install and configure MinIO Server from [here](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html#procedure).
- Install and configure MinIO Client from [here](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart).
- Install and configure MinIO Server from [here](https://silo.pigsty.io/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html#procedure).
- Install and configure MinIO Client from [here](https://silo.pigsty.io/reference/minio-mc.html#quickstart).
```
$ mc admin config get myminio | grep notify
+3 -3
View File
@@ -1,13 +1,13 @@
# Bucket Quota Configuration Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
![quota](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/quota/bucketquota.png)
![quota](https://raw.githubusercontent.com/pgsty/minio/master/docs/bucket/quota/bucketquota.png)
Buckets can be configured to have `Hard` quota - it disallows writes to the bucket after configured quota limit is reached.
## Prerequisites
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html#procedure).
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
- Install MinIO - [MinIO Quickstart Guide](https://silo.pigsty.io/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html#procedure).
- [Use `mc` with MinIO Server](https://silo.pigsty.io/reference/minio-mc.html#quickstart)
## Set bucket quota configuration
+4 -4
View File
@@ -1,6 +1,6 @@
# Bucket Replication Design [![slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/minio/minio/blob/master/docs/bucket/replication/README.md) first.
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/README.md) first.
## Overview
@@ -59,7 +59,7 @@ If 3 or more targets are participating in active-active replication, the replica
### Internal metadata for replication
`xl.meta` that is in use for [versioning](https://github.com/minio/minio/blob/master/docs/bucket/versioning/DESIGN.md) has additional metadata for replication of objects,delete markers and versioned deletes.
`xl.meta` that is in use for [versioning](https://github.com/pgsty/minio/blob/master/docs/bucket/versioning/DESIGN.md) has additional metadata for replication of objects,delete markers and versioned deletes.
### Metadata for object replication - on source
@@ -156,5 +156,5 @@ If 3 or more targets are participating in active-active replication, the replica
## Explore Further
- [MinIO Bucket Versioning Implementation](https://docs.min.io/community/minio-object-store/administration/object-management/object-versioning.html)
- [MinIO Client Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
- [MinIO Bucket Versioning Implementation](https://silo.pigsty.io/administration/object-management/object-versioning.html)
- [MinIO Client Quickstart Guide](https://silo.pigsty.io/reference/minio-mc.html#quickstart)
+11 -11
View File
@@ -2,9 +2,9 @@
Bucket replication is designed to replicate selected objects in a bucket to a destination bucket.
The contents of this page have been migrated to the new [MinIO Documentation: Bucket Replication](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) page. The [Bucket Replication](https://docs.min.io/community/minio-object-store/administration/bucket-replication/bucket-replication-requirements.html) page references dedicated tutorials for configuring one-way "Active-Passive" and two-way "Active-Active" bucket replication.
The contents of this page have been migrated to the new [MinIO Documentation: Bucket Replication](https://silo.pigsty.io/administration/bucket-replication.html) page. The [Bucket Replication](https://silo.pigsty.io/administration/bucket-replication/bucket-replication-requirements.html) page references dedicated tutorials for configuring one-way "Active-Passive" and two-way "Active-Active" bucket replication.
To replicate objects in a bucket to a destination bucket on a target site either in the same cluster or a different cluster, start by enabling [versioning](https://docs.min.io/community/minio-object-store/administration/object-management/object-versioning.html) for both source and destination buckets. Finally, the target site and the destination bucket need to be configured on the source MinIO server.
To replicate objects in a bucket to a destination bucket on a target site either in the same cluster or a different cluster, start by enabling [versioning](https://silo.pigsty.io/administration/object-management/object-versioning.html) for both source and destination buckets. Finally, the target site and the destination bucket need to be configured on the source MinIO server.
## Highlights
@@ -94,7 +94,7 @@ The access key provided for the replication *target* cluster should have these m
}
```
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/minio/minio/blob/master/docs/bucket/replication/setup_replication.sh)
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/setup_replication.sh)
To set up replication from a source bucket `srcbucket` on myminio cluster to a bucket `destbucket` on the target minio cluster with endpoint https://replica-endpoint:9000, use:
```
@@ -155,15 +155,15 @@ The replication configuration generated has the following format and can be expo
The replication configuration follows [AWS S3 Spec](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html). Any objects uploaded to the source bucket that meet replication criteria will now be automatically replicated by the MinIO server to the remote destination bucket. Replication can be disabled at any time by disabling specific rules in the configuration or deleting the replication configuration entirely.
When object locking is used in conjunction with replication, both source and destination buckets needs to have [object locking](https://docs.min.io/community/minio-object-store/administration/object-management/object-retention.html) enabled. Similarly objects encrypted on the server side, will be replicated if destination also supports encryption.
When object locking is used in conjunction with replication, both source and destination buckets needs to have [object locking](https://silo.pigsty.io/administration/object-management/object-retention.html) enabled. Similarly objects encrypted on the server side, will be replicated if destination also supports encryption.
Replication status can be seen in the metadata on the source and destination objects. On the source side, the `X-Amz-Replication-Status` changes from `PENDING` to `COMPLETED` or `FAILED` after replication attempt either succeeded or failed respectively. On the destination side, a `X-Amz-Replication-Status` status of `REPLICA` indicates that the object was replicated successfully. Any replication failures are automatically re-attempted during a periodic disk scanner cycle.
To perform bi-directional replication, repeat the above process on the target site - this time setting the source bucket as the replication target. It is recommended that replication be run in a system with at least two CPU's available to the process, so that replication can run in its own thread.
![put](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/replication/PUT_bucket_replication.png)
![put](https://raw.githubusercontent.com/pgsty/minio/master/docs/bucket/replication/PUT_bucket_replication.png)
![head](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/replication/HEAD_bucket_replication.png)
![head](https://raw.githubusercontent.com/pgsty/minio/master/docs/bucket/replication/HEAD_bucket_replication.png)
## Replica Modification sync
@@ -210,11 +210,11 @@ Also note that `mc` version `RELEASE.2021-09-02T09-21-27Z` or older supports onl
Status of delete marker replication can be viewed by doing a GET/HEAD on the object version - it will return a `X-Minio-Replication-DeleteMarker-Status` header and http response code of `405`. In the case of permanent deletes, if the delete replication is pending or failed to propagate to the target cluster, GET/HEAD will return additional `X-Minio-Replication-Delete-Status` header and a http response code of `405`.
![delete](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/replication/DELETE_bucket_replication.png)
![delete](https://raw.githubusercontent.com/pgsty/minio/master/docs/bucket/replication/DELETE_bucket_replication.png)
The status of replication can be monitored by configuring event notifications on the source and target buckets using `mc event add`.On the source side, the `s3:PutObject`, `s3:Replication:OperationCompletedReplication` and `s3:Replication:OperationFailedReplication` events show the status of replication in the `X-Amz-Replication-Status` metadata.
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see <https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md> for more details.
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see <https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/list.md> for more details.
### Sync/Async Replication
@@ -276,6 +276,6 @@ MinIO does not support SSE-C encrypted objects on replicated buckets, any applic
## Explore Further
- [MinIO Bucket Replication Design](https://github.com/minio/minio/blob/master/docs/bucket/replication/DESIGN.md)
- [MinIO Bucket Versioning Implementation](https://docs.min.io/community/minio-object-store/administration/object-management/object-retention.html)
- [MinIO Client Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
- [MinIO Bucket Replication Design](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/DESIGN.md)
- [MinIO Bucket Versioning Implementation](https://silo.pigsty.io/administration/object-management/object-retention.html)
- [MinIO Client Quickstart Guide](https://silo.pigsty.io/reference/minio-mc.html#quickstart)

Some files were not shown because too many files have changed in this diff Show More