Compare commits

...

33 Commits

Author SHA1 Message Date
Feng Ruohang 2616d43fbc fix(iam): enforce admin credential authorization boundaries
Honor self-service authorization denials, prevent session-limited credential delegation and updates, and protect root credential mutations under built-in IAM. Preserve empty-policy inheritance and damaged-account cleanup, normalize new service policy claims at issuance, and add signed Admin API and LDAP regressions.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-16 21:49:59 +08:00
Feng Ruohang 00a375c4e2 Merge pull request #238 from pgsty/codex/checksum-error-followup
Embed the Console versioning error fix at e5b165e4c0ee and add regression coverage for the algorithm-specific checksum messages merged in #229. Go CI, vulnerability checks, DCO and release-pipeline validation passed; the actual embedded Console and S3 checksum responses were also verified locally.

Fixes #231.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 21:47:26 +08:00
Feng Ruohang ab4c97e607 fix: embed Console versioning error handling and cover checksum messages
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 21:33:02 +08:00
Feng Ruohang 721dbfd1b7 Merge pull request #229 from yupoet/fix/putobject-checksum-error-message-226
Preserve the checksum rejection status and code while naming the checksum algorithm in PutObject and UploadPart error messages.

Validated on the branch updated to main: all six Go CI jobs and VulnCheck passed. Additional local coverage checked all five algorithms, wrapped errors, trailing checksums, multipart count mismatches and the generic fallback.

The maintainer explicitly accepted this contribution without the original author's DCO trailer as a one-time exception. Original authorship and the original commit are retained; no sign-off has been added on the author's behalf.

Fixes #226

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 21:22:45 +08:00
Feng Ruohang 2926474688 Merge branch 'main' into fix/putobject-checksum-error-message-226 2026-10-15 21:10:14 +08:00
Feng Ruohang a7370aa9ac Merge pull request #236 from clown15/fix-cleanup-stale-uploads-modtime
Parse native multipart directory timestamps before the Base64 compatibility path. Cover ambiguous plaintext IDs, cleanup metadata reads, expiration, and conservative legacy fallback.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:31:35 +08:00
Feng Ruohang a592d69f45 Merge pull request #232 from julienlau/julienlau/fix/commit-metadata-missing-503
Classify missing final-commit metadata as a retryable write failure, preserving storage RPC and quorum behavior. Cover real staged-metadata loss, API responses, successful quorums, and rollback boundaries.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:30:01 +08:00
Feng Ruohang bfc7906064 fix: prefer native multipart timestamps during cleanup
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:19:37 +08:00
Feng Ruohang eeff112491 test: verify commit metadata loss and write quorum boundaries
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:18:45 +08:00
Feng Ruohang a70156e45b Merge remote-tracking branch 'origin/main' into codex/pr236-maintainer-completion
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:12:09 +08:00
Feng Ruohang 99245e3ef8 Merge remote-tracking branch 'origin/main' into codex/pr232-maintainer-completion
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 18:12:09 +08:00
Feng Ruohang 7a3ca0f821 build: gate releases on Object Lock regressions (#237)
Run the existing Object Lock authorization and overwrite race regressions before release state changes, builds, packages, and uploads.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 13:20:15 +08:00
Feng Ruohang d94b3b3c92 Merge pull request #233 from julienlau/julienlau/fix/stale-metadata-leaks
Fix stale metadata in legacy xl.meta headers and scanner responses.

Keep scanner response resets local and cover reused legacy headers and both scanner response variants.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 10:01:39 +08:00
Feng Ruohang 86c5a92eef fix: keep scanner response resets local and strengthen regression tests
Keep the legacy header resets and clear the unused NSScanner response
field at each construction site. Remove the generic stream Recycler call:
Recycler transfers resource ownership and cannot be treated as a reset
hook followed by another insertion into the handler's pool.

Exercise the actual NSScanner handler across repeated scans with dirty
response allocations, so the regression test does not depend on sync.Pool
reusing an object. Check both response variants after wire serialization.
Cover legacy DecodeMsg and reused indexed metadata headers as well.

Retain the original contributor's commits and DCO sign-offs.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 09:48:16 +08:00
Feng Ruohang cdc3b8cb10 Merge remote-tracking branch 'origin/main' into codex/pr-233-maintainer-fix
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-15 09:42:46 +08:00
Feng Ruohang 0d3ba5d4cb Merge pull request #235 from pgsty/codex/retention-preflight-hardening
Retain the verified Object Lock write-lock defenses and the coordinated bilingual repair documentation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 23:42:33 +08:00
Feng Ruohang 71a1631f13 fix: reject object lock preflight read failures
Return storage read failures before accepting an object overwrite. Keep the null-version and multipart target checks under the destination write lock, and add slow-body, read-fault, legacy-upload and recovery-collision regressions to the Object Lock race gate.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 23:28:36 +08:00
clown15 2d57b975d1 cmd: fix fast-path modtime parsing in cleanupStaleUploadsOnDisk
Upload ID directories on disk are named as '<UUID>x<UnixNano>'.
Attempting to decode them as Base64 fails unconditionally, causing
the cleanup routine to bypass the timestamp check and fall back to
disk.ReadVersion on every active upload.

Directly parse the timestamp suffix from the plaintext directory
name to restore O(1) stale upload detection, while preserving Base64
decoding and ReadVersion fallback for legacy uploads.

Signed-off-by: clown15 <thsdhrhd15@naver.com>
2026-10-14 08:55:51 +00:00
Feng Ruohang 14042b5aab Merge pull request #234 from pgsty/codex/fix-retention-write-bypasses
fix: enforce retention authorization and immutable version writes

Include Go 1.27.2 security fixes and a compatible linter.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 16:02:58 +08:00
Feng Ruohang 38a5ab549c build: use golangci-lint compatible with Go 1.27.2
Pin golangci-lint v2.14.0, whose x/tools v0.50.0 decoder supports the
version 5 export data emitted by Go 1.27.2. The previous v2.13.1 binary
failed to import internal/cpu in Linux CI before running the checks.
Keep the existing linter selection and configuration.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 15:48:51 +08:00
Feng Ruohang fa565ec9fc build: update Go to 1.27.2 and x/net to 0.60.0
Use the October security fixes for reachable standard-library and HTTP/2
vulnerabilities found by govulncheck. Align the module requirement, build
dependency check and digest-pinned Docker build stages on Go 1.27.2.
Update only golang.org/x/net and its checksums; module tidying and credit
regeneration introduce no other dependency or license changes.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-10-14 15:39:25 +08:00
Feng Ruohang 3196480da9 fix: enforce retention authorization and immutable version writes
Close SN-2026-015 by requiring PutObjectRetention with request-derived
condition keys before reading bucket or object state. Weakening unexpired
GOVERNANCE retention also requires the bypass header and permission. Treat
retention removal as zero remaining days and avoid duration saturation for
distant dates.

Close SN-2026-016 by requiring replication trust for caller-selected target
version IDs and checking persisted multipart targets and locked null
versions under the destination write lock, including across storage pools.
Preserve versioning transitions, trusted replication and batch Snowball
version IDs, and fail closed when lock state cannot be read.

Add authorization, concurrent lock-update, multipart transition, cross-pool,
configuration-failure and retention-boundary regression coverage. Validate
with make test, focused race tests and the original independent probes.

Developed with Codex (Astra).

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit fe3935476b929e03193d7496a7448d760e635096)
2026-10-14 15:25:19 +08:00
Julien Laurenceau cb85461c42 test: cover renameData commit-metadata-missing conversion
Drive renameData() with stub disks to assert that an in-quorum
errCommitMetaMissing is converted to errErasureWriteQuorum (HTTP 503)
and that an in-quorum errFileNotFound still surfaces unchanged so
PutObject() keeps returning IncompleteBody for client disconnects.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:24:40 +02:00
Julien Laurenceau ba6e8c8d69 fix: reset Signature when unmarshaling legacy V1 xl.meta headers
The V1 header wire format has no Signature field, so decoding into a
reused header leaked the previous object's signature. Reset it along
with EcN/EcM so a decode fully replaces the header contents.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:12:49 +02:00
Julien Laurenceau a54d0c4354 fix: reset pooled NSScanner RPC responses to avoid stale usage data
NSScanner responses stream either a dataUsageEntry update or the
final dataUsageCache. The grid framework recycles stream responses
into a pool without resetting them, and the handler set only one
field per response - so a recycled response still carried the Final
(or Update) payload of an earlier scan. A remote node reading such a
response accepted the stale final cache, corrupting bucket usage
data across scanner cycles.

Honor Recycler types when pooling stream responses, implement it for
nsScannerResp, and clear the unset field at the response
construction sites for defense in depth.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:02:02 +02:00
Julien Laurenceau 00d4a206f7 fix: zero EcM when unmarshaling legacy xl.meta V2 headers
The legacy header decode paths zeroed EcN twice (z.EcN, z.EcN = 0, 0)
instead of EcN and EcM, so EcM retained its previous value when the
same structure was reused across decodes. Decoding an xl.meta written
by an older release would then leak the erasure metadata (data/parity
block counts) of a previously decoded object into it, potentially
corrupting quorum and healing decisions.

Also zero the fields on the V1 header path, which had no reset at
all.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 17:02:02 +02:00
Julien Laurenceau f480086b6b fix: return 503 instead of 404 when commit metadata is missing
When the staged xl.meta disappears between the data rename and the
meta-file commit inside RenameData(), the affected drives return
errFileNotFound. An in-quorum errFileNotFound is reduced by
reduceWriteQuorumErrs() and previously surfaced as:

- IncompleteBody (400) from PutObject(), misattributing the failure
  to a client disconnect, or
- ObjectNotFound (404 NoSuchKey) from CompleteMultipartUpload(),
  telling clients the object is absent so they skip retries.

Map errFileNotFound at the commit step to a dedicated internal
errCommitMetaMissing, convert it to errErasureWriteQuorum in
renameData(), and map it to InsufficientWriteQuorum (HTTP 503) in
toObjectErr() so clients retry. Below-quorum outcomes already return
InsufficientWriteQuorum and are unchanged, as is the IncompleteBody
mapping for staging failures of client-disconnected uploads.

Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
2026-10-12 16:11:06 +02:00
Paris Yu 968c30238c fix: name the checksum algorithm in PutObject/UploadPart mismatch errors
hash.ChecksumMismatch now carries the requested algorithm (CRC32, CRC32C,
SHA1, SHA256, CRC64NVME), and toAPIError renders it S3-style:

  The CRC32 you specified did not match the calculated checksum.

instead of the generic "The provided 'x-amz-checksum' header does not match
what was computed.", for all four mismatch construction sites (streaming
reader, trailing-checksum decode, and both Checksum.Matches paths). Mirrors
the wording already shipped for CompleteMultipartUpload in #225.

Fixes #226
2026-09-26 20:37:20 +08:00
Feng Ruohang bcb5fc9566 Merge pull request #225 from cbornet/error-messages
Align some checksum error messages with S3

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-26 09:03:50 +08:00
cbornet 394d9db212 Align some checksum error messages with S3
Signed-off-by: cbornet <cbornet@hotmail.com>
2026-09-25 17:44:52 +02:00
Feng Ruohang 69d5e92791 Merge pull request #220 from AleksaMCode/fix/lambda-targetidset-isempty
Fix `TargetIDSet.IsEmpty` and add regression test
2026-09-23 08:07:09 +08:00
AleksaMCode efd5698597 lambda/event: fix TargetIDSet.IsEmpty and add regression test
Signed-off-by: AleksaMCode <aleksamcode@gmail.com>
2026-09-17 22:06:44 +02:00
Feng Ruohang 2fde3cf535 docs: update ec description 2026-09-17 09:42:26 +08:00
52 changed files with 2965 additions and 288 deletions
+5
View File
@@ -102,6 +102,11 @@ jobs:
- name: Run tag replication tests under race detector - name: Run tag replication tests under race detector
run: go test -race ./cmd -run '^TestAPITagging' -count=1 -timeout=5m run: go test -race ./cmd -run '^TestAPITagging' -count=1 -timeout=5m
- name: Run Object Lock authorization and overwrite tests under race detector
env:
MINIO_API_REQUESTS_MAX: "10000"
run: go test -race -tags kqueue,dev ./cmd -run '^Test(APIPutObjectRetention|APIVersionAddressedWrite|PutObjectRetention|DirectoryObjectLock|ObjectLockOverwrite|APICopyObjectPreservesVersionAfterSuspension|MultipartNullObjectLock|PoolsObjectLockOverwrite|MultipartVersioningTransitions|PlanReview|RecoveredMultipartUpload)' -count=1 -timeout=5m
crosscompile: crosscompile:
name: Cross Compile name: Cross Compile
runs-on: ubuntu-latest runs-on: ubuntu-latest
+6
View File
@@ -70,6 +70,12 @@ jobs:
buildscripts/verify-rebrand.sh buildscripts/verify-rebrand.sh
dockerscripts/docker-entrypoint_test.sh dockerscripts/docker-entrypoint_test.sh
# Keep this gate aligned with the Object Lock race regression in go.yml.
- name: Gate release on Object Lock authorization and overwrite tests under race detector
env:
MINIO_API_REQUESTS_MAX: "10000"
run: go test -race -tags kqueue,dev ./cmd -run '^Test(APIPutObjectRetention|APIVersionAddressedWrite|PutObjectRetention|DirectoryObjectLock|ObjectLockOverwrite|APICopyObjectPreservesVersionAfterSuspension|MultipartNullObjectLock|PoolsObjectLockOverwrite|MultipartVersioningTransitions|PlanReview|RecoveredMultipartUpload)' -count=1 -timeout=5m
- name: Compute release variables - name: Compute release variables
env: env:
# Passed through the environment, never interpolated into the script # Passed through the environment, never interpolated into the script
+2 -2
View File
@@ -1,4 +1,4 @@
FROM golang:1.27.1-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS curl-build FROM golang:1.27.2-alpine@sha256:f92b6ef800e499660581efdabdf25d9d817a9d124eaf900924f0504e7e27e12d AS curl-build
ARG TARGETARCH ARG TARGETARCH
COPY dockerscripts/build-static-curl.sh /build/build-static-curl COPY dockerscripts/build-static-curl.sh /build/build-static-curl
RUN /bin/sh /build/build-static-curl RUN /bin/sh /build/build-static-curl
@@ -9,7 +9,7 @@ COPY --from=curl-build /go/bin/curl /curl
COPY --from=curl-build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=curl-build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/curl"] ENTRYPOINT ["/curl"]
FROM golang:1.27.1-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS build FROM golang:1.27.2-alpine@sha256:f92b6ef800e499660581efdabdf25d9d817a9d124eaf900924f0504e7e27e12d AS build
ARG TARGETARCH ARG TARGETARCH
+1 -1
View File
@@ -4,7 +4,7 @@ LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
GOOS ?= $(shell go env GOOS) GOOS ?= $(shell go env GOOS)
GOARCH ?= $(shell go env GOARCH) GOARCH ?= $(shell go env GOARCH)
GOLANGCI_VERSION ?= v2.13.1 GOLANGCI_VERSION ?= v2.14.0
VERSION ?= $(shell git describe --tags) VERSION ?= $(shell git describe --tags)
REPO ?= docker.io/pgsty REPO ?= docker.io/pgsty
+1 -1
View File
@@ -7,7 +7,7 @@ _init() {
## Minimum required versions for build dependencies ## Minimum required versions for build dependencies
GIT_VERSION="1.0" GIT_VERSION="1.0"
GO_VERSION="1.27.1" GO_VERSION="1.27.2"
OSX_VERSION="10.8" OSX_VERSION="10.8"
KNAME=$(uname -s) KNAME=$(uname -s)
ARCH=$(uname -m) ARCH=$(uname -m)
+3 -1
View File
@@ -249,8 +249,10 @@ func (a adminAPIHandlers) AddServiceAccountLDAP(w http.ResponseWriter, r *http.R
// In case of LDAP/OIDC we need to set `opts.claims` to ensure // In case of LDAP/OIDC we need to set `opts.claims` to ensure
// it is associated with the LDAP/OIDC user properly. // it is associated with the LDAP/OIDC user properly.
// The new credential's policy comes from the request, not the caller.
for k, v := range cred.Claims { for k, v := range cred.Claims {
if k == expClaim { switch k {
case expClaim, policy.SessionPolicyName, sessionPolicyNameExtracted, iamPolicyClaimNameSA():
continue continue
} }
opts.claims[k] = v opts.claims[k] = v
+507
View File
@@ -0,0 +1,507 @@
// Copyright (c) 2026 PGSTY
// SPDX-License-Identifier: AGPL-3.0-or-later
package cmd
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
polplugin "github.com/minio/minio/internal/config/policy/plugin"
"github.com/minio/mux"
xnet "github.com/pgsty/silo-pkg/v3/net"
"github.com/pgsty/silo-pkg/v3/policy"
)
func iamAdminFixture(t *testing.T) (context.Context, *IAMSys) {
t.Helper()
ctx, sys, obj := prepareIAMRevisionFixture(t)
globalObjLayerMutex.Lock()
globalObjectAPI = obj
globalObjLayerMutex.Unlock()
return ctx, sys
}
func iamAdminUser(ctx context.Context, t *testing.T, sys *IAMSys, name, policyName string) auth.Credentials {
t.Helper()
cred := auth.Credentials{AccessKey: name, SecretKey: "review-user-valid-secret"}
_, err := sys.CreateUser(ctx, name, madmin.AddOrUpdateUserReq{SecretKey: cred.SecretKey, Status: madmin.AccountEnabled})
mustIAM(t, err)
if policyName != "" {
_, err = sys.PolicyDBSet(ctx, name, policyName, regUser, false)
mustIAM(t, err)
}
return cred
}
func iamAdminServiceAccount(ctx context.Context, t *testing.T, sys *IAMSys, parent, key, document string) auth.Credentials {
t.Helper()
var sp *policy.Policy
var err error
if document != "" {
sp, err = policy.ParseConfig(strings.NewReader(document))
mustIAM(t, err)
}
cred, _, err := sys.NewServiceAccount(ctx, parent, nil, newServiceAccountOpts{accessKey: key, secretKey: "review-service-valid-secret", sessionPolicy: sp})
mustIAM(t, err)
return cred
}
func iamAdminRequest(t *testing.T, cred auth.Credentials, method, target string, payload any) *http.Request {
t.Helper()
var body []byte
if payload != nil {
raw, err := json.Marshal(payload)
mustIAM(t, err)
body, err = madmin.EncryptData(cred.SecretKey, raw)
mustIAM(t, err)
}
headers := map[string]string{}
if cred.IsTemp() && !cred.IsServiceAccount() {
headers["X-Amz-Security-Token"] = cred.SessionToken
}
req, err := newTestSignedRequestV4(method, "http://localhost"+target, int64(len(body)), bytes.NewReader(body), cred.AccessKey, cred.SecretKey, headers)
mustIAM(t, err)
mustIAM(t, req.ParseForm())
return req
}
func iamAdminCall(t *testing.T, cred auth.Credentials, method, target string, payload any, vars map[string]string, handler http.HandlerFunc) *httptest.ResponseRecorder {
t.Helper()
req := iamAdminRequest(t, cred, method, target, payload)
req = mux.SetURLVars(req, vars)
rec := httptest.NewRecorder()
req = req.WithContext(newContext(req, rec, "IAMAdminAuthorization"))
handler(rec, req)
return rec
}
func assertIAMAdminStatus(t *testing.T, operation string, rec *httptest.ResponseRecorder, want int) {
t.Helper()
if rec.Code != want {
t.Errorf("%s status=%d want=%d body=%s", operation, rec.Code, want, rec.Body.String())
return
}
if want == http.StatusForbidden || want == http.StatusNotFound {
var response APIErrorResponse
mustIAM(t, json.Unmarshal(rec.Body.Bytes(), &response))
wantCode := errorCodes.ToAPIErr(ErrAccessDenied).Code
if want == http.StatusNotFound {
wantCode = errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound).Code
}
if response.Code != wantCode {
t.Errorf("%s error code=%s want=%s", operation, response.Code, wantCode)
}
}
}
func iamAdminPlugin(t *testing.T, allow bool) {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]bool{"result": allow})
}))
t.Cleanup(server.Close)
u, err := xnet.ParseHTTPURL(server.URL)
mustIAM(t, err)
previous := newGlobalAuthZPluginFn()
setGlobalAuthZPlugin(polplugin.New(polplugin.Args{URL: u, CloseRespFn: func(r io.ReadCloser) { r.Close() }}))
t.Cleanup(func() { setGlobalAuthZPlugin(previous) })
}
func TestIAMAdminRootEmptyPolicy(t *testing.T) {
for _, doc := range []string{"", "null", "{}", `{"Statement":null}`} {
t.Run("policy="+doc, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "review-root-empty", doc)
sibling := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "review-root-sibling", "")
req := iamAdminRequest(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil)
checked, owner, code := checkKeyValid(req, caller.AccessKey)
if code != ErrNone {
t.Fatalf("checkKeyValid: %v", code)
}
active, _ := isAllowedBySessionPolicyForServiceAccount(policy.Args{Claims: checked.Claims})
t.Logf("owner=%v parsed_active_session_policy=%v", owner, active)
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "create", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-root-new", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccount), 200)
assertIAMAdminStatus(t, "update", iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+sibling.AccessKey, madmin.UpdateServiceAccountReq{NewName: "review"}, map[string]string{"accessKey": sibling.AccessKey}, api.UpdateServiceAccount), 204)
assertIAMAdminStatus(t, "delete", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+sibling.AccessKey, nil, map[string]string{"accessKey": sibling.AccessKey}, api.DeleteServiceAccount), 204)
assertIAMAdminStatus(t, "revoke", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 204)
})
}
}
func TestIAMAdminEmptyPolicyDelegation(t *testing.T) {
for _, parentType := range []string{"root", "user"} {
for _, document := range []string{"null", "{}", `{"Statement":null}`} {
t.Run(parentType+"/"+document, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := globalActiveCred
if parentType == "user" {
parent = iamAdminUser(ctx, t, sys, "empty-policy-parent", "readwrite")
}
sp, err := policy.ParseConfig(strings.NewReader(document))
mustIAM(t, err)
caller, _, err := sys.NewServiceAccount(ctx, parent.AccessKey, nil, newServiceAccountOpts{
accessKey: "empty-policy-caller", secretKey: "empty-policy-caller-secret", sessionPolicy: sp,
claims: map[string]any{"identity-attribute": "preserved"},
})
mustIAM(t, err)
const limitedPolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:ListServiceAccounts"]}]}`
for _, tc := range []struct {
policy string
listStatus int
allowObject bool
}{
{"", http.StatusOK, true},
{"null", http.StatusOK, true},
{"{}", http.StatusOK, true},
{`{"Statement":null}`, http.StatusOK, true},
{limitedPolicy, http.StatusOK, false},
{`{"Version":"2012-10-17"}`, http.StatusForbidden, false},
{`{"Statement":[]}`, http.StatusForbidden, false},
} {
expires := UTCNow().Add(time.Hour)
req := madmin.AddServiceAccountReq{Expiration: &expires}
if tc.policy != "" {
req.Policy = json.RawMessage(tc.policy)
}
api := adminAPIHandlers{}
rec := iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", req, nil, api.AddServiceAccount)
assertIAMAdminStatus(t, "create expiring child", rec, http.StatusOK)
data, err := madmin.DecryptData(caller.SecretKey, rec.Body)
mustIAM(t, err)
var response madmin.AddServiceAccountResp
mustIAM(t, json.Unmarshal(data, &response))
child, ok := sys.store.GetUser(response.Credentials.AccessKey)
if !ok {
t.Fatal("new service account missing")
}
// Exercise the issued key through authentication and authorization.
assertIAMAdminStatus(t, "use expiring child policy="+tc.policy, iamAdminCall(t, child.Credentials, "GET", "/minio/admin/v3/list-service-accounts", nil, nil, api.ListServiceAccounts), tc.listStatus)
signed := iamAdminRequest(t, child.Credentials, "GET", "/example/object", nil)
checked, owner, code := checkKeyValid(signed, child.Credentials.AccessKey)
if code != ErrNone {
t.Fatalf("child authentication: %v", code)
}
if checked.Claims["identity-attribute"] != "preserved" {
t.Fatal("creation lost the caller's identity context")
}
allowed := sys.IsAllowed(policy.Args{AccountName: checked.AccessKey, Claims: checked.Claims, IsOwner: owner, Action: policy.GetObjectAction, BucketName: "example", ObjectName: "object"})
if allowed != tc.allowObject {
t.Errorf("GetObject allowed=%v with requested policy=%s", allowed, tc.policy)
}
}
})
}
}
}
func TestIAMAdminCorruptServiceCleanup(t *testing.T) {
for _, status := range []string{auth.AccountOn, auth.AccountOff, "expired"} {
t.Run(status, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "review-admin", "consoleAdmin")
parent := iamAdminUser(ctx, t, sys, "review-parent", "readwrite")
svc := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-corrupt", "")
if status == auth.AccountOff {
_, err := sys.UpdateServiceAccount(ctx, svc.AccessKey, updateServiceAccountOpts{status: status})
mustIAM(t, err)
}
cache := sys.store.lock()
u := cache.iamUsersMap[svc.AccessKey]
u.Credentials.SessionToken = "broken-session-token"
if status == "expired" {
u.Credentials.Expiration = UTCNow().Add(-time.Minute)
}
cache.iamUsersMap[svc.AccessKey] = u
sys.store.unlock()
_, _, err := sys.GetServiceAccount(ctx, svc.AccessKey)
if err == nil {
t.Fatal("fault injection failed")
}
raw, exists := sys.store.GetUser(svc.AccessKey)
_, loginOK := sys.GetUser(ctx, svc.AccessKey)
t.Logf("metadata_exists=%v parent_known=%v GetUser_ok=%v", exists, raw.Credentials.ParentUser == parent.AccessKey, loginOK)
api := adminAPIHandlers{}
outsider := iamAdminUser(ctx, t, sys, "cleanup-outsider", "readwrite")
assertIAMAdminStatus(t, "unauthorized cleanup", iamAdminCall(t, outsider, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.DeleteServiceAccount), 404)
assertIAMAdminStatus(t, "admin cleanup", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.DeleteServiceAccount), 204)
if _, ok := sys.store.GetUser(svc.AccessKey); ok {
t.Fatal("deleted service account remains in the IAM cache")
}
})
}
}
func TestIAMAdminPolicylessSelf(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "review-policyless", "")
svc := iamAdminServiceAccount(ctx, t, sys, caller.AccessKey, "rv-policyless-child", "")
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "list", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil, nil, api.ListServiceAccounts), 200)
assertIAMAdminStatus(t, "info service", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.InfoServiceAccount), 200)
assertIAMAdminStatus(t, "info access key", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-access-key?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.InfoAccessKey), 200)
assertIAMAdminStatus(t, "delete", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+svc.AccessKey, nil, map[string]string{"accessKey": svc.AccessKey}, api.DeleteServiceAccount), 204)
assertIAMAdminStatus(t, "revoke", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 204)
assertIAMAdminStatus(t, "list another user", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts?user="+globalActiveCred.AccessKey, nil, nil, api.ListServiceAccounts), 403)
assertIAMAdminStatus(t, "revoke another user", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true&user="+globalActiveCred.AccessKey, nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
}
func TestIAMAdminExpiringInheritedPolicy(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "review-legacy-parent", "consoleAdmin")
document := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:CreateServiceAccount","admin:UpdateServiceAccount"]}]}`
// Model a credential persisted by the old handler: it copied its caller's
// policy claims but set the newly created service's type to inherited.
expires := UTCNow().Add(time.Hour)
caller, _, err := sys.NewServiceAccount(ctx, parent.AccessKey, nil, newServiceAccountOpts{
accessKey: "rv-exp-inherited", secretKey: "review-service-valid-secret", expiration: &expires,
claims: map[string]any{policy.SessionPolicyName: base64.StdEncoding.EncodeToString([]byte(document))},
})
mustIAM(t, err)
req := iamAdminRequest(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil)
checked, owner, code := checkKeyValid(req, caller.AccessKey)
if code != ErrNone {
t.Fatalf("checkKeyValid: %v", code)
}
args := policy.Args{AccountName: caller.AccessKey, Claims: checked.Claims, IsOwner: owner, Action: policy.CreateServiceAccountAdminAction, DenyOnly: true}
if !sys.IsAllowed(args) {
t.Fatal("fixture cannot create")
}
args.Action = policy.CreateUserAdminAction
args.DenyOnly = false
if sys.IsAllowed(args) {
t.Fatal("fixture not constrained")
}
t.Logf("service=%v temp=%v inherited=%v; create allowed, create-user denied", checked.IsServiceAccount(), checked.IsTemp(), checked.Claims[iamPolicyClaimNameSA()] == inheritedPolicyType)
api := adminAPIHandlers{}
rec := iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-escaped", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccount)
if rec.Code == 200 {
child, ok := sys.GetUser(ctx, "review-escaped")
if !ok {
t.Fatal("new key missing")
}
childReq := iamAdminRequest(t, child.Credentials, "GET", "/minio/admin/v3/list-service-accounts", nil)
childChecked, childOwner, childCode := checkKeyValid(childReq, child.Credentials.AccessKey)
if childCode != ErrNone {
t.Fatalf("child auth=%v", childCode)
}
allowed := sys.IsAllowed(policy.Args{AccountName: childChecked.AccessKey, Claims: childChecked.Claims, IsOwner: childOwner, Action: policy.CreateUserAdminAction})
t.Errorf("constrained inherited service created a new key; new key CreateUser allowed=%v", allowed)
} else {
assertIAMAdminStatus(t, "reject derived creation", rec, 403)
}
updated := iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+caller.AccessKey, madmin.UpdateServiceAccountReq{NewName: "metadata-only"}, map[string]string{"accessKey": caller.AccessKey}, api.UpdateServiceAccount)
if updated.Code == 204 {
after, ok := sys.GetUser(ctx, caller.AccessKey)
if !ok {
t.Fatal("updated key missing")
}
afterReq := iamAdminRequest(t, after.Credentials, "GET", "/minio/admin/v3/list-service-accounts", nil)
afterChecked, afterOwner, afterCode := checkKeyValid(afterReq, caller.AccessKey)
if afterCode != ErrNone {
t.Fatalf("updated auth=%v", afterCode)
}
allowed := sys.IsAllowed(policy.Args{AccountName: caller.AccessKey, Claims: afterChecked.Claims, IsOwner: afterOwner, Action: policy.CreateUserAdminAction})
t.Errorf("constrained inherited service updated itself; CreateUser allowed after update=%v", allowed)
} else {
assertIAMAdminStatus(t, "reject inherited self update", updated, 403)
}
}
func TestIAMAdminSessionPolicyCannotDelegate(t *testing.T) {
for _, temp := range []bool{false, true} {
t.Run(map[bool]string{false: "service", true: "sts"}[temp], func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "rv-parent", "consoleAdmin")
doc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:CreateServiceAccount","admin:UpdateServiceAccount"]}]}`
var caller auth.Credentials
if !temp {
caller = iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "review-restricted", doc)
} else {
var err error
caller, err = auth.GetNewCredentialsWithMetadata(map[string]any{parentClaim: parent.AccessKey, "exp": UTCNow().Add(time.Hour).Unix(), policy.SessionPolicyName: base64.StdEncoding.EncodeToString([]byte(doc))}, globalActiveCred.SecretKey)
mustIAM(t, err)
caller.ParentUser = parent.AccessKey
_, err = sys.SetTempUser(ctx, caller.AccessKey, caller, "")
mustIAM(t, err)
}
sibling := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-sibling", "")
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "restrict create", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-bad-new", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccount), 403)
assertIAMAdminStatus(t, "restrict LDAP create", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/idp/ldap/add-service-account", madmin.AddServiceAccountReq{AccessKey: "review-bad-new", SecretKey: "review-service-valid-secret"}, nil, api.AddServiceAccountLDAP), 403)
assertIAMAdminStatus(t, "restrict update", iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+sibling.AccessKey, madmin.UpdateServiceAccountReq{NewSecretKey: "review-known-new-secret"}, map[string]string{"accessKey": sibling.AccessKey}, api.UpdateServiceAccount), 403)
if _, ok := sys.store.GetUser("review-bad-new"); ok {
t.Fatal("denied creation persisted a credential")
}
u, ok := sys.store.GetUser(sibling.AccessKey)
if !ok || u.Credentials.SecretKey != sibling.SecretKey {
t.Fatal("denied update changed the target credential")
}
})
}
}
func TestIAMAdminSelfDenials(t *testing.T) {
for _, mode := range []string{"parent_deny", "group_deny", "session_implicit_deny", "session_explicit_deny", "plugin_deny"} {
t.Run(mode, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "rv-self-parent", "readwrite")
target := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-self-target", "")
caller := parent
doc := `{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["admin:ListServiceAccounts","admin:RemoveServiceAccount"]}]}`
switch mode {
case "parent_deny", "group_deny":
p, err := policy.ParseConfig(strings.NewReader(doc))
mustIAM(t, err)
_, err = sys.SetPolicy(ctx, "review-self-deny", *p)
mustIAM(t, err)
if mode == "group_deny" {
_, err = sys.AddUsersToGroup(ctx, "self-deny-group", []string{parent.AccessKey})
mustIAM(t, err)
_, err = sys.PolicyDBSet(ctx, "self-deny-group", "review-self-deny", regUser, true)
} else {
_, err = sys.PolicyDBSet(ctx, parent.AccessKey, "readwrite,review-self-deny", regUser, false)
}
mustIAM(t, err)
case "session_implicit_deny":
caller = iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-self-caller", `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::example/*"]}]}`)
case "session_explicit_deny":
caller = iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "rv-self-caller", doc)
case "plugin_deny":
iamAdminPlugin(t, false)
}
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "list", iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts", nil, nil, api.ListServiceAccounts), 403)
assertIAMAdminStatus(t, "info service", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.InfoServiceAccount), 403)
assertIAMAdminStatus(t, "info access key", iamAdminCall(t, caller, "GET", "/minio/admin/v3/info-access-key?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.InfoAccessKey), 403)
assertIAMAdminStatus(t, "delete", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 403)
assertIAMAdminStatus(t, "revoke", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
})
}
}
func TestIAMAdminNonRootCannotManageRoot(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "rv-console-admin", "consoleAdmin")
target := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "rv-root-target", "")
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "create root", iamAdminCall(t, caller, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{AccessKey: "rv-new-root", SecretKey: "review-service-valid-secret", TargetUser: globalActiveCred.AccessKey}, nil, api.AddServiceAccount), 403)
assertIAMAdminStatus(t, "update root", iamAdminCall(t, caller, "POST", "/minio/admin/v3/update-service-account?accessKey="+target.AccessKey, madmin.UpdateServiceAccountReq{NewSecretKey: "review-known-new-secret"}, map[string]string{"accessKey": target.AccessKey}, api.UpdateServiceAccount), 403)
assertIAMAdminStatus(t, "delete root", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 404)
assertIAMAdminStatus(t, "revoke root", iamAdminCall(t, caller, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true&user="+globalActiveCred.AccessKey, nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
// Restricted root-derived credentials and the site replicator are not root
// administrators, even when their effective policy allows these actions.
limited := iamAdminServiceAccount(ctx, t, sys, globalActiveCred.AccessKey, "limited-root", `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:*"]}]}`)
assertIAMAdminStatus(t, "limited root delete", iamAdminCall(t, limited, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 404)
assertIAMAdminStatus(t, "limited root revoke", iamAdminCall(t, limited, "POST", "/minio/admin/v3/revoke-tokens/builtin?fullRevoke=true", nil, map[string]string{"userProvider": madmin.BuiltinProvider}, api.RevokeTokens), 403)
replicator, _, err := sys.NewServiceAccount(ctx, globalActiveCred.AccessKey, nil, newServiceAccountOpts{accessKey: siteReplicatorSvcAcc, secretKey: "replicator-valid-secret", allowSiteReplicatorAccount: true})
mustIAM(t, err)
assertIAMAdminStatus(t, "replicator create root", iamAdminCall(t, replicator, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{}, nil, api.AddServiceAccount), 403)
// An external plugin remains authoritative, including for root targets.
iamAdminPlugin(t, true)
assertIAMAdminStatus(t, "plugin update root", iamAdminCall(t, limited, "POST", "/minio/admin/v3/update-service-account?accessKey="+target.AccessKey, madmin.UpdateServiceAccountReq{NewName: "plugin-authorized"}, map[string]string{"accessKey": target.AccessKey}, api.UpdateServiceAccount), 204)
assertIAMAdminStatus(t, "plugin create root", iamAdminCall(t, limited, "PUT", "/minio/admin/v3/add-service-account", madmin.AddServiceAccountReq{}, nil, api.AddServiceAccount), 200)
}
func TestIAMAdminDeleteCorruptServiceReplicates(t *testing.T) {
ctx, sys := iamAdminFixture(t)
caller := iamAdminUser(ctx, t, sys, "cleanup-admin", "consoleAdmin")
parent := iamAdminUser(ctx, t, sys, "cleanup-parent", "readwrite")
target := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "cleanup-target", "")
cache := sys.store.lock()
u := cache.iamUsersMap[target.AccessKey]
u.Credentials.SessionToken = "broken-session-token"
u.Credentials.Status = auth.AccountOff
cache.iamUsersMap[target.AccessKey] = u
sys.store.unlock()
var deletions atomic.Int32
peer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/minio/health/live" {
return
}
var batch iamRevisionBatch
if err := json.NewDecoder(r.Body).Decode(&batch); err != nil {
t.Error(err)
w.WriteHeader(http.StatusBadRequest)
return
}
if len(batch.Items) != 1 || batch.Items[0].SvcAccChange == nil || batch.Items[0].SvcAccChange.Delete == nil || batch.Items[0].SvcAccChange.Delete.AccessKey != target.AccessKey {
t.Error("replication did not carry the service account deletion")
} else {
deletions.Add(1)
}
_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "peer", Instance: "peer-boot", Digest: "deleted"}})
}))
t.Cleanup(peer.Close)
globalSiteReplicatorCred.RLock()
previousSecret := globalSiteReplicatorCred.secretKey
globalSiteReplicatorCred.RUnlock()
globalSiteReplicatorCred.Set(caller.SecretKey)
t.Cleanup(func() { globalSiteReplicatorCred.Set(previousSecret) })
globalSiteReplicationSys.Lock()
globalSiteReplicationSys.enabled = true
globalSiteReplicationSys.state = srState{ServiceAccountAccessKey: caller.AccessKey, Peers: map[string]madmin.PeerInfo{
globalDeploymentID(): {DeploymentID: globalDeploymentID(), Name: "local"},
"peer": {DeploymentID: "peer", Name: "peer", Endpoint: peer.URL},
}}
globalSiteReplicationSys.Unlock()
t.Cleanup(func() {
globalSiteReplicationSys.Lock()
globalSiteReplicationSys.enabled = false
globalSiteReplicationSys.Unlock()
})
api := adminAPIHandlers{}
assertIAMAdminStatus(t, "replicated cleanup", iamAdminCall(t, caller, "DELETE", "/minio/admin/v3/delete-service-account?accessKey="+target.AccessKey, nil, map[string]string{"accessKey": target.AccessKey}, api.DeleteServiceAccount), 204)
if deletions.Load() != 1 {
t.Fatalf("expected one immediate deletion notification, got %d", deletions.Load())
}
}
func TestIAMAdminListSelfAliases(t *testing.T) {
for _, kind := range []string{"service", "sts"} {
t.Run(kind, func(t *testing.T) {
ctx, sys := iamAdminFixture(t)
parent := iamAdminUser(ctx, t, sys, "alias-parent", "readwrite")
target := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "alias-target", "")
caller := iamAdminServiceAccount(ctx, t, sys, parent.AccessKey, "alias-caller", "")
if kind == "sts" {
var err error
caller, err = auth.GetNewCredentialsWithMetadata(map[string]any{parentClaim: parent.AccessKey, "exp": UTCNow().Add(time.Hour).Unix()}, globalActiveCred.SecretKey)
mustIAM(t, err)
caller.ParentUser = parent.AccessKey
_, err = sys.SetTempUser(ctx, caller.AccessKey, caller, "")
mustIAM(t, err)
}
api := adminAPIHandlers{}
for _, alias := range []string{"", caller.AccessKey, parent.AccessKey} {
rec := iamAdminCall(t, caller, "GET", "/minio/admin/v3/list-service-accounts?user="+alias, nil, nil, api.ListServiceAccounts)
assertIAMAdminStatus(t, "list self alias", rec, 200)
data, err := madmin.DecryptData(caller.SecretKey, rec.Body)
mustIAM(t, err)
var response madmin.ListServiceAccountsResp
mustIAM(t, json.Unmarshal(data, &response))
found := false
for _, account := range response.Accounts {
found = found || account.AccessKey == target.AccessKey
}
if !found {
t.Fatalf("self alias %q did not list the parent's service accounts", alias)
}
}
})
}
}
+139 -82
View File
@@ -714,8 +714,10 @@ func (a adminAPIHandlers) AddServiceAccount(w http.ResponseWriter, r *http.Reque
// In case of LDAP/OIDC we need to set `opts.claims` to ensure // In case of LDAP/OIDC we need to set `opts.claims` to ensure
// it is associated with the LDAP/OIDC user properly. // it is associated with the LDAP/OIDC user properly.
// The new credential's policy comes from the request, not the caller.
for k, v := range cred.Claims { for k, v := range cred.Claims {
if k == expClaim { switch k {
case expClaim, policy.SessionPolicyName, sessionPolicyNameExtracted, iamPolicyClaimNameSA():
continue continue
} }
opts.claims[k] = v opts.claims[k] = v
@@ -855,10 +857,9 @@ func (a adminAPIHandlers) UpdateServiceAccount(w http.ResponseWriter, r *http.Re
// Permission checks: // Permission checks:
// //
// 1. Any type of account (i.e. access keys (previously/still called service // 1. Updating requires policy.UpdateServiceAccountAdminAction. Under built-in
// accounts), STS accounts, internal IDP accounts, etc) with the // IAM, session-limited callers cannot update credentials, and only unrestricted
// policy.UpdateServiceAccountAdminAction permission can update any service // root credentials can manage root's service accounts.
// account.
// //
// 2. We would like to let a user update their own access keys, however it // 2. We would like to let a user update their own access keys, however it
// is currently blocked pending a re-design. Users are still able to delete // is currently blocked pending a re-design. Users are still able to delete
@@ -874,6 +875,11 @@ func (a adminAPIHandlers) UpdateServiceAccount(w http.ResponseWriter, r *http.Re
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL) writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return return
} }
if newGlobalAuthZPluginFn() == nil && (hasActiveSessionPolicy(cred) ||
(svcAccount.ParentUser == globalActiveCred.AccessKey && !canManageRootCredentials(cred))) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
var sp *policy.Policy var sp *policy.Policy
if len(updateReq.NewPolicy) > 0 { if len(updateReq.NewPolicy) > 0 {
@@ -951,23 +957,21 @@ func (a adminAPIHandlers) InfoServiceAccount(w http.ResponseWriter, r *http.Requ
return return
} }
if !globalIAMSys.IsAllowed(policy.Args{ requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey, AccountName: cred.AccessKey,
Groups: cred.Groups, Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction, Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred), ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner, IsOwner: owner,
Claims: cred.Claims, Claims: cred.Claims,
DenyOnly: requestUser == svcAccount.ParentUser,
}) { }) {
requestUser := cred.AccessKey writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
if cred.ParentUser != "" { return
requestUser = cred.ParentUser
}
if requestUser != svcAccount.ParentUser {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
} }
// if session policy is nil or empty, then it is implied policy // if session policy is nil or empty, then it is implied policy
@@ -1039,29 +1043,27 @@ func (a adminAPIHandlers) ListServiceAccounts(w http.ResponseWriter, r *http.Req
return return
} }
var targetAccount string selfUser := cred.AccessKey
if cred.ParentUser != "" {
// If listing is requested for a specific user (who is not the request selfUser = cred.ParentUser
// sender), check that the user has permissions. }
user := r.Form.Get("user") user := r.Form.Get("user")
if user != "" && user != cred.AccessKey { selfOnly := user == "" || user == cred.AccessKey || user == selfUser
if !globalIAMSys.IsAllowed(policy.Args{ if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey, AccountName: cred.AccessKey,
Groups: cred.Groups, Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction, Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred), ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner, IsOwner: owner,
Claims: cred.Claims, Claims: cred.Claims,
}) { DenyOnly: selfOnly,
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL) }) {
return writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
} return
targetAccount = user }
} else { targetAccount := user
targetAccount = cred.AccessKey if selfOnly {
if cred.ParentUser != "" { targetAccount = selfUser
targetAccount = cred.ParentUser
}
} }
serviceAccounts, err := globalIAMSys.ListServiceAccounts(ctx, targetAccount) serviceAccounts, err := globalIAMSys.ListServiceAccounts(ctx, targetAccount)
@@ -1110,7 +1112,7 @@ func (a adminAPIHandlers) DeleteServiceAccount(w http.ResponseWriter, r *http.Re
// Get current object layer instance. // Get current object layer instance.
objectAPI := newObjectLayerFn() objectAPI := newObjectLayerFn()
if objectAPI == nil || globalNotificationSys == nil { if objectAPI == nil || globalNotificationSys == nil || !globalIAMSys.Initialized() {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL) writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL)
return return
} }
@@ -1131,35 +1133,41 @@ func (a adminAPIHandlers) DeleteServiceAccount(w http.ResponseWriter, r *http.Re
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidArgument), r.URL) writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidArgument), r.URL)
return return
} }
// We do not care if service account is readable or not at this point, // Deletion needs the stored identity, not a valid token or session policy.
// since this is a delete call we shall allow it to be deleted if possible. // Keep disabled, expired and damaged service accounts removable, and retain
svcAccount, _, err := globalIAMSys.GetServiceAccount(ctx, serviceAccount) // their parent for authorization and the site replication hook below.
if errors.Is(err, errNoSuchServiceAccount) { u, ok := globalIAMSys.store.GetUser(serviceAccount)
if !ok || !u.Credentials.IsServiceAccount() {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL) writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL)
return return
} }
svcAccount := u.Credentials
adminPrivilege := globalIAMSys.IsAllowed(policy.Args{ parentUser := cred.AccessKey
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
isSelf := parentUser == svcAccount.ParentUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey, AccountName: cred.AccessKey,
Groups: cred.Groups, Groups: cred.Groups,
Action: policy.RemoveServiceAccountAdminAction, Action: policy.RemoveServiceAccountAdminAction,
ConditionValues: getConditionValues(r, "", cred), ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner, IsOwner: owner,
Claims: cred.Claims, Claims: cred.Claims,
}) DenyOnly: isSelf,
}) {
if !adminPrivilege { if !isSelf {
parentUser := cred.AccessKey // Hide the existence of another user's service account.
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
if svcAccount.ParentUser != "" && parentUser != svcAccount.ParentUser {
// The service account belongs to another user but return not
// found error to mitigate brute force attacks. or the
// serviceAccount doesn't exist.
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL) writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL)
return return
} }
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if newGlobalAuthZPluginFn() == nil && svcAccount.ParentUser == globalActiveCred.AccessKey && !canManageRootCredentials(cred) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminServiceAccountNotFound), r.URL)
return
} }
if err := globalIAMSys.DeleteServiceAccount(ctx, serviceAccount, true); err != nil { if err := globalIAMSys.DeleteServiceAccount(ctx, serviceAccount, true); err != nil {
@@ -2046,26 +2054,30 @@ func (a adminAPIHandlers) RevokeTokens(w http.ResponseWriter, r *http.Request) {
return return
} }
adminPrivilege := globalIAMSys.IsAllowed(policy.Args{ parentUser := cred.AccessKey
if cred.ParentUser != "" {
parentUser = cred.ParentUser
}
isSelfTarget := isTokenSelfRevoke || user == parentUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey, AccountName: cred.AccessKey,
Groups: cred.Groups, Groups: cred.Groups,
Action: policy.RemoveServiceAccountAdminAction, Action: policy.RemoveServiceAccountAdminAction,
ConditionValues: getConditionValues(r, "", cred), ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner, IsOwner: owner,
Claims: cred.Claims, Claims: cred.Claims,
}) DenyOnly: isSelfTarget,
}) {
if !adminPrivilege || isTokenSelfRevoke { writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
parentUser := cred.AccessKey return
if cred.ParentUser != "" { }
parentUser = cred.ParentUser if isSelfTarget {
}
if !isTokenSelfRevoke && user != parentUser {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
user = parentUser user = parentUser
} }
if newGlobalAuthZPluginFn() == nil && user == globalActiveCred.AccessKey && !canManageRootCredentials(cred) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
// Infer token revoke type from the request if requestor is STS. // Infer token revoke type from the request if requestor is STS.
if isTokenSelfRevoke && tokenRevokeType == "" && !fullRevoke { if isTokenSelfRevoke && tokenRevokeType == "" && !fullRevoke {
@@ -2112,29 +2124,22 @@ func (a adminAPIHandlers) InfoAccessKey(w http.ResponseWriter, r *http.Request)
u, ok := globalIAMSys.GetUser(ctx, accessKey) u, ok := globalIAMSys.GetUser(ctx, accessKey)
targetCred := u.Credentials targetCred := u.Credentials
if !globalIAMSys.IsAllowed(policy.Args{ requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
isSelf := ok && targetCred.ParentUser != "" && requestUser == targetCred.ParentUser
if !isAllowedServiceAccountAction(cred, policy.Args{
AccountName: cred.AccessKey, AccountName: cred.AccessKey,
Groups: cred.Groups, Groups: cred.Groups,
Action: policy.ListServiceAccountsAdminAction, Action: policy.ListServiceAccountsAdminAction,
ConditionValues: getConditionValues(r, "", cred), ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner, IsOwner: owner,
Claims: cred.Claims, Claims: cred.Claims,
DenyOnly: isSelf,
}) { }) {
// If requested user does not exist and requestor is not allowed to list service accounts, return access denied. writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
if !ok { return
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
requestUser := cred.AccessKey
if cred.ParentUser != "" {
requestUser = cred.ParentUser
}
if requestUser != targetCred.ParentUser {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
} }
if !ok { if !ok {
@@ -2908,6 +2913,44 @@ func addExpirationToCondValues(exp *time.Time, condValues map[string][]string) e
return nil return nil
} }
// hasActiveSessionPolicy follows the same credential dispatch as IAMSys.IsAllowed.
// Expiring service accounts also satisfy IsTemp and are evaluated as STS there.
// Callers use this only after authorization under built-in IAM.
func hasActiveSessionPolicy(cred auth.Credentials) bool {
args := policy.Args{Claims: cred.Claims}
if cred.IsTemp() {
hasSP, _ := isAllowedBySessionPolicy(args)
return hasSP
}
if cred.IsServiceAccount() {
if cred.Claims[iamPolicyClaimNameSA()] == inheritedPolicyType {
return false
}
hasSP, _ := isAllowedBySessionPolicyForServiceAccount(args)
return hasSP
}
return false
}
// canManageRootCredentials is an additional guard after built-in IAM authorization.
// The signature's owner flag is insufficient: historical empty session policies
// clear it even though the service account still inherits root's permissions.
func canManageRootCredentials(cred auth.Credentials) bool {
return cred.AccessKey == globalActiveCred.AccessKey ||
(cred.ParentUser == globalActiveCred.AccessKey && cred.AccessKey != siteReplicatorSvcAcc && !hasActiveSessionPolicy(cred))
}
// isAllowedServiceAccountAction preserves regular users' self inspection and
// cleanup even without a policy attachment, while enforcing explicit denies.
// Derived credentials and external authorization plugins use full IAM evaluation.
func isAllowedServiceAccountAction(cred auth.Credentials, args policy.Args) bool {
if args.DenyOnly && !args.IsOwner && !cred.IsTemp() && !cred.IsServiceAccount() && newGlobalAuthZPluginFn() == nil {
policies, err := globalIAMSys.PolicyDBGet(args.AccountName, args.Groups...)
return err == nil && globalIAMSys.GetCombinedPolicy(policies...).IsAllowed(args)
}
return globalIAMSys.IsAllowed(args)
}
func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.Credentials, newServiceAccountOpts, madmin.AddServiceAccountReq, string, APIError) { func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.Credentials, newServiceAccountOpts, madmin.AddServiceAccountReq, string, APIError) {
ctx := r.Context() ctx := r.Context()
@@ -2996,6 +3039,14 @@ func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.
}) { }) {
return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", errorCodes.ToAPIErr(ErrAccessDenied) return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", errorCodes.ToAPIErr(ErrAccessDenied)
} }
effectiveTarget := targetUser
if (targetUser == cred.AccessKey || targetUser == cred.ParentUser) && cred.ParentUser != "" {
effectiveTarget = cred.ParentUser
}
if newGlobalAuthZPluginFn() == nil && (hasActiveSessionPolicy(cred) ||
(effectiveTarget == globalActiveCred.AccessKey && !canManageRootCredentials(cred))) {
return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", errorCodes.ToAPIErr(ErrAccessDenied)
}
var sp *policy.Policy var sp *policy.Policy
if len(createReq.Policy) > 0 { if len(createReq.Policy) > 0 {
@@ -3003,6 +3054,12 @@ func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.
if err != nil { if err != nil {
return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", toAdminAPIErr(ctx, err) return ctx, auth.Credentials{}, newServiceAccountOpts{}, madmin.AddServiceAccountReq{}, "", toAdminAPIErr(ctx, err)
} }
// Empty service policies mean inheritance, including for expiring keys.
// Normalize the replication request as well as the local credential.
if sp.Version == "" && sp.Statements == nil && sp.ID == "" {
sp = nil
createReq.Policy = nil
}
} }
opts.sessionPolicy = sp opts.sessionPolicy = sp
+45
View File
@@ -0,0 +1,45 @@
// Copyright (c) 2026 PGSTY
//
// This file is part of SILO Object Storage.
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"fmt"
"net/http"
"testing"
"github.com/minio/minio/internal/hash"
)
func TestChecksumMismatchAPIMessage(t *testing.T) {
for _, algorithm := range []string{"CRC32", "CRC32C", "SHA1", "SHA256", "CRC64NVME", ""} {
t.Run(algorithm, func(t *testing.T) {
mismatch := hash.ChecksumMismatch{Algorithm: algorithm, Want: "expected", Got: "actual"}
want := errorCodes.ToAPIErr(ErrContentChecksumMismatch).Description
if algorithm != "" {
want = fmt.Sprintf("The %s you specified did not match the calculated checksum.", algorithm)
}
for _, err := range []error{mismatch, fmt.Errorf("upload: %w", mismatch)} {
got := toAPIError(t.Context(), err)
if got.HTTPStatusCode != http.StatusBadRequest || got.Code != "XAmzContentChecksumMismatch" || got.Description != want {
t.Fatalf("toAPIError(%v) = %+v, want 400 XAmzContentChecksumMismatch: %s", err, got, want)
}
}
})
}
}
+9 -1
View File
@@ -550,7 +550,7 @@ var errorCodes = errorCodeMap{
}, },
ErrBadDigest: { ErrBadDigest: {
Code: "BadDigest", Code: "BadDigest",
Description: "The Content-Md5 you specified did not match what we received.", Description: "The Content-MD5 you specified did not match what we received.",
HTTPStatusCode: http.StatusBadRequest, HTTPStatusCode: http.StatusBadRequest,
}, },
ErrEntityTooSmall: { ErrEntityTooSmall: {
@@ -2505,6 +2505,14 @@ func toAPIError(ctx context.Context, err error) APIError {
case errors.Is(err, errMissingPartChecksum): case errors.Is(err, errMissingPartChecksum):
apiErr.Description = strings.TrimPrefix(err.Error(), errMissingPartChecksum.Error()+": ") apiErr.Description = strings.TrimPrefix(err.Error(), errMissingPartChecksum.Error()+": ")
} }
// PutObject/UploadPart checksum mismatches name the algorithm in the
// message, matching S3 (e.g. "The CRC32 you specified did not match the
// calculated checksum."). Falls through to the generic table description
// when the algorithm is unknown.
var checksumMismatch hash.ChecksumMismatch
if errors.As(err, &checksumMismatch) && checksumMismatch.Algorithm != "" {
apiErr.Description = fmt.Sprintf("The %s you specified did not match the calculated checksum.", checksumMismatch.Algorithm)
}
switch apiErr.Code { switch apiErr.Code {
case "NotImplemented": case "NotImplemented":
apiErr = APIError{ apiErr = APIError{
-45
View File
@@ -28,13 +28,10 @@ import (
"mime" "mime"
"net/http" "net/http"
"net/url" "net/url"
"strconv"
"strings" "strings"
"sync/atomic" "sync/atomic"
"time"
"github.com/minio/minio/internal/auth" "github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
"github.com/minio/minio/internal/etag" "github.com/minio/minio/internal/etag"
"github.com/minio/minio/internal/hash" "github.com/minio/minio/internal/hash"
xhttp "github.com/minio/minio/internal/http" xhttp "github.com/minio/minio/internal/http"
@@ -733,48 +730,6 @@ func setAuthMiddleware(h http.Handler) http.Handler {
}) })
} }
func isPutRetentionAllowed(bucketName, objectName string, retDays int, retDate time.Time, retMode objectlock.RetMode, byPassSet bool, r *http.Request, cred auth.Credentials, owner bool) (s3Err APIErrorCode) {
var retSet bool
if cred.AccessKey == "" {
return ErrAccessDenied
}
conditions := getConditionValues(r, "", cred)
conditions["object-lock-mode"] = []string{string(retMode)}
conditions["object-lock-retain-until-date"] = []string{retDate.UTC().Format(time.RFC3339)}
if retDays > 0 {
conditions["object-lock-remaining-retention-days"] = []string{strconv.Itoa(retDays)}
}
if retMode == objectlock.RetGovernance && byPassSet {
byPassSet = globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.BypassGovernanceRetentionAction,
BucketName: bucketName,
ObjectName: objectName,
ConditionValues: conditions,
IsOwner: owner,
Claims: cred.Claims,
})
}
if globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.PutObjectRetentionAction,
BucketName: bucketName,
ConditionValues: conditions,
ObjectName: objectName,
IsOwner: owner,
Claims: cred.Claims,
}) {
retSet = true
}
if byPassSet || retSet {
return ErrNone
}
return ErrAccessDenied
}
// isPutActionAllowed - check if PUT operation is allowed on the resource, this // isPutActionAllowed - check if PUT operation is allowed on the resource, this
// call verifies bucket policies and IAM policies, supports multi user // call verifies bucket policies and IAM policies, supports multi user
// checks etc. // checks etc.
+5 -1
View File
@@ -583,7 +583,11 @@ func toObjectInfo(bucket, object string, objInfo minio.ObjectInfo) ObjectInfo {
func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLayer, remoteClnt *minio.Client, entries []ObjectInfo, prefix string) error { func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLayer, remoteClnt *minio.Client, entries []ObjectInfo, prefix string) error {
input := make(chan minio.SnowballObject, 1) input := make(chan minio.SnowballObject, 1)
opts := minio.SnowballOptions{ opts := minio.SnowballOptions{
Opts: minio.PutObjectOptions{}, Opts: minio.PutObjectOptions{
Internal: minio.AdvancedPutOptions{
ReplicationRequest: true,
},
},
InMemory: *r.Source.Snowball.InMemory, InMemory: *r.Source.Snowball.InMemory,
Compress: *r.Source.Snowball.Compress, Compress: *r.Source.Snowball.Compress,
SkipErrs: *r.Source.Snowball.SkipErrs, SkipErrs: *r.Source.Snowball.SkipErrs,
+6
View File
@@ -1273,6 +1273,12 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err)) writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
return return
} }
if isDirObject(object) {
if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, false); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
}
opts.WantChecksum = checksum opts.WantChecksum = checksum
fanOutOpts := fanOutOptions{Checksum: checksum} fanOutOpts := fanOutOptions{Checksum: checksum}
+122 -61
View File
@@ -20,8 +20,8 @@ package cmd
import ( import (
"context" "context"
"errors" "errors"
"math"
"net/http" "net/http"
"strconv"
"strings" "strings"
"time" "time"
@@ -164,76 +164,129 @@ func enforceRetentionBypassForDelete(ctx context.Context, r *http.Request, bucke
return nil return nil
} }
// enforceRetentionBypassForPut enforces whether an existing object under governance can be overwritten // isRetentionActionAllowed evaluates action for a PutObjectRetention request
// with governance bypass headers set in the request. // with the object-lock condition keys derived from the requested retention.
// Objects under site wide WORM cannot be overwritten. // The keys come from the request, not from the stored version, so the
// For objects in "Governance" mode, overwrite is allowed if a) object retention date is past OR // handler can decide s3:PutObjectRetention before reading any object state.
// governance bypass headers are set and user has governance bypass permissions. func isRetentionActionAllowed(action policy.Action, bucket, object string, objRetention *objectlock.ObjectRetention, r *http.Request, cred auth.Credentials, owner bool) bool {
// Objects in compliance mode can be overwritten only if retention date is being extended. No mode change is permitted. if cred.AccessKey == "" {
return false
}
t, err := objectlock.UTCNowNTP()
if err != nil {
return false
}
conditions := getConditionValues(r, "", cred)
conditions["object-lock-mode"] = []string{string(objRetention.Mode)}
conditions["object-lock-retain-until-date"] = []string{objRetention.RetainUntilDate.UTC().Format(time.RFC3339)}
// Relative days from now, for "object-lock-remaining-retention-days".
// Clearing retention (<Retention/>) has a zero RetainUntilDate, which must
// evaluate as 0 remaining days rather than a large positive math.Abs delta
// from year 0001.
days := 0
if !objRetention.RetainUntilDate.IsZero() && objRetention.RetainUntilDate.After(t) {
// Time.Sub saturates after about 292 years. Count seconds instead so
// maximum-retention policies also apply to distant RFC3339 dates.
seconds := objRetention.RetainUntilDate.Unix() - t.Unix()
days = int(seconds / 86400)
if seconds%86400 != 0 || objRetention.RetainUntilDate.Nanosecond() > t.Nanosecond() {
days++
}
}
conditions["object-lock-remaining-retention-days"] = []string{strconv.Itoa(days)}
return globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: action,
BucketName: bucket,
ConditionValues: conditions,
ObjectName: object,
IsOwner: owner,
Claims: cred.Claims,
})
}
// enforceRetentionBypassForPut enforces the stored retention of oi against a
// PutObjectRetention request. The caller must already hold
// s3:PutObjectRetention; this only decides changes that depend on the stored
// state. Expired or absent retention may be changed freely. Unexpired
// GOVERNANCE retention may be shortened, removed or have its mode changed only
// with the bypass header and s3:BypassGovernanceRetention. Unexpired
// COMPLIANCE retention may only be extended.
func enforceRetentionBypassForPut(ctx context.Context, r *http.Request, oi ObjectInfo, objRetention *objectlock.ObjectRetention, cred auth.Credentials, owner bool) error { func enforceRetentionBypassForPut(ctx context.Context, r *http.Request, oi ObjectInfo, objRetention *objectlock.ObjectRetention, cred auth.Credentials, owner bool) error {
byPassSet := objectlock.IsObjectLockGovernanceBypassSet(r.Header) ret := objectlock.GetObjectRetentionMeta(oi.UserDefined)
if !ret.Mode.Valid() {
return nil
}
t, err := objectlock.UTCNowNTP() t, err := objectlock.UTCNowNTP()
if err != nil { if err != nil {
internalLogIf(ctx, err, logger.WarningKind) internalLogIf(ctx, err, logger.WarningKind)
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID} return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
} }
if ret.RetainUntilDate.Before(t) {
return nil
}
// Pass in relative days from current time, to additionally weakens := objRetention.Mode != ret.Mode || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time)
// to verify "object-lock-remaining-retention-days" policy if any. if !weakens {
days := int(math.Ceil(math.Abs(objRetention.RetainUntilDate.Sub(t).Hours()) / 24)) return nil
}
ret := objectlock.GetObjectRetentionMeta(oi.UserDefined) switch ret.Mode {
if ret.Mode.Valid() { case objectlock.RetGovernance:
// Retention has expired you may change whatever you like. // https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-bypass-governance
if ret.RetainUntilDate.Before(t) { if !objectlock.IsObjectLockGovernanceBypassSet(r.Header) {
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name, return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
days, objRetention.RetainUntilDate.Time,
objRetention.Mode, byPassSet, r, cred,
owner)
if apiErr == ErrAccessDenied {
return errAuthentication
}
return nil
} }
if !isRetentionActionAllowed(policy.BypassGovernanceRetentionAction, oi.Bucket, oi.Name, objRetention, r, cred, owner) {
switch ret.Mode { return errAuthentication
case objectlock.RetGovernance:
govPerm := isPutRetentionAllowed(oi.Bucket, oi.Name, days,
objRetention.RetainUntilDate.Time, objRetention.Mode,
byPassSet, r, cred, owner)
// Governance mode retention period cannot be shortened, if x-amz-bypass-governance is not set.
if !byPassSet {
if objRetention.Mode != objectlock.RetGovernance || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
}
if govPerm == ErrAccessDenied {
return errAuthentication
}
return nil
case objectlock.RetCompliance:
// Compliance retention mode cannot be changed or shortened.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html#object-lock-retention-modes
if objRetention.Mode != objectlock.RetCompliance || objRetention.RetainUntilDate.Before(ret.RetainUntilDate.Time) {
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name,
days, objRetention.RetainUntilDate.Time, objRetention.Mode,
false, r, cred, owner)
if apiErr == ErrAccessDenied {
return errAuthentication
}
return nil
} }
return nil return nil
} // No pre-existing retention metadata present. default:
// Compliance retention mode cannot be changed or shortened.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html#object-lock-retention-modes
return ObjectLocked{Bucket: oi.Bucket, Object: oi.Name, VersionID: oi.VersionID}
}
}
apiErr := isPutRetentionAllowed(oi.Bucket, oi.Name, // checkObjectLockedForOverwrite re-checks under the object write lock whether an
days, objRetention.RetainUntilDate.Time, // in-place write to the null version (such as a directory object) would
objRetention.Mode, byPassSet, r, cred, owner) // overwrite a locked version in an Object Lock bucket.
if apiErr == ErrAccessDenied { func checkObjectLockedForOverwrite(ctx context.Context, bucket, object string, getObjectInfoFn GetObjectInfoFn) error {
return errAuthentication if isMinioMetaBucketName(bucket) {
return nil
}
retentionCfg, err := globalBucketObjectLockSys.Get(bucket)
if err != nil {
return err
}
if !retentionCfg.LockEnabled {
return nil
}
objInfo, err := getObjectInfoFn(ctx, bucket, object, ObjectOptions{
VersionID: nullVersionID,
NoLock: true,
NoAuditLog: true,
})
if err != nil {
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || errors.As(err, &MethodNotAllowed{}) {
return nil
}
return err
}
if objectlock.GetObjectLegalHoldMeta(objInfo.UserDefined).Status == objectlock.LegalHoldOn {
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined)
if r.Mode == objectlock.RetCompliance || r.Mode == objectlock.RetGovernance {
t, err := objectlock.UTCNowNTP()
if err != nil {
internalLogIf(ctx, err, logger.WarningKind)
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
if r.RetainUntilDate.After(t) {
return ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: nullVersionID}
}
} }
return nil return nil
} }
@@ -274,16 +327,24 @@ func checkPutObjectLockAllowed(ctx context.Context, rq *http.Request, bucket, ob
if err != nil { if err != nil {
return mode, retainDate, legalHold, toAPIErrorCode(ctx, err) return mode, retainDate, legalHold, toAPIErrorCode(ctx, err)
} }
if isDirObject(object) && opts.VersionID == "" {
opts.VersionID = nullVersionID
}
if opts.VersionID != "" && !replicaTrusted { if opts.VersionID != "" && !replicaTrusted {
if objInfo, err := getObjectInfoFn(ctx, bucket, object, opts); err == nil { objInfo, err := getObjectInfoFn(ctx, bucket, object, opts)
if err != nil {
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) && !errors.As(err, &MethodNotAllowed{}) {
return mode, retainDate, legalHold, toAPIErrorCode(ctx, err)
}
} else {
r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined) r := objectlock.GetObjectRetentionMeta(objInfo.UserDefined)
t, err := objectlock.UTCNowNTP() t, err := objectlock.UTCNowNTP()
if err != nil { if err != nil {
internalLogIf(ctx, err, logger.WarningKind) internalLogIf(ctx, err, logger.WarningKind)
return mode, retainDate, legalHold, ErrObjectLocked return mode, retainDate, legalHold, ErrObjectLocked
} }
if r.Mode == objectlock.RetCompliance && r.RetainUntilDate.After(t) { if (r.Mode == objectlock.RetCompliance || r.Mode == objectlock.RetGovernance) && r.RetainUntilDate.After(t) {
return mode, retainDate, legalHold, ErrObjectLocked return mode, retainDate, legalHold, ErrObjectLocked
} }
mode = r.Mode mode = r.Mode
+91
View File
@@ -20,6 +20,7 @@ package cmd
import ( import (
"context" "context"
"encoding/hex" "encoding/hex"
"errors"
"fmt" "fmt"
"math/rand" "math/rand"
"reflect" "reflect"
@@ -110,6 +111,96 @@ func TestReduceErrs(t *testing.T) {
} }
} }
// TestReduceWriteQuorumErrsCommitMetaMissing verifies that an in-quorum
// errCommitMetaMissing reduces to the sentinel, which renameData() converts
// to errErasureWriteQuorum, while below-quorum reduces directly to
// errErasureWriteQuorum.
func TestReduceWriteQuorumErrsCommitMetaMissing(t *testing.T) {
errs := make([]error, 16)
for i := range 9 {
errs[i] = errCommitMetaMissing
}
if err := reduceWriteQuorumErrs(t.Context(), errs, objectOpIgnoredErrs, 9); !errors.Is(err, errCommitMetaMissing) {
t.Errorf("expected %s, got %s", errCommitMetaMissing, err)
}
// The sentinel must be distinguishable from errFileNotFound so that
// commit failures are never mapped to ObjectNotFound (404).
if errors.Is(errCommitMetaMissing, errFileNotFound) {
t.Error("errCommitMetaMissing must not match errFileNotFound")
}
// Below quorum the reduction must yield a write-quorum failure (503);
// eight sentinels against eight committed drives is not a quorum.
belowQuorum := make([]error, 16)
for i := range 8 {
belowQuorum[i] = errCommitMetaMissing
}
if err := reduceWriteQuorumErrs(t.Context(), belowQuorum, objectOpIgnoredErrs, 9); !errors.Is(err, errErasureWriteQuorum) {
t.Errorf("expected %s, got %s", errErasureWriteQuorum, err)
}
}
// renameDataFailingDisk is a StorageAPI stub returning a fixed error
// from RenameData, enough to drive renameData() in unit tests.
type renameDataFailingDisk struct {
StorageAPI
err error
undoWrite bool
}
func (d *renameDataFailingDisk) RenameData(context.Context, string, string, FileInfo, string, string, RenameOptions) (RenameDataResp, error) {
return RenameDataResp{}, d.err
}
func (d *renameDataFailingDisk) DeleteVersion(_ context.Context, _, _ string, _ FileInfo, _ bool, opts DeleteOptions) error {
d.undoWrite = opts.UndoWrite
return nil
}
// TestRenameDataCommitMetaMissing verifies that renameData() converts an
// in-quorum errCommitMetaMissing into errErasureWriteQuorum (HTTP 503),
// while an in-quorum errFileNotFound keeps its meaning so that
// PutObject() still reports IncompleteBody for client disconnects.
func TestRenameDataCommitMetaMissing(t *testing.T) {
for _, tc := range []struct {
name string
failures int
diskErr error
wantErr error
}{
{"all_commit_metadata_missing", 16, errCommitMetaMissing, errErasureWriteQuorum},
{"quorum_commit_metadata_missing", 9, errCommitMetaMissing, errErasureWriteQuorum},
{"neither_side_has_quorum", 8, errCommitMetaMissing, errErasureWriteQuorum},
{"successful_write_quorum", 7, errCommitMetaMissing, nil},
{"staging_failure", 16, errFileNotFound, errFileNotFound},
} {
t.Run(tc.name, func(t *testing.T) {
metadata := make([]FileInfo, 16)
disks := make([]StorageAPI, 16)
for i := range disks {
metadata[i] = FileInfo{Deleted: true} // valid without erasure info
disk := &renameDataFailingDisk{}
if i < tc.failures {
disk.err = tc.diskErr
}
disks[i] = disk
}
_, _, _, err := renameData(t.Context(), disks, minioMetaTmpBucket, "tmp", metadata, "bucket", "object", 9)
if !errors.Is(err, tc.wantErr) {
t.Fatalf("expected %v, got %v", tc.wantErr, err)
}
for i, disk := range disks {
// Failed writes undo only the drives that committed. A
// successful quorum must never be rolled back.
wantUndo := tc.wantErr != nil && i >= tc.failures
if got := disk.(*renameDataFailingDisk).undoWrite; got != wantUndo {
t.Errorf("disk %d: UndoWrite = %v, want %v", i, got, wantUndo)
}
}
})
}
}
// TestHashOrder - test order of ints in array // TestHashOrder - test order of ints in array
func TestHashOrder(t *testing.T) { func TestHashOrder(t *testing.T) {
testCases := []struct { testCases := []struct {
+177
View File
@@ -0,0 +1,177 @@
// Copyright (c) 2026 clown15
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"encoding/base64"
"fmt"
"net/url"
"os"
"path/filepath"
"testing"
"time"
)
func TestExtractModTimeFromUploadIDDir(t *testing.T) {
now := time.Unix(1792056747, 123456789).UTC()
nano := now.UnixNano()
uuid := "d16ad42a-6fe5-4a3b-8092-69983142238f"
testCases := []struct {
name string
dirName string
expectedMod time.Time
wantZero bool
}{
{
name: "Modern Plaintext Format (<UUID>x<UnixNano>)",
dirName: fmt.Sprintf("%sx%d", uuid, nano),
expectedMod: time.Unix(0, nano),
wantZero: false,
},
{
// This valid UUID decodes to bytes containing 'x'. Checking
// for that byte before parsing plaintext loses the timestamp.
name: "Plaintext Also Decodes To Base64 Containing x",
dirName: fmt.Sprintf("4eacd477-e733-4851-b821-6fd42785ff87x%d", nano),
expectedMod: now,
},
{
name: "Legacy Format Without Timestamp (<UUID>)",
dirName: uuid,
expectedMod: time.Time{},
wantZero: true,
},
{
name: "Base64 Encoded Format",
dirName: base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("%sx%d", uuid, nano))),
expectedMod: time.Unix(0, nano),
wantZero: false,
},
{
name: "Encoded Format With Deployment Prefix",
dirName: base64.RawURLEncoding.EncodeToString([]byte(fmt.Sprintf("%s.%sx%d", uuid, uuid, nano))),
expectedMod: now,
},
{
name: "Malformed Suffix (<UUID>xinvalid)",
dirName: fmt.Sprintf("%sxnotanumber", uuid),
expectedMod: time.Time{},
wantZero: true,
},
{
name: "Timestamp Overflow",
dirName: uuid + "x9223372036854775808",
wantZero: true,
},
{
name: "Malformed Plaintext UUID",
dirName: fmt.Sprintf("not-a-uuidx%d", nano),
wantZero: true,
},
{
name: "Empty Directory Name",
dirName: "",
expectedMod: time.Time{},
wantZero: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
got := extractModTimeFromUploadIDDir(tc.dirName)
if tc.wantZero {
if !got.IsZero() {
t.Fatalf("expected zero time for %q, got %v", tc.dirName, got)
}
return
}
if !got.Equal(tc.expectedMod) {
t.Fatalf("extractModTimeFromUploadIDDir(%q) = %v; want %v", tc.dirName, got, tc.expectedMod)
}
})
}
}
type multipartCleanupDisk struct {
StorageAPI
root string
modTime time.Time
readErr error
reads int
}
func (d *multipartCleanupDisk) Endpoint() Endpoint {
return Endpoint{URL: &url.URL{Path: d.root}}
}
func (d *multipartCleanupDisk) ReadVersion(context.Context, string, string, string, string, ReadOptions) (FileInfo, error) {
d.reads++
return FileInfo{ModTime: d.modTime}, d.readErr
}
func (d *multipartCleanupDisk) StatVol(context.Context, string) (VolInfo, error) {
return VolInfo{Created: time.Now()}, nil
}
func TestCleanupStaleUploadsOnDisk(t *testing.T) {
const uuid = "d16ad42a-6fe5-4a3b-8092-69983142238f"
expiry := globalAPIConfig.getStaleUploadsExpiry()
for _, tc := range []struct {
name, prefix string
age time.Duration
legacy bool
readErr error
wantReads int
wantDeleted bool
}{
{name: "recent", prefix: uuid, age: expiry / 2, readErr: errFileNotFound},
{name: "plaintext_base64_collision", prefix: "4eacd477-e733-4851-b821-6fd42785ff87", age: expiry / 2, readErr: errFileNotFound},
{name: "expired", prefix: uuid, age: 2 * expiry, readErr: errFileNotFound, wantDeleted: true},
{name: "legacy_recent", prefix: uuid, age: expiry / 2, legacy: true, wantReads: 1},
{name: "legacy_expired", prefix: uuid, age: 2 * expiry, legacy: true, wantReads: 1, wantDeleted: true},
{name: "legacy_metadata_missing", prefix: uuid, age: 2 * expiry, legacy: true, readErr: errFileNotFound, wantReads: 1},
{name: "malformed_metadata_missing", prefix: "not-a-uuid", age: 2 * expiry, readErr: errFileNotFound, wantReads: 1},
} {
t.Run(tc.name, func(t *testing.T) {
modTime := time.Now().Add(-tc.age)
id := tc.prefix
if !tc.legacy {
id = fmt.Sprintf("%sx%d", tc.prefix, modTime.UnixNano())
}
disk := &multipartCleanupDisk{root: t.TempDir(), modTime: modTime, readErr: tc.readErr}
uploadDir := filepath.Join(disk.root, minioMetaMultipartBucket, "sha", id)
if err := os.MkdirAll(uploadDir, 0o755); err != nil {
t.Fatal(err)
}
erasureObjects{}.cleanupStaleUploadsOnDisk(t.Context(), disk)
if disk.reads != tc.wantReads {
t.Errorf("ReadVersion calls = %d, want %d", disk.reads, tc.wantReads)
}
_, err := os.Stat(uploadDir)
if tc.wantDeleted {
if !os.IsNotExist(err) {
t.Errorf("expired upload must be removed: %v", err)
}
} else if err != nil {
t.Errorf("upload must be preserved: %v", err)
}
})
}
}
+2 -2
View File
@@ -253,7 +253,7 @@ func testAPICompleteMultipartFullObjectChecksumMismatch(obj ObjectLayer, instanc
if apiErr.Code != "BadDigest" { if apiErr.Code != "BadDigest" {
t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code) t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code)
} }
if want := "The CRC32 checksum you specified did not match the calculated checksum."; apiErr.Message != want { if want := "The CRC32 you specified did not match the calculated checksum."; apiErr.Message != want {
t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message) t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message)
} }
@@ -361,7 +361,7 @@ func testAPICompleteMultipartCompositeChecksumMismatch(obj ObjectLayer, instance
if apiErr.Code != "BadDigest" { if apiErr.Code != "BadDigest" {
t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code) t.Fatalf("%s: expected BadDigest, got %q", instanceType, apiErr.Code)
} }
if want := "The CRC32 checksum you specified did not match the calculated checksum."; apiErr.Message != want { if want := "The CRC32 you specified did not match the calculated checksum."; apiErr.Message != want {
t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message) t.Fatalf("%s: expected message %q, got %q", instanceType, want, apiErr.Message)
} }
if _, err := obj.GetObjectInfo(t.Context(), bucketName, objectName, ObjectOptions{}); err == nil { if _, err := obj.GetObjectInfo(t.Context(), bucketName, objectName, ObjectOptions{}); err == nil {
+45 -10
View File
@@ -174,6 +174,24 @@ func (er erasureObjects) deleteAll(ctx context.Context, bucket, prefix string) {
wg.Wait() wg.Wait()
} }
// extractModTimeFromUploadIDDir extracts creation time from the upload ID directory on disk.
// On-disk directory names are of the form <UUID>x<UnixNano>.
// Parse the native format first: plaintext UUIDs can also be valid Base64.
// Preserve the previous encoded-format parser for compatibility.
func extractModTimeFromUploadIDDir(uploadIDDir string) time.Time {
if modTime, ok := multipartUploadTime(uploadIDDir); ok {
return modTime
}
if b64, err := base64.RawURLEncoding.DecodeString(uploadIDDir); err == nil {
if split := strings.Split(string(b64), "x"); len(split) == 2 {
if t, err := strconv.ParseInt(split[1], 10, 64); err == nil {
return time.Unix(0, t)
}
}
}
return time.Time{}
}
// Remove the old multipart uploads on the given disk. // Remove the old multipart uploads on the given disk.
func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk StorageAPI) { func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk StorageAPI) {
drivePath := disk.Endpoint().Path drivePath := disk.Endpoint().Path
@@ -181,16 +199,7 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
readDirFn(pathJoin(drivePath, minioMetaMultipartBucket), func(shaDir string, typ os.FileMode) error { readDirFn(pathJoin(drivePath, minioMetaMultipartBucket), func(shaDir string, typ os.FileMode) error {
readDirFn(pathJoin(drivePath, minioMetaMultipartBucket, shaDir), func(uploadIDDir string, typ os.FileMode) error { readDirFn(pathJoin(drivePath, minioMetaMultipartBucket, shaDir), func(uploadIDDir string, typ os.FileMode) error {
uploadIDPath := pathJoin(shaDir, uploadIDDir) uploadIDPath := pathJoin(shaDir, uploadIDDir)
var modTime time.Time modTime := extractModTimeFromUploadIDDir(uploadIDDir)
// Upload IDs are of the form base64_url(<UUID>x<UnixNano>), we can extract the time from the UUID.
if b64, err := base64.RawURLEncoding.DecodeString(uploadIDDir); err == nil {
if split := strings.Split(string(b64), "x"); len(split) == 2 {
t, err := strconv.ParseInt(split[1], 10, 64)
if err == nil {
modTime = time.Unix(0, t)
}
}
}
// Fallback for older uploads without time in the ID. // Fallback for older uploads without time in the ID.
if modTime.IsZero() { if modTime.IsZero() {
wait := deleteMultipartCleanupSleeper.Timer(ctx) wait := deleteMultipartCleanupSleeper.Timer(ctx)
@@ -1696,6 +1705,32 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
defer lk.Unlock(lkctx) defer lk.Unlock(lkctx)
} }
if !opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement {
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
if fi.VersionID == "" || fi.VersionID == nullVersionID {
if err := checkObjectLockedForOverwrite(ctx, bucket, object, getObjectInfo); err != nil {
return oi, toObjectErr(err, bucket, object)
}
} else {
_, gerr := getObjectInfo(ctx, bucket, object, ObjectOptions{
VersionID: fi.VersionID,
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoLock: true,
})
switch {
case gerr == nil, errors.As(gerr, &MethodNotAllowed{}):
return oi, toObjectErr(errAuthentication, bucket, object)
case isErrVersionNotFound(gerr) || isErrObjectNotFound(gerr):
default:
return oi, toObjectErr(gerr, bucket, object)
}
}
}
er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename(). er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename().
defer func() { defer func() {
+28
View File
@@ -150,6 +150,17 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
} }
} }
if !dstOpts.ReplicationRequest && !dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement &&
(versionID == "" || versionID == nullVersionID) {
getObjectInfo := er.getObjectInfo
if dstOpts.replicaObjectInfo != nil {
getObjectInfo = dstOpts.replicaObjectInfo
}
if err := checkObjectLockedForOverwrite(ctx, dstBucket, dstObject, getObjectInfo); err != nil {
return oi, toObjectErr(err, dstBucket, dstObject)
}
}
modTime = UTCNow() // We only preserve modTime if dstOpts.MTime is true. modTime = UTCNow() // We only preserve modTime if dstOpts.MTime is true.
// in all other cases mtime is latest. // in all other cases mtime is latest.
@@ -1072,6 +1083,12 @@ func renameData(ctx context.Context, disks []StorageAPI, srcBucket, srcEntry str
errs := g.Wait() errs := g.Wait()
err := reduceWriteQuorumErrs(ctx, errs, objectOpIgnoredErrs, writeQuorum) err := reduceWriteQuorumErrs(ctx, errs, objectOpIgnoredErrs, writeQuorum)
if errors.Is(err, errCommitMetaMissing) {
// Commit metadata went missing on a quorum of drives; the write
// did not commit. Return a 503 class error so clients retry
// instead of treating the object as absent (404).
err = errErasureWriteQuorum
}
if err != nil { if err != nil {
dg := errgroup.WithNErrs(len(disks)) dg := errgroup.WithNErrs(len(disks))
for index, nerr := range errs { for index, nerr := range errs {
@@ -1594,6 +1611,17 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
defer lk.Unlock(lkctx) defer lk.Unlock(lkctx)
} }
if !opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement &&
(fi.VersionID == "" || fi.VersionID == nullVersionID) {
getObjectInfo := er.getObjectInfo
if opts.replicaObjectInfo != nil {
getObjectInfo = opts.replicaObjectInfo
}
if err := checkObjectLockedForOverwrite(ctx, bucket, object, getObjectInfo); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
}
// Rename the successfully written temporary object to final location. // Rename the successfully written temporary object to final location.
onlineDisks, versions, oldDataDir, err := renameData(ctx, onlineDisks, minioMetaTmpBucket, tempObj, partsMetadata, bucket, object, writeQuorum) onlineDisks, versions, oldDataDir, err := renameData(ctx, onlineDisks, minioMetaTmpBucket, tempObj, partsMetadata, bucket, object, writeQuorum)
if err != nil { if err != nil {
@@ -265,6 +265,7 @@ func TestPoolsMultipartConditionalLatestVersionAndCallbackOnce(t *testing.T) {
if explicit { if explicit {
current = putConsistencyObject(t, z, bucket, object, 1, "latest-other-version", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)}) current = putConsistencyObject(t, z, bucket, object, 1, "latest-other-version", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
} }
opts.VersionID = ""
id, parts := multipartConditionUpload(t, z, bucket, object, 1, opts) id, parts := multipartConditionUpload(t, z, bucket, object, 1, opts)
called := 0 called := 0
opts.CheckPrecondFn = func(oi ObjectInfo) bool { called++; return oi.ETag != current.ETag } opts.CheckPrecondFn = func(oi ObjectInfo) bool { called++; return oi.ETag != current.ETag }
@@ -633,6 +633,8 @@ func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
} }
} }
// A public client cannot address a destination version, with or without a
// precondition; only trusted replication may (SN-2026-016).
func TestPoolsConditionalPutDestinationVersionHTTP(t *testing.T) { func TestPoolsConditionalPutDestinationVersionHTTP(t *testing.T) {
z, _ := consistencyPools(t) z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned") bucket, router := conditionalPutBucket(t, z, "versioned")
@@ -641,21 +643,14 @@ func TestPoolsConditionalPutDestinationVersionHTTP(t *testing.T) {
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)}) current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)() defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object) + "?versionId=" + old.VersionID url := getPutObjectURL("", bucket, object) + "?versionId=" + old.VersionID
for _, stale := range []bool{true, false} { for _, tag := range []string{old.ETag, current.ETag} {
tag, status := current.ETag, http.StatusOK
if stale {
tag, status = old.ETag, http.StatusPreconditionFailed
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", tag)}) put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", tag)})
if put.Code != status { if put.Code != http.StatusForbidden {
t.Fatalf("version-addressed public PUT: %d want %d: %s", put.Code, status, put.Body.String()) t.Fatalf("version-addressed public PUT: %d want 403: %s", put.Code, put.Body.String())
}
if got := put.Header()[xhttp.AmzVersionID]; !stale && (len(got) != 1 || got[0] != old.VersionID) {
t.Fatalf("write version changed: %v", put.Header())
} }
} }
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil) get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "replacement" { if get.Code != http.StatusOK || get.Body.String() != "old" {
t.Fatalf("addressed GET: %d %q", get.Code, get.Body.String()) t.Fatalf("addressed GET: %d %q", get.Code, get.Body.String())
} }
} }
+11 -2
View File
@@ -1199,6 +1199,10 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
if opts.ReplicaLockReconcile || opts.DataMovement { if opts.ReplicaLockReconcile || opts.DataMovement {
opts.ReplicaLockReconcile = true opts.ReplicaLockReconcile = true
}
// Public overwrites need all pools for Object Lock. Unaddressed replica
// callbacks retain their set-local reads unless reconciliation is needed.
if opts.ReplicaLockReconcile || !opts.ReplicationRequest {
opts.replicaObjectInfo = z.replicaObjectInfo opts.replicaObjectInfo = z.replicaObjectInfo
} }
oi, err := z.serverPools[idx].PutObject(ctx, bucket, object, data, opts) oi, err := z.serverPools[idx].PutObject(ctx, bucket, object, data, opts)
@@ -1510,9 +1514,9 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
return objInfo, err return objInfo, err
} }
if !z.SinglePool() && dstOpts.ReplicaLockReconcile { if !z.SinglePool() && (dstOpts.ReplicaLockReconcile || !dstOpts.ReplicationRequest) {
dstOpts.replicaObjectInfo = z.replicaObjectInfo dstOpts.replicaObjectInfo = z.replicaObjectInfo
if !dstOpts.DataMovement { if dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement {
defer func() { defer func() {
if err == nil { if err == nil {
err = z.retireReplicaCopies(ctx, dstBucket, dstObject, poolIdx, objInfo) err = z.retireReplicaCopies(ctx, dstBucket, dstObject, poolIdx, objInfo)
@@ -1549,13 +1553,16 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
ServerSideEncryption: dstOpts.ServerSideEncryption, ServerSideEncryption: dstOpts.ServerSideEncryption,
UserDefined: srcInfo.UserDefined, UserDefined: srcInfo.UserDefined,
Versioned: dstOpts.Versioned, Versioned: dstOpts.Versioned,
VersionSuspended: dstOpts.VersionSuspended,
VersionID: dstOpts.VersionID, VersionID: dstOpts.VersionID,
MTime: dstOpts.MTime, MTime: dstOpts.MTime,
NoLock: true, NoLock: true,
EncryptFn: dstOpts.EncryptFn, EncryptFn: dstOpts.EncryptFn,
WantChecksum: dstOpts.WantChecksum, WantChecksum: dstOpts.WantChecksum,
WantServerSideChecksumType: dstOpts.WantServerSideChecksumType, WantServerSideChecksumType: dstOpts.WantServerSideChecksumType,
ReplicationRequest: dstOpts.ReplicationRequest,
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile, ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
DataMovement: dstOpts.DataMovement,
replicaObjectInfo: dstOpts.replicaObjectInfo, replicaObjectInfo: dstOpts.replicaObjectInfo,
} }
@@ -2305,6 +2312,8 @@ func (z *erasureServerPools) CompleteMultipartUpload(ctx context.Context, bucket
if opts.ReplicaLockReconcile || opts.DataMovement { if opts.ReplicaLockReconcile || opts.DataMovement {
opts.ReplicaLockReconcile = true opts.ReplicaLockReconcile = true
}
if opts.ReplicaLockReconcile || !opts.ReplicationRequest {
opts.replicaObjectInfo = z.replicaObjectInfo opts.replicaObjectInfo = z.replicaObjectInfo
} }
+6
View File
@@ -869,11 +869,17 @@ func (s *erasureSets) CopyObject(ctx context.Context, srcBucket, srcObject, dstB
ServerSideEncryption: dstOpts.ServerSideEncryption, ServerSideEncryption: dstOpts.ServerSideEncryption,
UserDefined: srcInfo.UserDefined, UserDefined: srcInfo.UserDefined,
Versioned: dstOpts.Versioned, Versioned: dstOpts.Versioned,
VersionSuspended: dstOpts.VersionSuspended,
VersionID: dstOpts.VersionID, VersionID: dstOpts.VersionID,
MTime: dstOpts.MTime, MTime: dstOpts.MTime,
NoLock: dstOpts.NoLock,
EncryptFn: dstOpts.EncryptFn, EncryptFn: dstOpts.EncryptFn,
WantChecksum: dstOpts.WantChecksum, WantChecksum: dstOpts.WantChecksum,
WantServerSideChecksumType: dstOpts.WantServerSideChecksumType, WantServerSideChecksumType: dstOpts.WantServerSideChecksumType,
ReplicationRequest: dstOpts.ReplicationRequest,
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
DataMovement: dstOpts.DataMovement,
replicaObjectInfo: dstOpts.replicaObjectInfo,
} }
return dstSet.putObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, putOpts) return dstSet.putObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, putOpts)
+2
View File
@@ -24,6 +24,8 @@ func TestIAMPeerDeleteNotificationReloadsCommittedState(t *testing.T) {
resetTestGlobals() resetTestGlobals()
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
defer cancel() defer cancel()
// Both caches use object storage, independent of earlier etcd suites.
globalEtcdClient = nil
obj, disk, err := prepareFS(ctx) obj, disk, err := prepareFS(ctx)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
+2 -2
View File
@@ -162,7 +162,7 @@ func toObjectErr(oerr error, params ...string) error {
apiErr.Type = v.Type apiErr.Type = v.Type
} }
return apiErr return apiErr
case errErasureWriteQuorum.Error(): case errErasureWriteQuorum.Error(), errCommitMetaMissing.Error():
apiErr := InsufficientWriteQuorum{} apiErr := InsufficientWriteQuorum{}
if len(params) >= 1 { if len(params) >= 1 {
apiErr.Bucket = params[0] apiErr.Bucket = params[0]
@@ -668,7 +668,7 @@ var (
func completeMultipartChecksumMismatch(algorithm string) error { func completeMultipartChecksumMismatch(algorithm string) error {
description := "The checksum you specified did not match the calculated checksum." description := "The checksum you specified did not match the calculated checksum."
if algorithm != "" { if algorithm != "" {
description = fmt.Sprintf("The %s checksum you specified did not match the calculated checksum.", algorithm) description = fmt.Sprintf("The %s you specified did not match the calculated checksum.", algorithm)
} }
return fmt.Errorf("%w: %s", errCompleteMultipartChecksumMismatch, description) return fmt.Errorf("%w: %s", errCompleteMultipartChecksumMismatch, description)
} }
+5
View File
@@ -347,6 +347,11 @@ func putOpts(ctx context.Context, bucket, object, vid string, hdrs http.Header,
versionSuspended := globalBucketVersioningSys.PrefixSuspended(bucket, object) versionSuspended := globalBucketVersioningSys.PrefixSuspended(bucket, object)
vid = strings.TrimSpace(vid) vid = strings.TrimSpace(vid)
// A caller-chosen version ID writes that version in place. Only trusted
// replication may recreate a source version under its original ID.
if vid != "" && !trustedReplication {
return opts, errAuthentication
}
if vid != "" && vid != nullVersionID { if vid != "" && vid != nullVersionID {
_, err := uuid.Parse(vid) _, err := uuid.Parse(vid)
if err != nil { if err != nil {
+9 -4
View File
@@ -62,12 +62,13 @@ func testAPIPutObjectChunkedChecksum(obj ObjectLayer, instanceType, bucketName s
// A well-formed 4-byte value that does not match the content. // A well-formed 4-byte value that does not match the content.
wrongCRC := base64.StdEncoding.EncodeToString([]byte{0x01, 0x02, 0x03, 0x04}) wrongCRC := base64.StdEncoding.EncodeToString([]byte{0x01, 0x02, 0x03, 0x04})
apiCode := func(rec *httptest.ResponseRecorder) string { apiErrOf := func(rec *httptest.ResponseRecorder) APIErrorResponse {
var apiErr APIErrorResponse var apiErr APIErrorResponse
b, _ := io.ReadAll(rec.Body) b, _ := io.ReadAll(rec.Body)
_ = xml.Unmarshal(b, &apiErr) _ = xml.Unmarshal(b, &apiErr)
return apiErr.Code return apiErr
} }
apiCode := func(rec *httptest.ResponseRecorder) string { return apiErrOf(rec).Code }
// newChunkedJavaForm builds a non-trailer signed chunked PutObject request that // newChunkedJavaForm builds a non-trailer signed chunked PutObject request that
// mirrors the AWS Java SDK v2 wire form: the checksum value sits in the header // mirrors the AWS Java SDK v2 wire form: the checksum value sits in the header
@@ -131,8 +132,12 @@ func testAPIPutObjectChunkedChecksum(obj ObjectLayer, instanceType, bucketName s
if rec.Code != http.StatusBadRequest { if rec.Code != http.StatusBadRequest {
t.Fatalf("%s: chunked+wrong CRC32: expected 400, got %d", instanceType, rec.Code) t.Fatalf("%s: chunked+wrong CRC32: expected 400, got %d", instanceType, rec.Code)
} }
if code := apiCode(rec); code != "XAmzContentChecksumMismatch" { gotErr := apiErrOf(rec)
t.Fatalf("%s: chunked+wrong CRC32: want XAmzContentChecksumMismatch, got %q", instanceType, code) if gotErr.Code != "XAmzContentChecksumMismatch" {
t.Fatalf("%s: chunked+wrong CRC32: want XAmzContentChecksumMismatch, got %q", instanceType, gotErr.Code)
}
if gotErr.Message != "The CRC32 you specified did not match the calculated checksum." {
t.Fatalf("%s: chunked+wrong CRC32: message = %q, want S3 algorithm-specific wording", instanceType, gotErr.Message)
} }
} }
} }
+18 -11
View File
@@ -3435,27 +3435,16 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
return return
} }
// Check permissions to perform this object retention operation
if s3Error := authenticateRequest(ctx, r, policy.PutObjectRetentionAction); s3Error != ErrNone { if s3Error := authenticateRequest(ctx, r, policy.PutObjectRetentionAction); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL) writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
return return
} }
if _, err := objectAPI.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if !validateLengthAndChecksum(r) { if !validateLengthAndChecksum(r) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL) writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL)
return return
} }
if rcfg, _ := globalBucketObjectLockSys.Get(bucket); !rcfg.LockEnabled {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidBucketObjectLockConfiguration), r.URL)
return
}
objRetention, err := objectlock.ParseObjectRetention(r.Body) objRetention, err := objectlock.ParseObjectRetention(r.Body)
if err != nil { if err != nil {
apiErr := errorCodes.ToAPIErr(ErrMalformedXML) apiErr := errorCodes.ToAPIErr(ErrMalformedXML)
@@ -3467,6 +3456,24 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
reqInfo := logger.GetReqInfo(ctx) reqInfo := logger.GetReqInfo(ctx)
reqInfo.SetTags("retention", objRetention.String()) reqInfo.SetTags("retention", objRetention.String())
// Check permissions to perform this object retention operation. The
// object-lock condition keys come from the request body, so this is
// decided before any bucket or object state is read.
if !isRetentionActionAllowed(policy.PutObjectRetentionAction, bucket, object, objRetention, r, reqInfo.Cred, reqInfo.Owner) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
return
}
if _, err := objectAPI.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if rcfg, _ := globalBucketObjectLockSys.Get(bucket); !rcfg.LockEnabled {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidBucketObjectLockConfiguration), r.URL)
return
}
opts, err := getOpts(ctx, r, bucket, object) opts, err := getOpts(ctx, r, bucket, object)
if err != nil { if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL) writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
+1 -1
View File
@@ -1166,7 +1166,7 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
return return
} }
if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, false); s3Err != ErrNone { if _, _, _, s3Err := checkPutObjectLockAllowed(ctx, r, bucket, object, objectAPI.GetObjectInfo, ErrNone, ErrNone, trustedReplication); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL) writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return return
} }
+679
View File
@@ -0,0 +1,679 @@
// Copyright (c) 2015-2026 MinIO, Inc.
// Copyright (c) 2026 PGSTY
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
package cmd
import (
"archive/tar"
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/amztime"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
xhttp "github.com/minio/minio/internal/http"
"github.com/pgsty/silo-pkg/v3/policy"
)
func retentionAuthzRequest(t *testing.T, router http.Handler, creds auth.Credentials, method, target string, body []byte, headers map[string]string) *httptest.ResponseRecorder {
t.Helper()
var rd io.ReadSeeker
if body != nil {
rd = bytes.NewReader(body)
}
req, err := newTestSignedRequestV4(method, target, int64(len(body)), rd, creds.AccessKey, creds.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
return rec
}
func retentionBody(mode string, until time.Time) []byte {
if mode == "" {
return []byte(`<Retention xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></Retention>`)
}
return fmt.Appendf(nil, `<Retention xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Mode>%s</Mode><RetainUntilDate>%s</RetainUntilDate></Retention>`,
mode, until.UTC().Format(time.RFC3339))
}
func putRetention(t *testing.T, router http.Handler, cred auth.Credentials, bucket, key, vid, mode string, until time.Time, bypass bool) int {
t.Helper()
body := retentionBody(mode, until)
sum := md5.Sum(body)
headers := map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])}
if bypass {
headers[objectlock.AmzObjectLockBypassRetGovernance] = "true"
}
rec := retentionAuthzRequest(t, router, cred, http.MethodPut,
makeTestTargetURL("", bucket, key, url.Values{"retention": {""}, "versionId": {vid}}), body, headers)
return rec.Code
}
func newRetentionPolicyUser(t *testing.T, instanceType, policyJSON string) auth.Credentials {
t.Helper()
ctx := t.Context()
accessKey, secretKey, err := auth.GenerateCredentials()
if err != nil {
t.Fatalf("%s: generate credentials: %v", instanceType, err)
}
creds := auth.Credentials{AccessKey: accessKey, SecretKey: secretKey}
if _, err = globalIAMSys.CreateUser(ctx, creds.AccessKey, madmin.AddOrUpdateUserReq{
SecretKey: creds.SecretKey,
Status: madmin.AccountEnabled,
}); err != nil {
t.Fatalf("%s: create user: %v", instanceType, err)
}
parsed, err := policy.ParseConfig(strings.NewReader(policyJSON))
if err != nil {
t.Fatalf("%s: parse policy: %v", instanceType, err)
}
policyName := "retention-authz-" + mustGetUUID()
if _, err = globalIAMSys.SetPolicy(ctx, policyName, *parsed); err != nil {
t.Fatalf("%s: install policy: %v", instanceType, err)
}
if _, err = globalIAMSys.PolicyDBSet(ctx, creds.AccessKey, policyName, regUser, false); err != nil {
t.Fatalf("%s: attach policy: %v", instanceType, err)
}
return creds
}
// TestAPIPutObjectRetentionAuthorization covers SN-2026-015: the governance
// bypass header is intent only; s3:PutObjectRetention is always required and
// weakening unexpired GOVERNANCE also requires s3:BypassGovernanceRetention.
func TestAPIPutObjectRetentionAuthorization(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPIPutObjectRetentionAuthorization,
})
}
func testAPIPutObjectRetentionAuthorization(obj ObjectLayer, instanceType, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
enableBucketObjectLock(t, bucket)
noRet := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:GetObject"`)
retOnly := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:PutObjectRetention"`)
retBypass := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:PutObjectRetention","s3:BypassGovernanceRetention"`)
bypassOnly := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:BypassGovernanceRetention"`)
condDenyShort := newRetentionPolicyUser(t, instanceType, `{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObjectRetention", "s3:BypassGovernanceRetention"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"]
},
{
"Effect": "Deny",
"Action": ["s3:PutObjectRetention"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"],
"Condition": {"NumericLessThan": {"s3:object-lock-remaining-retention-days": "60"}}
}
]
}`)
explicitDenyBypass := newRetentionPolicyUser(t, instanceType, `{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"]
},
{
"Effect": "Deny",
"Action": ["s3:BypassGovernanceRetention"],
"Resource": ["arn:aws:s3:::`+bucket+`/*"]
}
]
}`)
far := UTCNow().Add(365 * 24 * time.Hour).Truncate(time.Second)
nearer := UTCNow().Add(30 * 24 * time.Hour).Truncate(time.Second)
farther := far.Add(24 * time.Hour)
put := func(t *testing.T, object, mode string) string {
t.Helper()
payload := []byte("retention authorization")
opts := ObjectOptions{Versioned: true, UserDefined: map[string]string{}}
if mode != "" {
opts.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] = mode
opts.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(far)
}
info, err := obj.PutObject(t.Context(), bucket, object,
mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""), opts)
if err != nil {
t.Fatal(err)
}
return info.VersionID
}
stored := func(t *testing.T, object, vid string) objectlock.ObjectRetention {
t.Helper()
info, err := obj.GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: vid})
if err != nil {
t.Fatal(err)
}
return objectlock.GetObjectRetentionMeta(info.UserDefined)
}
for _, test := range []struct {
name string
creds auth.Credentials
existing string // stored retention mode on the version, until far
mode string // requested mode, "" removes retention
until time.Time
bypass bool
want int
changed bool
}{
{name: "no-permission removal", creds: noRet, existing: "GOVERNANCE", bypass: true, want: http.StatusForbidden},
{name: "no-permission mode change", creds: noRet, existing: "GOVERNANCE", mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "no-permission new compliance", creds: noRet, mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "retention-only shorten with header", creds: retOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "retention-only removal with header", creds: retOnly, existing: "GOVERNANCE", bypass: true, want: http.StatusForbidden},
{name: "retention-only mode change with header", creds: retOnly, existing: "GOVERNANCE", mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "retention-only shorten without header", creds: retOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, want: http.StatusBadRequest},
{name: "retention-only extend with header", creds: retOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: farther, bypass: true, want: http.StatusOK, changed: true},
{name: "retention-only new retention", creds: retOnly, mode: "GOVERNANCE", until: nearer, want: http.StatusOK, changed: true},
{name: "bypass shorten with header", creds: retBypass, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusOK, changed: true},
{name: "bypass removal with header", creds: retBypass, existing: "GOVERNANCE", bypass: true, want: http.StatusOK, changed: true},
{name: "bypass shorten without header", creds: retBypass, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, want: http.StatusBadRequest},
{name: "bypass shorten compliance", creds: retBypass, existing: "COMPLIANCE", mode: "COMPLIANCE", until: nearer, bypass: true, want: http.StatusBadRequest},
{name: "bypass-only extend", creds: bypassOnly, existing: "GOVERNANCE", mode: "GOVERNANCE", until: farther, bypass: true, want: http.StatusForbidden},
{name: "conditional-deny-put-retention with bypass header", creds: condDenyShort, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "conditional-allow-put-retention satisfied", creds: condDenyShort, existing: "GOVERNANCE", mode: "GOVERNANCE", until: farther, bypass: true, want: http.StatusOK, changed: true},
{name: "explicit-deny-bypass shorten with header", creds: explicitDenyBypass, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusForbidden},
{name: "owner shorten with header", creds: root, existing: "GOVERNANCE", mode: "GOVERNANCE", until: nearer, bypass: true, want: http.StatusOK, changed: true},
} {
t.Run(instanceType+"/"+test.name, func(t *testing.T) {
object := "retention-authz/" + strings.ReplaceAll(test.name, " ", "-")
vid := put(t, object, test.existing)
before := stored(t, object, vid)
body := retentionBody(test.mode, test.until)
sum := md5.Sum(body)
headers := map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])}
if test.bypass {
headers[objectlock.AmzObjectLockBypassRetGovernance] = "true"
}
target := makeTestTargetURL("", bucket, object, url.Values{"retention": {""}, "versionId": {vid}})
rec := retentionAuthzRequest(t, router, test.creds, http.MethodPut, target, body, headers)
if rec.Code != test.want {
t.Fatalf("status = %d, want %d: %s", rec.Code, test.want, rec.Body.String())
}
after := stored(t, object, vid)
if changed := after.Mode != before.Mode || !after.RetainUntilDate.Equal(before.RetainUntilDate.Time); changed != test.changed {
t.Fatalf("retention changed = %v, want %v (before %s, after %s)", changed, test.changed, before.String(), after.String())
}
})
}
t.Run(instanceType+"/no-permission missing bucket", func(t *testing.T) {
body := retentionBody("", time.Time{})
sum := md5.Sum(body)
target := makeTestTargetURL("", "retention-authz-missing", "object", url.Values{"retention": {""}})
rec := retentionAuthzRequest(t, router, noRet, http.MethodPut, target, body,
map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403 before bucket lookup: %s", rec.Code, rec.Body.String())
}
})
}
// TestAPIVersionAddressedWriteRequiresReplicationTrust covers SN-2026-016: a
// caller-chosen destination versionId is honored only for trusted replication,
// and write paths cannot overwrite locked versions via directory keys or
// version-addressed multipart completions.
func TestAPIVersionAddressedWriteRequiresReplicationTrust(t *testing.T) {
defer DetectTestLeak(t)()
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: testAPIVersionAddressedWriteRequiresReplicationTrust,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
})
}
func testAPIVersionAddressedWriteRequiresReplicationTrust(obj ObjectLayer, instanceType, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instanceType, bucket, `"s3:GetObject","s3:PutObject"`)
original := []byte("original version content")
info, err := obj.PutObject(t.Context(), bucket, "addressed",
mustGetPutObjReader(t, bytes.NewReader(original), int64(len(original)), "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
vid := info.VersionID
unchanged := func(t *testing.T) {
t.Helper()
got, err := obj.GetObjectInfo(t.Context(), bucket, "addressed", ObjectOptions{VersionID: vid})
if err != nil {
t.Fatal(err)
}
if got.Size != int64(len(original)) || got.ETag != info.ETag {
t.Fatalf("version %s was rewritten: size %d etag %s", vid, got.Size, got.ETag)
}
}
for _, creds := range []struct {
name string
creds auth.Credentials
}{{"writer", writer}, {"owner", root}} {
t.Run(instanceType+"/"+creds.name+"/put", func(t *testing.T) {
body := []byte("replacement")
target := makeTestTargetURL("", bucket, "addressed", url.Values{"versionId": {vid}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPut, target, body, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
})
t.Run(instanceType+"/"+creds.name+"/null-version", func(t *testing.T) {
body := []byte("replacement")
target := makeTestTargetURL("", bucket, "addressed", url.Values{"versionId": {nullVersionID}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPut, target, body, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
})
t.Run(instanceType+"/"+creds.name+"/copy", func(t *testing.T) {
target := makeTestTargetURL("", bucket, "addressed", url.Values{"versionId": {vid}})
req, err := newTestSignedRequestV4(http.MethodPut, target, 0, nil, creds.creds.AccessKey, creds.creds.SecretKey,
map[string]string{xhttp.AmzCopySource: "/" + bucket + "/addressed?versionId=" + vid, xhttp.AmzMetadataDirective: "REPLACE"})
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
})
t.Run(instanceType+"/"+creds.name+"/multipart", func(t *testing.T) {
target := makeTestTargetURL("", bucket, "addressed", url.Values{"uploads": {""}, "versionId": {vid}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPost, target, nil, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
})
t.Run(instanceType+"/"+creds.name+"/complete-multipart-existing-version", func(t *testing.T) {
// Simulate a pending upload persisted with an existing version ID
// (e.g. initiated pre-upgrade or by a replication peer).
up, err := obj.NewMultipartUpload(t.Context(), bucket, "addressed", ObjectOptions{
Versioned: true,
VersionID: vid,
})
if err != nil {
t.Fatal(err)
}
defer obj.AbortMultipartUpload(t.Context(), bucket, "addressed", up.UploadID, ObjectOptions{})
partPayload := []byte("replacement-part")
pi, err := obj.PutObjectPart(t.Context(), bucket, "addressed", up.UploadID, 1,
mustGetPutObjReader(t, bytes.NewReader(partPayload), int64(len(partPayload)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
completeBody := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"%s"</ETag></Part></CompleteMultipartUpload>`, pi.ETag)
target := makeTestTargetURL("", bucket, "addressed", url.Values{"uploadId": {up.UploadID}})
rec := retentionAuthzRequest(t, router, creds.creds, http.MethodPost, target, completeBody, nil)
if rec.Code != http.StatusForbidden {
t.Fatalf("status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
})
}
t.Run(instanceType+"/locked-directory-object", func(t *testing.T) {
far := UTCNow().Add(365 * 24 * time.Hour).Truncate(time.Second)
for _, lockCase := range []struct {
name string
meta map[string]string
}{
{
name: "governance",
meta: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "GOVERNANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(far),
},
},
{
name: "compliance",
meta: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "COMPLIANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(far),
},
},
{
name: "legal-hold",
meta: map[string]string{
strings.ToLower(xhttp.AmzObjectLockLegalHold): "ON",
},
},
} {
t.Run(lockCase.name, func(t *testing.T) {
dirKey := "locked-dir-" + lockCase.name + "/"
if _, err := obj.PutObject(t.Context(), bucket, dirKey,
mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""),
ObjectOptions{Versioned: true, VersionID: nullVersionID, UserDefined: lockCase.meta}); err != nil {
t.Fatal(err)
}
target := makeTestTargetURL("", bucket, dirKey, nil)
rec := retentionAuthzRequest(t, router, writer, http.MethodPut, target, nil, nil)
if rec.Code != http.StatusBadRequest {
t.Fatalf("PUT locked dir status = %d, want 400: %s", rec.Code, rec.Body.String())
}
got, err := obj.GetObjectInfo(t.Context(), bucket, dirKey, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
for k, wantVal := range lockCase.meta {
if got.UserDefined[k] != wantVal {
t.Fatalf("metadata %s = %q, want %q", k, got.UserDefined[k], wantVal)
}
}
})
}
})
t.Run(instanceType+"/snowball-archive-version-id", func(t *testing.T) {
var tarBuf bytes.Buffer
tw := tar.NewWriter(&tarBuf)
entryData := []byte("snowball-overwrite")
if err := tw.WriteHeader(&tar.Header{
Typeflag: tar.TypeReg,
Name: "addressed",
Size: int64(len(entryData)),
Format: tar.FormatPAX,
PAXRecords: map[string]string{"minio.versionId": vid},
}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(entryData); err != nil {
t.Fatal(err)
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
// Untrusted writer cannot overwrite an existing version via PAX minio.versionId.
target := makeTestTargetURL("", bucket, "archive.tar", nil)
rec := retentionAuthzRequest(t, router, writer, http.MethodPut, target, tarBuf.Bytes(),
map[string]string{"x-amz-meta-snowball-auto-extract": "true"})
if rec.Code != http.StatusForbidden {
t.Fatalf("untrusted snowball status = %d, want 403: %s", rec.Code, rec.Body.String())
}
unchanged(t)
// Trusted batch replication (with outer MinIOSourceReplicationRequest) preserves PAX minio.versionId.
replicaVID := mustGetUUID()
var repBuf bytes.Buffer
repTW := tar.NewWriter(&repBuf)
repData := []byte("replicated-via-snowball")
if err := repTW.WriteHeader(&tar.Header{
Typeflag: tar.TypeReg,
Name: "snowball-replicated",
Size: int64(len(repData)),
Format: tar.FormatPAX,
PAXRecords: map[string]string{"minio.versionId": replicaVID},
}); err != nil {
t.Fatal(err)
}
if _, err := repTW.Write(repData); err != nil {
t.Fatal(err)
}
if err := repTW.Close(); err != nil {
t.Fatal(err)
}
repRec := retentionAuthzRequest(t, router, root, http.MethodPut, target, repBuf.Bytes(),
map[string]string{
"x-amz-meta-snowball-auto-extract": "true",
xhttp.MinIOSourceReplicationRequest: "true",
})
if repRec.Code != http.StatusOK {
t.Fatalf("trusted snowball status = %d, want 200: %s", repRec.Code, repRec.Body.String())
}
if _, err := obj.GetObjectInfo(t.Context(), bucket, "snowball-replicated", ObjectOptions{VersionID: replicaVID}); err != nil {
t.Fatalf("trusted snowball replication did not preserve version ID: %v", err)
}
})
t.Run(instanceType+"/trusted replication", func(t *testing.T) {
replicaVID := mustGetUUID()
body := []byte("replicated version")
target := makeTestTargetURL("", bucket, "replicated", url.Values{"versionId": {replicaVID}})
rec := retentionAuthzRequest(t, router, root, http.MethodPut, target, body,
map[string]string{xhttp.MinIOSourceReplicationRequest: "true"})
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200: %s", rec.Code, rec.Body.String())
}
if _, err := obj.GetObjectInfo(t.Context(), bucket, "replicated", ObjectOptions{VersionID: replicaVID}); err != nil {
t.Fatalf("trusted replication did not keep its version ID: %v", err)
}
})
}
func TestPutObjectRetentionClearMinimumDaysPolicy(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
for _, conditionalAllow := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/conditional-allow-%v", instance, conditionalAllow), func(t *testing.T) {
statement := `{"Effect":"Allow","Action":["s3:PutObjectRetention","s3:BypassGovernanceRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"]}, {"Effect":"Deny","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"],"Condition":{"NumericLessThan":{"s3:object-lock-remaining-retention-days":"60"}}}`
if conditionalAllow {
statement = `{"Effect":"Allow","Action":["s3:BypassGovernanceRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"]}, {"Effect":"Allow","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::` + bucket + `/*"],"Condition":{"NumericGreaterThanEquals":{"s3:object-lock-remaining-retention-days":"60"}}}`
}
cred := newRetentionPolicyUser(t, instance, `{"Version":"2012-10-17","Statement":[`+statement+`]}`)
key := fmt.Sprintf("review-minimum-%v", conditionalAllow)
far := UTCNow().Add(365 * 24 * time.Hour).Truncate(time.Second)
info, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""), ObjectOptions{
Versioned: true,
UserDefined: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "GOVERNANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(far),
},
})
if err != nil {
t.Fatal(err)
}
if status := putRetention(t, router, cred, bucket, key, info.VersionID, "GOVERNANCE", UTCNow().Add(30*24*time.Hour), true); status != http.StatusForbidden {
t.Fatalf("shortening to 30 days status = %d, want 403", status)
}
status := putRetention(t, router, cred, bucket, key, info.VersionID, "", time.Time{}, true)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: info.VersionID})
if err != nil {
t.Fatal(err)
}
ret := objectlock.GetObjectRetentionMeta(after.UserDefined)
if status != http.StatusForbidden || ret.Mode != objectlock.RetGovernance {
t.Fatalf("clearing retention status = %d, stored retention = %s; want 403 and original GOVERNANCE", status, ret.String())
}
})
}
},
})
}
type hookBeforeWrite struct {
ObjectLayer
key string
before func()
}
func (p *hookBeforeWrite) fire(key string) {
if key == p.key && p.before != nil {
f := p.before
p.before = nil
f()
}
}
func (p *hookBeforeWrite) PutObject(ctx context.Context, bucket, key string, r *PutObjReader, opts ObjectOptions) (ObjectInfo, error) {
p.fire(key)
return p.ObjectLayer.PutObject(ctx, bucket, key, r, opts)
}
func (p *hookBeforeWrite) CopyObject(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject string, srcInfo ObjectInfo, srcOpts, dstOpts ObjectOptions) (ObjectInfo, error) {
p.fire(dstObject)
return p.ObjectLayer.CopyObject(ctx, srcBucket, srcObject, dstBucket, dstObject, srcInfo, srcOpts, dstOpts)
}
func (p *hookBeforeWrite) CompleteMultipartUpload(ctx context.Context, bucket, key, uploadID string, parts []CompletePart, opts ObjectOptions) (ObjectInfo, error) {
p.fire(key)
return p.ObjectLayer.CompleteMultipartUpload(ctx, bucket, key, uploadID, parts, opts)
}
func TestDirectoryObjectLockAfterPreflight(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:GetObject","s3:PutObject"`)
srcDir := "other-dir/"
if rec := retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, srcDir, nil), nil, nil); rec.Code != http.StatusOK {
t.Fatalf("srcDir PUT: %d %s", rec.Code, rec.Body.String())
}
for _, mode := range []string{"GOVERNANCE", "COMPLIANCE", "LEGAL-HOLD"} {
for _, op := range []string{"PutObject", "CopyObject", "SelfCopy"} {
t.Run(instance+"/"+mode+"/"+op, func(t *testing.T) {
key := fmt.Sprintf("review-race-%s-%s/", mode, op)
if rec := retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, nil); rec.Code != http.StatusOK {
t.Fatalf("initial PUT: %d %s", rec.Code, rec.Body.String())
}
var locked ObjectInfo
hook := &hookBeforeWrite{ObjectLayer: obj, key: key}
hook.before = func() {
if mode == "LEGAL-HOLD" {
body := []byte(`<LegalHold><Status>ON</Status></LegalHold>`)
sum := md5.Sum(body)
rec := retentionAuthzRequest(t, router, root, http.MethodPut, makeTestTargetURL("", bucket, key, url.Values{"legal-hold": {""}, "versionId": {nullVersionID}}), body, map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
if rec.Code != http.StatusOK {
t.Fatalf("set legal hold: %d %s", rec.Code, rec.Body.String())
}
} else if status := putRetention(t, router, root, bucket, key, nullVersionID, mode, UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("set retention: %d", status)
}
var err error
locked, err = obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
}
setObjectLayer(hook)
defer setObjectLayer(obj)
var rec *httptest.ResponseRecorder
switch op {
case "PutObject":
rec = retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, nil)
case "CopyObject":
rec = retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, map[string]string{
xhttp.AmzCopySource: pathJoin(bucket, srcDir),
})
case "SelfCopy":
rec = retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, map[string]string{
xhttp.AmzCopySource: pathJoin(bucket, key), xhttp.AmzMetadataDirective: "REPLACE", "x-amz-meta-updated": "true",
})
}
if hook.before != nil {
t.Fatalf("%s never reached backend: %d %s", op, rec.Code, rec.Body.String())
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusBadRequest {
t.Fatalf("locked null write via %s: %d %s", op, rec.Code, rec.Body.String())
}
multipartConditionError(t, rec, "InvalidRequest")
retentionOverwriteUnchanged(t, locked, after)
})
}
}
},
})
}
func TestMultipartVersioningTransitions(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
t.Run(instance+"/unversioned-to-enabled", func(t *testing.T) {
key := "review-upload-before-versioning"
initRec := retentionAuthzRequest(t, router, root, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploads": {""}}), nil, nil)
if initRec.Code != http.StatusOK {
t.Fatalf("init: %d %s", initRec.Code, initRec.Body.String())
}
var mp InitiateMultipartUploadResponse
if err := xml.Unmarshal(initRec.Body.Bytes(), &mp); err != nil {
t.Fatal(err)
}
part := retentionAuthzRequest(t, router, root, http.MethodPut, getPutObjectPartURL("", bucket, key, mp.UploadID, "1"), []byte("ordinary pending upload"), nil)
if part.Code != http.StatusOK {
t.Fatalf("part: %d %s", part.Code, part.Body.String())
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, []byte(`<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
body := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>%s</ETag></Part></CompleteMultipartUpload>`, multipartConditionResponseETag(part))
rec := retentionAuthzRequest(t, router, root, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {mp.UploadID}}), body, nil)
if rec.Code != http.StatusOK {
t.Fatalf("ordinary upload initiated before versioning fails completion: %d %s", rec.Code, rec.Body.String())
}
})
t.Run(instance+"/addressed-upload-after-suspension", func(t *testing.T) {
key := "review-addressed-before-suspension"
original := []byte("original")
info, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(original), int64(len(original)), "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
up, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{Versioned: true, VersionID: info.VersionID})
if err != nil {
t.Fatal(err)
}
part := []byte("replaced existing version")
pi, err := obj.PutObjectPart(t.Context(), bucket, key, up.UploadID, 1, mustGetPutObjReader(t, bytes.NewReader(part), int64(len(part)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig, []byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
body := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"%s"</ETag></Part></CompleteMultipartUpload>`, pi.ETag)
rec := retentionAuthzRequest(t, router, writer, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}}), body, nil)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: info.VersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusForbidden || after.ETag != info.ETag {
t.Fatalf("suspended-bucket existing UUID overwritten: status=%d old-etag=%s new-etag=%s body=%s", rec.Code, info.ETag, after.ETag, rec.Body.String())
}
})
},
})
}
+546
View File
@@ -0,0 +1,546 @@
// Copyright (c) 2026 Feng Ruohang
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This file is part of Silo Object Storage stack.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"time"
"github.com/minio/minio/internal/amztime"
"github.com/minio/minio/internal/auth"
objectlock "github.com/minio/minio/internal/bucket/object/lock"
xhttp "github.com/minio/minio/internal/http"
)
func retentionOverwriteUnchanged(t *testing.T, before, after ObjectInfo) {
t.Helper()
if after.ETag != before.ETag || after.Size != before.Size || after.VersionID != before.VersionID || !after.ModTime.Equal(before.ModTime) {
t.Error("rejected write changed the stored object")
}
if objectlock.GetObjectRetentionMeta(after.UserDefined).String() != objectlock.GetObjectRetentionMeta(before.UserDefined).String() || objectlock.GetObjectLegalHoldMeta(after.UserDefined).Status != objectlock.GetObjectLegalHoldMeta(before.UserDefined).Status {
t.Error("rejected write changed the stored lock")
}
}
func TestObjectLockOverwriteConfigUnavailable(t *testing.T) {
previous := newObjectLayerFn()
setObjectLayer(nil)
defer setObjectLayer(previous)
err := checkObjectLockedForOverwrite(t.Context(), "lock-config-unavailable", "object", func(context.Context, string, string, ObjectOptions) (ObjectInfo, error) {
t.Fatal("object must not be read without a bucket configuration")
return ObjectInfo{}, nil
})
if !errors.Is(err, errServerNotInitialized) {
t.Fatalf("unavailable lock configuration must fail closed, got %v", err)
}
}
func TestPutObjectRetentionMaximumDaysPolicy(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, _ auth.Credentials, t *testing.T) {
for _, conditionalAllow := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/conditional-allow=%t", instance, conditionalAllow), func(t *testing.T) {
statement := fmt.Sprintf(`{"Effect":"Allow","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::%s/*"],"Condition":{"NumericLessThanEquals":{"s3:object-lock-remaining-retention-days":"200000"}}}`, bucket)
if !conditionalAllow {
statement = fmt.Sprintf(`{"Effect":"Allow","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::%s/*"]},{"Effect":"Deny","Action":["s3:PutObjectRetention"],"Resource":["arn:aws:s3:::%s/*"],"Condition":{"NumericGreaterThan":{"s3:object-lock-remaining-retention-days":"200000"}}}`, bucket, bucket)
}
cred := newRetentionPolicyUser(t, instance, `{"Version":"2012-10-17","Statement":[`+statement+`]}`)
key := fmt.Sprintf("maximum-retention-%t", conditionalAllow)
initial, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(nil), 0, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
// Both dates exceed time.Duration's range (~292 years). A saturated
// Time.Sub would report 106752 days and incorrectly allow 1000 years.
if status := putRetention(t, router, cred, bucket, key, initial.VersionID, "GOVERNANCE", UTCNow().AddDate(1000, 0, 0), false); status != http.StatusForbidden {
t.Fatalf("1000-year retention: want 403, got %d", status)
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
retentionOverwriteUnchanged(t, initial, after)
if status := putRetention(t, router, cred, bucket, key, initial.VersionID, "GOVERNANCE", UTCNow().AddDate(500, 0, 0), false); status != http.StatusOK {
t.Fatalf("500-year retention: want 200, got %d", status)
}
})
}
},
})
}
func TestAPICopyObjectPreservesVersionAfterSuspension(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
t.Run(instance, func(t *testing.T) {
key := "self-copy-after-suspension"
original, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("payload"), 7, "", ""), ObjectOptions{
Versioned: true, UserDefined: map[string]string{"x-amz-meta-state": "original"},
})
if err != nil {
t.Fatal(err)
}
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig,
[]byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
rec := retentionAuthzRequest(t, router, root, http.MethodPut, makeTestTargetURL("", bucket, key, nil), nil, map[string]string{
xhttp.AmzCopySource: "/" + bucket + "/" + key, xhttp.AmzMetadataDirective: "REPLACE", "x-amz-meta-state": "replacement",
})
if rec.Code != http.StatusOK {
t.Fatalf("copy: %d %s", rec.Code, rec.Body.String())
}
old, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: original.VersionID})
if err != nil {
t.Fatal(err)
}
if old.UserDefined["x-amz-meta-state"] != "original" {
t.Errorf("copy changed metadata of existing UUID %s: %v", original.VersionID, old.UserDefined)
}
current, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatalf("copy must create the null version: %v", err)
}
if current.UserDefined["X-Amz-Meta-State"] != "replacement" {
t.Errorf("new null version metadata: %v", current.UserDefined)
}
})
},
})
}
// A persisted null upload can survive an upgrade or a versioning transition.
// The completion must see locks committed after the handler's preflight.
func TestMultipartNullObjectLockAfterPreflight(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
for _, mode := range []string{"GOVERNANCE", "COMPLIANCE", "LEGAL-HOLD"} {
t.Run(instance+"/"+mode, func(t *testing.T) {
ctx := t.Context()
key := "multipart-null-" + mode
opts := ObjectOptions{Versioned: true, VersionID: nullVersionID, ReplicationRequest: true}
if _, err := obj.PutObject(ctx, bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), opts); err != nil {
t.Fatal(err)
}
mp, err := obj.NewMultipartUpload(ctx, bucket, key, opts)
if err != nil {
t.Fatal(err)
}
part, err := obj.PutObjectPart(ctx, bucket, key, mp.UploadID, 1, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
var locked ObjectInfo
hook := &hookBeforeWrite{ObjectLayer: obj, key: key, before: func() {
if mode == "LEGAL-HOLD" {
body := []byte(`<LegalHold><Status>ON</Status></LegalHold>`)
sum := md5.Sum(body)
rec := retentionAuthzRequest(t, router, root, http.MethodPut, makeTestTargetURL("", bucket, key, url.Values{"legal-hold": {""}, "versionId": {nullVersionID}}), body, map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
if rec.Code != http.StatusOK {
t.Fatalf("set legal hold: %d %s", rec.Code, rec.Body.String())
}
} else if status := putRetention(t, router, root, bucket, key, nullVersionID, mode, UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("set retention: %d", status)
}
var err error
locked, err = obj.GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
}}
setObjectLayer(hook)
defer setObjectLayer(obj)
body := []byte(multipartConditionCompleteBody([]CompletePart{{PartNumber: 1, ETag: part.ETag}}))
rec := retentionAuthzRequest(t, router, writer, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {mp.UploadID}}), body, nil)
if hook.before != nil || rec.Code != http.StatusBadRequest {
t.Fatalf("completion after locking: %d %s", rec.Code, rec.Body.String())
}
multipartConditionError(t, rec, "InvalidRequest")
after, err := obj.GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
retentionOverwriteUnchanged(t, locked, after)
parts, err := obj.ListObjectParts(ctx, bucket, key, mp.UploadID, 0, 100, ObjectOptions{})
if err != nil || len(parts.Parts) != 1 || parts.Parts[0].ETag != part.ETag {
t.Fatalf("rejected completion removed upload parts: %v %+v", err, parts.Parts)
}
})
}
},
})
}
// The newer, unlocked copy is in the write pool; the lock exists only in the
// other pool. A set-local check would miss it even while holding a write lock.
func TestPoolsObjectLockOverwrite(t *testing.T) {
z, _ := consistencyPools(t)
bucket, _, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: true, LockEnabled: true})
if err != nil {
t.Fatal(err)
}
for _, state := range []string{"GOVERNANCE", "COMPLIANCE", "LEGAL-HOLD", "expired", "unlocked", "unreadable"} {
for _, op := range []string{"put", "copy", "self-copy", "multipart"} {
t.Run(state+"/"+op, func(t *testing.T) {
ctx := t.Context()
key := encodeDirObject("null-" + state + "-" + op + "/")
meta := map[string]string{}
switch state {
case "GOVERNANCE", "COMPLIANCE":
meta[strings.ToLower(xhttp.AmzObjectLockMode)] = state
meta[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(UTCNow().Add(365 * 24 * time.Hour))
case "LEGAL-HOLD":
meta[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = "ON"
case "expired":
meta[strings.ToLower(xhttp.AmzObjectLockMode)] = "GOVERNANCE"
meta[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = amztime.ISO8601Format(UTCNow().Add(-24 * time.Hour))
}
seed := ObjectOptions{Versioned: true, VersionID: nullVersionID, ReplicationRequest: true, MTime: UTCNow().Add(-time.Hour), UserDefined: meta}
putConsistencyObject(t, z, bucket, key, 1, "original", seed)
seed.UserDefined = map[string]string{}
seed.MTime = UTCNow()
putConsistencyObject(t, z, bucket, key, 0, "original", seed)
before := make([]ObjectInfo, len(z.serverPools))
for i, pool := range z.serverPools {
var err error
before[i], err = pool.GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
}
defer conditionalPutPool(t, z, key, 0)()
var restore func()
if state == "unreadable" {
restore = conditionalPutSwapDisks(z.serverPools[1], key, func(d StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: d, bucket: bucket, object: key}
})
defer restore()
}
opts := ObjectOptions{Versioned: true, VersionID: nullVersionID, UserDefined: map[string]string{"x-amz-meta-updated": "true"}}
var writeErr error
switch op {
case "put":
_, writeErr = z.PutObject(ctx, bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("replacement"), 11, "", ""), opts)
case "copy", "self-copy":
source := key
if op == "copy" {
source = "source-" + state
putConsistencyObject(t, z, bucket, source, 0, "replacement", ObjectOptions{Versioned: true})
}
gr, gerr := z.GetObjectNInfo(ctx, bucket, source, nil, nil, ObjectOptions{})
if gerr != nil {
t.Fatal(gerr)
}
defer gr.Close()
info := gr.ObjInfo
info.UserDefined = opts.UserDefined
info.metadataOnly = op == "self-copy"
info.PutObjReader = mustGetPutObjReader(t, gr, info.Size, "", "")
info.Reader = info.PutObjReader.Reader
_, writeErr = z.CopyObject(ctx, bucket, source, bucket, key, info, ObjectOptions{VersionID: nullVersionID}, opts)
case "multipart":
id, parts := multipartConditionUpload(t, z, bucket, key, 0, opts)
_, writeErr = z.CompleteMultipartUpload(ctx, bucket, key, id, parts, ObjectOptions{Versioned: true})
}
allowed := state == "expired" || state == "unlocked"
if allowed {
if writeErr != nil {
t.Fatalf("unlocked write: %v", writeErr)
}
return
}
if state == "unreadable" {
if !isErrReadQuorum(writeErr) {
t.Fatalf("unreadable pool: want quorum error, got %v", writeErr)
}
restore()
} else if !errors.As(writeErr, &ObjectLocked{}) {
t.Fatalf("lock in other pool: want ObjectLocked, got %v", writeErr)
}
for i, original := range before {
after, gerr := z.serverPools[i].GetObjectInfo(ctx, bucket, key, ObjectOptions{VersionID: nullVersionID})
if gerr != nil {
t.Fatal(gerr)
}
retentionOverwriteUnchanged(t, original, after)
}
})
}
}
}
type planReviewGatedBody struct {
io.ReadCloser
first sync.Once
ready chan struct{}
release chan struct{}
}
func (g *planReviewGatedBody) Read(p []byte) (int, error) {
g.first.Do(func() { close(g.ready); <-g.release })
return g.ReadCloser.Read(p)
}
func TestPlanReviewSlowDirectoryPut(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
key := "slow-body-directory/"
before, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), ObjectOptions{Versioned: true, VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
payload := []byte("replacement-body")
req, err := newTestSignedRequestV4(http.MethodPut, makeTestTargetURL("", bucket, key, nil), int64(len(payload)), bytes.NewReader(payload), writer.AccessKey, writer.SecretKey, nil)
if err != nil {
t.Fatal(err)
}
gate := &planReviewGatedBody{ReadCloser: req.Body, ready: make(chan struct{}), release: make(chan struct{})}
req.Body = gate
var once sync.Once
release := func() { once.Do(func() { close(gate.release) }) }
defer release()
rec := httptest.NewRecorder()
done := make(chan struct{})
go func() { router.ServeHTTP(rec, req); close(done) }()
select {
case <-gate.ready:
case <-done:
t.Fatalf("PUT finished before body: %d %s", rec.Code, rec.Body.String())
case <-time.After(5 * time.Second):
t.Fatal("PUT did not reach body read")
}
locked := make(chan int, 1)
go func() {
locked <- putRetention(t, router, root, bucket, key, nullVersionID, "COMPLIANCE", UTCNow().Add(365*24*time.Hour), false)
}()
select {
case status := <-locked:
if status != http.StatusOK {
t.Fatalf("set lock while body paused: %d", status)
}
case <-time.After(5 * time.Second):
t.Fatal("retention update blocked behind paused body")
}
release()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("PUT did not finish")
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusBadRequest || after.ETag != before.ETag || objectlock.GetObjectRetentionMeta(after.UserDefined).Mode != objectlock.RetCompliance {
t.Fatal("paused ordinary PUT overwrote the COMPLIANCE null version after the lock was acknowledged")
}
})
},
})
}
type planReviewReadFault struct {
ObjectLayer
key string
fired bool
putCalled bool
}
func (p *planReviewReadFault) GetObjectInfo(ctx context.Context, bucket, key string, opts ObjectOptions) (ObjectInfo, error) {
if key == p.key && opts.VersionID == nullVersionID && !p.fired {
p.fired = true
return ObjectInfo{}, InsufficientReadQuorum{}
}
return p.ObjectLayer.GetObjectInfo(ctx, bucket, key, opts)
}
func (p *planReviewReadFault) PutObject(ctx context.Context, bucket, key string, data *PutObjReader, opts ObjectOptions) (ObjectInfo, error) {
if key == p.key {
p.putCalled = true
}
return p.ObjectLayer.PutObject(ctx, bucket, key, data, opts)
}
func TestPlanReviewPreflightReadFailure(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, _ auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
key := "locked-directory-read-fault/"
_, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), ObjectOptions{
Versioned: true, VersionID: nullVersionID, ReplicationRequest: true,
UserDefined: map[string]string{
strings.ToLower(xhttp.AmzObjectLockMode): "COMPLIANCE",
strings.ToLower(xhttp.AmzObjectLockRetainUntilDate): amztime.ISO8601Format(UTCNow().Add(365 * 24 * time.Hour)),
},
})
if err != nil {
t.Fatal(err)
}
initial, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
fault := &planReviewReadFault{ObjectLayer: obj, key: key}
setObjectLayer(fault)
defer setObjectLayer(obj)
rec := retentionAuthzRequest(t, router, writer, http.MethodPut, makeTestTargetURL("", bucket, key, nil), []byte("replacement"), nil)
if !fault.fired {
t.Fatal("preflight fault was not injected")
}
if fault.putCalled {
t.Fatal("preflight read error allowed PutObject to proceed to storage layer")
}
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: nullVersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code < 400 {
t.Errorf("metadata read quorum error was treated as absence: status=%d", rec.Code)
}
retentionOverwriteUnchanged(t, initial, after)
})
},
})
}
func TestPlanReviewLegacyMPUCompliance(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
key := "legacy-mpu-compliance"
initial, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewBufferString("original"), 8, "", ""), ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
up, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{Versioned: true, VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
payload := []byte("replacement-part")
part, err := obj.PutObjectPart(t.Context(), bucket, key, up.UploadID, 1, mustGetPutObjReader(t, bytes.NewReader(payload), int64(len(payload)), "", ""), ObjectOptions{})
if err != nil {
t.Fatal(err)
}
if status := putRetention(t, router, root, bucket, key, initial.VersionID, "COMPLIANCE", UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("set compliance: %d", status)
}
before, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
body := fmt.Appendf(nil, `<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"%s"</ETag></Part></CompleteMultipartUpload>`, part.ETag)
rec := retentionAuthzRequest(t, router, writer, http.MethodPost, makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}}), body, nil)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: initial.VersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusForbidden {
t.Errorf("legacy upload replaced locked UUID: status=%d", rec.Code)
}
retentionOverwriteUnchanged(t, before, after)
})
},
})
}
// A recovered upload retains an authentic server-generated version ID while a
// surviving replica reintroduces its already-completed version and lock.
// Only the fixture uses replication privileges; completion uses PutObject only.
func TestRecoveredMultipartUploadAfterReplica(t *testing.T) {
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
t: t,
makeBucketOptions: MakeBucketOptions{VersioningEnabled: true, LockEnabled: true},
objAPITest: func(obj ObjectLayer, instance, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
writer := newObjectAttributesAuthzUser(t, instance, bucket, `"s3:PutObject"`)
t.Run(instance, func(t *testing.T) {
z := obj.(*erasureServerPools)
key := "recovered-generated-upload"
up, err := obj.NewMultipartUpload(t.Context(), bucket, key, ObjectOptions{Versioned: true})
if err != nil {
t.Fatal(err)
}
er := z.serverPools[0].getHashedSet(key)
fi, _, err := er.checkUploadIDExists(t.Context(), bucket, key, up.UploadID, true)
if err != nil {
t.Fatal(err)
}
_, err = obj.PutObject(t.Context(), bucket, key,
mustGetPutObjReader(t, bytes.NewBufferString("protected-original"), 18, "", ""),
ObjectOptions{Versioned: true, VersionID: fi.VersionID, ReplicationRequest: true})
if err != nil {
t.Fatal(err)
}
if status := putRetention(t, router, root, bucket, key, fi.VersionID, "COMPLIANCE", UTCNow().Add(365*24*time.Hour), false); status != http.StatusOK {
t.Fatalf("setting compliance lock: %d", status)
}
before, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: fi.VersionID})
if err != nil {
t.Fatal(err)
}
partRec := retentionAuthzRequest(t, router, writer, http.MethodPut,
makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}, "partNumber": {"1"}}),
[]byte("attacker-replacement"), nil)
if partRec.Code != http.StatusOK {
t.Fatalf("upload part: %d %s", partRec.Code, partRec.Body.String())
}
etag := multipartConditionResponseETag(partRec)
body := []byte(multipartConditionCompleteBody([]CompletePart{{PartNumber: 1, ETag: etag}}))
rec := retentionAuthzRequest(t, router, writer, http.MethodPost,
makeTestTargetURL("", bucket, key, url.Values{"uploadId": {up.UploadID}}), body, nil)
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: fi.VersionID})
if err != nil {
t.Fatal(err)
}
if rec.Code != http.StatusForbidden {
t.Errorf("recovered upload bypassed existing UUID guard: status=%d", rec.Code)
}
retentionOverwriteUnchanged(t, before, after)
})
},
})
}
+1 -1
View File
@@ -238,7 +238,7 @@ func r5APIOrdering(t *testing.T, encrypted bool) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
original, err = obj.CompleteMultipartUpload(t.Context(), bucket, name, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{Versioned: true}) original, err = obj.CompleteMultipartUpload(t.Context(), bucket, name, mp.UploadID, []CompletePart{{PartNumber: 1, ETag: part.ETag}}, ObjectOptions{Versioned: true, ReplicationRequest: true})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+5
View File
@@ -70,6 +70,11 @@ var errDiskAccessDenied = StorageErr("drive access denied")
// errFileNotFound - cannot find the file. // errFileNotFound - cannot find the file.
var errFileNotFound = StorageErr("file not found") var errFileNotFound = StorageErr("file not found")
// errCommitMetaMissing - the staged xl.meta disappeared between the
// data rename and the meta-file commit; a write-quorum failure, not
// a missing object.
var errCommitMetaMissing = StorageErr("commit metadata is missing")
// errFileNotFound - cannot find requested file version. // errFileNotFound - cannot find requested file version.
var errFileVersionNotFound = StorageErr("file version not found") var errFileVersionNotFound = StorageErr("file version not found")
+2
View File
@@ -106,6 +106,8 @@ func toStorageErr(err error) error {
return errVolumeExists return errVolumeExists
case errFileNotFound.Error(): case errFileNotFound.Error():
return errFileNotFound return errFileNotFound
case errCommitMetaMissing.Error():
return errCommitMetaMissing
case errFileVersionNotFound.Error(): case errFileVersionNotFound.Error():
return errFileVersionNotFound return errFileVersionNotFound
case errFileNameTooLong.Error(): case errFileNameTooLong.Error():
+4
View File
@@ -290,6 +290,9 @@ func (s *storageRESTServer) NSScannerHandler(ctx context.Context, params *nsScan
for update := range updates { for update := range updates {
resp := storageNSScannerRPC.NewResponse() resp := storageNSScannerRPC.NewResponse()
resp.Update = &update resp.Update = &update
// Pooled responses may carry fields from a previous
// scan, always clear the fields not being set.
resp.Final = nil
out <- resp out <- resp
} }
}() }()
@@ -301,6 +304,7 @@ func (s *storageRESTServer) NSScannerHandler(ctx context.Context, params *nsScan
// Send final response. // Send final response.
resp := storageNSScannerRPC.NewResponse() resp := storageNSScannerRPC.NewResponse()
resp.Final = &ui resp.Final = &ui
resp.Update = nil
out <- resp out <- resp
return nil return nil
} }
+94
View File
@@ -0,0 +1,94 @@
// Copyright (c) 2026 Julien Laurenceau and Feng Ruohang
//
// This file is part of Silo Object Storage.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"context"
"testing"
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/grid"
)
type scannerResponseTestDisk struct {
StorageAPI
}
func (scannerResponseTestDisk) NSScanner(_ context.Context, cache dataUsageCache, updates chan<- dataUsageEntry, _ madmin.HealScanMode, _ func() bool) (dataUsageCache, error) {
defer close(updates)
updates <- dataUsageEntry{Size: int64(cache.Info.NextCycle)}
cache.Info.NextCycle++
return cache, nil
}
func TestNSScannerHandlerClearsStaleResponses(t *testing.T) {
endpoint, err := NewEndpoint(t.TempDir())
if err != nil {
t.Fatal(err)
}
globalLocalDrivesMu.Lock()
oldDrives, oldSets := globalLocalDrivesMap, globalLocalSetDrives
globalLocalDrivesMap = map[string]StorageAPI{endpoint.String(): scannerResponseTestDisk{}}
globalLocalSetDrives = nil
globalLocalDrivesMu.Unlock()
t.Cleanup(func() {
globalLocalDrivesMu.Lock()
globalLocalDrivesMap, globalLocalSetDrives = oldDrives, oldSets
globalLocalDrivesMu.Unlock()
})
oldRPC := storageNSScannerRPC
t.Cleanup(func() { storageNSScannerRPC = oldRPC })
// Every new allocation is dirty too: the check must not depend on
// sync.Pool returning a previously used object.
storageNSScannerRPC = grid.NewStream[*nsScannerOptions, grid.NoPayload, *nsScannerResp](grid.HandlerNSScanner,
func() *nsScannerOptions { return &nsScannerOptions{} }, nil,
func() *nsScannerResp {
return &nsScannerResp{Update: &dataUsageEntry{Size: -1}, Final: &dataUsageCache{}}
})
server := storageRESTServer{endpoint: endpoint}
for cycle := uint32(1); cycle <= 3; cycle++ {
out := make(chan *nsScannerResp, 2)
params := &nsScannerOptions{Cache: &dataUsageCache{Info: dataUsageCacheInfo{Name: "bucket", NextCycle: cycle}}}
if err := server.NSScannerHandler(t.Context(), params, out); err != nil {
t.Fatal(err)
}
if len(out) != 2 {
t.Fatalf("cycle %d: got %d responses, want an update and a final cache", cycle, len(out))
}
for i := 0; i < 2; i++ {
resp := <-out
wire, err := resp.MarshalMsg(nil)
storageNSScannerRPC.PutResponse(resp)
if err != nil {
t.Fatal(err)
}
var decoded nsScannerResp
if _, err := decoded.UnmarshalMsg(wire); err != nil {
t.Fatal(err)
}
if i == 0 {
if decoded.Update == nil || decoded.Final != nil || decoded.Update.Size != int64(cycle) {
t.Errorf("cycle %d: invalid update response: %+v", cycle, decoded)
}
} else if decoded.Update != nil || decoded.Final == nil || decoded.Final.Info.NextCycle != cycle+1 {
t.Errorf("cycle %d: invalid final response: %+v", cycle, decoded)
}
}
}
}
+20 -2
View File
@@ -44,6 +44,7 @@ import (
"github.com/minio/minio-go/v7/pkg/tags" "github.com/minio/minio-go/v7/pkg/tags"
xhttp "github.com/minio/minio/internal/http" xhttp "github.com/minio/minio/internal/http"
"github.com/pgsty/silo-pkg/v3/ldap" "github.com/pgsty/silo-pkg/v3/ldap"
"github.com/pgsty/silo-pkg/v3/policy"
) )
func runAllIAMSTSTests(suite *TestSuiteIAM, c *check) { func runAllIAMSTSTests(suite *TestSuiteIAM, c *check) {
@@ -2384,8 +2385,10 @@ func (s *TestSuiteIAM) TestIAMImport(c *check, exportedContent []byte, caseNum i
if err != nil { if err != nil {
c.Fatalf("import %d: Unable to list policies: %v", caseNum, err) c.Fatalf("import %d: Unable to list policies: %v", caseNum, err)
} }
defaultCannedPolicies := set.CreateStringSet("consoleAdmin", "readwrite", "readonly", defaultCannedPolicies := set.CreateStringSet()
"diagnostics", "writeonly") for _, p := range policy.DefaultPolicies {
defaultCannedPolicies.Add(p.Name)
}
for policy, policyBytes := range policyContentMap { for policy, policyBytes := range policyContentMap {
if defaultCannedPolicies.Contains(policy) { if defaultCannedPolicies.Contains(policy) {
continue continue
@@ -2991,6 +2994,21 @@ func (s *TestSuiteIAM) TestLDAPSTSServiceAccounts(c *check) {
// 6. Check that service account cannot be created for some other user. // 6. Check that service account cannot be created for some other user.
c.mustNotCreateSvcAccount(ctx, globalActiveCred.AccessKey, userAdmClient) c.mustNotCreateSvcAccount(ctx, globalActiveCred.AccessKey, userAdmClient)
// An empty policy must not contaminate an expiring LDAP child credential.
emptyPolicyCred, err := userAdmClient.AddServiceAccountLDAP(ctx, madmin.AddServiceAccountReq{Policy: []byte("null")})
if err != nil {
c.Fatalf("Unable to create empty-policy LDAP service account: %v", err)
}
emptyPolicyAdmin := s.getAdminClient(c, emptyPolicyCred.AccessKey, emptyPolicyCred.SecretKey, "")
expires := UTCNow().Add(time.Hour)
for _, childPolicy := range [][]byte{nil, []byte("null")} {
child, err := emptyPolicyAdmin.AddServiceAccountLDAP(ctx, madmin.AddServiceAccountReq{Expiration: &expires, Policy: childPolicy})
if err != nil {
c.Fatalf("Unable to create expiring LDAP child: %v", err)
}
c.assertSvcAccS3Access(ctx, s, child, bucket)
}
// Detach the policy from the user // Detach the policy from the user
if _, err = s.adm.DetachPolicyLDAP(ctx, userReq); err != nil { if _, err = s.adm.DetachPolicyLDAP(ctx, userReq); err != nil {
c.Fatalf("Unable to detach user policy: %v", err) c.Fatalf("Unable to detach user policy: %v", err)
+123
View File
@@ -18,10 +18,16 @@
package cmd package cmd
import ( import (
"context"
"errors"
"net/http"
"os" "os"
"path/filepath"
"runtime" "runtime"
"sync"
"syscall" "syscall"
"testing" "testing"
"time"
) )
func TestSysErrors(t *testing.T) { func TestSysErrors(t *testing.T) {
@@ -56,3 +62,120 @@ func TestSysErrors(t *testing.T) {
} }
} }
} }
// TestCommitMetaMissingMapping verifies the RPC and object-layer mapping
// of errCommitMetaMissing, returned when the staged xl.meta disappears
// between the data rename and the meta-file commit in RenameData().
func TestCommitMetaMissingMapping(t *testing.T) {
// The sentinel must survive a storage REST/grid round-trip; errors
// cross the wire as strings and are remapped by toStorageErr().
if got := toStorageErr(errors.New(errCommitMetaMissing.Error())); !errors.Is(got, errCommitMetaMissing) {
t.Fatalf("expected %s, got %s", errCommitMetaMissing, got)
}
if got := toStorageErr(errCommitMetaMissing); got != errCommitMetaMissing {
t.Fatalf("expected %s, got %s", errCommitMetaMissing, got)
}
// toObjectErr() must map the sentinel to InsufficientWriteQuorum
// (503), never to ObjectNotFound (404).
err := toObjectErr(errCommitMetaMissing, "bucket", "object")
qerr, ok := err.(InsufficientWriteQuorum)
if !ok {
t.Fatalf("expected InsufficientWriteQuorum, got %s (%T)", err, err)
}
if qerr.Bucket != "bucket" || qerr.Object != "object" {
t.Fatalf("unexpected bucket/object %s/%s", qerr.Bucket, qerr.Object)
}
apiErr := toAPIError(t.Context(), err)
if apiErr.Code != "SlowDownWrite" || apiErr.HTTPStatusCode != http.StatusServiceUnavailable {
t.Fatalf("expected SlowDownWrite/503, got %+v", apiErr)
}
}
// missingCommitMetaContext removes staged metadata at an existing cancellation
// check, after WriteAll and (for non-inline objects) the data rename succeeded.
// This deterministically models concurrent cleanup without changing production
// code or relying on a filesystem race. The context itself remains live.
type missingCommitMetaContext struct {
context.Context
mu sync.Mutex
meta, dataDest string
removed bool
removeErr error
}
func (c *missingCommitMetaContext) Done() <-chan struct{} {
c.mu.Lock()
defer c.mu.Unlock()
if !c.removed {
if _, err := os.Stat(c.meta); err == nil {
dataMoved := c.dataDest == ""
if !dataMoved {
_, err = os.Stat(c.dataDest)
dataMoved = err == nil
}
if dataMoved {
c.removeErr = os.Remove(c.meta)
c.removed = c.removeErr == nil
}
}
}
return c.Context.Done()
}
func TestXLStorageRenameDataCommitMetaMissing(t *testing.T) {
for _, tc := range []struct {
name string
dataFile bool
}{
{"empty_object", false},
{"data_directory", true},
} {
t.Run(tc.name, func(t *testing.T) {
disk, root, err := newXLStorageTestSetup(t)
if err != nil {
t.Fatal(err)
}
defer disk.Close()
s := disk.storage
if err := s.MakeVol(t.Context(), "bucket"); err != nil {
t.Fatal(err)
}
fi := newFileInfo("object", 2, 2)
fi.DataDir, fi.VersionID = mustGetUUID(), mustGetUUID()
fi.Erasure.Index = 1
fi.ModTime = time.Now().UTC()
fi.Fresh = true
ctx := &missingCommitMetaContext{
Context: t.Context(),
meta: filepath.Join(root, minioMetaTmpBucket, "staged", xlStorageFormatFile),
}
if tc.dataFile {
fi.Size = 1
fi.AddObjectPart(1, "", 1, 1, fi.ModTime, nil, nil)
part := filepath.Join(root, minioMetaTmpBucket, "staged", fi.DataDir, "part.1")
if err := os.MkdirAll(filepath.Dir(part), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(part, []byte("a"), 0o644); err != nil {
t.Fatal(err)
}
ctx.dataDest = filepath.Join(root, "bucket", "object", fi.DataDir)
}
_, err = s.RenameData(ctx, minioMetaTmpBucket, "staged", fi, "bucket", "object", RenameOptions{})
if !ctx.removed {
t.Fatalf("metadata removal did not execute: removal error = %v, RenameData error = %v", ctx.removeErr, err)
}
if !errors.Is(err, errCommitMetaMissing) {
t.Fatalf("expected commit metadata missing, got %v", err)
}
apiErr := toAPIError(t.Context(), toObjectErr(err, "bucket", "object"))
if apiErr.Code != "SlowDownWrite" || apiErr.HTTPStatusCode != http.StatusServiceUnavailable {
t.Fatalf("expected SlowDownWrite/503, got %+v", apiErr)
}
if _, err := os.Stat(filepath.Join(root, "bucket", "object", xlStorageFormatFile)); !os.IsNotExist(err) {
t.Fatalf("failed commit must not produce destination metadata: %v", err)
}
})
}
}
+6 -2
View File
@@ -40,6 +40,10 @@ func (x *xlMetaV2VersionHeader) unmarshalV(v uint8, bts []byte) (o []byte, err e
// unmarshalV1 decodes version 1, never released. // unmarshalV1 decodes version 1, never released.
func (x *xlMetaV2VersionHeader) unmarshalV1(bts []byte) (o []byte, err error) { func (x *xlMetaV2VersionHeader) unmarshalV1(bts []byte) (o []byte, err error) {
// Fields absent from the V1 wire format must be reset since the
// header may be reused across decodes.
x.Signature = [4]byte{}
x.EcN, x.EcM = 0, 0
var zb0001 uint32 var zb0001 uint32
zb0001, bts, err = msgp.ReadArrayHeaderBytes(bts) zb0001, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil { if err != nil {
@@ -134,7 +138,7 @@ type xlMetaV2VersionHeaderV2 struct {
// UnmarshalMsg implements msgp.Unmarshaler // UnmarshalMsg implements msgp.Unmarshaler
func (z *xlMetaV2VersionHeaderV2) UnmarshalMsg(bts []byte) (o []byte, err error) { func (z *xlMetaV2VersionHeaderV2) UnmarshalMsg(bts []byte) (o []byte, err error) {
z.EcN, z.EcN = 0, 0 z.EcN, z.EcM = 0, 0
var zb0001 uint32 var zb0001 uint32
zb0001, bts, err = msgp.ReadArrayHeaderBytes(bts) zb0001, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil { if err != nil {
@@ -184,7 +188,7 @@ func (z *xlMetaV2VersionHeaderV2) UnmarshalMsg(bts []byte) (o []byte, err error)
// DecodeMsg implements msgp.Decodable // DecodeMsg implements msgp.Decodable
func (z *xlMetaV2VersionHeaderV2) DecodeMsg(dc *msgp.Reader) (err error) { func (z *xlMetaV2VersionHeaderV2) DecodeMsg(dc *msgp.Reader) (err error) {
z.EcN, z.EcN = 0, 0 z.EcN, z.EcM = 0, 0
var zb0001 uint32 var zb0001 uint32
zb0001, err = dc.ReadArrayHeader() zb0001, err = dc.ReadArrayHeader()
if err != nil { if err != nil {
+90
View File
@@ -38,6 +38,7 @@ import (
"github.com/klauspost/compress/zstd" "github.com/klauspost/compress/zstd"
"github.com/minio/minio/internal/bucket/lifecycle" "github.com/minio/minio/internal/bucket/lifecycle"
xhttp "github.com/minio/minio/internal/http" xhttp "github.com/minio/minio/internal/http"
"github.com/tinylib/msgp/msgp"
) )
func TestReadXLMetaNoData(t *testing.T) { func TestReadXLMetaNoData(t *testing.T) {
@@ -1190,3 +1191,92 @@ func BenchmarkToFileInfoNoParts(b *testing.B) {
func BenchmarkToFileInfoWithParts(b *testing.B) { func BenchmarkToFileInfoWithParts(b *testing.B) {
benchmarkManyPartsOptionally(b, true) benchmarkManyPartsOptionally(b, true)
} }
// TestXLMetaV2HeaderLegacyUnmarshalResetsEc checks that decoding legacy
// version headers, which carry no erasure metadata fields, cannot leak
// stale EcN/EcM values from a previously decoded object into the same
// structure.
func TestXLMetaV2HeaderLegacyUnmarshalResetsEc(t *testing.T) {
base := xlMetaV2VersionHeader{
VersionID: [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16},
ModTime: time.Now().Unix(),
Signature: [4]byte{1, 2, 3, 4},
Type: ObjectType,
Flags: xlFlagInlineData,
}
// Legacy V2 header wire format: 5 fields, no EcN/EcM.
var legacy []byte
legacy = msgp.AppendArrayHeader(legacy, 5)
legacy = msgp.AppendBytes(legacy, base.VersionID[:])
legacy = msgp.AppendInt64(legacy, base.ModTime)
legacy = msgp.AppendBytes(legacy, base.Signature[:])
legacy = msgp.AppendUint8(legacy, uint8(base.Type))
legacy = msgp.AppendUint8(legacy, uint8(base.Flags))
for _, decoder := range []string{"UnmarshalMsg", "DecodeMsg", "loadIndexed"} {
t.Run(decoder, func(t *testing.T) {
header := base
header.EcN, header.EcM = 4, 8
x2 := xlMetaV2VersionHeaderV2{xlMetaV2VersionHeader: &header}
var err error
switch decoder {
case "UnmarshalMsg":
_, err = x2.UnmarshalMsg(legacy)
case "DecodeMsg":
err = x2.DecodeMsg(msgp.NewReader(bytes.NewReader(legacy)))
case "loadIndexed":
indexed := msgp.AppendUint(nil, 2)
indexed = msgp.AppendUint(indexed, xlMetaVersion)
indexed = msgp.AppendInt(indexed, 1)
indexed = msgp.AppendBytes(indexed, legacy)
indexed = msgp.AppendBytes(indexed, nil)
xl := xlMetaV2{versions: []xlMetaV2ShallowVersion{{header: header}}}
err = xl.loadIndexed(indexed, nil)
header = xl.versions[0].header
}
if err != nil {
t.Fatal(err)
}
if header != base {
t.Fatalf("legacy header: got %+v, want %+v", header, base)
}
})
}
// V1 headers (4 fields, never released) must zero the fields too.
var v1 []byte
v1 = msgp.AppendArrayHeader(v1, 4)
v1 = msgp.AppendBytes(v1, base.VersionID[:])
v1 = msgp.AppendInt64(v1, base.ModTime)
v1 = msgp.AppendUint8(v1, uint8(base.Type))
v1 = msgp.AppendUint8(v1, uint8(base.Flags))
header := base
header.EcN, header.EcM = 4, 8
header.Signature = [4]byte{9, 9, 9, 9}
if _, err := header.unmarshalV1(v1); err != nil {
t.Fatalf("legacy V1 header unmarshal failed: %v", err)
}
if header.EcN != 0 || header.EcM != 0 {
t.Fatalf("stale erasure metadata leaked on V1 decode: EcN %d, EcM %d", header.EcN, header.EcM)
}
if header.Signature != ([4]byte{}) {
t.Fatalf("stale signature leaked on V1 decode: %v", header.Signature)
}
// Current headers must keep their erasure metadata.
current := base
current.EcN, current.EcM = 4, 8
wire, err := current.MarshalMsg(nil)
if err != nil {
t.Fatalf("header marshal failed: %v", err)
}
decoded := base
if _, err := decoded.UnmarshalMsg(wire); err != nil {
t.Fatalf("header unmarshal failed: %v", err)
}
if decoded.EcN != 4 || decoded.EcM != 8 {
t.Fatalf("erasure metadata lost on current decode: EcN %d, EcM %d", decoded.EcN, decoded.EcM)
}
}
+9 -1
View File
@@ -2925,7 +2925,15 @@ func (s *xlStorage) RenameData(ctx context.Context, srcVolume, srcPath string, f
// if its a partial rename() do not attempt to delete recursively. // if its a partial rename() do not attempt to delete recursively.
// this can be healed since all parts are available. // this can be healed since all parts are available.
s.deleteFile(dstVolumeDir, dstDataPath, false, false) s.deleteFile(dstVolumeDir, dstDataPath, false, false)
return res, osErrToFileErr(err) err = osErrToFileErr(err)
if errors.Is(err, errFileNotFound) {
// The staged xl.meta disappeared between staging and commit.
// This is a write-quorum failure, not a missing object; use a
// dedicated error so the object layer returns 503 instead of
// mapping it to ObjectNotFound (404).
err = errCommitMetaMissing
}
return res, err
} }
if srcVolume != minioMetaMultipartBucket { if srcVolume != minioMetaMultipartBucket {
+3 -3
View File
@@ -10,13 +10,13 @@ Silo in distributed mode can help you setup a highly-available storage system wi
Distributed Silo provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/silo/blob/main/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed Silo is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed Silo. Distributed Silo provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/silo/blob/main/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed Silo is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed Silo.
If one or more drives are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data protection bits added automatically to provide additional safety for these objects. With the default `availability` storage-class optimization, Silo can increase parity for new objects when drives are offline, up to half the drives in the erasure set. Writes must still meet the quorum for the resulting shard layout. This does not guarantee unchanged availability or keep a two-drive `EC:1` set writable after one drive fails. See the [storage-class reference](https://silo.pgsty.com/reference/minio-server/settings/storage-class/) for details.
### High availability ### High availability
A stand-alone Silo server would go down if the server hosting the drives goes offline. In contrast, a distributed Silo setup with _m_ servers and _n_ drives will have your data safe as long as _m/2_ servers or _m*n_/2 or more drives are online. A stand-alone Silo server becomes unavailable if its host goes offline. In a distributed deployment, determine node-failure tolerance from how many drives each failed node removes from every erasure set and from the parity recorded for the affected objects. Aggregate online node or drive counts alone do not establish read or write quorum.
For example, an 16-server distributed setup with 200 drives per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 drives down Silo would continue to read and write objects. For example, if a 16-node pool has 16-drive erasure sets with one drive per node in each set, a fixed `EC:4` layout has a read and write quorum of 12. Losing four nodes leaves 12 drives in every set. This result depends on that drive placement and parity; it cannot be generalized to arbitrary failures of the same total number of drives. In a two-node, two-drive `EC:1` set, losing one node leaves only one drive, so existing intact objects can remain readable but writes cannot continue.
Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/pgsty/silo/blob/main/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class). Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/pgsty/silo/blob/main/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class).
+6 -6
View File
@@ -1,18 +1,18 @@
# Silo Erasure Code Quickstart Guide # Silo Erasure Code Quickstart Guide
Silo protects data against hardware failures and silent data corruption using erasure code and checksums. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data. Silo protects data against hardware failures and silent data corruption using erasure coding and checksums. Recovery depends on the healthy shards and metadata remaining in each object's erasure set. With maximum parity, an object can tolerate the loss of up to `floor(N/2)` shards in its `N`-drive set; the deployment's total online drive count is not sufficient to determine recoverability.
## What is Erasure Code? ## What is Erasure Code?
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Silo uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks. Erasure coding uses mathematical algorithms to reconstruct missing or corrupted data. Silo uses Reed-Solomon codes to split objects into data and parity shards. For a 12-drive erasure set, parity can range from `EC:0` (12 data shards and no parity) to `EC:6` (six data shards and six parity shards).
By default, Silo shards the objects across N/2 data and N/2 parity drives. Though, you can use [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class) to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures. The default parity depends on the erasure set size: `EC:0` for 1 drive, `EC:1` for 2–3 drives, `EC:2` for 4–5 drives, `EC:3` for 6–7 drives, and `EC:4` for 8–16 drives. See the [Silo storage-class reference](https://silo.pgsty.com/reference/minio-server/settings/storage-class/) for configuration details.
In 12 drive example above, with Silo server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives. In the 12-drive example, the default `EC:4` layout uses eight data shards and four parity shards. An existing object can tolerate four unavailable shards if the remaining shards and metadata are intact. Explicit `EC:6` instead uses six data and six parity shards, with a read quorum of six and a write quorum of seven.
## Why is Erasure Code useful? ## Why is Erasure Code useful?
Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in Silo erasure code you can lose as many as half of drives and still the data remains safe. Further, Silo's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As Silo encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. Silo's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available. Erasure coding protects objects against drive failures within their erasure set, up to the parity recorded for each object. Silo encodes objects individually and can heal them incrementally when enough healthy shards and consistent metadata remain. Failed drives still need repair or replacement to restore redundancy; erasure coding does not eliminate that maintenance.
![Erasure](https://github.com/pgsty/silo/blob/main/docs/screenshots/erasure-code.jpg?raw=true) ![Erasure](https://github.com/pgsty/silo/blob/main/docs/screenshots/erasure-code.jpg?raw=true)
@@ -64,4 +64,4 @@ podman run \
### 3. Test your setup ### 3. Test your setup
You may unplug drives randomly and continue to perform I/O on the system. In a test deployment, take drives offline and verify reads and writes against the quorum required by each erasure set and object layout. Restore the drives after each test; continued I/O depends on the remaining healthy shards and metadata.
+10 -18
View File
@@ -38,31 +38,23 @@ You can calculate _approximate_ storage usage ratio using the formula - total dr
### Allowed values for STANDARD storage class ### Allowed values for STANDARD storage class
`STANDARD` storage class implies more parity than `REDUCED_REDUNDANCY` class. So, `STANDARD` parity drives should be `STANDARD` supports `EC:0` without erasure-code redundancy and nonzero parity values up to `floor(N/2)`, where `N` is the number of drives in the erasure set. When both `STANDARD` and `REDUCED_REDUNDANCY` parity are nonzero, `STANDARD` must be greater than or equal to `REDUCED_REDUNDANCY`; equal parity is allowed.
- Greater than or equal to 2, if `REDUCED_REDUNDANCY` parity is not set. The default `STANDARD` parity is:
- Greater than `REDUCED_REDUNDANCY` parity, if it is set.
Parity blocks can not be higher than data blocks, so `STANDARD` storage class parity can not be higher than N/2. (N being total number of drives) | Erasure Set Size | Default Parity (EC:M) |
| --- | --- |
The default value for the `STANDARD` storage class depends on the number of volumes in the erasure set: | 1 | EC:0 |
| 2–3 | EC:1 |
| Erasure Set Size | Default Parity (EC:N) | | 4–5 | EC:2 |
|------------------|-----------------------| | 6–7 | EC:3 |
| 5 or fewer | EC:2 | | 8–16 | EC:4 |
| 6-7 | EC:3 |
| 8 or more | EC:4 |
For more complete documentation on Erasure Set sizing, see the [Silo Documentation on Erasure Sets](https://silo.pgsty.com/operations/concepts/erasure-coding/#minio-ec-erasure-set). For more complete documentation on Erasure Set sizing, see the [Silo Documentation on Erasure Sets](https://silo.pgsty.com/operations/concepts/erasure-coding/#minio-ec-erasure-set).
### Allowed values for REDUCED_REDUNDANCY storage class ### Allowed values for REDUCED_REDUNDANCY storage class
`REDUCED_REDUNDANCY` implies lesser parity than `STANDARD` class. So,`REDUCED_REDUNDANCY` parity drives should be `REDUCED_REDUNDANCY` parity can be zero or up to `floor(N/2)`. When both storage classes have nonzero parity, its parity must be less than or equal to `STANDARD` parity. The default is `EC:1` for multi-drive sets and `EC:0` for single-drive deployments. See the [Silo storage-class reference](https://silo.pgsty.com/reference/minio-server/settings/storage-class/) for the maintained configuration documentation.
- Less than N/2, if `STANDARD` parity is not set.
- Less than `STANDARD` Parity, if it is set.
Default value for `REDUCED_REDUNDANCY` storage class is `1`.
## Get started with Storage Class ## Get started with Storage Class
+3 -3
View File
@@ -1,10 +1,10 @@
module github.com/minio/minio module github.com/minio/minio
go 1.27.1 go 1.27.2
// Console and MC retain their historical module paths for best-effort upstream // Console and MC retain their historical module paths for best-effort upstream
// compatibility. Pin the maintained PGSTY implementations used by SILO. // compatibility. Pin the maintained PGSTY implementations used by SILO.
replace github.com/minio/console => github.com/pgsty/silo-console v0.0.0-20260916075814-1360e26d976d replace github.com/minio/console => github.com/pgsty/silo-console v0.0.0-20261015132217-e5b165e4c0ee
replace github.com/minio/mc => github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a replace github.com/minio/mc => github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a
@@ -294,7 +294,7 @@ require (
go.opentelemetry.io/otel/trace v1.45.0 // indirect go.opentelemetry.io/otel/trace v1.45.0 // indirect
go.uber.org/multierr v1.11.0 // indirect go.uber.org/multierr v1.11.0 // indirect
golang.org/x/mod v0.41.0 // indirect golang.org/x/mod v0.41.0 // indirect
golang.org/x/net v0.59.0 // indirect golang.org/x/net v0.60.0 // indirect
golang.org/x/text v0.42.0 // indirect golang.org/x/text v0.42.0 // indirect
golang.org/x/tools v0.50.0 // indirect golang.org/x/tools v0.50.0 // indirect
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
+4 -4
View File
@@ -547,8 +547,8 @@ github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwp
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a h1:FY1vcTQT67HGBe23ozlhOiQnWCzUCGmeSN5x6DT/kFs= github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a h1:FY1vcTQT67HGBe23ozlhOiQnWCzUCGmeSN5x6DT/kFs=
github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a/go.mod h1:6z9lX8kOvaWz2vLlP7BjjF3JWHnIH86XDA2HwcmpZH8= github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a/go.mod h1:6z9lX8kOvaWz2vLlP7BjjF3JWHnIH86XDA2HwcmpZH8=
github.com/pgsty/silo-console v0.0.0-20260916075814-1360e26d976d h1:A9ou1GkUckGZd3BOgEXeZ393nbYST967SKcI4Lgr/Vs= github.com/pgsty/silo-console v0.0.0-20261015132217-e5b165e4c0ee h1:e51AnrGe0Zq+nHTYaGfv/GkehQMO9Mekuw/ew6zYwOM=
github.com/pgsty/silo-console v0.0.0-20260916075814-1360e26d976d/go.mod h1:JS165CF2yoTBIx1X8k1IQ6vunJcZFG3pRZlWHJ8UT8o= github.com/pgsty/silo-console v0.0.0-20261015132217-e5b165e4c0ee/go.mod h1:jpt39bx4wjhXoFB+BTOa/bx/pmU3pXzp2cT7PnT21Lo=
github.com/pgsty/silo-pkg/v3 v3.14.1 h1:8MONT3Hky9EOnsuPk29590pEBE6MILaOnJtaejPPi7M= github.com/pgsty/silo-pkg/v3 v3.14.1 h1:8MONT3Hky9EOnsuPk29590pEBE6MILaOnJtaejPPi7M=
github.com/pgsty/silo-pkg/v3 v3.14.1/go.mod h1:jxKWxi52jSsDaDhXXPqjOyPDYdxx2iqdBXyLfM/xSIw= github.com/pgsty/silo-pkg/v3 v3.14.1/go.mod h1:jxKWxi52jSsDaDhXXPqjOyPDYdxx2iqdBXyLfM/xSIw=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM= github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
@@ -762,8 +762,8 @@ golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug
golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws= golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues= golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98= golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58= golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+1 -1
View File
@@ -24,7 +24,7 @@ type TargetIDSet map[TargetID]struct{}
// IsEmpty returns true if the set is empty. // IsEmpty returns true if the set is empty.
func (set TargetIDSet) IsEmpty() bool { func (set TargetIDSet) IsEmpty() bool {
return len(set) != 0 return len(set) == 0
} }
// Clone - returns copy of this set. // Clone - returns copy of this set.
@@ -22,6 +22,22 @@ import (
"testing" "testing"
) )
func TestTargetIDSetIsEmpty(t *testing.T) {
testCases := []struct {
set TargetIDSet
expected bool
}{
{NewTargetIDSet(), true},
{NewTargetIDSet(TargetID{"1", "webhook"}), false},
}
for i, testCase := range testCases {
if got := testCase.set.IsEmpty(); got != testCase.expected {
t.Fatalf("test %v: expected: %v, got: %v", i+1, testCase.expected, got)
}
}
}
func TestTargetIDSetClone(t *testing.T) { func TestTargetIDSetClone(t *testing.T) {
testCases := []struct { testCases := []struct {
set TargetIDSet set TargetIDSet
+73
View File
@@ -0,0 +1,73 @@
// Copyright (c) 2026 PGSTY
//
// This file is part of SILO Object Storage.
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package hash
import (
"bytes"
"errors"
"io"
"net/http"
"net/http/httptest"
"testing"
)
func TestChecksumMismatchAlgorithm(t *testing.T) {
content := []byte("actual content")
for _, typ := range []ChecksumType{ChecksumCRC32, ChecksumCRC32C, ChecksumSHA1, ChecksumSHA256, ChecksumCRC64NVME} {
t.Run(typ.String(), func(t *testing.T) {
wrong := NewChecksumFromData(typ, []byte("wrong content"))
check := func(t *testing.T, err error) {
t.Helper()
var mismatch ChecksumMismatch
if !errors.As(err, &mismatch) || mismatch.Algorithm != typ.String() {
t.Fatalf("error=%+v, want %s checksum mismatch", err, typ)
}
}
check(t, wrong.Matches(content, 0))
parts := NewChecksumFromData(typ, content)
parts.WantParts = 2
check(t, parts.Matches(content, 1))
for _, mode := range []string{"header", "trailer", "malformed-trailer"} {
t.Run(mode, func(t *testing.T) {
reader, err := NewReader(t.Context(), bytes.NewReader(content), int64(len(content)), "", "", int64(len(content)))
if err != nil {
t.Fatal(err)
}
req := httptest.NewRequest(http.MethodPut, "http://localhost/bucket/object", nil)
if mode == "header" {
req.Header.Set(typ.Key(), wrong.Encoded)
} else {
req.Header.Set("x-amz-trailer", typ.Key())
req.Trailer = make(http.Header)
value := wrong.Encoded
if mode == "malformed-trailer" {
value = "!invalid-base64!"
}
req.Trailer.Set(typ.Key(), value)
}
if err = reader.AddChecksum(req, false); err != nil {
t.Fatal(err)
}
_, err = io.ReadAll(reader)
check(t, err)
})
}
})
}
}
+6 -4
View File
@@ -584,15 +584,17 @@ func (c Checksum) Matches(content []byte, parts int) error {
sum := hasher.Sum(nil) sum := hasher.Sum(nil)
if c.WantParts > 0 && c.WantParts != parts { if c.WantParts > 0 && c.WantParts != parts {
return ChecksumMismatch{ return ChecksumMismatch{
Want: fmt.Sprintf("%s-%d", c.Encoded, c.WantParts), Algorithm: c.Type.String(),
Got: fmt.Sprintf("%s-%d", base64.StdEncoding.EncodeToString(sum), parts), Want: fmt.Sprintf("%s-%d", c.Encoded, c.WantParts),
Got: fmt.Sprintf("%s-%d", base64.StdEncoding.EncodeToString(sum), parts),
} }
} }
if !bytes.Equal(sum, c.Raw) { if !bytes.Equal(sum, c.Raw) {
return ChecksumMismatch{ return ChecksumMismatch{
Want: c.Encoded, Algorithm: c.Type.String(),
Got: base64.StdEncoding.EncodeToString(sum), Want: c.Encoded,
Got: base64.StdEncoding.EncodeToString(sum),
} }
} }
return nil return nil
+5 -2
View File
@@ -74,8 +74,11 @@ func (e SizeMismatch) Error() string {
// ChecksumMismatch - when content checksum does not match with what was sent from client. // ChecksumMismatch - when content checksum does not match with what was sent from client.
type ChecksumMismatch struct { type ChecksumMismatch struct {
Want string // Algorithm carries the requested checksum type (e.g. "CRC32") so API
Got string // error rendering can name it, matching S3's wording.
Algorithm string
Want string
Got string
} }
func (e ChecksumMismatch) Error() string { func (e ChecksumMismatch) Error() string {
+4 -3
View File
@@ -302,13 +302,14 @@ func (r *Reader) Read(p []byte) (int, error) {
r.contentHash.Encoded = r.trailer.Get(r.contentHash.Type.Key()) r.contentHash.Encoded = r.trailer.Get(r.contentHash.Type.Key())
r.contentHash.Raw, err = base64.StdEncoding.DecodeString(r.contentHash.Encoded) r.contentHash.Raw, err = base64.StdEncoding.DecodeString(r.contentHash.Encoded)
if err != nil || len(r.contentHash.Raw) == 0 { if err != nil || len(r.contentHash.Raw) == 0 {
return 0, ChecksumMismatch{Got: r.contentHash.Encoded} return 0, ChecksumMismatch{Algorithm: r.contentHash.Type.String(), Got: r.contentHash.Encoded}
} }
} }
if sum := r.contentHasher.Sum(nil); !bytes.Equal(r.contentHash.Raw, sum) { if sum := r.contentHasher.Sum(nil); !bytes.Equal(r.contentHash.Raw, sum) {
err := ChecksumMismatch{ err := ChecksumMismatch{
Want: r.contentHash.Encoded, Algorithm: r.contentHash.Type.String(),
Got: base64.StdEncoding.EncodeToString(sum), Want: r.contentHash.Encoded,
Got: base64.StdEncoding.EncodeToString(sum),
} }
return n, err return n, err
} }