Merge current main into multipart listing contribution

Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
2026-09-16 12:51:20 +08:00
20 changed files with 910 additions and 92 deletions
+3
View File
@@ -90,6 +90,9 @@ jobs:
- name: Run S3 Select tests under race detector
run: go test -race ./internal/s3select/... -count=1
- name: Run conditional PUT tests under race detector
run: go test -race ./cmd -run '^Test(PoolsConditionalPut|SinglePoolConditionalPutHTTP)' -count=1 -timeout=5m
crosscompile:
name: Cross Compile
runs-on: ubuntu-latest
+57 -3
View File
@@ -9,6 +9,33 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
### Authorization and security
- Restrict embedded Console's anonymous sharing proxy to object-content GETs
at the configured S3 origin, and reject every redirect. Internal metrics,
system paths and non-download S3 operations cannot be reached through it.
Normal public, presigned and versioned downloads remain available without a
new setting; a full sharing-disable switch is not introduced. See
[Console #56](https://github.com/pgsty/silo-console/pull/56) and the
[design record](https://github.com/pgsty/silo-console/issues/52).
Thanks to Jiri Pejchal (@jiri-pejchal) for the report.
- Persist IAM deletion revisions and parent revocation boundaries so stale site
events cannot restore deleted identities, policies or their older grants
(#191, #192). Peer deletion notifications reload committed storage; deliberate
recreation requires a newer revision, and credentials issued before the
parent's revocation remain invalid.
**Coordinated upgrade required:** upgrade every participating node and site.
Mixed old/new nodes sharing an IAM backend and rolling downgrade are
unsupported. Back up complete IAM storage and encryption material; an admin
export of live records omits deletion history. Reissue credentials for
recreated parents and explicitly reconcile pre-upgrade revocations whose
history is already lost. Restoring an older backup can lose later revocations;
keep affected sites isolated until reconciliation/rekeying is complete. See
[the operator runbook](https://github.com/pgsty/silo.pgsty.com/blob/29c7f220b3acc556ad570694056d35e11246f1b9/content/operations/replication/iam-upgrade.md).
- Enforce an absolute HTTP/1 request-header deadline through the connection
wrapper (#196). Repeated small reads no longer extend that deadline, and
`--read-header-timeout` / `MINIO_READ_HEADER_TIMEOUT` now reaches the HTTP
server. HTTP/1 request bodies retain the rolling idle timeout; this does not
impose a total upload/download duration. A shorter setting also constrains
TLS handshake reads. The wrapper's strict header mode is not applied to HTTP/2.
- Reject unsigned `x-amz-*` request headers that could turn a signed PUT into a
copy of another object accessible to the signer (SN-2026-011). The latest
public Server is affected; the fix is on main. See [the advisory ledger](docs/security/advisories.md).
@@ -30,17 +57,44 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
rolling upgrades, detection of any legacy keyless upload retains the prior
listing behavior until those uploads drain. See [issue #79](https://github.com/pgsty/silo/issues/79)
and its [design record](https://silo.pgsty.com/blog/design/list-multipart-uploads/).
- Preserve object tags during multi-pool metadata reconciliation by reading the
resolved tag field together with its revision (#189). Previously, reconciliation
could replace existing tags with an empty value.
- Preserve the tag revision on SSE-KMS metadata replication (#193), and advance
tag revisions monotonically on local PUT/DELETE tagging (#196). Empty tags
participate in reconciliation as an ordered deletion, preventing older
events from restoring removed tags. SSE-C key rotation also retains the tag
revision. Malformed historical revisions can fail and retry; their missing
history is not reconstructed by the upgrade.
- Complete delete-marker version purges and preserve their identity and retry
state through MRF recovery (#196). Recovery accepts a 405 marker response only
when its version, bucket, object name and modification time match the task.
Purge audit status is normalized from `COMPLETE` to `COMPLETED`.
Thanks to Julien Laurenceau (@julienlau) for the investigation and proposed
fix in #184 that helped shape this follow-up.
- Restore only the six replication-specific metadata fields after ordinary
request metadata extraction (#194). This prevents transport-only `aws-chunked`
from being stored as Content-Encoding while preserving the signed-header
protections. Trusted Snowball entries no longer inherit the outer archive's
ordinary metadata. Thanks to Mikhail Khadarenka (@chodorenko) for the fix in #187.
**Existing data:** these repairs prevent new errors; they do not scan or rewrite
historical object metadata, recover lost tags or prove that old purge work has
converged. Follow the [read-only audit procedure](https://github.com/pgsty/silo.pgsty.com/blob/29c7f220b3acc556ad570694056d35e11246f1b9/content/operations/replication/replica-metadata-audit.md)
before planning any repair of stored state.
- Evaluate conditional multipart completion against the logical current object
across all pools while holding the existing object lock. A stale `If-Match`
can no longer replace newer data in another pool, and the current ETag is no
longer rejected because the upload resides next to an older copy. Conditions
are evaluated once; a current delete marker counts as an absent object.
**Availability change:** if metadata cannot be read from any pool, conditional
**Availability change:** if any pool's metadata cannot be read, conditional
completion fails even when another pool can still serve GET/HEAD. This also
applies when the unreadable pool may not hold the object: absence cannot be
verified. Retry after the pool recovers. Unconditional completion and the
single-pool path retain their existing behavior.
Ordinary conditional PUT has a separate cross-pool precondition gap tracked
in [#199](https://github.com/pgsty/silo/issues/199); the multipart repair does
not resolve it.
- Reconcile ordinary single-object version DELETE across all pools, including
null versions, delete markers and unqualified directory-marker DELETE. This
@@ -70,7 +124,7 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
- Repair federated CopyObject checksums, destination timestamps, reserved
metadata, encrypted-object forwarding, legal hold and KMS context.
- Make resync counters, target selection, cancellation and worker lifetimes
reflect actual work; complete delete-marker purges and report bounded MRF drops.
reflect actual work, and report bounded MRF drops.
- Converge bucket metadata with deterministic source state, deletion tombstones,
creation time recovery and diagnostics. The mixed-version export gate requires
coordinated upgrades before tombstones are exported. See [the #77 record](docs/investigations/issue-77-current.md).
@@ -82,7 +136,7 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
- Restore embedded Console login over loopback TLS, trusted-proxy handling and
all four WebSocket connection limits. Preserve Go TLS defaults across transports.
- Directly require `github.com/pgsty/silo-pkg/v3` v3.14.0; select Console
`v0.0.0-20260913015128-417559bb2c97` and MC
`v0.0.0-20260916034812-56dfe455ac2f` and MC
`v0.0.0-20260913012246-4f609a4da3bb` with explicit PGSTY replacements.
- Pin upstream minio-go `v7.3.1-0.20260910142817-60bd07042d49`; refresh Go x/*
modules and security fixes including bounded AMQP frame handling. Keep Go
+7 -3
View File
@@ -1,8 +1,9 @@
# Contributors
SILO is built by **41 community contributors**, including its maintainers. This record covers every
SILO is built by **42 community contributors**, including its maintainers. This record covers every
human author who has opened an issue or pull request in the repositories listed below, in any state.
It was checked against the complete, paginated GitHub API records on **2026-09-07 03:00:14 UTC**.
The full API audit below dates to **2026-09-07 03:00:14 UTC**; later individually verified
reports include Jiri Pejchals Console #52, added on **2026-09-16**.
Contributors appear once in the avatar wall: authors of merged PRs first, other PR authors next,
and issue-only authors after them. Within each group, substantial features, security and correctness
@@ -31,6 +32,7 @@ the Git history and [NOTICE](NOTICE); this record covers activity in the PGSTY r
<a href="https://github.com/cbornet"><img src="https://silo.pgsty.com/images/contributors/cbornet.svg" width="60" height="60" alt="@cbornet" title="@cbornet — Reported multipart and streaming checksum defects and missing-bucket semantics"></a>
<a href="https://github.com/vampywiz17"><img src="https://silo.pgsty.com/images/contributors/vampywiz17.svg" width="60" height="60" alt="@vampywiz17" title="@vampywiz17 — Reported LDAP TLS and Console login regressions"></a>
<a href="https://github.com/orenyomtov"><img src="https://silo.pgsty.com/images/contributors/orenyomtov.svg" width="60" height="60" alt="@orenyomtov" title="@orenyomtov — Reported the unsigned-header CopyObject cross-object read (SN-2026-011)"></a>
<a href="https://github.com/jiri-pejchal"><img src="https://silo.pgsty.com/images/contributors/jiri-pejchal.svg" width="60" height="60" alt="@jiri-pejchal" title="@jiri-pejchal — Reported the Console share proxy exposure of internal metrics"></a>
<a href="https://github.com/mumu-lab"><img src="https://silo.pgsty.com/images/contributors/mumu-lab.svg" width="48" height="48" alt="@mumu-lab" title="@mumu-lab — Reported bucket quota metrics reading a deprecated field"></a>
<a href="https://github.com/jvasile"><img src="https://silo.pgsty.com/images/contributors/jvasile.svg" width="48" height="48" alt="@jvasile" title="@jvasile — Reported missing user, group, and defaults in Debian packages"></a>
<a href="https://github.com/pmezhuev"><img src="https://silo.pgsty.com/images/contributors/pmezhuev.svg" width="48" height="48" alt="@pmezhuev" title="@pmezhuev — Reported missing RPM package signatures"></a>
@@ -98,6 +100,7 @@ for their reports as well; the avatar wall and community total still count each
| [@cbornet](https://github.com/cbornet) | [pgsty/silo#31](https://github.com/pgsty/silo/issues/31) Multipart uploads with FULL_OBJECT CRC32 not working<br>[pgsty/silo#32](https://github.com/pgsty/silo/issues/32) `listObjects` should return `NoSuchBucket` when the bucket doesn't exist and prefix is passed<br>[pgsty/silo#107](https://github.com/pgsty/silo/issues/107) PutObject fails with chunked encoding and checksumType |
| [@vampywiz17](https://github.com/vampywiz17) | [pgsty/silo#15](https://github.com/pgsty/silo/issues/15) LDAP TLS regression in RELEASE.2026-03-21T00-00-00Z breaks built-in Console and external Console LDAP login on Kubernetes Tenant<br>[pgsty/silo#108](https://github.com/pgsty/silo/issues/108) Web Console login regression in RELEASE.2026-09-03T13-18-01Z (local and LDAP users fail) |
| [@orenyomtov](https://github.com/orenyomtov) | Private security disclosure: a presigned or signed PUT could be turned into a server-side CopyObject read of any object the signing key can reach via an unsigned `x-amz-copy-source` header. Fixed as [`SN-2026-011`](https://github.com/pgsty/silo/blob/main/docs/security/advisories.md) ([pgsty/silo#173](https://github.com/pgsty/silo/pull/173)) |
| [Jiri Pejchal (@jiri-pejchal)](https://github.com/jiri-pejchal) | [pgsty/silo-console#52](https://github.com/pgsty/silo-console/issues/52) — Reported anonymous access to internal metrics through the public object-sharing proxy, distinguished the demonstrated impact from hypothetical redirect abuse, and proposed server-side controls |
| [@mumu-lab](https://github.com/mumu-lab) | [pgsty/silo#106](https://github.com/pgsty/silo/issues/106) 监控指标读取已弃用的 BucketQuota.Quota 字段导致 Quota 指标无值 |
| [@jvasile](https://github.com/jvasile) | [pgsty/silo#33](https://github.com/pgsty/silo/issues/33) .deb doesn't create user/group/default files |
| [@pmezhuev](https://github.com/pmezhuev) | [pgsty/silo#43](https://github.com/pgsty/silo/issues/43) RPM package for RELEASE.2026-06-18T00-00-00Z is missing GPG signature |
@@ -125,7 +128,8 @@ for their reports as well; the avatar wall and community total still count each
## Audit scope
All issue and PR pages were read without a date cutoff. Counts below include automated accounts;
the **40-person** roll excludes the two bots, Copilot and dependabot. The maintained product stack
the historical **40-person** audit excluded the two bots, Copilot and dependabot.
The current 42-person list also includes subsequent verified reports. The maintained product stack
is SILO, Console, mcli, and silo-pkg; the SDK, KES, website, and older documentation repository were
also checked for community submissions.
+2 -1
View File
@@ -123,7 +123,7 @@ Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); e
## Contributors
**41 community contributors** build SILO, Console, mcli, shared packages, and related projects. The list includes maintainers and every human Issue or PR author, ordered by merged PRs, other PRs, then issue reports. Gold rings highlight significant contributions.
**42 community contributors** build SILO, Console, mcli, shared packages, and related projects. The list includes maintainers and every human Issue or PR author, ordered by merged PRs, other PRs, then issue reports. Gold rings highlight significant contributions.
<p align="center">
<a href="https://github.com/Vonng"><img src="https://silo.pgsty.com/images/contributors/Vonng.svg" width="60" height="60" alt="@Vonng" title="@Vonng — Maintains SILO, Console, mcli, shared packages, releases, and documentation"></a>
@@ -144,6 +144,7 @@ Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); e
<a href="https://github.com/cbornet"><img src="https://silo.pgsty.com/images/contributors/cbornet.svg" width="60" height="60" alt="@cbornet" title="@cbornet — Reported multipart and streaming checksum defects and missing-bucket semantics"></a>
<a href="https://github.com/vampywiz17"><img src="https://silo.pgsty.com/images/contributors/vampywiz17.svg" width="60" height="60" alt="@vampywiz17" title="@vampywiz17 — Reported LDAP TLS and Console login regressions"></a>
<a href="https://github.com/orenyomtov"><img src="https://silo.pgsty.com/images/contributors/orenyomtov.svg" width="60" height="60" alt="@orenyomtov" title="@orenyomtov — Reported the unsigned-header CopyObject cross-object read (SN-2026-011)"></a>
<a href="https://github.com/jiri-pejchal"><img src="https://silo.pgsty.com/images/contributors/jiri-pejchal.svg" width="60" height="60" alt="@jiri-pejchal" title="@jiri-pejchal — Reported the Console share proxy exposure of internal metrics"></a>
<a href="https://github.com/mumu-lab"><img src="https://silo.pgsty.com/images/contributors/mumu-lab.svg" width="48" height="48" alt="@mumu-lab" title="@mumu-lab — Reported bucket quota metrics reading a deprecated field"></a>
<a href="https://github.com/jvasile"><img src="https://silo.pgsty.com/images/contributors/jvasile.svg" width="48" height="48" alt="@jvasile" title="@jvasile — Reported missing user, group, and defaults in Debian packages"></a>
<a href="https://github.com/pmezhuev"><img src="https://silo.pgsty.com/images/contributors/pmezhuev.svg" width="48" height="48" alt="@pmezhuev" title="@pmezhuev — Reported missing RPM package signatures"></a>
+2 -1
View File
@@ -102,7 +102,7 @@ S3 API、`MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*
## 贡献者
**41 位社区贡献者**共同建设 SILO、Console、mcli、公共包与相关项目。名单包含维护者,以及所有提出 Issue 或 PR 的真人作者;按已合并 PR、其他 PR、Issue 报告排序,黄圈标记显著贡献。
**42 位社区贡献者**共同建设 SILO、Console、mcli、公共包与相关项目。名单包含维护者,以及所有提出 Issue 或 PR 的真人作者;按已合并 PR、其他 PR、Issue 报告排序,黄圈标记显著贡献。
<p align="center">
<a href="https://github.com/Vonng"><img src="https://silo.pgsty.com/images/contributors/Vonng.svg" width="60" height="60" alt="@Vonng" title="@Vonng — 维护 SILO、Console、mcli、公共包、发行与文档"></a>
@@ -123,6 +123,7 @@ S3 API、`MINIO_*` 环境变量、`minio_*` 指标、`x-minio-*` 头、`/minio/*
<a href="https://github.com/cbornet"><img src="https://silo.pgsty.com/images/contributors/cbornet.svg" width="60" height="60" alt="@cbornet" title="@cbornet — 报告分片与流式校验和缺陷及缺失桶语义问题"></a>
<a href="https://github.com/vampywiz17"><img src="https://silo.pgsty.com/images/contributors/vampywiz17.svg" width="60" height="60" alt="@vampywiz17" title="@vampywiz17 — 报告 LDAP TLS 与 Console 登录回归"></a>
<a href="https://github.com/orenyomtov"><img src="https://silo.pgsty.com/images/contributors/orenyomtov.svg" width="60" height="60" alt="@orenyomtov" title="@orenyomtov — 报告未签名头导致的 CopyObject 跨对象读取(SN-2026-011"></a>
<a href="https://github.com/jiri-pejchal"><img src="https://silo.pgsty.com/images/contributors/jiri-pejchal.svg" width="60" height="60" alt="@jiri-pejchal" title="@jiri-pejchal — 报告 Console 分享代理暴露内部指标的问题"></a>
<a href="https://github.com/mumu-lab"><img src="https://silo.pgsty.com/images/contributors/mumu-lab.svg" width="48" height="48" alt="@mumu-lab" title="@mumu-lab — 报告桶配额指标读取已弃用字段的问题"></a>
<a href="https://github.com/jvasile"><img src="https://silo.pgsty.com/images/contributors/jvasile.svg" width="48" height="48" alt="@jvasile" title="@jvasile — 报告 Debian 包缺少用户、用户组与默认配置"></a>
<a href="https://github.com/pmezhuev"><img src="https://silo.pgsty.com/images/contributors/pmezhuev.svg" width="48" height="48" alt="@pmezhuev" title="@pmezhuev — 报告 RPM 包缺少 GPG 签名"></a>
@@ -0,0 +1,721 @@
// Copyright (c) 2026 Feng Ruohang
//
// This file is part of Silo Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cmd
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"fmt"
"maps"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/kms"
)
// Change allocation capacity only; metadata and data use real fixture disks.
// Restoring the adapters also lets encrypted fixtures move the write target
// between requests without changing the production allocation policy.
type conditionalPutCapacityDisk struct {
StorageAPI
full bool
}
func (d conditionalPutCapacityDisk) DiskInfo(ctx context.Context, opts DiskInfoOptions) (DiskInfo, error) {
info, err := d.StorageAPI.DiskInfo(ctx, opts)
info.Total, info.Used = 1<<40, 0
if d.full {
info.Used = info.Total - (1 << 20)
}
info.Free = info.Total - info.Used
return info, err
}
// Keep getDisks immutable while background IAM and storage readers use it.
// GetDisks takes this same mutex when it copies the backing disk list.
func conditionalPutSwapDisks(pool *erasureSets, object string, wrap func(StorageAPI) StorageAPI) func() {
setIndex := pool.getHashedSet(object).setIndex
pool.erasureDisksMu.Lock()
previous := pool.erasureDisks[setIndex]
disks := append([]StorageAPI(nil), previous...)
for i, disk := range disks {
disks[i] = wrap(disk)
}
pool.erasureDisks[setIndex] = disks
pool.erasureDisksMu.Unlock()
return func() {
pool.erasureDisksMu.Lock()
pool.erasureDisks[setIndex] = previous
pool.erasureDisksMu.Unlock()
}
}
func conditionalPutPool(t *testing.T, z *erasureServerPools, object string, target int) func() {
t.Helper()
var restore []func()
for i, pool := range z.serverPools {
restore = append(restore, conditionalPutSwapDisks(pool, object, func(disk StorageAPI) StorageAPI {
return conditionalPutCapacityDisk{StorageAPI: disk, full: i != target}
}))
}
return func() {
for _, fn := range restore {
fn()
}
}
}
func conditionalPutBucket(t *testing.T, z *erasureServerPools, mode string) (string, http.Handler) {
t.Helper()
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{VersioningEnabled: mode != "unversioned"})
if err != nil {
t.Fatal(err)
}
if mode == "suspended" {
if _, err := globalBucketMetadataSys.Update(t.Context(), bucket, bucketVersioningConfig,
[]byte(`<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>`)); err != nil {
t.Fatal(err)
}
}
return bucket, router
}
func TestPoolsConditionalPutHTTP(t *testing.T) {
for _, mode := range []string{"unversioned", "versioned", "suspended"} {
t.Run(mode, func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, mode)
for target := range 2 {
for _, tc := range []struct {
name string
oldCopy, localCurrent bool
condition string
status int
}{
{"stale-etag-accepted", true, false, "old", http.StatusPreconditionFailed},
{"current-etag-rejected", true, false, "current", http.StatusOK},
{"create-only-overwrites-other-pool", false, false, "none", http.StatusPreconditionFailed},
{"current-etag-missing-in-write-pool", false, false, "current", http.StatusOK},
{"control-current-in-write-pool", true, true, "current", http.StatusOK},
{"control-stale-etag-rejected", true, true, "old", http.StatusPreconditionFailed},
{"none-match-current-etag", true, false, "none-current", http.StatusPreconditionFailed},
{"none-match-old-etag", true, false, "none-old", http.StatusOK},
} {
t.Run(fmt.Sprintf("target=%d/%s", target, tc.name), func(t *testing.T) {
object := fmt.Sprintf("%d-%s", target, tc.name)
currentPool := 1 - target
if tc.localCurrent {
currentPool = target
}
opts := ObjectOptions{Versioned: mode == "versioned", VersionSuspended: mode == "suspended", MTime: UTCNow().Add(-time.Hour)}
oldETag := "absent-old"
if tc.oldCopy {
oldETag = putConsistencyObject(t, z, bucket, object, 1-currentPool, "old", opts).ETag
}
opts.MTime = UTCNow().Add(-time.Minute)
current := putConsistencyObject(t, z, bucket, object, currentPool, "current", opts)
defer conditionalPutPool(t, z, object, target)()
idx, err := z.getWritePoolIdx(t.Context(), bucket, object, 11, false)
if err != nil || idx != target {
t.Fatalf("allocation target=%d: idx=%d err=%v", target, idx, err)
}
headers := map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", current.ETag)}
if tc.condition == "old" {
headers[xhttp.IfMatch] = fmt.Sprintf("%q", oldETag)
}
if tc.condition == "none" {
headers = map[string]string{xhttp.IfNoneMatch: "*"}
}
if tc.condition == "none-current" {
headers = map[string]string{xhttp.IfNoneMatch: fmt.Sprintf("%q", current.ETag)}
}
if tc.condition == "none-old" {
headers = map[string]string{xhttp.IfNoneMatch: fmt.Sprintf("%q", oldETag)}
}
url := getPutObjectURL("", bucket, object)
before := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if before.Code != http.StatusOK || before.Body.String() != "current" || multipartConditionResponseETag(before) != current.ETag {
t.Fatalf("invalid current object: %d %q %v", before.Code, before.Body.String(), before.Header())
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != tc.status {
t.Errorf("PUT status=%d want=%d: %s", put.Code, tc.status, put.Body.String())
}
if put.Code == http.StatusPreconditionFailed {
multipartConditionError(t, put, "PreconditionFailed")
if multipartConditionResponseETag(put) != current.ETag || put.Header().Get(xhttp.LastModified) != before.Header().Get(xhttp.LastModified) {
t.Errorf("412 headers do not describe current object: %v", put.Header())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
wantBody, wantETag := "current", current.ETag
if tc.status == http.StatusOK {
wantBody, wantETag = "replacement", fmt.Sprintf("%x", md5.Sum([]byte("replacement")))
}
t.Logf("PUT %d; GET %d bytes=%q ETag=%s", put.Code, get.Code, get.Body.String(), multipartConditionResponseETag(get))
if get.Code != http.StatusOK || get.Body.String() != wantBody || multipartConditionResponseETag(get) != wantETag {
t.Errorf("GET=%d bytes=%q ETag=%s; want %q %s", get.Code, get.Body.String(), multipartConditionResponseETag(get), wantBody, wantETag)
}
})
}
}
})
}
}
func TestPoolsConditionalPutHTTPAbsence(t *testing.T) {
for _, state := range []string{"missing", "uuid-marker", "null-marker"} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/match=%t", state, match), func(t *testing.T) {
z, _ := consistencyPools(t)
mode := "versioned"
if state == "null-marker" {
mode = "suspended"
}
bucket, router := conditionalPutBucket(t, z, mode)
object := "absent-key"
if state != "missing" {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
vid := mustGetUUID()
if state == "null-marker" {
vid = nullVersionID
}
if _, err := z.serverPools[1].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: vid, DeleteMarker: true, MTime: UTCNow().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
defer conditionalPutPool(t, z, object, 0)()
headers := map[string]string{xhttp.IfNoneMatch: "*"}
want := http.StatusOK
if match {
headers = map[string]string{xhttp.IfMatch: "*"}
want = http.StatusNotFound
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url, "new", headers)
if put.Code != want {
t.Fatalf("PUT %d want %d: %s", put.Code, want, put.Body.String())
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if match {
multipartConditionError(t, put, "NoSuchKey")
if get.Code != http.StatusNotFound {
t.Fatalf("failed PUT exposed data: %d %s", get.Code, get.Body.String())
}
} else if get.Code != http.StatusOK || get.Body.String() != "new" || multipartConditionResponseETag(get) != multipartConditionResponseETag(put) {
t.Fatalf("successful create GET: %d %q %v", get.Code, get.Body.String(), get.Header())
}
})
}
}
}
func TestPoolsConditionalPutUnreadable(t *testing.T) {
for faultPool := range 2 {
for _, present := range []bool{false, true} {
for _, match := range []bool{false, true} {
t.Run(fmt.Sprintf("fault-pool=%d/present=%t/match=%t", faultPool, present, match), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
object := "unreadable-key"
if present {
putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{MTime: UTCNow().Add(-time.Hour)})
putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
}
defer conditionalPutPool(t, z, object, 0)()
restoreFault := conditionalPutSwapDisks(z.serverPools[faultPool], object, func(disk StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
})
defer restoreFault()
headers := map[string]string{xhttp.IfNoneMatch: "*"}
if match {
headers = map[string]string{xhttp.IfMatch: "*"}
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != http.StatusServiceUnavailable {
t.Errorf("unverified PUT must fail: %d %s", put.Code, put.Body.String())
}
called := 0
_, err := z.PutObject(t.Context(), bucket, object, mustGetPutObjReader(t, strings.NewReader("replacement"), 11, "", ""), ObjectOptions{HasIfMatch: match, CheckPrecondFn: func(ObjectInfo) bool { called++; return false }})
if !isErrReadQuorum(err) || called != 0 {
t.Errorf("lookup error=%v callback calls=%d", err, called)
}
restoreFault()
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if present {
if get.Code != http.StatusOK || get.Body.String() != "current" || multipartConditionResponseETag(get) != fmt.Sprintf("%x", md5.Sum([]byte("current"))) {
t.Fatalf("failed PUT changed object: %d %q", get.Code, get.Body.String())
}
} else if get.Code != http.StatusNotFound {
t.Fatalf("failed PUT created object: %d %q", get.Code, get.Body.String())
}
})
}
}
}
}
func TestPoolsConditionalPutEncryptedETag(t *testing.T) {
for _, kind := range []string{"SSE-C", "SSE-S3", "SSE-KMS"} {
t.Run(kind, func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
oldKMS, oldTLS := GlobalKMS, globalIsTLS
GlobalKMS, globalIsTLS = kms.NewStub("conditional-put-key"), true
defer func() { GlobalKMS, globalIsTLS = oldKMS, oldTLS }()
headers := map[string]string{xhttp.AmzServerSideEncryption: xhttp.AmzEncryptionAES}
readHeaders := map[string]string{}
if kind == "SSE-C" {
key := bytes.Repeat([]byte{0x42}, 32)
digest := md5.Sum(key)
headers = map[string]string{
xhttp.AmzServerSideEncryptionCustomerAlgorithm: xhttp.AmzEncryptionAES,
xhttp.AmzServerSideEncryptionCustomerKey: base64.StdEncoding.EncodeToString(key),
xhttp.AmzServerSideEncryptionCustomerKeyMD5: base64.StdEncoding.EncodeToString(digest[:]),
}
readHeaders = maps.Clone(headers)
}
if kind == "SSE-KMS" {
headers[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionKMS
headers[xhttp.AmzServerSideEncryptionKmsID] = "conditional-put-key"
}
object := "encrypted-key"
url := getPutObjectURL("", bucket, object)
restore := conditionalPutPool(t, z, object, 0)
old := multipartConditionRequest(t, router, http.MethodPut, url, "old", headers)
restore()
restore = conditionalPutPool(t, z, object, 1)
current := multipartConditionRequest(t, router, http.MethodPut, url, "current", headers)
restore()
if old.Code != http.StatusOK || current.Code != http.StatusOK {
t.Fatalf("encrypted setup: %d %s / %d %s", old.Code, old.Body.String(), current.Code, current.Body.String())
}
defer conditionalPutPool(t, z, object, 0)()
for _, stale := range []bool{true, false} {
h := maps.Clone(headers)
h[xhttp.IfMatch] = fmt.Sprintf("%q", multipartConditionResponseETag(current))
wantStatus, wantBody, wantETag := http.StatusOK, "replacement", ""
if stale {
h[xhttp.IfMatch] = fmt.Sprintf("%q", multipartConditionResponseETag(old))
wantStatus, wantBody, wantETag = http.StatusPreconditionFailed, "current", multipartConditionResponseETag(current)
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", h)
if put.Code != wantStatus {
t.Fatalf("encrypted condition: %d want %d: %s", put.Code, wantStatus, put.Body.String())
}
if !stale {
wantETag = multipartConditionResponseETag(put)
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", readHeaders)
if get.Code != http.StatusOK || get.Body.String() != wantBody || multipartConditionResponseETag(get) != wantETag {
t.Fatalf("encrypted GET: %d %q %v", get.Code, get.Body.String(), get.Header())
}
}
})
}
}
func TestPoolsConditionalPutVersionSelection(t *testing.T) {
for _, kind := range []string{"public-version", "replica", "replica-preserve-etag", "movement", "no-lock", "tie", "draining"} {
t.Run(kind, func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "version-selection"
addressed := putConsistencyObject(t, z, bucket, object, 1, "addressed", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
opts := ObjectOptions{Versioned: true, VersionID: addressed.VersionID, HasIfMatch: true}
want := current
switch kind {
case "replica":
opts.ReplicaLockReconcile, opts.ReplicationRequest = true, true
want = addressed
case "replica-preserve-etag":
opts.PreserveETag = addressed.ETag
opts.ReplicaLockReconcile, opts.ReplicationRequest = true, true
want = addressed
case "movement":
opts.DataMovement, opts.SrcPoolIdx = true, 1
want = addressed
case "tie":
want = putConsistencyObject(t, z, bucket, object, 0, "tie-winner", ObjectOptions{Versioned: true, MTime: current.ModTime})
case "draining":
z.poolMetaMutex.Lock()
z.poolMeta.Pools[1].Decommission = &PoolDecommissionInfo{}
z.poolMetaMutex.Unlock()
}
defer conditionalPutPool(t, z, object, 0)()
ctx := t.Context()
if kind == "no-lock" {
lk := z.NewNSLock(bucket, object)
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
if err != nil {
t.Fatal(err)
}
defer lk.Unlock(lkctx)
ctx, opts.NoLock = lkctx.Context(), true
}
called := 0
opts.UserDefined = make(map[string]string)
opts.CheckPrecondFn = func(oi ObjectInfo) bool {
called++
if oi.ETag != want.ETag || oi.VersionID != want.VersionID {
t.Errorf("comparison ETag/version=%s/%s want %s/%s", oi.ETag, oi.VersionID, want.ETag, want.VersionID)
}
return oi.ETag != want.ETag
}
oi, err := z.PutObject(ctx, bucket, object, mustGetPutObjReader(t, strings.NewReader("replacement"), 11, "", ""), opts)
if err != nil || called != 1 {
t.Fatalf("PUT err=%v callback calls=%d", err, called)
}
if oi.VersionID != addressed.VersionID {
t.Fatalf("destination version changed: %s", oi.VersionID)
}
if opts.PreserveETag != "" && oi.ETag != opts.PreserveETag {
t.Fatalf("PreserveETag changed: %s", oi.ETag)
}
})
}
}
func TestPoolsConditionalPutReplicaDuplicateHTTP(t *testing.T) {
for _, null := range []bool{false, true} {
t.Run(fmt.Sprintf("null=%t", null), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "replica-duplicate"
vid := mustGetUUID()
if null {
vid = nullVersionID
}
addressed := putConsistencyObject(t, z, bucket, object, 1, "addressed", ObjectOptions{Versioned: true, VersionID: vid, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
headers := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA",
xhttp.MinIOSourceETag: addressed.ETag,
xhttp.MinIOSourceMTime: addressed.ModTime.Format(time.RFC3339Nano),
}
url := getPutObjectURL("", bucket, object)
put := multipartConditionRequest(t, router, http.MethodPut, url+"?versionId="+vid, "addressed", headers)
if put.Code != http.StatusPreconditionFailed {
t.Fatalf("replica duplicate: %d %s", put.Code, put.Body.String())
}
multipartConditionError(t, put, "PreconditionFailed")
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "current" || multipartConditionResponseETag(get) != current.ETag {
t.Fatalf("duplicate changed current object: %d %q", get.Code, get.Body.String())
}
get = multipartConditionRequest(t, router, http.MethodGet, url+"?versionId="+vid, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "addressed" || multipartConditionResponseETag(get) != addressed.ETag {
t.Fatalf("duplicate changed addressed version: %d %q", get.Code, get.Body.String())
}
})
}
}
func TestPoolsConditionalPutConcurrentHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "unversioned")
for _, match := range []bool{false, true} {
for iteration := range 5 {
t.Run(fmt.Sprintf("match=%t/iteration=%d", match, iteration), func(t *testing.T) {
object := fmt.Sprintf("concurrent-%t-%d", match, iteration)
headers := map[string]string{xhttp.IfNoneMatch: "*"}
if match {
oi := putConsistencyObject(t, z, bucket, object, 1, "old", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
headers = map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", oi.ETag)}
}
defer conditionalPutPool(t, z, object, 0)()
start := make(chan struct{})
results := make(chan *httptest.ResponseRecorder, 2)
url := getPutObjectURL("", bucket, object)
for i := range 2 {
body := fmt.Sprintf("writer-%d", i)
req, err := newTestSignedRequestV4(http.MethodPut, url, int64(len(body)), strings.NewReader(body), globalActiveCred.AccessKey, globalActiveCred.SecretKey, headers)
if err != nil {
t.Fatal(err)
}
go func() {
<-start
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
results <- rec
}()
}
close(start)
success, failed, winnerETag := 0, 0, ""
for range 2 {
result := <-results
switch result.Code {
case http.StatusOK:
success++
winnerETag = multipartConditionResponseETag(result)
case http.StatusPreconditionFailed:
failed++
multipartConditionError(t, result, "PreconditionFailed")
default:
t.Errorf("unexpected PUT %d: %s", result.Code, result.Body.String())
}
}
if success != 1 || failed != 1 {
t.Fatalf("success=%d precondition failures=%d", success, failed)
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || multipartConditionResponseETag(get) != winnerETag || fmt.Sprintf("%x", md5.Sum(get.Body.Bytes())) != winnerETag {
t.Fatalf("winner lost: %d %q %v", get.Code, get.Body.String(), get.Header())
}
})
}
}
}
func TestPoolsConditionalPutSerializesMutation(t *testing.T) {
for _, deletion := range []bool{false, true} {
t.Run(fmt.Sprintf("delete=%t", deletion), func(t *testing.T) {
z, bucket := consistencyPools(t)
object := "conditional-mutation"
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
defer cancel()
gate := &consistencyGateReader{Reader: strings.NewReader("replacement"), entered: make(chan struct{}), resume: make(chan struct{})}
release := func() { gate.release.Do(func() { close(gate.resume) }) }
defer release()
reader := mustGetPutObjReader(t, gate, 11, "", "")
written := make(chan error, 1)
go func() {
_, err := z.PutObject(ctx, bucket, object, reader, ObjectOptions{HasIfMatch: true, CheckPrecondFn: func(oi ObjectInfo) bool { return oi.ETag != current.ETag }})
written <- err
}()
select {
case <-gate.entered:
case err := <-written:
t.Fatalf("PUT failed before body read: %v", err)
case <-ctx.Done():
t.Fatal(ctx.Err())
}
mutated := make(chan error, 1)
go func() {
var err error
if deletion {
_, err = z.DeleteObject(ctx, bucket, object, ObjectOptions{})
} else {
_, err = z.PutObjectMetadata(ctx, bucket, object, ObjectOptions{EvalMetadataFn: func(oi *ObjectInfo, _ error) (ReplicateDecision, error) {
oi.UserDefined["x-amz-meta-after-put"] = "present"
return ReplicateDecision{}, nil
}})
}
mutated <- err
}()
select {
case err := <-mutated:
t.Fatalf("mutation escaped PUT lock: %v", err)
case <-time.After(100 * time.Millisecond):
}
release()
if err := <-written; err != nil {
t.Fatal(err)
}
if err := <-mutated; err != nil {
t.Fatal(err)
}
oi, err := z.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
if deletion {
if !isErrObjectNotFound(err) {
t.Fatalf("delete lost: %v", err)
}
} else if err != nil || oi.UserDefined["x-amz-meta-after-put"] != "present" || oi.ETag != fmt.Sprintf("%x", md5.Sum([]byte("replacement"))) {
t.Fatalf("metadata/PUT lost: %+v %v", oi, err)
}
})
}
}
func TestSinglePoolConditionalPutHTTP(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
obj, dirs, err := prepareErasure16(ctx)
if err != nil {
cancel()
t.Fatal(err)
}
z := obj.(*erasureServerPools)
t.Cleanup(func() { cancel(); z.Shutdown(context.Background()); removeRoots(dirs) })
if !z.SinglePool() {
t.Fatal("fixture is not a single pool")
}
bucket, router := conditionalPutBucket(t, z, "unversioned")
object := "single-pool-condition"
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object)
for _, tc := range []struct {
body, match, none string
status int
}{
{"missing", "*", "", http.StatusNotFound},
{"first", "", "*", http.StatusOK},
{"blocked", "", "*", http.StatusPreconditionFailed},
{"blocked", "stale", "", http.StatusPreconditionFailed},
{"second", fmt.Sprintf("%x", md5.Sum([]byte("first"))), "", http.StatusOK},
} {
rec := multipartConditionRequest(t, router, http.MethodPut, url, tc.body, map[string]string{xhttp.IfMatch: tc.match, xhttp.IfNoneMatch: tc.none})
if rec.Code != tc.status {
t.Fatalf("PUT %d want %d: %s", rec.Code, tc.status, rec.Body.String())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "second" {
t.Fatalf("single-pool GET: %d %q", get.Code, get.Body.String())
}
}
// An internal replica callback without an addressed version is not a public
// condition. Preserve its availability when another pool is unreadable. An
// addressed replica already requires all pools for lock/tag reconciliation.
func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
for _, addressed := range []bool{false, true} {
t.Run(fmt.Sprintf("addressed=%t", addressed), func(t *testing.T) {
z, _ := consistencyPools(t)
mode := "unversioned"
if addressed {
mode = "versioned"
}
bucket, router := conditionalPutBucket(t, z, mode)
object := "replica-availability"
oi := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: addressed, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
restoreFault := conditionalPutSwapDisks(z.serverPools[1], object, func(disk StorageAPI) StorageAPI {
return consistencyReadFaultDisk{StorageAPI: disk, bucket: bucket, object: object}
})
defer restoreFault()
headers := map[string]string{
xhttp.MinIOSourceReplicationRequest: "true",
xhttp.AmzBucketReplicationStatus: "REPLICA",
xhttp.MinIOSourceETag: fmt.Sprintf("%x", md5.Sum([]byte("replacement"))),
}
url := getPutObjectURL("", bucket, object)
wantStatus, wantBody := http.StatusOK, "replacement"
if addressed {
url += "?versionId=" + oi.VersionID
wantStatus, wantBody = http.StatusServiceUnavailable, "old"
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
if put.Code != wantStatus {
t.Fatalf("replica PUT: %d want %d: %s", put.Code, wantStatus, put.Body.String())
}
restoreFault()
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != wantBody {
t.Fatalf("replica GET: %d %q", get.Code, get.Body.String())
}
})
}
}
func TestPoolsConditionalPutDestinationVersionHTTP(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "client-version"
old := putConsistencyObject(t, z, bucket, object, 0, "old", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
current := putConsistencyObject(t, z, bucket, object, 1, "current", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Minute)})
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object) + "?versionId=" + old.VersionID
for _, stale := range []bool{true, false} {
tag, status := current.ETag, http.StatusOK
if stale {
tag, status = old.ETag, http.StatusPreconditionFailed
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfMatch: fmt.Sprintf("%q", tag)})
if put.Code != status {
t.Fatalf("version-addressed public PUT: %d want %d: %s", put.Code, status, put.Body.String())
}
if got := put.Header()[xhttp.AmzVersionID]; !stale && (len(got) != 1 || got[0] != old.VersionID) {
t.Fatalf("write version changed: %v", put.Header())
}
}
get := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
if get.Code != http.StatusOK || get.Body.String() != "replacement" {
t.Fatalf("addressed GET: %d %q", get.Code, get.Body.String())
}
}
func TestPoolsConditionalPutDeleteMarkerTie(t *testing.T) {
for markerPool := range 2 {
t.Run(fmt.Sprintf("marker-pool=%d", markerPool), func(t *testing.T) {
z, _ := consistencyPools(t)
bucket, router := conditionalPutBucket(t, z, "versioned")
object := "marker-tie"
mtime := UTCNow().Add(-time.Minute)
putConsistencyObject(t, z, bucket, object, 1-markerPool, "live", ObjectOptions{Versioned: true, MTime: mtime})
if _, err := z.serverPools[markerPool].DeleteObject(t.Context(), bucket, object, ObjectOptions{Versioned: true, VersionID: mustGetUUID(), DeleteMarker: true, MTime: mtime}); err != nil {
t.Fatal(err)
}
defer conditionalPutPool(t, z, object, 0)()
url := getPutObjectURL("", bucket, object)
before := multipartConditionRequest(t, router, http.MethodGet, url, "", nil)
want := http.StatusPreconditionFailed
if markerPool == 0 {
want = http.StatusOK
if before.Code != http.StatusNotFound {
t.Fatalf("GET tie: %d", before.Code)
}
} else if before.Code != http.StatusOK {
t.Fatalf("GET tie: %d", before.Code)
}
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", map[string]string{xhttp.IfNoneMatch: "*"})
if put.Code != want {
t.Fatalf("PUT tie: %d want %d: %s", put.Code, want, put.Body.String())
}
})
}
}
// Overlap fixture changes with the real IAM Walk reader instead of relying on
// its periodic refresh timer to expose an unsynchronized disk-adapter swap.
func TestPoolsConditionalPutFixtureConcurrentIAM(t *testing.T) {
z, _ := consistencyPools(t)
conditionalPutBucket(t, z, "unversioned")
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
started, finished := make(chan struct{}), make(chan error, 1)
iam := globalIAMSys
go func() {
close(started)
for range 20 {
if err := iam.Load(ctx, false); err != nil {
finished <- err
return
}
}
finished <- nil
}()
<-started
for i := range 5000 {
restore := conditionalPutPool(t, z, "fixture-concurrent-iam", i%2)
restore()
}
if err := <-finished; err != nil {
t.Fatal(err)
}
}
+34
View File
@@ -1149,6 +1149,40 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
}
opts.NoLock = true
// Public write conditions compare the logical current object while the
// pools-layer write lock is held. The destination selected by capacity may
// be empty or stale, and draining pools can still hold the current object.
// Replica callbacks retain their existing addressed-version semantics and
// metadata reconciliation at the set layer.
if opts.CheckPrecondFn != nil && !opts.ReplicationRequest &&
!opts.ReplicaLockReconcile && !opts.DataMovement {
copies, lerr := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{
VersionID: "", // Compare the current object, not the write's version.
Versioned: opts.Versioned,
VersionSuspended: opts.VersionSuspended,
NoAuditLog: true,
})
var latest ObjectInfo
if lerr == nil {
latest = copies[0].ObjInfo
if latest.DeleteMarker {
lerr = toObjectErr(errFileNotFound, bucket, object)
}
}
// An unreadable pool may hold the newest object; it is not absence.
if lerr != nil && !isErrObjectNotFound(lerr) && !isErrVersionNotFound(lerr) {
return ObjectInfo{}, lerr
}
if lerr == nil && opts.CheckPrecondFn(latest) {
return ObjectInfo{}, PreConditionFailed{}
}
if lerr != nil && opts.HasIfMatch {
return ObjectInfo{}, lerr
}
// Do not repeat an accepted condition against the destination's copy.
opts.CheckPrecondFn = nil
}
idx, err := z.getWritePoolIdx(ctx, bucket, object, data.Size(), true)
if err != nil {
return ObjectInfo{}, err
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,8 +58,8 @@ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/
silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
./mc ready sitea
./mc ready siteb
@@ -83,7 +83,7 @@ done
echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket
remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket)
sleep 1
@@ -137,7 +137,7 @@ fi
echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket-version/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket-version
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket-version
./mc mb sitea/bucket-version/directory/
@@ -37,7 +37,7 @@ set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -70,9 +70,9 @@ silo server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/
silo server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo12345@127.0.0.1:9006
./mc ready sitea
./mc ready siteb
@@ -93,73 +93,73 @@ export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
echo "adding replication rule for a -> b : ${remote_arn}"
sleep 1
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for b -> a : ${remote_arn}"
./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for a -> c : ${remote_arn}"
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for c -> a : ${remote_arn}"
./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for b -> c : ${remote_arn}"
./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for c -> b : ${remote_arn}"
./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for olockbucket a -> b : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for olockbucket b -> a : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for olockbucket a -> c : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for olockbucket c -> a : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for olockbucket b -> c : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for olockbucket c -> b : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \
--remote-bucket http://minio:silo12345@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
@@ -204,33 +204,33 @@ head -c 221227088 </dev/urandom >200M
sleep 10
echo "Verifying ETag for all objects"
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
# additional tests for encryption object alignment
go install -v github.com/minio/multipart-debug@latest
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo12345 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt
dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt
sudo apt install jq -y
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo12345 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo12345 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo12345 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
sleep 10
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -70,10 +70,10 @@ silo server --address 127.0.0.1:9008 "http://127.0.0.1:9007/tmp/multisited/data/
# Wait to make sure all Silo instances are up
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008
export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo12345@127.0.0.1:9006
export MC_HOST_sited=http://minio:silo12345@127.0.0.1:9008
./mc ready sitea
./mc ready siteb
@@ -89,7 +89,7 @@ export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008
sleep 10s
## Add warm tier
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo123 --bucket bucket
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo12345 --bucket bucket
## Add ILM rules
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2"
+2 -2
View File
@@ -38,7 +38,7 @@ pid=$!
mc ready mysilo
mc admin user add mysilo/ silo123 silo123
mc admin user add mysilo/ silo123 silo12345
mc admin policy create mysilo/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json
mc admin policy create mysilo/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json
@@ -50,7 +50,7 @@ mc admin policy attach mysilo consoleAdmin --user silo123
mc mb -l mysilo/test-bucket
mc mb -l mysilo/multi-key-poc
export MC_HOST_mysilo1="http://silo123:silo123@localhost:9000/"
export MC_HOST_mysilo1="http://silo123:silo12345@localhost:9000/"
mc cp /etc/issue mysilo1/test-bucket
ret=$?
+4 -4
View File
@@ -34,7 +34,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,9 +58,9 @@ site2_pid=$!
silo server --config-dir /tmp/silo-ldap --address ":9003" /tmp/silo-ldap-idp3/{1...4} >/tmp/silo3_1.log 2>&1 &
site3_pid=$!
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
export MC_HOST_silo1=http://minio:silo12345@localhost:9001
export MC_HOST_silo2=http://minio:silo12345@localhost:9002
export MC_HOST_silo3=http://minio:silo12345@localhost:9003
./mc ready silo1
./mc ready silo2
+4 -4
View File
@@ -31,7 +31,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -56,9 +56,9 @@ if [ ! -f ./mc ]; then
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
export MC_HOST_silo1=http://minio:silo12345@localhost:9001
export MC_HOST_silo2=http://minio:silo12345@localhost:9002
export MC_HOST_silo3=http://minio:silo12345@localhost:9003
./mc ready silo1
./mc ready silo2
@@ -34,7 +34,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,13 +58,13 @@ site3_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9030" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_2.log 2>&1 &
site3_pid2=$!
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
export MC_HOST_silo1=http://minio:silo12345@localhost:9001
export MC_HOST_silo2=http://minio:silo12345@localhost:9002
export MC_HOST_silo3=http://minio:silo12345@localhost:9003
export MC_HOST_silo10=http://minio:silo123@localhost:9010
export MC_HOST_silo20=http://minio:silo123@localhost:9020
export MC_HOST_silo30=http://minio:silo123@localhost:9030
export MC_HOST_silo10=http://minio:silo12345@localhost:9010
export MC_HOST_silo20=http://minio:silo12345@localhost:9020
export MC_HOST_silo30=http://minio:silo12345@localhost:9030
./mc ready silo1
./mc ready silo2
@@ -33,7 +33,7 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
# Download AWS CLI
echo -n "Download and install AWS CLI"
@@ -51,7 +51,7 @@ cat >~/.aws/credentials <<EOF
[enterprise]
region = us-east-1
aws_access_key_id = minio
aws_secret_access_key = silo123
aws_secret_access_key = silo12345
EOF
# Create certificates for TLS enabled Silo
@@ -65,8 +65,8 @@ echo "done"
# Start Silo instances
echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
@@ -75,8 +75,8 @@ if [ ! -f ./mc ]; then
echo "done"
fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure
./mc ready silo2 --insecure
@@ -29,7 +29,7 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled Silo
@@ -43,8 +43,8 @@ echo "done"
# Start Silo instances
echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
@@ -53,8 +53,8 @@ if [ ! -f ./mc ]; then
echo "done"
fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure
./mc ready silo2 --insecure
@@ -232,12 +232,12 @@ fi
./mc cat silo2/test-bucket/custpartsize --insecure >/dev/null || exit_1
echo -n "Starting Silo instances with different kms key ..."
CI=on MINIO_KMS_SECRET_KEY=silo3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/silo3/disk{1...4} >/tmp/silo3_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=silo4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/silo4/disk{1...4} >/tmp/silo4_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=silo3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/silo3/disk{1...4} >/tmp/silo3_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=silo4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/silo4/disk{1...4} >/tmp/silo4_1.log 2>&1 &
echo "done"
export MC_HOST_silo3=https://minio:silo123@localhost:9003
export MC_HOST_silo4=https://minio:silo123@localhost:9004
export MC_HOST_silo3=https://minio:silo12345@localhost:9003
export MC_HOST_silo4=https://minio:silo12345@localhost:9004
./mc ready silo3 --insecure
./mc ready silo4 --insecure
@@ -25,7 +25,7 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled Silo
@@ -49,8 +49,8 @@ if [ ! -f ./mc ]; then
echo "done"
fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure
./mc ready silo2 --insecure
@@ -25,7 +25,7 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_ROOT_PASSWORD="silo12345"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled Silo
@@ -49,8 +49,8 @@ if [ ! -f ./mc ]; then
echo "done"
fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure
./mc ready silo2 --insecure
+1 -1
View File
@@ -4,7 +4,7 @@ go 1.27.1
// Console and MC retain their historical module paths for best-effort upstream
// compatibility. Pin the maintained PGSTY implementations used by SILO.
replace github.com/minio/console => github.com/pgsty/silo-console v0.0.0-20260913015128-417559bb2c97
replace github.com/minio/console => github.com/pgsty/silo-console v0.0.0-20260916034812-56dfe455ac2f
replace github.com/minio/mc => github.com/pgsty/mc v0.0.0-20260913012246-4f609a4da3bb
+2 -2
View File
@@ -547,8 +547,8 @@ github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwp
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pgsty/mc v0.0.0-20260913012246-4f609a4da3bb h1:S7IAYBoKvFRqrw5MUsQE34/q4cKC4K7gUmnlEGxWtqo=
github.com/pgsty/mc v0.0.0-20260913012246-4f609a4da3bb/go.mod h1:kJN7dsWtSUhXd2vNPXdjDi/or6lJKlSLfb56cBfMckA=
github.com/pgsty/silo-console v0.0.0-20260913015128-417559bb2c97 h1:FppTgZy7ZPmAdWjGPi8IEXHSFI8aawze5p9r4XQy+pE=
github.com/pgsty/silo-console v0.0.0-20260913015128-417559bb2c97/go.mod h1:YtRQZ6jYXRUE03oPA+IMGtflQN6nCvDWdKtroA7tfKo=
github.com/pgsty/silo-console v0.0.0-20260916034812-56dfe455ac2f h1:ull9m/nXOEMfggKtMQP2idYXPh6l6chCUJ9hJHYyzDc=
github.com/pgsty/silo-console v0.0.0-20260916034812-56dfe455ac2f/go.mod h1:YtRQZ6jYXRUE03oPA+IMGtflQN6nCvDWdKtroA7tfKo=
github.com/pgsty/silo-pkg/v3 v3.14.0 h1:RCuVkzr6mdjbkV/Rv4OVO/XgdMBE0XYvUnT6GFuihFk=
github.com/pgsty/silo-pkg/v3 v3.14.0/go.mod h1:c26IoMVITlP1+Sirl0AsHwoijlsSC9wPpyCuvhb3Axc=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=