Compare commits

...

7 Commits

Author SHA1 Message Date
Feng Ruohang e791640dac docs: describe merged conditional PUT behavior and recovery guidance
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 11:21:16 +08:00
Feng Ruohang 9b4ae82a29 Merge pull request #207 from pgsty/codex/conditional-put-pools
fix(pools): evaluate cross-pool PUT conditions against the current object
2026-09-16 11:17:02 +08:00
Feng Ruohang fb7c406ddc Merge pull request #206 from pgsty/codex/main-consolidation-20260916
test: restore valid credentials in integration fixtures
2026-09-16 08:18:56 +08:00
Feng Ruohang 416826f61f Merge pull request #205 from pgsty/codex/release-notes-followups
docs: complete September correctness and upgrade notes
2026-09-16 08:16:23 +08:00
Feng Ruohang a2e2f3ee82 test: restore valid credentials in integration fixtures
Port the shell-fixture changes from ebc9937d97b27871dcc4bb91d4b5771d3550b76a. Match the existing minimum secret length consistently across startup, aliases and helper commands.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:08:08 +08:00
Feng Ruohang 70c7ec4a9f docs: link reviewed runbook sources before website deployment
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 08:04:53 +08:00
Feng Ruohang 2b722b7e92 docs: complete September correctness and upgrade notes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-09-16 07:51:11 +08:00
12 changed files with 141 additions and 83 deletions
+60 -2
View File
@@ -9,6 +9,25 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
### Authorization and security ### Authorization and security
- Persist IAM deletion revisions and parent revocation boundaries so stale site
events cannot restore deleted identities, policies or their older grants
(#191, #192). Peer deletion notifications reload committed storage; deliberate
recreation requires a newer revision, and credentials issued before the
parent's revocation remain invalid.
**Coordinated upgrade required:** upgrade every participating node and site.
Mixed old/new nodes sharing an IAM backend and rolling downgrade are
unsupported. Back up complete IAM storage and encryption material; an admin
export of live records omits deletion history. Reissue credentials for
recreated parents and explicitly reconcile pre-upgrade revocations whose
history is already lost. Restoring an older backup can lose later revocations;
keep affected sites isolated until reconciliation/rekeying is complete. See
[the operator runbook](https://github.com/pgsty/silo.pgsty.com/blob/7bd2d57c2ce5aaa804d0b1a2fe0e5eed69d15235/content/operations/replication/iam-upgrade.md).
- Enforce an absolute HTTP/1 request-header deadline through the connection
wrapper (#196). Repeated small reads no longer extend that deadline, and
`--read-header-timeout` / `MINIO_READ_HEADER_TIMEOUT` now reaches the HTTP
server. HTTP/1 request bodies retain the rolling idle timeout; this does not
impose a total upload/download duration. A shorter setting also constrains
TLS handshake reads. The wrapper's strict header mode is not applied to HTTP/2.
- Reject unsigned `x-amz-*` request headers that could turn a signed PUT into a - Reject unsigned `x-amz-*` request headers that could turn a signed PUT into a
copy of another object accessible to the signer (SN-2026-011). The latest copy of another object accessible to the signer (SN-2026-011). The latest
public Server is affected; the fix is on main. See [the advisory ledger](docs/security/advisories.md). public Server is affected; the fix is on main. See [the advisory ledger](docs/security/advisories.md).
@@ -22,17 +41,56 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
### Object storage and replication ### Object storage and replication
- Preserve object tags during multi-pool metadata reconciliation by reading the
resolved tag field together with its revision (#189). Previously, reconciliation
could replace existing tags with an empty value.
- Preserve the tag revision on SSE-KMS metadata replication (#193), and advance
tag revisions monotonically on local PUT/DELETE tagging (#196). Empty tags
participate in reconciliation as an ordered deletion, preventing older
events from restoring removed tags. SSE-C key rotation also retains the tag
revision. Malformed historical revisions can fail and retry; their missing
history is not reconstructed by the upgrade.
- Complete delete-marker version purges and preserve their identity and retry
state through MRF recovery (#196). Recovery accepts a 405 marker response only
when its version, bucket, object name and modification time match the task.
Purge audit status is normalized from `COMPLETE` to `COMPLETED`.
Thanks to Julien Laurenceau (@julienlau) for the investigation and proposed
fix in #184 that helped shape this follow-up.
- Restore only the six replication-specific metadata fields after ordinary
request metadata extraction (#194). This prevents transport-only `aws-chunked`
from being stored as Content-Encoding while preserving the signed-header
protections. Trusted Snowball entries no longer inherit the outer archive's
ordinary metadata. Thanks to Mikhail Khadarenka (@chodorenko) for the fix in #187.
**Existing data:** these repairs prevent new errors; they do not scan or rewrite
historical object metadata, recover lost tags or prove that old purge work has
converged. Follow the [read-only audit procedure](https://github.com/pgsty/silo.pgsty.com/blob/7bd2d57c2ce5aaa804d0b1a2fe0e5eed69d15235/content/operations/replication/replica-metadata-audit.md)
before planning any repair of stored state.
- Evaluate conditional multipart completion against the logical current object - Evaluate conditional multipart completion against the logical current object
across all pools while holding the existing object lock. A stale `If-Match` across all pools while holding the existing object lock. A stale `If-Match`
can no longer replace newer data in another pool, and the current ETag is no can no longer replace newer data in another pool, and the current ETag is no
longer rejected because the upload resides next to an older copy. Conditions longer rejected because the upload resides next to an older copy. Conditions
are evaluated once; a current delete marker counts as an absent object. are evaluated once; a current delete marker counts as an absent object.
**Availability change:** if metadata cannot be read from any pool, conditional **Availability change:** if any pool's metadata cannot be read, conditional
completion fails even when another pool can still serve GET/HEAD. This also completion fails even when another pool can still serve GET/HEAD. This also
applies when the unreadable pool may not hold the object: absence cannot be applies when the unreadable pool may not hold the object: absence cannot be
verified. Retry after the pool recovers. Unconditional completion and the verified. Retry after the pool recovers. Unconditional completion and the
single-pool path retain their existing behavior. single-pool path retain their existing behavior.
- Evaluate ordinary multi-pool conditional PUT against the logical current
object across all pools, including draining pools, under the existing object
lock (#207). A stale destination copy no longer accepts a stale ETag or rejects
the current one; a current delete marker is treated as absence.
**Availability change:** if any pool's object metadata cannot be verified,
the condition fails even when GET can use another pool; read-quorum failures
return 503. Restore readability or heal before retrying. Unconditional PUT,
single-pool conditions and internal replication retain their existing behavior.
A public condition with a destination `versionId` compares the current object
while preserving the requested write version. This change does not retire
stale copies in other pools, undo historical accepted overwrites or provide
a new global clock-ordering guarantee. The multipart-completion repair in #190
neither introduced nor repaired this separate PUT defect.
- Reconcile ordinary single-object version DELETE across all pools, including - Reconcile ordinary single-object version DELETE across all pools, including
null versions, delete markers and unqualified directory-marker DELETE. This null versions, delete markers and unqualified directory-marker DELETE. This
applies the deletion to every resolved pool copy under existing quorum applies the deletion to every resolved pool copy under existing quorum
@@ -61,7 +119,7 @@ and [complete commit range](https://github.com/pgsty/silo/compare/RELEASE.2026-0
- Repair federated CopyObject checksums, destination timestamps, reserved - Repair federated CopyObject checksums, destination timestamps, reserved
metadata, encrypted-object forwarding, legal hold and KMS context. metadata, encrypted-object forwarding, legal hold and KMS context.
- Make resync counters, target selection, cancellation and worker lifetimes - Make resync counters, target selection, cancellation and worker lifetimes
reflect actual work; complete delete-marker purges and report bounded MRF drops. reflect actual work, and report bounded MRF drops.
- Converge bucket metadata with deterministic source state, deletion tombstones, - Converge bucket metadata with deterministic source state, deletion tombstones,
creation time recovery and diagnostics. The mixed-version export gate requires creation time recovery and diagnostics. The mixed-version export gate requires
coordinated upgrades before tombstones are exported. See [the #77 record](docs/investigations/issue-77-current.md). coordinated upgrades before tombstones are exported. See [the #77 record](docs/investigations/issue-77-current.md).
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,8 +58,8 @@ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/
silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001 export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004 export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
./mc ready sitea ./mc ready sitea
./mc ready siteb ./mc ready siteb
@@ -83,7 +83,7 @@ done
echo "adding replication rule for site a -> site b" echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket/ \ ./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket --remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket
remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket) remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket)
sleep 1 sleep 1
@@ -137,7 +137,7 @@ fi
echo "adding replication rule for site a -> site b" echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket-version/ \ ./mc replicate add sitea/bucket-version/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket-version --remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket-version
./mc mb sitea/bucket-version/directory/ ./mc mb sitea/bucket-version/directory/
@@ -37,7 +37,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -70,9 +70,9 @@ silo server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/
silo server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 & "http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001 export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004 export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006 export MC_HOST_sitec=http://minio:silo12345@127.0.0.1:9006
./mc ready sitea ./mc ready sitea
./mc ready siteb ./mc ready siteb
@@ -93,73 +93,73 @@ export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
echo "adding replication rule for a -> b : ${remote_arn}" echo "adding replication rule for a -> b : ${remote_arn}"
sleep 1 sleep 1
./mc replicate add sitea/bucket/ \ ./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for b -> a : ${remote_arn}" echo "adding replication rule for b -> a : ${remote_arn}"
./mc replicate add siteb/bucket/ \ ./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for a -> c : ${remote_arn}" echo "adding replication rule for a -> c : ${remote_arn}"
./mc replicate add sitea/bucket/ \ ./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for c -> a : ${remote_arn}" echo "adding replication rule for c -> a : ${remote_arn}"
./mc replicate add sitec/bucket/ \ ./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for b -> c : ${remote_arn}" echo "adding replication rule for b -> c : ${remote_arn}"
./mc replicate add siteb/bucket/ \ ./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
echo "adding replication rule for c -> b : ${remote_arn}" echo "adding replication rule for c -> b : ${remote_arn}"
./mc replicate add sitec/bucket/ \ ./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
echo "adding replication rule for olockbucket a -> b : ${remote_arn}" echo "adding replication rule for olockbucket a -> b : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \ ./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for olockbucket b -> a : ${remote_arn}" echo "adding replication rule for olockbucket b -> a : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \ ./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for olockbucket a -> c : ${remote_arn}" echo "adding replication rule for olockbucket a -> c : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \ ./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for olockbucket c -> a : ${remote_arn}" echo "adding replication rule for olockbucket c -> a : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \ ./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for olockbucket b -> c : ${remote_arn}" echo "adding replication rule for olockbucket b -> c : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \ ./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
echo "adding replication rule for olockbucket c -> b : ${remote_arn}" echo "adding replication rule for olockbucket c -> b : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \ ./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \ --remote-bucket http://minio:silo12345@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
@@ -204,33 +204,33 @@ head -c 221227088 </dev/urandom >200M
sleep 10 sleep 10
echo "Verifying ETag for all objects" echo "Verifying ETag for all objects"
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo12345 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
# additional tests for encryption object alignment # additional tests for encryption object alignment
go install -v github.com/minio/multipart-debug@latest go install -v github.com/minio/multipart-debug@latest
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart new --bucket bucket --object new-test-encrypted-object --encrypt) upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo12345 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt
dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt
sudo apt install jq -y sudo apt install jq -y
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag) etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo12345 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag) etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo12345 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2} multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo12345 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
sleep 10 sleep 10
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -70,10 +70,10 @@ silo server --address 127.0.0.1:9008 "http://127.0.0.1:9007/tmp/multisited/data/
# Wait to make sure all Silo instances are up # Wait to make sure all Silo instances are up
export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001 export MC_HOST_sitea=http://minio:silo12345@127.0.0.1:9001
export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004 export MC_HOST_siteb=http://minio:silo12345@127.0.0.1:9004
export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006 export MC_HOST_sitec=http://minio:silo12345@127.0.0.1:9006
export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008 export MC_HOST_sited=http://minio:silo12345@127.0.0.1:9008
./mc ready sitea ./mc ready sitea
./mc ready siteb ./mc ready siteb
@@ -89,7 +89,7 @@ export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008
sleep 10s sleep 10s
## Add warm tier ## Add warm tier
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo123 --bucket bucket ./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo12345 --bucket bucket
## Add ILM rules ## Add ILM rules
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2" ./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2"
+2 -2
View File
@@ -38,7 +38,7 @@ pid=$!
mc ready mysilo mc ready mysilo
mc admin user add mysilo/ silo123 silo123 mc admin user add mysilo/ silo123 silo12345
mc admin policy create mysilo/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json mc admin policy create mysilo/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json
mc admin policy create mysilo/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json mc admin policy create mysilo/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json
@@ -50,7 +50,7 @@ mc admin policy attach mysilo consoleAdmin --user silo123
mc mb -l mysilo/test-bucket mc mb -l mysilo/test-bucket
mc mb -l mysilo/multi-key-poc mc mb -l mysilo/multi-key-poc
export MC_HOST_mysilo1="http://silo123:silo123@localhost:9000/" export MC_HOST_mysilo1="http://silo123:silo12345@localhost:9000/"
mc cp /etc/issue mysilo1/test-bucket mc cp /etc/issue mysilo1/test-bucket
ret=$? ret=$?
+4 -4
View File
@@ -34,7 +34,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,9 +58,9 @@ site2_pid=$!
silo server --config-dir /tmp/silo-ldap --address ":9003" /tmp/silo-ldap-idp3/{1...4} >/tmp/silo3_1.log 2>&1 & silo server --config-dir /tmp/silo-ldap --address ":9003" /tmp/silo-ldap-idp3/{1...4} >/tmp/silo3_1.log 2>&1 &
site3_pid=$! site3_pid=$!
export MC_HOST_silo1=http://minio:silo123@localhost:9001 export MC_HOST_silo1=http://minio:silo12345@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002 export MC_HOST_silo2=http://minio:silo12345@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003 export MC_HOST_silo3=http://minio:silo12345@localhost:9003
./mc ready silo1 ./mc ready silo1
./mc ready silo2 ./mc ready silo2
+4 -4
View File
@@ -31,7 +31,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -56,9 +56,9 @@ if [ ! -f ./mc ]; then
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi fi
export MC_HOST_silo1=http://minio:silo123@localhost:9001 export MC_HOST_silo1=http://minio:silo12345@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002 export MC_HOST_silo2=http://minio:silo12345@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003 export MC_HOST_silo3=http://minio:silo12345@localhost:9003
./mc ready silo1 ./mc ready silo1
./mc ready silo2 ./mc ready silo2
@@ -34,7 +34,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -58,13 +58,13 @@ site3_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9030" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_2.log 2>&1 & silo server --config-dir /tmp/silo-internal --address ":9030" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_2.log 2>&1 &
site3_pid2=$! site3_pid2=$!
export MC_HOST_silo1=http://minio:silo123@localhost:9001 export MC_HOST_silo1=http://minio:silo12345@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002 export MC_HOST_silo2=http://minio:silo12345@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003 export MC_HOST_silo3=http://minio:silo12345@localhost:9003
export MC_HOST_silo10=http://minio:silo123@localhost:9010 export MC_HOST_silo10=http://minio:silo12345@localhost:9010
export MC_HOST_silo20=http://minio:silo123@localhost:9020 export MC_HOST_silo20=http://minio:silo12345@localhost:9020
export MC_HOST_silo30=http://minio:silo123@localhost:9030 export MC_HOST_silo30=http://minio:silo12345@localhost:9030
./mc ready silo1 ./mc ready silo1
./mc ready silo2 ./mc ready silo2
@@ -33,7 +33,7 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
# Download AWS CLI # Download AWS CLI
echo -n "Download and install AWS CLI" echo -n "Download and install AWS CLI"
@@ -51,7 +51,7 @@ cat >~/.aws/credentials <<EOF
[enterprise] [enterprise]
region = us-east-1 region = us-east-1
aws_access_key_id = minio aws_access_key_id = minio
aws_secret_access_key = silo123 aws_secret_access_key = silo12345
EOF EOF
# Create certificates for TLS enabled Silo # Create certificates for TLS enabled Silo
@@ -65,8 +65,8 @@ echo "done"
# Start Silo instances # Start Silo instances
echo -n "Starting Silo instances ..." echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
@@ -75,8 +75,8 @@ if [ ! -f ./mc ]; then
echo "done" echo "done"
fi fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001 export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002 export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure ./mc ready silo1 --insecure
./mc ready silo2 --insecure ./mc ready silo2 --insecure
@@ -29,7 +29,7 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA" TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled Silo # Create certificates for TLS enabled Silo
@@ -43,8 +43,8 @@ echo "done"
# Start Silo instances # Start Silo instances
echo -n "Starting Silo instances ..." echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
@@ -53,8 +53,8 @@ if [ ! -f ./mc ]; then
echo "done" echo "done"
fi fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001 export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002 export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure ./mc ready silo1 --insecure
./mc ready silo2 --insecure ./mc ready silo2 --insecure
@@ -232,12 +232,12 @@ fi
./mc cat silo2/test-bucket/custpartsize --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/custpartsize --insecure >/dev/null || exit_1
echo -n "Starting Silo instances with different kms key ..." echo -n "Starting Silo instances with different kms key ..."
CI=on MINIO_KMS_SECRET_KEY=silo3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/silo3/disk{1...4} >/tmp/silo3_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=silo3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/silo3/disk{1...4} >/tmp/silo3_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=silo4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/silo4/disk{1...4} >/tmp/silo4_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=silo4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo12345 silo server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/silo4/disk{1...4} >/tmp/silo4_1.log 2>&1 &
echo "done" echo "done"
export MC_HOST_silo3=https://minio:silo123@localhost:9003 export MC_HOST_silo3=https://minio:silo12345@localhost:9003
export MC_HOST_silo4=https://minio:silo123@localhost:9004 export MC_HOST_silo4=https://minio:silo12345@localhost:9004
./mc ready silo3 --insecure ./mc ready silo3 --insecure
./mc ready silo4 --insecure ./mc ready silo4 --insecure
@@ -25,7 +25,7 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA" TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled Silo # Create certificates for TLS enabled Silo
@@ -49,8 +49,8 @@ if [ ! -f ./mc ]; then
echo "done" echo "done"
fi fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001 export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002 export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure ./mc ready silo1 --insecure
./mc ready silo2 --insecure ./mc ready silo2 --insecure
@@ -25,7 +25,7 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123" export MINIO_ROOT_PASSWORD="silo12345"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA" TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled Silo # Create certificates for TLS enabled Silo
@@ -49,8 +49,8 @@ if [ ! -f ./mc ]; then
echo "done" echo "done"
fi fi
export MC_HOST_silo1=https://minio:silo123@localhost:9001 export MC_HOST_silo1=https://minio:silo12345@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002 export MC_HOST_silo2=https://minio:silo12345@localhost:9002
./mc ready silo1 --insecure ./mc ready silo1 --insecure
./mc ready silo2 --insecure ./mc ready silo2 --insecure