astelm/minio
1
0
Fork 0
mirror of https://github.com/pgsty/minio.git synced 2026-08-09 15:53:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

test: pin the external fixtures and run the suites against the silo binary

Browse Source 
The test and verification scripts invoked ./minio and pulled their tooling from
upstream infrastructure with no integrity check. Every `curl | tar` of a client
or an old server binary was an unverified execution path in a script that
regularly runs as a privileged user, and several fetched a floating "latest".

Two installers replace all of it:

- install-mcli.sh resolves a pinned pgsty/mc release, downloads the archive and
  its checksum manifest, requires exactly one valid manifest entry for the
  asset, verifies it, and installs. MCLI_BIN with a mandatory MCLI_SHA256 lets
  an offline or air-gapped run supply its own binary, still checksum-checked.
- install-verified-fixture.sh takes source, expected SHA-256 and target, and
  refuses anything that does not match. Sources may be a URL or a local file.

Every script that previously downloaded mc now calls install-mcli.sh. The three
places that genuinely need an upstream artifact - the old MinIO server binary
for the LDAP IAM upgrade-import test, the 2021 mc for the three-site
replication test, and the functional-tests.sh fixture - go through
install-verified-fixture.sh with the digest recorded inline. Those dl.min.io
URLs remain on purpose: they are historical upstream artifacts needed to prove
upgrade compatibility, and they are now pinned and verified rather than
trusted.

The scripts otherwise switch to ./silo, silo.service, the silo container and
compose service names, and SILO_CONFIG_DIR. run-multi-site-minio-idp.sh is
renamed to run-multi-site-silo-idp.sh with the Makefile target following.
buildscripts/minio-upgrade.sh keeps its name and its `minio server` argv - it
exists to test the MinIO-to-Silo upgrade, so the old side must stay old - but
it is now pinned to an image digest rather than a tag, and its `docker system
prune` and `docker volume prune` calls are removed. Those ran unfiltered
against the developer's whole Docker installation; the resiliency tests had the
same problem and lose their prune and `docker ps -q` sweeps too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Feng Ruohang
2026-08-06 08:48:58 +08:00
parent bd8df51665
commit 6613c2a3cb
52 changed files with 2046 additions and 1989 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/site-replication/ldap.yaml
+1 -1
View File
@@ -9,6 +9,6 @@ services:
- "389:389"
- "636:636"
environment:
LDAP_ORGANIZATION: "MinIO Inc"
LDAP_ORGANIZATION: "Silo Inc"
LDAP_DOMAIN: "min.io"
LDAP_ADMIN_PASSWORD: "admin"
docs/site-replication/run-multi-site-ldap.sh
+105 -106
View File
@@ -4,24 +4,24 @@
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
cat /tmp/minio1_2.log
echo "minio2 ============"
cat /tmp/minio2_1.log
cat /tmp/minio2_2.log
echo "minio3 ============"
cat /tmp/minio3_1.log
cat /tmp/minio3_2.log
echo "silo1 ============"
cat /tmp/silo1_1.log
cat /tmp/silo1_2.log
echo "silo2 ============"
cat /tmp/silo2_1.log
cat /tmp/silo2_2.log
echo "silo3 ============"
cat /tmp/silo3_1.log
cat /tmp/silo3_2.log
exit 1
}
cleanup() {
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/minio-ldap-idp{1,2,3}
echo "Cleaning up instances of Silo"
pkill silo
pkill -9 silo
rm -rf /tmp/silo-ldap-idp{1,2,3}
}
cleanup
@@ -34,7 +34,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -48,73 +48,72 @@ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN="ou=swengg,dc=min,dc=io"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member=%d))"
if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi
minio server --config-dir /tmp/minio-ldap --address ":9001" /tmp/minio-ldap-idp1/{1...4} >/tmp/minio1_1.log 2>&1 &
silo server --config-dir /tmp/silo-ldap --address ":9001" /tmp/silo-ldap-idp1/{1...4} >/tmp/silo1_1.log 2>&1 &
site1_pid=$!
minio server --config-dir /tmp/minio-ldap --address ":9002" /tmp/minio-ldap-idp2/{1...4} >/tmp/minio2_1.log 2>&1 &
silo server --config-dir /tmp/silo-ldap --address ":9002" /tmp/silo-ldap-idp2/{1...4} >/tmp/silo2_1.log 2>&1 &
site2_pid=$!
minio server --config-dir /tmp/minio-ldap --address ":9003" /tmp/minio-ldap-idp3/{1...4} >/tmp/minio3_1.log 2>&1 &
silo server --config-dir /tmp/silo-ldap --address ":9003" /tmp/silo-ldap-idp3/{1...4} >/tmp/silo3_1.log 2>&1 &
site3_pid=$!
export MC_HOST_minio1=http://minio:minio123@localhost:9001
export MC_HOST_minio2=http://minio:minio123@localhost:9002
export MC_HOST_minio3=http://minio:minio123@localhost:9003
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
./mc ready minio1
./mc ready minio2
./mc ready minio3
./mc ready silo1
./mc ready silo2
./mc ready silo3
./mc admin replicate add minio1 minio2 minio3
./mc admin replicate add silo1 silo2 silo3
./mc idp ldap policy attach minio1 consoleAdmin --user="uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc idp ldap policy attach silo1 consoleAdmin --user="uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
sleep 5
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin policy create minio1 rw ./docs/site-replication/rw.json
./mc admin user info silo2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin policy create silo1 rw ./docs/site-replication/rw.json
sleep 5
./mc admin policy info minio2 rw >/dev/null 2>&1
./mc admin policy info minio3 rw >/dev/null 2>&1
./mc admin policy info silo2 rw >/dev/null 2>&1
./mc admin policy info silo3 rw >/dev/null 2>&1
./mc admin policy remove minio3 rw
./mc admin policy remove silo3 rw
sleep 10
./mc admin policy info minio1 rw
./mc admin policy info silo1 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info minio2 rw
./mc admin policy info silo2 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin user info minio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.."
exit_1
fi
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.."
exit_1
fi
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.."
exit_1
fi
# LDAP simple user
./mc admin user svcacct add minio2 dillon --access-key testsvc --secret-key testsvc123
./mc admin user svcacct add silo2 dillon --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.."
exit_1
@@ -122,33 +121,33 @@ fi
sleep 10
./mc idp ldap policy entities minio1
./mc idp ldap policy entities minio2
./mc idp ldap policy entities minio3
./mc idp ldap policy entities silo1
./mc idp ldap policy entities silo2
./mc idp ldap policy entities silo3
./mc admin service restart minio1
./mc admin service restart minio2
./mc admin service restart minio3
./mc admin service restart silo1
./mc admin service restart silo2
./mc admin service restart silo3
sleep 10
./mc idp ldap policy entities minio1
./mc idp ldap policy entities minio2
./mc idp ldap policy entities minio3
./mc idp ldap policy entities silo1
./mc idp ldap policy entities silo2
./mc idp ldap policy entities silo3
./mc admin user svcacct info minio1 testsvc
./mc admin user svcacct info silo1 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info minio2 testsvc
./mc admin user svcacct info silo2 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info minio3 testsvc
./mc admin user svcacct info silo3 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
@@ -158,25 +157,25 @@ MC_HOST_svc1=http://testsvc:testsvc123@localhost:9001 ./mc ls svc1
MC_HOST_svc2=http://testsvc:testsvc123@localhost:9002 ./mc ls svc2
MC_HOST_svc3=http://testsvc:testsvc123@localhost:9003 ./mc ls svc3
./mc admin user svcacct rm minio1 testsvc
./mc admin user svcacct rm silo1 testsvc
if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.."
exit_1
fi
./mc admin user info minio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.."
exit_1
fi
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.."
exit_1
fi
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info silo3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.."
exit_1
@@ -184,154 +183,154 @@ fi
sleep 10
./mc admin user svcacct info minio2 testsvc
./mc admin user svcacct info silo2 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc admin user svcacct info minio3 testsvc
./mc admin user svcacct info silo3 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc mb minio1/newbucket
# copy large upload to newbucket on minio1
./mc mb silo1/newbucket
# copy large upload to newbucket on silo1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/newbucket
./mc cp ./lrgfile silo1/newbucket
# create a bucket bucket2 on minio1.
./mc mb minio1/bucket2
# create a bucket bucket2 on silo1.
./mc mb silo1/bucket2
sleep 5
./mc stat --no-list minio2/newbucket
./mc stat --no-list silo2/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket
./mc stat --no-list silo3/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc cp README.md minio2/newbucket/
./mc cp README.md silo2/newbucket/
sleep 5
./mc stat --no-list minio1/newbucket/README.md
./mc stat --no-list silo1/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket/README.md
./mc stat --no-list silo3/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
sleep 10
./mc stat --no-list minio3/newbucket/lrgfile
./mc stat --no-list silo3/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
actual_checksum=$(./mc cat silo3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit
fi
rm ./lrgfile
./mc rm -r --versions --force minio1/newbucket/lrgfile
./mc rm -r --versions --force silo1/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
sleep 5
./mc stat --no-list minio1/newbucket/lrgfile
./mc stat --no-list silo1/newbucket/lrgfile
if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.."
exit_1
fi
vID=$(./mc stat --no-list minio2/newbucket/README.md --json | jq .versionID)
vID=$(./mc stat --no-list silo2/newbucket/README.md --json | jq .versionID)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc tag set --version-id "${vID}" minio2/newbucket/README.md "key=val"
./mc tag set --version-id "${vID}" silo2/newbucket/README.md "key=val"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
./mc tag remove --version-id "${vID}" minio2/newbucket/README.md
./mc tag remove --version-id "${vID}" silo2/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting tag removal to be successful. exiting.."
exit_1
fi
sleep 5
replStatus_minio2=$(./mc stat --no-list minio2/newbucket/README.md --json | jq -r .replicationStatus)
replStatus_silo2=$(./mc stat --no-list silo2/newbucket/README.md --json | jq -r .replicationStatus)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
if [ ${replStatus_minio2} != "COMPLETED" ]; then
if [ ${replStatus_silo2} != "COMPLETED" ]; then
echo "expected tag removal to have replicated, exiting..."
exit_1
fi
./mc rm minio3/newbucket/README.md
./mc rm silo3/newbucket/README.md
sleep 5
./mc stat --no-list minio2/newbucket/README.md
./mc stat --no-list silo2/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc stat --no-list minio1/newbucket/README.md
./mc stat --no-list silo1/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc mb --with-lock minio3/newbucket-olock
./mc mb --with-lock silo3/newbucket-olock
sleep 5
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
enabled_silo2=$(./mc stat --json silo2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio2}" != "Enabled" ]; then
if [ "${enabled_silo2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
enabled_silo1=$(./mc stat --json silo1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio1}" != "Enabled" ]; then
if [ "${enabled_silo1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
# "Test if most recent tag update is replicated"
./mc tag set minio2/newbucket "key=val1"
./mc tag set silo2/newbucket "key=val1"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
@@ -339,49 +338,49 @@ fi
sleep 10
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# stop minio1
# stop silo1
kill -9 ${site1_pid}
# Update tag on minio2/newbucket when minio1 is down
./mc tag set minio2/newbucket "key=val2"
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
./mc mb minio2/newbucket2
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
./mc rb minio2/bucket2
# Update tag on silo2/newbucket when silo1 is down
./mc tag set silo2/newbucket "key=val2"
# create a new bucket on silo2. This should replicate to silo1 after it comes online.
./mc mb silo2/newbucket2
# delete bucket2 on silo2. This should replicate to silo1 after it comes online.
./mc rb silo2/bucket2
# Restart minio1 instance
minio server --config-dir /tmp/minio-ldap --address ":9001" /tmp/minio-ldap-idp1/{1...4} >/tmp/minio1_1.log 2>&1 &
# Restart silo1 instance
silo server --config-dir /tmp/silo-ldap --address ":9001" /tmp/silo-ldap-idp1/{1...4} >/tmp/silo1_1.log 2>&1 &
sleep 200
# Test whether most recent tag update on minio2 is replicated to minio1
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
# Test whether most recent tag update on silo2 is replicated to silo1
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Test if bucket created/deleted when minio1 is down healed
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
# Test if bucket created/deleted when silo1 is down healed
diff -q <(./mc ls silo1) <(./mc ls silo2) 1>/dev/null
if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1
fi
# force a resync after removing all site replication
./mc admin replicate rm --all --force minio1
./mc rb minio2 --force --dangerous
./mc admin replicate add minio1 minio2
./mc admin replicate resync start minio1 minio2
./mc admin replicate rm --all --force silo1
./mc rb silo2 --force --dangerous
./mc admin replicate add silo1 silo2
./mc admin replicate resync start silo1 silo2
sleep 30
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
./mc ls -r --versions silo1/newbucket >/tmp/silo1.txt
./mc ls -r --versions silo2/newbucket >/tmp/silo2.txt
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
out=$(diff -qpruN /tmp/silo1.txt /tmp/silo2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out"
docs/site-replication/run-multi-site-minio-idp.sh
-458
View File
@@ -1,458 +0,0 @@
#!/usr/bin/env bash
# shellcheck disable=SC2120
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
cat /tmp/minio1_2.log
echo "minio2 ============"
cat /tmp/minio2_1.log
cat /tmp/minio2_2.log
echo "minio3 ============"
cat /tmp/minio3_1.log
cat /tmp/minio3_2.log
exit 1
}
cleanup() {
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/minio-internal-idp{1,2,3}
}
cleanup
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
minio server --config-dir /tmp/minio-internal --address ":9001" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_1.log 2>&1 &
site1_pid1=$!
minio server --config-dir /tmp/minio-internal --address ":9010" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_2.log 2>&1 &
site1_pid2=$!
minio server --config-dir /tmp/minio-internal --address ":9002" http://localhost:9002/tmp/minio-internal-idp2/{1...4} http://localhost:9020/tmp/minio-internal-idp2/{5...8} >/tmp/minio2_1.log 2>&1 &
site2_pid1=$!
minio server --config-dir /tmp/minio-internal --address ":9020" http://localhost:9002/tmp/minio-internal-idp2/{1...4} http://localhost:9020/tmp/minio-internal-idp2/{5...8} >/tmp/minio2_2.log 2>&1 &
site2_pid2=$!
minio server --config-dir /tmp/minio-internal --address ":9003" http://localhost:9003/tmp/minio-internal-idp3/{1...4} http://localhost:9030/tmp/minio-internal-idp3/{5...8} >/tmp/minio3_1.log 2>&1 &
site3_pid1=$!
minio server --config-dir /tmp/minio-internal --address ":9030" http://localhost:9003/tmp/minio-internal-idp3/{1...4} http://localhost:9030/tmp/minio-internal-idp3/{5...8} >/tmp/minio3_2.log 2>&1 &
site3_pid2=$!
export MC_HOST_minio1=http://minio:minio123@localhost:9001
export MC_HOST_minio2=http://minio:minio123@localhost:9002
export MC_HOST_minio3=http://minio:minio123@localhost:9003
export MC_HOST_minio10=http://minio:minio123@localhost:9010
export MC_HOST_minio20=http://minio:minio123@localhost:9020
export MC_HOST_minio30=http://minio:minio123@localhost:9030
./mc ready minio1
./mc ready minio2
./mc ready minio3
./mc ready minio10
./mc ready minio20
./mc ready minio30
./mc admin replicate add minio1 minio2
site_enabled=$(./mc admin replicate info minio1)
site_enabled_peer=$(./mc admin replicate info minio10)
[[ $site_enabled =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
[[ $site_enabled_peer =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
./mc admin user add minio1 foobar foo12345
## add foobar-g group with foobar
./mc admin group add minio2 foobar-g foobar
./mc admin policy attach minio1 consoleAdmin --user=foobar
sleep 5
./mc admin user info minio2 foobar
./mc admin group info minio1 foobar-g
./mc admin policy create minio1 rw ./docs/site-replication/rw.json
sleep 5
./mc admin policy info minio2 rw >/dev/null 2>&1
./mc admin replicate status minio1
## Add a new empty site
./mc admin replicate add minio1 minio2 minio3
sleep 10
./mc admin policy info minio3 rw >/dev/null 2>&1
./mc admin policy remove minio3 rw
./mc admin replicate status minio3
sleep 10
./mc admin policy info minio1 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info minio2 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info minio3 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin user info minio1 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'minio1', exiting.."
exit_1
fi
./mc admin user info minio2 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'minio2', exiting.."
exit_1
fi
./mc admin user info minio3 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'minio3', exiting.."
exit_1
fi
./mc admin group info minio3 foobar-g
if [ $? -ne 0 ]; then
echo "group mapping missing on 'minio3', exiting.."
exit_1
fi
./mc admin user svcacct add minio2 foobar --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.."
exit_1
fi
./mc admin user svcacct add minio2 minio --access-key testsvc2 --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding root svc account testsvc2 failed, exiting.."
exit_1
fi
sleep 10
export MC_HOST_rootsvc=http://testsvc2:testsvc123@localhost:9002
./mc ls rootsvc
if [ $? -ne 0 ]; then
echo "root service account not inherited root permissions, exiting.."
exit_1
fi
./mc admin user svcacct info minio1 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info minio2 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct rm minio1 testsvc
if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.."
exit_1
fi
sleep 10
./mc admin user svcacct info minio2 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc admin user svcacct info minio3 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc mb minio1/newbucket
# copy large upload to newbucket on minio1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/newbucket
sleep 5
./mc stat --no-list minio2/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
err_minio2=$(./mc stat --no-list minio2/newbucket/xxx --json | jq -r .error.cause.message)
if [ $? -ne 0 ]; then
echo "expecting object to be missing. exiting.."
exit_1
fi
if [ "${err_minio2}" != "Object does not exist" ]; then
echo "expected to see Object does not exist error, exiting..."
exit_1
fi
./mc cp README.md minio2/newbucket/
sleep 5
./mc stat --no-list minio1/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
sleep 10
./mc stat --no-list minio3/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit
fi
rm ./lrgfile
./mc rm -r --versions --force minio1/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
sleep 5
./mc stat --no-list minio1/newbucket/lrgfile
if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.."
exit_1
fi
vID=$(./mc stat --no-list minio2/newbucket/README.md --json | jq .versionID)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc tag set --version-id "${vID}" minio2/newbucket/README.md "key=val"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list minio1/newbucket/README.md --version-id "${vID}" --json | jq -r .tagset.key)
if [ "${val}" != "val" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
./mc tag remove --version-id "${vID}" minio2/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting tag removal to be successful. exiting.."
exit_1
fi
sleep 5
replStatus_minio2=$(./mc stat --no-list minio2/newbucket/README.md --json | jq -r .replicationStatus)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
if [ ${replStatus_minio2} != "COMPLETED" ]; then
echo "expected tag removal to have replicated, exiting..."
exit_1
fi
./mc rm minio3/newbucket/README.md
sleep 5
./mc stat --no-list minio2/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc stat --no-list minio1/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc mb --with-lock minio3/newbucket-olock
sleep 5
set -x
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
set +x
# "Test if most recent tag update is replicated"
./mc tag set minio2/newbucket "key=val1"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Create user with policy consoleAdmin on minio1
./mc admin user add minio1 foobarx foobar123
if [ $? -ne 0 ]; then
echo "adding user failed, exiting.."
exit_1
fi
./mc admin policy attach minio1 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "adding policy mapping failed, exiting.."
exit_1
fi
sleep 10
# unset policy for foobarx in minio2
./mc admin policy detach minio2 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "unset policy mapping failed, exiting.."
exit_1
fi
# create a bucket bucket2 on minio1.
./mc mb minio1/bucket2
sleep 10
# Test whether policy detach replicated to minio1
policy=$(./mc admin user info minio1 foobarx --json | jq -r .policyName)
if [ "${policy}" != "null" ]; then
echo "expected policy detach to have replicated, exiting..."
exit_1
fi
kill -9 ${site1_pid1} ${site1_pid2}
# Update tag on minio2/newbucket when minio1 is down
./mc tag set minio2/newbucket "key=val2"
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
./mc mb minio2/newbucket2
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
./mc rb minio2/bucket2
# Restart minio1 instance
minio server --config-dir /tmp/minio-internal --address ":9001" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_1.log 2>&1 &
minio server --config-dir /tmp/minio-internal --address ":9010" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_2.log 2>&1 &
sleep 200
# Test whether most recent tag update on minio2 is replicated to minio1
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Test if bucket created/deleted when minio1 is down healed
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1
fi
# force a resync after removing all site replication
./mc admin replicate rm --all --force minio1
./mc rb minio2 --force --dangerous
./mc admin replicate add minio1 minio2
./mc admin replicate resync start minio1 minio2
sleep 30
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out"
exit 1
fi
docs/site-replication/run-multi-site-oidc.sh
+81 -82
View File
@@ -4,21 +4,21 @@
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
echo "minio2 ============"
cat /tmp/minio2_1.log
echo "minio3 ============"
cat /tmp/minio3_1.log
echo "silo1 ============"
cat /tmp/silo1_1.log
echo "silo2 ============"
cat /tmp/silo2_1.log
echo "silo3 ============"
cat /tmp/silo3_1.log
exit 1
}
cleanup() {
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/minio{1,2,3}
echo "Cleaning up instances of Silo"
pkill silo
pkill -9 silo
rm -rf /tmp/silo{1,2,3}
}
cleanup
@@ -31,7 +31,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -42,64 +42,63 @@ export MINIO_IDENTITY_OPENID_CLAIM_NAME="groups"
export MINIO_IDENTITY_OPENID_SCOPES="openid,groups"
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:10000/oauth_callback"
minio server --address ":9001" --console-address ":10000" /tmp/minio1/{1...4} >/tmp/minio1_1.log 2>&1 &
silo server --address ":9001" --console-address ":10000" /tmp/silo1/{1...4} >/tmp/silo1_1.log 2>&1 &
site1_pid=$!
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:11000/oauth_callback"
minio server --address ":9002" --console-address ":11000" /tmp/minio2/{1...4} >/tmp/minio2_1.log 2>&1 &
silo server --address ":9002" --console-address ":11000" /tmp/silo2/{1...4} >/tmp/silo2_1.log 2>&1 &
site2_pid=$!
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:12000/oauth_callback"
minio server --address ":9003" --console-address ":12000" /tmp/minio3/{1...4} >/tmp/minio3_1.log 2>&1 &
silo server --address ":9003" --console-address ":12000" /tmp/silo3/{1...4} >/tmp/silo3_1.log 2>&1 &
site3_pid=$!
if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi
export MC_HOST_minio1=http://minio:minio123@localhost:9001
export MC_HOST_minio2=http://minio:minio123@localhost:9002
export MC_HOST_minio3=http://minio:minio123@localhost:9003
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
./mc ready minio1
./mc ready minio2
./mc ready minio3
./mc ready silo1
./mc ready silo2
./mc ready silo3
./mc admin replicate add minio1 minio2 minio3
./mc admin replicate add silo1 silo2 silo3
./mc admin policy create minio1 projecta ./docs/site-replication/rw.json
./mc admin policy create silo1 projecta ./docs/site-replication/rw.json
sleep 5
./mc admin policy info minio2 projecta >/dev/null 2>&1
./mc admin policy info silo2 projecta >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo "expecting the command to succeed, exiting.."
exit_1
fi
./mc admin policy info minio3 projecta >/dev/null 2>&1
./mc admin policy info silo3 projecta >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo "expecting the command to succeed, exiting.."
exit_1
fi
./mc admin policy remove minio3 projecta
./mc admin policy remove silo3 projecta
sleep 10
./mc admin policy info minio1 projecta
./mc admin policy info silo1 projecta
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info minio2 projecta
./mc admin policy info silo2 projecta
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy create minio1 projecta ./docs/site-replication/rw.json
./mc admin policy create silo1 projecta ./docs/site-replication/rw.json
sleep 5
# Generate STS credential with STS call to minio1
# Generate STS credential with STS call to silo1
STS_CRED=$(MINIO_ENDPOINT=http://localhost:9001 go run ./docs/site-replication/gen-oidc-sts-cred.go)
MC_HOST_foo=http://${STS_CRED}@localhost:9001 ./mc ls foo
@@ -110,7 +109,7 @@ fi
sleep 2
# Check that the STS credential works on minio2 and minio3.
# Check that the STS credential works on silo2 and silo3.
MC_HOST_foo=http://${STS_CRED}@localhost:9002 ./mc ls foo
if [ $? -ne 0 ]; then
echo "Expected sts credential to work, exiting.."
@@ -126,7 +125,7 @@ fi
STS_ACCESS_KEY=$(echo ${STS_CRED} | cut -d ':' -f 1)
# Create service account for STS user
./mc admin user svcacct add minio2 $STS_ACCESS_KEY --access-key testsvc --secret-key testsvc123
./mc admin user svcacct add silo2 $STS_ACCESS_KEY --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.."
exit_1
@@ -134,192 +133,192 @@ fi
sleep 10
./mc admin user svcacct info minio1 testsvc
./mc admin user svcacct info silo1 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info minio2 testsvc
./mc admin user svcacct info silo2 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct rm minio1 testsvc
./mc admin user svcacct rm silo1 testsvc
if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.."
exit_1
fi
sleep 10
./mc admin user svcacct info minio2 testsvc
./mc admin user svcacct info silo2 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc admin user svcacct info minio3 testsvc
./mc admin user svcacct info silo3 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
# create a bucket bucket2 on minio1.
./mc mb minio1/bucket2
# create a bucket bucket2 on silo1.
./mc mb silo1/bucket2
./mc mb minio1/newbucket
./mc mb silo1/newbucket
# copy large upload to newbucket on minio1
# copy large upload to newbucket on silo1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/newbucket
./mc cp ./lrgfile silo1/newbucket
sleep 5
./mc stat --no-list minio2/newbucket
./mc stat --no-list silo2/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket
./mc stat --no-list silo3/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc cp README.md minio2/newbucket/
./mc cp README.md silo2/newbucket/
sleep 5
./mc stat --no-list minio1/newbucket/README.md
./mc stat --no-list silo1/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket/README.md
./mc stat --no-list silo3/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc rm minio3/newbucket/README.md
./mc rm silo3/newbucket/README.md
sleep 5
./mc stat --no-list minio2/newbucket/README.md
./mc stat --no-list silo2/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc stat --no-list minio1/newbucket/README.md
./mc stat --no-list silo1/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
sleep 10
./mc stat --no-list minio3/newbucket/lrgfile
./mc stat --no-list silo3/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
actual_checksum=$(./mc cat silo3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit
fi
rm ./lrgfile
./mc rm -r --versions --force minio1/newbucket/lrgfile
./mc rm -r --versions --force silo1/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
sleep 5
./mc stat --no-list minio1/newbucket/lrgfile
./mc stat --no-list silo1/newbucket/lrgfile
if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.."
exit_1
fi
./mc mb --with-lock minio3/newbucket-olock
./mc mb --with-lock silo3/newbucket-olock
sleep 5
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
enabled_silo2=$(./mc stat --json silo2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio2}" != "Enabled" ]; then
if [ "${enabled_silo2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
enabled_silo1=$(./mc stat --json silo1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio1}" != "Enabled" ]; then
if [ "${enabled_silo1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
# "Test if most recent tag update is replicated"
./mc tag set minio2/newbucket "key=val1"
./mc tag set silo2/newbucket "key=val1"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 10
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# stop minio1 instance
# stop silo1 instance
kill -9 ${site1_pid}
# Update tag on minio2/newbucket when minio1 is down
./mc tag set minio2/newbucket "key=val2"
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
./mc mb minio2/newbucket2
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
./mc rb minio2/bucket2
# Update tag on silo2/newbucket when silo1 is down
./mc tag set silo2/newbucket "key=val2"
# create a new bucket on silo2. This should replicate to silo1 after it comes online.
./mc mb silo2/newbucket2
# delete bucket2 on silo2. This should replicate to silo1 after it comes online.
./mc rb silo2/bucket2
# Restart minio1 instance
minio server --address ":9001" --console-address ":10000" /tmp/minio1/{1...4} >/tmp/minio1_1.log 2>&1 &
# Restart silo1 instance
silo server --address ":9001" --console-address ":10000" /tmp/silo1/{1...4} >/tmp/silo1_1.log 2>&1 &
sleep 200
# Test whether most recent tag update on minio2 is replicated to minio1
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
# Test whether most recent tag update on silo2 is replicated to silo1
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Test if bucket created/deleted when minio1 is down healed
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
# Test if bucket created/deleted when silo1 is down healed
diff -q <(./mc ls silo1) <(./mc ls silo2) 1>/dev/null
if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1
fi
# force a resync after removing all site replication
./mc admin replicate rm --all --force minio1
./mc rb minio2 --force --dangerous
./mc admin replicate add minio1 minio2
./mc admin replicate resync start minio1 minio2
./mc admin replicate rm --all --force silo1
./mc rb silo2 --force --dangerous
./mc admin replicate add silo1 silo2
./mc admin replicate resync start silo1 silo2
sleep 30
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
./mc ls -r --versions silo1/newbucket >/tmp/silo1.txt
./mc ls -r --versions silo2/newbucket >/tmp/silo2.txt
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
out=$(diff -qpruN /tmp/silo1.txt /tmp/silo2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out"
docs/site-replication/run-multi-site-silo-idp.sh
Executable
+457
View File
@@ -0,0 +1,457 @@
#!/usr/bin/env bash
# shellcheck disable=SC2120
exit_1() {
cleanup
echo "silo1 ============"
cat /tmp/silo1_1.log
cat /tmp/silo1_2.log
echo "silo2 ============"
cat /tmp/silo2_1.log
cat /tmp/silo2_2.log
echo "silo3 ============"
cat /tmp/silo3_1.log
cat /tmp/silo3_2.log
exit 1
}
cleanup() {
echo "Cleaning up instances of Silo"
pkill silo
pkill -9 silo
rm -rf /tmp/silo-internal-idp{1,2,3}
}
cleanup
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
if [ ! -f ./mc ]; then
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi
silo server --config-dir /tmp/silo-internal --address ":9001" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_1.log 2>&1 &
site1_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9010" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_2.log 2>&1 &
site1_pid2=$!
silo server --config-dir /tmp/silo-internal --address ":9002" http://localhost:9002/tmp/silo-internal-idp2/{1...4} http://localhost:9020/tmp/silo-internal-idp2/{5...8} >/tmp/silo2_1.log 2>&1 &
site2_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9020" http://localhost:9002/tmp/silo-internal-idp2/{1...4} http://localhost:9020/tmp/silo-internal-idp2/{5...8} >/tmp/silo2_2.log 2>&1 &
site2_pid2=$!
silo server --config-dir /tmp/silo-internal --address ":9003" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_1.log 2>&1 &
site3_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9030" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_2.log 2>&1 &
site3_pid2=$!
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
export MC_HOST_silo10=http://minio:silo123@localhost:9010
export MC_HOST_silo20=http://minio:silo123@localhost:9020
export MC_HOST_silo30=http://minio:silo123@localhost:9030
./mc ready silo1
./mc ready silo2
./mc ready silo3
./mc ready silo10
./mc ready silo20
./mc ready silo30
./mc admin replicate add silo1 silo2
site_enabled=$(./mc admin replicate info silo1)
site_enabled_peer=$(./mc admin replicate info silo10)
[[ $site_enabled =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
[[ $site_enabled_peer =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
./mc admin user add silo1 foobar foo12345
## add foobar-g group with foobar
./mc admin group add silo2 foobar-g foobar
./mc admin policy attach silo1 consoleAdmin --user=foobar
sleep 5
./mc admin user info silo2 foobar
./mc admin group info silo1 foobar-g
./mc admin policy create silo1 rw ./docs/site-replication/rw.json
sleep 5
./mc admin policy info silo2 rw >/dev/null 2>&1
./mc admin replicate status silo1
## Add a new empty site
./mc admin replicate add silo1 silo2 silo3
sleep 10
./mc admin policy info silo3 rw >/dev/null 2>&1
./mc admin policy remove silo3 rw
./mc admin replicate status silo3
sleep 10
./mc admin policy info silo1 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info silo2 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info silo3 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin user info silo1 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'silo1', exiting.."
exit_1
fi
./mc admin user info silo2 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'silo2', exiting.."
exit_1
fi
./mc admin user info silo3 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'silo3', exiting.."
exit_1
fi
./mc admin group info silo3 foobar-g
if [ $? -ne 0 ]; then
echo "group mapping missing on 'silo3', exiting.."
exit_1
fi
./mc admin user svcacct add silo2 foobar --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.."
exit_1
fi
./mc admin user svcacct add silo2 minio --access-key testsvc2 --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding root svc account testsvc2 failed, exiting.."
exit_1
fi
sleep 10
export MC_HOST_rootsvc=http://testsvc2:testsvc123@localhost:9002
./mc ls rootsvc
if [ $? -ne 0 ]; then
echo "root service account not inherited root permissions, exiting.."
exit_1
fi
./mc admin user svcacct info silo1 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info silo2 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct rm silo1 testsvc
if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.."
exit_1
fi
sleep 10
./mc admin user svcacct info silo2 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc admin user svcacct info silo3 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc mb silo1/newbucket
# copy large upload to newbucket on silo1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile silo1/newbucket
sleep 5
./mc stat --no-list silo2/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc stat --no-list silo3/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
err_silo2=$(./mc stat --no-list silo2/newbucket/xxx --json | jq -r .error.cause.message)
if [ $? -ne 0 ]; then
echo "expecting object to be missing. exiting.."
exit_1
fi
if [ "${err_silo2}" != "Object does not exist" ]; then
echo "expected to see Object does not exist error, exiting..."
exit_1
fi
./mc cp README.md silo2/newbucket/
sleep 5
./mc stat --no-list silo1/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc stat --no-list silo3/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
sleep 10
./mc stat --no-list silo3/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
actual_checksum=$(./mc cat silo3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit
fi
rm ./lrgfile
./mc rm -r --versions --force silo1/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
sleep 5
./mc stat --no-list silo1/newbucket/lrgfile
if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.."
exit_1
fi
vID=$(./mc stat --no-list silo2/newbucket/README.md --json | jq .versionID)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc tag set --version-id "${vID}" silo2/newbucket/README.md "key=val"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list silo1/newbucket/README.md --version-id "${vID}" --json | jq -r .tagset.key)
if [ "${val}" != "val" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
./mc tag remove --version-id "${vID}" silo2/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting tag removal to be successful. exiting.."
exit_1
fi
sleep 5
replStatus_silo2=$(./mc stat --no-list silo2/newbucket/README.md --json | jq -r .replicationStatus)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
if [ ${replStatus_silo2} != "COMPLETED" ]; then
echo "expected tag removal to have replicated, exiting..."
exit_1
fi
./mc rm silo3/newbucket/README.md
sleep 5
./mc stat --no-list silo2/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc stat --no-list silo1/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc mb --with-lock silo3/newbucket-olock
sleep 5
set -x
enabled_silo2=$(./mc stat --json silo2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_silo2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
enabled_silo1=$(./mc stat --json silo1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_silo1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
set +x
# "Test if most recent tag update is replicated"
./mc tag set silo2/newbucket "key=val1"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Create user with policy consoleAdmin on silo1
./mc admin user add silo1 foobarx foobar123
if [ $? -ne 0 ]; then
echo "adding user failed, exiting.."
exit_1
fi
./mc admin policy attach silo1 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "adding policy mapping failed, exiting.."
exit_1
fi
sleep 10
# unset policy for foobarx in silo2
./mc admin policy detach silo2 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "unset policy mapping failed, exiting.."
exit_1
fi
# create a bucket bucket2 on silo1.
./mc mb silo1/bucket2
sleep 10
# Test whether policy detach replicated to silo1
policy=$(./mc admin user info silo1 foobarx --json | jq -r .policyName)
if [ "${policy}" != "null" ]; then
echo "expected policy detach to have replicated, exiting..."
exit_1
fi
kill -9 ${site1_pid1} ${site1_pid2}
# Update tag on silo2/newbucket when silo1 is down
./mc tag set silo2/newbucket "key=val2"
# create a new bucket on silo2. This should replicate to silo1 after it comes online.
./mc mb silo2/newbucket2
# delete bucket2 on silo2. This should replicate to silo1 after it comes online.
./mc rb silo2/bucket2
# Restart silo1 instance
silo server --config-dir /tmp/silo-internal --address ":9001" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_1.log 2>&1 &
silo server --config-dir /tmp/silo-internal --address ":9010" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_2.log 2>&1 &
sleep 200
# Test whether most recent tag update on silo2 is replicated to silo1
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Test if bucket created/deleted when silo1 is down healed
diff -q <(./mc ls silo1) <(./mc ls silo2) 1>/dev/null
if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1
fi
# force a resync after removing all site replication
./mc admin replicate rm --all --force silo1
./mc rb silo2 --force --dangerous
./mc admin replicate add silo1 silo2
./mc admin replicate resync start silo1 silo2
sleep 30
./mc ls -r --versions silo1/newbucket >/tmp/silo1.txt
./mc ls -r --versions silo2/newbucket >/tmp/silo2.txt
out=$(diff -qpruN /tmp/silo1.txt /tmp/silo2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out"
exit 1
fi
docs/site-replication/run-replication-with-checksum-header.sh
+41 -42
View File
@@ -4,18 +4,18 @@
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
echo "minio2 ============"
cat /tmp/minio2_1.log
echo "silo1 ============"
cat /tmp/silo1_1.log
echo "silo2 ============"
cat /tmp/silo2_1.log
exit 1
}
cleanup() {
echo -n "Cleaning up instances of MinIO ..."
pkill -9 minio || sudo pkill -9 minio
rm -rf /tmp/minio{1,2}
echo -n "Cleaning up instances of Silo ..."
pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/silo{1,2}
echo "done"
}
@@ -33,7 +33,7 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_PASSWORD="silo123"
# Download AWS CLI
echo -n "Download and install AWS CLI"
@@ -51,11 +51,11 @@ cat >~/.aws/credentials <<EOF
[enterprise]
region = us-east-1
aws_access_key_id = minio
aws_secret_access_key = minio123
aws_secret_access_key = silo123
EOF
# Create certificates for TLS enabled MinIO
echo -n "Setup certs for MinIO instances ..."
# Create certificates for TLS enabled Silo
echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost
mkdir -p /tmp/certs
@@ -63,24 +63,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done"
# Start MinIO instances
echo -n "Starting MinIO instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
# Start Silo instances
echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
echo -n "Downloading Silo client ..."
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
echo "done"
fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure
./mc ready minio2 --insecure
./mc ready silo1 --insecure
./mc ready silo2 --insecure
# Prepare data for tests
echo -n "Preparing test data ..."
@@ -91,16 +90,16 @@ shred -s 500M /tmp/data/mpartobj
echo "done"
# Add replication site
./mc admin replicate add minio1 minio2 --insecure
./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete
sleep 30
# Create bucket in source cluster
echo "Create bucket in source MinIO instance"
./mc mb minio1/test-bucket --insecure
echo "Create bucket in source Silo instance"
./mc mb silo1/test-bucket --insecure
# Load objects to source site with checksum header
echo "Loading objects to source MinIO instance"
echo "Loading objects to source Silo instance"
OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj --body /tmp/data/obj --no-verify-ssl --profile enterprise
@@ -128,20 +127,20 @@ sleep 120
# List the objects from replicated site
echo "Objects from replicated site"
./mc ls minio2/test-bucket --insecure
count1=$(./mc ls minio2/test-bucket/obj --insecure | wc -l)
./mc ls silo2/test-bucket --insecure
count1=$(./mc ls silo2/test-bucket/obj --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/obj not replicated"
echo "BUG: object silo1/test-bucket/obj not replicated"
exit_1
fi
count2=$(./mc ls minio2/test-bucket/mpartobj --insecure | wc -l)
count2=$(./mc ls silo2/test-bucket/mpartobj --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/mpartobj not replicated"
echo "BUG: object silo1/test-bucket/mpartobj not replicated"
exit_1
fi
# Stat the objects from source site
echo "Stat minio1/test-bucket/obj"
echo "Stat silo1/test-bucket/obj"
SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
@@ -150,7 +149,7 @@ SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
# Stat the objects from replicated site
echo "Stat minio2/test-bucket/obj"
echo "Stat silo2/test-bucket/obj"
DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
@@ -180,10 +179,10 @@ echo "Set default encryption on "
# test if checksum header is replicated for encrypted objects
# Enable SSE KMS for test-bucket bucket
./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure
./mc encrypt set sse-kms minio-default-key silo1/test-bucket --insecure
# Load objects to source site with checksum header
echo "Loading objects to source MinIO instance"
echo "Loading objects to source Silo instance"
OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj2 --body /tmp/data/obj --no-verify-ssl --profile enterprise
@@ -211,20 +210,20 @@ sleep 120
# List the objects from replicated site
echo "Objects from replicated site"
./mc ls minio2/test-bucket --insecure
count1=$(./mc ls minio2/test-bucket/obj2 --insecure | wc -l)
./mc ls silo2/test-bucket --insecure
count1=$(./mc ls silo2/test-bucket/obj2 --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/obj2 not replicated"
echo "BUG: object silo1/test-bucket/obj2 not replicated"
exit_1
fi
count2=$(./mc ls minio2/test-bucket/mpartobj2 --insecure | wc -l)
count2=$(./mc ls silo2/test-bucket/mpartobj2 --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/mpartobj2 not replicated"
echo "BUG: object silo1/test-bucket/mpartobj2 not replicated"
exit_1
fi
# Stat the objects from source site
echo "Stat minio1/test-bucket/obj2"
echo "Stat silo1/test-bucket/obj2"
SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
@@ -233,7 +232,7 @@ SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
# Stat the objects from replicated site
echo "Stat minio2/test-bucket/obj2"
echo "Stat silo2/test-bucket/obj2"
DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
docs/site-replication/run-sse-kms-object-replication.sh
+95 -96
View File
@@ -4,23 +4,23 @@
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
echo "minio2 ============"
cat /tmp/minio2_1.log
echo "minio3 ============"
cat /tmp/minio3_1.log
echo "minio4 ============"
cat /tmp/minio4_1.log
echo "silo1 ============"
cat /tmp/silo1_1.log
echo "silo2 ============"
cat /tmp/silo2_1.log
echo "silo3 ============"
cat /tmp/silo3_1.log
echo "silo4 ============"
cat /tmp/silo4_1.log
exit 1
}
cleanup() {
echo -n "Cleaning up instances of MinIO ..."
pkill -9 minio || sudo pkill -9 minio
echo -n "Cleaning up instances of Silo ..."
pkill -9 silo || sudo pkill -9 silo
pkill -9 kes || sudo pkill -9 kes
rm -rf ${PWD}/keys
rm -rf /tmp/minio{1,2,3,4}
rm -rf /tmp/silo{1,2,3,4}
echo "done"
}
@@ -29,11 +29,11 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_PASSWORD="silo123"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled MinIO
echo -n "Setup certs for MinIO instances ..."
# Create certificates for TLS enabled Silo
echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost
mkdir -p /tmp/certs
@@ -41,24 +41,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done"
# Start MinIO instances
echo -n "Starting MinIO instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
# Start Silo instances
echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
echo -n "Downloading Silo client ..."
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
echo "done"
fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure
./mc ready minio2 --insecure
./mc ready silo1 --insecure
./mc ready silo2 --insecure
# Prepare data for tests
echo -n "Preparing test data ..."
@@ -73,181 +72,181 @@ shred -s 500M /tmp/data/custpartsize
echo "done"
# Add replication site
./mc admin replicate add minio1 minio2 --insecure
./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete
sleep 30
# Create bucket in source cluster
echo "Create bucket in source MinIO instance"
./mc mb minio1/test-bucket --insecure
echo "Create bucket in source Silo instance"
./mc mb silo1/test-bucket --insecure
# Enable SSE KMS for the bucket
./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure
./mc encrypt set sse-kms minio-default-key silo1/test-bucket --insecure
# Load objects to source site
echo "Loading objects to source MinIO instance"
./mc cp /tmp/data/encrypted minio1/test-bucket --insecure
./mc cp /tmp/data/mpartobj minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize minio1/test-bucket --insecure
./mc put /tmp/data/custpartsize minio1/test-bucket --insecure --part-size 50MiB
echo "Loading objects to source Silo instance"
./mc cp /tmp/data/encrypted silo1/test-bucket --insecure
./mc cp /tmp/data/mpartobj silo1/test-bucket/mpartobj --enc-c "silo1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize silo1/test-bucket --insecure
./mc put /tmp/data/custpartsize silo1/test-bucket --insecure --part-size 50MiB
sleep 120
# List the objects from source site
echo "Objects from source instance"
./mc ls minio1/test-bucket --insecure
count1=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l)
./mc ls silo1/test-bucket --insecure
count1=$(./mc ls silo1/test-bucket/encrypted --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/encrypted not found"
echo "BUG: object silo1/test-bucket/encrypted not found"
exit_1
fi
count2=$(./mc ls minio1/test-bucket/mpartobj --insecure | wc -l)
count2=$(./mc ls silo1/test-bucket/mpartobj --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/mpartobj not found"
echo "BUG: object silo1/test-bucket/mpartobj not found"
exit_1
fi
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l)
count3=$(./mc ls silo1/test-bucket/defpartsize --insecure | wc -l)
if [ "${count3}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/defpartsize not found"
echo "BUG: object silo1/test-bucket/defpartsize not found"
exit_1
fi
count4=$(./mc ls minio1/test-bucket/custpartsize --insecure | wc -l)
count4=$(./mc ls silo1/test-bucket/custpartsize --insecure | wc -l)
if [ "${count4}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/custpartsize not found"
echo "BUG: object silo1/test-bucket/custpartsize not found"
exit_1
fi
# List the objects from replicated site
echo "Objects from replicated instance"
./mc ls minio2/test-bucket --insecure
repcount1=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l)
./mc ls silo2/test-bucket --insecure
repcount1=$(./mc ls silo2/test-bucket/encrypted --insecure | wc -l)
if [ "${repcount1}" -ne 1 ]; then
echo "BUG: object test-bucket/encrypted not replicated"
exit_1
fi
repcount2=$(./mc ls minio2/test-bucket/mpartobj --insecure | wc -l)
repcount2=$(./mc ls silo2/test-bucket/mpartobj --insecure | wc -l)
if [ "${repcount2}" -ne 1 ]; then
echo "BUG: object test-bucket/mpartobj not replicated"
exit_1
fi
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l)
repcount3=$(./mc ls silo2/test-bucket/defpartsize --insecure | wc -l)
if [ "${repcount3}" -ne 1 ]; then
echo "BUG: object test-bucket/defpartsize not replicated"
exit_1
fi
repcount4=$(./mc ls minio2/test-bucket/custpartsize --insecure | wc -l)
repcount4=$(./mc ls silo2/test-bucket/custpartsize --insecure | wc -l)
if [ "${repcount4}" -ne 1 ]; then
echo "BUG: object test-bucket/custpartsize not replicated"
exit_1
fi
# Stat the objects from source site
echo "Stat minio1/test-bucket/encrypted"
./mc stat --no-list minio1/test-bucket/encrypted --insecure --json
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --insecure --json)
echo "Stat silo1/test-bucket/encrypted"
./mc stat --no-list silo1/test-bucket/encrypted --insecure --json
stat_out1=$(./mc stat --no-list silo1/test-bucket/encrypted --insecure --json)
src_obj1_algo=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
src_obj1_keyid=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio1/test-bucket/defpartsize"
./mc stat --no-list minio1/test-bucket/defpartsize --insecure --json
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --insecure --json)
echo "Stat silo1/test-bucket/defpartsize"
./mc stat --no-list silo1/test-bucket/defpartsize --insecure --json
stat_out2=$(./mc stat --no-list silo1/test-bucket/defpartsize --insecure --json)
src_obj2_algo=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
src_obj2_keyid=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio1/test-bucket/custpartsize"
./mc stat --no-list minio1/test-bucket/custpartsize --insecure --json
stat_out3=$(./mc stat --no-list minio1/test-bucket/custpartsize --insecure --json)
echo "Stat silo1/test-bucket/custpartsize"
./mc stat --no-list silo1/test-bucket/custpartsize --insecure --json
stat_out3=$(./mc stat --no-list silo1/test-bucket/custpartsize --insecure --json)
src_obj3_algo=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
src_obj3_keyid=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio1/test-bucket/mpartobj"
./mc stat --no-list minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out4=$(./mc stat --no-list minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo1/test-bucket/mpartobj"
./mc stat --no-list silo1/test-bucket/mpartobj --enc-c "silo1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out4=$(./mc stat --no-list silo1/test-bucket/mpartobj --enc-c "silo1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj4_etag=$(echo "${stat_out4}" | jq '.etag')
src_obj4_size=$(echo "${stat_out4}" | jq '.size')
src_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Stat the objects from replicated site
echo "Stat minio2/test-bucket/encrypted"
./mc stat --no-list minio2/test-bucket/encrypted --insecure --json
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --insecure --json)
echo "Stat silo2/test-bucket/encrypted"
./mc stat --no-list silo2/test-bucket/encrypted --insecure --json
stat_out1_rep=$(./mc stat --no-list silo2/test-bucket/encrypted --insecure --json)
rep_obj1_algo=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
rep_obj1_keyid=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio2/test-bucket/defpartsize"
./mc stat --no-list minio2/test-bucket/defpartsize --insecure --json
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --insecure --json)
echo "Stat silo2/test-bucket/defpartsize"
./mc stat --no-list silo2/test-bucket/defpartsize --insecure --json
stat_out2_rep=$(./mc stat --no-list silo2/test-bucket/defpartsize --insecure --json)
rep_obj2_algo=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
rep_obj2_keyid=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio2/test-bucket/custpartsize"
./mc stat --no-list minio2/test-bucket/custpartsize --insecure --json
stat_out3_rep=$(./mc stat --no-list minio2/test-bucket/custpartsize --insecure --json)
echo "Stat silo2/test-bucket/custpartsize"
./mc stat --no-list silo2/test-bucket/custpartsize --insecure --json
stat_out3_rep=$(./mc stat --no-list silo2/test-bucket/custpartsize --insecure --json)
rep_obj3_algo=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
rep_obj3_keyid=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio2/test-bucket/mpartobj"
./mc stat --no-list minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out4_rep=$(./mc stat --no-list minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo2/test-bucket/mpartobj"
./mc stat --no-list silo2/test-bucket/mpartobj --enc-c "silo2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out4_rep=$(./mc stat --no-list silo2/test-bucket/mpartobj --enc-c "silo2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj4_etag=$(echo "${stat_out4}" | jq '.etag')
rep_obj4_size=$(echo "${stat_out4}" | jq '.size')
rep_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Check the algo and keyId of replicated objects
if [ "${rep_obj1_algo}" != "${src_obj1_algo}" ]; then
echo "BUG: Algorithm: '${rep_obj1_algo}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_algo}'"
echo "BUG: Algorithm: '${rep_obj1_algo}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_algo}'"
exit_1
fi
if [ "${rep_obj1_keyid}" != "${src_obj1_keyid}" ]; then
echo "BUG: KeyId: '${rep_obj1_keyid}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_keyid}'"
echo "BUG: KeyId: '${rep_obj1_keyid}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_keyid}'"
exit_1
fi
if [ "${rep_obj2_algo}" != "${src_obj2_algo}" ]; then
echo "BUG: Algorithm: '${rep_obj2_algo}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_algo}'"
echo "BUG: Algorithm: '${rep_obj2_algo}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_algo}'"
exit_1
fi
if [ "${rep_obj2_keyid}" != "${src_obj2_keyid}" ]; then
echo "BUG: KeyId: '${rep_obj2_keyid}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_keyid}'"
echo "BUG: KeyId: '${rep_obj2_keyid}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_keyid}'"
exit_1
fi
if [ "${rep_obj3_algo}" != "${src_obj3_algo}" ]; then
echo "BUG: Algorithm: '${rep_obj3_algo}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_algo}'"
echo "BUG: Algorithm: '${rep_obj3_algo}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_algo}'"
exit_1
fi
if [ "${rep_obj3_keyid}" != "${src_obj3_keyid}" ]; then
echo "BUG: KeyId: '${rep_obj3_keyid}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_keyid}'"
echo "BUG: KeyId: '${rep_obj3_keyid}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_keyid}'"
exit_1
fi
# Check the etag, size and md5 of replicated SSEC object
if [ "${rep_obj4_etag}" != "${src_obj4_etag}" ]; then
echo "BUG: Etag: '${rep_obj4_etag}' of replicated object: 'minio2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_etag}'"
echo "BUG: Etag: '${rep_obj4_etag}' of replicated object: 'silo2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_etag}'"
exit_1
fi
if [ "${rep_obj4_size}" != "${src_obj4_size}" ]; then
echo "BUG: Size: '${rep_obj4_size}' of replicated object: 'minio2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_size}'"
echo "BUG: Size: '${rep_obj4_size}' of replicated object: 'silo2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_size}'"
exit_1
fi
if [ "${src_obj4_md5}" != "${rep_obj4_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/mpartobj' doesn't match with source. Expected: '${src_obj4_md5}', Found: '${rep_obj4_md5}'"
echo "BUG: MD5 checksum of object 'silo2/test-bucket/mpartobj' doesn't match with source. Expected: '${src_obj4_md5}', Found: '${rep_obj4_md5}'"
exit_1
fi
# Check content of replicated objects
./mc cat minio2/test-bucket/encrypted --insecure
./mc cat minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat minio2/test-bucket/defpartsize --insecure >/dev/null || exit_1
./mc cat minio2/test-bucket/custpartsize --insecure >/dev/null || exit_1
./mc cat silo2/test-bucket/encrypted --insecure
./mc cat silo2/test-bucket/mpartobj --enc-c "silo2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat silo2/test-bucket/defpartsize --insecure >/dev/null || exit_1
./mc cat silo2/test-bucket/custpartsize --insecure >/dev/null || exit_1
echo -n "Starting MinIO instances with different kms key ..."
CI=on MINIO_KMS_SECRET_KEY=minio3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/minio3/disk{1...4} >/tmp/minio3_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/minio4/disk{1...4} >/tmp/minio4_1.log 2>&1 &
echo -n "Starting Silo instances with different kms key ..."
CI=on MINIO_KMS_SECRET_KEY=silo3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/silo3/disk{1...4} >/tmp/silo3_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=silo4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/silo4/disk{1...4} >/tmp/silo4_1.log 2>&1 &
echo "done"
export MC_HOST_minio3=https://minio:minio123@localhost:9003
export MC_HOST_minio4=https://minio:minio123@localhost:9004
export MC_HOST_silo3=https://minio:silo123@localhost:9003
export MC_HOST_silo4=https://minio:silo123@localhost:9004
./mc ready minio3 --insecure
./mc ready minio4 --insecure
./mc ready silo3 --insecure
./mc ready silo4 --insecure
./mc admin replicate add minio3 minio4 --insecure
./mc mb minio3/bucket --insecure
./mc cp --insecure --enc-kms minio3/bucket=minio3-default-key /tmp/data/encrypted minio3/bucket/x
./mc admin replicate add silo3 silo4 --insecure
./mc mb silo3/bucket --insecure
./mc cp --insecure --enc-kms silo3/bucket=silo3-default-key /tmp/data/encrypted silo3/bucket/x
sleep 10
st=$(./mc stat --json --no-list --insecure minio3/bucket/x | jq -r .replicationStatus)
st=$(./mc stat --json --no-list --insecure silo3/bucket/x | jq -r .replicationStatus)
if [ "${st}" != "FAILED" ]; then
echo "BUG: Replication succeeded when kms key is different"
exit_1
docs/site-replication/run-ssec-object-replication-with-compression.sh
+63 -64
View File
@@ -4,19 +4,19 @@
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
echo "minio2 ============"
cat /tmp/minio2_1.log
echo "silo1 ============"
cat /tmp/silo1_1.log
echo "silo2 ============"
cat /tmp/silo2_1.log
exit 1
}
cleanup() {
echo -n "Cleaning up instances of MinIO ..."
pkill minio || sudo pkill minio
pkill -9 minio || sudo pkill -9 minio
rm -rf /tmp/minio{1,2}
echo -n "Cleaning up instances of Silo ..."
pkill silo || sudo pkill silo
pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/silo{1,2}
echo "done"
}
@@ -25,11 +25,11 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_PASSWORD="silo123"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled MinIO
echo -n "Setup certs for MinIO instances ..."
# Create certificates for TLS enabled Silo
echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost
mkdir -p /tmp/certs
@@ -37,24 +37,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done"
# Start MinIO instances
echo -n "Starting MinIO instances ..."
minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
# Start Silo instances
echo -n "Starting Silo instances ..."
silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
echo -n "Downloading Silo client ..."
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
echo "done"
fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure
./mc ready minio2 --insecure
./mc ready silo1 --insecure
./mc ready silo2 --insecure
# Prepare data for tests
echo -n "Preparing test data ..."
@@ -67,129 +66,129 @@ touch /tmp/data/mpartobj.txt
shred -s 500M /tmp/data/mpartobj.txt
echo "done"
# Enable compression for site minio1
./mc admin config set minio1 compression enable=on extensions=".txt" --insecure
./mc admin config set minio1 compression allow_encryption=off --insecure
# Enable compression for site silo1
./mc admin config set silo1 compression enable=on extensions=".txt" --insecure
./mc admin config set silo1 compression allow_encryption=off --insecure
# Create bucket in source cluster
echo "Create bucket in source MinIO instance"
./mc mb minio1/test-bucket --insecure
echo "Create bucket in source Silo instance"
./mc mb silo1/test-bucket --insecure
# Load objects to source site
echo "Loading objects to source MinIO instance"
./mc cp /tmp/data/plainfile minio1/test-bucket --insecure
./mc cp /tmp/data/encrypted minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
echo "Loading objects to source Silo instance"
./mc cp /tmp/data/plainfile silo1/test-bucket --insecure
./mc cp /tmp/data/encrypted silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
# Below should fail as compression and SSEC used at the same time
# DISABLED: We must check the response header to see if compression was actually applied
#RESULT=$({ ./mc put /tmp/data/mpartobj.txt minio1/test-bucket/mpartobj.txt --enc-c "minio1/test-bucket/mpartobj.txt=${TEST_MINIO_ENC_KEY}" --insecure; } 2>&1)
#RESULT=$({ ./mc put /tmp/data/mpartobj.txt silo1/test-bucket/mpartobj.txt --enc-c "silo1/test-bucket/mpartobj.txt=${TEST_MINIO_ENC_KEY}" --insecure; } 2>&1)
#if [[ ${RESULT} != *"Server side encryption specified with SSE-C with compression not allowed"* ]]; then
# echo "BUG: Loading an SSE-C object to site with compression should fail. Succeeded though."
# exit_1
#fi
# Add replication site
./mc admin replicate add minio1 minio2 --insecure
./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete
sleep 30
# List the objects from source site
echo "Objects from source instance"
./mc ls minio1/test-bucket --insecure
count1=$(./mc ls minio1/test-bucket/plainfile --insecure | wc -l)
./mc ls silo1/test-bucket --insecure
count1=$(./mc ls silo1/test-bucket/plainfile --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/plainfile not found"
echo "BUG: object silo1/test-bucket/plainfile not found"
exit_1
fi
count2=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l)
count2=$(./mc ls silo1/test-bucket/encrypted --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/encrypted not found"
echo "BUG: object silo1/test-bucket/encrypted not found"
exit_1
fi
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l)
count3=$(./mc ls silo1/test-bucket/defpartsize --insecure | wc -l)
if [ "${count3}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/defpartsize not found"
echo "BUG: object silo1/test-bucket/defpartsize not found"
exit_1
fi
sleep 120
# List the objects from replicated site
echo "Objects from replicated instance"
./mc ls minio2/test-bucket --insecure
repcount1=$(./mc ls minio2/test-bucket/plainfile --insecure | wc -l)
./mc ls silo2/test-bucket --insecure
repcount1=$(./mc ls silo2/test-bucket/plainfile --insecure | wc -l)
if [ "${repcount1}" -ne 1 ]; then
echo "BUG: object test-bucket/plainfile not replicated"
exit_1
fi
repcount2=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l)
repcount2=$(./mc ls silo2/test-bucket/encrypted --insecure | wc -l)
if [ "${repcount2}" -ne 1 ]; then
echo "BUG: object test-bucket/encrypted not replicated"
exit_1
fi
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l)
repcount3=$(./mc ls silo2/test-bucket/defpartsize --insecure | wc -l)
if [ "${repcount3}" -ne 1 ]; then
echo "BUG: object test-bucket/defpartsize not replicated"
exit_1
fi
# Stat the SSEC objects from source site
echo "Stat minio1/test-bucket/encrypted"
./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo1/test-bucket/encrypted"
./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1=$(./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj1_etag=$(echo "${stat_out1}" | jq '.etag')
src_obj1_size=$(echo "${stat_out1}" | jq '.size')
src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio1/test-bucket/defpartsize"
./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo1/test-bucket/defpartsize"
./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2=$(./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj2_etag=$(echo "${stat_out2}" | jq '.etag')
src_obj2_size=$(echo "${stat_out2}" | jq '.size')
src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Stat the SSEC objects from replicated site
echo "Stat minio2/test-bucket/encrypted"
./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo2/test-bucket/encrypted"
./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1_rep=$(./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag')
rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size')
rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio2/test-bucket/defpartsize"
./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo2/test-bucket/defpartsize"
./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2_rep=$(./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag')
rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size')
rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Check the etag and size of replicated SSEC objects
if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
exit_1
fi
if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
exit_1
fi
if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
exit_1
fi
if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
exit_1
fi
# Check content of replicated SSEC objects
./mc cat minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cat minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cat silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
# Check the MD5 checksums of encrypted objects from source and target
if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
echo "BUG: MD5 checksum of object 'silo2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
exit_1
fi
if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
echo "BUG: MD5 checksum of object 'silo2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
exit_1
fi
docs/site-replication/run-ssec-object-replication.sh
+73 -74
View File
@@ -4,19 +4,19 @@
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
echo "minio2 ============"
cat /tmp/minio2_1.log
echo "silo1 ============"
cat /tmp/silo1_1.log
echo "silo2 ============"
cat /tmp/silo2_1.log
exit 1
}
cleanup() {
echo -n "Cleaning up instances of MinIO ..."
pkill minio || sudo pkill minio
pkill -9 minio || sudo pkill -9 minio
rm -rf /tmp/minio{1,2}
echo -n "Cleaning up instances of Silo ..."
pkill silo || sudo pkill silo
pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/silo{1,2}
echo "done"
}
@@ -25,11 +25,11 @@ cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_PASSWORD="silo123"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled MinIO
echo -n "Setup certs for MinIO instances ..."
# Create certificates for TLS enabled Silo
echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost
mkdir -p /tmp/certs
@@ -37,24 +37,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done"
# Start MinIO instances
echo -n "Starting MinIO instances ..."
minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
# Start Silo instances
echo -n "Starting Silo instances ..."
silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
echo -n "Downloading Silo client ..."
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
echo "done"
fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002
export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure
./mc ready minio2 --insecure
./mc ready silo1 --insecure
./mc ready silo2 --insecure
# Prepare data for tests
echo -n "Preparing test data ..."
@@ -68,155 +67,155 @@ shred -s 500M /tmp/data/custpartsize
echo "done"
# Add replication site
./mc admin replicate add minio1 minio2 --insecure
./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete
sleep 30
# Create bucket in source cluster
echo "Create bucket in source MinIO instance"
./mc mb minio1/test-bucket --insecure
echo "Create bucket in source Silo instance"
./mc mb silo1/test-bucket --insecure
# Load objects to source site
echo "Loading objects to source MinIO instance"
echo "Loading objects to source Silo instance"
set -x
./mc cp /tmp/data/plainfile minio1/test-bucket --insecure
./mc cp /tmp/data/encrypted minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
./mc put /tmp/data/custpartsize minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --part-size 50MiB
./mc cp /tmp/data/plainfile silo1/test-bucket --insecure
./mc cp /tmp/data/encrypted silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
./mc put /tmp/data/custpartsize silo1/test-bucket/custpartsize --enc-c "silo1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --part-size 50MiB
set +x
sleep 120
# List the objects from source site
echo "Objects from source instance"
./mc ls minio1/test-bucket --insecure
count1=$(./mc ls minio1/test-bucket/plainfile --insecure | wc -l)
./mc ls silo1/test-bucket --insecure
count1=$(./mc ls silo1/test-bucket/plainfile --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/plainfile not found"
echo "BUG: object silo1/test-bucket/plainfile not found"
exit_1
fi
count2=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l)
count2=$(./mc ls silo1/test-bucket/encrypted --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/encrypted not found"
echo "BUG: object silo1/test-bucket/encrypted not found"
exit_1
fi
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l)
count3=$(./mc ls silo1/test-bucket/defpartsize --insecure | wc -l)
if [ "${count3}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/defpartsize not found"
echo "BUG: object silo1/test-bucket/defpartsize not found"
exit_1
fi
count4=$(./mc ls minio1/test-bucket/custpartsize --insecure | wc -l)
count4=$(./mc ls silo1/test-bucket/custpartsize --insecure | wc -l)
if [ "${count4}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/custpartsize not found"
echo "BUG: object silo1/test-bucket/custpartsize not found"
exit_1
fi
# List the objects from replicated site
echo "Objects from replicated instance"
./mc ls minio2/test-bucket --insecure
repcount1=$(./mc ls minio2/test-bucket/plainfile --insecure | wc -l)
./mc ls silo2/test-bucket --insecure
repcount1=$(./mc ls silo2/test-bucket/plainfile --insecure | wc -l)
if [ "${repcount1}" -ne 1 ]; then
echo "BUG: object test-bucket/plainfile not replicated"
exit_1
fi
repcount2=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l)
repcount2=$(./mc ls silo2/test-bucket/encrypted --insecure | wc -l)
if [ "${repcount2}" -ne 1 ]; then
echo "BUG: object test-bucket/encrypted not replicated"
exit_1
fi
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l)
repcount3=$(./mc ls silo2/test-bucket/defpartsize --insecure | wc -l)
if [ "${repcount3}" -ne 1 ]; then
echo "BUG: object test-bucket/defpartsize not replicated"
exit_1
fi
repcount4=$(./mc ls minio2/test-bucket/custpartsize --insecure | wc -l)
repcount4=$(./mc ls silo2/test-bucket/custpartsize --insecure | wc -l)
if [ "${repcount4}" -ne 1 ]; then
echo "BUG: object test-bucket/custpartsize not replicated"
exit_1
fi
# Stat the SSEC objects from source site
echo "Stat minio1/test-bucket/encrypted"
./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo1/test-bucket/encrypted"
./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1=$(./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj1_etag=$(echo "${stat_out1}" | jq '.etag')
src_obj1_size=$(echo "${stat_out1}" | jq '.size')
src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio1/test-bucket/defpartsize"
./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo1/test-bucket/defpartsize"
./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2=$(./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj2_etag=$(echo "${stat_out2}" | jq '.etag')
src_obj2_size=$(echo "${stat_out2}" | jq '.size')
src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio1/test-bucket/custpartsize"
./mc stat --no-list minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out3=$(./mc stat --no-list minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo1/test-bucket/custpartsize"
./mc stat --no-list silo1/test-bucket/custpartsize --enc-c "silo1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out3=$(./mc stat --no-list silo1/test-bucket/custpartsize --enc-c "silo1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj3_etag=$(echo "${stat_out3}" | jq '.etag')
src_obj3_size=$(echo "${stat_out3}" | jq '.size')
src_obj3_md5=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Stat the SSEC objects from replicated site
echo "Stat minio2/test-bucket/encrypted"
./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo2/test-bucket/encrypted"
./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1_rep=$(./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag')
rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size')
rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio2/test-bucket/defpartsize"
./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo2/test-bucket/defpartsize"
./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2_rep=$(./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag')
rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size')
rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio2/test-bucket/custpartsize"
./mc stat --no-list minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out3_rep=$(./mc stat --no-list minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
echo "Stat silo2/test-bucket/custpartsize"
./mc stat --no-list silo2/test-bucket/custpartsize --enc-c "silo2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out3_rep=$(./mc stat --no-list silo2/test-bucket/custpartsize --enc-c "silo2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj3_etag=$(echo "${stat_out3_rep}" | jq '.etag')
rep_obj3_size=$(echo "${stat_out3_rep}" | jq '.size')
rep_obj3_md5=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Check the etag and size of replicated SSEC objects
if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
exit_1
fi
if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
exit_1
fi
if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
exit_1
fi
if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
exit_1
fi
if [ "${rep_obj3_etag}" != "${src_obj3_etag}" ]; then
echo "BUG: Etag: '${rep_obj3_etag}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_etag}'"
echo "BUG: Etag: '${rep_obj3_etag}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_etag}'"
exit_1
fi
if [ "${rep_obj3_size}" != "${src_obj3_size}" ]; then
echo "BUG: Size: '${rep_obj3_size}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_size}'"
echo "BUG: Size: '${rep_obj3_size}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_size}'"
exit_1
fi
# Check content of replicated SSEC objects
./mc cat minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cat minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cat silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat silo2/test-bucket/custpartsize --enc-c "silo2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
# Check the MD5 checksums of encrypted objects from source and target
if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
echo "BUG: MD5 checksum of object 'silo2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
exit_1
fi
if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
echo "BUG: MD5 checksum of object 'silo2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
exit_1
fi
if [ "${src_obj3_md5}" != "${rep_obj3_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/custpartsize' doesn't match with source. Expected: '${src_obj3_md5}', Found: '${rep_obj3_md5}'"
echo "BUG: MD5 checksum of object 'silo2/test-bucket/custpartsize' doesn't match with source. Expected: '${src_obj3_md5}', Found: '${rep_obj3_md5}'"
exit_1
fi