test: pin the external fixtures and run the suites against the silo binary

The test and verification scripts invoked ./minio and pulled their tooling from
upstream infrastructure with no integrity check. Every `curl | tar` of a client
or an old server binary was an unverified execution path in a script that
regularly runs as a privileged user, and several fetched a floating "latest".

Two installers replace all of it:

- install-mcli.sh resolves a pinned pgsty/mc release, downloads the archive and
  its checksum manifest, requires exactly one valid manifest entry for the
  asset, verifies it, and installs. MCLI_BIN with a mandatory MCLI_SHA256 lets
  an offline or air-gapped run supply its own binary, still checksum-checked.
- install-verified-fixture.sh takes source, expected SHA-256 and target, and
  refuses anything that does not match. Sources may be a URL or a local file.

Every script that previously downloaded mc now calls install-mcli.sh. The three
places that genuinely need an upstream artifact - the old MinIO server binary
for the LDAP IAM upgrade-import test, the 2021 mc for the three-site
replication test, and the functional-tests.sh fixture - go through
install-verified-fixture.sh with the digest recorded inline. Those dl.min.io
URLs remain on purpose: they are historical upstream artifacts needed to prove
upgrade compatibility, and they are now pinned and verified rather than
trusted.

The scripts otherwise switch to ./silo, silo.service, the silo container and
compose service names, and SILO_CONFIG_DIR. run-multi-site-minio-idp.sh is
renamed to run-multi-site-silo-idp.sh with the Makefile target following.
buildscripts/minio-upgrade.sh keeps its name and its `minio server` argv - it
exists to test the MinIO-to-Silo upgrade, so the old side must stay old - but
it is now pinned to an image digest rather than a tag, and its `docker system
prune` and `docker volume prune` calls are removed. Those ran unfiltered
against the developer's whole Docker installation; the resiliency tests had the
same problem and lose their prune and `docker ps -q` sweeps too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Feng Ruohang
2026-08-06 08:48:58 +08:00
parent bd8df51665
commit 6613c2a3cb
52 changed files with 2046 additions and 1989 deletions
+22 -23
View File
@@ -3,7 +3,7 @@
set -x set -x
export MINIO_CI_CD=1 export MINIO_CI_CD=1
killall -9 minio killall -9 silo
rm -rf ${HOME}/tmp/dist rm -rf ${HOME}/tmp/dist
@@ -19,28 +19,27 @@ done
echo $args echo $args
for ((i = 0; i < $((nr_servers)); i++)); do for ((i = 0; i < $((nr_servers)); i++)); do
(minio server --address ":$((9100 + i))" $args 2>&1 >/tmp/log$i.txt) & (silo server --address ":$((9100 + i))" $args 2>&1 >/tmp/log$i.txt) &
done done
sleep 10s sleep 10s
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O ./mc https://dl.minio.io/client/mc/release/linux-amd64/./mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
set +e set +e
export MC_HOST_minioadm=http://minioadmin:minioadmin@localhost:9100/ export MC_HOST_siloadm=http://minioadmin:minioadmin@localhost:9100/
./mc ready minioadm ./mc ready siloadm
./mc ls minioadm/ ./mc ls siloadm/
./mc admin config set minioadm/ api root_access=off ./mc admin config set siloadm/ api root_access=off
sleep 3s # let things settle a little sleep 3s # let things settle a little
./mc ls minioadm/ ./mc ls siloadm/
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "listing succeeded, 'minioadmin' was not disabled" echo "listing succeeded, 'minioadmin' was not disabled"
exit 1 exit 1
@@ -48,38 +47,38 @@ fi
set -e set -e
killall -9 minio killall -9 silo
export MINIO_API_ROOT_ACCESS=on export MINIO_API_ROOT_ACCESS=on
for ((i = 0; i < $((nr_servers)); i++)); do for ((i = 0; i < $((nr_servers)); i++)); do
(minio server --address ":$((9100 + i))" $args 2>&1 >/tmp/log$i.txt) & (silo server --address ":$((9100 + i))" $args 2>&1 >/tmp/log$i.txt) &
done done
set +e set +e
./mc ready minioadm/ ./mc ready siloadm/
./mc ls minioadm/ ./mc ls siloadm/
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "listing failed, 'minioadmin' should be enabled" echo "listing failed, 'minioadmin' should be enabled"
exit 1 exit 1
fi fi
killall -9 minio killall -9 silo
rm -rf /tmp/multisitea/ rm -rf /tmp/multisitea/
rm -rf /tmp/multisiteb/ rm -rf /tmp/multisiteb/
echo "Setup site-replication and then disable root credentials" echo "Setup site-replication and then disable root credentials"
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001 export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001
@@ -96,19 +95,19 @@ export MC_HOST_siteb=http://minioadmin:minioadmin@127.0.0.1:9004
./mc admin user info siteb foobar ./mc admin user info siteb foobar
killall -9 minio killall -9 silo
echo "turning off root access, however site replication must continue" echo "turning off root access, however site replication must continue"
export MINIO_API_ROOT_ACCESS=off export MINIO_API_ROOT_ACCESS=off
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://foobar:foo12345@127.0.0.1:9001 export MC_HOST_sitea=http://foobar:foo12345@127.0.0.1:9001
+19 -30
View File
@@ -5,45 +5,34 @@ set -o pipefail
set -x set -x
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
function start_minio_4drive() { function start_silo_4drive() {
start_port=$1 start_port=$1
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
export MC_HOST_minio="http://minio:minio123@127.0.0.1:${start_port}/" export MC_HOST_silo="http://silo:silo123@127.0.0.1:${start_port}/"
unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects
export MINIO_CI_CD=1 export MINIO_CI_CD=1
mkdir ${WORK_DIR} mkdir ${WORK_DIR}
C_PWD=${PWD}
if [ ! -x "$PWD/mc" ]; then if [ ! -x "$PWD/mc" ]; then
MC_BUILD_DIR="mc-$RANDOM" "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "$PWD/mc"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "$C_PWD/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
fi fi
"${MINIO[@]}" --address ":$start_port" "${WORK_DIR}/disk{1...4}" >"${WORK_DIR}/server1.log" 2>&1 & "${SILO[@]}" --address ":$start_port" "${WORK_DIR}/disk{1...4}" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$! pid=$!
disown $pid disown $pid
sleep 5 sleep 5
@@ -56,30 +45,30 @@ function start_minio_4drive() {
exit 1 exit 1
fi fi
"${PWD}/mc" mb --with-versioning minio/bucket "${PWD}/mc" mb --with-versioning silo/bucket
for i in $(seq 1 4); do for i in $(seq 1 4); do
"${PWD}/mc" cp /etc/hosts minio/bucket/testobj "${PWD}/mc" cp /etc/hosts silo/bucket/testobj
sudo chown -R root. "${WORK_DIR}/disk${i}" sudo chown -R root. "${WORK_DIR}/disk${i}"
"${PWD}/mc" cp /etc/hosts minio/bucket/testobj "${PWD}/mc" cp /etc/hosts silo/bucket/testobj
sudo chown -R ${USER}. "${WORK_DIR}/disk${i}" sudo chown -R ${USER}. "${WORK_DIR}/disk${i}"
done done
for vid in $("${PWD}/mc" ls --json --versions minio/bucket/testobj | jq -r .versionId); do for vid in $("${PWD}/mc" ls --json --versions silo/bucket/testobj | jq -r .versionId); do
"${PWD}/mc" cat --vid "${vid}" minio/bucket/testobj | md5sum "${PWD}/mc" cat --vid "${vid}" silo/bucket/testobj | md5sum
done done
pkill minio pkill silo
sleep 3 sleep 3
} }
function main() { function main() {
start_port=$(shuf -i 10000-65000 -n 1) start_port=$(shuf -i 10000-65000 -n 1)
start_minio_4drive ${start_port} start_silo_4drive ${start_port}
} }
function purge() { function purge() {
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
set -euo pipefail
if [ "$#" -ne 1 ]; then
echo "usage: $0 TARGET" >&2
exit 2
fi
target=$1
target_dir=$(dirname "${target}")
if [ ! -d "${target_dir}" ]; then
echo "target directory does not exist: ${target_dir}" >&2
exit 1
fi
sha256_file() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{print $1}'
else
shasum -a 256 "$1" | awk '{print $1}'
fi
}
if [ -n "${MCLI_BIN:-}" ]; then
if [ ! -f "${MCLI_BIN}" ]; then
echo "MCLI_BIN is not a regular file: ${MCLI_BIN}" >&2
exit 1
fi
if ! printf '%s\n' "${MCLI_SHA256:-}" | grep -Eq '^[0-9a-fA-F]{64}$'; then
echo "MCLI_SHA256 must contain the expected SHA-256 for MCLI_BIN" >&2
exit 1
fi
actual=$(sha256_file "${MCLI_BIN}")
if [ "${actual}" != "${MCLI_SHA256,,}" ]; then
echo "MCLI_BIN checksum mismatch: expected ${MCLI_SHA256,,}, got ${actual}" >&2
exit 1
fi
install -m 0755 "${MCLI_BIN}" "${target}"
exit 0
fi
release=${MCLI_RELEASE:-RELEASE.2026-08-04T00-00-00Z}
version_hyphen=${release#RELEASE.}
package_version=$(printf '%s\n' "${version_hyphen}" | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2})-([0-9]{2})-([0-9]{2})Z$/\1\2\3\4\5\6.0.0/')
if [ "${package_version}" = "${version_hyphen}" ]; then
echo "invalid MCLI_RELEASE: ${release}" >&2
exit 1
fi
case $(uname -s) in
Linux) os=linux ;;
Darwin) os=darwin ;;
*) echo "unsupported mcli host OS: $(uname -s)" >&2; exit 1 ;;
esac
case $(uname -m) in
x86_64 | amd64) arch=amd64 ;;
aarch64 | arm64) arch=arm64 ;;
*) echo "unsupported mcli host architecture: $(uname -m)" >&2; exit 1 ;;
esac
archive="mcli_${package_version}_${os}_${arch}.tar.gz"
checksums="mcli_${package_version}_checksums.txt"
base_url="https://github.com/pgsty/mc/releases/download/${release}"
tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/silo-mcli.XXXXXX")
trap 'rm -rf "${tmp_dir}"' EXIT
curl --fail --location --retry 3 --silent --show-error \
"${base_url}/${checksums}" --output "${tmp_dir}/${checksums}"
curl --fail --location --retry 3 --silent --show-error \
"${base_url}/${archive}" --output "${tmp_dir}/${archive}"
expected=$(awk -v asset="${archive}" '
{
name=$2
sub(/^\*/, "", name)
if (name == asset && $1 ~ /^[0-9a-fA-F]{64}$/) print tolower($1)
}
' "${tmp_dir}/${checksums}")
if ! printf '%s\n' "${expected}" | grep -Eq '^[0-9a-f]{64}$'; then
echo "checksum manifest does not contain exactly one valid entry for ${archive}" >&2
exit 1
fi
actual=$(sha256_file "${tmp_dir}/${archive}")
if [ "${actual}" != "${expected}" ]; then
echo "downloaded ${archive} checksum mismatch" >&2
exit 1
fi
tar -xzf "${tmp_dir}/${archive}" -C "${tmp_dir}" mcli
install -m 0755 "${tmp_dir}/mcli" "${target}"
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
set -euo pipefail
if [ "$#" -ne 3 ]; then
echo "usage: $0 SOURCE SHA256 TARGET" >&2
exit 2
fi
source_ref=$1
expected=${2,,}
target=$3
if ! printf '%s\n' "${expected}" | grep -Eq '^[0-9a-f]{64}$'; then
echo "expected checksum must be a lowercase SHA-256 digest" >&2
exit 1
fi
if [ ! -d "$(dirname "${target}")" ]; then
echo "target directory does not exist: $(dirname "${target}")" >&2
exit 1
fi
tmp_file=$(mktemp "${TMPDIR:-/tmp}/silo-fixture.XXXXXX")
trap 'rm -f "${tmp_file}"' EXIT
case ${source_ref} in
https://*)
curl --fail --location --retry 3 --silent --show-error \
"${source_ref}" --output "${tmp_file}"
;;
*)
if [ ! -f "${source_ref}" ]; then
echo "fixture is not a regular file: ${source_ref}" >&2
exit 1
fi
cp "${source_ref}" "${tmp_file}"
;;
esac
if command -v sha256sum >/dev/null 2>&1; then
actual=$(sha256sum "${tmp_file}" | awk '{print $1}')
else
actual=$(shasum -a 256 "${tmp_file}" | awk '{print $1}')
fi
if [ "${actual}" != "${expected}" ]; then
echo "fixture checksum mismatch: expected ${expected}, got ${actual}" >&2
exit 1
fi
install -m 0755 "${tmp_file}" "${target}"
@@ -1,7 +1,7 @@
#!/bin/bash #!/bin/bash
# This script is used to test the migration of IAM content from old minio # This script tests IAM migration from an old MinIO compatibility fixture into
# instance to new minio instance. # the current Silo server.
# #
# To run it locally, start the LDAP server in github.com/minio/minio-iam-testing # To run it locally, start the LDAP server in github.com/minio/minio-iam-testing
# repo (e.g. make podman-run), and then run this script. # repo (e.g. make podman-run), and then run this script.
@@ -15,6 +15,7 @@
OLD_VERSION=RELEASE.2024-03-26T22-10-45Z OLD_VERSION=RELEASE.2024-03-26T22-10-45Z
OLD_BINARY_LINK=https://dl.min.io/server/minio/release/linux-amd64/archive/minio.${OLD_VERSION} OLD_BINARY_LINK=https://dl.min.io/server/minio/release/linux-amd64/archive/minio.${OLD_VERSION}
OLD_BINARY_SHA256=2050199d89e3057571620a1d453118fed5bd2de9d4f3b266b11365fdf984d676
__init__() { __init__() {
if which curl &>/dev/null; then if which curl &>/dev/null; then
@@ -27,17 +28,16 @@ __init__() {
export GOPATH=/tmp/gopath export GOPATH=/tmp/gopath
export PATH="${PATH}":"${GOPATH}"/bin export PATH="${PATH}":"${GOPATH}"/bin
if which mc &>/dev/null; then if [ ! -x "${GOPATH}/bin/mc" ]; then
echo "mc is already installed" echo "Installing verified compatible client fixture"
else mkdir -p "${GOPATH}/bin"
echo "Installing mc:" "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "${GOPATH}/bin/mc"
go install github.com/minio/mc@latest
fi fi
if [ ! -x ./minio.${OLD_VERSION} ]; then if [ ! -x ./minio.${OLD_VERSION} ]; then
echo "Downloading minio.${OLD_VERSION} binary" echo "Installing verified upstream compatibility fixture minio.${OLD_VERSION}"
curl -o minio.${OLD_VERSION} ${OLD_BINARY_LINK} "$(git rev-parse --show-toplevel)/buildscripts/install-verified-fixture.sh" \
chmod +x minio.${OLD_VERSION} "${OLD_BINARY_LINK}" "${OLD_BINARY_SHA256}" "minio.${OLD_VERSION}"
fi fi
if [ -z "$_MINIO_LDAP_TEST_SERVER" ]; then if [ -z "$_MINIO_LDAP_TEST_SERVER" ]; then
@@ -49,7 +49,7 @@ __init__() {
} }
create_iam_content_in_old_minio() { create_iam_content_in_old_minio() {
echo "Creating IAM content in old minio instance." echo "Creating IAM content in the old MinIO compatibility fixture."
MINIO_CI_CD=1 ./minio.${OLD_VERSION} server /tmp/data/{1...4} & MINIO_CI_CD=1 ./minio.${OLD_VERSION} server /tmp/data/{1...4} &
sleep 5 sleep 5
@@ -80,9 +80,9 @@ create_iam_content_in_old_minio() {
} }
import_iam_content_in_new_minio() { import_iam_content_in_new_minio() {
echo "Importing IAM content in new minio instance." echo "Importing IAM content into the current Silo instance."
# Assume current minio binary exists. # Assume the current Silo binary exists.
MINIO_CI_CD=1 ./minio server /tmp/data/{1...4} & MINIO_CI_CD=1 ./silo server /tmp/data/{1...4} &
sleep 5 sleep 5
set -x set -x
+113 -102
View File
@@ -1,113 +1,124 @@
#!/bin/bash #!/usr/bin/env bash
trap 'cleanup $LINENO' ERR set -euo pipefail
# Exercise both directions of the on-disk compatibility contract using an
# immutable pre-rebrand image and a container built from the current checkout.
# Every Docker resource is uniquely named and removed explicitly; this test
# never prunes unrelated images, containers, networks, or volumes.
repo_dir="$(git rev-parse --show-toplevel)"
old_image="${OLD_IMAGE:-docker.io/pgsty/minio@sha256:b6bfe7239bfc83fb90d31612d9704d86039dd714f7904b3f1ad68f211e602372}"
new_image="${NEW_IMAGE:-silo-upgrade-test:dev}"
suffix="$(date +%s)-$$"
network="silo-upgrade-net-${suffix}"
volume="silo-upgrade-data-${suffix}"
old_container="silo-upgrade-old-${suffix}"
new_container="silo-upgrade-new-${suffix}"
rollback_container="silo-upgrade-rollback-${suffix}"
root_user=silo-upgrade-admin
root_password=silo-upgrade-secret-123
# shellcheck disable=SC2120
cleanup() { cleanup() {
MINIO_VERSION=dev /tmp/gopath/bin/docker-compose \ status=$?
-f "buildscripts/upgrade-tests/compose.yml" \ trap - EXIT
down || true if [ "${status}" -ne 0 ]; then
for name in "${old_container}" "${new_container}" "${rollback_container}"; do
MINIO_VERSION=dev /tmp/gopath/bin/docker-compose \ docker logs "${name}" 2>/dev/null | tail -n 80 >&2 || true
-f "buildscripts/upgrade-tests/compose.yml" \
rm || true
for volume in $(docker volume ls -q | grep upgrade); do
docker volume rm ${volume} || true
done
docker volume prune -f
docker system prune -f || true
docker volume prune -f || true
docker volume rm $(docker volume ls -q -f dangling=true) || true
}
verify_checksum_after_heal() {
local sum1
sum1=$(curl -s "$2" | sha256sum)
mc admin heal --json -r "$1" >/dev/null # test after healing
local sum1_heal
sum1_heal=$(curl -s "$2" | sha256sum)
if [ "${sum1_heal}" != "${sum1}" ]; then
echo "mismatch expected ${sum1_heal}, got ${sum1}"
exit 1
fi
}
verify_checksum_mc() {
local expected
expected=$(mc cat "$1" | sha256sum)
local got
got=$(mc cat "$2" | sha256sum)
if [ "${expected}" != "${got}" ]; then
echo "mismatch - expected ${expected}, got ${got}"
exit 1
fi
echo "matches - ${expected}, got ${got}"
}
add_alias() {
for i in $(seq 1 4); do
echo "... attempting to add alias $i"
until (mc alias set minio http://127.0.0.1:9000 minioadmin minioadmin); do
echo "...waiting... for 5secs" && sleep 5
done done
fi
if [ "${KEEP_UPGRADE_TEST_RESOURCES:-0}" = 1 ]; then
printf 'Retained Docker resources for inspection: %s %s\n' "${network}" "${volume}" >&2
exit "${status}"
fi
docker rm -f "${old_container}" "${new_container}" "${rollback_container}" >/dev/null 2>&1 || true
docker network rm "${network}" >/dev/null 2>&1 || true
docker volume rm "${volume}" >/dev/null 2>&1 || true
exit "${status}"
}
trap cleanup EXIT
trap 'exit 130' INT TERM
wait_ready() {
name="$1"
ready=""
for _ in $(seq 1 90); do
if docker logs "${name}" 2>&1 | grep -q 'API:'; then
ready=1
break
fi
if [ "$(docker inspect -f '{{.State.Running}}' "${name}")" != true ]; then
break
fi
sleep 1
done done
if [ -z "${ready}" ]; then
echo "Sleeping for nginx" echo "Server did not become ready: ${name}" >&2
sleep 20 return 1
fi
} }
__init__() { start_server() {
sudo apt install curl -y name="$1"
export GOPATH=/tmp/gopath image="$2"
export PATH=${PATH}:${GOPATH}/bin shift 2
docker run -d --name "${name}" --network "${network}" \
go install github.com/minio/mc@latest --mount "source=${volume},target=/data" \
-e MINIO_CI_CD=1 -e MINIO_ROOT_USER="${root_user}" -e MINIO_ROOT_PASSWORD="${root_password}" \
## this is needed because github actions don't have "${image}" "$@" >/dev/null
## docker-compose on all runners wait_ready "${name}"
COMPOSE_VERSION=v2.35.1 docker exec "${name}" mcli alias set local http://127.0.0.1:9000 "${root_user}" "${root_password}" >/dev/null
mkdir -p /tmp/gopath/bin/
wget -O /tmp/gopath/bin/docker-compose https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-linux-x86_64
chmod +x /tmp/gopath/bin/docker-compose
cleanup
TAG=minio/minio:dev make docker
MINIO_VERSION=RELEASE.2019-12-19T22-52-26Z docker-compose \
-f "buildscripts/upgrade-tests/compose.yml" \
up -d --build
add_alias
mc mb minio/minio-test/
mc cp ./minio minio/minio-test/to-read/
mc cp /etc/hosts minio/minio-test/to-read/hosts
mc anonymous set download minio/minio-test
verify_checksum_mc ./minio minio/minio-test/to-read/minio
curl -s http://127.0.0.1:9000/minio-test/to-read/hosts | sha256sum
MINIO_VERSION=dev /tmp/gopath/bin/docker-compose -f "buildscripts/upgrade-tests/compose.yml" stop
} }
main() { stop_server() {
MINIO_VERSION=dev /tmp/gopath/bin/docker-compose -f "buildscripts/upgrade-tests/compose.yml" up -d --build name="$1"
docker stop -t 20 "${name}" >/dev/null
add_alias test "$(docker inspect -f '{{.State.ExitCode}}' "${name}")" = 0
docker logs "${name}" 2>&1 | grep -q 'Exiting on signal'
verify_checksum_after_heal minio/minio-test http://127.0.0.1:9000/minio-test/to-read/hosts docker rm "${name}" >/dev/null
verify_checksum_mc ./minio minio/minio-test/to-read/minio
verify_checksum_mc /etc/hosts minio/minio-test/to-read/hosts
cleanup
} }
(__init__ "$@" && main "$@") command -v docker >/dev/null
docker info >/dev/null
if ! docker image inspect "${old_image}" >/dev/null 2>&1; then
docker pull "${old_image}"
fi
if [ "${SILO_UPGRADE_SKIP_BUILD:-0}" != 1 ]; then
make -C "${repo_dir}" docker TAG="${new_image}"
fi
docker image inspect "${new_image}" >/dev/null
docker network create "${network}" >/dev/null
docker volume create "${volume}" >/dev/null
start_server "${old_container}" "${old_image}" minio server /data --address :9000
docker exec "${old_container}" mcli mb local/compat >/dev/null
docker exec "${old_container}" mcli version enable local/compat >/dev/null
printf 'old-version-1\n' | docker exec -i "${old_container}" mcli pipe local/compat/versioned.txt >/dev/null
printf 'old-version-2\n' | docker exec -i "${old_container}" mcli pipe local/compat/versioned.txt >/dev/null
test "$(docker exec "${old_container}" mcli ls --versions local/compat/versioned.txt | grep -c 'versioned.txt')" -ge 2
docker exec "${old_container}" mcli mb --with-lock local/locked >/dev/null
printf 'locked-by-old\n' | docker exec -i "${old_container}" mcli pipe local/locked/object.txt >/dev/null
dd if=/dev/zero bs=1048576 count=70 2>/dev/null | docker exec -i "${old_container}" mcli pipe local/compat/multipart.bin >/dev/null
test "$(docker exec "${old_container}" mcli stat --json local/compat/multipart.bin | jq -r '.size')" = 73400320
docker exec "${old_container}" mcli admin user add local migration-user migration-secret-123 >/dev/null
docker exec "${old_container}" mcli admin policy attach local readwrite --user migration-user >/dev/null
stop_server "${old_container}"
start_server "${new_container}" "${new_image}" silo server /data --address :9000
test "$(docker exec "${new_container}" mcli cat local/compat/versioned.txt)" = old-version-2
test "$(docker exec "${new_container}" mcli cat local/locked/object.txt)" = locked-by-old
test "$(docker exec "${new_container}" mcli stat --json local/compat/multipart.bin | jq -r '.size')" = 73400320
docker exec "${new_container}" mcli admin user info local migration-user >/dev/null
docker exec "${new_container}" mcli alias set migrated http://127.0.0.1:9000 migration-user migration-secret-123 >/dev/null
printf 'written-by-silo\n' | docker exec -i "${new_container}" mcli pipe migrated/compat/silo.txt >/dev/null
stop_server "${new_container}"
start_server "${rollback_container}" "${old_image}" minio server /data --address :9000
test "$(docker exec "${rollback_container}" mcli cat local/compat/versioned.txt)" = old-version-2
test "$(docker exec "${rollback_container}" mcli cat local/compat/silo.txt)" = written-by-silo
test "$(docker exec "${rollback_container}" mcli stat --json local/compat/multipart.bin | jq -r '.size')" = 73400320
stop_server "${rollback_container}"
echo "MinIO-to-Silo data upgrade and rollback checks passed"
+19 -30
View File
@@ -5,16 +5,16 @@ if [ -n "$TEST_DEBUG" ]; then
fi fi
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
@@ -30,9 +30,9 @@ catch() {
echo "error on line $1" echo "error on line $1"
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio || true pkill silo || true
pkill -9 minio || true pkill -9 silo || true
purge "$WORK_DIR" purge "$WORK_DIR"
if [ $# -ne 0 ]; then if [ $# -ne 0 ]; then
exit $# exit $#
@@ -41,32 +41,21 @@ catch() {
catch catch
function start_minio_10drive() { function start_silo_10drive() {
start_port=$1 start_port=$1
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
export MC_HOST_minio="http://minio:minio123@127.0.0.1:${start_port}/" export MC_HOST_silo="http://silo:silo123@127.0.0.1:${start_port}/"
unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects
export MINIO_CI_CD=1 export MINIO_CI_CD=1
mkdir ${WORK_DIR} mkdir ${WORK_DIR}
C_PWD=${PWD}
if [ ! -x "$PWD/mc" ]; then if [ ! -x "$PWD/mc" ]; then
MC_BUILD_DIR="mc-$RANDOM" "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "$PWD/mc"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "$C_PWD/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
fi fi
"${MINIO[@]}" --address ":$start_port" "${WORK_DIR}/disk{1...10}" >"${WORK_DIR}/server1.log" 2>&1 & "${SILO[@]}" --address ":$start_port" "${WORK_DIR}/disk{1...10}" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$! pid=$!
disown $pid disown $pid
sleep 5 sleep 5
@@ -79,10 +68,10 @@ function start_minio_10drive() {
exit 1 exit 1
fi fi
"${PWD}/mc" mb --with-versioning minio/bucket "${PWD}/mc" mb --with-versioning silo/bucket
export AWS_ACCESS_KEY_ID=minio export AWS_ACCESS_KEY_ID=silo
export AWS_SECRET_ACCESS_KEY=minio123 export AWS_SECRET_ACCESS_KEY=silo123
aws --endpoint-url http://localhost:"$start_port" s3api create-multipart-upload --bucket bucket --key obj-1 >upload-id.json aws --endpoint-url http://localhost:"$start_port" s3api create-multipart-upload --bucket bucket --key obj-1 >upload-id.json
uploadId=$(jq -r '.UploadId' upload-id.json) uploadId=$(jq -r '.UploadId' upload-id.json)
@@ -120,7 +109,7 @@ EOF
function main() { function main() {
start_port=$(shuf -i 10000-65000 -n 1) start_port=$(shuf -i 10000-65000 -n 1)
start_minio_10drive ${start_port} start_silo_10drive ${start_port}
} }
main "$@" main "$@"
+14 -23
View File
@@ -6,38 +6,29 @@ set -x
set -e set -e
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
function start_minio_5drive() { function start_silo_5drive() {
start_port=$1 start_port=$1
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
export MC_HOST_minio="http://minio:minio123@127.0.0.1:${start_port}/" export MC_HOST_silo="http://silo:silo123@127.0.0.1:${start_port}/"
unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects
export MINIO_CI_CD=1 export MINIO_CI_CD=1
MC_BUILD_DIR="mc-$RANDOM" mkdir -p "${WORK_DIR}"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "${WORK_DIR}/mc"
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "$WORK_DIR/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
"${WORK_DIR}/mc" cp --quiet -r "buildscripts/cicd-corpus/" "${WORK_DIR}/cicd-corpus/" "${WORK_DIR}/mc" cp --quiet -r "buildscripts/cicd-corpus/" "${WORK_DIR}/cicd-corpus/"
"${MINIO[@]}" --address ":$start_port" "${WORK_DIR}/cicd-corpus/disk{1...5}" >"${WORK_DIR}/server1.log" 2>&1 & "${SILO[@]}" --address ":$start_port" "${WORK_DIR}/cicd-corpus/disk{1...5}" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$! pid=$!
disown $pid disown $pid
sleep 5 sleep 5
@@ -50,16 +41,16 @@ function start_minio_5drive() {
exit 1 exit 1
fi fi
"${WORK_DIR}/mc" stat minio/bucket/testobj "${WORK_DIR}/mc" stat silo/bucket/testobj
pkill minio pkill silo
sleep 3 sleep 3
} }
function main() { function main() {
start_port=$(shuf -i 10000-65000 -n 1) start_port=$(shuf -i 10000-65000 -n 1)
start_minio_5drive ${start_port} start_silo_5drive ${start_port}
} }
function purge() { function purge() {
+30 -36
View File
@@ -5,48 +5,42 @@ set -o pipefail
set -x set -x
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO_OLD=("$PWD/minio.RELEASE.2020-10-28T08-16-50Z" --config-dir "$MINIO_CONFIG_DIR" server) MINIO_OLD=("$PWD/minio.RELEASE.2020-10-28T08-16-50Z" --config-dir "$SILO_CONFIG_DIR" server)
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
function download_old_release() { function download_old_release() {
if [ ! -f minio.RELEASE.2020-10-28T08-16-50Z ]; then if [ ! -x minio.RELEASE.2020-10-28T08-16-50Z ]; then
curl --silent -O https://dl.minio.io/server/minio/release/linux-amd64/archive/minio.RELEASE.2020-10-28T08-16-50Z : "${SILO_LEGACY_FIXTURE_2020:?set SILO_LEGACY_FIXTURE_2020 to the audited legacy binary}"
chmod a+x minio.RELEASE.2020-10-28T08-16-50Z : "${SILO_LEGACY_SHA256_2020:?set SILO_LEGACY_SHA256_2020 to its audited SHA-256}"
"$(git rev-parse --show-toplevel)/buildscripts/install-verified-fixture.sh" \
"${SILO_LEGACY_FIXTURE_2020}" "${SILO_LEGACY_SHA256_2020}" \
"minio.RELEASE.2020-10-28T08-16-50Z"
fi fi
} }
function verify_rewrite() { function verify_rewrite() {
start_port=$1 start_port=$1
export MINIO_ACCESS_KEY=minio export MINIO_ACCESS_KEY=silo
export MINIO_SECRET_KEY=minio123 export MINIO_SECRET_KEY=silo123
export MC_HOST_minio="http://minio:minio123@127.0.0.1:${start_port}/" export MC_HOST_silo="http://silo:silo123@127.0.0.1:${start_port}/"
unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects
export MINIO_CI_CD=1 export MINIO_CI_CD=1
MC_BUILD_DIR="mc-$RANDOM" mkdir -p "${WORK_DIR}"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "${WORK_DIR}/mc"
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "$WORK_DIR/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
"${MINIO_OLD[@]}" --address ":$start_port" "${WORK_DIR}/xl{1...16}" >"${WORK_DIR}/server1.log" 2>&1 & "${MINIO_OLD[@]}" --address ":$start_port" "${WORK_DIR}/xl{1...16}" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$! pid=$!
disown $pid disown $pid
"${WORK_DIR}/mc" ready minio/ "${WORK_DIR}/mc" ready silo/
if ! ps -p ${pid} 1>&2 >/dev/null; then if ! ps -p ${pid} 1>&2 >/dev/null; then
echo "server1 log:" echo "server1 log:"
@@ -56,30 +50,30 @@ function verify_rewrite() {
exit 1 exit 1
fi fi
"${WORK_DIR}/mc" mb minio/healing-rewrite-bucket --quiet --with-lock "${WORK_DIR}/mc" mb silo/healing-rewrite-bucket --quiet --with-lock
"${WORK_DIR}/mc" cp \ "${WORK_DIR}/mc" cp \
buildscripts/verify-build.sh \ buildscripts/verify-build.sh \
minio/healing-rewrite-bucket/ \ silo/healing-rewrite-bucket/ \
--disable-multipart --quiet --disable-multipart --quiet
"${WORK_DIR}/mc" cp \ "${WORK_DIR}/mc" cp \
buildscripts/verify-build.sh \ buildscripts/verify-build.sh \
minio/healing-rewrite-bucket/ \ silo/healing-rewrite-bucket/ \
--disable-multipart --quiet --disable-multipart --quiet
"${WORK_DIR}/mc" cp \ "${WORK_DIR}/mc" cp \
buildscripts/verify-build.sh \ buildscripts/verify-build.sh \
minio/healing-rewrite-bucket/ \ silo/healing-rewrite-bucket/ \
--disable-multipart --quiet --disable-multipart --quiet
kill ${pid} kill ${pid}
sleep 3 sleep 3
"${MINIO[@]}" --address ":$start_port" "${WORK_DIR}/xl{1...16}" >"${WORK_DIR}/server1.log" 2>&1 & "${SILO[@]}" --address ":$start_port" "${WORK_DIR}/xl{1...16}" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$! pid=$!
disown $pid disown $pid
"${WORK_DIR}/mc" ready minio/ "${WORK_DIR}/mc" ready silo/
if ! ps -p ${pid} 1>&2 >/dev/null; then if ! ps -p ${pid} 1>&2 >/dev/null; then
echo "server1 log:" echo "server1 log:"
@@ -92,8 +86,8 @@ function verify_rewrite() {
if ! ./s3-check-md5 \ if ! ./s3-check-md5 \
-debug \ -debug \
-versions \ -versions \
-access-key minio \ -access-key silo \
-secret-key minio123 \ -secret-key silo123 \
-endpoint "http://127.0.0.1:${start_port}/" 2>&1 | grep INTACT; then -endpoint "http://127.0.0.1:${start_port}/" 2>&1 | grep INTACT; then
echo "server1 log:" echo "server1 log:"
cat "${WORK_DIR}/server1.log" cat "${WORK_DIR}/server1.log"
@@ -101,7 +95,7 @@ function verify_rewrite() {
mkdir -p inspects mkdir -p inspects
( (
cd inspects cd inspects
"${WORK_DIR}/mc" admin inspect minio/healing-rewrite-bucket/verify-build.sh/** "${WORK_DIR}/mc" admin inspect silo/healing-rewrite-bucket/verify-build.sh/**
) )
"${WORK_DIR}/mc" mb play/inspects "${WORK_DIR}/mc" mb play/inspects
@@ -111,14 +105,14 @@ function verify_rewrite() {
exit 1 exit 1
fi fi
go run ./buildscripts/heal-manual.go "127.0.0.1:${start_port}" "minio" "minio123" go run ./buildscripts/heal-manual.go "127.0.0.1:${start_port}" "silo" "silo123"
sleep 1 sleep 1
if ! ./s3-check-md5 \ if ! ./s3-check-md5 \
-debug \ -debug \
-versions \ -versions \
-access-key minio \ -access-key silo \
-secret-key minio123 \ -secret-key silo123 \
-endpoint http://127.0.0.1:${start_port}/ 2>&1 | grep INTACT; then -endpoint http://127.0.0.1:${start_port}/ 2>&1 | grep INTACT; then
echo "server1 log:" echo "server1 log:"
cat "${WORK_DIR}/server1.log" cat "${WORK_DIR}/server1.log"
@@ -126,7 +120,7 @@ function verify_rewrite() {
mkdir -p inspects mkdir -p inspects
( (
cd inspects cd inspects
"${WORK_DIR}/mc" admin inspect minio/healing-rewrite-bucket/verify-build.sh/** "${WORK_DIR}/mc" admin inspect silo/healing-rewrite-bucket/verify-build.sh/**
) )
"${WORK_DIR}/mc" mb play/inspects "${WORK_DIR}/mc" mb play/inspects
+21 -32
View File
@@ -5,16 +5,16 @@ if [ -n "$TEST_DEBUG" ]; then
fi fi
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
@@ -30,9 +30,9 @@ catch() {
echo "error on line $1" echo "error on line $1"
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio || true pkill silo || true
pkill -9 minio || true pkill -9 silo || true
purge "$WORK_DIR" purge "$WORK_DIR"
if [ $# -ne 0 ]; then if [ $# -ne 0 ]; then
exit $# exit $#
@@ -70,31 +70,20 @@ function send_put_object_request() {
return 0 return 0
} }
function test_minio_with_timeout() { function test_silo_with_timeout() {
start_port=$1 start_port=$1
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
export MC_HOST_minio="http://minio:minio123@127.0.0.1:${start_port}/" export MC_HOST_silo="http://silo:silo123@127.0.0.1:${start_port}/"
export MINIO_CI_CD=1 export MINIO_CI_CD=1
mkdir ${WORK_DIR} mkdir ${WORK_DIR}
C_PWD=${PWD}
if [ ! -x "$PWD/mc" ]; then if [ ! -x "$PWD/mc" ]; then
MC_BUILD_DIR="mc-$RANDOM" "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "$PWD/mc"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "$C_PWD/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
fi fi
"${MINIO[@]}" --address ":$start_port" --read-header-timeout ${srv_hdr_timeout}s --idle-timeout ${srv_idle_timeout}s "${WORK_DIR}/disk/" >"${WORK_DIR}/server1.log" 2>&1 & "${SILO[@]}" --address ":$start_port" --read-header-timeout ${srv_hdr_timeout}s --idle-timeout ${srv_idle_timeout}s "${WORK_DIR}/disk/" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$! pid=$!
disown $pid disown $pid
sleep 1 sleep 1
@@ -109,20 +98,20 @@ function test_minio_with_timeout() {
set -e set -e
"${PWD}/mc" mb minio/testbucket "${PWD}/mc" mb silo/testbucket
"${PWD}/mc" anonymous set public minio/testbucket "${PWD}/mc" anonymous set public silo/testbucket
# slow header writing # slow header writing
send_put_object_request 20 0 && exit -1 send_put_object_request 20 0 && exit -1
"${PWD}/mc" stat minio/testbucket/testobject && exit -1 "${PWD}/mc" stat silo/testbucket/testobject && exit -1
# quick header write and slow bodywrite # quick header write and slow bodywrite
send_put_object_request 0 40 && exit -1 send_put_object_request 0 40 && exit -1
"${PWD}/mc" stat minio/testbucket/testobject && exit -1 "${PWD}/mc" stat silo/testbucket/testobject && exit -1
# quick header and body write # quick header and body write
send_put_object_request 1 1 || exit -1 send_put_object_request 1 1 || exit -1
"${PWD}/mc" stat minio/testbucket/testobject || exit -1 "${PWD}/mc" stat silo/testbucket/testobject || exit -1
} }
function main() { function main() {
@@ -131,7 +120,7 @@ function main() {
export srv_idle_timeout=5 export srv_idle_timeout=5
export -f gen_put_request export -f gen_put_request
test_minio_with_timeout ${start_port} test_silo_with_timeout ${start_port}
} }
main "$@" main "$@"
+56 -66
View File
@@ -5,8 +5,8 @@ set -e
set -E set -E
set -o pipefail set -o pipefail
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
@@ -15,196 +15,196 @@ WORK_DIR="$PWD/.verify-$RANDOM"
export MINT_MODE=core export MINT_MODE=core
export MINT_DATA_DIR="$WORK_DIR/data" export MINT_DATA_DIR="$WORK_DIR/data"
export SERVER_ENDPOINT="127.0.0.1:9000" export SERVER_ENDPOINT="127.0.0.1:9000"
export MC_HOST_verify="http://minio:minio123@${SERVER_ENDPOINT}/" export MC_HOST_verify="http://silo:silo123@${SERVER_ENDPOINT}/"
export MC_HOST_verify_ipv6="http://minio:minio123@[::1]:9000/" export MC_HOST_verify_ipv6="http://silo:silo123@[::1]:9000/"
export ACCESS_KEY="minio" export ACCESS_KEY="silo"
export SECRET_KEY="minio123" export SECRET_KEY="silo123"
export ENABLE_HTTPS=0 export ENABLE_HTTPS=0
export GO111MODULE=on export GO111MODULE=on
export GOGC=25 export GOGC=25
export ENABLE_ADMIN=1 export ENABLE_ADMIN=1
export MINIO_CI_CD=1 export MINIO_CI_CD=1
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR") SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR")
FILE_1_MB="$MINT_DATA_DIR/datafile-1-MB" FILE_1_MB="$MINT_DATA_DIR/datafile-1-MB"
FILE_65_MB="$MINT_DATA_DIR/datafile-65-MB" FILE_65_MB="$MINT_DATA_DIR/datafile-65-MB"
FUNCTIONAL_TESTS="$WORK_DIR/functional-tests.sh" FUNCTIONAL_TESTS="$WORK_DIR/functional-tests.sh"
function start_minio_fs() { function start_silo_fs() {
export MINIO_ROOT_USER=$ACCESS_KEY export MINIO_ROOT_USER=$ACCESS_KEY
export MINIO_ROOT_PASSWORD=$SECRET_KEY export MINIO_ROOT_PASSWORD=$SECRET_KEY
"${MINIO[@]}" server "${WORK_DIR}/fs-disk" >"$WORK_DIR/fs-minio.log" 2>&1 & "${SILO[@]}" server "${WORK_DIR}/fs-disk" >"$WORK_DIR/fs-silo.log" 2>&1 &
"${WORK_DIR}/mc" ready verify "${WORK_DIR}/mc" ready verify
} }
function start_minio_erasure() { function start_silo_erasure() {
"${MINIO[@]}" server "${WORK_DIR}/erasure-disk1" "${WORK_DIR}/erasure-disk2" "${WORK_DIR}/erasure-disk3" "${WORK_DIR}/erasure-disk4" >"$WORK_DIR/erasure-minio.log" 2>&1 & "${SILO[@]}" server "${WORK_DIR}/erasure-disk1" "${WORK_DIR}/erasure-disk2" "${WORK_DIR}/erasure-disk3" "${WORK_DIR}/erasure-disk4" >"$WORK_DIR/erasure-silo.log" 2>&1 &
"${WORK_DIR}/mc" ready verify "${WORK_DIR}/mc" ready verify
} }
function start_minio_erasure_sets() { function start_silo_erasure_sets() {
export MINIO_ENDPOINTS="${WORK_DIR}/erasure-disk-sets{1...32}" export MINIO_ENDPOINTS="${WORK_DIR}/erasure-disk-sets{1...32}"
"${MINIO[@]}" server >"$WORK_DIR/erasure-minio-sets.log" 2>&1 & "${SILO[@]}" server >"$WORK_DIR/erasure-silo-sets.log" 2>&1 &
"${WORK_DIR}/mc" ready verify "${WORK_DIR}/mc" ready verify
} }
function start_minio_pool_erasure_sets() { function start_silo_pool_erasure_sets() {
export MINIO_ROOT_USER=$ACCESS_KEY export MINIO_ROOT_USER=$ACCESS_KEY
export MINIO_ROOT_PASSWORD=$SECRET_KEY export MINIO_ROOT_PASSWORD=$SECRET_KEY
export MINIO_ENDPOINTS="http://127.0.0.1:9000${WORK_DIR}/pool-disk-sets{1...4} http://127.0.0.1:9001${WORK_DIR}/pool-disk-sets{5...8}" export MINIO_ENDPOINTS="http://127.0.0.1:9000${WORK_DIR}/pool-disk-sets{1...4} http://127.0.0.1:9001${WORK_DIR}/pool-disk-sets{5...8}"
"${MINIO[@]}" server --address ":9000" >"$WORK_DIR/pool-minio-9000.log" 2>&1 & "${SILO[@]}" server --address ":9000" >"$WORK_DIR/pool-silo-9000.log" 2>&1 &
"${MINIO[@]}" server --address ":9001" >"$WORK_DIR/pool-minio-9001.log" 2>&1 & "${SILO[@]}" server --address ":9001" >"$WORK_DIR/pool-silo-9001.log" 2>&1 &
"${WORK_DIR}/mc" ready verify "${WORK_DIR}/mc" ready verify
} }
function start_minio_pool_erasure_sets_ipv6() { function start_silo_pool_erasure_sets_ipv6() {
export MINIO_ROOT_USER=$ACCESS_KEY export MINIO_ROOT_USER=$ACCESS_KEY
export MINIO_ROOT_PASSWORD=$SECRET_KEY export MINIO_ROOT_PASSWORD=$SECRET_KEY
export MINIO_ENDPOINTS="http://[::1]:9000${WORK_DIR}/pool-disk-sets-ipv6{1...4} http://[::1]:9001${WORK_DIR}/pool-disk-sets-ipv6{5...8}" export MINIO_ENDPOINTS="http://[::1]:9000${WORK_DIR}/pool-disk-sets-ipv6{1...4} http://[::1]:9001${WORK_DIR}/pool-disk-sets-ipv6{5...8}"
"${MINIO[@]}" server --address="[::1]:9000" >"$WORK_DIR/pool-minio-ipv6-9000.log" 2>&1 & "${SILO[@]}" server --address="[::1]:9000" >"$WORK_DIR/pool-silo-ipv6-9000.log" 2>&1 &
"${MINIO[@]}" server --address="[::1]:9001" >"$WORK_DIR/pool-minio-ipv6-9001.log" 2>&1 & "${SILO[@]}" server --address="[::1]:9001" >"$WORK_DIR/pool-silo-ipv6-9001.log" 2>&1 &
"${WORK_DIR}/mc" ready verify_ipv6 "${WORK_DIR}/mc" ready verify_ipv6
} }
function start_minio_dist_erasure() { function start_silo_dist_erasure() {
export MINIO_ROOT_USER=$ACCESS_KEY export MINIO_ROOT_USER=$ACCESS_KEY
export MINIO_ROOT_PASSWORD=$SECRET_KEY export MINIO_ROOT_PASSWORD=$SECRET_KEY
export MINIO_ENDPOINTS="http://127.0.0.1:9000${WORK_DIR}/dist-disk1 http://127.0.0.1:9001${WORK_DIR}/dist-disk2 http://127.0.0.1:9002${WORK_DIR}/dist-disk3 http://127.0.0.1:9003${WORK_DIR}/dist-disk4" export MINIO_ENDPOINTS="http://127.0.0.1:9000${WORK_DIR}/dist-disk1 http://127.0.0.1:9001${WORK_DIR}/dist-disk2 http://127.0.0.1:9002${WORK_DIR}/dist-disk3 http://127.0.0.1:9003${WORK_DIR}/dist-disk4"
for i in $(seq 0 3); do for i in $(seq 0 3); do
"${MINIO[@]}" server --address ":900${i}" >"$WORK_DIR/dist-minio-900${i}.log" 2>&1 & "${SILO[@]}" server --address ":900${i}" >"$WORK_DIR/dist-silo-900${i}.log" 2>&1 &
done done
"${WORK_DIR}/mc" ready verify "${WORK_DIR}/mc" ready verify
} }
function run_test_fs() { function run_test_fs() {
start_minio_fs start_silo_fs
(cd "$WORK_DIR" && "$FUNCTIONAL_TESTS") (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
rv=$? rv=$?
pkill minio pkill silo
sleep 3 sleep 3
if [ "$rv" -ne 0 ]; then if [ "$rv" -ne 0 ]; then
cat "$WORK_DIR/fs-minio.log" cat "$WORK_DIR/fs-silo.log"
fi fi
rm -f "$WORK_DIR/fs-minio.log" rm -f "$WORK_DIR/fs-silo.log"
return "$rv" return "$rv"
} }
function run_test_erasure_sets() { function run_test_erasure_sets() {
start_minio_erasure_sets start_silo_erasure_sets
(cd "$WORK_DIR" && "$FUNCTIONAL_TESTS") (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
rv=$? rv=$?
pkill minio pkill silo
sleep 3 sleep 3
if [ "$rv" -ne 0 ]; then if [ "$rv" -ne 0 ]; then
cat "$WORK_DIR/erasure-minio-sets.log" cat "$WORK_DIR/erasure-silo-sets.log"
fi fi
rm -f "$WORK_DIR/erasure-minio-sets.log" rm -f "$WORK_DIR/erasure-silo-sets.log"
return "$rv" return "$rv"
} }
function run_test_pool_erasure_sets() { function run_test_pool_erasure_sets() {
start_minio_pool_erasure_sets start_silo_pool_erasure_sets
(cd "$WORK_DIR" && "$FUNCTIONAL_TESTS") (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
rv=$? rv=$?
pkill minio pkill silo
sleep 3 sleep 3
if [ "$rv" -ne 0 ]; then if [ "$rv" -ne 0 ]; then
for i in $(seq 0 1); do for i in $(seq 0 1); do
echo "server$i log:" echo "server$i log:"
cat "$WORK_DIR/pool-minio-900$i.log" cat "$WORK_DIR/pool-silo-900$i.log"
done done
fi fi
for i in $(seq 0 1); do for i in $(seq 0 1); do
rm -f "$WORK_DIR/pool-minio-900$i.log" rm -f "$WORK_DIR/pool-silo-900$i.log"
done done
return "$rv" return "$rv"
} }
function run_test_pool_erasure_sets_ipv6() { function run_test_pool_erasure_sets_ipv6() {
start_minio_pool_erasure_sets_ipv6 start_silo_pool_erasure_sets_ipv6
export SERVER_ENDPOINT="[::1]:9000" export SERVER_ENDPOINT="[::1]:9000"
(cd "$WORK_DIR" && "$FUNCTIONAL_TESTS") (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
rv=$? rv=$?
pkill minio pkill silo
sleep 3 sleep 3
if [ "$rv" -ne 0 ]; then if [ "$rv" -ne 0 ]; then
for i in $(seq 0 1); do for i in $(seq 0 1); do
echo "server$i log:" echo "server$i log:"
cat "$WORK_DIR/pool-minio-ipv6-900$i.log" cat "$WORK_DIR/pool-silo-ipv6-900$i.log"
done done
fi fi
for i in $(seq 0 1); do for i in $(seq 0 1); do
rm -f "$WORK_DIR/pool-minio-ipv6-900$i.log" rm -f "$WORK_DIR/pool-silo-ipv6-900$i.log"
done done
return "$rv" return "$rv"
} }
function run_test_erasure() { function run_test_erasure() {
start_minio_erasure start_silo_erasure
(cd "$WORK_DIR" && "$FUNCTIONAL_TESTS") (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
rv=$? rv=$?
pkill minio pkill silo
sleep 3 sleep 3
if [ "$rv" -ne 0 ]; then if [ "$rv" -ne 0 ]; then
cat "$WORK_DIR/erasure-minio.log" cat "$WORK_DIR/erasure-silo.log"
fi fi
rm -f "$WORK_DIR/erasure-minio.log" rm -f "$WORK_DIR/erasure-silo.log"
return "$rv" return "$rv"
} }
function run_test_dist_erasure() { function run_test_dist_erasure() {
start_minio_dist_erasure start_silo_dist_erasure
(cd "$WORK_DIR" && "$FUNCTIONAL_TESTS") (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
rv=$? rv=$?
pkill minio pkill silo
sleep 3 sleep 3
if [ "$rv" -ne 0 ]; then if [ "$rv" -ne 0 ]; then
echo "server1 log:" echo "server1 log:"
cat "$WORK_DIR/dist-minio-9000.log" cat "$WORK_DIR/dist-silo-9000.log"
echo "server2 log:" echo "server2 log:"
cat "$WORK_DIR/dist-minio-9001.log" cat "$WORK_DIR/dist-silo-9001.log"
echo "server3 log:" echo "server3 log:"
cat "$WORK_DIR/dist-minio-9002.log" cat "$WORK_DIR/dist-silo-9002.log"
echo "server4 log:" echo "server4 log:"
cat "$WORK_DIR/dist-minio-9003.log" cat "$WORK_DIR/dist-silo-9003.log"
fi fi
rm -f "$WORK_DIR/dist-minio-9000.log" "$WORK_DIR/dist-minio-9001.log" "$WORK_DIR/dist-minio-9002.log" "$WORK_DIR/dist-minio-9003.log" rm -f "$WORK_DIR/dist-silo-9000.log" "$WORK_DIR/dist-silo-9001.log" "$WORK_DIR/dist-silo-9002.log" "$WORK_DIR/dist-silo-9003.log"
return "$rv" return "$rv"
} }
@@ -216,31 +216,21 @@ function purge() {
function __init__() { function __init__() {
echo "Initializing environment" echo "Initializing environment"
mkdir -p "$WORK_DIR" mkdir -p "$WORK_DIR"
mkdir -p "$MINIO_CONFIG_DIR" mkdir -p "$SILO_CONFIG_DIR"
mkdir -p "$MINT_DATA_DIR" mkdir -p "$MINT_DATA_DIR"
MC_BUILD_DIR="mc-$RANDOM" "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" "${WORK_DIR}/mc"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "${WORK_DIR}/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
shred -n 1 -s 1M - 1>"$FILE_1_MB" 2>/dev/null shred -n 1 -s 1M - 1>"$FILE_1_MB" 2>/dev/null
shred -n 1 -s 65M - 1>"$FILE_65_MB" 2>/dev/null shred -n 1 -s 65M - 1>"$FILE_65_MB" 2>/dev/null
## version is purposefully set to '3' for minio to migrate configuration file ## version is purposefully set to '3' for minio to migrate configuration file
echo '{"version": "3", "credential": {"accessKey": "minio", "secretKey": "minio123"}, "region": "us-east-1"}' >"$MINIO_CONFIG_DIR/config.json" echo '{"version": "3", "credential": {"accessKey": "silo", "secretKey": "silo123"}, "region": "us-east-1"}' >"$SILO_CONFIG_DIR/config.json"
if ! wget -q -O "$FUNCTIONAL_TESTS" https://raw.githubusercontent.com/minio/mc/master/functional-tests.sh; then "$(git rev-parse --show-toplevel)/buildscripts/install-verified-fixture.sh" \
echo "failed to download https://raw.githubusercontent.com/minio/mc/master/functional-tests.sh" https://raw.githubusercontent.com/pgsty/mc/4c4dcc4b55baf238cd0c81030d77945b3828f157/functional-tests.sh \
exit 1 9b98c8152b294d567b9bc732869226dd4f65d0f4d84092dc066a900a66a9e22c \
fi "$FUNCTIONAL_TESTS"
sed -i 's|-sS|-sSg|g' "$FUNCTIONAL_TESTS" sed -i 's|-sS|-sSg|g' "$FUNCTIONAL_TESTS"
chmod a+x "$FUNCTIONAL_TESTS" chmod a+x "$FUNCTIONAL_TESTS"
@@ -4,18 +4,18 @@
set -E set -E
set -o pipefail set -o pipefail
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
function start_minio_3_node() { function start_silo_3_node() {
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
export MINIO_ERASURE_SET_DRIVE_COUNT=6 export MINIO_ERASURE_SET_DRIVE_COUNT=6
export MINIO_CI_CD=1 export MINIO_CI_CD=1
@@ -25,26 +25,26 @@ function start_minio_3_node() {
args="$args http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/1/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/2/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/3/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/4/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/5/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/6/" args="$args http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/1/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/2/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/3/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/4/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/5/ http://127.0.0.1:$((start_port + i))${WORK_DIR}/$i/6/"
done done
"${MINIO[@]}" --address ":$((start_port + 1))" $args >"${WORK_DIR}/dist-minio-server1.log" 2>&1 & "${SILO[@]}" --address ":$((start_port + 1))" $args >"${WORK_DIR}/dist-silo-server1.log" 2>&1 &
pid1=$! pid1=$!
disown ${pid1} disown ${pid1}
"${MINIO[@]}" --address ":$((start_port + 2))" $args >"${WORK_DIR}/dist-minio-server2.log" 2>&1 & "${SILO[@]}" --address ":$((start_port + 2))" $args >"${WORK_DIR}/dist-silo-server2.log" 2>&1 &
pid2=$! pid2=$!
disown $pid2 disown $pid2
"${MINIO[@]}" --address ":$((start_port + 3))" $args >"${WORK_DIR}/dist-minio-server3.log" 2>&1 & "${SILO[@]}" --address ":$((start_port + 3))" $args >"${WORK_DIR}/dist-silo-server3.log" 2>&1 &
pid3=$! pid3=$!
disown $pid3 disown $pid3
export MC_HOST_myminio="http://minio:minio123@127.0.0.1:$((start_port + 1))" export MC_HOST_mysilo="http://silo:silo123@127.0.0.1:$((start_port + 1))"
timeout 15m /tmp/mc ready myminio || fail timeout 15m /tmp/mc ready mysilo || fail
# Wait for all drives to be online and formatted # Wait for all drives to be online and formatted
while [ $(/tmp/mc admin info --json myminio | jq '.info.servers[].drives[].state | select(. != "ok")' | wc -l) -gt 0 ]; do sleep 1; done while [ $(/tmp/mc admin info --json mysilo | jq '.info.servers[].drives[].state | select(. != "ok")' | wc -l) -gt 0 ]; do sleep 1; done
# Wait for all drives to be healed # Wait for all drives to be healed
while [ $(/tmp/mc admin info --json myminio | jq '.info.servers[].drives[].healing | select(. != null) | select(. == true)' | wc -l) -gt 0 ]; do sleep 1; done while [ $(/tmp/mc admin info --json mysilo | jq '.info.servers[].drives[].healing | select(. != null) | select(. == true)' | wc -l) -gt 0 ]; do sleep 1; done
# Wait for Status: in MinIO output # Wait for Status: in MinIO output
while true; do while true; do
@@ -66,26 +66,26 @@ function start_minio_3_node() {
done done
if ! ps -p $pid1 1>&2 >/dev/null; then if ! ps -p $pid1 1>&2 >/dev/null; then
echo "minio-server-1 is not running." && fail echo "silo-server-1 is not running." && fail
fi fi
if ! ps -p $pid2 1>&2 >/dev/null; then if ! ps -p $pid2 1>&2 >/dev/null; then
echo "minio-server-2 is not running." && fail echo "silo-server-2 is not running." && fail
fi fi
if ! ps -p $pid3 1>&2 >/dev/null; then if ! ps -p $pid3 1>&2 >/dev/null; then
echo "minio-server-3 is not running." && fail echo "silo-server-3 is not running." && fail
fi fi
if ! pkill minio; then if ! pkill silo; then
fail fail
fi fi
sleep 1 sleep 1
if pgrep minio; then if pgrep silo; then
# forcibly killing, to proceed further properly. # forcibly killing, to proceed further properly.
if ! pkill -9 minio; then if ! pkill -9 silo; then
echo "no minio process running anymore, proceed." echo "no Silo process running anymore, proceed."
fi fi
fi fi
} }
@@ -93,7 +93,7 @@ function start_minio_3_node() {
function fail() { function fail() {
for i in $(seq 1 3); do for i in $(seq 1 3); do
echo "server$i log:" echo "server$i log:"
cat "${WORK_DIR}/dist-minio-server$i.log" cat "${WORK_DIR}/dist-silo-server$i.log"
done done
echo "FAILED" echo "FAILED"
purge "$WORK_DIR" purge "$WORK_DIR"
@@ -101,7 +101,7 @@ function fail() {
} }
function check_online() { function check_online() {
if ! grep -q 'API:' ${WORK_DIR}/dist-minio-*.log; then if ! grep -q 'API:' ${WORK_DIR}/dist-silo-*.log; then
echo "1" echo "1"
fi fi
} }
@@ -113,19 +113,18 @@ function purge() {
function __init__() { function __init__() {
echo "Initializing environment" echo "Initializing environment"
mkdir -p "$WORK_DIR" mkdir -p "$WORK_DIR"
mkdir -p "$MINIO_CONFIG_DIR" mkdir -p "$SILO_CONFIG_DIR"
## version is purposefully set to '3' for minio to migrate configuration file ## version is purposefully set to '3' for minio to migrate configuration file
echo '{"version": "3", "credential": {"accessKey": "minio", "secretKey": "minio123"}, "region": "us-east-1"}' >"$MINIO_CONFIG_DIR/config.json" echo '{"version": "3", "credential": {"accessKey": "silo", "secretKey": "silo123"}, "region": "us-east-1"}' >"$SILO_CONFIG_DIR/config.json"
if [ ! -f /tmp/mc ]; then if [ ! -f /tmp/mc ]; then
wget --quiet -O /tmp/mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" /tmp/mc
chmod +x /tmp/mc
fi fi
} }
function perform_test() { function perform_test() {
start_minio_3_node $2 start_silo_3_node $2
echo "Testing Distributed Erasure setup healing of drives" echo "Testing Distributed Erasure setup healing of drives"
echo "Remove the contents of the disks belonging to '${1}' erasure set" echo "Remove the contents of the disks belonging to '${1}' erasure set"
@@ -133,7 +132,7 @@ function perform_test() {
rm -rf ${WORK_DIR}/${1}/*/ rm -rf ${WORK_DIR}/${1}/*/
set -x set -x
start_minio_3_node $2 start_silo_3_node $2
} }
function main() { function main() {
+10 -10
View File
@@ -4,20 +4,20 @@ set -E
set -o pipefail set -o pipefail
set -x set -x
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
WORK_DIR="$(mktemp -d)" WORK_DIR="$(mktemp -d)"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
function start_minio() { function start_silo() {
start_port=$1 start_port=$1
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects unset MINIO_KMS_AUTO_ENCRYPTION # do not auto-encrypt objects
unset MINIO_CI_CD unset MINIO_CI_CD
unset CI unset CI
@@ -28,7 +28,7 @@ function start_minio() {
done done
for i in $(seq 1 4); do for i in $(seq 1 4); do
"${MINIO[@]}" --address ":$((start_port + i))" ${args[@]} 2>&1 >"${WORK_DIR}/server$i.log" & "${SILO[@]}" --address ":$((start_port + i))" ${args[@]} 2>&1 >"${WORK_DIR}/server$i.log" &
done done
# Wait until all nodes return 403 # Wait until all nodes return 403
@@ -60,7 +60,7 @@ function prepare_block_devices() {
# Start a distributed MinIO setup, unmount one disk and check if it is formatted # Start a distributed MinIO setup, unmount one disk and check if it is formatted
function main() { function main() {
start_port=$(shuf -i 10000-65000 -n 1) start_port=$(shuf -i 10000-65000 -n 1)
start_minio ${start_port} start_silo ${start_port}
# Unmount the disk, after the unmount the device id # Unmount the disk, after the unmount the device id
# /tmp/xxx/mnt/disk4 will be the same as '/' and it # /tmp/xxx/mnt/disk4 will be the same as '/' and it
@@ -82,7 +82,7 @@ function main() {
} }
function cleanup() { function cleanup() {
pkill minio pkill silo
sudo umount ${WORK_DIR}/mnt/disk{1..3}/ sudo umount ${WORK_DIR}/mnt/disk{1..3}/
sudo rm /dev/minio-loopdisk* sudo rm /dev/minio-loopdisk*
rm -rf "$WORK_DIR" rm -rf "$WORK_DIR"
+30 -31
View File
@@ -4,23 +4,23 @@
set -E set -E
set -o pipefail set -o pipefail
if [ ! -x "$PWD/minio" ]; then if [ ! -x "$PWD/silo" ]; then
echo "minio executable binary not found in current directory" echo "Silo executable binary not found in current directory"
exit 1 exit 1
fi fi
WORK_DIR="$PWD/.verify-$RANDOM" WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio" SILO_CONFIG_DIR="$WORK_DIR/.silo"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server) SILO=("$PWD/silo" --config-dir "$SILO_CONFIG_DIR" server)
GOPATH=/tmp/gopath GOPATH=/tmp/gopath
function start_minio_3_node() { function start_silo_3_node() {
for i in $(seq 1 3); do for i in $(seq 1 3); do
rm "${WORK_DIR}/dist-minio-server$i.log" rm "${WORK_DIR}/dist-silo-server$i.log"
done done
export MINIO_ROOT_USER=minio export MINIO_ROOT_USER=silo
export MINIO_ROOT_PASSWORD=minio123 export MINIO_ROOT_PASSWORD=silo123
export MINIO_ERASURE_SET_DRIVE_COUNT=6 export MINIO_ERASURE_SET_DRIVE_COUNT=6
export MINIO_CI_CD=1 export MINIO_CI_CD=1
@@ -34,51 +34,51 @@ function start_minio_3_node() {
args="$args http://127.0.0.1:$((start_port + 1))${WORK_DIR}/1/${d}/ http://127.0.0.1:$((start_port + 2))${WORK_DIR}/2/${d}/ http://127.0.0.1:$((start_port + 3))${WORK_DIR}/3/${d}/ " args="$args http://127.0.0.1:$((start_port + 1))${WORK_DIR}/1/${d}/ http://127.0.0.1:$((start_port + 2))${WORK_DIR}/2/${d}/ http://127.0.0.1:$((start_port + 3))${WORK_DIR}/3/${d}/ "
done done
"${MINIO[@]}" --address ":$((start_port + 1))" $args >"${WORK_DIR}/dist-minio-server1.log" 2>&1 & "${SILO[@]}" --address ":$((start_port + 1))" $args >"${WORK_DIR}/dist-silo-server1.log" 2>&1 &
pid1=$! pid1=$!
disown ${pid1} disown ${pid1}
"${MINIO[@]}" --address ":$((start_port + 2))" $args >"${WORK_DIR}/dist-minio-server2.log" 2>&1 & "${SILO[@]}" --address ":$((start_port + 2))" $args >"${WORK_DIR}/dist-silo-server2.log" 2>&1 &
pid2=$! pid2=$!
disown $pid2 disown $pid2
"${MINIO[@]}" --address ":$((start_port + 3))" $args >"${WORK_DIR}/dist-minio-server3.log" 2>&1 & "${SILO[@]}" --address ":$((start_port + 3))" $args >"${WORK_DIR}/dist-silo-server3.log" 2>&1 &
pid3=$! pid3=$!
disown $pid3 disown $pid3
export MC_HOST_myminio="http://minio:minio123@127.0.0.1:$((start_port + 1))" export MC_HOST_mysilo="http://silo:silo123@127.0.0.1:$((start_port + 1))"
timeout 15m /tmp/mc ready myminio || fail timeout 15m /tmp/mc ready mysilo || fail
[ ${first_time} -eq 0 ] && upload_objects [ ${first_time} -eq 0 ] && upload_objects
[ ${first_time} -ne 0 ] && sleep 120 [ ${first_time} -ne 0 ] && sleep 120
if ! ps -p $pid1 1>&2 >/dev/null; then if ! ps -p $pid1 1>&2 >/dev/null; then
echo "minio server 1 is not running" && fail echo "silo server 1 is not running" && fail
fi fi
if ! ps -p $pid2 1>&2 >/dev/null; then if ! ps -p $pid2 1>&2 >/dev/null; then
echo "minio server 2 is not running" && fail echo "silo server 2 is not running" && fail
fi fi
if ! ps -p $pid3 1>&2 >/dev/null; then if ! ps -p $pid3 1>&2 >/dev/null; then
echo "minio server 3 is not running" && fail echo "silo server 3 is not running" && fail
fi fi
if ! pkill minio; then if ! pkill silo; then
fail fail
fi fi
sleep 1 sleep 1
if pgrep minio; then if pgrep silo; then
# forcibly killing, to proceed further properly. # forcibly killing, to proceed further properly.
if ! pkill -9 minio; then if ! pkill -9 silo; then
echo "no minio process running anymore, proceed." echo "no Silo process running anymore, proceed."
fi fi
fi fi
} }
function check_heal() { function check_heal() {
if ! grep -q 'API:' ${WORK_DIR}/dist-minio-*.log; then if ! grep -q 'API:' ${WORK_DIR}/dist-silo-*.log; then
return 1 return 1
fi fi
@@ -103,9 +103,9 @@ function purge() {
function fail() { function fail() {
for i in $(seq 1 3); do for i in $(seq 1 3); do
echo "server$i log:" echo "server$i log:"
cat "${WORK_DIR}/dist-minio-server$i.log" cat "${WORK_DIR}/dist-silo-server$i.log"
done done
pkill -9 minio pkill -9 silo
echo "FAILED" echo "FAILED"
purge "$WORK_DIR" purge "$WORK_DIR"
exit 1 exit 1
@@ -114,28 +114,27 @@ function fail() {
function __init__() { function __init__() {
echo "Initializing environment" echo "Initializing environment"
mkdir -p "$WORK_DIR" mkdir -p "$WORK_DIR"
mkdir -p "$MINIO_CONFIG_DIR" mkdir -p "$SILO_CONFIG_DIR"
## version is purposefully set to '3' for minio to migrate configuration file ## version is purposefully set to '3' for minio to migrate configuration file
echo '{"version": "3", "credential": {"accessKey": "minio", "secretKey": "minio123"}, "region": "us-east-1"}' >"$MINIO_CONFIG_DIR/config.json" echo '{"version": "3", "credential": {"accessKey": "silo", "secretKey": "silo123"}, "region": "us-east-1"}' >"$SILO_CONFIG_DIR/config.json"
if [ ! -f /tmp/mc ]; then if [ ! -f /tmp/mc ]; then
wget --quiet -O /tmp/mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" /tmp/mc
chmod +x /tmp/mc
fi fi
} }
function upload_objects() { function upload_objects() {
/tmp/mc mb myminio/testbucket/ /tmp/mc mb mysilo/testbucket/
for ((i = 0; i < 20; i++)); do for ((i = 0; i < 20; i++)); do
echo "my content" | /tmp/mc pipe myminio/testbucket/file-$i echo "my content" | /tmp/mc pipe mysilo/testbucket/file-$i
done done
} }
function perform_test() { function perform_test() {
start_port=$2 start_port=$2
start_minio_3_node $start_port start_silo_3_node $start_port
echo "Testing Distributed Erasure setup healing of drives" echo "Testing Distributed Erasure setup healing of drives"
echo "Remove the contents of the disks belonging to '${1}' node" echo "Remove the contents of the disks belonging to '${1}' node"
@@ -143,7 +142,7 @@ function perform_test() {
rm -rf ${WORK_DIR}/${1}/*/ rm -rf ${WORK_DIR}/${1}/*/
set -x set -x
start_minio_3_node $start_port start_silo_3_node $start_port
check_heal ${1} check_heal ${1}
rv=$? rv=$?
+9 -10
View File
@@ -16,9 +16,9 @@ catch() {
done done
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/multisitea rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb rm -rf /tmp/multisiteb
if [ $# -ne 0 ]; then if [ $# -ne 0 ]; then
@@ -39,21 +39,20 @@ unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
# Wait to make sure all MinIO instances are up # Wait to make sure all Silo instances are up
export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001 export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001
export MC_HOST_siteb=http://minioadmin:minioadmin@127.0.0.1:9004 export MC_HOST_siteb=http://minioadmin:minioadmin@127.0.0.1:9004
+35 -36
View File
@@ -18,9 +18,9 @@ catch() {
cat /tmp/dc2.log cat /tmp/dc2.log
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
set +e set +e
pkill minio pkill silo
pkill mc pkill mc
rm -rf /tmp/xl/ rm -rf /tmp/xl/
if [ $# -ne 0 ]; then if [ $# -ne 0 ]; then
@@ -34,7 +34,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -44,8 +44,7 @@ unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
mkdir -p /tmp/xl/1/ /tmp/xl/2/ mkdir -p /tmp/xl/1/ /tmp/xl/2/
@@ -54,40 +53,40 @@ export MINIO_KMS_SECRET_KEY="my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmG
export MINIO_ROOT_USER="minioadmin" export MINIO_ROOT_USER="minioadmin"
export MINIO_ROOT_PASSWORD="minioadmin" export MINIO_ROOT_PASSWORD="minioadmin"
./minio server --address ":9001" /tmp/xl/1/{1...4}/ 2>&1 >/tmp/dc1.log & ./silo server --address ":9001" /tmp/xl/1/{1...4}/ 2>&1 >/tmp/dc1.log &
pid1=$! pid1=$!
./minio server --address ":9002" /tmp/xl/2/{1...4}/ 2>&1 >/tmp/dc2.log & ./silo server --address ":9002" /tmp/xl/2/{1...4}/ 2>&1 >/tmp/dc2.log &
pid2=$! pid2=$!
sleep 3 sleep 3
export MC_HOST_myminio1=http://minioadmin:minioadmin@localhost:9001 export MC_HOST_mysilo1=http://minioadmin:minioadmin@localhost:9001
export MC_HOST_myminio2=http://minioadmin:minioadmin@localhost:9002 export MC_HOST_mysilo2=http://minioadmin:minioadmin@localhost:9002
./mc ready myminio1 ./mc ready mysilo1
./mc ready myminio2 ./mc ready mysilo2
./mc mb myminio1/testbucket/ ./mc mb mysilo1/testbucket/
./mc version enable myminio1/testbucket/ ./mc version enable mysilo1/testbucket/
./mc mb myminio2/testbucket/ ./mc mb mysilo2/testbucket/
./mc version enable myminio2/testbucket/ ./mc version enable mysilo2/testbucket/
./mc replicate add myminio1/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9002/testbucket/ --priority 1 ./mc replicate add mysilo1/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9002/testbucket/ --priority 1
# Test replication of delete markers and permanent deletes # Test replication of delete markers and permanent deletes
./mc cp README.md myminio1/testbucket/dir/file ./mc cp README.md mysilo1/testbucket/dir/file
./mc cp README.md myminio1/testbucket/dir/file ./mc cp README.md mysilo1/testbucket/dir/file
sleep 1s sleep 1s
echo "=== myminio1" echo "=== mysilo1"
./mc ls --versions myminio1/testbucket/dir/file ./mc ls --versions mysilo1/testbucket/dir/file
echo "=== myminio2" echo "=== mysilo2"
./mc ls --versions myminio2/testbucket/dir/file ./mc ls --versions mysilo2/testbucket/dir/file
versionId="$(./mc ls --json --versions myminio1/testbucket/dir/ | tail -n1 | jq -r .versionId)" versionId="$(./mc ls --json --versions mysilo1/testbucket/dir/ | tail -n1 | jq -r .versionId)"
export AWS_ACCESS_KEY_ID=minioadmin export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin export AWS_SECRET_ACCESS_KEY=minioadmin
@@ -95,23 +94,23 @@ export AWS_REGION=us-east-1
aws s3api --endpoint-url http://localhost:9001 delete-object --bucket testbucket --key dir/file --version-id "$versionId" aws s3api --endpoint-url http://localhost:9001 delete-object --bucket testbucket --key dir/file --version-id "$versionId"
./mc ls -r --versions myminio1/testbucket >/tmp/myminio1.txt ./mc ls -r --versions mysilo1/testbucket >/tmp/mysilo1.txt
./mc ls -r --versions myminio2/testbucket >/tmp/myminio2.txt ./mc ls -r --versions mysilo2/testbucket >/tmp/mysilo2.txt
out=$(diff -qpruN /tmp/myminio1.txt /tmp/myminio2.txt) out=$(diff -qpruN /tmp/mysilo1.txt /tmp/mysilo2.txt)
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication: $out" echo "BUG: expected no missing entries after replication: $out"
exit 1 exit 1
fi fi
./mc rm myminio1/testbucket/dir/file ./mc rm mysilo1/testbucket/dir/file
sleep 1s sleep 1s
./mc ls -r --versions myminio1/testbucket >/tmp/myminio1.txt ./mc ls -r --versions mysilo1/testbucket >/tmp/mysilo1.txt
./mc ls -r --versions myminio2/testbucket >/tmp/myminio2.txt ./mc ls -r --versions mysilo2/testbucket >/tmp/mysilo2.txt
out=$(diff -qpruN /tmp/myminio1.txt /tmp/myminio2.txt) out=$(diff -qpruN /tmp/mysilo1.txt /tmp/mysilo2.txt)
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication: $out" echo "BUG: expected no missing entries after replication: $out"
@@ -122,24 +121,24 @@ fi
set -x set -x
./mc mb myminio1/foobucket/ myminio2/foobucket/ --with-versioning ./mc mb mysilo1/foobucket/ mysilo2/foobucket/ --with-versioning
./mc replicate add myminio1/foobucket --remote-bucket http://minioadmin:minioadmin@localhost:9002/foobucket/ --priority 1 ./mc replicate add mysilo1/foobucket --remote-bucket http://minioadmin:minioadmin@localhost:9002/foobucket/ --priority 1
./mc cp README.md myminio1/foobucket/dir/file ./mc cp README.md mysilo1/foobucket/dir/file
versionId="$(./mc ls --json --versions myminio1/foobucket/dir/ | jq -r .versionId)" versionId="$(./mc ls --json --versions mysilo1/foobucket/dir/ | jq -r .versionId)"
kill ${pid2} && wait ${pid2} || true kill ${pid2} && wait ${pid2} || true
aws s3api --endpoint-url http://localhost:9001 delete-object --bucket foobucket --key dir/file --version-id "$versionId" aws s3api --endpoint-url http://localhost:9001 delete-object --bucket foobucket --key dir/file --version-id "$versionId"
out="$(./mc ls myminio1/foobucket/dir/)" out="$(./mc ls mysilo1/foobucket/dir/)"
if [ "$out" != "" ]; then if [ "$out" != "" ]; then
echo "BUG: non versioned listing should not show pending/failed replicated delete:" echo "BUG: non versioned listing should not show pending/failed replicated delete:"
echo "$out" echo "$out"
exit 1 exit 1
fi fi
out="$(./mc ls --versions myminio1/foobucket/dir/)" out="$(./mc ls --versions mysilo1/foobucket/dir/)"
if [ "$out" != "" ]; then if [ "$out" != "" ]; then
echo "BUG: versioned listing should not show pending/failed replicated deletes:" echo "BUG: versioned listing should not show pending/failed replicated deletes:"
echo "$out" echo "$out"
@@ -18,9 +18,9 @@ catch() {
done done
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/multisitea rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb rm -rf /tmp/multisiteb
rm -rf /tmp/data rm -rf /tmp/data
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -45,22 +45,21 @@ unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://minio:minio123@127.0.0.1:9001 export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:minio123@127.0.0.1:9004 export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
./mc ready sitea ./mc ready sitea
./mc ready siteb ./mc ready siteb
@@ -84,7 +83,7 @@ done
echo "adding replication rule for site a -> site b" echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket/ \ ./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket --remote-bucket http://minio:silo123@127.0.0.1:9004/bucket
remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket) remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket)
sleep 1 sleep 1
@@ -138,7 +137,7 @@ fi
echo "adding replication rule for site a -> site b" echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket-version/ \ ./mc replicate add sitea/bucket-version/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket-version --remote-bucket http://minio:silo123@127.0.0.1:9004/bucket-version
./mc mb sitea/bucket-version/directory/ ./mc mb sitea/bucket-version/directory/
@@ -20,9 +20,9 @@ catch() {
done done
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/multisitea rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb rm -rf /tmp/multisiteb
rm -rf /tmp/multisitec rm -rf /tmp/multisitec
@@ -37,7 +37,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -46,32 +46,33 @@ unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME unset MINIO_KMS_KES_KEY_NAME
go install -v github.com/minio/mc@master "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
cp -a $(go env GOPATH)/bin/mc ./mc
if [ ! -f mc.RELEASE.2021-03-12T03-36-59Z ]; then if [ ! -x mc.RELEASE.2021-03-12T03-36-59Z ]; then
wget -q -O mc.RELEASE.2021-03-12T03-36-59Z https://dl.minio.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2021-03-12T03-36-59Z && "$(git rev-parse --show-toplevel)/buildscripts/install-verified-fixture.sh" \
chmod +x mc.RELEASE.2021-03-12T03-36-59Z https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2021-03-12T03-36-59Z \
8296d4aa02a4cd9cb812bae4d0f65a2bb4003e1475192dcfc1fd42a726250ffa \
mc.RELEASE.2021-03-12T03-36-59Z
fi fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
minio server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_1.log 2>&1 & "http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_1.log 2>&1 &
minio server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 & "http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
export MC_HOST_sitea=http://minio:minio123@127.0.0.1:9001 export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:minio123@127.0.0.1:9004 export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:minio123@127.0.0.1:9006 export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
./mc ready sitea ./mc ready sitea
./mc ready siteb ./mc ready siteb
@@ -92,73 +93,73 @@ export MC_HOST_sitec=http://minio:minio123@127.0.0.1:9006
echo "adding replication rule for a -> b : ${remote_arn}" echo "adding replication rule for a -> b : ${remote_arn}"
sleep 1 sleep 1
./mc replicate add sitea/bucket/ \ ./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket \ --remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for b -> a : ${remote_arn}" echo "adding replication rule for b -> a : ${remote_arn}"
./mc replicate add siteb/bucket/ \ ./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/bucket \ --remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for a -> c : ${remote_arn}" echo "adding replication rule for a -> c : ${remote_arn}"
./mc replicate add sitea/bucket/ \ ./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/bucket \ --remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for c -> a : ${remote_arn}" echo "adding replication rule for c -> a : ${remote_arn}"
./mc replicate add sitec/bucket/ \ ./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/bucket \ --remote-bucket http://minio:silo123@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for b -> c : ${remote_arn}" echo "adding replication rule for b -> c : ${remote_arn}"
./mc replicate add siteb/bucket/ \ ./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/bucket \ --remote-bucket http://minio:silo123@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
echo "adding replication rule for c -> b : ${remote_arn}" echo "adding replication rule for c -> b : ${remote_arn}"
./mc replicate add sitec/bucket/ \ ./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket \ --remote-bucket http://minio:silo123@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
echo "adding replication rule for olockbucket a -> b : ${remote_arn}" echo "adding replication rule for olockbucket a -> b : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \ ./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/olockbucket \ --remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for olockbucket b -> a : ${remote_arn}" echo "adding replication rule for olockbucket b -> a : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \ ./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/olockbucket \ --remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1 sleep 1
echo "adding replication rule for olockbucket a -> c : ${remote_arn}" echo "adding replication rule for olockbucket a -> c : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \ ./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/olockbucket \ --remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for olockbucket c -> a : ${remote_arn}" echo "adding replication rule for olockbucket c -> a : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \ ./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/olockbucket \ --remote-bucket http://minio:silo123@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1 sleep 1
echo "adding replication rule for olockbucket b -> c : ${remote_arn}" echo "adding replication rule for olockbucket b -> c : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \ ./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/olockbucket \ --remote-bucket http://minio:silo123@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
echo "adding replication rule for olockbucket c -> b : ${remote_arn}" echo "adding replication rule for olockbucket c -> b : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \ ./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/olockbucket \ --remote-bucket http://minio:silo123@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3 --replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1 sleep 1
@@ -203,33 +204,33 @@ head -c 221227088 </dev/urandom >200M
sleep 10 sleep 10
echo "Verifying ETag for all objects" echo "Verifying ETag for all objects"
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9001/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9002/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9003/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9004/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9005/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9006/ -bucket bucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9001/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9002/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9003/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9004/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9005/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9006/ -bucket olockbucket ./s3-check-md5 -versions -access-key minio -secret-key silo123 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
# additional tests for encryption object alignment # additional tests for encryption object alignment
go install -v github.com/minio/multipart-debug@latest go install -v github.com/minio/multipart-debug@latest
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey minio123 multipart new --bucket bucket --object new-test-encrypted-object --encrypt) upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt
dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt
sudo apt install jq -y sudo apt install jq -y
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey minio123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag) etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey minio123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag) etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey silo123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey minio123 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2} multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey silo123 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
sleep 10 sleep 10
@@ -16,9 +16,9 @@ catch() {
done done
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/multisitea rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb rm -rf /tmp/multisiteb
rm -rf /tmp/multisitec rm -rf /tmp/multisitec
@@ -35,7 +35,7 @@ set -e
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -45,36 +45,35 @@ unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
minio server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_1.log 2>&1 & "http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_1.log 2>&1 &
minio server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 & "http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
minio server --address 127.0.0.1:9007 "http://127.0.0.1:9007/tmp/multisited/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9007 "http://127.0.0.1:9007/tmp/multisited/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9008/tmp/multisited/data/disterasure/xl{5...8}" >/tmp/sited_1.log 2>&1 & "http://127.0.0.1:9008/tmp/multisited/data/disterasure/xl{5...8}" >/tmp/sited_1.log 2>&1 &
minio server --address 127.0.0.1:9008 "http://127.0.0.1:9007/tmp/multisited/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9008 "http://127.0.0.1:9007/tmp/multisited/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9008/tmp/multisited/data/disterasure/xl{5...8}" >/tmp/sited_2.log 2>&1 & "http://127.0.0.1:9008/tmp/multisited/data/disterasure/xl{5...8}" >/tmp/sited_2.log 2>&1 &
# Wait to make sure all MinIO instances are up # Wait to make sure all Silo instances are up
export MC_HOST_sitea=http://minio:minio123@127.0.0.1:9001 export MC_HOST_sitea=http://minio:silo123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:minio123@127.0.0.1:9004 export MC_HOST_siteb=http://minio:silo123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:minio123@127.0.0.1:9006 export MC_HOST_sitec=http://minio:silo123@127.0.0.1:9006
export MC_HOST_sited=http://minio:minio123@127.0.0.1:9008 export MC_HOST_sited=http://minio:silo123@127.0.0.1:9008
./mc ready sitea ./mc ready sitea
./mc ready siteb ./mc ready siteb
@@ -90,7 +89,7 @@ export MC_HOST_sited=http://minio:minio123@127.0.0.1:9008
sleep 10s sleep 10s
## Add warm tier ## Add warm tier
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key minio123 --bucket bucket ./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key silo123 --bucket bucket
## Add ILM rules ## Add ILM rules
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2" ./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2"
+17 -17
View File
@@ -9,7 +9,7 @@ export CI=1
make || exit 255 make || exit 255
killall -9 minio || true killall -9 silo || true
rm -rf /tmp/xl/ rm -rf /tmp/xl/
mkdir -p /tmp/xl/1/ /tmp/xl/2/ mkdir -p /tmp/xl/1/ /tmp/xl/2/
@@ -26,39 +26,39 @@ for i in $(seq 1 $NODES); do
done done
for i in $(seq 1 $NODES); do for i in $(seq 1 $NODES); do
./minio server --address "127.0.0.1:$((9000 + i))" ${args1[@]} & # | tee /tmp/minio/node.$i & ./silo server --address "127.0.0.1:$((9000 + i))" ${args1[@]} & # | tee /tmp/minio/node.$i &
./minio server --address "127.0.0.1:$((9100 + i))" ${args2[@]} & # | tee /tmp/minio/node.$i & ./silo server --address "127.0.0.1:$((9100 + i))" ${args2[@]} & # | tee /tmp/minio/node.$i &
done done
sleep 10 sleep 10
./mc alias set myminio1 http://localhost:9001 minioadmin minioadmin ./mc alias set mysilo1 http://localhost:9001 minioadmin minioadmin
./mc alias set myminio2 http://localhost:9101 minioadmin minioadmin ./mc alias set mysilo2 http://localhost:9101 minioadmin minioadmin
./mc ready myminio1 ./mc ready mysilo1
./mc ready myminio2 ./mc ready mysilo2
sleep 1 sleep 1
./mc mb myminio1/testbucket/ --with-lock ./mc mb mysilo1/testbucket/ --with-lock
./mc mb myminio2/testbucket/ --with-lock ./mc mb mysilo2/testbucket/ --with-lock
./mc encrypt set sse-s3 my-minio-key myminio1/testbucket/ ./mc encrypt set sse-s3 my-minio-key mysilo1/testbucket/
./mc encrypt set sse-s3 my-minio-key myminio2/testbucket/ ./mc encrypt set sse-s3 my-minio-key mysilo2/testbucket/
./mc replicate add myminio1/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9101/testbucket --priority 1 ./mc replicate add mysilo1/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9101/testbucket --priority 1
./mc replicate add myminio2/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9001/testbucket --priority 1 ./mc replicate add mysilo2/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9001/testbucket --priority 1
sleep 1 sleep 1
cp README.md internal.tar cp README.md internal.tar
./mc cp internal.tar myminio1/testbucket/dir/1.tar ./mc cp internal.tar mysilo1/testbucket/dir/1.tar
./mc cp internal.tar myminio2/testbucket/dir/2.tar ./mc cp internal.tar mysilo2/testbucket/dir/2.tar
sleep 1 sleep 1
./mc ls -r --versions myminio1/testbucket/dir/ >/tmp/dir_1.txt ./mc ls -r --versions mysilo1/testbucket/dir/ >/tmp/dir_1.txt
./mc ls -r --versions myminio2/testbucket/dir/ >/tmp/dir_2.txt ./mc ls -r --versions mysilo2/testbucket/dir/ >/tmp/dir_2.txt
out=$(diff -qpruN /tmp/dir_1.txt /tmp/dir_2.txt) out=$(diff -qpruN /tmp/dir_1.txt /tmp/dir_2.txt)
ret=$? ret=$?
@@ -15,8 +15,8 @@ exit_1() {
} }
cleanup() { cleanup() {
echo -n "Cleaning up instances of MinIO ..." echo -n "Cleaning up instances of Silo ..."
pkill -9 minio || sudo pkill -9 minio pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/sitea rm -rf /tmp/sitea
rm -rf /tmp/siteb rm -rf /tmp/siteb
echo "done" echo "done"
@@ -29,23 +29,22 @@ export MINIO_BROWSER=off
make install-race make install-race
# Start MinIO instances # Start Silo instances
echo -n "Starting MinIO instances ..." echo -n "Starting Silo instances ..."
minio server --address 127.0.0.1:9001 --console-address ":10000" "http://127.0.0.1:9001/tmp/sitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9001 --console-address ":10000" "http://127.0.0.1:9001/tmp/sitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/sitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://127.0.0.1:9002/tmp/sitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/sitea/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/sitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/sitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://127.0.0.1:9002/tmp/sitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 --console-address ":10001" "http://127.0.0.1:9003/tmp/siteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9003 --console-address ":10001" "http://127.0.0.1:9003/tmp/siteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/siteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 & "http://127.0.0.1:9004/tmp/siteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/siteb/data/disterasure/xl{1...4}" \ silo server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/siteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/siteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 & "http://127.0.0.1:9004/tmp/siteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001 export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001
+6 -7
View File
@@ -16,9 +16,9 @@ catch() {
cat "/tmp/sitea_2.log" cat "/tmp/sitea_2.log"
fi fi
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/multisitea rm -rf /tmp/multisitea
if [ $# -ne 0 ]; then if [ $# -ne 0 ]; then
exit $# exit $#
@@ -39,14 +39,13 @@ unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
minio server -S /tmp/no-certs --address ":9001" "http://localhost:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server -S /tmp/no-certs --address ":9001" "http://localhost:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://localhost:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 & "http://localhost:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server -S /tmp/no-certs --address ":9002" "http://localhost:9001/tmp/multisitea/data/disterasure/xl{1...4}" \ silo server -S /tmp/no-certs --address ":9002" "http://localhost:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://localhost:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 & "http://localhost:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
export MC_HOST_sitea=http://minioadmin:minioadmin@localhost:9002 export MC_HOST_sitea=http://minioadmin:minioadmin@localhost:9002
+37 -38
View File
@@ -4,12 +4,11 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
rm -rf /tmp/xl rm -rf /tmp/xl
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
export CI=true export CI=true
@@ -19,50 +18,50 @@ export MINIO_COMPRESSION_MIME_TYPES="application/*"
export MINIO_COMPRESSION_ALLOW_ENCRYPTION="on" export MINIO_COMPRESSION_ALLOW_ENCRYPTION="on"
export MINIO_KMS_AUTO_ENCRYPTION=on export MINIO_KMS_AUTO_ENCRYPTION=on
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$! pid=$!
./mc ready myminio ./mc ready mysilo
./mc admin user add myminio/ minio123 minio123 ./mc admin user add mysilo/ silo123 silo123
./mc admin user add myminio/ minio12345 minio12345 ./mc admin user add mysilo/ silo12345 silo12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json ./mc admin policy create mysilo/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json ./mc admin policy create mysilo/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123 ./mc admin policy attach mysilo/ rw --user=silo123
./mc admin policy attach myminio/ lake --user=minio12345 ./mc admin policy attach mysilo/ lake --user=silo12345
./mc mb -l myminio/versioned ./mc mb -l mysilo/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers) ## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null ./mc rm -r --force mysilo/versioned >/dev/null
## mirror again to create another set of version on top ## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum) expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l) user_count=$(./mc admin user list mysilo/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l) policy_count=$(./mc admin policy list mysilo/ | wc -l)
kill $pid kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$! pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) & (silo server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$! pid_2=$!
sleep 30 sleep 30
./mc ready myminio ./mc ready mysilo
expanded_user_count=$(./mc admin user list myminio/ | wc -l) expanded_user_count=$(./mc admin user list mysilo/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l) expanded_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup" echo "BUG: original user count differs from expanded setup"
@@ -74,21 +73,21 @@ if [ $policy_count -ne $expanded_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion" echo "expected versioning enabled after expansion"
exit 1 exit 1
fi fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null ./mc mirror cmd mysilo/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt ./mc ls -r mysilo/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt ./mc ls -r --versions mysilo/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1} ./mc admin decom start mysilo/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do until $(./mc admin decom status mysilo/ | grep -q Complete); do
echo "waiting for decom to finish..." echo "waiting for decom to finish..."
sleep 1 sleep 1
done done
@@ -98,15 +97,15 @@ kill $pid_2
sleep 5 sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) & (silo server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$! pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio ./mc ready mysilo
decom_user_count=$(./mc admin user list myminio/ | wc -l) decom_user_count=$(./mc admin user list mysilo/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l) decom_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission" echo "BUG: original user count differs after decommission"
@@ -118,21 +117,21 @@ if [ $policy_count -ne $decom_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission" echo "BUG: expected versioning enabled after decommission"
exit 1 exit 1
fi fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum) got_checksum=$(./mc cat mysilo/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}" echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1 exit 1
fi fi
./mc ls -r myminio/versioned >decommissioned_ns.txt ./mc ls -r mysilo/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt ./mc ls -r --versions mysilo/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt) out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$? ret=$?
+55 -56
View File
@@ -4,13 +4,12 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
pkill kes pkill kes
rm -rf /tmp/xl rm -rf /tmp/xl
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
if [ ! -f ./kes ]; then if [ ! -f ./kes ]; then
@@ -34,56 +33,56 @@ export MINIO_KMS_KES_ENDPOINT=https://127.0.0.1:7373
export MINIO_KMS_KES_API_KEY="${API_KEY}" export MINIO_KMS_KES_API_KEY="${API_KEY}"
export MINIO_KMS_KES_KEY_NAME=minio-default-key export MINIO_KMS_KES_KEY_NAME=minio-default-key
export MINIO_KMS_KES_CAPATH=public.crt export MINIO_KMS_KES_CAPATH=public.crt
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$! pid=$!
./mc ready myminio ./mc ready mysilo
./mc admin user add myminio/ minio123 minio123 ./mc admin user add mysilo/ silo123 silo123
./mc admin user add myminio/ minio12345 minio12345 ./mc admin user add mysilo/ silo12345 silo12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json ./mc admin policy create mysilo/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json ./mc admin policy create mysilo/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123 ./mc admin policy attach mysilo/ rw --user=silo123
./mc admin policy attach myminio/ lake --user=minio12345 ./mc admin policy attach mysilo/ lake --user=silo12345
./mc mb -l myminio/versioned ./mc mb -l mysilo/versioned
./mc mb -l myminio/versioned-1 ./mc mb -l mysilo/versioned-1
./mc encrypt set sse-s3 myminio/versioned ./mc encrypt set sse-s3 mysilo/versioned
./mc encrypt set sse-kms minio-default-key myminio/versioned-1 ./mc encrypt set sse-kms minio-default-key mysilo/versioned-1
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
./mc mirror internal myminio/versioned-1/ --quiet >/dev/null ./mc mirror internal mysilo/versioned-1/ --quiet >/dev/null
## Soft delete (creates delete markers) ## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null ./mc rm -r --force mysilo/versioned >/dev/null
./mc rm -r --force myminio/versioned-1 >/dev/null ./mc rm -r --force mysilo/versioned-1 >/dev/null
## mirror again to create another set of version on top ## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
./mc mirror internal myminio/versioned-1/ --quiet >/dev/null ./mc mirror internal mysilo/versioned-1/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum) expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l) user_count=$(./mc admin user list mysilo/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l) policy_count=$(./mc admin policy list mysilo/ | wc -l)
kill $pid kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$! pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) & (silo server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$! pid_2=$!
./mc ready myminio ./mc ready mysilo
expanded_user_count=$(./mc admin user list myminio/ | wc -l) expanded_user_count=$(./mc admin user list mysilo/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l) expanded_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ "$user_count" -ne "$expanded_user_count" ]; then if [ "$user_count" -ne "$expanded_user_count" ]; then
echo "BUG: original user count differs from expanded setup" echo "BUG: original user count differs from expanded setup"
@@ -95,45 +94,45 @@ if [ "$policy_count" -ne "$expanded_policy_count" ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion" echo "expected versioning enabled after expansion"
exit 1 exit 1
fi fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled" ./mc encrypt info mysilo/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected encryption enabled after expansion" echo "expected encryption enabled after expansion"
exit 1 exit 1
fi fi
./mc version info myminio/versioned-1 | grep -q "versioning is enabled" ./mc version info mysilo/versioned-1 | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion" echo "expected versioning enabled after expansion"
exit 1 exit 1
fi fi
./mc encrypt info myminio/versioned-1 | grep -q "Auto encryption 'sse-kms' is enabled" ./mc encrypt info mysilo/versioned-1 | grep -q "Auto encryption 'sse-kms' is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected encryption enabled after expansion" echo "expected encryption enabled after expansion"
exit 1 exit 1
fi fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null ./mc mirror cmd mysilo/versioned/ --quiet >/dev/null
./mc mirror cmd myminio/versioned-1/ --quiet >/dev/null ./mc mirror cmd mysilo/versioned-1/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt ./mc ls -r mysilo/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt ./mc ls -r --versions mysilo/versioned/ >expanded_ns_versions.txt
./mc ls -r myminio/versioned-1/ >expanded_ns_1.txt ./mc ls -r mysilo/versioned-1/ >expanded_ns_1.txt
./mc ls -r --versions myminio/versioned-1/ >expanded_ns_versions_1.txt ./mc ls -r --versions mysilo/versioned-1/ >expanded_ns_versions_1.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1} ./mc admin decom start mysilo/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do until $(./mc admin decom status mysilo/ | grep -q Complete); do
echo "waiting for decom to finish..." echo "waiting for decom to finish..."
sleep 1s sleep 1s
done done
@@ -143,17 +142,17 @@ kill $pid_2
sleep 5s sleep 5s
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) & (silo server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$! pid=$!
sleep 30s sleep 30s
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio ./mc ready mysilo
decom_user_count=$(./mc admin user list myminio/ | wc -l) decom_user_count=$(./mc admin user list mysilo/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l) decom_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ "$user_count" -ne "$decom_user_count" ]; then if [ "$user_count" -ne "$decom_user_count" ]; then
echo "BUG: original user count differs after decommission" echo "BUG: original user count differs after decommission"
@@ -165,50 +164,50 @@ if [ "$policy_count" -ne "$decom_policy_count" ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission" echo "BUG: expected versioning enabled after decommission"
exit 1 exit 1
fi fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled" ./mc encrypt info mysilo/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected encryption enabled after expansion" echo "BUG: expected encryption enabled after expansion"
exit 1 exit 1
fi fi
./mc version info myminio/versioned-1 | grep -q "versioning is enabled" ./mc version info mysilo/versioned-1 | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission" echo "BUG: expected versioning enabled after decommission"
exit 1 exit 1
fi fi
./mc encrypt info myminio/versioned-1 | grep -q "Auto encryption 'sse-kms' is enabled" ./mc encrypt info mysilo/versioned-1 | grep -q "Auto encryption 'sse-kms' is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected encryption enabled after expansion" echo "BUG: expected encryption enabled after expansion"
exit 1 exit 1
fi fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum) got_checksum=$(./mc cat mysilo/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}" echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1 exit 1
fi fi
got_checksum_1=$(./mc cat myminio/versioned-1/dsync/drwmutex.go | md5sum) got_checksum_1=$(./mc cat mysilo/versioned-1/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum_1}" ]; then if [ "${expected_checksum}" != "${got_checksum_1}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum_1}" echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum_1}"
exit 1 exit 1
fi fi
./mc ls -r myminio/versioned >decommissioned_ns.txt ./mc ls -r mysilo/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt ./mc ls -r --versions mysilo/versioned >decommissioned_ns_versions.txt
./mc ls -r myminio/versioned-1 >decommissioned_ns_1.txt ./mc ls -r mysilo/versioned-1 >decommissioned_ns_1.txt
./mc ls -r --versions myminio/versioned-1 >decommissioned_ns_versions_1.txt ./mc ls -r --versions mysilo/versioned-1 >decommissioned_ns_versions_1.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt) out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$? ret=$?
+40 -41
View File
@@ -4,61 +4,60 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
rm -rf /tmp/xl rm -rf /tmp/xl
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
export CI=true export CI=true
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$! pid=$!
./mc ready myminio ./mc ready mysilo
./mc admin user add myminio/ minio123 minio123 ./mc admin user add mysilo/ silo123 silo123
./mc admin user add myminio/ minio12345 minio12345 ./mc admin user add mysilo/ silo12345 silo12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json ./mc admin policy create mysilo/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json ./mc admin policy create mysilo/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123 ./mc admin policy attach mysilo/ rw --user=silo123
./mc admin policy attach myminio/ lake --user=minio12345 ./mc admin policy attach mysilo/ lake --user=silo12345
./mc mb -l myminio/versioned ./mc mb -l mysilo/versioned
./mc encrypt set sse-s3 myminio/versioned ./mc encrypt set sse-s3 mysilo/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers) ## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null ./mc rm -r --force mysilo/versioned >/dev/null
## mirror again to create another set of version on top ## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum) expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l) user_count=$(./mc admin user list mysilo/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l) policy_count=$(./mc admin policy list mysilo/ | wc -l)
kill $pid kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$! pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) & (silo server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$! pid_2=$!
./mc ready myminio ./mc ready mysilo
expanded_user_count=$(./mc admin user list myminio/ | wc -l) expanded_user_count=$(./mc admin user list mysilo/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l) expanded_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup" echo "BUG: original user count differs from expanded setup"
@@ -70,28 +69,28 @@ if [ $policy_count -ne $expanded_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion" echo "expected versioning enabled after expansion"
exit 1 exit 1
fi fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled" ./mc encrypt info mysilo/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected encryption enabled after expansion" echo "expected encryption enabled after expansion"
exit 1 exit 1
fi fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null ./mc mirror cmd mysilo/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt ./mc ls -r mysilo/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt ./mc ls -r --versions mysilo/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1} ./mc admin decom start mysilo/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do until $(./mc admin decom status mysilo/ | grep -q Complete); do
echo "waiting for decom to finish..." echo "waiting for decom to finish..."
sleep 1 sleep 1
done done
@@ -101,15 +100,15 @@ kill $pid_2
sleep 5 sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) & (silo server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$! pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio ./mc ready mysilo
decom_user_count=$(./mc admin user list myminio/ | wc -l) decom_user_count=$(./mc admin user list mysilo/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l) decom_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission" echo "BUG: original user count differs after decommission"
@@ -121,28 +120,28 @@ if [ $policy_count -ne $decom_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission" echo "BUG: expected versioning enabled after decommission"
exit 1 exit 1
fi fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled" ./mc encrypt info mysilo/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected encryption enabled after expansion" echo "BUG: expected encryption enabled after expansion"
exit 1 exit 1
fi fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum) got_checksum=$(./mc cat mysilo/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}" echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1 exit 1
fi fi
./mc ls -r myminio/versioned >decommissioned_ns.txt ./mc ls -r mysilo/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt ./mc ls -r --versions mysilo/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt) out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$? ret=$?
+37 -38
View File
@@ -4,61 +4,60 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
rm -rf /tmp/xl rm -rf /tmp/xl
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
export CI=true export CI=true
export MINIO_KMS_AUTO_ENCRYPTION=on export MINIO_KMS_AUTO_ENCRYPTION=on
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$! pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9000/"
./mc ready myminio ./mc ready mysilo
./mc admin user add myminio/ minio123 minio123 ./mc admin user add mysilo/ silo123 silo123
./mc admin user add myminio/ minio12345 minio12345 ./mc admin user add mysilo/ silo12345 silo12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json ./mc admin policy create mysilo/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json ./mc admin policy create mysilo/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123 ./mc admin policy attach mysilo/ rw --user=silo123
./mc admin policy attach myminio/ lake --user=minio12345 ./mc admin policy attach mysilo/ lake --user=silo12345
./mc mb -l myminio/versioned ./mc mb -l mysilo/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers) ## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null ./mc rm -r --force mysilo/versioned >/dev/null
## mirror again to create another set of version on top ## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum) expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l) user_count=$(./mc admin user list mysilo/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l) policy_count=$(./mc admin policy list mysilo/ | wc -l)
kill $pid kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$! pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) & (silo server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$! pid_2=$!
./mc ready myminio ./mc ready mysilo
expanded_user_count=$(./mc admin user list myminio/ | wc -l) expanded_user_count=$(./mc admin user list mysilo/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l) expanded_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup" echo "BUG: original user count differs from expanded setup"
@@ -70,21 +69,21 @@ if [ $policy_count -ne $expanded_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion" echo "expected versioning enabled after expansion"
exit 1 exit 1
fi fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null ./mc mirror cmd mysilo/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt ./mc ls -r mysilo/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt ./mc ls -r --versions mysilo/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1} ./mc admin decom start mysilo/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do until $(./mc admin decom status mysilo/ | grep -q Complete); do
echo "waiting for decom to finish..." echo "waiting for decom to finish..."
sleep 1 sleep 1
done done
@@ -94,15 +93,15 @@ kill $pid_2
sleep 5 sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) & (silo server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$! pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio ./mc ready mysilo
decom_user_count=$(./mc admin user list myminio/ | wc -l) decom_user_count=$(./mc admin user list mysilo/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l) decom_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission" echo "BUG: original user count differs after decommission"
@@ -114,15 +113,15 @@ if [ $policy_count -ne $decom_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission" echo "BUG: expected versioning enabled after decommission"
exit 1 exit 1
fi fi
./mc ls -r myminio/versioned >decommissioned_ns.txt ./mc ls -r mysilo/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt ./mc ls -r --versions mysilo/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt) out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$? ret=$?
@@ -138,7 +137,7 @@ if [ $ret -ne 0 ]; then
exit 1 exit 1
fi fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum) got_checksum=$(./mc cat mysilo/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}" echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1 exit 1
+49 -50
View File
@@ -4,67 +4,66 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
rm -rf /tmp/xl rm -rf /tmp/xl
rm -rf /tmp/xltier rm -rf /tmp/xltier
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
export CI=true export CI=true
export MINIO_SCANNER_SPEED=fastest export MINIO_SCANNER_SPEED=fastest
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/tmp/decom.log) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/tmp/decom.log) &
pid=$! pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9000/"
./mc ready myminio ./mc ready mysilo
./mc admin user add myminio/ minio123 minio123 ./mc admin user add mysilo/ silo123 silo123
./mc admin user add myminio/ minio12345 minio12345 ./mc admin user add mysilo/ silo12345 silo12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json ./mc admin policy create mysilo/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json ./mc admin policy create mysilo/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123 ./mc admin policy attach mysilo/ rw --user=silo123
./mc admin policy attach myminio/ lake --user=minio12345 ./mc admin policy attach mysilo/ lake --user=silo12345
./mc mb -l myminio/versioned ./mc mb -l mysilo/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers) ## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null ./mc rm -r --force mysilo/versioned >/dev/null
## mirror again to create another set of version on top ## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null ./mc mirror internal mysilo/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum) expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l) user_count=$(./mc admin user list mysilo/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l) policy_count=$(./mc admin policy list mysilo/ | wc -l)
## create a warm tier instance ## create a warm tier instance
(minio server /tmp/xltier/{1...4}/disk{0...1} --address :9002 2>&1 >/dev/null) & (silo server /tmp/xltier/{1...4}/disk{0...1} --address :9002 2>&1 >/dev/null) &
export MC_HOST_mytier="http://minioadmin:minioadmin@localhost:9002/" export MC_HOST_mytier="http://minioadmin:minioadmin@localhost:9002/"
./mc ready myminio ./mc ready mysilo
./mc mb -l myminio/bucket2 ./mc mb -l mysilo/bucket2
./mc mb -l mytier/tiered ./mc mb -l mytier/tiered
## create a tier and set up ilm policy to tier immediately ## create a tier and set up ilm policy to tier immediately
./mc admin tier add minio myminio TIER1 --endpoint http://localhost:9002 --access-key minioadmin --secret-key minioadmin --bucket tiered --prefix prefix5/ ./mc admin tier add minio mysilo TIER1 --endpoint http://localhost:9002 --access-key minioadmin --secret-key minioadmin --bucket tiered --prefix prefix5/
./mc ilm add myminio/bucket2 --transition-days 0 --transition-tier TIER1 --transition-days 0 ./mc ilm add mysilo/bucket2 --transition-days 0 --transition-tier TIER1 --transition-days 0
## mirror some content to bucket2 and capture versions tiered ## mirror some content to bucket2 and capture versions tiered
./mc mirror internal myminio/bucket2/ --quiet >/dev/null ./mc mirror internal mysilo/bucket2/ --quiet >/dev/null
./mc ls -r myminio/bucket2/ >bucket2_ns.txt ./mc ls -r mysilo/bucket2/ >bucket2_ns.txt
./mc ls -r --versions myminio/bucket2/ >bucket2_ns_versions.txt ./mc ls -r --versions mysilo/bucket2/ >bucket2_ns_versions.txt
sleep 30 sleep 30
@@ -72,16 +71,16 @@ sleep 30
kill $pid kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$! pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) & (silo server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$! pid_2=$!
./mc ready myminio ./mc ready mysilo
expanded_user_count=$(./mc admin user list myminio/ | wc -l) expanded_user_count=$(./mc admin user list mysilo/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l) expanded_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup" echo "BUG: original user count differs from expanded setup"
@@ -93,22 +92,22 @@ if [ $policy_count -ne $expanded_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion" echo "expected versioning enabled after expansion"
exit 1 exit 1
fi fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null ./mc mirror cmd mysilo/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt ./mc ls -r mysilo/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt ./mc ls -r --versions mysilo/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1} ./mc admin decom start mysilo/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
count=0 count=0
until $(./mc admin decom status myminio/ | grep -q Complete); do until $(./mc admin decom status mysilo/ | grep -q Complete); do
echo "waiting for decom to finish..." echo "waiting for decom to finish..."
count=$((count + 1)) count=$((count + 1))
if [ ${count} -eq 120 ]; then if [ ${count} -eq 120 ]; then
@@ -122,16 +121,16 @@ kill $pid_2
sleep 5 sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) & (silo server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$! pid=$!
sleep 5 sleep 5
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio ./mc ready mysilo
decom_user_count=$(./mc admin user list myminio/ | wc -l) decom_user_count=$(./mc admin user list mysilo/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l) decom_policy_count=$(./mc admin policy list mysilo/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission" echo "BUG: original user count differs after decommission"
@@ -143,15 +142,15 @@ if [ $policy_count -ne $decom_policy_count ]; then
exit 1 exit 1
fi fi
./mc version info myminio/versioned | grep -q "versioning is enabled" ./mc version info mysilo/versioned | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission" echo "BUG: expected versioning enabled after decommission"
exit 1 exit 1
fi fi
./mc ls -r myminio/versioned >decommissioned_ns.txt ./mc ls -r mysilo/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt ./mc ls -r --versions mysilo/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt) out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$? ret=$?
@@ -167,22 +166,22 @@ if [ $ret -ne 0 ]; then
exit 1 exit 1
fi fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum) got_checksum=$(./mc cat mysilo/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}" echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1 exit 1
fi fi
# after decommissioning, compare listings in bucket2 and tiered # after decommissioning, compare listings in bucket2 and tiered
./mc version info myminio/bucket2 | grep -q "versioning is enabled" ./mc version info mysilo/bucket2 | grep -q "versioning is enabled"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission on bucket2" echo "BUG: expected versioning enabled after decommission on bucket2"
exit 1 exit 1
fi fi
./mc ls -r myminio/bucket2 >decommissioned_bucket2_ns.txt ./mc ls -r mysilo/bucket2 >decommissioned_bucket2_ns.txt
./mc ls -r --versions myminio/bucket2 >decommissioned_bucket2_ns_versions.txt ./mc ls -r --versions mysilo/bucket2 >decommissioned_bucket2_ns_versions.txt
./mc ls -r --versions mytier/tiered/ >tiered_ns_versions2.txt ./mc ls -r --versions mytier/tiered/ >tiered_ns_versions2.txt
out=$(diff -qpruN bucket2_ns.txt decommissioned_bucket2_ns.txt) out=$(diff -qpruN bucket2_ns.txt decommissioned_bucket2_ns.txt)
@@ -206,7 +205,7 @@ if [ $ret -ne 0 ]; then
exit 1 exit 1
fi fi
got_checksum=$(./mc cat myminio/bucket2/dsync/drwmutex.go | md5sum) got_checksum=$(./mc cat mysilo/bucket2/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects with tiering: expected ${expected_checksum} got ${got_checksum}" echo "BUG: decommission failed on encrypted objects with tiering: expected ${expected_checksum} got ${got_checksum}"
exit 1 exit 1
@@ -3,9 +3,9 @@
set -e set -e
cleanup() { cleanup() {
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio || true pkill silo || true
pkill -9 minio || true pkill -9 silo || true
rm -rf /tmp/xl/ || true rm -rf /tmp/xl/ || true
rm -rf /tmp/minio.configfile.{1,2,3,4} || true rm -rf /tmp/minio.configfile.{1,2,3,4} || true
} }
@@ -28,8 +28,7 @@ if [ ! -f ./mc ]; then
;; ;;
esac esac
wget -O mc https://dl.minio.io/client/mc/release/${os,,}-${arch,,}/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
for i in $(seq 1 4); do for i in $(seq 1 4); do
@@ -55,29 +54,29 @@ pools: # Specify the nodes and drives with pools
EOF EOF
done done
minio server --config /tmp/minio.configfile.1 >/tmp/minio1_1.log 2>&1 & silo server --config /tmp/minio.configfile.1 >/tmp/silo1_1.log 2>&1 &
site1_pid=$! site1_pid=$!
minio server --config /tmp/minio.configfile.2 >/tmp/minio2_1.log 2>&1 & silo server --config /tmp/minio.configfile.2 >/tmp/silo2_1.log 2>&1 &
site2_pid=$! site2_pid=$!
minio server --config /tmp/minio.configfile.3 >/tmp/minio3_1.log 2>&1 & silo server --config /tmp/minio.configfile.3 >/tmp/silo3_1.log 2>&1 &
site3_pid=$! site3_pid=$!
minio server --config /tmp/minio.configfile.4 >/tmp/minio4_1.log 2>&1 & silo server --config /tmp/minio.configfile.4 >/tmp/silo4_1.log 2>&1 &
site4_pid=$! site4_pid=$!
export MC_HOST_minio1=http://minr0otUS2r:pBU94AGAY85e@localhost:9001 export MC_HOST_silo1=http://minr0otUS2r:pBU94AGAY85e@localhost:9001
export MC_HOST_minio3=http://minr0otUS2r:pBU94AGAY85e@localhost:9003 export MC_HOST_silo3=http://minr0otUS2r:pBU94AGAY85e@localhost:9003
./mc ready minio1 ./mc ready silo1
./mc ready minio3 ./mc ready silo3
./mc mb minio1/testbucket ./mc mb silo1/testbucket
# copy large upload to newbucket on minio1 # copy large upload to newbucket on silo1
truncate -s 17M lrgfile truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum) expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/testbucket ./mc cp ./lrgfile silo1/testbucket
actual_checksum=$(./mc cat minio3/testbucket/lrgfile | md5sum) actual_checksum=$(./mc cat silo3/testbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "unexpected object checksum, expected: ${expected_checksum} got: ${actual_checksum}" echo "unexpected object checksum, expected: ${expected_checksum} got: ${actual_checksum}"
@@ -85,7 +84,7 @@ if [ "${expected_checksum}" != "${actual_checksum}" ]; then
fi fi
# Compare the difference of the list of disks and their location, with the below expected output # Compare the difference of the list of disks and their location, with the below expected output
diff <(./mc admin info minio1 --json | jq -r '.info.servers[].drives[] | "\(.pool_index),\(.set_index),\(.disk_index) \(.endpoint)"' | sort) <( diff <(./mc admin info silo1 --json | jq -r '.info.servers[].drives[] | "\(.pool_index),\(.set_index),\(.disk_index) \(.endpoint)"' | sort) <(
cat <<EOF cat <<EOF
0,0,0 http://localhost:9001/tmp/xl/node9001/mnt/disk1 0,0,0 http://localhost:9001/tmp/xl/node9001/mnt/disk1
0,0,1 http://localhost:9002/tmp/xl/node9002/mnt/disk1 0,0,1 http://localhost:9002/tmp/xl/node9002/mnt/disk1
@@ -4,14 +4,13 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
docker rm -f $(docker ps -aq) docker rm -f $(docker ps -aq)
rm -rf /tmp/ldap{1..4} rm -rf /tmp/ldap{1..4}
rm -rf /tmp/ldap1{1..4} rm -rf /tmp/ldap1{1..4}
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
mc -v mc -v
@@ -24,47 +23,47 @@ make docker-images
make docker-run make docker-run
cd - cd -
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:22000" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:22000"
export MC_HOST_myminio1="http://minioadmin:minioadmin@localhost:24000" export MC_HOST_mysilo1="http://minioadmin:minioadmin@localhost:24000"
# Start MinIO instance # Start Silo instance
export CI=true export CI=true
(minio server --address :22000 --console-address :10000 http://localhost:22000/tmp/ldap{1...4} 2>&1 >/dev/null) & (silo server --address :22000 --console-address :10000 http://localhost:22000/tmp/ldap{1...4} 2>&1 >/dev/null) &
sleep 30 sleep 30
./mc ready myminio ./mc ready mysilo
./mc idp ldap add myminio server_addr=localhost:389 server_insecure=on \ ./mc idp ldap add mysilo server_addr=localhost:389 server_insecure=on \
lookup_bind_dn=cn=admin,dc=min,dc=io lookup_bind_password=admin \ lookup_bind_dn=cn=admin,dc=min,dc=io lookup_bind_password=admin \
user_dn_search_base_dn=dc=min,dc=io user_dn_search_filter="(uid=%s)" \ user_dn_search_base_dn=dc=min,dc=io user_dn_search_filter="(uid=%s)" \
group_search_base_dn=ou=swengg,dc=min,dc=io group_search_filter="(&(objectclass=groupOfNames)(member=%d))" group_search_base_dn=ou=swengg,dc=min,dc=io group_search_filter="(&(objectclass=groupOfNames)(member=%d))"
./mc admin service restart myminio --json ./mc admin service restart mysilo --json
./mc ready myminio ./mc ready mysilo
./mc admin cluster iam import myminio docs/distributed/samples/myminio-iam-info.zip ./mc admin cluster iam import mysilo docs/distributed/samples/myminio-iam-info.zip
sleep 10 sleep 10
# Verify the list of users and service accounts from the import # Verify the list of users and service accounts from the import
./mc admin user list myminio ./mc admin user list mysilo
USER_COUNT=$(./mc admin user list myminio | wc -l) USER_COUNT=$(./mc admin user list mysilo | wc -l)
if [ "${USER_COUNT}" -ne 2 ]; then if [ "${USER_COUNT}" -ne 2 ]; then
echo "BUG: Expected no of users: 2 Found: ${USER_COUNT}" echo "BUG: Expected no of users: 2 Found: ${USER_COUNT}"
exit 1 exit 1
fi fi
./mc admin user svcacct list myminio "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json ./mc admin user svcacct list mysilo "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json
SVCACCT_COUNT_1=$(./mc admin user svcacct list myminio "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l) SVCACCT_COUNT_1=$(./mc admin user svcacct list mysilo "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_1}" -ne 2 ]; then if [ "${SVCACCT_COUNT_1}" -ne 2 ]; then
echo "BUG: Expected svcacct count for 'uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_1}" echo "BUG: Expected svcacct count for 'uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_1}"
exit 1 exit 1
fi fi
./mc admin user svcacct list myminio "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json ./mc admin user svcacct list mysilo "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json
SVCACCT_COUNT_2=$(./mc admin user svcacct list myminio "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l) SVCACCT_COUNT_2=$(./mc admin user svcacct list mysilo "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_2}" -ne 2 ]; then if [ "${SVCACCT_COUNT_2}" -ne 2 ]; then
echo "BUG: Expected svcacct count for 'uid=dillon,ou=people,ou=swengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_2}" echo "BUG: Expected svcacct count for 'uid=dillon,ou=people,ou=swengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_2}"
exit 1 exit 1
fi fi
# Kill MinIO and LDAP to start afresh with missing groups/DN # Kill Silo and LDAP to start afresh with missing groups/DN
pkill minio pkill silo
docker rm -f $(docker ps -aq) docker rm -f $(docker ps -aq)
rm -rf /tmp/ldap{1..4} rm -rf /tmp/ldap{1..4}
@@ -76,40 +75,40 @@ make docker-images
make docker-run make docker-run
cd - cd -
(minio server --address ":24000" --console-address :10000 http://localhost:24000/tmp/ldap1{1...4} 2>&1 >/dev/null) & (silo server --address ":24000" --console-address :10000 http://localhost:24000/tmp/ldap1{1...4} 2>&1 >/dev/null) &
sleep 30 sleep 30
./mc ready myminio1 ./mc ready mysilo1
./mc idp ldap add myminio1 server_addr=localhost:389 server_insecure=on \ ./mc idp ldap add mysilo1 server_addr=localhost:389 server_insecure=on \
lookup_bind_dn=cn=admin,dc=min,dc=io lookup_bind_password=admin \ lookup_bind_dn=cn=admin,dc=min,dc=io lookup_bind_password=admin \
user_dn_search_base_dn=dc=min,dc=io user_dn_search_filter="(uid=%s)" \ user_dn_search_base_dn=dc=min,dc=io user_dn_search_filter="(uid=%s)" \
group_search_base_dn=ou=hwengg,dc=min,dc=io group_search_filter="(&(objectclass=groupOfNames)(member=%d))" group_search_base_dn=ou=hwengg,dc=min,dc=io group_search_filter="(&(objectclass=groupOfNames)(member=%d))"
./mc admin service restart myminio1 --json ./mc admin service restart mysilo1 --json
./mc ready myminio1 ./mc ready mysilo1
./mc admin cluster iam import myminio1 docs/distributed/samples/myminio-iam-info.zip ./mc admin cluster iam import mysilo1 docs/distributed/samples/myminio-iam-info.zip
sleep 10 sleep 10
# Verify the list of users and service accounts from the import # Verify the list of users and service accounts from the import
./mc admin user list myminio1 ./mc admin user list mysilo1
USER_COUNT=$(./mc admin user list myminio1 | wc -l) USER_COUNT=$(./mc admin user list mysilo1 | wc -l)
if [ "${USER_COUNT}" -ne 1 ]; then if [ "${USER_COUNT}" -ne 1 ]; then
echo "BUG: Expected no of users: 1 Found: ${USER_COUNT}" echo "BUG: Expected no of users: 1 Found: ${USER_COUNT}"
exit 1 exit 1
fi fi
./mc admin user svcacct list myminio1 "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json ./mc admin user svcacct list mysilo1 "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json
SVCACCT_COUNT_1=$(./mc admin user svcacct list myminio1 "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l) SVCACCT_COUNT_1=$(./mc admin user svcacct list mysilo1 "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_1}" -ne 2 ]; then if [ "${SVCACCT_COUNT_1}" -ne 2 ]; then
echo "BUG: Expected svcacct count for 'uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_1}" echo "BUG: Expected svcacct count for 'uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_1}"
exit 1 exit 1
fi fi
./mc admin user svcacct list myminio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json ./mc admin user svcacct list mysilo1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json
SVCACCT_COUNT_2=$(./mc admin user svcacct list myminio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l) SVCACCT_COUNT_2=$(./mc admin user svcacct list mysilo1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_2}" -ne 0 ]; then if [ "${SVCACCT_COUNT_2}" -ne 0 ]; then
echo "BUG: Expected svcacct count for 'uid=dillon,ou=people,ou=swengg,dc=min,dc=io': 0. Found: ${SVCACCT_COUNT_2}" echo "BUG: Expected svcacct count for 'uid=dillon,ou=people,ou=swengg,dc=min,dc=io': 0. Found: ${SVCACCT_COUNT_2}"
exit 1 exit 1
fi fi
# Finally kill running processes # Finally kill running processes
pkill minio pkill silo
docker rm -f $(docker ps -aq) docker rm -f $(docker ps -aq)
+20 -21
View File
@@ -4,20 +4,19 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
docker rm -f $(docker ps -aq) docker rm -f $(docker ps -aq)
rm -rf /tmp/openid{1..4} rm -rf /tmp/openid{1..4}
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:22000" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:22000"
# The service account used below is already present in iam configuration getting imported # The service account used below is already present in iam configuration getting imported
export MC_HOST_myminio1="http://dillon-service-2:dillon-service-2@localhost:22000" export MC_HOST_mysilo1="http://dillon-service-2:dillon-service-2@localhost:22000"
# Start MinIO instance # Start Silo instance
export CI=true export CI=true
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
mc -v mc -v
@@ -30,12 +29,12 @@ mc -v
cd - cd -
) )
(minio server --address :22000 --console-address :10000 http://localhost:22000/tmp/openid{1...4} 2>&1 >/tmp/server.log) & (silo server --address :22000 --console-address :10000 http://localhost:22000/tmp/openid{1...4} 2>&1 >/tmp/server.log) &
./mc ready myminio ./mc ready mysilo
./mc mb myminio/test-bucket ./mc mb mysilo/test-bucket
./mc cp /etc/hosts myminio/test-bucket ./mc cp /etc/hosts mysilo/test-bucket
./mc idp openid add myminio \ ./mc idp openid add mysilo \
config_url="http://localhost:5556/dex/.well-known/openid-configuration" \ config_url="http://localhost:5556/dex/.well-known/openid-configuration" \
client_id="minio-client-app" \ client_id="minio-client-app" \
client_secret="minio-client-app-secret" \ client_secret="minio-client-app-secret" \
@@ -44,39 +43,39 @@ mc -v
display_name="Login via dex1" \ display_name="Login via dex1" \
role_policy="consoleAdmin" role_policy="consoleAdmin"
./mc admin service restart myminio --json ./mc admin service restart mysilo --json
./mc ready myminio ./mc ready mysilo
./mc admin cluster iam import myminio docs/distributed/samples/myminio-iam-info-openid.zip ./mc admin cluster iam import mysilo docs/distributed/samples/myminio-iam-info-openid.zip
# Verify if buckets / objects accessible using service account # Verify if buckets / objects accessible using service account
echo "Verifying buckets and objects access for the imported service account" echo "Verifying buckets and objects access for the imported service account"
./mc ls myminio1/ --json ./mc ls mysilo1/ --json
BKT_COUNT=$(./mc ls myminio1/ --json | jq '.key' | wc -l) BKT_COUNT=$(./mc ls mysilo1/ --json | jq '.key' | wc -l)
if [ "${BKT_COUNT}" -ne 1 ]; then if [ "${BKT_COUNT}" -ne 1 ]; then
echo "BUG: Expected no of bucket: 1, Found: ${BKT_COUNT}" echo "BUG: Expected no of bucket: 1, Found: ${BKT_COUNT}"
exit 1 exit 1
fi fi
BKT_NAME=$(./mc ls myminio1/ --json | jq '.key' | sed 's/"//g' | sed 's\/\\g') BKT_NAME=$(./mc ls mysilo1/ --json | jq '.key' | sed 's/"//g' | sed 's\/\\g')
if [[ ${BKT_NAME} != "test-bucket" ]]; then if [[ ${BKT_NAME} != "test-bucket" ]]; then
echo "BUG: Expected bucket: test-bucket, Found: ${BKT_NAME}" echo "BUG: Expected bucket: test-bucket, Found: ${BKT_NAME}"
exit 1 exit 1
fi fi
./mc ls myminio1/test-bucket ./mc ls mysilo1/test-bucket
OBJ_COUNT=$(./mc ls myminio1/test-bucket --json | jq '.key' | wc -l) OBJ_COUNT=$(./mc ls mysilo1/test-bucket --json | jq '.key' | wc -l)
if [ "${OBJ_COUNT}" -ne 1 ]; then if [ "${OBJ_COUNT}" -ne 1 ]; then
echo "BUG: Expected no of objects: 1, Found: ${OBJ_COUNT}" echo "BUG: Expected no of objects: 1, Found: ${OBJ_COUNT}"
exit 1 exit 1
fi fi
OBJ_NAME=$(./mc ls myminio1/test-bucket --json | jq '.key' | sed 's/"//g') OBJ_NAME=$(./mc ls mysilo1/test-bucket --json | jq '.key' | sed 's/"//g')
if [[ ${OBJ_NAME} != "hosts" ]]; then if [[ ${OBJ_NAME} != "hosts" ]]; then
echo "BUG: Expected object: hosts, Found: ${BKT_NAME}" echo "BUG: Expected object: hosts, Found: ${BKT_NAME}"
exit 1 exit 1
fi fi
# Finally kill running processes # Finally kill running processes
pkill minio pkill silo
docker rm -f $(docker ps -aq) docker rm -f $(docker ps -aq)
+1 -1
View File
@@ -1,5 +1,5 @@
. { . {
etcd churchofminio.com { etcd silo.example {
endpoint http://localhost:2379 http://localhost:4001 endpoint http://localhost:2379 http://localhost:4001
upstream /etc/resolv.conf upstream /etc/resolv.conf
} }
+18 -19
View File
@@ -4,12 +4,11 @@ if [ -n "$TEST_DEBUG" ]; then
set -x set -x
fi fi
pkill minio pkill silo
pkill kes pkill kes
rm -rf /tmp/xl rm -rf /tmp/xl
go install -v github.com/minio/mc@master "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
cp -a $(go env GOPATH)/bin/mc ./mc
if [ ! -f ./kes ]; then if [ ! -f ./kes ]; then
wget --quiet -O kes https://github.com/minio/kes/releases/latest/download/kes-linux-amd64 && wget --quiet -O kes https://github.com/minio/kes/releases/latest/download/kes-linux-amd64 &&
@@ -32,49 +31,49 @@ export MINIO_KMS_KES_ENDPOINT=https://127.0.0.1:7373
export MINIO_KMS_KES_API_KEY="${API_KEY}" export MINIO_KMS_KES_API_KEY="${API_KEY}"
export MINIO_KMS_KES_KEY_NAME=minio-default-key export MINIO_KMS_KES_KEY_NAME=minio-default-key
export MINIO_KMS_KES_CAPATH=public.crt export MINIO_KMS_KES_CAPATH=public.crt
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/" export MC_HOST_mysilo="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) & (silo server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$! pid=$!
mc ready myminio mc ready mysilo
mc admin user add myminio/ minio123 minio123 mc admin user add mysilo/ silo123 silo123
mc admin policy create myminio/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json mc admin policy create mysilo/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json
mc admin policy create myminio/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json mc admin policy create mysilo/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json
mc admin policy attach myminio deny-non-sse-kms-pol --user minio123 mc admin policy attach mysilo deny-non-sse-kms-pol --user silo123
mc admin policy attach myminio deny-invalid-sse-kms-pol --user minio123 mc admin policy attach mysilo deny-invalid-sse-kms-pol --user silo123
mc admin policy attach myminio consoleAdmin --user minio123 mc admin policy attach mysilo consoleAdmin --user silo123
mc mb -l myminio/test-bucket mc mb -l mysilo/test-bucket
mc mb -l myminio/multi-key-poc mc mb -l mysilo/multi-key-poc
export MC_HOST_myminio1="http://minio123:minio123@localhost:9000/" export MC_HOST_mysilo1="http://silo123:silo123@localhost:9000/"
mc cp /etc/issue myminio1/test-bucket mc cp /etc/issue mysilo1/test-bucket
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: PutObject to bucket: test-bucket should succeed. Failed" echo "BUG: PutObject to bucket: test-bucket should succeed. Failed"
exit 1 exit 1
fi fi
mc cp /etc/issue myminio1/multi-key-poc | grep -q "Insufficient permissions to access this path" mc cp /etc/issue mysilo1/multi-key-poc | grep -q "Insufficient permissions to access this path"
ret=$? ret=$?
if [ $ret -eq 0 ]; then if [ $ret -eq 0 ]; then
echo "BUG: PutObject to bucket: multi-key-poc without sse-kms should fail. Succedded" echo "BUG: PutObject to bucket: multi-key-poc without sse-kms should fail. Succedded"
exit 1 exit 1
fi fi
mc cp /etc/hosts myminio1/multi-key-poc/hosts --enc-kms "myminio1/multi-key-poc/hosts=minio-default-key" mc cp /etc/hosts mysilo1/multi-key-poc/hosts --enc-kms "mysilo1/multi-key-poc/hosts=minio-default-key"
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: PutObject to bucket: multi-key-poc with valid sse-kms should succeed. Failed" echo "BUG: PutObject to bucket: multi-key-poc with valid sse-kms should succeed. Failed"
exit 1 exit 1
fi fi
mc cp /etc/issue myminio1/multi-key-poc/issue --enc-kms "myminio1/multi-key-poc/issue=minio-default-key-xxx" | grep "Insufficient permissions to access this path" mc cp /etc/issue mysilo1/multi-key-poc/issue --enc-kms "mysilo1/multi-key-poc/issue=minio-default-key-xxx" | grep "Insufficient permissions to access this path"
ret=$? ret=$?
if [ $ret -eq 0 ]; then if [ $ret -eq 0 ]; then
echo "BUG: PutObject to bucket: multi-key-poc with invalid sse-kms should fail. Succeeded" echo "BUG: PutObject to bucket: multi-key-poc with invalid sse-kms should fail. Succeeded"
@@ -1,9 +1,9 @@
version: '3.7' version: '3.7'
# Settings and configurations that are common for all containers # Settings and configurations that are common for all containers
x-minio-common: &minio-common x-silo-common: &silo-common
image: quay.io/minio/minio:RELEASE.2025-09-06T17-38-46Z image: docker.io/pgsty/silo:latest
command: server --console-address ":9001" http://minio{1...4}/data{1...2} command: server --console-address ":9001" http://silo{1...4}/data{1...2}
expose: expose:
- "9000" - "9000"
- "9001" - "9001"
@@ -16,34 +16,34 @@ x-minio-common: &minio-common
timeout: 5s timeout: 5s
retries: 5 retries: 5
# starts 4 docker containers running minio server instances. # starts 4 docker containers running silo server instances.
# using nginx reverse proxy, load balancing, you can access # using nginx reverse proxy, load balancing, you can access
# it through port 9000. # it through port 9000.
services: services:
minio1: silo1:
<<: *minio-common <<: *silo-common
hostname: minio1 hostname: silo1
volumes: volumes:
- data1-1:/data1 - data1-1:/data1
- data1-2:/data2 - data1-2:/data2
minio2: silo2:
<<: *minio-common <<: *silo-common
hostname: minio2 hostname: silo2
volumes: volumes:
- data2-1:/data1 - data2-1:/data1
- data2-2:/data2 - data2-2:/data2
minio3: silo3:
<<: *minio-common <<: *silo-common
hostname: minio3 hostname: silo3
volumes: volumes:
- data3-1:/data1 - data3-1:/data1
- data3-2:/data2 - data3-2:/data2
minio4: silo4:
<<: *minio-common <<: *silo-common
hostname: minio4 hostname: silo4
volumes: volumes:
- data4-1:/data1 - data4-1:/data1
- data4-2:/data2 - data4-2:/data2
@@ -57,10 +57,10 @@ services:
- "9000:9000" - "9000:9000"
- "9001:9001" - "9001:9001"
depends_on: depends_on:
- minio1 - silo1
- minio2 - silo2
- minio3 - silo3
- minio4 - silo4
## By default this config uses default local driver, ## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration. ## For custom volumes replace with volume driver configuration.
+10 -10
View File
@@ -22,19 +22,19 @@ http {
# include /etc/nginx/conf.d/*.conf; # include /etc/nginx/conf.d/*.conf;
upstream minio { upstream silo {
server minio1:9000; server silo1:9000;
server minio2:9000; server silo2:9000;
server minio3:9000; server silo3:9000;
server minio4:9000; server silo4:9000;
} }
upstream console { upstream console {
ip_hash; ip_hash;
server minio1:9001; server silo1:9001;
server minio2:9001; server silo2:9001;
server minio3:9001; server silo3:9001;
server minio4:9001; server silo4:9001;
} }
server { server {
@@ -63,7 +63,7 @@ http {
proxy_set_header Connection ""; proxy_set_header Connection "";
chunked_transfer_encoding off; chunked_transfer_encoding off;
proxy_pass http://minio; proxy_pass http://silo;
} }
} }
+20 -20
View File
@@ -1,9 +1,9 @@
# Settings and configurations that are common for all containers # Settings and configurations that are common for all containers
x-minio-common: &minio-common x-silo-common: &silo-common
build: build:
context: ../../. context: ../../.
dockerfile: Dockerfile dockerfile: Dockerfile.goreleaser
command: server --console-address ":9001" http://minio{1...4}/data{1...8} command: server --console-address ":9001" http://silo{1...4}/data{1...8}
expose: expose:
- "9000" - "9000"
- "9001" - "9001"
@@ -15,13 +15,13 @@ x-minio-common: &minio-common
timeout: 5s timeout: 5s
retries: 5 retries: 5
# starts 4 docker containers running minio server instances. # Starts four containers running Silo server instances.
# using nginx reverse proxy, load balancing, you can access # using nginx reverse proxy, load balancing, you can access
# it through port 9000. # it through port 9000.
services: services:
minio1: silo1:
<<: *minio-common <<: *silo-common
hostname: minio1 hostname: silo1
volumes: volumes:
- data1-1:/data1 - data1-1:/data1
- data1-2:/data2 - data1-2:/data2
@@ -32,9 +32,9 @@ services:
- data1-7:/data7 - data1-7:/data7
- data1-8:/data8 - data1-8:/data8
minio2: silo2:
<<: *minio-common <<: *silo-common
hostname: minio2 hostname: silo2
volumes: volumes:
- data2-1:/data1 - data2-1:/data1
- data2-2:/data2 - data2-2:/data2
@@ -45,9 +45,9 @@ services:
- data2-7:/data7 - data2-7:/data7
- data2-8:/data8 - data2-8:/data8
minio3: silo3:
<<: *minio-common <<: *silo-common
hostname: minio3 hostname: silo3
volumes: volumes:
- data3-1:/data1 - data3-1:/data1
- data3-2:/data2 - data3-2:/data2
@@ -58,9 +58,9 @@ services:
- data3-7:/data7 - data3-7:/data7
- data3-8:/data8 - data3-8:/data8
minio4: silo4:
<<: *minio-common <<: *silo-common
hostname: minio4 hostname: silo4
volumes: volumes:
- data4-1:/data1 - data4-1:/data1
- data4-2:/data2 - data4-2:/data2
@@ -80,10 +80,10 @@ services:
- "9000:9000" - "9000:9000"
- "9001:9001" - "9001:9001"
depends_on: depends_on:
- minio1 - silo1
- minio2 - silo2
- minio3 - silo3
- minio4 - silo4
## By default this config uses default local driver, ## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration. ## For custom volumes replace with volume driver configuration.
+10 -10
View File
@@ -22,19 +22,19 @@ http {
# include /etc/nginx/conf.d/*.conf; # include /etc/nginx/conf.d/*.conf;
upstream minio { upstream silo {
server minio1:9000; server silo1:9000;
server minio2:9000; server silo2:9000;
server minio3:9000; server silo3:9000;
server minio4:9000; server silo4:9000;
} }
upstream console { upstream console {
ip_hash; ip_hash;
server minio1:9001; server silo1:9001;
server minio2:9001; server silo2:9001;
server minio3:9001; server silo3:9001;
server minio4:9001; server silo4:9001;
} }
server { server {
@@ -63,7 +63,7 @@ http {
proxy_set_header Connection ""; proxy_set_header Connection "";
chunked_transfer_encoding off; chunked_transfer_encoding off;
proxy_pass http://minio; proxy_pass http://silo;
} }
} }
+2 -2
View File
@@ -4,11 +4,11 @@
echo "script failed" >resiliency-initial.log # assume initial state echo "script failed" >resiliency-initial.log # assume initial state
echo "sleep to wait for MinIO Server to be ready prior mc commands" echo "sleep to wait for Silo Server to be ready prior mc commands"
# https://github.com/minio/mc/issues/3599 # https://github.com/minio/mc/issues/3599
MINIO_SERVER_URL="http://127.0.0.1:9000" MINIO_SERVER_URL="http://127.0.0.1:9000"
ALIAS_NAME=myminio ALIAS_NAME=mysilo
BUCKET="test-bucket" BUCKET="test-bucket"
SRC_DIR="/tmp/data" SRC_DIR="/tmp/data"
INLINED_DIR="/tmp/inlined" INLINED_DIR="/tmp/inlined"
+86 -97
View File
@@ -3,19 +3,8 @@
TESTS_RUN_STATUS=1 TESTS_RUN_STATUS=1
function cleanup() { function cleanup() {
echo "Cleaning up MinIO deployment" echo "Cleaning up Silo deployment"
docker compose -f "${DOCKER_COMPOSE_FILE}" down --volumes docker compose -f "${DOCKER_COMPOSE_FILE}" down --volumes --remove-orphans
for container in $(docker ps -q); do
echo Removing docker $container
docker rm -f $container >/dev/null 2>&1
docker wait $container
done
}
function cleanup_and_prune() {
cleanup
docker system prune --volumes --force
docker image prune --all --force
} }
function verify_resiliency() { function verify_resiliency() {
@@ -56,53 +45,53 @@ function test_resiliency_success_with_server_down() {
echo echo
echo -e "${GREEN}Running test_resiliency_success_with_server_down ...${NC}" echo -e "${GREEN}Running test_resiliency_success_with_server_down ...${NC}"
# Stop one node # Stop one node
docker stop resiliency-minio1-1 docker stop resiliency-silo1-1
sleep 10 sleep 10
verify_resiliency "${FUNCNAME[0]}" verify_resiliency "${FUNCNAME[0]}"
# Finally restart the node # Finally restart the node
docker start resiliency-minio1-1 docker start resiliency-silo1-1
./mc ready myminio ./mc ready mysilo
} }
function test_resiliency_failure_with_server_down_and_single_disk_offline() { function test_resiliency_failure_with_server_down_and_single_disk_offline() {
echo echo
echo -e "${GREEN}Running test_resiliency_failure_with_server_down_and_single_disk_offline ...${NC}" echo -e "${GREEN}Running test_resiliency_failure_with_server_down_and_single_disk_offline ...${NC}"
# Stop one node # Stop one node
docker stop resiliency-minio1-1 docker stop resiliency-silo1-1
# In additional, suspend one more disk per set in order to induce a failure # In additional, suspend one more disk per set in order to induce a failure
docker exec resiliency-minio2-1 /bin/sh -c "mv /data2/.minio.sys /data2/.minio.bkp && touch /data2/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "mv /data2/.minio.sys /data2/.minio.bkp && touch /data2/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "mv /data6/.minio.sys /data6/.minio.bkp && touch /data6/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "mv /data6/.minio.sys /data6/.minio.bkp && touch /data6/.minio.sys"
sleep 10 sleep 10
verify_resiliency_failure "${FUNCNAME[0]}" verify_resiliency_failure "${FUNCNAME[0]}"
# Enable the disks back on nodes # Enable the disks back on nodes
docker exec resiliency-minio2-1 /bin/sh -c "rm -rf /data2/.minio.sys && mv /data2/.minio.bkp /data2/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "rm -rf /data2/.minio.sys && mv /data2/.minio.bkp /data2/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "rm -rf /data6/.minio.sys && mv /data6/.minio.bkp /data6/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "rm -rf /data6/.minio.sys && mv /data6/.minio.bkp /data6/.minio.sys"
# Finally restart the node # Finally restart the node
docker start resiliency-minio1-1 docker start resiliency-silo1-1
./mc ready myminio ./mc ready mysilo
} }
function test_resiliency_failure_with_servers_down() { function test_resiliency_failure_with_servers_down() {
echo echo
echo -e "${GREEN}Running test_resiliency_failure_with_servers_down ...${NC}" echo -e "${GREEN}Running test_resiliency_failure_with_servers_down ...${NC}"
# Stop two nodes # Stop two nodes
docker stop resiliency-minio1-1 docker stop resiliency-silo1-1
docker stop resiliency-minio2-1 docker stop resiliency-silo2-1
sleep 10 sleep 10
verify_resiliency_failure "${FUNCNAME[0]}" verify_resiliency_failure "${FUNCNAME[0]}"
# Restart the nodes # Restart the nodes
docker start resiliency-minio1-1 docker start resiliency-silo1-1
docker start resiliency-minio2-1 docker start resiliency-silo2-1
./mc ready myminio ./mc ready mysilo
} }
function test_resiliency_success_with_disks_offline() { function test_resiliency_success_with_disks_offline() {
@@ -110,29 +99,29 @@ function test_resiliency_success_with_disks_offline() {
echo -e "${GREEN}Running test_resiliency_success_with_disks_offline ...${NC}" echo -e "${GREEN}Running test_resiliency_success_with_disks_offline ...${NC}"
# There are 8 disks on each node with EC:4 and two erasure sets. # There are 8 disks on each node with EC:4 and two erasure sets.
# We should be able to safely suspend one disk per set from each server. # We should be able to safely suspend one disk per set from each server.
docker exec resiliency-minio1-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio1-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
sleep 10 sleep 10
verify_resiliency "${FUNCNAME[0]}" verify_resiliency "${FUNCNAME[0]}"
# Finally enable the disks back on nodes # Finally enable the disks back on nodes
docker exec resiliency-minio1-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio1-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
./mc ready myminio ./mc ready mysilo
} }
function test_resiliency_failure_with_too_many_disks_offline() { function test_resiliency_failure_with_too_many_disks_offline() {
@@ -141,33 +130,33 @@ function test_resiliency_failure_with_too_many_disks_offline() {
# There are 8 disks on each node with EC:4 and two erasure sets. # There are 8 disks on each node with EC:4 and two erasure sets.
# We should be able to safely suspend one disk per set from each server. # We should be able to safely suspend one disk per set from each server.
# suspending one additional disk from each set should cause failures # suspending one additional disk from each set should cause failures
docker exec resiliency-minio1-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "mv /data1/.minio.sys /data1/.minio.bkp && touch /data1/.minio.sys"
docker exec resiliency-minio1-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "mv /data5/.minio.sys /data5/.minio.bkp && touch /data5/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "mv /data2/.minio.sys /data2/.minio.bkp && touch /data2/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "mv /data2/.minio.sys /data2/.minio.bkp && touch /data2/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "mv /data6/.minio.sys /data6/.minio.bkp && touch /data6/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "mv /data6/.minio.sys /data6/.minio.bkp && touch /data6/.minio.sys"
sleep 10 sleep 10
verify_resiliency_failure "${FUNCNAME[0]}" verify_resiliency_failure "${FUNCNAME[0]}"
# Finally enable the disks back on nodes # Finally enable the disks back on nodes
docker exec resiliency-minio1-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "rm -rf /data1/.minio.sys && mv /data1/.minio.bkp /data1/.minio.sys"
docker exec resiliency-minio1-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo1-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio4-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys" docker exec resiliency-silo4-1 /bin/sh -c "rm -rf /data5/.minio.sys && mv /data5/.minio.bkp /data5/.minio.sys"
docker exec resiliency-minio2-1 /bin/sh -c "rm -rf /data2/.minio.sys && mv /data2/.minio.bkp /data2/.minio.sys" docker exec resiliency-silo2-1 /bin/sh -c "rm -rf /data2/.minio.sys && mv /data2/.minio.bkp /data2/.minio.sys"
docker exec resiliency-minio3-1 /bin/sh -c "rm -rf /data6/.minio.sys && mv /data6/.minio.bkp /data6/.minio.sys" docker exec resiliency-silo3-1 /bin/sh -c "rm -rf /data6/.minio.sys && mv /data6/.minio.bkp /data6/.minio.sys"
./mc ready myminio ./mc ready mysilo
} }
function find_erasure_set_for_file() { function find_erasure_set_for_file() {
@@ -175,7 +164,7 @@ function find_erasure_set_for_file() {
local FILE=$1 local FILE=$1
local DIR=$2 local DIR=$2
# Check for existence of file in erasure set 1 # Check for existence of file in erasure set 1
docker exec resiliency-minio1-1 /bin/sh -c "stat /data1/test-bucket/$DIR/$FILE/xl.meta" >/dev/null 2>&1 docker exec resiliency-silo1-1 /bin/sh -c "stat /data1/test-bucket/$DIR/$FILE/xl.meta" >/dev/null 2>&1
STATUS=$? STATUS=$?
if [ $STATUS -eq 0 ]; then if [ $STATUS -eq 0 ]; then
DATA_DRIVE=1 DATA_DRIVE=1
@@ -183,7 +172,7 @@ function find_erasure_set_for_file() {
if [ $DATA_DRIVE -eq -1 ]; then if [ $DATA_DRIVE -eq -1 ]; then
# Check for existence of file in erasure set 2 # Check for existence of file in erasure set 2
docker exec resiliency-minio1-1 /bin/sh -c "stat /data5/test-bucket/$DIR/$FILE/xl.meta" >/dev/null 2>&1 docker exec resiliency-silo1-1 /bin/sh -c "stat /data5/test-bucket/$DIR/$FILE/xl.meta" >/dev/null 2>&1
STATUS=$? STATUS=$?
if [ $STATUS -eq 0 ]; then if [ $STATUS -eq 0 ]; then
DATA_DRIVE=5 DATA_DRIVE=5
@@ -208,28 +197,28 @@ function test_resiliency_healing_missing_xl_metas() {
fi fi
# Remove single xl.meta -- status still green # Remove single xl.meta -- status still green
OUTPUT=$(docker exec resiliency-minio1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta")
WANT='{ "before": { "color": "green", "missing": 1, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "green", "missing": 1, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
# Remove two xl.meta's -- status becomes yellow # Remove two xl.meta's -- status becomes yellow
OUTPUT=$(docker exec resiliency-minio1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta")
OUTPUT=$(docker exec resiliency-minio2-1 /bin/sh -c "rm /data$((DATA_DRIVE + 1))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo2-1 /bin/sh -c "rm /data$((DATA_DRIVE + 1))/test-bucket/initial-data/$FILE/xl.meta")
WANT='{ "before": { "color": "yellow", "missing": 2, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "yellow", "missing": 2, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
# Remove three xl.meta's -- status becomes red (3 missing) # Remove three xl.meta's -- status becomes red (3 missing)
OUTPUT=$(docker exec resiliency-minio1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta")
OUTPUT=$(docker exec resiliency-minio2-1 /bin/sh -c "rm /data$((DATA_DRIVE + 1))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo2-1 /bin/sh -c "rm /data$((DATA_DRIVE + 1))/test-bucket/initial-data/$FILE/xl.meta")
OUTPUT=$(docker exec resiliency-minio3-1 /bin/sh -c "rm /data$((DATA_DRIVE + 2))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo3-1 /bin/sh -c "rm /data$((DATA_DRIVE + 2))/test-bucket/initial-data/$FILE/xl.meta")
WANT='{ "before": { "color": "red", "missing": 3, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "red", "missing": 3, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
# Remove four xl.meta's -- status becomes red (4 missing) # Remove four xl.meta's -- status becomes red (4 missing)
OUTPUT=$(docker exec resiliency-minio1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo1-1 /bin/sh -c "rm /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/xl.meta")
OUTPUT=$(docker exec resiliency-minio2-1 /bin/sh -c "rm /data$((DATA_DRIVE + 1))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo2-1 /bin/sh -c "rm /data$((DATA_DRIVE + 1))/test-bucket/initial-data/$FILE/xl.meta")
OUTPUT=$(docker exec resiliency-minio3-1 /bin/sh -c "rm /data$((DATA_DRIVE + 2))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo3-1 /bin/sh -c "rm /data$((DATA_DRIVE + 2))/test-bucket/initial-data/$FILE/xl.meta")
OUTPUT=$(docker exec resiliency-minio4-1 /bin/sh -c "rm /data$((DATA_DRIVE + 3))/test-bucket/initial-data/$FILE/xl.meta") OUTPUT=$(docker exec resiliency-silo4-1 /bin/sh -c "rm /data$((DATA_DRIVE + 3))/test-bucket/initial-data/$FILE/xl.meta")
WANT='{ "before": { "color": "red", "missing": 4, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "red", "missing": 4, "corrupted": 0 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
} }
@@ -250,22 +239,22 @@ function test_resiliency_healing_truncated_parts() {
fi fi
# Truncate single part -- status still green # Truncate single part -- status still green
OUTPUT=$(docker exec resiliency-minio1-1 /bin/sh -c "truncate --size=10K /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/*/part.1") OUTPUT=$(docker exec resiliency-silo1-1 /bin/sh -c "truncate --size=10K /data$((DATA_DRIVE))/test-bucket/initial-data/$FILE/*/part.1")
WANT='{ "before": { "color": "green", "missing": 0, "corrupted": 1 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "green", "missing": 0, "corrupted": 1 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
# Truncate two parts -- status becomes yellow (2 missing) # Truncate two parts -- status becomes yellow (2 missing)
OUTPUT=$(docker exec resiliency-minio2-1 /bin/sh -c "truncate --size=10K /data{$((DATA_DRIVE))..$((DATA_DRIVE + 1))}/test-bucket/initial-data/$FILE/*/part.1") OUTPUT=$(docker exec resiliency-silo2-1 /bin/sh -c "truncate --size=10K /data{$((DATA_DRIVE))..$((DATA_DRIVE + 1))}/test-bucket/initial-data/$FILE/*/part.1")
WANT='{ "before": { "color": "yellow", "missing": 0, "corrupted": 2 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "yellow", "missing": 0, "corrupted": 2 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
# Truncate three parts -- status becomes red (3 missing) # Truncate three parts -- status becomes red (3 missing)
OUTPUT=$(docker exec resiliency-minio3-1 /bin/sh -c "truncate --size=10K /data{$((DATA_DRIVE))..$((DATA_DRIVE + 2))}/test-bucket/initial-data/$FILE/*/part.1") OUTPUT=$(docker exec resiliency-silo3-1 /bin/sh -c "truncate --size=10K /data{$((DATA_DRIVE))..$((DATA_DRIVE + 2))}/test-bucket/initial-data/$FILE/*/part.1")
WANT='{ "before": { "color": "red", "missing": 0, "corrupted": 3 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "red", "missing": 0, "corrupted": 3 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
# Truncate four parts -- status becomes red (4 missing) # Truncate four parts -- status becomes red (4 missing)
OUTPUT=$(docker exec resiliency-minio4-1 /bin/sh -c "truncate --size=10K /data{$((DATA_DRIVE))..$((DATA_DRIVE + 3))}/test-bucket/initial-data/$FILE/*/part.1") OUTPUT=$(docker exec resiliency-silo4-1 /bin/sh -c "truncate --size=10K /data{$((DATA_DRIVE))..$((DATA_DRIVE + 3))}/test-bucket/initial-data/$FILE/*/part.1")
WANT='{ "before": { "color": "red", "missing": 0, "corrupted": 4 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }' WANT='{ "before": { "color": "red", "missing": 0, "corrupted": 4 }, "after": { "color": "green", "missing": 0, "corrupted": 0 }, "args": {"file": "'${FILE}'", "dir": "'${DIR}'"} }'
verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}" verify_resiliency_healing "${FUNCNAME[0]}" "${WANT}"
} }
@@ -275,19 +264,19 @@ function induce_bitrot() {
local DIR=$2 local DIR=$2
local FILE=$3 local FILE=$3
# Figure out the UUID of the directory where the `part.*` files are stored # Figure out the UUID of the directory where the `part.*` files are stored
UUID=$(docker exec resiliency-minio$NODE-1 /bin/sh -c "ls -l $DIR/test-bucket/initial-data/$FILE/*/part.1") UUID=$(docker exec resiliency-silo$NODE-1 /bin/sh -c "ls -l $DIR/test-bucket/initial-data/$FILE/*/part.1")
UUID=$(echo $UUID | cut -d " " -f 9 | cut -d "/" -f 6) UUID=$(echo $UUID | cut -d " " -f 9 | cut -d "/" -f 6)
# Determine head and tail size of file where we will introduce bitrot # Determine head and tail size of file where we will introduce bitrot
FILE_SIZE=$(docker exec resiliency-minio$NODE-1 /bin/sh -c "stat --printf="%s" $DIR/test-bucket/initial-data/$FILE/$UUID/part.1") FILE_SIZE=$(docker exec resiliency-silo$NODE-1 /bin/sh -c "stat --printf="%s" $DIR/test-bucket/initial-data/$FILE/$UUID/part.1")
TAIL_SIZE=$((FILE_SIZE - 32 * 2)) TAIL_SIZE=$((FILE_SIZE - 32 * 2))
# Extract head and tail of file # Extract head and tail of file
$(docker exec resiliency-minio$NODE-1 /bin/sh -c "cat $DIR/test-bucket/initial-data/$FILE/$UUID/part.1 | head --bytes 32 > /tmp/head") $(docker exec resiliency-silo$NODE-1 /bin/sh -c "cat $DIR/test-bucket/initial-data/$FILE/$UUID/part.1 | head --bytes 32 > /tmp/head")
$(docker exec resiliency-minio$NODE-1 /bin/sh -c "cat $DIR/test-bucket/initial-data/$FILE/$UUID/part.1 | tail --bytes $TAIL_SIZE > /tmp/tail") $(docker exec resiliency-silo$NODE-1 /bin/sh -c "cat $DIR/test-bucket/initial-data/$FILE/$UUID/part.1 | tail --bytes $TAIL_SIZE > /tmp/tail")
# Corrupt the part by writing head twice followed by tail # Corrupt the part by writing head twice followed by tail
$(docker exec resiliency-minio$NODE-1 /bin/sh -c "cat /tmp/head /tmp/head /tmp/tail > $DIR/test-bucket/initial-data/$FILE/$UUID/part.1") $(docker exec resiliency-silo$NODE-1 /bin/sh -c "cat /tmp/head /tmp/head /tmp/tail > $DIR/test-bucket/initial-data/$FILE/$UUID/part.1")
} }
function test_resiliency_healing_induced_bitrot() { function test_resiliency_healing_induced_bitrot() {
@@ -338,15 +327,15 @@ function induce_bitrot_for_xlmeta() {
local FILE=$3 local FILE=$3
# Determine head and tail size of file where we will introduce bitrot # Determine head and tail size of file where we will introduce bitrot
FILE_SIZE=$(docker exec resiliency-minio$NODE-1 /bin/sh -c "stat --printf="%s" $DIR/test-bucket/inlined-data/$FILE/xl.meta") FILE_SIZE=$(docker exec resiliency-silo$NODE-1 /bin/sh -c "stat --printf="%s" $DIR/test-bucket/inlined-data/$FILE/xl.meta")
HEAD_SIZE=$((FILE_SIZE - 32 * 2)) HEAD_SIZE=$((FILE_SIZE - 32 * 2))
# Extract head and tail of file # Extract head and tail of file
$(docker exec resiliency-minio$NODE-1 /bin/sh -c "cat $DIR/test-bucket/inlined-data/$FILE/xl.meta | head --bytes $HEAD_SIZE > /head") $(docker exec resiliency-silo$NODE-1 /bin/sh -c "cat $DIR/test-bucket/inlined-data/$FILE/xl.meta | head --bytes $HEAD_SIZE > /head")
$(docker exec resiliency-minio$NODE-1 /bin/sh -c "cat $DIR/test-bucket/inlined-data/$FILE/xl.meta | tail --bytes 32 > /tail") $(docker exec resiliency-silo$NODE-1 /bin/sh -c "cat $DIR/test-bucket/inlined-data/$FILE/xl.meta | tail --bytes 32 > /tail")
# Corrupt xl.meta by writing head followed by tail twice # Corrupt xl.meta by writing head followed by tail twice
$(docker exec resiliency-minio$NODE-1 /bin/sh -c "cat /head /tail tmp/tail > $DIR/test-bucket/inlined-data/$FILE/xl.meta") $(docker exec resiliency-silo$NODE-1 /bin/sh -c "cat /head /tail tmp/tail > $DIR/test-bucket/inlined-data/$FILE/xl.meta")
} }
function test_resiliency_healing_inlined_metadata() { function test_resiliency_healing_inlined_metadata() {
@@ -393,21 +382,21 @@ function test_resiliency_healing_inlined_metadata() {
function main() { function main() {
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget -q https://dl.minio.io/client/mc/release/linux-amd64/mc && chmod +x ./mc "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi fi
export MC_HOST_myminio=http://minioadmin:minioadmin@localhost:9000 export MC_HOST_mysilo=http://minioadmin:minioadmin@localhost:9000
cleanup_and_prune cleanup
# Run resiliency tests against MinIO # Run resiliency tests against Silo
docker compose -f "${DOCKER_COMPOSE_FILE}" up -d docker compose -f "${DOCKER_COMPOSE_FILE}" up -d
# Initial setup # Initial setup
docs/resiliency/resiliency-initial-script.sh docs/resiliency/resiliency-initial-script.sh
RESULT=$(grep "script passed" <resiliency-initial.log) RESULT=$(grep "script passed" <resiliency-initial.log)
if [ "$RESULT" != "script passed" ]; then if [ "$RESULT" != "script passed" ]; then
cleanup_and_prune cleanup
exit 1 exit 1
fi fi
@@ -426,7 +415,7 @@ function main() {
rv=1 rv=1
fi fi
cleanup_and_prune cleanup
exit $rv exit $rv
} }
@@ -2,7 +2,7 @@
echo "script failed" >resiliency-verify-failure.log # assume initial state echo "script failed" >resiliency-verify-failure.log # assume initial state
ALIAS_NAME=myminio ALIAS_NAME=mysilo
BUCKET="test-bucket" BUCKET="test-bucket"
DEST_DIR="/tmp/dest" DEST_DIR="/tmp/dest"
@@ -8,7 +8,7 @@ DIR=$(echo $1 | jq -r '.args.dir')
DEEP=$(echo $1 | jq -r '.args.deep') DEEP=$(echo $1 | jq -r '.args.deep')
WANT=$(echo $1 | jq 'del(.args)') # ... and remove args from wanted result WANT=$(echo $1 | jq 'del(.args)') # ... and remove args from wanted result
ALIAS_NAME=myminio ALIAS_NAME=mysilo
BUCKET="test-bucket" BUCKET="test-bucket"
JQUERY='select(.name=="'"${BUCKET}"'/'"${DIR}"'/'"${FILE}"'") | {"before":{"color": .before.color, "missing": .before.missing, "corrupted": .before.corrupted},"after":{"color": .after.color, "missing": .after.missing, "corrupted": .after.corrupted}}' JQUERY='select(.name=="'"${BUCKET}"'/'"${DIR}"'/'"${FILE}"'") | {"before":{"color": .before.color, "missing": .before.missing, "corrupted": .before.corrupted},"after":{"color": .after.color, "missing": .after.missing, "corrupted": .after.corrupted}}'
if [ "$DEEP" = "true" ]; then if [ "$DEEP" = "true" ]; then
+1 -1
View File
@@ -2,7 +2,7 @@
echo "script failed" >resiliency-verify.log # assume initial state echo "script failed" >resiliency-verify.log # assume initial state
ALIAS_NAME=myminio ALIAS_NAME=mysilo
BUCKET="test-bucket" BUCKET="test-bucket"
SRC_DIR="/tmp/data" SRC_DIR="/tmp/data"
DEST_DIR="/tmp/dest" DEST_DIR="/tmp/dest"
+1 -1
View File
@@ -4,7 +4,7 @@ import boto3
s3 = boto3.client('s3', s3 = boto3.client('s3',
endpoint_url='http://localhost:9000', endpoint_url='http://localhost:9000',
aws_access_key_id='minio', aws_access_key_id='minio',
aws_secret_access_key='minio123', aws_secret_access_key='silo123',
region_name='us-east-1') region_name='us-east-1')
r = s3.select_object_content( r = s3.select_object_content(
+1 -1
View File
@@ -9,6 +9,6 @@ services:
- "389:389" - "389:389"
- "636:636" - "636:636"
environment: environment:
LDAP_ORGANIZATION: "MinIO Inc" LDAP_ORGANIZATION: "Silo Inc"
LDAP_DOMAIN: "min.io" LDAP_DOMAIN: "min.io"
LDAP_ADMIN_PASSWORD: "admin" LDAP_ADMIN_PASSWORD: "admin"
+105 -106
View File
@@ -4,24 +4,24 @@
exit_1() { exit_1() {
cleanup cleanup
echo "minio1 ============" echo "silo1 ============"
cat /tmp/minio1_1.log cat /tmp/silo1_1.log
cat /tmp/minio1_2.log cat /tmp/silo1_2.log
echo "minio2 ============" echo "silo2 ============"
cat /tmp/minio2_1.log cat /tmp/silo2_1.log
cat /tmp/minio2_2.log cat /tmp/silo2_2.log
echo "minio3 ============" echo "silo3 ============"
cat /tmp/minio3_1.log cat /tmp/silo3_1.log
cat /tmp/minio3_2.log cat /tmp/silo3_2.log
exit 1 exit 1
} }
cleanup() { cleanup() {
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/minio-ldap-idp{1,2,3} rm -rf /tmp/silo-ldap-idp{1,2,3}
} }
cleanup cleanup
@@ -34,7 +34,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -48,73 +48,72 @@ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN="ou=swengg,dc=min,dc=io"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member=%d))" export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member=%d))"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
minio server --config-dir /tmp/minio-ldap --address ":9001" /tmp/minio-ldap-idp1/{1...4} >/tmp/minio1_1.log 2>&1 & silo server --config-dir /tmp/silo-ldap --address ":9001" /tmp/silo-ldap-idp1/{1...4} >/tmp/silo1_1.log 2>&1 &
site1_pid=$! site1_pid=$!
minio server --config-dir /tmp/minio-ldap --address ":9002" /tmp/minio-ldap-idp2/{1...4} >/tmp/minio2_1.log 2>&1 & silo server --config-dir /tmp/silo-ldap --address ":9002" /tmp/silo-ldap-idp2/{1...4} >/tmp/silo2_1.log 2>&1 &
site2_pid=$! site2_pid=$!
minio server --config-dir /tmp/minio-ldap --address ":9003" /tmp/minio-ldap-idp3/{1...4} >/tmp/minio3_1.log 2>&1 & silo server --config-dir /tmp/silo-ldap --address ":9003" /tmp/silo-ldap-idp3/{1...4} >/tmp/silo3_1.log 2>&1 &
site3_pid=$! site3_pid=$!
export MC_HOST_minio1=http://minio:minio123@localhost:9001 export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_minio2=http://minio:minio123@localhost:9002 export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_minio3=http://minio:minio123@localhost:9003 export MC_HOST_silo3=http://minio:silo123@localhost:9003
./mc ready minio1 ./mc ready silo1
./mc ready minio2 ./mc ready silo2
./mc ready minio3 ./mc ready silo3
./mc admin replicate add minio1 minio2 minio3 ./mc admin replicate add silo1 silo2 silo3
./mc idp ldap policy attach minio1 consoleAdmin --user="uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc idp ldap policy attach silo1 consoleAdmin --user="uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
sleep 5 sleep 5
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
./mc admin policy create minio1 rw ./docs/site-replication/rw.json ./mc admin policy create silo1 rw ./docs/site-replication/rw.json
sleep 5 sleep 5
./mc admin policy info minio2 rw >/dev/null 2>&1 ./mc admin policy info silo2 rw >/dev/null 2>&1
./mc admin policy info minio3 rw >/dev/null 2>&1 ./mc admin policy info silo3 rw >/dev/null 2>&1
./mc admin policy remove minio3 rw ./mc admin policy remove silo3 rw
sleep 10 sleep 10
./mc admin policy info minio1 rw ./mc admin policy info silo1 rw
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.." echo "expecting the command to fail, exiting.."
exit_1 exit_1
fi fi
./mc admin policy info minio2 rw ./mc admin policy info silo2 rw
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.." echo "expecting the command to fail, exiting.."
exit_1 exit_1
fi fi
./mc admin user info minio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.." echo "policy mapping missing, exiting.."
exit_1 exit_1
fi fi
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.." echo "policy mapping missing, exiting.."
exit_1 exit_1
fi fi
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.." echo "policy mapping missing, exiting.."
exit_1 exit_1
fi fi
# LDAP simple user # LDAP simple user
./mc admin user svcacct add minio2 dillon --access-key testsvc --secret-key testsvc123 ./mc admin user svcacct add silo2 dillon --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.." echo "adding svc account failed, exiting.."
exit_1 exit_1
@@ -122,33 +121,33 @@ fi
sleep 10 sleep 10
./mc idp ldap policy entities minio1 ./mc idp ldap policy entities silo1
./mc idp ldap policy entities minio2 ./mc idp ldap policy entities silo2
./mc idp ldap policy entities minio3 ./mc idp ldap policy entities silo3
./mc admin service restart minio1 ./mc admin service restart silo1
./mc admin service restart minio2 ./mc admin service restart silo2
./mc admin service restart minio3 ./mc admin service restart silo3
sleep 10 sleep 10
./mc idp ldap policy entities minio1 ./mc idp ldap policy entities silo1
./mc idp ldap policy entities minio2 ./mc idp ldap policy entities silo2
./mc idp ldap policy entities minio3 ./mc idp ldap policy entities silo3
./mc admin user svcacct info minio1 testsvc ./mc admin user svcacct info silo1 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.." echo "svc account not mirrored, exiting.."
exit_1 exit_1
fi fi
./mc admin user svcacct info minio2 testsvc ./mc admin user svcacct info silo2 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.." echo "svc account not mirrored, exiting.."
exit_1 exit_1
fi fi
./mc admin user svcacct info minio3 testsvc ./mc admin user svcacct info silo3 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.." echo "svc account not mirrored, exiting.."
exit_1 exit_1
@@ -158,25 +157,25 @@ MC_HOST_svc1=http://testsvc:testsvc123@localhost:9001 ./mc ls svc1
MC_HOST_svc2=http://testsvc:testsvc123@localhost:9002 ./mc ls svc2 MC_HOST_svc2=http://testsvc:testsvc123@localhost:9002 ./mc ls svc2
MC_HOST_svc3=http://testsvc:testsvc123@localhost:9003 ./mc ls svc3 MC_HOST_svc3=http://testsvc:testsvc123@localhost:9003 ./mc ls svc3
./mc admin user svcacct rm minio1 testsvc ./mc admin user svcacct rm silo1 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.." echo "removing svc account failed, exiting.."
exit_1 exit_1
fi fi
./mc admin user info minio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.." echo "policy mapping missing, exiting.."
exit_1 exit_1
fi fi
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.." echo "policy mapping missing, exiting.."
exit_1 exit_1
fi fi
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" ./mc admin user info silo3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "policy mapping missing, exiting.." echo "policy mapping missing, exiting.."
exit_1 exit_1
@@ -184,154 +183,154 @@ fi
sleep 10 sleep 10
./mc admin user svcacct info minio2 testsvc ./mc admin user svcacct info silo2 testsvc
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.." echo "svc account found after delete, exiting.."
exit_1 exit_1
fi fi
./mc admin user svcacct info minio3 testsvc ./mc admin user svcacct info silo3 testsvc
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.." echo "svc account found after delete, exiting.."
exit_1 exit_1
fi fi
./mc mb minio1/newbucket ./mc mb silo1/newbucket
# copy large upload to newbucket on minio1 # copy large upload to newbucket on silo1
truncate -s 17M lrgfile truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum) expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/newbucket ./mc cp ./lrgfile silo1/newbucket
# create a bucket bucket2 on minio1. # create a bucket bucket2 on silo1.
./mc mb minio1/bucket2 ./mc mb silo1/bucket2
sleep 5 sleep 5
./mc stat --no-list minio2/newbucket ./mc stat --no-list silo2/newbucket
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.." echo "expecting bucket to be present. exiting.."
exit_1 exit_1
fi fi
./mc stat --no-list minio3/newbucket ./mc stat --no-list silo3/newbucket
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.." echo "expecting bucket to be present. exiting.."
exit_1 exit_1
fi fi
./mc cp README.md minio2/newbucket/ ./mc cp README.md silo2/newbucket/
sleep 5 sleep 5
./mc stat --no-list minio1/newbucket/README.md ./mc stat --no-list silo1/newbucket/README.md
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.." echo "expecting object to be present. exiting.."
exit_1 exit_1
fi fi
./mc stat --no-list minio3/newbucket/README.md ./mc stat --no-list silo3/newbucket/README.md
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.." echo "expecting object to be present. exiting.."
exit_1 exit_1
fi fi
sleep 10 sleep 10
./mc stat --no-list minio3/newbucket/lrgfile ./mc stat --no-list silo3/newbucket/lrgfile
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.." echo "expected object to be present, exiting.."
exit_1 exit_1
fi fi
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum) actual_checksum=$(./mc cat silo3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}" echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit exit
fi fi
rm ./lrgfile rm ./lrgfile
./mc rm -r --versions --force minio1/newbucket/lrgfile ./mc rm -r --versions --force silo1/newbucket/lrgfile
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.." echo "expected object to be present, exiting.."
exit_1 exit_1
fi fi
sleep 5 sleep 5
./mc stat --no-list minio1/newbucket/lrgfile ./mc stat --no-list silo1/newbucket/lrgfile
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.." echo "expected object to be deleted permanently after replication, exiting.."
exit_1 exit_1
fi fi
vID=$(./mc stat --no-list minio2/newbucket/README.md --json | jq .versionID) vID=$(./mc stat --no-list silo2/newbucket/README.md --json | jq .versionID)
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.." echo "expecting object to be present. exiting.."
exit_1 exit_1
fi fi
./mc tag set --version-id "${vID}" minio2/newbucket/README.md "key=val" ./mc tag set --version-id "${vID}" silo2/newbucket/README.md "key=val"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.." echo "expecting tag set to be successful. exiting.."
exit_1 exit_1
fi fi
sleep 5 sleep 5
./mc tag remove --version-id "${vID}" minio2/newbucket/README.md ./mc tag remove --version-id "${vID}" silo2/newbucket/README.md
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting tag removal to be successful. exiting.." echo "expecting tag removal to be successful. exiting.."
exit_1 exit_1
fi fi
sleep 5 sleep 5
replStatus_minio2=$(./mc stat --no-list minio2/newbucket/README.md --json | jq -r .replicationStatus) replStatus_silo2=$(./mc stat --no-list silo2/newbucket/README.md --json | jq -r .replicationStatus)
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.." echo "expecting object to be present. exiting.."
exit_1 exit_1
fi fi
if [ ${replStatus_minio2} != "COMPLETED" ]; then if [ ${replStatus_silo2} != "COMPLETED" ]; then
echo "expected tag removal to have replicated, exiting..." echo "expected tag removal to have replicated, exiting..."
exit_1 exit_1
fi fi
./mc rm minio3/newbucket/README.md ./mc rm silo3/newbucket/README.md
sleep 5 sleep 5
./mc stat --no-list minio2/newbucket/README.md ./mc stat --no-list silo2/newbucket/README.md
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.." echo "expected file to be deleted, exiting.."
exit_1 exit_1
fi fi
./mc stat --no-list minio1/newbucket/README.md ./mc stat --no-list silo1/newbucket/README.md
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.." echo "expected file to be deleted, exiting.."
exit_1 exit_1
fi fi
./mc mb --with-lock minio3/newbucket-olock ./mc mb --with-lock silo3/newbucket-olock
sleep 5 sleep 5
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled) enabled_silo2=$(./mc stat --json silo2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..." echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1 exit_1
fi fi
if [ "${enabled_minio2}" != "Enabled" ]; then if [ "${enabled_silo2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..." echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1 exit_1
fi fi
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled) enabled_silo1=$(./mc stat --json silo1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..." echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1 exit_1
fi fi
if [ "${enabled_minio1}" != "Enabled" ]; then if [ "${enabled_silo1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..." echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1 exit_1
fi fi
# "Test if most recent tag update is replicated" # "Test if most recent tag update is replicated"
./mc tag set minio2/newbucket "key=val1" ./mc tag set silo2/newbucket "key=val1"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.." echo "expecting tag set to be successful. exiting.."
exit_1 exit_1
@@ -339,49 +338,49 @@ fi
sleep 10 sleep 10
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key) val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..." echo "expected bucket tag to have replicated, exiting..."
exit_1 exit_1
fi fi
# stop minio1 # stop silo1
kill -9 ${site1_pid} kill -9 ${site1_pid}
# Update tag on minio2/newbucket when minio1 is down # Update tag on silo2/newbucket when silo1 is down
./mc tag set minio2/newbucket "key=val2" ./mc tag set silo2/newbucket "key=val2"
# create a new bucket on minio2. This should replicate to minio1 after it comes online. # create a new bucket on silo2. This should replicate to silo1 after it comes online.
./mc mb minio2/newbucket2 ./mc mb silo2/newbucket2
# delete bucket2 on minio2. This should replicate to minio1 after it comes online. # delete bucket2 on silo2. This should replicate to silo1 after it comes online.
./mc rb minio2/bucket2 ./mc rb silo2/bucket2
# Restart minio1 instance # Restart silo1 instance
minio server --config-dir /tmp/minio-ldap --address ":9001" /tmp/minio-ldap-idp1/{1...4} >/tmp/minio1_1.log 2>&1 & silo server --config-dir /tmp/silo-ldap --address ":9001" /tmp/silo-ldap-idp1/{1...4} >/tmp/silo1_1.log 2>&1 &
sleep 200 sleep 200
# Test whether most recent tag update on minio2 is replicated to minio1 # Test whether most recent tag update on silo2 is replicated to silo1
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key) val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..." echo "expected bucket tag to have replicated, exiting..."
exit_1 exit_1
fi fi
# Test if bucket created/deleted when minio1 is down healed # Test if bucket created/deleted when silo1 is down healed
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null diff -q <(./mc ls silo1) <(./mc ls silo2) 1>/dev/null
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..." echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1 exit_1
fi fi
# force a resync after removing all site replication # force a resync after removing all site replication
./mc admin replicate rm --all --force minio1 ./mc admin replicate rm --all --force silo1
./mc rb minio2 --force --dangerous ./mc rb silo2 --force --dangerous
./mc admin replicate add minio1 minio2 ./mc admin replicate add silo1 silo2
./mc admin replicate resync start minio1 minio2 ./mc admin replicate resync start silo1 silo2
sleep 30 sleep 30
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt ./mc ls -r --versions silo1/newbucket >/tmp/silo1.txt
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt ./mc ls -r --versions silo2/newbucket >/tmp/silo2.txt
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt) out=$(diff -qpruN /tmp/silo1.txt /tmp/silo2.txt)
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out" echo "BUG: expected no missing entries after replication resync: $out"
@@ -1,458 +0,0 @@
#!/usr/bin/env bash
# shellcheck disable=SC2120
exit_1() {
cleanup
echo "minio1 ============"
cat /tmp/minio1_1.log
cat /tmp/minio1_2.log
echo "minio2 ============"
cat /tmp/minio2_1.log
cat /tmp/minio2_2.log
echo "minio3 ============"
cat /tmp/minio3_1.log
cat /tmp/minio3_2.log
exit 1
}
cleanup() {
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/minio-internal-idp{1,2,3}
}
cleanup
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
minio server --config-dir /tmp/minio-internal --address ":9001" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_1.log 2>&1 &
site1_pid1=$!
minio server --config-dir /tmp/minio-internal --address ":9010" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_2.log 2>&1 &
site1_pid2=$!
minio server --config-dir /tmp/minio-internal --address ":9002" http://localhost:9002/tmp/minio-internal-idp2/{1...4} http://localhost:9020/tmp/minio-internal-idp2/{5...8} >/tmp/minio2_1.log 2>&1 &
site2_pid1=$!
minio server --config-dir /tmp/minio-internal --address ":9020" http://localhost:9002/tmp/minio-internal-idp2/{1...4} http://localhost:9020/tmp/minio-internal-idp2/{5...8} >/tmp/minio2_2.log 2>&1 &
site2_pid2=$!
minio server --config-dir /tmp/minio-internal --address ":9003" http://localhost:9003/tmp/minio-internal-idp3/{1...4} http://localhost:9030/tmp/minio-internal-idp3/{5...8} >/tmp/minio3_1.log 2>&1 &
site3_pid1=$!
minio server --config-dir /tmp/minio-internal --address ":9030" http://localhost:9003/tmp/minio-internal-idp3/{1...4} http://localhost:9030/tmp/minio-internal-idp3/{5...8} >/tmp/minio3_2.log 2>&1 &
site3_pid2=$!
export MC_HOST_minio1=http://minio:minio123@localhost:9001
export MC_HOST_minio2=http://minio:minio123@localhost:9002
export MC_HOST_minio3=http://minio:minio123@localhost:9003
export MC_HOST_minio10=http://minio:minio123@localhost:9010
export MC_HOST_minio20=http://minio:minio123@localhost:9020
export MC_HOST_minio30=http://minio:minio123@localhost:9030
./mc ready minio1
./mc ready minio2
./mc ready minio3
./mc ready minio10
./mc ready minio20
./mc ready minio30
./mc admin replicate add minio1 minio2
site_enabled=$(./mc admin replicate info minio1)
site_enabled_peer=$(./mc admin replicate info minio10)
[[ $site_enabled =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
[[ $site_enabled_peer =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
./mc admin user add minio1 foobar foo12345
## add foobar-g group with foobar
./mc admin group add minio2 foobar-g foobar
./mc admin policy attach minio1 consoleAdmin --user=foobar
sleep 5
./mc admin user info minio2 foobar
./mc admin group info minio1 foobar-g
./mc admin policy create minio1 rw ./docs/site-replication/rw.json
sleep 5
./mc admin policy info minio2 rw >/dev/null 2>&1
./mc admin replicate status minio1
## Add a new empty site
./mc admin replicate add minio1 minio2 minio3
sleep 10
./mc admin policy info minio3 rw >/dev/null 2>&1
./mc admin policy remove minio3 rw
./mc admin replicate status minio3
sleep 10
./mc admin policy info minio1 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info minio2 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info minio3 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin user info minio1 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'minio1', exiting.."
exit_1
fi
./mc admin user info minio2 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'minio2', exiting.."
exit_1
fi
./mc admin user info minio3 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'minio3', exiting.."
exit_1
fi
./mc admin group info minio3 foobar-g
if [ $? -ne 0 ]; then
echo "group mapping missing on 'minio3', exiting.."
exit_1
fi
./mc admin user svcacct add minio2 foobar --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.."
exit_1
fi
./mc admin user svcacct add minio2 minio --access-key testsvc2 --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding root svc account testsvc2 failed, exiting.."
exit_1
fi
sleep 10
export MC_HOST_rootsvc=http://testsvc2:testsvc123@localhost:9002
./mc ls rootsvc
if [ $? -ne 0 ]; then
echo "root service account not inherited root permissions, exiting.."
exit_1
fi
./mc admin user svcacct info minio1 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info minio2 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct rm minio1 testsvc
if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.."
exit_1
fi
sleep 10
./mc admin user svcacct info minio2 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc admin user svcacct info minio3 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc mb minio1/newbucket
# copy large upload to newbucket on minio1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/newbucket
sleep 5
./mc stat --no-list minio2/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
err_minio2=$(./mc stat --no-list minio2/newbucket/xxx --json | jq -r .error.cause.message)
if [ $? -ne 0 ]; then
echo "expecting object to be missing. exiting.."
exit_1
fi
if [ "${err_minio2}" != "Object does not exist" ]; then
echo "expected to see Object does not exist error, exiting..."
exit_1
fi
./mc cp README.md minio2/newbucket/
sleep 5
./mc stat --no-list minio1/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc stat --no-list minio3/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
sleep 10
./mc stat --no-list minio3/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit
fi
rm ./lrgfile
./mc rm -r --versions --force minio1/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
sleep 5
./mc stat --no-list minio1/newbucket/lrgfile
if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.."
exit_1
fi
vID=$(./mc stat --no-list minio2/newbucket/README.md --json | jq .versionID)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc tag set --version-id "${vID}" minio2/newbucket/README.md "key=val"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list minio1/newbucket/README.md --version-id "${vID}" --json | jq -r .tagset.key)
if [ "${val}" != "val" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
./mc tag remove --version-id "${vID}" minio2/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting tag removal to be successful. exiting.."
exit_1
fi
sleep 5
replStatus_minio2=$(./mc stat --no-list minio2/newbucket/README.md --json | jq -r .replicationStatus)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
if [ ${replStatus_minio2} != "COMPLETED" ]; then
echo "expected tag removal to have replicated, exiting..."
exit_1
fi
./mc rm minio3/newbucket/README.md
sleep 5
./mc stat --no-list minio2/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc stat --no-list minio1/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc mb --with-lock minio3/newbucket-olock
sleep 5
set -x
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_minio1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
set +x
# "Test if most recent tag update is replicated"
./mc tag set minio2/newbucket "key=val1"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Create user with policy consoleAdmin on minio1
./mc admin user add minio1 foobarx foobar123
if [ $? -ne 0 ]; then
echo "adding user failed, exiting.."
exit_1
fi
./mc admin policy attach minio1 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "adding policy mapping failed, exiting.."
exit_1
fi
sleep 10
# unset policy for foobarx in minio2
./mc admin policy detach minio2 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "unset policy mapping failed, exiting.."
exit_1
fi
# create a bucket bucket2 on minio1.
./mc mb minio1/bucket2
sleep 10
# Test whether policy detach replicated to minio1
policy=$(./mc admin user info minio1 foobarx --json | jq -r .policyName)
if [ "${policy}" != "null" ]; then
echo "expected policy detach to have replicated, exiting..."
exit_1
fi
kill -9 ${site1_pid1} ${site1_pid2}
# Update tag on minio2/newbucket when minio1 is down
./mc tag set minio2/newbucket "key=val2"
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
./mc mb minio2/newbucket2
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
./mc rb minio2/bucket2
# Restart minio1 instance
minio server --config-dir /tmp/minio-internal --address ":9001" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_1.log 2>&1 &
minio server --config-dir /tmp/minio-internal --address ":9010" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_2.log 2>&1 &
sleep 200
# Test whether most recent tag update on minio2 is replicated to minio1
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Test if bucket created/deleted when minio1 is down healed
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1
fi
# force a resync after removing all site replication
./mc admin replicate rm --all --force minio1
./mc rb minio2 --force --dangerous
./mc admin replicate add minio1 minio2
./mc admin replicate resync start minio1 minio2
sleep 30
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out"
exit 1
fi
+81 -82
View File
@@ -4,21 +4,21 @@
exit_1() { exit_1() {
cleanup cleanup
echo "minio1 ============" echo "silo1 ============"
cat /tmp/minio1_1.log cat /tmp/silo1_1.log
echo "minio2 ============" echo "silo2 ============"
cat /tmp/minio2_1.log cat /tmp/silo2_1.log
echo "minio3 ============" echo "silo3 ============"
cat /tmp/minio3_1.log cat /tmp/silo3_1.log
exit 1 exit 1
} }
cleanup() { cleanup() {
echo "Cleaning up instances of MinIO" echo "Cleaning up instances of Silo"
pkill minio pkill silo
pkill -9 minio pkill -9 silo
rm -rf /tmp/minio{1,2,3} rm -rf /tmp/silo{1,2,3}
} }
cleanup cleanup
@@ -31,7 +31,7 @@ unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw= export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
@@ -42,64 +42,63 @@ export MINIO_IDENTITY_OPENID_CLAIM_NAME="groups"
export MINIO_IDENTITY_OPENID_SCOPES="openid,groups" export MINIO_IDENTITY_OPENID_SCOPES="openid,groups"
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:10000/oauth_callback" export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:10000/oauth_callback"
minio server --address ":9001" --console-address ":10000" /tmp/minio1/{1...4} >/tmp/minio1_1.log 2>&1 & silo server --address ":9001" --console-address ":10000" /tmp/silo1/{1...4} >/tmp/silo1_1.log 2>&1 &
site1_pid=$! site1_pid=$!
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:11000/oauth_callback" export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:11000/oauth_callback"
minio server --address ":9002" --console-address ":11000" /tmp/minio2/{1...4} >/tmp/minio2_1.log 2>&1 & silo server --address ":9002" --console-address ":11000" /tmp/silo2/{1...4} >/tmp/silo2_1.log 2>&1 &
site2_pid=$! site2_pid=$!
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:12000/oauth_callback" export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:12000/oauth_callback"
minio server --address ":9003" --console-address ":12000" /tmp/minio3/{1...4} >/tmp/minio3_1.log 2>&1 & silo server --address ":9003" --console-address ":12000" /tmp/silo3/{1...4} >/tmp/silo3_1.log 2>&1 &
site3_pid=$! site3_pid=$!
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
fi fi
export MC_HOST_minio1=http://minio:minio123@localhost:9001 export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_minio2=http://minio:minio123@localhost:9002 export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_minio3=http://minio:minio123@localhost:9003 export MC_HOST_silo3=http://minio:silo123@localhost:9003
./mc ready minio1 ./mc ready silo1
./mc ready minio2 ./mc ready silo2
./mc ready minio3 ./mc ready silo3
./mc admin replicate add minio1 minio2 minio3 ./mc admin replicate add silo1 silo2 silo3
./mc admin policy create minio1 projecta ./docs/site-replication/rw.json ./mc admin policy create silo1 projecta ./docs/site-replication/rw.json
sleep 5 sleep 5
./mc admin policy info minio2 projecta >/dev/null 2>&1 ./mc admin policy info silo2 projecta >/dev/null 2>&1
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting the command to succeed, exiting.." echo "expecting the command to succeed, exiting.."
exit_1 exit_1
fi fi
./mc admin policy info minio3 projecta >/dev/null 2>&1 ./mc admin policy info silo3 projecta >/dev/null 2>&1
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting the command to succeed, exiting.." echo "expecting the command to succeed, exiting.."
exit_1 exit_1
fi fi
./mc admin policy remove minio3 projecta ./mc admin policy remove silo3 projecta
sleep 10 sleep 10
./mc admin policy info minio1 projecta ./mc admin policy info silo1 projecta
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.." echo "expecting the command to fail, exiting.."
exit_1 exit_1
fi fi
./mc admin policy info minio2 projecta ./mc admin policy info silo2 projecta
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.." echo "expecting the command to fail, exiting.."
exit_1 exit_1
fi fi
./mc admin policy create minio1 projecta ./docs/site-replication/rw.json ./mc admin policy create silo1 projecta ./docs/site-replication/rw.json
sleep 5 sleep 5
# Generate STS credential with STS call to minio1 # Generate STS credential with STS call to silo1
STS_CRED=$(MINIO_ENDPOINT=http://localhost:9001 go run ./docs/site-replication/gen-oidc-sts-cred.go) STS_CRED=$(MINIO_ENDPOINT=http://localhost:9001 go run ./docs/site-replication/gen-oidc-sts-cred.go)
MC_HOST_foo=http://${STS_CRED}@localhost:9001 ./mc ls foo MC_HOST_foo=http://${STS_CRED}@localhost:9001 ./mc ls foo
@@ -110,7 +109,7 @@ fi
sleep 2 sleep 2
# Check that the STS credential works on minio2 and minio3. # Check that the STS credential works on silo2 and silo3.
MC_HOST_foo=http://${STS_CRED}@localhost:9002 ./mc ls foo MC_HOST_foo=http://${STS_CRED}@localhost:9002 ./mc ls foo
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "Expected sts credential to work, exiting.." echo "Expected sts credential to work, exiting.."
@@ -126,7 +125,7 @@ fi
STS_ACCESS_KEY=$(echo ${STS_CRED} | cut -d ':' -f 1) STS_ACCESS_KEY=$(echo ${STS_CRED} | cut -d ':' -f 1)
# Create service account for STS user # Create service account for STS user
./mc admin user svcacct add minio2 $STS_ACCESS_KEY --access-key testsvc --secret-key testsvc123 ./mc admin user svcacct add silo2 $STS_ACCESS_KEY --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.." echo "adding svc account failed, exiting.."
exit_1 exit_1
@@ -134,192 +133,192 @@ fi
sleep 10 sleep 10
./mc admin user svcacct info minio1 testsvc ./mc admin user svcacct info silo1 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.." echo "svc account not mirrored, exiting.."
exit_1 exit_1
fi fi
./mc admin user svcacct info minio2 testsvc ./mc admin user svcacct info silo2 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.." echo "svc account not mirrored, exiting.."
exit_1 exit_1
fi fi
./mc admin user svcacct rm minio1 testsvc ./mc admin user svcacct rm silo1 testsvc
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.." echo "removing svc account failed, exiting.."
exit_1 exit_1
fi fi
sleep 10 sleep 10
./mc admin user svcacct info minio2 testsvc ./mc admin user svcacct info silo2 testsvc
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.." echo "svc account found after delete, exiting.."
exit_1 exit_1
fi fi
./mc admin user svcacct info minio3 testsvc ./mc admin user svcacct info silo3 testsvc
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.." echo "svc account found after delete, exiting.."
exit_1 exit_1
fi fi
# create a bucket bucket2 on minio1. # create a bucket bucket2 on silo1.
./mc mb minio1/bucket2 ./mc mb silo1/bucket2
./mc mb minio1/newbucket ./mc mb silo1/newbucket
# copy large upload to newbucket on minio1 # copy large upload to newbucket on silo1
truncate -s 17M lrgfile truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum) expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/newbucket ./mc cp ./lrgfile silo1/newbucket
sleep 5 sleep 5
./mc stat --no-list minio2/newbucket ./mc stat --no-list silo2/newbucket
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.." echo "expecting bucket to be present. exiting.."
exit_1 exit_1
fi fi
./mc stat --no-list minio3/newbucket ./mc stat --no-list silo3/newbucket
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.." echo "expecting bucket to be present. exiting.."
exit_1 exit_1
fi fi
./mc cp README.md minio2/newbucket/ ./mc cp README.md silo2/newbucket/
sleep 5 sleep 5
./mc stat --no-list minio1/newbucket/README.md ./mc stat --no-list silo1/newbucket/README.md
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.." echo "expecting object to be present. exiting.."
exit_1 exit_1
fi fi
./mc stat --no-list minio3/newbucket/README.md ./mc stat --no-list silo3/newbucket/README.md
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.." echo "expecting object to be present. exiting.."
exit_1 exit_1
fi fi
./mc rm minio3/newbucket/README.md ./mc rm silo3/newbucket/README.md
sleep 5 sleep 5
./mc stat --no-list minio2/newbucket/README.md ./mc stat --no-list silo2/newbucket/README.md
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.." echo "expected file to be deleted, exiting.."
exit_1 exit_1
fi fi
./mc stat --no-list minio1/newbucket/README.md ./mc stat --no-list silo1/newbucket/README.md
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.." echo "expected file to be deleted, exiting.."
exit_1 exit_1
fi fi
sleep 10 sleep 10
./mc stat --no-list minio3/newbucket/lrgfile ./mc stat --no-list silo3/newbucket/lrgfile
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.." echo "expected object to be present, exiting.."
exit_1 exit_1
fi fi
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum) actual_checksum=$(./mc cat silo3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}" echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit exit
fi fi
rm ./lrgfile rm ./lrgfile
./mc rm -r --versions --force minio1/newbucket/lrgfile ./mc rm -r --versions --force silo1/newbucket/lrgfile
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.." echo "expected object to be present, exiting.."
exit_1 exit_1
fi fi
sleep 5 sleep 5
./mc stat --no-list minio1/newbucket/lrgfile ./mc stat --no-list silo1/newbucket/lrgfile
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.." echo "expected object to be deleted permanently after replication, exiting.."
exit_1 exit_1
fi fi
./mc mb --with-lock minio3/newbucket-olock ./mc mb --with-lock silo3/newbucket-olock
sleep 5 sleep 5
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled) enabled_silo2=$(./mc stat --json silo2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..." echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1 exit_1
fi fi
if [ "${enabled_minio2}" != "Enabled" ]; then if [ "${enabled_silo2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..." echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1 exit_1
fi fi
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled) enabled_silo1=$(./mc stat --json silo1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..." echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1 exit_1
fi fi
if [ "${enabled_minio1}" != "Enabled" ]; then if [ "${enabled_silo1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..." echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1 exit_1
fi fi
# "Test if most recent tag update is replicated" # "Test if most recent tag update is replicated"
./mc tag set minio2/newbucket "key=val1" ./mc tag set silo2/newbucket "key=val1"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.." echo "expecting tag set to be successful. exiting.."
exit_1 exit_1
fi fi
sleep 10 sleep 10
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key) val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..." echo "expected bucket tag to have replicated, exiting..."
exit_1 exit_1
fi fi
# stop minio1 instance # stop silo1 instance
kill -9 ${site1_pid} kill -9 ${site1_pid}
# Update tag on minio2/newbucket when minio1 is down # Update tag on silo2/newbucket when silo1 is down
./mc tag set minio2/newbucket "key=val2" ./mc tag set silo2/newbucket "key=val2"
# create a new bucket on minio2. This should replicate to minio1 after it comes online. # create a new bucket on silo2. This should replicate to silo1 after it comes online.
./mc mb minio2/newbucket2 ./mc mb silo2/newbucket2
# delete bucket2 on minio2. This should replicate to minio1 after it comes online. # delete bucket2 on silo2. This should replicate to silo1 after it comes online.
./mc rb minio2/bucket2 ./mc rb silo2/bucket2
# Restart minio1 instance # Restart silo1 instance
minio server --address ":9001" --console-address ":10000" /tmp/minio1/{1...4} >/tmp/minio1_1.log 2>&1 & silo server --address ":9001" --console-address ":10000" /tmp/silo1/{1...4} >/tmp/silo1_1.log 2>&1 &
sleep 200 sleep 200
# Test whether most recent tag update on minio2 is replicated to minio1 # Test whether most recent tag update on silo2 is replicated to silo1
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key) val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..." echo "expected bucket tag to have replicated, exiting..."
exit_1 exit_1
fi fi
# Test if bucket created/deleted when minio1 is down healed # Test if bucket created/deleted when silo1 is down healed
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null diff -q <(./mc ls silo1) <(./mc ls silo2) 1>/dev/null
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..." echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1 exit_1
fi fi
# force a resync after removing all site replication # force a resync after removing all site replication
./mc admin replicate rm --all --force minio1 ./mc admin replicate rm --all --force silo1
./mc rb minio2 --force --dangerous ./mc rb silo2 --force --dangerous
./mc admin replicate add minio1 minio2 ./mc admin replicate add silo1 silo2
./mc admin replicate resync start minio1 minio2 ./mc admin replicate resync start silo1 silo2
sleep 30 sleep 30
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt ./mc ls -r --versions silo1/newbucket >/tmp/silo1.txt
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt ./mc ls -r --versions silo2/newbucket >/tmp/silo2.txt
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt) out=$(diff -qpruN /tmp/silo1.txt /tmp/silo2.txt)
ret=$? ret=$?
if [ $ret -ne 0 ]; then if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out" echo "BUG: expected no missing entries after replication resync: $out"
+457
View File
@@ -0,0 +1,457 @@
#!/usr/bin/env bash
# shellcheck disable=SC2120
exit_1() {
cleanup
echo "silo1 ============"
cat /tmp/silo1_1.log
cat /tmp/silo1_2.log
echo "silo2 ============"
cat /tmp/silo2_1.log
cat /tmp/silo2_2.log
echo "silo3 ============"
cat /tmp/silo3_1.log
cat /tmp/silo3_2.log
exit 1
}
cleanup() {
echo "Cleaning up instances of Silo"
pkill silo
pkill -9 silo
rm -rf /tmp/silo-internal-idp{1,2,3}
}
cleanup
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="silo123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
if [ ! -f ./mc ]; then
"$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
fi
silo server --config-dir /tmp/silo-internal --address ":9001" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_1.log 2>&1 &
site1_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9010" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_2.log 2>&1 &
site1_pid2=$!
silo server --config-dir /tmp/silo-internal --address ":9002" http://localhost:9002/tmp/silo-internal-idp2/{1...4} http://localhost:9020/tmp/silo-internal-idp2/{5...8} >/tmp/silo2_1.log 2>&1 &
site2_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9020" http://localhost:9002/tmp/silo-internal-idp2/{1...4} http://localhost:9020/tmp/silo-internal-idp2/{5...8} >/tmp/silo2_2.log 2>&1 &
site2_pid2=$!
silo server --config-dir /tmp/silo-internal --address ":9003" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_1.log 2>&1 &
site3_pid1=$!
silo server --config-dir /tmp/silo-internal --address ":9030" http://localhost:9003/tmp/silo-internal-idp3/{1...4} http://localhost:9030/tmp/silo-internal-idp3/{5...8} >/tmp/silo3_2.log 2>&1 &
site3_pid2=$!
export MC_HOST_silo1=http://minio:silo123@localhost:9001
export MC_HOST_silo2=http://minio:silo123@localhost:9002
export MC_HOST_silo3=http://minio:silo123@localhost:9003
export MC_HOST_silo10=http://minio:silo123@localhost:9010
export MC_HOST_silo20=http://minio:silo123@localhost:9020
export MC_HOST_silo30=http://minio:silo123@localhost:9030
./mc ready silo1
./mc ready silo2
./mc ready silo3
./mc ready silo10
./mc ready silo20
./mc ready silo30
./mc admin replicate add silo1 silo2
site_enabled=$(./mc admin replicate info silo1)
site_enabled_peer=$(./mc admin replicate info silo10)
[[ $site_enabled =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
[[ $site_enabled_peer =~ "is not enabled" ]] && {
echo "expected both peers to have same information"
exit_1
}
./mc admin user add silo1 foobar foo12345
## add foobar-g group with foobar
./mc admin group add silo2 foobar-g foobar
./mc admin policy attach silo1 consoleAdmin --user=foobar
sleep 5
./mc admin user info silo2 foobar
./mc admin group info silo1 foobar-g
./mc admin policy create silo1 rw ./docs/site-replication/rw.json
sleep 5
./mc admin policy info silo2 rw >/dev/null 2>&1
./mc admin replicate status silo1
## Add a new empty site
./mc admin replicate add silo1 silo2 silo3
sleep 10
./mc admin policy info silo3 rw >/dev/null 2>&1
./mc admin policy remove silo3 rw
./mc admin replicate status silo3
sleep 10
./mc admin policy info silo1 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info silo2 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin policy info silo3 rw
if [ $? -eq 0 ]; then
echo "expecting the command to fail, exiting.."
exit_1
fi
./mc admin user info silo1 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'silo1', exiting.."
exit_1
fi
./mc admin user info silo2 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'silo2', exiting.."
exit_1
fi
./mc admin user info silo3 foobar
if [ $? -ne 0 ]; then
echo "policy mapping missing on 'silo3', exiting.."
exit_1
fi
./mc admin group info silo3 foobar-g
if [ $? -ne 0 ]; then
echo "group mapping missing on 'silo3', exiting.."
exit_1
fi
./mc admin user svcacct add silo2 foobar --access-key testsvc --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding svc account failed, exiting.."
exit_1
fi
./mc admin user svcacct add silo2 minio --access-key testsvc2 --secret-key testsvc123
if [ $? -ne 0 ]; then
echo "adding root svc account testsvc2 failed, exiting.."
exit_1
fi
sleep 10
export MC_HOST_rootsvc=http://testsvc2:testsvc123@localhost:9002
./mc ls rootsvc
if [ $? -ne 0 ]; then
echo "root service account not inherited root permissions, exiting.."
exit_1
fi
./mc admin user svcacct info silo1 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct info silo2 testsvc
if [ $? -ne 0 ]; then
echo "svc account not mirrored, exiting.."
exit_1
fi
./mc admin user svcacct rm silo1 testsvc
if [ $? -ne 0 ]; then
echo "removing svc account failed, exiting.."
exit_1
fi
sleep 10
./mc admin user svcacct info silo2 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc admin user svcacct info silo3 testsvc
if [ $? -eq 0 ]; then
echo "svc account found after delete, exiting.."
exit_1
fi
./mc mb silo1/newbucket
# copy large upload to newbucket on silo1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile silo1/newbucket
sleep 5
./mc stat --no-list silo2/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
./mc stat --no-list silo3/newbucket
if [ $? -ne 0 ]; then
echo "expecting bucket to be present. exiting.."
exit_1
fi
err_silo2=$(./mc stat --no-list silo2/newbucket/xxx --json | jq -r .error.cause.message)
if [ $? -ne 0 ]; then
echo "expecting object to be missing. exiting.."
exit_1
fi
if [ "${err_silo2}" != "Object does not exist" ]; then
echo "expected to see Object does not exist error, exiting..."
exit_1
fi
./mc cp README.md silo2/newbucket/
sleep 5
./mc stat --no-list silo1/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc stat --no-list silo3/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
sleep 10
./mc stat --no-list silo3/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
actual_checksum=$(./mc cat silo3/newbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
exit
fi
rm ./lrgfile
./mc rm -r --versions --force silo1/newbucket/lrgfile
if [ $? -ne 0 ]; then
echo "expected object to be present, exiting.."
exit_1
fi
sleep 5
./mc stat --no-list silo1/newbucket/lrgfile
if [ $? -eq 0 ]; then
echo "expected object to be deleted permanently after replication, exiting.."
exit_1
fi
vID=$(./mc stat --no-list silo2/newbucket/README.md --json | jq .versionID)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
./mc tag set --version-id "${vID}" silo2/newbucket/README.md "key=val"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list silo1/newbucket/README.md --version-id "${vID}" --json | jq -r .tagset.key)
if [ "${val}" != "val" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
./mc tag remove --version-id "${vID}" silo2/newbucket/README.md
if [ $? -ne 0 ]; then
echo "expecting tag removal to be successful. exiting.."
exit_1
fi
sleep 5
replStatus_silo2=$(./mc stat --no-list silo2/newbucket/README.md --json | jq -r .replicationStatus)
if [ $? -ne 0 ]; then
echo "expecting object to be present. exiting.."
exit_1
fi
if [ ${replStatus_silo2} != "COMPLETED" ]; then
echo "expected tag removal to have replicated, exiting..."
exit_1
fi
./mc rm silo3/newbucket/README.md
sleep 5
./mc stat --no-list silo2/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc stat --no-list silo1/newbucket/README.md
if [ $? -eq 0 ]; then
echo "expected file to be deleted, exiting.."
exit_1
fi
./mc mb --with-lock silo3/newbucket-olock
sleep 5
set -x
enabled_silo2=$(./mc stat --json silo2/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_silo2}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
enabled_silo1=$(./mc stat --json silo1/newbucket-olock | jq -r .ObjectLock.enabled)
if [ $? -ne 0 ]; then
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
exit_1
fi
if [ "${enabled_silo1}" != "Enabled" ]; then
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
exit_1
fi
set +x
# "Test if most recent tag update is replicated"
./mc tag set silo2/newbucket "key=val1"
if [ $? -ne 0 ]; then
echo "expecting tag set to be successful. exiting.."
exit_1
fi
sleep 5
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val1" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Create user with policy consoleAdmin on silo1
./mc admin user add silo1 foobarx foobar123
if [ $? -ne 0 ]; then
echo "adding user failed, exiting.."
exit_1
fi
./mc admin policy attach silo1 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "adding policy mapping failed, exiting.."
exit_1
fi
sleep 10
# unset policy for foobarx in silo2
./mc admin policy detach silo2 consoleAdmin --user=foobarx
if [ $? -ne 0 ]; then
echo "unset policy mapping failed, exiting.."
exit_1
fi
# create a bucket bucket2 on silo1.
./mc mb silo1/bucket2
sleep 10
# Test whether policy detach replicated to silo1
policy=$(./mc admin user info silo1 foobarx --json | jq -r .policyName)
if [ "${policy}" != "null" ]; then
echo "expected policy detach to have replicated, exiting..."
exit_1
fi
kill -9 ${site1_pid1} ${site1_pid2}
# Update tag on silo2/newbucket when silo1 is down
./mc tag set silo2/newbucket "key=val2"
# create a new bucket on silo2. This should replicate to silo1 after it comes online.
./mc mb silo2/newbucket2
# delete bucket2 on silo2. This should replicate to silo1 after it comes online.
./mc rb silo2/bucket2
# Restart silo1 instance
silo server --config-dir /tmp/silo-internal --address ":9001" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_1.log 2>&1 &
silo server --config-dir /tmp/silo-internal --address ":9010" http://localhost:9001/tmp/silo-internal-idp1/{1...4} http://localhost:9010/tmp/silo-internal-idp1/{5...8} >/tmp/silo1_2.log 2>&1 &
sleep 200
# Test whether most recent tag update on silo2 is replicated to silo1
val=$(./mc tag list silo1/newbucket --json | jq -r .tagset | jq -r .key)
if [ "${val}" != "val2" ]; then
echo "expected bucket tag to have replicated, exiting..."
exit_1
fi
# Test if bucket created/deleted when silo1 is down healed
diff -q <(./mc ls silo1) <(./mc ls silo2) 1>/dev/null
if [ $? -ne 0 ]; then
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
exit_1
fi
# force a resync after removing all site replication
./mc admin replicate rm --all --force silo1
./mc rb silo2 --force --dangerous
./mc admin replicate add silo1 silo2
./mc admin replicate resync start silo1 silo2
sleep 30
./mc ls -r --versions silo1/newbucket >/tmp/silo1.txt
./mc ls -r --versions silo2/newbucket >/tmp/silo2.txt
out=$(diff -qpruN /tmp/silo1.txt /tmp/silo2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication resync: $out"
exit 1
fi
@@ -4,18 +4,18 @@
exit_1() { exit_1() {
cleanup cleanup
echo "minio1 ============" echo "silo1 ============"
cat /tmp/minio1_1.log cat /tmp/silo1_1.log
echo "minio2 ============" echo "silo2 ============"
cat /tmp/minio2_1.log cat /tmp/silo2_1.log
exit 1 exit 1
} }
cleanup() { cleanup() {
echo -n "Cleaning up instances of MinIO ..." echo -n "Cleaning up instances of Silo ..."
pkill -9 minio || sudo pkill -9 minio pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/minio{1,2} rm -rf /tmp/silo{1,2}
echo "done" echo "done"
} }
@@ -33,7 +33,7 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
# Download AWS CLI # Download AWS CLI
echo -n "Download and install AWS CLI" echo -n "Download and install AWS CLI"
@@ -51,11 +51,11 @@ cat >~/.aws/credentials <<EOF
[enterprise] [enterprise]
region = us-east-1 region = us-east-1
aws_access_key_id = minio aws_access_key_id = minio
aws_secret_access_key = minio123 aws_secret_access_key = silo123
EOF EOF
# Create certificates for TLS enabled MinIO # Create certificates for TLS enabled Silo
echo -n "Setup certs for MinIO instances ..." echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost ./certgen --host localhost
mkdir -p /tmp/certs mkdir -p /tmp/certs
@@ -63,24 +63,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done" echo "done"
# Start MinIO instances # Start Silo instances
echo -n "Starting MinIO instances ..." echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..." echo -n "Downloading Silo client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
echo "done" echo "done"
fi fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001 export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002 export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure ./mc ready silo1 --insecure
./mc ready minio2 --insecure ./mc ready silo2 --insecure
# Prepare data for tests # Prepare data for tests
echo -n "Preparing test data ..." echo -n "Preparing test data ..."
@@ -91,16 +90,16 @@ shred -s 500M /tmp/data/mpartobj
echo "done" echo "done"
# Add replication site # Add replication site
./mc admin replicate add minio1 minio2 --insecure ./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete # sleep for replication to complete
sleep 30 sleep 30
# Create bucket in source cluster # Create bucket in source cluster
echo "Create bucket in source MinIO instance" echo "Create bucket in source Silo instance"
./mc mb minio1/test-bucket --insecure ./mc mb silo1/test-bucket --insecure
# Load objects to source site with checksum header # Load objects to source site with checksum header
echo "Loading objects to source MinIO instance" echo "Loading objects to source Silo instance"
OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64) OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj --body /tmp/data/obj --no-verify-ssl --profile enterprise aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj --body /tmp/data/obj --no-verify-ssl --profile enterprise
@@ -128,20 +127,20 @@ sleep 120
# List the objects from replicated site # List the objects from replicated site
echo "Objects from replicated site" echo "Objects from replicated site"
./mc ls minio2/test-bucket --insecure ./mc ls silo2/test-bucket --insecure
count1=$(./mc ls minio2/test-bucket/obj --insecure | wc -l) count1=$(./mc ls silo2/test-bucket/obj --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/obj not replicated" echo "BUG: object silo1/test-bucket/obj not replicated"
exit_1 exit_1
fi fi
count2=$(./mc ls minio2/test-bucket/mpartobj --insecure | wc -l) count2=$(./mc ls silo2/test-bucket/mpartobj --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/mpartobj not replicated" echo "BUG: object silo1/test-bucket/mpartobj not replicated"
exit_1 exit_1
fi fi
# Stat the objects from source site # Stat the objects from source site
echo "Stat minio1/test-bucket/obj" echo "Stat silo1/test-bucket/obj"
SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise) SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise) SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256') SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
@@ -150,7 +149,7 @@ SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag') SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
# Stat the objects from replicated site # Stat the objects from replicated site
echo "Stat minio2/test-bucket/obj" echo "Stat silo2/test-bucket/obj"
DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise) DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise) DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256') DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
@@ -180,10 +179,10 @@ echo "Set default encryption on "
# test if checksum header is replicated for encrypted objects # test if checksum header is replicated for encrypted objects
# Enable SSE KMS for test-bucket bucket # Enable SSE KMS for test-bucket bucket
./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure ./mc encrypt set sse-kms minio-default-key silo1/test-bucket --insecure
# Load objects to source site with checksum header # Load objects to source site with checksum header
echo "Loading objects to source MinIO instance" echo "Loading objects to source Silo instance"
OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64) OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj2 --body /tmp/data/obj --no-verify-ssl --profile enterprise aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj2 --body /tmp/data/obj --no-verify-ssl --profile enterprise
@@ -211,20 +210,20 @@ sleep 120
# List the objects from replicated site # List the objects from replicated site
echo "Objects from replicated site" echo "Objects from replicated site"
./mc ls minio2/test-bucket --insecure ./mc ls silo2/test-bucket --insecure
count1=$(./mc ls minio2/test-bucket/obj2 --insecure | wc -l) count1=$(./mc ls silo2/test-bucket/obj2 --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/obj2 not replicated" echo "BUG: object silo1/test-bucket/obj2 not replicated"
exit_1 exit_1
fi fi
count2=$(./mc ls minio2/test-bucket/mpartobj2 --insecure | wc -l) count2=$(./mc ls silo2/test-bucket/mpartobj2 --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/mpartobj2 not replicated" echo "BUG: object silo1/test-bucket/mpartobj2 not replicated"
exit_1 exit_1
fi fi
# Stat the objects from source site # Stat the objects from source site
echo "Stat minio1/test-bucket/obj2" echo "Stat silo1/test-bucket/obj2"
SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise) SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise) SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256') SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
@@ -233,7 +232,7 @@ SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag') SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
# Stat the objects from replicated site # Stat the objects from replicated site
echo "Stat minio2/test-bucket/obj2" echo "Stat silo2/test-bucket/obj2"
DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise) DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise) DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256') DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
@@ -4,23 +4,23 @@
exit_1() { exit_1() {
cleanup cleanup
echo "minio1 ============" echo "silo1 ============"
cat /tmp/minio1_1.log cat /tmp/silo1_1.log
echo "minio2 ============" echo "silo2 ============"
cat /tmp/minio2_1.log cat /tmp/silo2_1.log
echo "minio3 ============" echo "silo3 ============"
cat /tmp/minio3_1.log cat /tmp/silo3_1.log
echo "minio4 ============" echo "silo4 ============"
cat /tmp/minio4_1.log cat /tmp/silo4_1.log
exit 1 exit 1
} }
cleanup() { cleanup() {
echo -n "Cleaning up instances of MinIO ..." echo -n "Cleaning up instances of Silo ..."
pkill -9 minio || sudo pkill -9 minio pkill -9 silo || sudo pkill -9 silo
pkill -9 kes || sudo pkill -9 kes pkill -9 kes || sudo pkill -9 kes
rm -rf ${PWD}/keys rm -rf ${PWD}/keys
rm -rf /tmp/minio{1,2,3,4} rm -rf /tmp/silo{1,2,3,4}
echo "done" echo "done"
} }
@@ -29,11 +29,11 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA" TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled MinIO # Create certificates for TLS enabled Silo
echo -n "Setup certs for MinIO instances ..." echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost ./certgen --host localhost
mkdir -p /tmp/certs mkdir -p /tmp/certs
@@ -41,24 +41,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done" echo "done"
# Start MinIO instances # Start Silo instances
echo -n "Starting MinIO instances ..." echo -n "Starting Silo instances ..."
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..." echo -n "Downloading Silo client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
echo "done" echo "done"
fi fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001 export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002 export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure ./mc ready silo1 --insecure
./mc ready minio2 --insecure ./mc ready silo2 --insecure
# Prepare data for tests # Prepare data for tests
echo -n "Preparing test data ..." echo -n "Preparing test data ..."
@@ -73,181 +72,181 @@ shred -s 500M /tmp/data/custpartsize
echo "done" echo "done"
# Add replication site # Add replication site
./mc admin replicate add minio1 minio2 --insecure ./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete # sleep for replication to complete
sleep 30 sleep 30
# Create bucket in source cluster # Create bucket in source cluster
echo "Create bucket in source MinIO instance" echo "Create bucket in source Silo instance"
./mc mb minio1/test-bucket --insecure ./mc mb silo1/test-bucket --insecure
# Enable SSE KMS for the bucket # Enable SSE KMS for the bucket
./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure ./mc encrypt set sse-kms minio-default-key silo1/test-bucket --insecure
# Load objects to source site # Load objects to source site
echo "Loading objects to source MinIO instance" echo "Loading objects to source Silo instance"
./mc cp /tmp/data/encrypted minio1/test-bucket --insecure ./mc cp /tmp/data/encrypted silo1/test-bucket --insecure
./mc cp /tmp/data/mpartobj minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure ./mc cp /tmp/data/mpartobj silo1/test-bucket/mpartobj --enc-c "silo1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize minio1/test-bucket --insecure ./mc cp /tmp/data/defpartsize silo1/test-bucket --insecure
./mc put /tmp/data/custpartsize minio1/test-bucket --insecure --part-size 50MiB ./mc put /tmp/data/custpartsize silo1/test-bucket --insecure --part-size 50MiB
sleep 120 sleep 120
# List the objects from source site # List the objects from source site
echo "Objects from source instance" echo "Objects from source instance"
./mc ls minio1/test-bucket --insecure ./mc ls silo1/test-bucket --insecure
count1=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l) count1=$(./mc ls silo1/test-bucket/encrypted --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/encrypted not found" echo "BUG: object silo1/test-bucket/encrypted not found"
exit_1 exit_1
fi fi
count2=$(./mc ls minio1/test-bucket/mpartobj --insecure | wc -l) count2=$(./mc ls silo1/test-bucket/mpartobj --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/mpartobj not found" echo "BUG: object silo1/test-bucket/mpartobj not found"
exit_1 exit_1
fi fi
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l) count3=$(./mc ls silo1/test-bucket/defpartsize --insecure | wc -l)
if [ "${count3}" -ne 1 ]; then if [ "${count3}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/defpartsize not found" echo "BUG: object silo1/test-bucket/defpartsize not found"
exit_1 exit_1
fi fi
count4=$(./mc ls minio1/test-bucket/custpartsize --insecure | wc -l) count4=$(./mc ls silo1/test-bucket/custpartsize --insecure | wc -l)
if [ "${count4}" -ne 1 ]; then if [ "${count4}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/custpartsize not found" echo "BUG: object silo1/test-bucket/custpartsize not found"
exit_1 exit_1
fi fi
# List the objects from replicated site # List the objects from replicated site
echo "Objects from replicated instance" echo "Objects from replicated instance"
./mc ls minio2/test-bucket --insecure ./mc ls silo2/test-bucket --insecure
repcount1=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l) repcount1=$(./mc ls silo2/test-bucket/encrypted --insecure | wc -l)
if [ "${repcount1}" -ne 1 ]; then if [ "${repcount1}" -ne 1 ]; then
echo "BUG: object test-bucket/encrypted not replicated" echo "BUG: object test-bucket/encrypted not replicated"
exit_1 exit_1
fi fi
repcount2=$(./mc ls minio2/test-bucket/mpartobj --insecure | wc -l) repcount2=$(./mc ls silo2/test-bucket/mpartobj --insecure | wc -l)
if [ "${repcount2}" -ne 1 ]; then if [ "${repcount2}" -ne 1 ]; then
echo "BUG: object test-bucket/mpartobj not replicated" echo "BUG: object test-bucket/mpartobj not replicated"
exit_1 exit_1
fi fi
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l) repcount3=$(./mc ls silo2/test-bucket/defpartsize --insecure | wc -l)
if [ "${repcount3}" -ne 1 ]; then if [ "${repcount3}" -ne 1 ]; then
echo "BUG: object test-bucket/defpartsize not replicated" echo "BUG: object test-bucket/defpartsize not replicated"
exit_1 exit_1
fi fi
repcount4=$(./mc ls minio2/test-bucket/custpartsize --insecure | wc -l) repcount4=$(./mc ls silo2/test-bucket/custpartsize --insecure | wc -l)
if [ "${repcount4}" -ne 1 ]; then if [ "${repcount4}" -ne 1 ]; then
echo "BUG: object test-bucket/custpartsize not replicated" echo "BUG: object test-bucket/custpartsize not replicated"
exit_1 exit_1
fi fi
# Stat the objects from source site # Stat the objects from source site
echo "Stat minio1/test-bucket/encrypted" echo "Stat silo1/test-bucket/encrypted"
./mc stat --no-list minio1/test-bucket/encrypted --insecure --json ./mc stat --no-list silo1/test-bucket/encrypted --insecure --json
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --insecure --json) stat_out1=$(./mc stat --no-list silo1/test-bucket/encrypted --insecure --json)
src_obj1_algo=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption"') src_obj1_algo=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
src_obj1_keyid=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"') src_obj1_keyid=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio1/test-bucket/defpartsize" echo "Stat silo1/test-bucket/defpartsize"
./mc stat --no-list minio1/test-bucket/defpartsize --insecure --json ./mc stat --no-list silo1/test-bucket/defpartsize --insecure --json
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --insecure --json) stat_out2=$(./mc stat --no-list silo1/test-bucket/defpartsize --insecure --json)
src_obj2_algo=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption"') src_obj2_algo=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
src_obj2_keyid=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"') src_obj2_keyid=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio1/test-bucket/custpartsize" echo "Stat silo1/test-bucket/custpartsize"
./mc stat --no-list minio1/test-bucket/custpartsize --insecure --json ./mc stat --no-list silo1/test-bucket/custpartsize --insecure --json
stat_out3=$(./mc stat --no-list minio1/test-bucket/custpartsize --insecure --json) stat_out3=$(./mc stat --no-list silo1/test-bucket/custpartsize --insecure --json)
src_obj3_algo=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption"') src_obj3_algo=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
src_obj3_keyid=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"') src_obj3_keyid=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio1/test-bucket/mpartobj" echo "Stat silo1/test-bucket/mpartobj"
./mc stat --no-list minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo1/test-bucket/mpartobj --enc-c "silo1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out4=$(./mc stat --no-list minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out4=$(./mc stat --no-list silo1/test-bucket/mpartobj --enc-c "silo1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj4_etag=$(echo "${stat_out4}" | jq '.etag') src_obj4_etag=$(echo "${stat_out4}" | jq '.etag')
src_obj4_size=$(echo "${stat_out4}" | jq '.size') src_obj4_size=$(echo "${stat_out4}" | jq '.size')
src_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') src_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Stat the objects from replicated site # Stat the objects from replicated site
echo "Stat minio2/test-bucket/encrypted" echo "Stat silo2/test-bucket/encrypted"
./mc stat --no-list minio2/test-bucket/encrypted --insecure --json ./mc stat --no-list silo2/test-bucket/encrypted --insecure --json
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --insecure --json) stat_out1_rep=$(./mc stat --no-list silo2/test-bucket/encrypted --insecure --json)
rep_obj1_algo=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"') rep_obj1_algo=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
rep_obj1_keyid=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"') rep_obj1_keyid=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio2/test-bucket/defpartsize" echo "Stat silo2/test-bucket/defpartsize"
./mc stat --no-list minio2/test-bucket/defpartsize --insecure --json ./mc stat --no-list silo2/test-bucket/defpartsize --insecure --json
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --insecure --json) stat_out2_rep=$(./mc stat --no-list silo2/test-bucket/defpartsize --insecure --json)
rep_obj2_algo=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"') rep_obj2_algo=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
rep_obj2_keyid=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"') rep_obj2_keyid=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio2/test-bucket/custpartsize" echo "Stat silo2/test-bucket/custpartsize"
./mc stat --no-list minio2/test-bucket/custpartsize --insecure --json ./mc stat --no-list silo2/test-bucket/custpartsize --insecure --json
stat_out3_rep=$(./mc stat --no-list minio2/test-bucket/custpartsize --insecure --json) stat_out3_rep=$(./mc stat --no-list silo2/test-bucket/custpartsize --insecure --json)
rep_obj3_algo=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"') rep_obj3_algo=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
rep_obj3_keyid=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"') rep_obj3_keyid=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
echo "Stat minio2/test-bucket/mpartobj" echo "Stat silo2/test-bucket/mpartobj"
./mc stat --no-list minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo2/test-bucket/mpartobj --enc-c "silo2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out4_rep=$(./mc stat --no-list minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out4_rep=$(./mc stat --no-list silo2/test-bucket/mpartobj --enc-c "silo2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj4_etag=$(echo "${stat_out4}" | jq '.etag') rep_obj4_etag=$(echo "${stat_out4}" | jq '.etag')
rep_obj4_size=$(echo "${stat_out4}" | jq '.size') rep_obj4_size=$(echo "${stat_out4}" | jq '.size')
rep_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') rep_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Check the algo and keyId of replicated objects # Check the algo and keyId of replicated objects
if [ "${rep_obj1_algo}" != "${src_obj1_algo}" ]; then if [ "${rep_obj1_algo}" != "${src_obj1_algo}" ]; then
echo "BUG: Algorithm: '${rep_obj1_algo}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_algo}'" echo "BUG: Algorithm: '${rep_obj1_algo}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_algo}'"
exit_1 exit_1
fi fi
if [ "${rep_obj1_keyid}" != "${src_obj1_keyid}" ]; then if [ "${rep_obj1_keyid}" != "${src_obj1_keyid}" ]; then
echo "BUG: KeyId: '${rep_obj1_keyid}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_keyid}'" echo "BUG: KeyId: '${rep_obj1_keyid}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_keyid}'"
exit_1 exit_1
fi fi
if [ "${rep_obj2_algo}" != "${src_obj2_algo}" ]; then if [ "${rep_obj2_algo}" != "${src_obj2_algo}" ]; then
echo "BUG: Algorithm: '${rep_obj2_algo}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_algo}'" echo "BUG: Algorithm: '${rep_obj2_algo}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_algo}'"
exit_1 exit_1
fi fi
if [ "${rep_obj2_keyid}" != "${src_obj2_keyid}" ]; then if [ "${rep_obj2_keyid}" != "${src_obj2_keyid}" ]; then
echo "BUG: KeyId: '${rep_obj2_keyid}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_keyid}'" echo "BUG: KeyId: '${rep_obj2_keyid}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_keyid}'"
exit_1 exit_1
fi fi
if [ "${rep_obj3_algo}" != "${src_obj3_algo}" ]; then if [ "${rep_obj3_algo}" != "${src_obj3_algo}" ]; then
echo "BUG: Algorithm: '${rep_obj3_algo}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_algo}'" echo "BUG: Algorithm: '${rep_obj3_algo}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_algo}'"
exit_1 exit_1
fi fi
if [ "${rep_obj3_keyid}" != "${src_obj3_keyid}" ]; then if [ "${rep_obj3_keyid}" != "${src_obj3_keyid}" ]; then
echo "BUG: KeyId: '${rep_obj3_keyid}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_keyid}'" echo "BUG: KeyId: '${rep_obj3_keyid}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_keyid}'"
exit_1 exit_1
fi fi
# Check the etag, size and md5 of replicated SSEC object # Check the etag, size and md5 of replicated SSEC object
if [ "${rep_obj4_etag}" != "${src_obj4_etag}" ]; then if [ "${rep_obj4_etag}" != "${src_obj4_etag}" ]; then
echo "BUG: Etag: '${rep_obj4_etag}' of replicated object: 'minio2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_etag}'" echo "BUG: Etag: '${rep_obj4_etag}' of replicated object: 'silo2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_etag}'"
exit_1 exit_1
fi fi
if [ "${rep_obj4_size}" != "${src_obj4_size}" ]; then if [ "${rep_obj4_size}" != "${src_obj4_size}" ]; then
echo "BUG: Size: '${rep_obj4_size}' of replicated object: 'minio2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_size}'" echo "BUG: Size: '${rep_obj4_size}' of replicated object: 'silo2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_size}'"
exit_1 exit_1
fi fi
if [ "${src_obj4_md5}" != "${rep_obj4_md5}" ]; then if [ "${src_obj4_md5}" != "${rep_obj4_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/mpartobj' doesn't match with source. Expected: '${src_obj4_md5}', Found: '${rep_obj4_md5}'" echo "BUG: MD5 checksum of object 'silo2/test-bucket/mpartobj' doesn't match with source. Expected: '${src_obj4_md5}', Found: '${rep_obj4_md5}'"
exit_1 exit_1
fi fi
# Check content of replicated objects # Check content of replicated objects
./mc cat minio2/test-bucket/encrypted --insecure ./mc cat silo2/test-bucket/encrypted --insecure
./mc cat minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/mpartobj --enc-c "silo2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat minio2/test-bucket/defpartsize --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/defpartsize --insecure >/dev/null || exit_1
./mc cat minio2/test-bucket/custpartsize --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/custpartsize --insecure >/dev/null || exit_1
echo -n "Starting MinIO instances with different kms key ..." echo -n "Starting Silo instances with different kms key ..."
CI=on MINIO_KMS_SECRET_KEY=minio3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/minio3/disk{1...4} >/tmp/minio3_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=silo3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/silo3/disk{1...4} >/tmp/silo3_1.log 2>&1 &
CI=on MINIO_KMS_SECRET_KEY=minio4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/minio4/disk{1...4} >/tmp/minio4_1.log 2>&1 & CI=on MINIO_KMS_SECRET_KEY=silo4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=silo123 silo server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/silo4/disk{1...4} >/tmp/silo4_1.log 2>&1 &
echo "done" echo "done"
export MC_HOST_minio3=https://minio:minio123@localhost:9003 export MC_HOST_silo3=https://minio:silo123@localhost:9003
export MC_HOST_minio4=https://minio:minio123@localhost:9004 export MC_HOST_silo4=https://minio:silo123@localhost:9004
./mc ready minio3 --insecure ./mc ready silo3 --insecure
./mc ready minio4 --insecure ./mc ready silo4 --insecure
./mc admin replicate add minio3 minio4 --insecure ./mc admin replicate add silo3 silo4 --insecure
./mc mb minio3/bucket --insecure ./mc mb silo3/bucket --insecure
./mc cp --insecure --enc-kms minio3/bucket=minio3-default-key /tmp/data/encrypted minio3/bucket/x ./mc cp --insecure --enc-kms silo3/bucket=silo3-default-key /tmp/data/encrypted silo3/bucket/x
sleep 10 sleep 10
st=$(./mc stat --json --no-list --insecure minio3/bucket/x | jq -r .replicationStatus) st=$(./mc stat --json --no-list --insecure silo3/bucket/x | jq -r .replicationStatus)
if [ "${st}" != "FAILED" ]; then if [ "${st}" != "FAILED" ]; then
echo "BUG: Replication succeeded when kms key is different" echo "BUG: Replication succeeded when kms key is different"
exit_1 exit_1
@@ -4,19 +4,19 @@
exit_1() { exit_1() {
cleanup cleanup
echo "minio1 ============" echo "silo1 ============"
cat /tmp/minio1_1.log cat /tmp/silo1_1.log
echo "minio2 ============" echo "silo2 ============"
cat /tmp/minio2_1.log cat /tmp/silo2_1.log
exit 1 exit 1
} }
cleanup() { cleanup() {
echo -n "Cleaning up instances of MinIO ..." echo -n "Cleaning up instances of Silo ..."
pkill minio || sudo pkill minio pkill silo || sudo pkill silo
pkill -9 minio || sudo pkill -9 minio pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/minio{1,2} rm -rf /tmp/silo{1,2}
echo "done" echo "done"
} }
@@ -25,11 +25,11 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA" TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled MinIO # Create certificates for TLS enabled Silo
echo -n "Setup certs for MinIO instances ..." echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost ./certgen --host localhost
mkdir -p /tmp/certs mkdir -p /tmp/certs
@@ -37,24 +37,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done" echo "done"
# Start MinIO instances # Start Silo instances
echo -n "Starting MinIO instances ..." echo -n "Starting Silo instances ..."
minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 & silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 & silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..." echo -n "Downloading Silo client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
echo "done" echo "done"
fi fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001 export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002 export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure ./mc ready silo1 --insecure
./mc ready minio2 --insecure ./mc ready silo2 --insecure
# Prepare data for tests # Prepare data for tests
echo -n "Preparing test data ..." echo -n "Preparing test data ..."
@@ -67,129 +66,129 @@ touch /tmp/data/mpartobj.txt
shred -s 500M /tmp/data/mpartobj.txt shred -s 500M /tmp/data/mpartobj.txt
echo "done" echo "done"
# Enable compression for site minio1 # Enable compression for site silo1
./mc admin config set minio1 compression enable=on extensions=".txt" --insecure ./mc admin config set silo1 compression enable=on extensions=".txt" --insecure
./mc admin config set minio1 compression allow_encryption=off --insecure ./mc admin config set silo1 compression allow_encryption=off --insecure
# Create bucket in source cluster # Create bucket in source cluster
echo "Create bucket in source MinIO instance" echo "Create bucket in source Silo instance"
./mc mb minio1/test-bucket --insecure ./mc mb silo1/test-bucket --insecure
# Load objects to source site # Load objects to source site
echo "Loading objects to source MinIO instance" echo "Loading objects to source Silo instance"
./mc cp /tmp/data/plainfile minio1/test-bucket --insecure ./mc cp /tmp/data/plainfile silo1/test-bucket --insecure
./mc cp /tmp/data/encrypted minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure ./mc cp /tmp/data/encrypted silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure ./mc cp /tmp/data/defpartsize silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
# Below should fail as compression and SSEC used at the same time # Below should fail as compression and SSEC used at the same time
# DISABLED: We must check the response header to see if compression was actually applied # DISABLED: We must check the response header to see if compression was actually applied
#RESULT=$({ ./mc put /tmp/data/mpartobj.txt minio1/test-bucket/mpartobj.txt --enc-c "minio1/test-bucket/mpartobj.txt=${TEST_MINIO_ENC_KEY}" --insecure; } 2>&1) #RESULT=$({ ./mc put /tmp/data/mpartobj.txt silo1/test-bucket/mpartobj.txt --enc-c "silo1/test-bucket/mpartobj.txt=${TEST_MINIO_ENC_KEY}" --insecure; } 2>&1)
#if [[ ${RESULT} != *"Server side encryption specified with SSE-C with compression not allowed"* ]]; then #if [[ ${RESULT} != *"Server side encryption specified with SSE-C with compression not allowed"* ]]; then
# echo "BUG: Loading an SSE-C object to site with compression should fail. Succeeded though." # echo "BUG: Loading an SSE-C object to site with compression should fail. Succeeded though."
# exit_1 # exit_1
#fi #fi
# Add replication site # Add replication site
./mc admin replicate add minio1 minio2 --insecure ./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete # sleep for replication to complete
sleep 30 sleep 30
# List the objects from source site # List the objects from source site
echo "Objects from source instance" echo "Objects from source instance"
./mc ls minio1/test-bucket --insecure ./mc ls silo1/test-bucket --insecure
count1=$(./mc ls minio1/test-bucket/plainfile --insecure | wc -l) count1=$(./mc ls silo1/test-bucket/plainfile --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/plainfile not found" echo "BUG: object silo1/test-bucket/plainfile not found"
exit_1 exit_1
fi fi
count2=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l) count2=$(./mc ls silo1/test-bucket/encrypted --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/encrypted not found" echo "BUG: object silo1/test-bucket/encrypted not found"
exit_1 exit_1
fi fi
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l) count3=$(./mc ls silo1/test-bucket/defpartsize --insecure | wc -l)
if [ "${count3}" -ne 1 ]; then if [ "${count3}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/defpartsize not found" echo "BUG: object silo1/test-bucket/defpartsize not found"
exit_1 exit_1
fi fi
sleep 120 sleep 120
# List the objects from replicated site # List the objects from replicated site
echo "Objects from replicated instance" echo "Objects from replicated instance"
./mc ls minio2/test-bucket --insecure ./mc ls silo2/test-bucket --insecure
repcount1=$(./mc ls minio2/test-bucket/plainfile --insecure | wc -l) repcount1=$(./mc ls silo2/test-bucket/plainfile --insecure | wc -l)
if [ "${repcount1}" -ne 1 ]; then if [ "${repcount1}" -ne 1 ]; then
echo "BUG: object test-bucket/plainfile not replicated" echo "BUG: object test-bucket/plainfile not replicated"
exit_1 exit_1
fi fi
repcount2=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l) repcount2=$(./mc ls silo2/test-bucket/encrypted --insecure | wc -l)
if [ "${repcount2}" -ne 1 ]; then if [ "${repcount2}" -ne 1 ]; then
echo "BUG: object test-bucket/encrypted not replicated" echo "BUG: object test-bucket/encrypted not replicated"
exit_1 exit_1
fi fi
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l) repcount3=$(./mc ls silo2/test-bucket/defpartsize --insecure | wc -l)
if [ "${repcount3}" -ne 1 ]; then if [ "${repcount3}" -ne 1 ]; then
echo "BUG: object test-bucket/defpartsize not replicated" echo "BUG: object test-bucket/defpartsize not replicated"
exit_1 exit_1
fi fi
# Stat the SSEC objects from source site # Stat the SSEC objects from source site
echo "Stat minio1/test-bucket/encrypted" echo "Stat silo1/test-bucket/encrypted"
./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out1=$(./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj1_etag=$(echo "${stat_out1}" | jq '.etag') src_obj1_etag=$(echo "${stat_out1}" | jq '.etag')
src_obj1_size=$(echo "${stat_out1}" | jq '.size') src_obj1_size=$(echo "${stat_out1}" | jq '.size')
src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio1/test-bucket/defpartsize" echo "Stat silo1/test-bucket/defpartsize"
./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out2=$(./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj2_etag=$(echo "${stat_out2}" | jq '.etag') src_obj2_etag=$(echo "${stat_out2}" | jq '.etag')
src_obj2_size=$(echo "${stat_out2}" | jq '.size') src_obj2_size=$(echo "${stat_out2}" | jq '.size')
src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Stat the SSEC objects from replicated site # Stat the SSEC objects from replicated site
echo "Stat minio2/test-bucket/encrypted" echo "Stat silo2/test-bucket/encrypted"
./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out1_rep=$(./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag') rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag')
rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size') rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size')
rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio2/test-bucket/defpartsize" echo "Stat silo2/test-bucket/defpartsize"
./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out2_rep=$(./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag') rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag')
rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size') rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size')
rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Check the etag and size of replicated SSEC objects # Check the etag and size of replicated SSEC objects
if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'" echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
exit_1 exit_1
fi fi
if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'" echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
exit_1 exit_1
fi fi
if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'" echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
exit_1 exit_1
fi fi
if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'" echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
exit_1 exit_1
fi fi
# Check content of replicated SSEC objects # Check content of replicated SSEC objects
./mc cat minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure ./mc cat silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cat minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
# Check the MD5 checksums of encrypted objects from source and target # Check the MD5 checksums of encrypted objects from source and target
if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'" echo "BUG: MD5 checksum of object 'silo2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
exit_1 exit_1
fi fi
if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'" echo "BUG: MD5 checksum of object 'silo2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
exit_1 exit_1
fi fi
@@ -4,19 +4,19 @@
exit_1() { exit_1() {
cleanup cleanup
echo "minio1 ============" echo "silo1 ============"
cat /tmp/minio1_1.log cat /tmp/silo1_1.log
echo "minio2 ============" echo "silo2 ============"
cat /tmp/minio2_1.log cat /tmp/silo2_1.log
exit 1 exit 1
} }
cleanup() { cleanup() {
echo -n "Cleaning up instances of MinIO ..." echo -n "Cleaning up instances of Silo ..."
pkill minio || sudo pkill minio pkill silo || sudo pkill silo
pkill -9 minio || sudo pkill -9 minio pkill -9 silo || sudo pkill -9 silo
rm -rf /tmp/minio{1,2} rm -rf /tmp/silo{1,2}
echo "done" echo "done"
} }
@@ -25,11 +25,11 @@ cleanup
export MINIO_CI_CD=1 export MINIO_CI_CD=1
export MINIO_BROWSER=off export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio" export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123" export MINIO_ROOT_PASSWORD="silo123"
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA" TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
# Create certificates for TLS enabled MinIO # Create certificates for TLS enabled Silo
echo -n "Setup certs for MinIO instances ..." echo -n "Setup certs for Silo instances ..."
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
./certgen --host localhost ./certgen --host localhost
mkdir -p /tmp/certs mkdir -p /tmp/certs
@@ -37,24 +37,23 @@ mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
mv private.key /tmp/certs || sudo mv private.key /tmp/certs mv private.key /tmp/certs || sudo mv private.key /tmp/certs
echo "done" echo "done"
# Start MinIO instances # Start Silo instances
echo -n "Starting MinIO instances ..." echo -n "Starting Silo instances ..."
minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 & silo server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/silo1/{1...4}/disk{1...4} /tmp/silo1/{5...8}/disk{1...4} >/tmp/silo1_1.log 2>&1 &
minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 & silo server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/silo2/{1...4}/disk{1...4} /tmp/silo2/{5...8}/disk{1...4} >/tmp/silo2_1.log 2>&1 &
echo "done" echo "done"
if [ ! -f ./mc ]; then if [ ! -f ./mc ]; then
echo -n "Downloading MinIO client ..." echo -n "Downloading Silo client ..."
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc && "$(git rev-parse --show-toplevel)/buildscripts/install-mcli.sh" ./mc
chmod +x mc
echo "done" echo "done"
fi fi
export MC_HOST_minio1=https://minio:minio123@localhost:9001 export MC_HOST_silo1=https://minio:silo123@localhost:9001
export MC_HOST_minio2=https://minio:minio123@localhost:9002 export MC_HOST_silo2=https://minio:silo123@localhost:9002
./mc ready minio1 --insecure ./mc ready silo1 --insecure
./mc ready minio2 --insecure ./mc ready silo2 --insecure
# Prepare data for tests # Prepare data for tests
echo -n "Preparing test data ..." echo -n "Preparing test data ..."
@@ -68,155 +67,155 @@ shred -s 500M /tmp/data/custpartsize
echo "done" echo "done"
# Add replication site # Add replication site
./mc admin replicate add minio1 minio2 --insecure ./mc admin replicate add silo1 silo2 --insecure
# sleep for replication to complete # sleep for replication to complete
sleep 30 sleep 30
# Create bucket in source cluster # Create bucket in source cluster
echo "Create bucket in source MinIO instance" echo "Create bucket in source Silo instance"
./mc mb minio1/test-bucket --insecure ./mc mb silo1/test-bucket --insecure
# Load objects to source site # Load objects to source site
echo "Loading objects to source MinIO instance" echo "Loading objects to source Silo instance"
set -x set -x
./mc cp /tmp/data/plainfile minio1/test-bucket --insecure ./mc cp /tmp/data/plainfile silo1/test-bucket --insecure
./mc cp /tmp/data/encrypted minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure ./mc cp /tmp/data/encrypted silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cp /tmp/data/defpartsize minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure ./mc cp /tmp/data/defpartsize silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
./mc put /tmp/data/custpartsize minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --part-size 50MiB ./mc put /tmp/data/custpartsize silo1/test-bucket/custpartsize --enc-c "silo1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --part-size 50MiB
set +x set +x
sleep 120 sleep 120
# List the objects from source site # List the objects from source site
echo "Objects from source instance" echo "Objects from source instance"
./mc ls minio1/test-bucket --insecure ./mc ls silo1/test-bucket --insecure
count1=$(./mc ls minio1/test-bucket/plainfile --insecure | wc -l) count1=$(./mc ls silo1/test-bucket/plainfile --insecure | wc -l)
if [ "${count1}" -ne 1 ]; then if [ "${count1}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/plainfile not found" echo "BUG: object silo1/test-bucket/plainfile not found"
exit_1 exit_1
fi fi
count2=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l) count2=$(./mc ls silo1/test-bucket/encrypted --insecure | wc -l)
if [ "${count2}" -ne 1 ]; then if [ "${count2}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/encrypted not found" echo "BUG: object silo1/test-bucket/encrypted not found"
exit_1 exit_1
fi fi
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l) count3=$(./mc ls silo1/test-bucket/defpartsize --insecure | wc -l)
if [ "${count3}" -ne 1 ]; then if [ "${count3}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/defpartsize not found" echo "BUG: object silo1/test-bucket/defpartsize not found"
exit_1 exit_1
fi fi
count4=$(./mc ls minio1/test-bucket/custpartsize --insecure | wc -l) count4=$(./mc ls silo1/test-bucket/custpartsize --insecure | wc -l)
if [ "${count4}" -ne 1 ]; then if [ "${count4}" -ne 1 ]; then
echo "BUG: object minio1/test-bucket/custpartsize not found" echo "BUG: object silo1/test-bucket/custpartsize not found"
exit_1 exit_1
fi fi
# List the objects from replicated site # List the objects from replicated site
echo "Objects from replicated instance" echo "Objects from replicated instance"
./mc ls minio2/test-bucket --insecure ./mc ls silo2/test-bucket --insecure
repcount1=$(./mc ls minio2/test-bucket/plainfile --insecure | wc -l) repcount1=$(./mc ls silo2/test-bucket/plainfile --insecure | wc -l)
if [ "${repcount1}" -ne 1 ]; then if [ "${repcount1}" -ne 1 ]; then
echo "BUG: object test-bucket/plainfile not replicated" echo "BUG: object test-bucket/plainfile not replicated"
exit_1 exit_1
fi fi
repcount2=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l) repcount2=$(./mc ls silo2/test-bucket/encrypted --insecure | wc -l)
if [ "${repcount2}" -ne 1 ]; then if [ "${repcount2}" -ne 1 ]; then
echo "BUG: object test-bucket/encrypted not replicated" echo "BUG: object test-bucket/encrypted not replicated"
exit_1 exit_1
fi fi
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l) repcount3=$(./mc ls silo2/test-bucket/defpartsize --insecure | wc -l)
if [ "${repcount3}" -ne 1 ]; then if [ "${repcount3}" -ne 1 ]; then
echo "BUG: object test-bucket/defpartsize not replicated" echo "BUG: object test-bucket/defpartsize not replicated"
exit_1 exit_1
fi fi
repcount4=$(./mc ls minio2/test-bucket/custpartsize --insecure | wc -l) repcount4=$(./mc ls silo2/test-bucket/custpartsize --insecure | wc -l)
if [ "${repcount4}" -ne 1 ]; then if [ "${repcount4}" -ne 1 ]; then
echo "BUG: object test-bucket/custpartsize not replicated" echo "BUG: object test-bucket/custpartsize not replicated"
exit_1 exit_1
fi fi
# Stat the SSEC objects from source site # Stat the SSEC objects from source site
echo "Stat minio1/test-bucket/encrypted" echo "Stat silo1/test-bucket/encrypted"
./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out1=$(./mc stat --no-list silo1/test-bucket/encrypted --enc-c "silo1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj1_etag=$(echo "${stat_out1}" | jq '.etag') src_obj1_etag=$(echo "${stat_out1}" | jq '.etag')
src_obj1_size=$(echo "${stat_out1}" | jq '.size') src_obj1_size=$(echo "${stat_out1}" | jq '.size')
src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio1/test-bucket/defpartsize" echo "Stat silo1/test-bucket/defpartsize"
./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out2=$(./mc stat --no-list silo1/test-bucket/defpartsize --enc-c "silo1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj2_etag=$(echo "${stat_out2}" | jq '.etag') src_obj2_etag=$(echo "${stat_out2}" | jq '.etag')
src_obj2_size=$(echo "${stat_out2}" | jq '.size') src_obj2_size=$(echo "${stat_out2}" | jq '.size')
src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio1/test-bucket/custpartsize" echo "Stat silo1/test-bucket/custpartsize"
./mc stat --no-list minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo1/test-bucket/custpartsize --enc-c "silo1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out3=$(./mc stat --no-list minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out3=$(./mc stat --no-list silo1/test-bucket/custpartsize --enc-c "silo1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
src_obj3_etag=$(echo "${stat_out3}" | jq '.etag') src_obj3_etag=$(echo "${stat_out3}" | jq '.etag')
src_obj3_size=$(echo "${stat_out3}" | jq '.size') src_obj3_size=$(echo "${stat_out3}" | jq '.size')
src_obj3_md5=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') src_obj3_md5=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Stat the SSEC objects from replicated site # Stat the SSEC objects from replicated site
echo "Stat minio2/test-bucket/encrypted" echo "Stat silo2/test-bucket/encrypted"
./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out1_rep=$(./mc stat --no-list silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag') rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag')
rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size') rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size')
rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio2/test-bucket/defpartsize" echo "Stat silo2/test-bucket/defpartsize"
./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out2_rep=$(./mc stat --no-list silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag') rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag')
rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size') rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size')
rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
echo "Stat minio2/test-bucket/custpartsize" echo "Stat silo2/test-bucket/custpartsize"
./mc stat --no-list minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json ./mc stat --no-list silo2/test-bucket/custpartsize --enc-c "silo2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
stat_out3_rep=$(./mc stat --no-list minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json) stat_out3_rep=$(./mc stat --no-list silo2/test-bucket/custpartsize --enc-c "silo2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
rep_obj3_etag=$(echo "${stat_out3_rep}" | jq '.etag') rep_obj3_etag=$(echo "${stat_out3_rep}" | jq '.etag')
rep_obj3_size=$(echo "${stat_out3_rep}" | jq '.size') rep_obj3_size=$(echo "${stat_out3_rep}" | jq '.size')
rep_obj3_md5=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"') rep_obj3_md5=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
# Check the etag and size of replicated SSEC objects # Check the etag and size of replicated SSEC objects
if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'" echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
exit_1 exit_1
fi fi
if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'" echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'silo2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
exit_1 exit_1
fi fi
if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'" echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
exit_1 exit_1
fi fi
if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'" echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'silo2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
exit_1 exit_1
fi fi
if [ "${rep_obj3_etag}" != "${src_obj3_etag}" ]; then if [ "${rep_obj3_etag}" != "${src_obj3_etag}" ]; then
echo "BUG: Etag: '${rep_obj3_etag}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_etag}'" echo "BUG: Etag: '${rep_obj3_etag}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_etag}'"
exit_1 exit_1
fi fi
if [ "${rep_obj3_size}" != "${src_obj3_size}" ]; then if [ "${rep_obj3_size}" != "${src_obj3_size}" ]; then
echo "BUG: Size: '${rep_obj3_size}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_size}'" echo "BUG: Size: '${rep_obj3_size}' of replicated object: 'silo2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_size}'"
exit_1 exit_1
fi fi
# Check content of replicated SSEC objects # Check content of replicated SSEC objects
./mc cat minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure ./mc cat silo2/test-bucket/encrypted --enc-c "silo2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
./mc cat minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/defpartsize --enc-c "silo2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
./mc cat minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1 ./mc cat silo2/test-bucket/custpartsize --enc-c "silo2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
# Check the MD5 checksums of encrypted objects from source and target # Check the MD5 checksums of encrypted objects from source and target
if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'" echo "BUG: MD5 checksum of object 'silo2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
exit_1 exit_1
fi fi
if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'" echo "BUG: MD5 checksum of object 'silo2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
exit_1 exit_1
fi fi
if [ "${src_obj3_md5}" != "${rep_obj3_md5}" ]; then if [ "${src_obj3_md5}" != "${rep_obj3_md5}" ]; then
echo "BUG: MD5 checksum of object 'minio2/test-bucket/custpartsize' doesn't match with source. Expected: '${src_obj3_md5}', Found: '${rep_obj3_md5}'" echo "BUG: MD5 checksum of object 'silo2/test-bucket/custpartsize' doesn't match with source. Expected: '${src_obj3_md5}', Found: '${rep_obj3_md5}'"
exit_1 exit_1
fi fi
+1 -1
View File
@@ -1,5 +1,5 @@
[main] [main]
summary=Maximum server performance for MinIO summary=Maximum server performance for Silo
[vm] [vm]
transparent_hugepage=madvise transparent_hugepage=madvise