Condensing the READMEs dropped the systemd drop-in that keeps data ownership
stable when silo.service takes over an existing minio.service, which the
rebrand guard pins in both files precisely so it cannot vanish quietly. The
path is back as a clause in the install note rather than a section, so the
guard is satisfied and the migration-critical detail stays discoverable in
the repository, not only on the portal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
GitHub generates no contributor graph for forks, so this fork had no
attribution record at all. CONTRIBUTORS.md becomes that record, in three
tiers: the four contributors with code merged into main, each listed with
the pull request and the commit that carries their authorship; the five who
opened pull requests; and the twenty-two who filed the bug reports and
compatibility findings that shaped the releases.
Both READMEs are condensed and gain a Contributors section that shows the
avatars and points at the full record, with README_ZH realigned to README
section for section.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
GPG-signing the RPMs rewrites their bytes after release.yml has already
generated SBOMs, the packages checksum manifest, and attestations from
the as-built files. The new workflow_dispatch-only finalize-release lane
runs between signing and publishing: it refuses non-Draft releases,
verifies the signed RPMs against their sha256 sidecars and the committed
PGSTY public key, confirms every other package still matches the
original manifest, regenerates the RPM SBOMs and the manifest from the
published bytes, cosign-signs the manifest under the workflow identity,
and attests the finalized set before replacing the assets in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The substring grep for the archive line also captured the archive's
.sbom.json entry, whose file is deliberately not downloaded in this
lane, so the sha256sum check failed on every run. awk now matches the
manifest filename column exactly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rpm carried a bare -1 release while the deb carried none. Both now
ship the PGDG-style 1PGSTY segment: silo-VER-1PGSTY.arch.rpm and
silo_VER-1PGSTY_arch.deb, with nfpm rendering it as the RPM Release tag
and the Debian revision respectively. The apk stays bare because Alpine
pkgrel admits only -r<integer>.
sign-release-rpms.sh declares the value once as expected_release;
package-release.sh names artifacts from its own PKG_RELEASE copy, and
test-release.yml evals the signing script's value and asserts both the
download names and the packaged RPM Release/DEB Version against it, so
the copies cannot drift apart silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The go list -deps closure is GOOS-specific, so the CREDITS generated on
darwin (carrying go-m1cpu) could never match a Linux regeneration
(ethtool, numcpus), and the check-gen guard failed on its first CI run.
Generation now pins linux/amd64 with the release build tags, so the
output is identical on every host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundle mcli RELEASE.2026-08-06T00-00-00Z in the container image (pin
and both architecture digests; interop with the new server was
verified end to end during release acceptance) and align the CI
fixture default in install-mcli.sh.
Point the Helm chart defaults at the first real silo image tag:
docker.io/pgsty/silo is a fresh repository, so the inherited
RELEASE.2026-08-04 default could never pull. Chart version moves to
7.0.1 with appVersion RELEASE.2026-08-06T00-00-00Z; the packaging
tripwire in verify-helm-migration.sh follows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gocritic's ifElseChain check fails make lint on the --json/--quiet
branch introduced by b6d47b739; the quality job runs make lint on
every push to main. Behavior is unchanged (verified by re-running the
json/quiet success and failure cases byte-for-byte).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Findings from an adversarial review (Codex, gpt-5.6-sol at max effort)
of 2ff594f4b and 4c34d2309, each independently verified before fixing:
- SBOM generation: buildx attaches a provenance attestation, so every
per-arch digest names an OCI index; Syft's platform default on an
amd64 runner cannot resolve an arm64-only index and the step dies.
Pass --platform explicitly on all four Syft calls (the two classic
lanes had the same latent defect - the renamed workflow has not run
yet, which is why it never fired).
- Release ordering: the HEALTHCHECK survival check now runs against
the pushed architecture image before the versioned and rolling
multi-arch manifests are created, so a broken health config blocks
their promotion; the comment now states honestly that the
arch-suffixed tags are already public at that point.
- Gate assertions: tar's member-argument mode exits non-zero on any
missing name, which under pipefail masked a found forbidden file
when exactly one of them existed; -tv prints symlinks as
'name -> target', defeating $-anchored greps; and the licenses
check proved only one-of-three. Export the rootfs once and assert
every required and forbidden entry individually (busybox/sh and
usr/bin/mc[li] now covered), and match the image healthcheck as an
exact array instead of a substring.
- Probe target vs CLI-configured servers: a probe process cannot see
PID 1's argv, so --url gains EnvVar MINIO_HEALTHCHECK_URL as the
documented way to point the baked-in HEALTHCHECK at a server whose
address/TLS comes from command-line arguments (verified end to end:
server on --address :9010, env var alone turns the container
healthy). Baseline regenerated for the new env token.
- IPv6 zone identifiers: serialize probe URLs via url.URL.String()
so [fe80::1%eth0]:9000 becomes a valid %25-escaped URL (tests added).
- Boolean flags: read --json/--quiet via Bool() so --json=false is
false, instead of IsSet() which treats any occurrence as true.
- Docker's HEALTHCHECK timeout raised to 10s: an outer deadline equal
to the probe's own 5s always SIGKILLed the probe before it could
print its diagnostic line.
- test-release path filter now also triggers on cmd/healthcheck-main.go
and cmd/main.go, so subcommand regressions run the image gate.
Not adopted: require_text's comment-insensitivity in verify-rebrand.sh
(snapshot-tripwire by design, consistent with its other assertions -
the semantic check lives in the CI gate now), and full
staging-then-promote tag publishing (a workflow-wide redesign shared
with the classic lanes, tracked as follow-up).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Publish pgsty/silo:<RELEASE>-distroless (and a rolling :distroless tag)
alongside the classic image: gcr.io/distroless/static-debian12 plus
exactly one program, the silo binary. No shell, no mc, no curl, no
entrypoint script - the binary is the ENTRYPOINT and the baked-in
exec-form HEALTHCHECK runs 'silo healthcheck ready'. The classic image
and its mc-based health checks are deliberately unchanged.
Design: silo.pgsty.com/compatibility/feature/healthcheck/
/data is created in the image layer, world-writable, because Docker
seeds fresh volumes from the layer mountpoint and no entrypoint exists
to repair ownership at runtime (issue #55); the parent directory is
copied from a throwaway busybox stage since COPY of a directory copies
contents rather than the entry itself, which would silently leave
/data at root:0755 and break every non-root run. The MINIO_USERNAME
drop-user path is not supported in this variant; use --user.
test-release.yml now builds the real Dockerfile.distroless on every
gate run (it has no download stages, so it stays offline) and asserts:
the HEALTHCHECK survives into the image config, /data ships 0777, no
shell and no /usr/bin/minio are present, Docker's health state turns
healthy from the baked probe alone, the probe binary execs without a
shell, and SIGTERM still stops the server gracefully - as root and as
--user 1001:1001. docker-release.yml gains the distroless build lanes,
multi-arch manifests, SBOM and provenance attestations, and a
release-blocking check that the pushed manifest still carries the
HEALTHCHECK (a Docker extension absent from the OCI image spec).
Verified locally on linux/arm64: full gate assertions plus bare
'docker run ... healthcheck' exit-code semantics and --version
passthrough.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add 'silo healthcheck [live|ready|cluster|cluster-read]', a thin
anonymous HTTP client for the server's own /minio/health/* endpoints,
so containers without a shell, curl, or mc can still run health
checks. Design: silo.pgsty.com/compatibility/feature/healthcheck/
The check vocabulary maps 1:1 onto the health API paths; the probe
target is derived from the server's own --address/MINIO_ADDRESS
contract with HTTPS auto-detected from the certs directory, and can
be overridden with --url. Exit codes are 0/1 only (Docker reserves 2);
diagnostics (x-minio-server-status, quorum headers) go into a single
output line for docker inspect. The request is strictly anonymous (a
credentialed request would be rejected by the reserved-path guard),
the transport bypasses HTTP_PROXY, and certificate verification is
skipped to match kubelet HTTPS probe behavior. Cluster checks default
to a 15s deadline so the server's 10s cluster_deadline can elapse.
Compatibility notes: the preserved /minio/health/* path literals and
the MINIO_ADDRESS env var are upstream wire/config surface, reused on
purpose; the rebrand-guard baseline is regenerated for the new route
literals (tests included) with zero new exported symbols. The docker
entrypoint argv translation learns the new command name.
Verified: unit tests, entrypoint tests, go vet, plus an end-to-end
run against a live server covering all four checks, --maintenance
(412), --json, usage errors, unreachable and timeout paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RPM, DEB and APK payloads carried no license material at all, and
the OCI image copied LICENSE and CREDITS but not NOTICE. For an AGPL
distribution that is not defensible packaging. Every delivery channel
now carries the same license set:
- nfpm packages install LICENSE and NOTICE under /usr/share/doc/silo/.
The entries are declared once per packager because nfpm only honors
type: license on rpm and silently drops such entries from deb and apk
-- the rpm keeps its %license flag, deb and apk carry plain files at
the same path. CREDITS stays out of the OS packages deliberately: at
1.8MB it would dominate an otherwise ~40MB payload.
- The OCI image gains /licenses/NOTICE next to the existing LICENSE and
CREDITS, in the Dockerfile and in both context-staging paths (the
make docker target and docker-release.yml).
- The GoReleaser archives list their files explicitly; the default
globs would keep missing NOTICE.
NOTICE's first line now reads 2015-2025 to match the banner constants
fixed in b57275be3 -- it had kept upstream's stale 2015-2023 claim --
and verify-rebrand.sh pins both NOTICE copyright lines.
The payload locks move in the same commit: sign-release-rpms.sh expects
the six-file listing, and the test-release validation asserts presence,
count, the rpm %license flag, and sha256 equality of the packaged
LICENSE and NOTICE against the repository files across all three
formats and both architectures.
Also dropped the unit's WorkingDirectory=/usr/local, a leftover from
upstream's /usr/local/bin install layout. Debian policy 9.1.2 keeps
packages out of /usr/local entirely; the binary already installs to
/usr/bin and the service does not depend on its working directory.
Verified locally with nfpm v2.47.0 (the CI-pinned version): built all
six packages via package-release.sh, then ran the test-release
validation step verbatim (extracted from the workflow YAML) and the
signing script's payload comparison against them in an Ubuntu 24.04
container -- both pass. Built the arm64 image from the real
Dockerfile.goreleaser and confirmed /licenses holds CREDITS, LICENSE
and NOTICE. make rebrand-guard passes with the new NOTICE pins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The shipped CREDITS predated the fork's dependency work: 38 of the
modules in the current go.mod, among them lestrrat-go/jwx, go-tpm,
go-spiffe and the charmbracelet family, had no entry at all, while
entries lingered for modules no longer in the build. An attribution
file that silently drifts from go.mod is worse than none, so it is now
generated, not curated.
buildscripts/gen-credits.sh (make credits) rebuilds the file from the
licenses of every module go list -deps reports for the main package --
the set actually linked into the silo binary, so test-only and tool
dependencies stay out. Entries keep the established name / URL /
license-text layout. New over the old file:
- Modules replaced in go.mod are annotated with the repository that
actually serves them, so the pgsty/silo-console, pgsty/mc and
pgsty/silo-pkg forks are named next to their upstream import paths.
- Bundled NOTICE files are reproduced after the license text, which
Apache License 2.0 section 4(d) requires when redistributing; 23
modules carry one.
- minio/colorjson, minio/csvparser and minio/filepath publish no
license file at all; they repackage Go standard library code and
their sources carry the Go Authors' BSD-style header, so the Go
project license is reproduced for them with a note saying why.
- The Go license text itself comes from the pinned golang.org/x/sys
module rather than GOROOT, because Homebrew's Go omits
GOROOT/LICENSE and the module copy is version-locked.
check-gen now runs the generator and fails on a CREDITS diff, the same
treatment go.mod and go.sum already get, so dependency changes cannot
leave stale attributions behind. Output is deterministic: two runs are
byte-identical, and LC_ALL=C sorting plus version-pinned inputs keep it
that way across machines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Pins pgsty/silo-console at 72fc0a5e, the v2.1.1 tag, replacing the
v2.1.0 commit from earlier today. The pseudo-version form stays because
the compatibility module path github.com/minio/console carries no /v2
suffix, so a v2.x semver tag is not directly selectable.
v2.1.1 is 5 commits over v2.1.0. The one substantive change for the
server is a fix that keeps unresolved i18n placeholders out of the
metrics dashboard legends; the rest is the v2.1.1 changelog, dropping
the container-image advertising from the Console docs, and regenerated
embedded web assets.
Verified: go mod tidy -compat=1.26 is a clean no-op afterwards, which is
what make check-gen asserts in CI, and go build ./... and go vet ./...
pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Brings the server in line with the contribution and copyright policy
already adopted in pgsty/mc.
Copyright terms were derived from the clock: startupBanner overwrote the
ldflags-injected CopyrightYear with time.Now().Year() and printed it as
the end of MinIO, Inc.'s term, so every January would have extended the
upstream copyright claim past the 2025 end of upstream development, and
release builds silently discarded the injected year. Both banners now
credit MinIO, Inc. for 2015-2025 and PGSTY from 2025 through the
release-stamped year, falling back to the current year in source builds.
NOTICE names PGSTY as the holder of the fork's modifications, matching
the banners and the packaging vendor field.
Contributions are accepted inbound=outbound under AGPL-3.0-or-later with
no CLA - the core is Copyright MinIO, Inc., so the combined work can
never be relicensed and a CLA would buy nothing. What a fork carrying a
downstream delta does need is provenance, so DCO 1.1 sign-off is now
mandatory and enforced by a workflow that rejects unsigned non-bot
commits. CONTRIBUTING documents sign-off, repair, cherry-pick provenance,
dual copyright headers, trailer preservation across squash merges, and
that assistive-tooling trailers carry no authorship or copyright claim.
verify-rebrand.sh pins the copyright split and the policy files so
neither can regress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
actions/checkout@v4 and actions/setup-go@v5 declare runs.using: node20, so
every Go CI job printed the runner's Node 20 deprecation warning. The
release, test-release and docker-release workflows carried six more node20
actions that never appeared in that warning only because those workflows
run on a tag or a dispatch rather than on every push: upload-artifact v4,
goreleaser-action v6, build-push v6 and the docker qemu/buildx/login v3
line. All of them move to the current majors, which declare node24. The
SHA-pinned rest already qualified -- attest and sbom-action are node24,
cosign-installer is composite.
Checked against what those majors removed. build-push v7 dropped
DOCKER_BUILD_NO_SUMMARY and DOCKER_BUILD_EXPORT_RETENTION_DAYS and
setup-buildx v4 dropped its deprecated inputs, none of which are used here.
checkout v7 refuses to check out fork PRs under pull_request_target and
workflow_run, and no workflow here triggers on either.
setup-go v6 exports GOTOOLCHAIN=local, the one change with teeth. go.mod
declares `go 1.26.5` and carries no toolchain directive, every job resolves
that same version through go-version-file, and nothing in the module graph
asks for more, so there is nothing left for the go command to fetch:
`go build ./...` passes under GOTOOLCHAIN=local. The tools installed from
outside the module are in range too -- nfpm v2.47.0 requires 1.26.4 and
govulncheck v1.6.0 requires 1.25.0 -- and golangci-lint arrives as a
prebuilt binary from upstream's install script, not through the toolchain.
The runner floor that comes with node24 (v2.327.1) is met by GitHub-hosted
runners, and every job here is ubuntu-latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pins pgsty/silo-console at 343d9442, the v2.1.0 tag, replacing the v2.0.0 commit
from 2026-08-04. The pseudo-version is still the form used here because the
compatibility module path github.com/minio/console carries no /v2 suffix, so a
v2.x semver tag is not directly selectable; the comment above the directive now
names the release the commit actually corresponds to.
v2.1.0 is 23 commits over v2.0.0. The substantive ones for the server are a
hand-rolled EN/ZH i18n layer with a language toggle covering login, navigation,
dashboard, buckets and object browser, identity and configuration, event
destinations, observability and admin tools; a dashboard migration to MinIO
Metrics V3; and removal of the remaining SUBNET references from health
reporting, which matches this repository's own decision to close the upstream
telemetry surfaces. The rest is UI polish and regenerated embedded web assets.
Verified: go mod tidy -compat=1.26 is a clean no-op afterwards, which is what
make check-gen asserts in CI, and go build ./... and go vet ./... pass.
Not included here: the GitHub Action version bumps and the .gitignore entry
sitting in the working tree are someone else's change and stay uncommitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit added a rename note to both READMEs and broke the gate I
added one commit earlier. That gate rejected the string pgsty/minio anywhere
outside three guard files, which was the wrong invariant: the point is that the
old name must never be a live target, not that it must never be spoken. A README
explaining where the MinIO-named artifacts went has to name them, and doing so
is the opposite of stranding a reader on a dead repository.
Split into two rules that say what is actually meant:
- No live URL may resolve to the old repository anywhere, READMEs included -
github.com/pgsty/minio and hub.docker.com/r/pgsty/minio are rejected outright.
This is the rule that protects users.
- The bare name is allowed only in five files: the pinned pre-rebrand image
digest in the upgrade test, the two guards that refuse a legacy image, and the
two READMEs.
Both halves are negative-tested: adding a github.com/pgsty/minio link to
README.md fails rule one, and adding a bare pgsty/minio to docs/docker/README.md
fails rule two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Someone arriving at main after the rename has one question the README did not
answer: where did the MinIO-named builds go. Both READMEs now say it directly —
the archived minio branch and the releases up to RELEASE.2026-08-04T00-00-00Z
hold the artifacts maintained under the original identity, and those assets and
the docker.io/pgsty/minio image stay published and unmodified.
The note also records the rename itself with its date, and repeats the one fact
that determines whether an operator has any work to do: only product and
artifact names changed. MINIO_* variables, minio_* metrics, x-minio-* headers,
/minio/* routes and the on-disk format are the same.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The transitional references land in one commit, because they are only correct
together: the repository is pgsty/silo, its default branch is main, and nothing
in the tree should point a user at the old names.
Changed:
- Workflow branch filters. go.yml and vulncheck.yml gated on `branches: master`
for both push and pull_request, so renaming the default branch would have
taken automatic CI offline with no error and no signal - the workflows would
simply never trigger again. They now name main.
- Release target. goreleaser's `release.github.name` becomes silo, which is
what actually decides where a tagged build publishes. sign-release-rpms.sh's
GH_REPO default follows.
- The OCI `image.source` label, the Helm chart `sources` entry, the security
advisory link in the issue-template config, and the go.mod comment citing the
LDAP TLS fix.
- 115 occurrences across README, README_ZH, SECURITY, CONTRIBUTING and 30 docs
pages, including 72 links that also carried the master branch in their path.
Those matter most: GitHub redirects clone, fetch, push and web URLs after a
rename, but raw.githubusercontent.com does not, and neither follows a branch
rename - every one of those links would 404 twice over.
- Three error strings in cmd/erasure-sets.go, cmd/storage-errors.go and
internal/config/errors.go that print an issue URL to operators. These are Go
string literals inside rebrand-guard's brand allowlist, so the baseline is
regenerated. The regeneration removes exactly those three entries and adds
none; all twelve other protected sets, including the 9014 exported symbols,
are byte-identical.
- The transitional-naming disclaimers in README, README_ZH, SECURITY and
CONTRIBUTING are dropped, since they no longer describe anything.
Deliberately unchanged, all three because they exist to reject or freeze the old
name rather than to point at it:
- buildscripts/minio-upgrade.sh pins pgsty/minio@sha256:b6bfe72... - the frozen
pre-rebrand image is the control group for the MinIO-to-Silo upgrade test.
- helm-migration-guard rejects any rendered container still pulling pgsty/minio.
- verify-rebrand.sh rejects the same in the delivery surfaces.
Also unchanged: docs/config/README.md links to pgsty/mc/blob/master, and that
repository's default branch really is still master. It moves when mc does.
verify-rebrand.sh gains three assertions so this cannot silently regress: no
source reference may name pgsty/minio outside the three allowlisted guards, no
link may target pgsty/silo's master branch, and go.yml and vulncheck.yml must
filter on main. Both new rejections were negative-tested - reintroducing a
master branch filter and adding a pgsty/minio URL each fail the gate with the
specific message.
This commit assumes the rename actually happens. Until the GitHub branch and
repository renames are executed, the links it introduces do not resolve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README, README_ZH, SECURITY, COMPLIANCE, CONTRIBUTING, NOTICE,
code_of_conduct, the vulnerability and PR-etiquette documents, the GitHub issue
and pull request templates, and the docs/ tree all present Silo as the product.
The Grafana dashboards under docs/metrics/prometheus/grafana/ have their panel
titles and descriptions rebranded while every minio_* query, label and
expression is left alone, so existing alerts and recording rules keep matching.
The distinction the review demanded is applied per hit rather than by
search-and-replace:
- Product and command text becomes Silo and silo: install and run instructions,
systemd examples, compose services, download links, badges.
- Protocol and interface text keeps MinIO: MINIO_* variables, minio_* metrics,
x-minio-* headers, /minio/* routes, .minio.sys, arn:minio, and API field and
error names.
- Attribution keeps MinIO and gains the fork's own: the AGPL obligations,
original copyright, CREDITS and NOTICE stay, with the modification notice
added alongside rather than replacing them.
- Historical and third-party references are left as facts, not rewritten for
brand tidiness.
README and README_ZH each carry an explicit non-affiliation notice, document
the side-by-side package migration including the
/etc/systemd/system/silo.service.d/10-legacy-user.conf drop-in for keeping a
legacy UID/GID, and state that recursive chown is never performed. The trademark
attribution uses the policy's approved "based on MinIO technology" wording, not
the shortened form the policy rejects.
github.com/pgsty/minio links are left in place and labelled transitional. The
repository has not been renamed, and rewriting them now would produce documented
URLs that 404 until the cutover; they change in the cutover commit together with
the goreleaser release target, the OCI source label and the raw-content branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test and verification scripts invoked ./minio and pulled their tooling from
upstream infrastructure with no integrity check. Every `curl | tar` of a client
or an old server binary was an unverified execution path in a script that
regularly runs as a privileged user, and several fetched a floating "latest".
Two installers replace all of it:
- install-mcli.sh resolves a pinned pgsty/mc release, downloads the archive and
its checksum manifest, requires exactly one valid manifest entry for the
asset, verifies it, and installs. MCLI_BIN with a mandatory MCLI_SHA256 lets
an offline or air-gapped run supply its own binary, still checksum-checked.
- install-verified-fixture.sh takes source, expected SHA-256 and target, and
refuses anything that does not match. Sources may be a URL or a local file.
Every script that previously downloaded mc now calls install-mcli.sh. The three
places that genuinely need an upstream artifact - the old MinIO server binary
for the LDAP IAM upgrade-import test, the 2021 mc for the three-site
replication test, and the functional-tests.sh fixture - go through
install-verified-fixture.sh with the digest recorded inline. Those dl.min.io
URLs remain on purpose: they are historical upstream artifacts needed to prove
upgrade compatibility, and they are now pinned and verified rather than
trusted.
The scripts otherwise switch to ./silo, silo.service, the silo container and
compose service names, and SILO_CONFIG_DIR. run-multi-site-minio-idp.sh is
renamed to run-multi-site-silo-idp.sh with the Makefile target following.
buildscripts/minio-upgrade.sh keeps its name and its `minio server` argv - it
exists to test the MinIO-to-Silo upgrade, so the old side must stay old - but
it is now pinned to an image digest rather than a tag, and its `docker system
prune` and `docker volume prune` calls are removed. Those ran unfiltered
against the developer's whole Docker installation; the resiliency tests had the
same problem and lose their prune and `docker ps -q` sweeps too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A rebrand of this size cannot be reviewed by reading the diff. The risk is not
what was renamed but what was renamed by accident: an environment variable, a
metric, a header, a route, a storage marker or an exported symbol that some
deployment depends on. So the invariants get a machine check.
rebrand-guard extracts the identifiers a product rename must not touch - the
go.mod module path, github.com/minio/* imports, MINIO_* variables, minio_*
metric literals, x-minio-* headers, route string literals, the hardcoded grid
routes in internal/grid/manager.go, the reserved bucket root, .minio.sys
markers, arn:minio and minio:s3 policy values, every exported symbol in cmd/
and internal/, and every branded string literal - and compares them to a
checked-in baseline. Any removal or addition fails the build, in both
directions, so a future change cannot quietly drop one either.
The recorded delta against the pre-rebrand tree is: module path, 137 imports,
19 metrics, 15 storage markers, 58 policy values and all 9014 exported symbols
byte-identical; environment 437 -> 436, headers 85 -> 84, routes 331 -> 330.
Each of those three is accounted for. MINIO_CONFIG_DIR and MINIO_VERSION were a
shell local and a compose variable in test scripts, never server settings.
x-minio-common was a docker-compose extension key, never an HTTP header.
/api/health/upload was the SUBNET upload path in the deleted callhome.go.
verify-rebrand.sh covers what the identifier guard cannot see: that the retired
upstream delivery paths stay deleted, that the packaging and container files
name silo, that silo.service keeps its Conflicts and both EnvironmentFile
lines, that silo.env contains no active assignment that would shadow the legacy
file, that no active delivery surface still publishes pgsty/minio, and that no
non-test Go source contains a dl.min.io, subnet.min.io, api.min.io,
slack.min.io or play.min.io URL.
Both, plus the entrypoint argv test, run in go.yml, release.yml and
test-release.yml. test-release.yml also gets the path filters right: it was
watching minio.service, which no longer exists, so the packaging gate would
have gone silently dormant. It now watches the unit, defaults, sysusers,
lifecycle scripts, helm chart and both guards, and asserts the four-file
payload, the absence of any cross-name Provides/Obsoletes/Conflicts, and that a
container started with the legacy `minio server` argv really runs silo as PID 1.
release.yml and docker-release.yml additionally verify that the checked-out
revision matches GITHUB_SHA before building. GitHub's OIDC certificate records
GITHUB_SHA, not the ref passed to actions/checkout, so a manual dispatch from
the wrong ref would have produced provenance describing different source from
the bytes being published. They now emit SBOMs and attestations for archives,
packages and images, and docker-release verifies the release workflow's
attestation on every artifact it consumes. Images publish to pgsty/silo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
helm/minio becomes helm/silo: chart name silo, version 6.0.0 -> 7.0.0, the
MinIO wordmark icon replaced with the project's own, image.repository and
mcImage.repository pointing at pgsty/silo, and the container command changed to
silo. User-visible titles, comments and documentation links are rebranded. The
MINIO_* environment variables and every existing values key are kept - the
first Silo chart is a rename, not a values-schema migration.
The hard problem is that a chart rename normally rewrites Kubernetes resource
identity, and a StatefulSet's selector and volumeClaimTemplate are immutable.
An existing release upgraded carelessly would either fail or orphan its PVCs.
Two things address that:
- Templates no longer derive the container name from .Chart.Name. It comes from
a helper, so nameOverride can pin it, which means an existing release can be
upgraded with nameOverride=minio, fullnameOverride=<existing-fullname> and
serviceAccount.name=minio-sa and render byte-stable identity while switching
chart and image.
- helm-migration-guard and verify-helm-migration.sh make that a gate rather
than a documented hope. The script lints the chart, renders it in distributed
and standalone modes plus the optional templates, then renders the legacy
chart from a pinned commit and the new chart with those three overrides and
compares resource identity. The guard additionally rejects any rendered
container still pulling pgsty/minio or invoking /usr/bin/minio. It runs
through a pinned alpine/helm image when helm is not installed locally, so the
gate does not depend on the developer's machine. Currently green over 7
compared resources.
Rollback is asymmetric and the README says so: the old chart with the new image
survives via the entrypoint argv shim, but the new chart with an old MinIO
image does not, because `silo server` is not a command that binary knows. Only
`helm rollback` is supported, never an image-only downgrade.
Not addressed here: the default image tag is pgsty/silo:RELEASE.2026-08-04T00-00-00Z,
which does not exist yet. The chart must not be published until the first Silo
image is pushed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dockerfile.goreleaser now copies silo to /usr/bin/silo, defaults to
CMD ["silo"], and labels the image as Silo. MINIO_UPDATE_MINISIGN_PUBKEY is
removed from the image environment: with the updater permanently disabled it
was dead weight, and leaving upstream's verification key in a Silo image
implied a trust relationship that does not exist. The MINIO_* runtime
environment variables, ports, volume and health endpoints are unchanged.
The image keeps shipping mcli with an /usr/bin/mc symlink. That is the client,
not a MinIO-branded alias for the server binary, and the Helm post-install job
and existing container scripts call it by name.
docker-entrypoint.sh translates a legacy first argument: `minio server /data`
becomes `silo server /data`, so an existing `command: minio server ...` in
compose or a Pod spec keeps working across the image swap. The translation is
argv-level only - no file named minio is installed, and an explicitly
overridden `entrypoint: /usr/bin/minio` still fails, which is the honest
outcome since that path genuinely no longer exists.
The entrypoint also fixes an unrelated startup hazard it was already carrying:
when the image runs under an arbitrary UID, HOME points at an unreadable /root
and the server probes its default config directory during initialization. It
now falls back to /tmp when HOME is unset, /root, missing or unwritable.
docker-entrypoint_test.sh pins all of it - empty argv, legacy minio, native
silo, bare flags, and an explicit shell command that must stay explicit. It is
wired into make rebrand-guard and into the go.yml, release.yml and
test-release.yml gates, so removing the shim breaks CI rather than breaking
users. Passing locally.
download-static-curl.sh gains checksum verification for the curl it fetches
into the build stage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything a user installs is renamed, and the package finally installs enough
to be startable on a clean host.
Artifact names
- goreleaser.yml: build id, binary, archive and checksum manifest become silo_*.
release.github.name stays "minio" with a comment - the GitHub repository has
not been renamed yet, and pointing at pgsty/silo before the rename would 404.
Also adds per-archive SPDX SBOMs and a keyless cosign signature over the
checksum manifest, so the signed manifest covers archives and SBOMs together.
- nfpm.yml: package name silo, and the binary moves from /usr/local/bin/minio
to /usr/bin/silo. /usr/local is not on the default PATH of a systemd unit and
is not FHS-correct for a distribution package.
- package-release.sh, sign-release-rpms.sh and verify-build-provenance.sh follow
the new names; the RPM signing script asserts NAME=silo and the new four-file
payload. nfpm is now invoked from the repository root so relative script paths
in the config resolve regardless of the caller's directory.
Package relationships are deliberately empty
No Provides, Obsoletes, Replaces or package-level Conflicts. Obsoletes: minio
cannot distinguish a pgsty package from upstream's own identically named one,
so an unattended dnf upgrade could silently swap a different vendor's product
for this one. With no relationships, both packages coexist, their file sets do
not overlap, and migration and rollback are single explicit commands. The
mutual exclusion lives in the unit instead: silo.service carries
Conflicts=minio.service plus After=minio.service.
Payload, from two files to four
- /usr/bin/silo
- /usr/lib/systemd/system/silo.service
- /etc/default/silo, installed config|noreplace
- /usr/lib/sysusers.d/silo.conf
The old package shipped a unit referencing an account nothing created, so a
clean install could not start. postinstall.sh now creates the silo system
account through systemd-sysusers, useradd or BusyBox adduser in that order and
runs daemon-reload. It never stops a service, never chowns data and never
touches /etc/default/minio. preremove.sh disables silo.service only on a real
removal - Debian "remove", RPM 0, Alpine's dotted version - so upgrades leave
the running service alone. lifecycle_test.sh exercises both against a stubbed
PATH, so a green run cannot create an account or touch the host.
silo.service reads /etc/default/minio then /etc/default/silo, in that order, so
an existing node's MINIO_* values keep working and the new file overrides them.
The packaged silo.env therefore ships comments only: any active assignment
would shadow the legacy file with an empty value.
Makefile: build/install/install-race produce ./silo, and the docker target now
assembles a context from a locally built linux binary plus Dockerfile.goreleaser
instead of the deleted Dockerfile. The hotfix, hotfix-push, docker-hotfix and
docker-hotfix-push targets are gone - they downloaded upstream's pkger, signed
with upstream's minisign key and scp'd to dl-N.minio.io. verifiers now depends
on a new rebrand-guard target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two coupled changes that must land together, because the same files carry both:
the server now identifies itself as Silo, and every path that would have called
home to a MinIO-operated service is closed.
Product identity
- build-constants.go: store name, UA name and startup banner become Silo. The
Go identifiers (MinioStoreName, MinioBannerName, ...) keep their names on
purpose - renaming exported symbols would churn the compatibility surface for
a cosmetic gain, and the rebrand guard freezes that surface.
- main.go, server-startup-msg.go, ftp-server.go and the user-visible log, help
and error strings across cmd/ and internal/ switch to Silo. Original MinIO
copyright, LICENSE, NOTICE and CREDITS are untouched; --version now prints
the upstream copyright, the pgsty modification notice, and the trademark
policy's approved "based on MinIO technology" attribution.
- api-headers.go: the HTTP Server header becomes "Silo". This is the one
externally observable identity change, so TestCommonHeadersUseSiloProductName
pins it - probes that sniff for "MinIO" must move to capability detection.
- Prometheus metric HELP strings keep their MinIO wording. They are part of the
metrics contract the guard protects, not product copy.
Configuration directory
- config-dir.go: new installs use ~/.silo. If only ~/.minio exists it is still
read, with a one-time notice and no files moved. If both exist ~/.silo wins
and an ambiguity warning is emitted; an explicit --config-dir always wins.
Covered by TestSelectDefaultConfigDir. The internal .minio.sys layout is
never renamed - this rule applies to the user config directory only.
Upstream service lockdown
- globalInplaceUpdateDisabled is now true at initialization rather than being
set from MINIO_UPDATE. common-main.go still parses MINIO_UPDATE so upgrading
nodes do not fail on an unknown key, but warns that the value is ignored;
there is no way to re-enable the updater. TestInplaceUpdateCannotBeEnabled
guards that. Without this, an admin with mc could have overwritten
/usr/bin/silo with an upstream MinIO binary.
- verifyBinary and commitBinary refuse early; the ServerUpdate v1/v2 admin
routes and the peer-rest update endpoints stay registered and keep returning
the existing programmatic error, so clients see a stable failure rather than
a 404.
- MinioReleaseBaseURL and defaultMinisignPubkey are emptied: no dl.min.io
download root, and upstream's minisign key is no longer a trust root for
anything this fork ships.
- cmd/callhome.go is deleted and internal/config/subnet/ is reduced to parsing
its old keys and reporting that the integration is disabled. config-current.go
warns instead of failing when callhome or SUBNET settings are present, so an
upgraded node with those keys still starts.
- internal/config/errors.go replaces the MinIO Slack and support entry points
with Silo documentation and issue links. Error codes and programmatic fields
are unchanged.
Verified: the compatibility baseline is unchanged except for the deliberate
removal of the /api/health/upload SUBNET route; go build, go vet and the full
cmd/ and internal/ unit suites pass; a locally built binary starts, serves
S3/Admin/metrics on the unchanged /minio/* routes, answers with Server: Silo,
and falls back to a pre-existing ~/.minio with the expected notice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tree still carried a complete second publishing pipeline inherited from
upstream that pgsty has never operated, plus MinIO-branded artwork the fork has
no right to redistribute. None of it is a pgsty release artifact, so the
history-is-immutable rule does not protect it: nothing here was ever published
under this repository's name, and git history plus the pre-cutover tag keep the
provenance.
Removed, by category:
- charts.min.io machinery: CNAME (charts.min.io), _config.yml, index.yaml,
helm-reindex.sh and 90 helm-releases/minio-*.tgz. GitHub Pages is not enabled
on this repository, so this was dormant - but enabling Pages for any reason
would have stood up a convincing impostor of the official MinIO chart
repository from our own domain.
- Dockerfiles that consume upstream binaries: Dockerfile, Dockerfile.cicd,
Dockerfile.hotfix, Dockerfile.release, Dockerfile.release.old_cpu,
Dockerfile.scratch and docker-buildx.sh. These fetch or expect a MinIO-built
binary; a Silo image must only ever contain an artifact this repository built
from this commit. Dockerfile.goreleaser, the one the release workflow
actually uses, stays.
- buildscripts/upgrade-tests/ (compose.yml, minio.env, nginx.conf), which
pinned minio/minio images through a MINIO_VERSION compose variable. The
upgrade test is rebuilt on image digests in a later commit.
- docs/hotfixes.md, describing an upstream hotfix distribution channel that
does not exist for this fork.
- MinIO trademark artwork: .github/logo.svg (a recoloured MinIO wordmark), the
docs/screenshots/ set, the Grafana dashboard screenshots and
docs/federation/lookup/bucket-lookup.png. The dashboards themselves are kept
and rebranded separately; only the images carrying MinIO chrome go.
Deliberately not touched: published releases, tags, image digests and package
signatures remain byte-identical. This commit only removes paths that were
never part of a pgsty release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release image (Dockerfile.goreleaser) was only ever built by
docker-release.yml, which is workflow_dispatch only - so the runtime layer and
entrypoint were never exercised by CI until an actual publish, where a broken
COPY path or a signal-handling regression would surface at the worst possible
moment. MC already covers this in its test-release; the server did not.
Add an offline, deterministic smoke: assemble a minimal image from the
linux/amd64 binary goreleaser already built and the real entrypoint, then start
the server and `docker stop` it on both the default and the MINIO_USERNAME
drop-privilege paths, asserting PID 1 is minio, the exit is 0, the shutdown is
sub-timeout, and the "Exiting on signal" log is present. The mcli-download build
stage is skipped deliberately - it hits the GitHub API and would make this gate
flaky and non-reproducible; its simple, checksum-guarded logic still runs at
publish time.
This is the regression guard for the exec-into-chroot entrypoint fix. Also add
dockerscripts/docker-entrypoint.sh to the pull_request paths so an entrypoint
change actually triggers this pipeline. Verified locally end to end: the fixed
entrypoint passes both paths; reverting the exec makes the drop-privilege path
time out to exit 137 and the step fails.
Co-authored-by: Claude <noreply@anthropic.com>
Three gate-correctness fixes:
- lint: `command typos && typos ./ || echo skipping` ran typos twice (POSIX
`command` executes it) and, via `&& ... || echo`, turned a real typo finding
(nonzero exit) into the "not installed" message with exit 0 - so spelling
issues could never fail the gate. Use `command -v` to test presence and run
typos once, letting its findings surface.
- getdeps: `curl -sSfL ... | sh` took the pipeline's exit from sh, and make's
/bin/sh has no pipefail, so a failed or partial download of the golangci-lint
installer was reported as success and a truncated script could run. Download
to a temp file under `set -e` (with a trap to clean it up) and execute that,
so a curl failure aborts the target. Verified: a 404 now exits nonzero
instead of silently continuing.
- release: workflow_dispatch checked out github.ref (the branch it ran from)
while naming artifacts after the input tag, so a release could be built from
one ref and published under another. Pin checkout to the requested tag.
Not addressed here: the server's release container image is still only built
by the publish-time docker-release workflow, never in CI (MC covers this in
its test-release). Flagged for a separate decision.
Co-authored-by: Claude <noreply@anthropic.com>
nFPM placed minio.service at /lib/systemd/system. On merged-usr EL/Fedora the
file lands correctly through the symlink, but the RPM database records /lib/...,
so `rpm -qf` disowns it and - more importantly - systemd's
%transfiletriggerin watches /usr/lib/systemd/system, so the path recorded as
/lib/... never fires the automatic `daemon-reload` on install or removal. Move
the unit to the canonical /usr/lib/systemd/system, which both restores that
trigger and makes the package own the path it ships.
The destination is asserted verbatim in several places, all updated in lockstep
so the packaging gate still passes: the six name/payload/sha checks in
test-release.yml (rpm/deb/apk) and expected_payload in sign-release-rpms.sh.
Verified by building a real deb with the new config: the unit is at
./usr/lib/systemd/system/minio.service.
Also drop the `# Built for ${project.name}...` line from minio.service. nFPM
expands variables in content src paths, not file bodies, so that pkger-era
placeholder was being written verbatim into every installed unit.
Not addressed here, deliberately: the package still does not create the
minio-user account the unit references. That matches upstream MinIO's own
packages (both rely on the documented manual useradd) and is not a fork
regression, so it stays a documentation step rather than a scriptlet.
Co-authored-by: Claude <noreply@anthropic.com>
A gitignored .goreleaser.yml sat in the repo root, so a bare `goreleaser`
invocation - the default discovery path - picked it up instead of the
.github/goreleaser.yml that CI passes with --config. The two had diverged
completely: the shadow published the GitHub release directly (draft: false),
pushed pgsty/minio:latest from a dockers section, and stamped a different
vendor and artifact naming. Since RPM signing runs on the maintainer's own
machine, that shadow config was one stray `goreleaser release` away from
publishing a mislabelled release and overwriting the Docker latest tag.
Delete the file and drop its .gitignore entry so any future .goreleaser.yml
shows up as untracked and is caught by the release workflow's clean-checkout
check, rather than silently steering local builds. CI is unaffected: it names
its config explicitly. (A copy of the removed file is preserved out of tree
for reference.)
Co-authored-by: Claude <noreply@anthropic.com>
gen-ldflags injected -X cmd.GOPATH / cmd.GOROOT from the builder's environment,
baking absolute paths like /Users/<user>/go into every released binary. That
defeats -trimpath and makes the build unreproducible: a third party rebuilding
the same tag gets different bytes and cannot verify checksums.txt.
The values only seed logger.Init's source-path trim list, and under -trimpath
the binary's paths are already relative, so there is no build-machine prefix
left to trim - the trim list also still gets runtime.GOROOT() and
build.Default.GOPATH at run time. Dropping the two stamps changes no observable
logging behaviour; cmd.GOPATH/GOROOT keep the empty defaults a plain go build
leaves.
Verified: gen-ldflags output no longer contains cmd.GOPATH/GOROOT; a
-trimpath release build has zero occurrences of the builder path (was 1);
Version, ReleaseTag and CommitID stamps are intact.
Co-authored-by: Claude <noreply@anthropic.com>
The compute step spliced ${{ github.event.inputs.tag }} straight into the
run: body, so a dispatch tag containing shell metacharacters was parsed as
script - arbitrary code in a job holding a contents:write token. Pass the
input through the environment instead (INPUT_TAG), the same shape
docker-release.yml already uses, so the value reaches bash as data.
Second vector: the old format check compared the sed output against the input
to decide validity, and sed anchors ^...$ per line. A tag carrying a newline
passed the check on its first line and the remaining lines flowed into
$GITHUB_ENV, setting arbitrary variables (PATH, LD_PRELOAD, ...) for every
later step. Replace it with a bash =~ whitelist anchored to the whole string,
which rejects any multi-line value up front.
Verified: a legal RELEASE.* tag is accepted and yields the right PKG_VERSION;
a `"; touch ...; #` injection and a newline-carrying PATH/LD_PRELOAD payload
are both rejected; an empty input falls back to the git ref name.
Co-authored-by: Claude <noreply@anthropic.com>
The crosscompile gate built 15 OS/arch combinations, ten of which we never
ship (ppc64le, mips64, s390x, mips, riscv64, 386, arm, freebsd, netbsd,
openbsd). On a cold CI cache each target costs ~80s, so the full set ran ~21
min and tripped the job's 20-min timeout, cancelling the only CI run for the
release HEAD. Trim the list to the exact goos/goarch matrix the release
actually produces (see .github/goreleaser.yml): linux, darwin, windows on
amd64 and arm64.
This also closes a coverage gap: windows/arm64 is published but was not being
compile-checked. The trimmed set builds all six in well under the timeout.
Note: netbsd is no longer compile-checked here, so the go-systemd v22.6.0 pin
(kept because v22.7.0 does not build on netbsd) loses its CI guard. The pin is
retained deliberately - it is harmless on the platforms we ship and upgrading
go-systemd is a separate decision - but a netbsd regression in a dependency
would now surface only if that build is exercised out of band.
Co-authored-by: Claude <noreply@anthropic.com>
The two chroot branches that drop privileges when MINIO_USERNAME/GROUPNAME
(and optionally MINIO_UID/GID) are set ran chroot as a child of the entry
shell, leaving the shell as PID 1. A SIGTERM from `docker stop` or an
orchestrator then went to the shell, which does not forward it, so MinIO was
never asked to shut down and was killed after the stop timeout (exit 137) with
no "Exiting on signal" log - risking in-flight requests and data at the flush
boundary. The default branch already exec's; these two now do too, so MinIO
runs as PID 1 and receives the signal directly.
Verified in a faithful reproduction of the release runtime layer: all three
paths (default, USERNAME only, USERNAME+UID/GID) now stop in ~0.2s with exit 0
and log the graceful shutdown, where the two drop-privilege paths previously
timed out to exit 137.
Co-authored-by: Claude <noreply@anthropic.com>
Make the repository's actual quality contract visible in CI. Check formatting, build and vet the tree, run cmd and internal tests, enforce pinned lint and generated-file cleanliness, exercise S3 Select under the race detector, and cross-compile every supported target.
Keep each concern in a separate job so failures identify the missing guarantee instead of hiding behind a single aggregate test result.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
SelectResults spawns a parser goroutine that drains and closes the
response body when the stream ends; deferring res.Close() had the test
drain and close the same bytes.Reader concurrently, which the race
detector catches reliably. Give the test body a close signal and wait
for the parser to finish instead of competing with it.
The double-close lives in minio-go's client parser, which no server
code path uses; it remains worth an upstream report.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Expand the generated-file gate beyond *_gen.go and go.sum to cover generated tests, msgp output, stringer files, go.mod, and untracked generated artifacts.
Regenerate the 19 stale stringer outputs with the go.mod-pinned x/tools version. Runtime String output is unchanged, and a repeated generation pass now leaves the tree byte-identical.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Installing golangci-lint from the moving master branch made local and CI results depend on the day they ran. Pin v2.11.3 and install it into a versioned path so version changes cannot accidentally reuse a stale binary.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
The release workflow built and pushed the container images in the same
run that created the draft, so pgsty/minio:latest moved to a build
nobody had signed or published yet. Abandoning that draft left latest
pointing at it with no way to notice, and the images were the one
artifact of a release that escaped the draft gate entirely.
Image publishing moves to its own dispatch-triggered workflow that
takes a tag and refuses to touch anything that is not a published,
non-prerelease release, and that is not the latest one - since it moves
the latest image tag, running it for an older release would silently
roll users back. It builds from the archives attached to that release,
checked against the published checksums, rather than rebuilding from
source: the image then contains the same binary the tarball does, by
construction rather than by assumption.
GoReleaser loses its dockers and docker_manifests sections along with
the QEMU, buildx and registry-login steps that only existed to serve
them, and the release job drops packages: write, which was granted for
a GHCR push that never happened.
Releasing now has one more manual step. That is the point: tagging no
longer moves docker latest, so the tag can be cut, inspected, signed
and published before anything reaches users who pull by tag.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Refuse dirty release checkouts and verify that every GoReleaser binary records the tagged revision with vcs.modified=false before packaging or publication.
Exercise the complete nFPM output in the release test pipeline: assert the six public names, validate every checksum and package identity field, and prove that RPM, DEB, and APK payloads contain the exact source binary and systemd unit. Validate release scripts and make their identity expectations the single source of truth.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace minio/pkger's hard-coded upstream identity with an in-tree nFPM configuration. Packages now name PGSTY as vendor and maintainer, use the SILO homepage and SPDX license, and preserve the established package names, versions, payload paths, modes, and checksum format.
Resolve both the release binary and systemd unit independently of the caller's working directory. The release and test workflows share the same package script, while the signing script consumes the same final metadata contract.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
GitHub Actions cannot hold the Pigsty RPM signing key, so the release workflow must stop before publication. Create releases as append-only drafts, refuse to overwrite existing assets, and make publication an explicit action after local signing and review.
Add a maintainer-side signing command that downloads the two RPMs, verifies checksums and package identity, signs them with the Pigsty key, regenerates checksums, and only replaces draft assets when --upload is explicitly requested. Published releases are never modified.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Consolidate the release-cycle security ledger after the implementation commits are stable. Document internode containment, policy-condition source hardening, the opt-in trusted-proxy boundary, and the bucket/object authorization tightening with fork-local identifiers where no CVE exists.
Use silo-pkg v3.11.0 as the maintained dependency reference, explain compatibility and migration behavior, and link each entry to the rewritten commit that actually carries the fix.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Keep unpublished security working notes, local agent instructions, and maintainer scratch material out of the public source tree. Already tracked security documentation remains tracked; the ignore rules only prevent accidental additions.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Point both the server and bundled client jobs at the release tag being cut, keep chart appVersion aligned, and retain the maintained pgsty/minio image selected by the public chart baseline.
Stop creating the console/console123 administrator by default. Leave an explicit change-me example while rendering no user-creation job unless an operator opts in. Chart packaging and repository indexing remain separate release steps.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
ReadParts built its disk-health trace path from partMetaPaths[0] with
no length check, so a caller passing no paths at all indexed an empty
slice. xlStorage.ReadParts handles an empty list perfectly well - it
returns an empty result - so the panic came entirely from the metrics
bookkeeping wrapped around it.
The reachable caller is ReadPartsHandler, which decodes its path list
from a msgpack request body that an authenticated peer controls, and
neither the handler, the storage-REST client, nor the path guard
rejects an empty one: guardPaths ranges over the slice, so an empty
slice passes vacuously.
net/http recovers a panicking handler, so this is not a crash - which
is what makes it worth fixing rather than merely tidy. ReadPartsHandler
calls keepHTTPResponseAlive before it calls ReadParts, and that helper
spawns a goroutine whose only exit is receiving from the channel done()
writes. Panicking in between skips both done(err) and done(nil), so the
process survives and the keep-alive goroutine and its ten-second ticker
stay parked forever - one per request, driven by a request body the
caller chooses. Repeating one malformed frame exhausts the node.
The fix follows DeleteVersions in the same decorator, which already
guards the identical "merely for tracing" lookup; ReadParts was the one
method missing the pattern. It also covers the second entry point, the
per-disk errgroup in readParts, where a panic has no recover at all and
would take the process down. That path is screened at the S3 boundary
today, so this is defence in depth there.
No error is returned for the empty case. The storage layer's answer to
an empty list is an empty result, and turning that into an error would
be a behaviour change on a path that is merely degenerate.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
joinErrs ranged over its own empty accumulator string instead of the
errs slice, so the loop body never executed and the function
unconditionally returned "". Its only caller feeds the merrs tag of the
DeleteDanglingObject audit event, so every dangling deletion was
recorded without the per-drive metadata errors: the record showed what
was deleted but not which drives errored or why quorum was lost.
Range over errs instead. The existing separator logic is already right
once the loop runs, since a leading nil error appends "<nil>" and every
later element gets its comma. Upstream's open minio/minio#21580 fixes
the same bug with a strings.Builder rewrite, not taken here: the
function runs once per dangling deletion over a drive-count-sized
slice, and the one-word change is the entire defect.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
The discrete PostgreSQL connection path concatenated raw values into a libpq keyword/value string. Whitespace, quotes, or backslashes could split a value into additional parameters or make an otherwise valid configuration fail to parse; the path also used the unsupported keyword username instead of user.
Render every generated value as a single-quoted libpq parameter, escape quotes and backslashes, and use the correct user key. Keep the existing connection_string form untouched. The earlier attempt to register migrated PostgreSQL and MySQL fields is deliberately absent because those key names collide with the legacy connection-string tokenizer.
Focused tests cover ordinary values, whitespace, quotes, backslashes, and parameter-shaped input.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
GetNotifyNATS reads user_credentials, nkey_seed and tls_handshake_first
and GetNotifyAMQP reads immediate, but none of them were registered in
DefaultNATSKVS/DefaultAMQPKVS or the help schema, so CheckValidKeys
rejected any enable=on target carrying them. Worse, the legacy config
migration wrote exactly these keys - including the env var name
MINIO_NOTIFY_NATS_USER_CREDENTIALS used as a config key, because the
NATSUserCredentials constant doubled as both - so a migrated NATS config
failed validation on every load, and the FetchEnabledTargets fail-fast
then silently disabled all bucket notification targets.
- Register user_credentials/nkey_seed/tls_handshake_first (NATS) and
immediate (AMQP) in the default KVS and help schema; split
NATSUserCredentials into a real config key plus EnvNATSUserCredentials
(all env var names byte-stable)
- Fix legacy migration: SetNotifyNATS writes the proper key;
SetNotifyAMQP no longer writes cfg.Immediate under the internal key
and now carries both immediate and internal
- Tolerate the legacy MINIO_NOTIFY_NATS_USER_CREDENTIALS key written by
pre-fix migrations (NATS-scoped, load path only) with fallback read;
env > user_credentials > legacy key
- Print key names only, never values, in the invalid-keys error of both
CheckValidKeys forms; rejected values can carry credentials
- Add an AST-based audit test asserting parser reads, migration writes
and help entries stay within the registered key set for all ten notify
subsystems, with floor assertions so collector drift fails loudly
- Document (unchanged) FetchEnabledTargets fail-fast and pin it with a
characterization test
Known same-class gap left in place and pinned by the audit's allowlist:
SetNotifyPostgres/SetNotifyMySQL write five unregistered DSN-era keys;
tracked for a follow-up issue.
Closes#39
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
The address MinIO attributes a request to is read from X-Forwarded-For,
X-Real-IP or RFC 7239 Forwarded, and never from the connection unless all
three are absent. It becomes aws:SourceIp and the audit remotehost field,
so any client that can reach the API port chooses the value an IpAddress
condition is evaluated against and the address every logged action is
attributed to.
MINIO_API_TRUSTED_PROXIES now selects who may make that claim:
unset the historical behaviour, unchanged
none no forwarded header is believed; the TCP peer wins
<CIDRs> believed only from listed peers, chains read right-to-left
Reading right-to-left is what makes an appending proxy safe: each hop
appends the peer it actually saw, so an entry a client injected can only
sit to the left of one a proxy wrote. The stock nginx recipe
$proxy_add_x_forwarded_for appends, which leaves the client's entry
left-most - exactly where the untrusted path reads - so a deployment with
no direct route to the API port was forgeable too.
_MINIO_API_XFF_HEADER is deliberately untouched, in semantics and in read
timing. Widening it to mean "trust nothing" was implemented and reverted:
it is the only part of this change that could alter a deployed
configuration, and the new variable expresses the same guarantee at no
compatibility cost. Upstream's TestXFFDisabled is retained verbatim.
Notes on the allow-list mode, all covered by tests:
- it must name proxies, not the subnet they sit in; listed entries are
skipped while walking, so a range covering clients lets them forge
- a cluster must list its own nodes, because MinIO forwards between
them and a client can force a hop via the ListObjectsV2 token
- loopback is trusted as a peer, not as a chain entry, so FTP and SFTP
keep attributing their sessions
- the node-to-node forwarder drops X-Real-IP and Forwarded from a peer
not entitled to have set them
- the walk scans the header in place and stops after 100 hops, so a
long chain costs neither allocation nor unbounded work
No behaviour change for any deployment that does not set the new
variable: the untrusted path is a verbatim copy of the previous function
body, differentially verified against it over ~5.1M header combinations.
The LDAP STS allow-list now shares the list parser as pure code motion,
verified identical across every combination of 37 allow-list values and
21 peer addresses.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
A bucket policy that allows s3:DeleteObject only when s3:versionid is null
-- Condition {"Null": {"s3:versionid": "true"}}, the idiom for "let clients
delete current objects but not roll back versions" -- denied every delete,
including the version-less ones it was meant to permit (upstream issue
minio/minio#21735).
getConditionValues wrote "versionid": {""} unconditionally. The condition
engine decides Null by slice length (nullfunc.evaluate), so a present-but-
empty value reads as "key present": Null:true never matched and Null:false
always did. Absent and empty were indistinguishable.
Writing the key only when the request names a version fixes the reported
case but, alone, opens a worse one. DeleteObjects carries each object's
version in the XML body, which getConditionValues -- reading only r.Form --
never sees. A body version would then vanish from the map, read as null,
and a policy meant to protect old versions would authorize deleting a
specific one. So authorization also rebinds versionid to the effective,
server-resolved reqInfo.VersionID for DeleteObjectAction: the per-entry
body value that checkRequestAuthTypeWithVID already sets in the
DeleteObjects loop, deleting the key when that value is empty. A
query-level ?versionId on a DeleteObjects POST no longer leaks into any
entry's decision.
Finally, trim the version the condition builder reads. newContext and
getOpts both TrimSpace it before the object layer acts, so an untrimmed
value here let a padded ?versionId=V%20 present a different s3:versionid
than the version actually operated on, sidestepping a Deny keyed on
StringEquals s3:versionid. DeleteObjectAction was already immune via the
trimmed reqInfo value; this covers GetObject, tagging, retention, and the
copy-source read.
Tests: an end-to-end DeleteObjects against a Null:{s3:versionid:true}
policy over versioned objects (with a decoy query versionId proving the
per-entry body value wins), and a unit test asserting key presence,
trimming, and the copy-source fallback.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Policy evaluation mixed server-derived identity and transport values with raw headers and query parameters. A client could therefore shadow internal condition keys, synthesize LDAP or JWT resource variables, substitute request tags for stored tags, or make a condition observe a value different from the one the handler actually used.
Partition condition sources, reserve internal names, adopt exact-name lookup from silo-pkg, and bind authorization to the effective request state. Preserve compatible query forms for storage class and upload tags with explicit header precedence, while restricting signature age and existing-object tags to authenticated or server-resolved values.
Tests sweep every supported key across header and query routes and exercise LDAP/OIDC variables, object-lock spelling, STS tags, metadata extraction, and end-to-end policy decisions.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Upgrade directly to silo-pkg v3.11.0, the fork's first non-colliding release, and carry the completed minio/minio#20449 hardening without exposing the retired v3.7.0, v3.8.0, or v3.8.1 tags in the rewritten history.
Twelve sensitive bucket-level writes now require the bare bucket resource in addition to the historical bucket/ form, so an object-only bucket/* grant cannot delete a bucket or change protections. Read/list behavior, ordinary tenant self-service, Deny statements, and NotResource exclusions retain their compatible behavior; MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH restores the old matcher when migration requires it.
End-to-end tests cover direct clients, session policies, service accounts, wildcard edge cases, compatible resource pairs, and real bucket deletion.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
sort.SliceIsSorted with a strict '<' predicate treats equal neighbours
as sorted, so a completion list like [1,1] passed the order check and
the same part was assembled into the object twice: a single uploaded
5 MiB part produced a 10 MiB object. Replace the check with an explicit
strictly-increasing scan that rejects repeats with InvalidPartOrder
before anything is assembled. Gaps and lists not starting at part 1
remain legal, matching AWS semantics.
The regression test drives the real CompleteMultipartUpload handler on
both Erasure backends and asserts that rejected completions leave no
object behind and keep the upload retryable.
Closes#49
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
httpStreamResponse, streamHTTPResponse, waitForHTTPStream, and their
8k buffer pool lost their last caller when ReadMultipleHandler went
away (73ac52472). Nothing references them anywhere in the tree, and
the storage REST wire surface is untouched - these encoded a framing
no registered handler still speaks.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
ReadPartsHandler completed its keepalive stream before reporting storage failures, then tried to write an ordinary error response after the body was already owned. Clients consequently decoded the error text as msgpack and lost the real failure.
Send failures through the keepalive completion channel and mark success only after ReadParts returns cleanly, preserving the existing wire framing and error identity.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
AppendFile, DeleteVersions, and ReadFile sized memory directly from peer-controlled Content-Length or query parameters. Tiny requests could therefore reserve gigabytes before delivering a body, while negative declarations could reach make and panic.
Cap preallocation without capping accepted append bodies, grow version slices as entries decode, reject negative counts, and enforce the format-implied 5 GiB ceiling on legacy whole-file reads. Regression tests drive raw handler inputs, measure total allocations, and retain legitimate round trips.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Malformed erasure layouts can divide by zero, while negative part sizes collapse expected shard sizes to zero and make truncated data appear healthy. Boundary validation alone is insufficient because poisoned metadata may already exist on disk or arrive through local heal paths.
Reject non-positive block sizes at the sole Erasure constructor, guard the metadata arithmetic helpers and rebalance calculation, refuse negative part sizes before persistence, and make CheckParts and VerifyFile reject previously stored poison. Tests cover both shard-size implementations, construction, persistence, local verification, and the wire boundary.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Storage REST request bodies and Grid RPC frames bypass the HTTP validity middleware, allowing wire-supplied paths and malformed FileInfo values to reach xlStorage unchecked.
Wrap the remotely exposed StorageAPI with a guard that covers every path-bearing method, including nested metadata fields. Reject traversal and destructive volume-root aliases before path cleaning can erase them, and validate erasure geometry and part sizes at the same wire boundary. Keep a raw-volume check in getVolDir for peer-S3 calls that bypass the wrapper.
Reflection, fuzz, traversal, peer-S3, compatibility, and malformed-erasure tests pin the complete method surface and prove that legal object names remain accepted.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
WithDeadline runs its work function on a goroutine of its own, so a panic
inside it is reachable by no recover() the caller can install: net/http
and internal/grid each recover only on the goroutine they own. Left
unhandled it terminates the process, which turns any malformed internode
payload that trips a bug in work() into a remote node kill.
Eleven call sites run wire-derived storage work through this path. The
project already recovers panics at both request boundaries; this extends
the same policy to the one goroutine those recovers cannot reach.
Stack dumps are capped over the process lifetime. The panic is by
definition reachable from untrusted input, so writing a full stack per
occurrence would trade a node kill for unbounded log amplification, one
small request each. The error is returned every time regardless, so the
caller's own rate-limited logging still sees each occurrence.
Inherited from upstream; the fork added no lines to this path.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Mark trackingResponseWriter committed when Write or an effective Flush implicitly sends a 200 response. This keeps duplicate-response detection aligned with the actual writer chain while preserving no-op Flush behavior when unsupported.
Add direct and gzip-streaming regression coverage for implicit headers, Flush delegation, and suppression of a second response.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace bytebufferpool-backed return values with bytes.Buffer storage so
the downloaded and compressed slices remain valid after downloadBinary
returns. Close the zstd encoder on copy failure and propagate final close
errors.
Add round-trip and pool-reuse regression coverage for both returned
buffers.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Wire the preallocated pooled shard slices into parallelReader instead
of discarding them and allocating a buffer for each disk.
Keep readerToBuf as a permutation while preferred readers are reordered.
The former assignments could duplicate a buffer slot after multiple
swaps, causing concurrent writes and a possible decode stall. Add pool
aliasing and mapping regression coverage.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
hash.Checksum.AddPart returned before seeding the accumulator when the
part was empty, so a multipart object with no content at all ended up
with no checksum instead of the checksum of zero bytes. Completing such
an upload failed with XAmzContentChecksumMismatch when the client
supplied the correct object checksum, and stored an empty checksum when
it did not.
Run the type check and the first checksum seeding before the zero size
early return. Appending zero bytes still leaves an existing accumulator
unchanged, so only the all empty case changes: a zero length part
followed by content already merged correctly, because prepending no bytes
does not alter a CRC.
AddPart has a single production caller, the multipart completion path, and
its part checksum type is derived from the upload's own checksum type, so
the type check now reached for zero sized parts cannot fire there.
Add a table test over CRC32, CRC32C and CRC64NVME covering every position
an empty part can take, and an API level zero length full object upload
that exercises the persisted AppendTo/ReadCheckSums round trip.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
CompleteMultipartUpload compared the per-part checksum taken from the
request body against the stored part checksum unconditionally, so a
client that sent only PartNumber and ETag for each part failed with
InvalidPart.
AWS S3 requires part level checksums in the completion body only for
composite checksum types. For full object types the client sends the
object level checksum in the request headers instead and does not retain
per-part values - that is the point of FULL_OBJECT. Reproduced with
boto3 1.43.58: it puts x-amz-checksum-crc32 on each UploadPart and the
full object checksum on the completion headers, but emits only ETag and
PartNumber in the completion body. A caller could only get such an upload
through by collecting the per-part checksums from the UploadPart
responses and echoing them back, which is exactly the bookkeeping
FULL_OBJECT exists to avoid and which no off-the-shelf SDK call does.
minio-go does echo them, which is why mc never hit this.
Treat the part checksum as optional when the upload declared a full
object checksum type and the client sent no part checksum at all. A part
carrying any checksum is still validated against the stored one -
including one sent under the wrong algorithm, which cannot match and is
rejected - composite uploads keep requiring a checksum for every part,
and the merged object checksum is still computed from the server stored,
upload time validated part checksums, never from client supplied values,
so integrity is unchanged.
Covered by API level tests over CRC32, CRC32C and CRC64NVME on both the
single drive and erasure backends, with guards for a wrong object
checksum, a wrong part checksum, a part checksum under another algorithm,
a mix of present and omitted part checksums, an absent object checksum,
and composite uploads still requiring every part checksum.
Fixes#31
Reported-by: Christophe Bornet <cbornet@users.noreply.github.com>
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace Georg Mangold's console fork (v1.9.1) with pgsty/silo-console
at the v2.0.0 release, Pigsty's maintained console carrying the SILO
identity, the redesigned and hardened web app, and regenerated embedded
assets. The fork keeps the upstream github.com/minio/console module
path without a /v2 suffix, so the replace pins v2.0.0's tagged commit
(b952a120) as a pseudo-version rather than the tag itself.
highwayhash v1.0.4 and go-m1cpu v0.2.2 follow from the new console's
requirements; the previous console resolved a v1.0.4 pre-release and
v0.2.1. Full-tree build verified against the new pin.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace github.com/minio/mc with the pinned pgsty/mc 2026-08-01 release while preserving the upstream module identity. Refresh the module graph, including the fork-required etcd 3.6.9 patch update.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Move MinIO from v1.18.6 to v1.18.7 to pick up the GO-2026-5841
fix. The vulnerable dictionary symbols are not reachable in this tree,
but keeping the direct compression dependency patched avoids carrying
the affected release.
Verified with go mod verify, full go test and go vet runs, and
govulncheck reporting no reachable symbol or imported-package
vulnerabilities.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
thrift 0.23.0 compares an int against math.MaxUint32, which does not
compile on 32-bit platforms and has broken linux/arm and linux/386
builds since the 2026-06-18 dependency refresh; 0.24.0 carries the
upstream fix. The library only reaches us through fraugster/parquet-go
for S3 Select, whose decode tests still pass.
go-systemd v22.7.0 moved CLOCK_MONOTONIC into a file built for every
unix platform and no longer compiles on NetBSD. Pin v22.6.0 with a
replace directive - a plain require cannot hold because Console pulls
v22.7.0 back in through MVS - until upstream ships the fix. The server
only consumes daemon.SdNotify, which both versions provide.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Require Go 1.26.5, gRPC 1.82.1, and x/text 0.39.0 while retaining the dependency graph's existing pins wherever MVS permits. Restore the blocking govulncheck job and correct the documented security advisories.
Use the maintained Silo image for both the server and post-install mc jobs. The image bundles mcli with an mc compatibility link.
Bump the chart major version because changing the default registry can affect admission and pull policies. Keep Helm repository packaging and indexes outside this source-chart change.
Replace upstream docs.min.io references with the SILO documentation domain and remove the upstream community path prefix.
Co-authored-by: waterkip <6317502+waterkip@users.noreply.github.com>
Update the module Go directive and release Docker build images from Go 1.26.2 to Go 1.26.4 so local, CI, hotfix, and release builds use the same patched toolchain.
Keep ordinary Go module requirements and replacements unchanged; this intentionally avoids a third-party dependency refresh while allowing container system packages to refresh through the newer golang Alpine base image and existing apk resolution.
Update the security advisory index to record the Go 1.26.4 toolchain bump alongside the earlier Go 1.26.2 security update.
Verified with go build ./..., go vet ./cmd/, and focused cmd tests. go mod tidy -diff was attempted as a read-only dependency drift check but could not complete because proxy.golang.org timed out while fetching uncached transitive test modules.
Co-authored-by: Codex <codex@openai.com>
The internode storage-REST ReadMultiple endpoint (/rmpl) joined
attacker-controlled Bucket/Prefix/Files into a filesystem path with no
validation, letting a peer with internode credentials read files outside
the drive root (GHSA-xh8f-g2qw-gcm7).
ReadMultiple has had no production caller since upstream #20390 removed
the last one (listParts) in Sep 2024; multipart now uses ReadParts (/rps).
Following the upstream fix, remove the whole API instead of validating
paths: route constant and registration, server handler, REST client
wrapper, the StorageAPI/xlStorage/xlStorageDiskIDCheck methods, the
storageMetricReadMultiple metric, and the ReadMultipleReq/Resp datatypes.
Regenerated the msgp and stringer outputs; storageRESTVersion stays at v63.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Route JSON Lines through the bounded PReader path so oversized records are rejected consistently instead of bypassing the limit on SIMD-capable CPUs.
Preserve S3 Select error codes in stream error events, wrap JSON parser errors as JSONParsingError, and flush completed records before returning a terminal error event. Add regression coverage for oversized JSON Lines input and error code preservation.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Remove the per-username LDAP STS throttle bucket and keep the limiter keyed only by source IP. A username bucket is shared across all clients and lets one source keep a known account's bucket drained with bad-password attempts, locking the legitimate user out before LDAP bind.
For trusted proxies, stop trusting the left-most forwarded address. Resolve X-Forwarded-For right-to-left, skip trusted proxy hops, reject catch-all trusted-proxy CIDRs, and intentionally ignore RFC 7239 Forwarded for this security-sensitive bucket. Document that X-Real-IP is trusted verbatim and must be overwritten by the trusted proxy, not passed through from clients.
Update focused limiter, source-IP, trusted-proxy, and LDAP config tests to cover source-only buckets, spoofed appended XFF, multi-hop trusted proxies, Forwarded fallback, catch-all rejection, and the X-Real-IP deployment contract.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Allows the trackingResponseWriter to properly proxy flush
requests to the underlying http.Flusher. This ensures
compatibility with streaming responses.
Update the STS, security, select, and Docker documentation to reflect the recent hardening work, including LDAP STS throttling details, OIDC JWT verification changes, and the new pgsty-specific security policy and advisory index.
Rewrite repository and raw-document links that still pointed at minio/minio so the docs consistently reference pgsty/minio instead.
The core idea is to keep the documentation aligned with the fork's actual security behavior, ownership, and upgrade guidance without mixing in unrelated code changes.
Prevent LDAP STS reservation cancel paths from over-crediting rate-limit buckets by capping refill and refund capacity against in-flight reservations.
Add an explicit trusted-proxy allowlist for LDAP STS source bucketing, prefer clean X-Real-IP values on trusted peers, and extend tests/docs for the new behavior.
Use the socket peer address for LDAP STS per-IP rate limiting instead of the generic forwarded-header-aware helper. This keeps the security-sensitive rate-limit key from trusting spoofable X-Forwarded-For, X-Real-IP, and Forwarded headers while leaving the rest of the source-IP behavior unchanged.
Add focused regression coverage for RemoteAddr parsing, header spoofing, and peer-address bucket selection.
Update go.mod and all golang build images from 1.26.1 to 1.26.2 to pick up the upstream 2026-04-07 security release.
This includes fixes for CVE-2026-32280 / CVE-2026-32281 in crypto/x509, CVE-2026-32283 in crypto/tls, and the related toolchain and standard library security fixes shipped in go1.26.2, without changing any unrelated dependencies.
Track issue #28 / GHSA-9c4q-hq6p-c237 as fake CVE-2026-40028. Close the Snowball auto-extract auth gap in PutObjectExtractHandler by treating authTypeStreamingUnsignedTrailer the same as ordinary PUTs: honor X-Amz-Decoded-Content-Length, initialize newUnsignedV4ChunkedReader(), and verify the SigV4 request before any tar bytes reach untar(). This removes the forged-signature write primitive that let a single request fan out into arbitrary extracted object creation.
Add regression coverage for forged-signature Snowball unsigned-trailer writes, anonymous Snowball requests against non-public buckets, and legitimate signed Snowball extraction with trailing CRC32 trailers. Validate the new tests against the vulnerable parent and patched tree, and confirm with containerized before/after smoke runs that the exploit succeeds pre-fix, fails post-fix, and normal signed Snowball uploads still extract correctly.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Track issue #27 / GHSA-hv4r-mvr4-25vw as fake CVE-2026-40027. Close the unsigned-trailer trust flaw that let query-string credentials skip signature verification in PutObject and PutObjectPart by moving presigned rejection and SigV4 verification into newUnsignedV4ChunkedReader(), so authTypeStreamingUnsignedTrailer can no longer silently downgrade query auth into an anonymous body read.
Add focused regression coverage for forged query-string-only unsigned-trailer PUTs and multipart uploads, mixed header/query auth rejection, and anonymous unsigned-trailer writes that remain allowed only when bucket policy explicitly permits them. Validate the new tests against the vulnerable parent and confirm with before/after live-server runs that presigned unsigned-trailer attacks are rejected while legitimate header-authenticated and policy-driven flows still work.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Enforce the 1 MiB maxCharsPerRecord limit while splitting CSV and line-delimited JSON input so oversized records are rejected before they can be buffered and parsed.
Return OverMaxRecordSize for these failures instead of collapsing them into InternalError, and preserve splitter errors in the JSON worker so oversized-record failures are not lost after successful partial decode.
Close the replication-header trust flaw that allowed ordinary PutObject and CopyObject requests to smuggle X-Minio-Replication-* headers into X-Minio-Internal-* SSE metadata and write objects into an unreadable state. Stop accepting replication-only metadata in the default extraction path, restore it only after a trusted replication write has passed ReplicateObjectAction, and tighten CopyObject by sanitizing replication-only request headers before metadata, precondition, and SSE-C source handling consume them. Also gate replica status writes on the same trusted replication path and restore replication SSE metadata in multipart and snowball upload flows so legitimate replication continues to work.
Add focused regression coverage for untrusted PUT and COPY header poisoning at the handler layer, plus helper tests for trusted vs untrusted metadata extraction and CopyObject header sanitization. Validate the new tests against both the patched tree and the vulnerable HEAD baseline, and confirm with live server before/after runs that malicious PUT/COPY requests no longer turn objects unreadable.
Co-authored-by: Codex <codex@openai.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Prevent username enumeration in AssumeRoleWithLDAPIdentity by returning the same external STS error for unknown users and invalid passwords, while preserving LDAP infrastructure failures as upstream errors so they continue to surface as 500s and remain visible in server logs.
Add a small in-memory rate limiter for LDAP STS login attempts, keyed by source IP and normalized username, and add regression coverage for auth failure classification, throttling, and Docker-backed LDAP end-to-end flows.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Close the OIDC JWT algorithm confusion flaw in AssumeRoleWithWebIdentity by restoring a JWKS-only verification path. Stop injecting the client secret into the verifier keyring and restrict accepted signing methods to the asymmetric algorithms already supported by the existing JWKS flow.
Add regression coverage to verify HS256 tokens are rejected, RS256 tokens remain valid, and JWKS refresh and retry logic cannot bypass the method allowlist.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Replace minio/pkg/v3 with pgsty/minio-pkg/v3 v3.6.3 to fix LDAP TLS
regression where DialURL() was not passing TLS config for ldaps://
connections, causing InsecureSkipVerify and RootCAs to be silently
ignored (x509: certificate signed by unknown authority).
Pin four dependencies to avoid breaking changes introduced in 5abd9a80f:
- go-ldap/ldap/v3 v3.4.12: v3.4.13 rewrote GetLDAPError() internals
- IBM/sarama v1.45.1: v1.46.0 changed Kafka protocol version negotiation
- lib/pq v1.10.9: v1.11.0 treats nil []byte as NULL and drops PG <14
- etcd v3.6.8: stay on intermediate version per policy
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
We maintain our own release pipeline (release.yml, test-release.yml)
and have no use for the upstream test/lint/integration workflows.
They reference infrastructure and secrets we don't have, and the
PR-triggered jobs never fire since we don't take external pull requests.
Closes#14
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rework Dockerfile.goreleaser to download the latest mcli binary from
pgsty/mc GitHub releases, verify its SHA-256 checksum, and install both
mcli and mc (symlink) into the final image alongside minio and curl.
Also add download-static-curl.sh to goreleaser extra_files and enable
workflow_dispatch for the release workflow.
Go 1.26.1 tightens a few toolchain checks that older builds tolerated.\n\nCast aliased replication status values back to their defining type before calling the generated msgp helpers, and replace Sprintf+WriteString pairs with direct Fprintf calls where needed.\n\nThese are compatibility-only source changes to keep the cmd package building cleanly under the newer linker/toolchain.
Switch to community-maintained console fork (georgmangold/console v1.9.1)
and update dependencies accordingly. Fix go vet format directive in
grid_test.go and adapt test status code for Go 1.26 HTTP semantics.
- Revert console dependency from stripped v1.7.7-pre to v1.7.6,
restoring the full embedded management console
- Rewrite README disclaimer with proper trademark attribution and
nominative fair use language for AGPL compliance
- Update documentation links and Go module paths to this repository
- Restore docs removed upstream (hotfixes.md, metrics/v3.md)
- Restore feature request issue template
- Update Go version to 1.26.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix(api): Don't send responses twice.
In some cases multiple responses are being sent for one request, causing
the API server to incorrectly drop connections.
This change introduces a ResponseWriter which tracks whether a
response has already been sent. This is used to prevent a response being
sent if something already has (e.g. by a preconditions check function).
Fixes#21633.
Co-authored-by: Menno Finlay-Smits <hello@menno.io>
This fixes a security issue where sub-policy attached to a service
account or STS account is not properly validated under certain "own"
account operations (like creating new service accounts). This allowed a
service account to create new service accounts for the same user
bypassing the inline policy restriction.
fix: use correct dummy ARN for claim-based OIDC provider
When listing OIDC access keys, use the correct ARN when looking up the provider configuration for the claim-based provider. Without this it was impossible to list access keys for a claim-based provider, only for a role-policy-based provider.
Fixesminio/minio#21548
Commit b6eb8dff64 renamed compression
setting environment variables to follow consistent style.
Although it preserved backward compatibility for the most part (i.e. it
handled MINIO_COMPRESS_ALLOW_ENCRYPTION, MINIO_COMPRESS_EXTENSIONS, and
MINIO_COMPRESS_MIME_TYPES), MINIO_COMPRESS_ENABLE was left behind.
Additionally, due to incorrect fallback ordering, and DefaultKVS
containing enable=off allow_encryption=off (so kvs.Get should've been
tried last), that commit broke MINIO_COMPRESS_ALLOW_ENCRYPTION (even
though it appeared to be handled), and even older MINIO_COMPRESS, too.
The legacy MIME types and extensions variables take precedence over both
config and new variables, so they don't need fixing.
`go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...` executed.
`go generate ./...` ran afterwards to keep generated.
* Store `ctx.Done` channel in a struct instead of a `ctx`. See: https://go.dev/blog/context-and-structs
* Return from `handleListener` on `ctx` cancellation, preventing goroutine leaks
* Simplify `handleListener` by removing the `send` closure. The `handleListener` is inlined by the compiler
* Return the first error from `Close`
* Preallocate slice in `Addrs`
* Reduce duplication in handling `opts.Trace`
* http/listener: revert error propagation from Close()
* http/listener: preserve original listener address in Addr()
* Preserve the original address when calling Addr() with multiple listeners
* Remove unused listeners from the slice
RoleARN is a required parameter in AssumeRoleWithWebIdentity,
according to the standard AWS implementation, and the official
AWS SDKs and CLI will not allow you to assume a role from a JWT
without also specifying a RoleARN. This meant that it was not
possible to use the official SDKs for claim-based OIDC with Minio
(minio/minio#21421), since Minio required you to _omit_ the RoleARN in this case.
minio/minio#21468 attempted to fix this by disabling the validation
of the RoleARN when a claim-based provider was configured, but this had
the side effect of making it impossible to have a mixture of claim-based
and role-based OIDC providers configured at the same time - every
authentication would be treated as claim-based, ignoring the RoleARN entirely.
This is an alternative fix, whereby:
- _if_ the `RoleARN` is one that Minio knows about, then use the associated role policy
- if the `RoleARN` is not recognised, but there is a claim-based provider configured, then ignore the role ARN and attempt authentication with the claim-based provider
- if the `RoleARN` is not recognised, and there is _no_ claim-based provider, then return an error.
Fixes incorrect application of ilm expiry rules on versioned objects
when replication is enabled.
Regression from https://github.com/minio/minio/pull/20441 which sends
DeleteObject calls to all pools. This is a problem for replication + ilm
scenario since replicated version can end up in a pool by itself instead of
pool where remaining object versions reside.
For example, if the delete marker is set on pool1 and object versions exist on
pool2, the second rule below will cause the delete marker to be expired by ilm
policy since it is the single version present in pool1
```
{
"Rules": [
{
"ID": "cs6il1ri2hp48g71mdjg",
"NoncurrentVersionExpiration": {
"NoncurrentDays": 14
},
"Status": "Enabled"
},
{
"Expiration": {
"ExpiredObjectDeleteMarker": true
},
"ID": "cs6inj3i2hp4po19cil0",
"Status": "Enabled"
}
]
}
```
this is minor PR that supports building on RISC-V 64,
this PR is for compilation only. There is no guarantee
that code is tested and will work in production.
S3 listing call is usually sent with a 'max-keys' parameter. This
'max-keys' will also be passed to WalkDir() call. However, when ILM is
enabled in a bucket and some objects are skipped, the listing can
return IsTruncated set to false even if there are more entries in
the drives.
The reason is that drives stop feeding the listing code because it has
max-keys parameter and the listing code thinks listing is finished
because it is being fed anymore.
Ask the drives to not stop listing and relies on the context
cancellation to stop listing in the drives as fast as possible.
This commit removes FIPS 140-2 related code for the following
reasons:
- FIPS 140-2 is a compliance, not a security requirement. Being
FIPS 140-2 compliant has no security implication on its own.
From a tech. perspetive, a FIPS 140-2 compliant implementation
is not necessarily secure and a non-FIPS 140-2 compliant implementation
is not necessarily insecure. It depends on the concret design and
crypto primitives/constructions used.
- The boringcrypto branch used to achieve FIPS 140-2 compliance was never
officially supported by the Go team and is now in maintainance mode.
It is replaced by a built-in FIPS 140-3 module. It will be removed
eventually. Ref: https://github.com/golang/go/issues/69536
- FIPS 140-2 modules are no longer re-certified after Sep. 2026.
Ref: https://csrc.nist.gov/projects/cryptographic-module-validation-program
Signed-off-by: Andreas Auernhammer <github@aead.dev>
Fixes#21249
Example params: `-ftp=force-tls=true -ftp="tls-private-key=ftp/private.key" -ftp="tls-public-cert=ftp/public.crt"`
If MinIO is set up for TLS those certs will be used.
we had a chicken and egg problem with this feature even
when used with kes the credentials generation would
not work in correct sequence causing setup/deployment
disruptions.
This PR streamlines all of this properly to ensure that
this functionality works as advertised.
On buffered streams the final entries could be missing, if a lot
are delivered when stream ends.
Fixes end-of-stream cancelling return of final entries by canceling
with the StreamEOF error.
in a specific corner case when you only have dangling
objects with single shard left over, we end up a situation
where healing is unable to list this dangling object to
purge due to the fact that listing logic expected only
`len(disks)/2+1` - where as when you make this choice you
end up with a situation that the drive where this object
is present is not part of your expected disks list, causing
it to be never listed and ignored into perpetuity.
change the logic such that HealObjects() would be able
to listAndHeal() per set properly on all its drives, since
there is really no other way to do this cleanly, however
instead of "listing" on all erasure sets simultaneously, we
list on '3' at a time. So in a large enough cluster this is
fairly staggered.
To make sure that no objects were skipped for any reason,
decommissioning does a second phase of listing to check if there
are some objects that need to be decommissioned. However, the code
forgot to skip orphan delete markers since the decom code already
skips it.
Make the code ignore delete markers in in the verification phase.
Co-authored-by: Anis Eleuch <anis@min.io>
There is prefix in json, but not in the equivalent command line. Although the role of prefix has been explained in the previous example, I think it should be supplemented.
This is a security incident fix, it would seem like since
the implementation of unsigned payload trailer on PUTs,
we do not validate the signature of the incoming request.
The signature can be invalid and is totally being ignored,
this in-turn allows any arbitrary secret to upload objects
given the user has "WRITE" permissions on the bucket, since
acces-key is a public information in general exposes these
potential users with WRITE on the bucket to be used by any
arbitrary client to make a fake request to MinIO the signature
under Authorization: header is totally ignored.
A test has been added to cover this scenario and fail
appropriately.
- Move VersionPurgeStatus into replication package
- ilm: Evaluate policy w/ obj retention/replication
- lifecycle: Use Evaluator to enforce ILM in scanner
- Unit tests covering ILM, replication and retention
- Simplify NewEvaluator constructor
When decommissioning is started, the list of buckets to decommission is
calculated, however, a bucket can be removed before decommissioning reaches
it. This will cause an infinite loop of listing error complaining about
the non-existence of the bucket. This commit will ignore
errVolumeNotFound to skip the not found bucket.
Enforce a bucket count limit on metrics for v2 calls.
If people hit this limit, they should move to v3, as certain calls explode with high bucket count.
Reviewers: This *should* only affect v2 calls, but the complexity is overwhelming.
If a user attempts to authenticate with a key but does not have an
sshpubkey attribute in LDAP, the server allows the connection, which
means the server trusted the key without reason. This is now fixed,
and a test has been added for validation.
Allow multiple private keys and extract all files from streams.
Place files in the folder with `.enc` removed.
Do basic checks so streams cannot traverse outside of the folder.
This commit adds the `MINIO_KMS_REPLICATE_KEYID` env. variable.
By default - if not specified or not set to `off` - MinIO will
replicate the KMS key ID of an object.
If `MINIO_KMS_REPLICATE_KEYID=off`, MinIO does not include the
object's KMS Key ID when replicating an object. However, it always
sets the SSE-KMS encryption header. This ensures that the object
gets encrypted using SSE-KMS. The target site chooses the KMS key
ID that gets used based on the site and bucket config.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
This commit allows clients to provide a set of intermediate CA
certificates (up to `MaxIntermediateCAs`) that the server will
use as intermediate CAs when verifying the trust chain from the
client leaf certificate up to one trusted root CA.
This is required if the client leaf certificate is not issued by
a trusted CA directly but by an intermediate CA. Without this commit,
MinIO rejects such certificates.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
If object is uploaded with tags, the internal tagging-timestamp tracked
for replication will be missing. Default to ModTime in such cases to
allow tags to be synced correctly.
Also fixing a regression in fetching tags and tag comparison
* fix: proxy requests to honor global transport
Load the globalProxyEndpoint properly
also, currently, the proxy requests will fail silently for batch cancel
even if the proxy fails; instead,d properly send the corresponding error back
for such proxy failures if opted
* pass the transport to the GetProxyEnpoints function
---------
Co-authored-by: Praveen raj Mani <praveen@minio.io>
Reject new lock requests immediately when 1000 goroutines are queued
for the local lock mutex.
We do not reject unlocking, refreshing, or maintenance; they add to the count.
The limit is set to allow for bursty behavior but prevent requests from
overloading the server completely.
Currently, DeleteObjects() tries to find the object's pool before
sending a delete request. This only works well when an object has
multiple versions in different pools since looking for the pool does
not consider the version-id. When an S3 client wants to
remove a version-id that exists in pool 2, the delete request will be
directed to pool one because it has another version of the same object.
This commit will remove looking for pool logic and will send a delete
request to all pools in parallel. This should not cause any performance
regression in most of the cases since the object will unlikely exist
in only one pool, and the performance price will be similar to
getPoolIndex() in that case.
Earlier, cluster and bucket metrics were named
`minio_usage_last_activity_nano_seconds`.
The bucket level is now named as
`minio_bucket_usage_last_activity_nano_seconds`
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
When compression is enabled, the final object size is not calculated in
that case, we need to make sure that the provided buffer is always
more significant than the shard size, the bitrot will always calculate
the hash of blocks with shard size, except the last block.
Before https://github.com/minio/minio/pull/20575, files could pick up indices
from unrelated files if no index was added.
This would result in these files not being consistent across a set.
When loading, search for the compression indicators and check if they
are within the problematic date range, and clean up any parts that have
an index but shouldn't.
The test validates that the signature matches the one in files stored without an index.
Bumps xlMetaVersion, so this check doesn't have to be made for future versions.
It is possible delete marker was received on old pool as decom
move in progress, this PR allows decom retry to ensure these
delete markers are moved to new pool so that decommission can
be completed.
Fixes#20819
Add profiling potential crash wourkaround
Using admin traces could potentially crash the server (or handler more likely) due to upstream divide by 0: https://github.com/felixge/fgprof/pull/34
Ensure the profile always runs 100ms before stopping, so sample count isn't 0 (default sample rate ~10ms/sample, but allow for cpu starvation)
If one object has many parts where all parts are readable but some parts
are missing from some drives, this object can be sometimes un-healable,
which is wrong.
This commit will avoid reading from drives that have missing, corrupted or
outdated xl.meta. It will also check if any part is unreadable to avoid
healing in that case.
we do not need to hold the read locks at the higher
layer instead before reading the body, instead hold
the read locks properly at the time of renamePart()
for protection from racy part overwrites to compete
with concurrent completeMultipart().
CheckParts call can take time to verify 10k parts of a single object in a single drive.
To avoid an internal dealine of one minute in the single handler RPC, this commit will
switch to streaming RPC instead.
This API had missing permissions checking, allowing a user to change
their policy mapping by:
1. Craft iam-info.zip file: Update own user permission in
user_mappings.json
2. Upload it via `mc admin cluster iam import nobody iam-info.zip`
Here `nobody` can be a user with pretty much any kind of permission (but
not anonymous) and this ends up working.
Some more detailed steps - start from a fresh setup:
```
./minio server /tmp/d{1...4} &
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc admin user add myminio nobody nobody123
mc admin policy attach myminio readwrite nobody nobody123
mc alias set nobody http://localhost:9000 nobody nobody123
mc admin cluster iam export myminio
mkdir /tmp/x && mv myminio-iam-info.zip /tmp/x
cd /tmp/x
unzip myminio-iam-info.zip
echo '{"nobody":{"version":1,"policy":"consoleAdmin","updatedAt":"2024-08-13T19:47:10.1Z"}}' > \
iam-assets/user_mappings.json
zip -r myminio-iam-info-updated.zip iam-assets/
mc admin cluster iam import nobody ./myminio-iam-info-updated.zip
mc admin service restart nobody
```
`mc admin heal ALIAS/bucket/object` does not have any flag to heal
object noncurrent versions, this commit will make healing of the object
noncurrent versions implicitly asked.
This also fixes the 'mc admin heal ALIAS/bucket/object' that does not work
correctly when the bucket is versioned. This has been broken since Apr 2023.
Golang http.Server will call SetReadDeadline overwriting the previous
deadline configuration set after a new connection Accept in the custom
listener code. Therefore, --idle-timeout was not correctly respected.
Make http.Server read/write timeout similar to --idle-timeout.
The code assigns corrupted state to a drive for any unexpected error,
which is confusing for users. This change will make sure to assign
corrupted state only for corrupted parts or xl.meta. Use unknown state
with a explanation for any unexpected error, like canceled, deadline
errors, drive timeout, ...
Also make sure to return the bucket/object name when the object is not
found or marked not found by the heal dangling code.
* Add the policy name to the audit log tags when doing policy-based API calls
* Audit log the retention settings requested in the API call
* Audit log of retention on PutObjectRetention API path too
The RemoveUser API only removes internal users, and it reports success
when it didnt find the internal user account for deletion. When provided
with a service account, it should not report success as that is misleading.
Update github.com/cosnicolaou/pbzip2 to latest version for
significant performance improvements. This update brings a 45%
reduction in processing time.
Previously, not setting http.Config.HTTPTimeout for logger webhook
was resulting in a timeout of 0, and causing "deadline exceeded"
errors in log webhook.
This change introduces a new env variable for configuring log webhook
timeout and more importantly sets it when the config is initialised.
Manual heal can return XMinioHealInvalidClientToken if the manual
healing is started in the first node, and the next mc call to get the
heal status is landed on another node. The reason is that redirection
based on the token ID is not able to redirect requests to the first node
due to a typo.
This also affects the batch cancel command if the batch is being done in
the first node, the user will never be able to cancel it due to the same
bug.
HTTP likes to slap an infinite read deadline on a connection and
do a blocking read while the response is being written.
This effectively means that a reading deadline becomes the
request-response deadline.
Instead of enforcing our timeout, we pass it through and keep
"infinite deadline" is sticky on connections.
However, we still "record" when reads are aborted, so we never overwrite that.
The HTTP server should have `ReadTimeout` and `IdleTimeout` set for the deadline to be effective.
Use --idle-timeout for incoming connections.
Since DeadlineConn would send deadline updates directly upstream,
it would race with Read/Write operations. The stdlib will perform a read,
but do an async SetReadDeadLine(unix(1)) to cancel the Read in
`abortPendingRead`. In this case, the Read may override the
deadline intended to cancel the read.
Stop updating deadlines if a deadline in the past is seen and when Close is called.
A mutex now protects all upstream deadline calls to avoid races.
This should fix the short-term buildup of...
```
365 @ 0x44112e 0x4756b9 0x475699 0x483525 0x732286 0x737407 0x73816b 0x479601
# 0x475698 sync.runtime_notifyListWait+0x138 runtime/sema.go:569
# 0x483524 sync.(*Cond).Wait+0x84 sync/cond.go:70
# 0x732285 net/http.(*connReader).abortPendingRead+0xa5 net/http/server.go:729
# 0x737406 net/http.(*response).finishRequest+0x86 net/http/server.go:1676
# 0x73816a net/http.(*conn).serve+0x62a net/http/server.go:2050
```
AFAICT Only affects internode calls that create a connection (non-grid).
These are needed checks for the functions to be un-crashable with any input
given to `msgUnPath` (tested with fuzzing).
Both conditions would result in a crash, which prevents that. Some
additional upstream checks are needed.
Fixes#20610
This PR fixes a regression introduced in https://github.com/minio/minio/pull/19797
by restoring the healing ability of transitioned objects
Bonus: support for transitioned objects to carry original
The object name is for future reverse lookups if necessary.
Also fix parity calculation for tiered objects to n/2 for n/2 == (parity)
Existing implementation runs IAM purge routines for expired LDAP and
OIDC accounts with a probability of 0.25 after every IAM refresh. This
change ensures that they are run once in each hour.
xlStorage.Healing() returns nil if there is an error reading
.healing.bin or if this latter is empty. healing.bin update()
call returns early if .healing.bin is empty; hence, no further update
of .healing.bin is possible.
A .healing.bin can be empty if os.Open() with O_TRUNC is successful
but the next Write returns an error.
To avoid this weird situation, avoid making healingTracker.update()
to return early if .healing.bin is empty, so write again.
This commit also fixes wrong error log printing when an object is
healed in another drive in the same erasure set but not in the drive
that is actively healing by fresh drive healing code. Currently, it prints
<nil> instead of a factual error.
* heal: Scan .minio.sys metadata only during site-wide heal (#137)
mc admin heal always invoke .minio.sys heal, but sometimes, this latter
contains a lot of data, many service accounts, STS accounts etc, which
makes mc admin heal command very slow.
Only invoke .minio.sys healing when no bucket was specified in `mc admin
heal` command.
Healing a large object with a normal scan mode where no parts read
is involved can still fail after 30 seconds if an object has
There are too many parts when hard disks are being used mainly.
The reason is there is a general deadline that checks for all parts we
do a deadline per part.
When listing objects with metadata, avoid returning an "expires" time
metadata value when its value is the zero time as this means that no
expires value is set on the object.
The condition were incorrect as we were comparing the filter
value against the modification time object.
For example if created after filter date is after modification
time of object, that means object was created before the filter
time and should be skipped while replication because as per the
filter we need only the objects created after the filter date.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
When Keycloak vendor is set, the code will start to clean up service
accounts that parents do not exist anymore. However, the code will also
look for the parent user of site-replicator-0, MINIO_ROOT_USER, which
obviously does not exist in Keycloak. Therefore, the site-replicator-0
will be removed automatically.
This commit will avoid cleaning up service accounts generated from
the root user.
The verify file handler response format was changed from gob to msgp
since two months but we forgot updating the verify handler client.
VerifyFile is only called during a heal deep scan (bitrot check).
HealObject() will fail in that case and will mark all disks corrupted and
will return early (as unrecoverable object but it will also not be
removed)
It is a bit rare for HealObject to be called with a deep scan flag. It
is called when a HealObject with a normal scan (e.g. new drive healing)
detects a bitrot corruption, therefore healing objects with a detected
bitrot corruption will fail.
in cases where we cannot possibly know a way to read and
construct the object, it is impossible to achieve any form of
quorum via xl.meta while we have sufficient responses from
all the drives, we should return object not found.
- PutObjectMetadata()
- PutObjectTags()
- DeleteObjectTags()
- TransitionObject()
- RestoreTransitionObject()
Also improve the behavior of multipart code across
pool locks, hold locks only once per upload ID for
- CompleteMultipartUpload()
- AbortMultipartUpload()
- ListObjectParts() (read-lock)
- GetMultipartInfo() (read-lock)
- PutObjectPart() (read-lock)
This avoids lock attempts across pools for no
reason, this increases O(n) when there are n-pools.
multi-object deletion may or may not compete with locks
granted for other callers, causing concurrent operations
to succeed on each other.
A continuation of the PR https://github.com/minio/minio/pull/20356
When custom authorization via plugin is enabled, the console will now
render the UI as if all actions are allowed. Since server cannot
determine the exact policy allowed for a user via the plugin, this is
acceptable to do. If a particular action is actually not allowed by the
plugin the call will result in an error.
Previously the server was evaluating a policy when custom authZ is
enabled - this is fixed now.
The "unlimited" value on PPC wasn't exactly the same as amd64.
Instead compare against an "unreasonably big value".
Would cause OOM in anything using the concurrent request limit.
Add TTFB for all requests in metrics-v3 in addition to the existing
GetObject. Also for the requests that do not return a body in the
response, calculate TTFB as the HTTP status code and the headers are
sent.
A batch job will fail if the retry attempt is not provided. The reason
is that the code mistakenly gets the retry attempts from the job status
rather than the job yaml file.
This will also set a default empty prefix for batch expiration.
Also this will avoid trimming the prefix since the yaml decoder already
does that if no quotes were provided, and we should not trim if quotes
were provided and the user provided a leading or a trailing space.
Tests if imported service accounts have
required access to buckets and objects.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
- PutObject() for multi-pooled was holding large
region locks, which was not necessary. This affects
almost all slowpoke clients and lengthy uploads.
- Re-arrange locks for CompleteMultipart, PutObject
to be close to rename()
Currently, it is not possible to remove a tier if it is not accessible
or contains some data, add a force flag to make the removal successful
in that case.
When the encryption and compression are both enabled, the
the server will avoid compressing the data for no apparent reason
This commit will enable it and update unit tests.
Download static cURL into release Docker image for all supported architectures.
Currently, the static cURL executable is only downloaded for the `amd64` architecture. However, `arm64` and `ppc64le` variants are also [available](https://github.com/moparisthebest/static-curl/releases/tag/v8.7.1).
postUpload() incorrectly saves actual size as '-1'
we should save correct size when its possible.
Bonus: fix the PutObjectPart() write locker, instead
of holding a lock before we read the client stream.
We should hold it only when we need to commit the parts.
this cache will be honored only when `prefix=""` while
performing ListMultipartUploads() operation.
This is mainly to satisfy applications like alluxio
for their underfs implementation and tests.
replaces https://github.com/minio/minio/pull/20181
AFAICT we send a canceled context to unlock (and thereby releaseAll). This will cause network calls to fail.
Instead use background and add 30s timeout.
Current implementation retries forever until our
log buffer is full, and we start dropping events.
This PR allows you to set a value until we give
up on existing audit/logger batches to proceed to
process the new ones.
Bonus:
- do not blow up buffers beyond batchSize value
- do not leak the ticker if the worker returns
The items will be saved per target batch and will
be committed to the queue store when the batch is full
Also, periodically commit the batched items to the queue store
based on configured commit_timeout; default is 30s;
Bonus: compress queue store multi writes
rebalance metadata is good to have only,
if it cannot be loaded when starting MinIO
for some reason we can possibly ignore it
and move on and let user start rebalance
again if needed.
readParts requires that both part.N and part.N.meta files be present.
This change addresses an issue with how an error to return to the upper
layers was picked from most drives where a UploadPart operation
had failed.
By default, even if MINIO_BROWSER=off set code tries to get free
port available for the console.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
S3 spec does not accept an ILM XML document containing both <Filter>
and <Prefix> XML tags, even if both are empty. That is why we added
a 'set' field in some lifecycle structures to decide when and when not to
show a tag. However, we forgot to disallow marshaling of Filter when
'set' is set to false.
This will fix ILM document replication in a site replication
configuration in some cases.
When the prefix field is not provided in the remote source of a yaml
replication job, the code fails to do listing and makes replication
successful. This commit fixes it.
This change adds a consistent nonce to ensure
that multipart uploads are deterministic on a
per-part basis.
Thanks to @klauspost for the work here minio/sio@3cd3734
locks handed by different pools would become non-compete for
multi-object delete request, this is wrong for obvious
reasons.
New locking implementation and revamp will rewrite multi-object
lock anyway, this is a workaround for now.
Currently, the bucket events and replication targets are only reloaded
with buckets that failed to load during the first cluster startup,
which is wrong because if one bucket change was done in one node but
that node was not able to notify other nodes; the other nodes will
reload the bucket metadata config but fails to set the events and bucket
targets in the memory.
when a hung drive is hot-unplugged, the server might go
into a loop where the previous `format.json` is somehow
still accessible to the process, we try to re-init() drives,
but that seems to cause a previous goroutine to hang around
since it is not canceled away when the drive is closed.
Bonus: add deadline for immediate purge routine, to unblock
it if the drive is blocking mutations.
if a user policy is found, avoid reading from the drives
for missing group mappings, group mappings are not mandatory
and conditional.
This PR restores the older behavior while making sure that
if a direct user policy is not found, we would still attempt
to load from the group from the drives.
This commit simplifies and optimizes the decryption of large (multipart)
objects. This PR does two things:
- Re-write the init logic for the decryption reader
- Reduce the number of OEK decryptions
Before, the init logic copied some SSE HTTP request headers to
parse them later. This is simplified to parsing them right away. This
removes some fields from the decryption reader struct.
Further, the decryption reader decrypted the OEK using the client-provided
key (SSE-C) or the KMS (SSE-S3 / SSE-KMS) for each part. This is redundant
since the OEK is the same for all parts. In particular, a KMS call might be a
network request. Now, the OEK is decrypted once for the entire multipart object.
This should improve latency when reading encrypted multipart objects
and reduce requests to the KMS.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
Use Walk(), which is a recursive listing with versioning, to check if
the bucket has some objects before being removed. This is beneficial
because the bucket can contain multiple dangling objects in multiple
drives.
Also, this will prevent a bug where a bucket is deleted in a deployment
that has many erasure sets but the bucket contains one or few objects
not spread to enough erasure sets.
Currently, retry healing of a new drive healing does not reset
HealedBuckets means that the next healing retry will skip those
buckets. The commit will fix this behavior.
Also, the skipped objects counter will include objects uploaded
that are uploaded after the healing is started.
sftp sends local requests to the S3 port while passing the session token
header when the account corresponds to a service account. However, this
is not permitted and will throw an error: "The security token included in the
request is invalid"
This commit will avoid passing the session token to the upper layer that
initializes MinIO client to avoid this error.
Sometimes, we need historical information in .healing.bin, such as the
number of expired objects that the healing avoids to heal and that can
create drive usage disparency in the same erasure set. For that reason,
this commit will not remove .healing.bin anymore and it will have a new
field called Finished so we know healing is finished in that drive.
Services are unfrozen before `initBackgroundReplication` is finished. This means that
the globalReplicationStats write is racy. Switch to an atomic pointer.
Provide the `ReplicationPool` with the stats, so it doesn't have to be grabbed
from the atomic pointer on every use.
All other loads and checks are nil, and calls return empty values when stats
still haven't been initialized.
* Allow a maximum of 10 seconds to start profiling operations.
* Download up to 16 profiles concurrently, but only allow 10 seconds for
each (does not include write time).
* Add cluster info as the first operation.
* Ignore remote download errors.
* Stop remote profiles if the request is terminated.
If the site replication is enabled and the code tries to extract jwt
claims while the site replication service account credentials are still
not loaded yet, the code will enter an infinite loop, causing in a
high CPU usage.
Another possibility of the infinite loop is having some service accounts
created by an old deployment version where the service account JWT was
signed by the root credentials, but not anymore.
This commit will remove the possibility of the infinite loop in the code
and add root credential fallback to extract claims from old service
accounts.
move away from map[string]interface{} to map[string]string
to simplify the audit, and also provide concise information.
avoids large allocations under load(), reduces the amount
of audit information generated, as the current implementation
was a bit free-form. instead all datastructures must be
flattened.
Previously, we checked if we had a quorum on the DataDir value.
We are removing this check, which allows reading objects with different
DataDir values in a few drives (due to a rebalance-stop race bug)
provided their eTags or ModTimes match.
Since a lot of operations load from storage, do remote calls, add a 10 second timeout to each operation.
This should make `mc admin info` return values even under extreme conditions.
- optimize writing part.N.meta by writing both part.N
and its meta in sequence without network component.
- remove part.N.meta, part.N which were partially success
ful, in quorum loss situations during renamePart()
- allow for strict read quorum check arbitrated via ETag
for the given part number, this makes it double safer
upon final commit.
- return an appropriate error when read quorum is missing,
instead of returning InvalidPart{}, which is non-retryable
error. This kind of situation can happen when many
nodes are going offline in rotation, an example of such
a restart() behavior is statefulset updates in k8s.
fixes#20091
during rebalance stop, it can possibly happen that
Put() would race by overwriting the same object again.
This may very well if done "successfully" it can
potentially proceed to delete the object from the pool,
causing data loss.
This PR enhances #20233 to handle more scenarios such
as these.
Rebalance-stop can race with ongoing rebalance operations. This change
prevents these operations from overwriting objects by checking the source
and destination pool indices are different.
This commit replaces the LDAP client TLS config and
adds a custom list of TLS cipher suites which support
RSA key exchange (RSA kex).
Some LDAP server connections experience a significant slowdown
when these cipher suites are not available. The Go TLS stack
disables them by default. (Can be enabled via GODEBUG=tlsrsakex=1).
fixes https://github.com/minio/minio/issues/20214
With a custom list of TLS ciphers, Go can pick the TLS RSA key-exchange
cipher. Ref:
```
if c.CipherSuites != nil {
return c.CipherSuites
}
if tlsrsakex.Value() == "1" {
return defaultCipherSuitesWithRSAKex
}
```
Ref: https://cs.opensource.google/go/go/+/refs/tags/go1.22.5:src/crypto/tls/common.go;l=1017
Signed-off-by: Andreas Auernhammer <github@aead.dev>
this allows for de-duplicating the callers when called
concurrently, allowing for bucketmetadata reads to be
single call. All concurrent callers will get the same data
as the first one.
Fix a regression in #19733 where TTFB metrics for all APIs except
GetObject were removed in v2 and v3 metrics. This causes breakage for
existing v2 metrics users. Instead we continue to send TTFB for all APIs
in V2 but only send for GetObject in V3.
allow multipart uploads expiration to be dyamic
It would seem like the new values will take effect
only after a restart for changes in multipart_expiration.
This PR fixes this by making it dynamic as it should have
been.
deadlines per moveToTrash() allows for a more granular timeout
approach for syscalls, instead of an aggregate timeout.
This PR also enhances multipart state cleanup to be optimal by
removing 100's of multipart network rename() calls into single
network call.
context deadline was introduced to avoid a slow transfer from blocking
replication queue(s) shared by other buckets that may not be under throttling.
This PR removes this context deadline for larger objects since they are
anyway restricted to a limited set of workers. Otherwise, objects would
get dequeued when the throttle limit is exceeded and cannot proceed
within the deadline.
epoll contention on TCP causes latency build-up when
we have high volume ingress. This PR is an attempt to
relieve this pressure.
upstream issue https://github.com/golang/go/issues/65064
It seems to be a deeper problem; haven't yet tried the fix
provide in this issue, but however this change without
changing the compiler helps.
Of course, this is a workaround for now, hoping for a
more comprehensive fix from Go runtime.
- ReadVersion
- ReadFile
- ReadXL
Further changes include to
- Compact internode resource RPC paths
- Compact internode query params
To optimize on parsing by gorilla/mux as the
length of this string increases latency in
gorilla/mux - reduce to a meaningful string.
the main reason is to let Go net/http perform necessary
book keeping properly, and in essential from consistency
point of view its GETs all the way.
Deprecate sendFile() as its buggy inside Go runtime.
For a non-tiered object, MinIO requires that EcM (# of data blocks) of
xl.meta agree, corresponding to the number of data blocks needed to
read this object.
OTOH, tiered objects have metadata in the hot tier and data in the
warm tier. The data and its integrity are offloaded to the warm tier. This
allows us to reduce the read quorum from EcM (typically > N/2, where N -
erasure stripe width) to N/2 + 1. The simple majority of metadata
ensures consensus on what the object is and where it is
located.
When a drive is in a failed state when a single node multiple drives
deployment is started, a replacement of a fresh disk will not be
properly healed unless the user restarts the node.
Fix this by always adding the new fresh disk to globalLocalDrivesMap. Also
remove globalLocalDrives for simplification, a map to store local node
drives can still be used since the order of local drives of a node is
not defined.
kms: Expose API available when bucket federation is enabled
When bucket federation feature is enabled, KMS API will not work, such
as `mc admin kms key list`
The commit will fix the issue by disabling bucket forwarding when this
is a KMS request.
Tracing syscalls, opening and reading an `xl.meta` looks like this:
```
openat(AT_FDCWD, "/mnt/drive1/ss8-old/testbucket/ObjSize4MiBThreads72/(554O51H/peTb(0iztdbTKw59.csv/xl.meta", O_RDONLY|O_NOATIME|O_CLOEXEC) = 34 <0.000>
fcntl(34, F_GETFL) = 0x48000 (flags O_RDONLY|O_LARGEFILE|O_NOATIME) <0.000>
fcntl(34, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOATIME) = 0 <0.000>
epoll_ctl(4, EPOLL_CTL_ADD, 34, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=3172471557, u64=8145488475984499461}}) = -1 EPERM (Operation not permitted) <0.000>
fcntl(34, F_GETFL) = 0x48800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOATIME) <0.000>
fcntl(34, F_SETFL, O_RDONLY|O_LARGEFILE|O_NOATIME) = 0 <0.000>
fstat(34, {st_mode=S_IFREG|0644, st_size=354, ...}) = 0 <0.000>
read(34, "XL2 \1\0\3\0\306\0\0\1P\2\2\1\304$\225\304\20\0\0\0\0\0\0\0\0\0\0\0"..., 354) = 354 <0.000>
close(34) = 0 <0.000>
```
Everything until `fstat` is the `os.Open` call.
Looking at the code: https://github.com/golang/go/blob/master/src/os/file_unix.go#L212-L243
It seems for every file it "tries" to see if it is pollable. This causes `syscall.SetNonblock(fd, true)` to be called. This is the first `F_SETFL`.
It then calls `f.pfd.Init("file", true)`. This will attempt to set it as pollable using `epoll_ctl`. This will always fail for files. It therefore calls `syscall.SetNonblock(fd, false)` resulting in the second `F_SETFL`.
If we set the `O_NONBLOCK` call on the initial open, we should avoid the 4 `fcntl` syscalls per file.
I don't see any way to avoid the `epoll_ctl` call, since kind is either `kindOpenFile` or `kindNonBlock`, so "pollable" will always be true. However avoiding 4 of 6 syscalls still seems worth it.
This should not have any effect, since files will end up with "nonblock" anyway.
allow non-inlined on disk to be inlined via
an unversioned ReadVersion() call, we only
need ReadXL() to resolve objects with multiple
versions only.
The choice of this block makes it to be dynamic
and chosen by the user via `mc admin config set`
Other bonus things
- Start measuring internode TTFB performance.
- Set TCP_NODELAY, TCP_CORK for low latency
Use `runtime.Gosched()` if we have less than maxMergeMessages and the
queue is empty. Up maxMergeMessages to 50 to merge more messages into
a single write.
Add length check for an early bailout on readAllInto when we know packet length.
This commit enforces FIPS-compliant TLS ciphers in FIPS mode
by importing the `fipsonly` module.
Otherwise, MinIO still accepts non-FIPS compliant TLS connections.
removes contentious usage of mutexes in LRU, which
were never really reused in any manner; we do not
need it.
To trust hosts, the correct way is TLS certs; this PR completely
removes this dependency, which has never been useful.
```
0 0% 100% 25.83s 26.76% github.com/hashicorp/golang-lru/v2/expirable.(*LRU[...])
0 0% 100% 28.03s 29.04% github.com/hashicorp/golang-lru/v2/expirable.(*LRU[...])
```
Bonus: use `x-minio-time` as a nanosecond to avoid unnecessary
parsing logic of time strings instead of using a more
straightforward mechanism.
- Also, fix failure reporting at the end.
- Also, avoid parsing report objects when listing or resuming jobs, this
does not cause any bugs, it is only printing, not useful errors.
Split the read and write sides of handleMessages into two separate functions
Cosmetic. The only non-copy-and-paste change is that `cancel(ErrDisconnected)` is moved
into the defer on `readStream`.
add unit test for v3 metrics for all its exposed endpoints
Bonus:
- support OpenMetrics encoding
- adds boot time for prometheus
- continueOnError is better to serve as
much metrics as possible.
```
curl http://localhost:9000/minio/metrics/v3/cluster/usage/buckets
```
Did not work as documented, due to the fact that there was a typo
in the bucket usage metrics registration group. This endpoint is
a cluster endpoint and does not require any `buckets` argument.
When creating the async listing, if the first request does not return within 3
minutes, it is stopped, since it isn't being kept alive.
Keep updating `lastHandout` while we are waiting for the initial request to be fulfilled.
When a fresh drive healing is finished, add more checks for the drive listing
errors. If any, re-list and heal again. Although this is an infrequent use
case to have listPathRaw() returning nil when minDisks is set to 1, we
still need to handle all possible use cases to avoid missing healing
any object.
Also, check for HealObject result to decide of an object is healed in the
fresh disk since HealObject returns nil if an object is healed in any
disk, and not in the new fresh drive.
In regular listing, this commit will avoid showing an object when its
latest version has a pending or failed deletion. In replicated setup.
It will also prevent showing older versions in the same case.
Also, sort the error map for multiple sites in ascending order
of deployment IDs, so that the error message generated is always
definitive order and same.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Add `ConnDialer` to abstract connection creation.
- `IncomingConn(ctx context.Context, conn net.Conn)` is provided as an entry point for
incoming custom connections.
- `ConnectWS` is provided to create web socket connections.
If `SkipReader` is called with a small initial buffer it may be doing a huge number if Reads to skip the requested number of bytes. If a small buffer is provided grab a 32K buffer and use that.
Fixes slow execution of `testAPIGetObjectWithMPHandler`.
Bonuses:
* Use `-short` with `-race` test.
* Do all suite test types with `-short`.
* Enable compressed+encrypted in `testAPIGetObjectWithMPHandler`.
* Disable big file tests in `testAPIGetObjectWithMPHandler` when using `-short`.
The code was advertenly passing max openfds to debug.SetMemoryLimit(),
fixing this accelerate go test in my machine.
This is only a testing bug, since the server context has always a valid
MaxMem, so the buggy code was never called in users environments.
Currently the status of a completed or failed batch is held in the
memory, a simple restart will lose the status and the user will not
have any visibility of the job that was long running.
In addition to the metrics, add a new API that reads the batch status
from the drives. A batch job will be cleaned up three days after
completion.
Also add the batch type in the batch id, the reason is that the batch
job request is removed immediately when the job is finished, then we
do not know the type of batch job anymore, hence a difficulty to locate
the job report
Hot load a policy document when during account authorization evaluation
to avoid returning 403 during server startup, when not all policies are
already loaded.
Add this support for group policies as well.
you can attempt a rebalance first i.e, start with 2 pools.
```
mc admin rebalance start alias/
```
and after that you can add a new pool, this would
potentially crash.
```
Jun 27 09:22:19 xxx minio[7828]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 27 09:22:19 xxx minio[7828]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x22cc225]
Jun 27 09:22:19 xxx minio[7828]: goroutine 1 [running]:
Jun 27 09:22:19 xxx minio[7828]: github.com/minio/minio/cmd.(*erasureServerPools).findIndex(...)
```
without atomic load() it is possible that for
a slow receiver we would get into a hot-loop, when
logCh is full and there are many incoming callers.
to avoid this as a workaround enable BATCH_SIZE
greater than 100 to ensure that your slow receiver
receives data in bulk to avoid being throttled in
some manner.
this PR however fixes the unprotected access to
the current workers value.
specifying customer HTTP client makes the gcs SDK
ignore the passed credentials, instead let the GCS
SDK manage the transport.
this PR fixes#19922 a regression from #19565
Currently, bucket metadata is being loaded serially inside ListBuckets
Objet API. Fix that by loading the bucket metadata as the number of
erasure sets * 10, which is a good approximation.
When a reconnection happens, `handleMessages` must be able to complete and exit.
This can be prevented in a full queue.
Deadlock chain (May 10th release)
```
1 @ 0x44110e 0x453125 0x109f88c 0x109f7d5 0x10a472c 0x10a3f72 0x10a34ed 0x4795e1
# 0x109f88b github.com/minio/minio/internal/grid.(*Connection).send+0x3eb github.com/minio/minio/internal/grid/connection.go:548
# 0x109f7d4 github.com/minio/minio/internal/grid.(*Connection).queueMsg+0x334 github.com/minio/minio/internal/grid/connection.go:586
# 0x10a472b github.com/minio/minio/internal/grid.(*Connection).handleAckMux+0xab github.com/minio/minio/internal/grid/connection.go:1284
# 0x10a3f71 github.com/minio/minio/internal/grid.(*Connection).handleMsg+0x231 github.com/minio/minio/internal/grid/connection.go:1211
# 0x10a34ec github.com/minio/minio/internal/grid.(*Connection).handleMessages.func1+0x6cc github.com/minio/minio/internal/grid/connection.go:1019
---> blocks ---> via (Connection).handleMsgWg
1 @ 0x44110e 0x454165 0x454134 0x475325 0x486b08 0x10a161a 0x10a1465 0x2470e67 0x7395a9 0x20e61af 0x20e5f1f 0x7395a9 0x22f781c 0x7395a9 0x22f89a5 0x7395a9 0x22f6e82 0x7395a9 0x22f49a2 0x7395a9 0x2206e45 0x7395a9 0x22f4d9c 0x7395a9 0x210ba06 0x7395a9 0x23089c2 0x7395a9 0x22f86e9 0x7395a9 0xd42582 0x2106c04
# 0x475324 sync.runtime_Semacquire+0x24 runtime/sema.go:62
# 0x486b07 sync.(*WaitGroup).Wait+0x47 sync/waitgroup.go:116
# 0x10a1619 github.com/minio/minio/internal/grid.(*Connection).reconnected+0xb9 github.com/minio/minio/internal/grid/connection.go:857
# 0x10a1464 github.com/minio/minio/internal/grid.(*Connection).handleIncoming+0x384 github.com/minio/minio/internal/grid/connection.go:825
```
Add a queue cleaner in reconnected that will pop old messages so `handleMessages` can
send messages without blocking and exit appropriately for the connection to be re-established.
Messages are likely dropped by the remote, but we may have some that can succeed,
so we only drop when running out of space.
Add the inlined data as base64 encoded field and try to add a string version if feasible.
Example:
```
λ xl-meta -data xl.meta
{
"8e03504e-1123-4957-b272-7bc53eda0d55": {
"bitrot_valid": true,
"bytes": 58,
"data_base64": "Z29sYW5nLm9yZy94L3N5cyB2MC4xNS4wIC8=",
"data_string": "golang.org/x/sys v0.15.0 /"
}
```
The string will have quotes, newlines escaped to produce valid JSON.
If content isn't valid utf8 or the encoding otherwise fails, only the base64 data will be added.
`-export` can still be used separately to extract the data as files (including bitrot).
* Prevents blocking when losing quorum (standard on cluster restarts).
* Time out to prevent endless buildup. Timed-out remote locks will be canceled because they miss the refresh anyway.
* Reduces latency for all calls since the wall time for the roundtrip to remotes no longer adds to the requests.
since the connection is active, the
response recorder body can grow endlessly
causing leak, as this bytes buffer is
never given back to GC due to an goroutine.
skip healing properly in scanner when drive is hotplugged
due to how the state is passed around the SkipHealing
might not be the true state() of the system always, causing
a situation where we might healing from the scanner on the
same drive which is being. Due to this competing heals get
triggered that slow each other down.
due to a historic bug in CopyObject() where
an inlined object loses its metadata, the
check causes an incorrect fallback verifying
data-dir.
CopyObject() bug was fixed in ffa91f9794 however
the occurrence of this problem is historic, so
the aforementioned check is stretching too much.
Bonus: simplify fileInfoRaw() to read xl.json as well,
also recreate buckets properly.
* Multipart SSEC checksums were not transferred.
* Remove key mismatch logging. This key is user-controlled with SSEC.
* If the source is SSEC and the destination reports ErrSSEEncryptedObject,
assume replication is good.
avoid concurrent callers for LoadUser() to even initiate
object read() requests, if an on-going operation is in progress.
this avoids many callers hitting the drives causing I/O
spikes, also allows for loading credentials faster.
This commit fixes an issue in the KES client configuration
that can cause the following error when connecting to KES:
```
ERROR Failed to connect to KMS: failed to generate data key with KMS key: tls: client certificate is required
```
The Go TLS stack seems to not send a client certificate if it
thinks the client certificate cannot be validated by the peer.
In case of an API key, we don't care about this since we use
public key pinning and the X.509 certificate is just a transport
encoding.
The `GetClientCertificate` seems to be honored always such that
this error does not occur.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
the reason for this is to avoid STS mappings to be
purged without a successful load of other policies,
and all the credentials only loaded successfully
are properly handled.
This also avoids unnecessary cache store which was
implemented earlier for optimization.
Directory objects are used by applications that simulate the folder
structure of an on-disk filesystem. These are zero-byte objects with names
ending with '/'. They are only used to check whether a 'folder' exists in
the namespace.
StartSize starts with the raw free space of all disks in the given pool,
however during the status, CurrentSize is not showing the current free
raw space, as expected at least by `mc admin decom status` since it was
written.
Go's net/http is notoriously difficult to have a streaming
deadlines per READ/WRITE on the net.Conn if we add them they
interfere with the Go's internal requirements for a HTTP
connection.
Remove this support for now
fixes#19853
Add partial shard reconstruction
* Add partial shard reconstruction
* Fix padding causing the last shard to be rejected
* Add md5 checks on single parts
* Move md5 verified to `verified/filename.ext`
* Move complete (without md5) to `complete/filename.ext.partno`
It's not pretty, but at least now the md5 gives some confidence it works correctly.
In the very rare case when all drives in a erasure set need to be healed,
remove .healing.bin from all drives, otherwise it will be stuck in a
loop
Also, fix a unit test that fails sometimes due to wrong test.
since #19688 there was a regression introduced during drive
lookups for single node multi-drive setups, drive replacement
would not work correctly without this PR.
This does not fix any current issue, but merging https://github.com/minio/madmin-go/pull/282
can lose the validation of the service account expiration time.
Add more defensive code for now. In the future, we should avoid doing
validation in another library.
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.
```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```
although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
fix: authenticate LDAP via actual DN instead of normalized DN
Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.
fixes#19757
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
Add combination of multiple parts.
Parts will be reconstructed and saved separately and can manually be combined to the complete object.
Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
Do not log errors on oneway streams when sending ping fails. Instead, cancel the stream.
This also makes sure pings are sent when blocked on sending responses.
This commit will fix one rare case of a multipart object that
can be read in theory but GetObject API returned an error.
It turned out that a six years old code was marking a drive offline
when the bitrot streaming fails to read a part in a disk with any error.
This can affect reading a subsequent part, though having enough shards,
but unable to construct because one drive was marked offline earlier.
This commit will remove the drive marking offline code. It will also
close the bitrotstreaming reader before marking it as nil.
Adds `-xver` which can be used with `-export` and `-combine` to attempt to combine files across versions if data is suspected to be the same. Overlapping data is compared.
Bonus: Make `inspect` accept wildcards.
Currently, on enabling callhome (or restarting the server), the callhome
job gets scheduled. This means that one has to wait for 24hrs (the
default frequency duration) to see it in action and to figure out if it
is working as expected.
It will be a better user experience to perform the first callhome
execution immediately after enabling it (or on server start if already
enabled).
Also, generate audit event on callhome execution, setting the error
field in case the execution has failed.
* Store ModTime in the upload ID; return it when listing instead of the current time.
* Use this ModTime to expire and skip reading the file info.
* Consistent upload sorting in listing (since it now has the ModTime).
* Exclude healing disks to avoid returning an empty list.
```
==================
WARNING: DATA RACE
Read at 0x0000082be990 by goroutine 205:
github.com/minio/minio/cmd.setCommonHeaders()
Previous write at 0x0000082be990 by main goroutine:
github.com/minio/minio/cmd.lookupConfigs()
```
Recent Veeam is very picky about storage class names. Add `_MINIO_VEEAM_FORCE_SC` env var.
It will override the storage class returned by the storage backend if it is non-standard
and we detect a Veeam client by checking the User Agent.
Applies to HeadObject/GetObject/ListObject*
add deadlines that can be dynamically changed via
the drive max timeout values.
Bonus: optimize "file not found" case and hung drives/network - circuit break the check and return right
away instead of waiting.
Do not log errors on oneway streams when sending ping fails. Instead cancel the stream.
This also makes sure pings are sent when blocked on sending responses.
I will do a separate PR that includes this and adds pings to two-way streams as well as tests for pings.
as that is the only API where the TTFB metric is beneficial, and
capturing this for all APIs exponentially increases the response size in
large clusters.
Replace the `io.Pipe` from streamingBitrotWriter -> CreateFile with a fixed size ring buffer.
This will add an output buffer for encoded shards to be written to disk - potentially via RPC.
This will remove blocking when `(*streamingBitrotWriter).Write` is called, and it writes hashes and data.
With current settings, the write looks like this:
```
Outbound
┌───────────────────┐ ┌────────────────┐ ┌───────────────┐ ┌────────────────┐
│ │ Parr. │ │ (http body) │ │ │ │
│ Bitrot Hash │ Write │ Pipe │ Read │ HTTP buffer │ Write (syscall) │ TCP Buffer │
│ Erasure Shard │ ──────────► │ (unbuffered) │ ────────────► │ (64K Max) │ ───────────────────► │ (4MB) │
│ │ │ │ │ (io.Copy) │ │ │
└───────────────────┘ └────────────────┘ └───────────────┘ └────────────────┘
```
We write a Hash (32 bytes). Since the pipe is unbuffered, it will block until the 32 bytes have
been delivered to the TCP buffer, and the next Read hits the Pipe.
Then we write the shard data. This will typically be bigger than 64KB, so it will block until two blocks
have been read from the pipe.
When we insert a ring buffer:
```
Outbound
┌───────────────────┐ ┌────────────────┐ ┌───────────────┐ ┌────────────────┐
│ │ │ │ (http body) │ │ │ │
│ Bitrot Hash │ Write │ Ring Buffer │ Read │ HTTP buffer │ Write (syscall) │ TCP Buffer │
│ Erasure Shard │ ──────────► │ (2MB) │ ────────────► │ (64K Max) │ ───────────────────► │ (4MB) │
│ │ │ │ │ (io.Copy) │ │ │
└───────────────────┘ └────────────────┘ └───────────────┘ └────────────────┘
```
The hash+shard will fit within the ring buffer, so writes will not block - but will complete after a
memcopy. Reads can fill the 64KB buffer if there is data for it.
If the network is congested, the ring buffer will become filled, and all syscalls will be on full buffers.
Only when the ring buffer is filled will erasure coding start blocking.
Since there is always "space" to write output data, we remove the parallel writing since we are
always writing to memory now, and the goroutine synchronization overhead probably not worth taking.
If the output were blocked in the existing, we would still wait for it to unblock in parallel write, so it would
make no difference there - except now the ring buffer smoothes out the load.
There are some micro-optimizations we could look at later. The biggest is that, in most cases,
we could encode directly to the ring buffer - if we are not at a boundary. Also, "force filling" the
Read requests (i.e., blocking until a full read can be completed) could be investigated and maybe
allow concurrent memory on read and write.
Metrics being added:
- read_tolerance: No of drive failures that can be tolerated without
disrupting read operations
- write_tolerance: No of drive failures that can be tolerated without
disrupting write operations
- read_health: Health of the erasure set in a pool for read operations
(1=healthy, 0=unhealthy)
- write_health: Health of the erasure set in a pool for write operations
(1=healthy, 0=unhealthy)
Adds regression test for #19699
Failures are a bit luck based, since it requires objects to be placed on different sets.
However this generates a failure prior to #19699
* Revert "Revert "Fix incorrect merging of slash-suffixed objects (#19699)""
This reverts commit f30417d9a8.
* Don't override when suffix doesn't match. Instead rely on quorum for each.
Instead of having "online" and "healing" as two metrics, replace with a
single metric "health" which can have following values:
0 = offline
1 = healthy
2 = healing
If two objects share everything but one object has a slash prefix, those would be merged in listings,
with secondary properties used for a tiebreak.
Example: An object with the key `prefix/obj` would be merged with an object named `prefix/obj/`.
While this violates the [no object can be a prefix of another](https://min.io/docs/minio/linux/operations/concepts/thresholds.html#conflicting-objects), let's resolve these.
If we have an object with 'name' and a directory named 'name/' discard the directory only - but allow objects
of 'name' and 'name/' (xldir) to be uniquely returned.
Regression from #15772
canceled callers might linger around longer,
can potentially overwhelm the system. Instead
provider a caller context and canceled callers
don't hold on to them.
Bonus: we have no reason to cache errors, we should
never cache errors otherwise we can potentially have
quorum errors creeping in unexpectedly. We should
let the cache when invalidating hit the actual resources
instead.
LastPong is saved as nanoseconds after a connection or reconnection but
saved as seconds when receiving a pong message. The code deciding if
a pong is too old can be skewed since it assumes LastPong is only in
seconds.
Accept multipart uploads where the combined checksum provides the expected part count.
It seems this was added by AWS to make the API more consistent, even if the
data is entirely superfluous on multiple levels.
Improves AWS S3 compatibility.
This commit adds support for MinKMS. Now, there are three KMS
implementations in `internal/kms`: Builtin, MinIO KES and MinIO KMS.
Adding another KMS integration required some cleanup. In particular:
- Various KMS APIs that haven't been and are not used have been
removed. A lot of the code was broken anyway.
- Metrics are now monitored by the `kms.KMS` itself. For basic
metrics this is simpler than collecting metrics for external
servers. In particular, each KES server returns its own metrics
and no cluster-level view.
- The builtin KMS now uses the same en/decryption implemented by
MinKMS and KES. It still supports decryption of the previous
ciphertext format. It's backwards compatible.
- Data encryption keys now include a master key version since MinKMS
supports multiple versions (~4 billion in total and 10000 concurrent)
per key name.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
If used, 'opts.Marker` will cause many missed entries since results are returned
unsorted, and pools are serialized.
Switch to fully concurrent listing and merging across pools to return sorted entries.
It is expected that whoever is using the credentials which has
the proper set of permissions must be able to run.
`mc support perf object`
While the root login is disabled.
fixes#19648
AWS S3 returns the actual object size as part of XML
response for InvalidRange error, this is used apparently
by SDKs to retry the request without the range.
'opts.Marker` is causing many missed entries if used since results are returned unsorted. Also since pools are serialized.
Switch to do fully concurrent listing and merging across pools to return sorted entries.
Returning errors on listings is impossible with the current API, so document that.
Return an error at once if no drives are found instead of just returning an empty listing and no error.
This is to support deployments migrating from a multi-pooled
wider stripe to lower stripe. MINIO_STORAGE_CLASS_STANDARD
is still expected to be same for all pools. So you can satisfy
adding custom drive count based pools by adjusting the storage
class value.
```
version: v2
address: ':9000'
rootUser: 'minioadmin'
rootPassword: 'minioadmin'
console-address: ':9001'
pools: # Specify the nodes and drives with pools
-
args:
- 'node{11...14}.example.net/data{1...4}'
-
args:
- 'node{15...18}.example.net/data{1...4}'
-
args:
- 'node{19...22}.example.net/data{1...4}'
-
args:
- 'node{23...34}.example.net/data{1...10}'
set-drive-count: 6
```
ILM actions due to ExpiredObjectDeleteAllVersions and
DelMarkerExpiration are ignored when object locking is enabled on a
bucket.
Note: This applies to object versions which may not have retention
configured on them. This applies to all object versions in this bucket,
including those created before the retention config was applied.
Per-bucket metrics endpoints always start with /bucket and the bucket
name is appended to the path. e.g. if the collector path is /bucket/api,
the endpoint for the bucket "mybucket" would be
/minio/metrics/v3/bucket/api/mybucket
Change the existing bucket api endpoint accordingly from /api/bucket to
/bucket/api
The `Token` parameter is a sensitive value that should not be output in the Audit log for STS AssumeRoleWithCustomToken API.
Bonus: Add a simple tool that echoes audit logs to the console.
When listing, with drives returning `errFileNotFound,` `errVolumeNotFound`, or `errUnformattedDisk,`,
we could get below `minDisks` drives being left.
This would result in a quorum never being reachable for any object. Therefore, the listing
would continue, but no results would ever be produced.
Include `fnf` in the mindisk check since it is incremented on these errors. This will stop
listing when minDisks are left.
Allow `opts.minDisks` to not return errVolumeNotFound or errFileNotFound and return that.
That will allow for good results even if disks return something else.
We switch `errUnformattedDisk` to a regular error. If we have enough of those, we should just fail.
Typically not all drives are connected, so we delay 3 minutes before resuming.
This greatly reduces risk of starting to list unconnected drives, or drives we risk being disconnected soon.
This delay is not applied when starting with an admin call.
ConsoleUI like applications rely on combination of
ListServiceAccounts() and InfoServiceAccount() to populate
UI elements, however individually these calls can be slow
causing the entire UI to load sluggishly.
i.e., this rule element doesn't apply to DEL markers.
This is a breaking change to how ExpiredObejctDeleteAllVersions
functions today. This is necessary to avoid the following highly probable
footgun scenario in the future.
Scenario:
The user uses tags-based filtering to select an object's time to live(TTL).
The application sometimes deletes objects, too, making its latest
version a DEL marker. The previous implementation skipped tag-based filters
if the newest version was DEL marker, voiding the tag-based TTL. The user is
surprised to find objects that have expired sooner than expected.
* Add DelMarkerExpiration action
This ILM action removes all versions of an object if its
the latest version is a DEL marker.
```xml
<DelMarkerObjectExpiration>
<Days> 10 </Days>
</DelMarkerObjectExpiration>
```
1. Applies only to objects whose,
• The latest version is a DEL marker.
• satisfies the number of days criteria
2. Deletes all versions of this object
3. Associated rule can't have tag-based filtering
Includes,
- New bucket event type for deletion due to DelMarkerExpiration
calling a remote target remove with a perfectly
well constructed ARN can lead to a crash for a bucket
with no replication configured.
This PR fixes, and adds a crash check for ImportMetadata
as well.
Algorithms are comma separated.
Note that valid values does not in all cases represent default values.
`--sftp=pub-key-algos=...` specifies the supported client public key
authentication algorithms. Note that this doesn't include certificate types
since those use the underlying algorithm. This list is sent to the client if
it supports the server-sig-algs extension. Order is irrelevant.
Valid values
```
ssh-ed25519
sk-ssh-ed25519@openssh.comsk-ecdsa-sha2-nistp256@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
rsa-sha2-256
rsa-sha2-512
ssh-rsa
ssh-dss
```
`--sftp=kex-algos=...` specifies the supported key-exchange algorithms in preference order.
Valid values:
```
curve25519-sha256
curve25519-sha256@libssh.org
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group14-sha256
diffie-hellman-group16-sha512
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
```
`--sftp=cipher-algos=...` specifies the allowed cipher algorithms.
If unspecified then a sensible default is used.
Valid values:
```
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.comaes256-gcm@openssh.comchacha20-poly1305@openssh.com
arcfour256
arcfour128
arcfour
aes128-cbc
3des-cbc
```
`--sftp=mac-algos=...` specifies a default set of MAC algorithms in preference order.
This is based on RFC 4253, section 6.4, but with hmac-md5 variants removed because they have
reached the end of their useful life.
Valid values:
```
hmac-sha2-256-etm@openssh.comhmac-sha2-512-etm@openssh.com
hmac-sha2-256
hmac-sha2-512
hmac-sha1
hmac-sha1-96
```
This would reduce the size of data in response of metrics
listing. While graphing we can default these metrics with
a zero value if not found.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Unfreeze as soon as the incoming connection is terminated and don't wait for everything to complete.
We don't want to keep the services frozen if something becomes stuck.
- handle errFileCorrupt properly
- micro-optimization of sending done() response quicker
to close the goroutine.
- fix logger.Event() usage in a couple of places
- handle the rest of the client to return a different error other than
lastErr() when the client is closed.
listPathRaw() counts errDiskNotFound as a valid error to indicate a
listing stream end. However, storage.WalkDir() is allowed to return
errDiskNotFound anytime since grid.ErrDisconnected is converted to
errDiskNotFound.
This affects fresh disk healing and should affect S3 listing as well.
endpoint: /minio/metrics/v3/system/process
metrics:
- locks_read_total
- locks_write_total
- cpu_total_seconds
- go_routine_total
- io_rchar_bytes
- io_read_bytes
- io_wchar_bytes
- io_write_bytes
- start_time_seconds
- uptime_seconds
- file_descriptor_limit_total
- file_descriptor_open_total
- syscall_read_total
- syscall_write_total
- resident_memory_bytes
- virtual_memory_bytes
- virtual_memory_max_bytes
Since the standard process collector implements only a subset of these
metrics, remove it and implement our own custom process collector that
captures all the process metrics we need.
Since the object is being permanently deleted, the lack of read quorum should not
matter as long as sufficient disks are online to complete the deletion with parity
requirements.
If several pools have the same object with insufficient read quorum, attempt to
delete object from all the pools where it exists
At server startup, LDAP configuration is validated against the LDAP
server. If the LDAP server is down at that point, we need to cleanly
disable LDAP configuration. Previously, LDAP would remain configured but
error out in strange ways because initialization did not complete
without errors.
When importing access keys (i.e. service accounts) for LDAP accounts,
we are requiring groups to exist under one of the configured group base
DNs. This is not correct. This change fixes this by only checking for
existence and storing the normalized form of the group DN - we do not
return an error if the group is not under a base DN.
Test is updated to illustrate an import failure that would happen
without this change.
Existing IAM import logic for LDAP creates new mappings when the
normalized form of the mapping key differs from the existing mapping key
in storage. This change effectively replaces the existing mapping key by
first deleting it and then recreating with the normalized form of the
mapping key.
For e.g. if an older deployment had a policy mapped to a user DN -
`UID=alice1,OU=people,OU=hwengg,DC=min,DC=io`
instead of adding a mapping for the normalized form -
`uid=alice1,ou=people,ou=hwengg,dc=min,dc=io`
we should replace the existing mapping.
This ensures that duplicates mappings won't remain after the import.
Some additional cleanup cases are also covered. If there are multiple
mappings for the name normalized key such as:
`UID=alice1,OU=people,OU=hwengg,DC=min,DC=io`
`uid=alice1,ou=people,ou=hwengg,DC=min,DC=io`
`uid=alice1,ou=people,ou=hwengg,dc=min,dc=io`
we check if the list of policies mapped to all these keys are exactly
the same, and if so remove all of them and create a single mapping with
the normalized key. However, if the policies mapped to such keys differ,
the import operation returns an error as the server cannot automatically
pick the "right" list of policies to map.
When inspecting files like `.minio.sys/pool.bin` that may be present on multiple sets, use signature to separate them.
Also fixes null versions to actually be useful with `-export -combine`.
`minio_cluster_webhook_queue_length` was wrongly defined as `counter`
where-as it should be `gauge`
Following were wrongly defined as `gauge` when they should actually be
`counter`:
- minio_bucket_replication_sent_bytes
- minio_bucket_replication_received_bytes
- minio_bucket_replication_total_failed_bytes
- minio_bucket_replication_total_failed_count
When LDAP is enabled, previously we were:
- rejecting creation of users and groups via the IAM import functionality
- throwing a `not a valid DN` error when non-LDAP group mappings are present
This change allows for these cases as we need to support situations
where the MinIO server contains users, groups and policy mappings
created before LDAP was enabled.
instead upon any error in renameData(), we still
preserve the existing dataDir in some form for
recoverability in strange situations such as out
of disk space type errors.
Bonus: avoid running list and heal() instead allow
versions disparity to return the actual versions,
uuid to heal. Currently limit this to 100 versions
and lesser disparate objects.
an undo now reverts back the xl.meta from xl.meta.bkp
during overwrites on such flaky setups.
Bonus: Save N depth syscalls via skipping the parents
upon overwrites and versioned updates.
Flaky setup examples are stretch clusters with regular
packet drops etc, we need to add some defensive code
around to avoid dangling objects.
RenameData could start operating on inline data after timing out
and the call returned due to WithDeadline.
This could cause a buffer to write to the inline data being written.
Since no writes are in `RenameData` and the call is canceled,
this doesn't present a corruption issue. But a race is a race and
should be fixed.
Copy inline data to a fresh buffer.
This PR makes a feasible approach to handle all the scenarios
that we must face to avoid returning "panic."
Instead, we must return "errServerNotInitialized" when a
bucketMetadataSys.Get() is called, allowing the caller to
retry their operation and wait.
Bonus fix the way data-usage-cache stores the object.
Instead of storing usage-cache.bin with the bucket as
`.minio.sys/buckets`, the `buckets` must be relative
to the bucket `.minio.sys` as part of the object name.
Otherwise, there is no way to decommission entries at
`.minio.sys/buckets` and their final erasure set positions.
A bucket must never have a `/` in it. Adds code to read()
from existing data-usage.bin upon upgrade.
This PR fixes a few things
- FIPS support for missing for remote transports, causing
MinIO could end up using non-FIPS Ciphers in FIPS mode
- Avoids too many transports, they all do the same thing
to make connection pooling work properly re-use them.
- globalTCPOptions must be set before setting transport
to make sure the client conn deadlines are honored properly.
- GCS warm tier must re-use our transport
- Re-enable trailing headers support.
This reverts commit 928c0181bf.
This change was not correct, reverting.
We track 3 states with the ProxyRequest header - if replication process wants
to know if object is already replicated with a HEAD, it shouldn't proxy back
- Poorna
AWS S3 trailing header support was recently enabled on the warm tier
client connection to MinIO type remote tiers. With this enabled, we are
seeing the following error message at http transport layer.
> Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request"; err=<nil>
This is an interim fix until we identify the root cause for this behaviour in the
minio-go client package.
Keep the EC in header, so it can be retrieved easily for dynamic quorum calculations.
To not force a full metadata decode on every read the value will be 0/0 for data written in previous versions.
Size is expected to increase by 2 bytes per version, since all valid values can be represented with 1 byte each.
Example:
```
λ xl-meta xl.meta
{
"Versions": [
{
"Header": {
"EcM": 4,
"EcN": 8,
"Flags": 6,
"ModTime": "2024-04-17T11:46:25.325613+02:00",
"Signature": "0a409875",
"Type": 1,
"VersionID": "8e03504e11234957b2727bc53eda0d55"
},
...
```
Not used for operations yet.
Follow up for #19528
If there are multiple existing DN mappings for the same normalized DN,
if they all have the same policy mapping value, we pick one of them of
them instead of returning an import error.
This is a change to IAM export/import functionality. For LDAP enabled
setups, it performs additional validations:
- for policy mappings on LDAP users and groups, it ensures that the
corresponding user or group DN exists and if so uses a normalized form
of these DNs for storage
- for access keys (service accounts), it updates (i.e. validates
existence and normalizes) the internally stored parent user DN and group
DNs.
This allows for a migration path for setups in which LDAP mappings have
been stored in previous versions of the server, where the name of the
mapping file stored on drives is not in a normalized form.
An administrator needs to execute:
`mc admin iam export ALIAS`
followed by
`mc admin iam import ALIAS /path/to/export/file`
The validations are more strict and returns errors when multiple
mappings are found for the same user/group DN. This is to ensure the
mappings stored by the server are unambiguous and to reduce the
potential for confusion.
Bonus **bug fix**: IAM export of access keys (service accounts) did not
export key name, description and expiration. This is fixed in this
change too.
Reading the list metacache is not protected by a lock; the code retries when it fails
to read the metacache object, however, it forgot to re-read the metacache object
from the drives, which is necessary, especially if the metacache object is inlined.
This commit will ensure that we always re-read the metacache object from the drives
when it is retrying.
When resuming a versioned listing where `version-id-marker=null`, the `null` object would
always be returned, causing duplicate entries to be returned.
Add check against empty version
unlinking() at two different locations on a disk when there
are lots to purge, this can lead to huge IOwaits, instead
rely on rename() to .trash to avoid running multiple unlinks()
in parallel.
since mid 2018 we do not have any deployments
without deployment-id, it is time to put this
code to rest, this PR removes this old code as
its no longer valuable.
on setups with 1000's of drives these are all
quite expensive operations.
The rest of the peer clients were not consistent across nodes. So, meta cache requests
would not go to the same server if a continuation happens on a different node.
As node metrics should be scraped per node basis, use a sample
configuartion using all the nodes in targets.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
When no results match or another error occurs, add an error to the stream. Keep the "inspect-input.txt" as the only thing in the zip for reference.
Example:
```
λ mc support inspect --airgap myminio/testbucket/fjghfjh/**
mc: Using public key from C:\Users\klaus\mc\support_public.pem
File data successfully downloaded as inspect-data.enc
λ inspect inspect-data.enc
Using private key from support_private.pem
output written to inspect-data.zip
2024/04/11 14:10:51 next stream: GetRawData: No files matched the given pattern
λ unzip -l inspect-data.zip
Archive: inspect-data.zip
Length Date Time Name
--------- ---------- ----- ----
222 2024-04-11 14:10 inspect-input.txt
--------- -------
222 1 file
λ
```
Modifies inspect to read until end of stream to report the error.
Bonus: Add legacy commandline params
Add following metrics:
- used_inodes
- total_inodes
- healing
- online
- reads_per_sec
- reads_kb_per_sec
- reads_await
- writes_per_sec
- writes_kb_per_sec
- writes_await
- perc_util
To be able to calculate the `per_sec` values, we capture the IOStats-related
data in the beginning (along with the time at which they were captured),
and compare them against the current values subsequently. This is because
dividing by "time since server uptime." doesn't work in k8s environments.
the disk location never changes in the lifetime of a
MinIO cluster, even if it did validate this close to the
disk instead at the higher layer.
Return appropriate errors indicating an invalid drive, so
that the drive is not recognized as part of a valid
drive.
we have had numerous reports on some config
values not having default values, causing
features misbehaving and not having default
values set properly.
This PR tries to address all these concerns
once and for all.
Each new sub-system that gets added
- must check for invalid keys
- must have default values set
- must not "return err" when being saved into
a global state() instead collate as part of
other subsystem errors allow other sub-systems
to independently initialize.
* Allow specifying the local server, with env variable _MINIO_SERVER_LOCAL, in systems where the hostname cannot be resolved to local IP
* Limit scope of the _MINIO_SERVER_LOCAL solution to only containerized implementations
Return an error when the user specifies endpoints for both source
and target. This can generate many type of errors as the code considers
a deployment remote if its endpoint is specified.
HealObject() does not return an error in some cases, for example, when
an object is successfully reconstructed in one disk but fails with other
disks, another case is when a disk does not have the object is temporarily
disconnected
Add the After heal drives result in the audit output for better
analysis.
Set object's modTime when being restored
restored here refers to making a temporary local copy in the hot tier
for a tiered object using the RestoreObject API
we have been using an LRU caching for internode
auth tokens, migrate to using a typed implementation
and also do not cache auth tokens when its an error.
This fixes a regression from #19358 which prevents policy mappings
created in the latest release from being displayed in policy entity
listing APIs.
This is due to the possibility that the base DNs in the LDAP config are
not in a normalized form and #19358 introduced normalized of mapping
keys (user DNs and group DNs). When listing, we check if the policy
mappings are on entities that parse as valid DNs that are descendants of
the base DNs in the config.
Test added that demonstrates a failure without this fix.
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
Using oidc.redirectUri in the values.yaml only works for the deployment.
When using the statefulset the environment variable
MINIO_IDENTITY_OPENID_REDIRECT_URI is not set. This leads to errors with
oicd providers. For example keycloak throws the error 'invalid
redirect_uri'.
This pull request fixes that.
This commit replaces the `KMS.Stat` API call with a
`KMS.GenerateKey` call. This approach is more reliable
since data key generation also works when the KMS backend
is unavailable (temp. offline), but KES has cached the
key. Ref: KES offline caching.
With this change, it is less likely that MinIO readiness
checks fail in cases where the KMS backend is offline.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
Make sure to pass a nil pointer as a Transport to minio-go when the API config
is not initialized, this will make sure that we do not pass an interface
with a known type but a nil value.
This will also fix the update of the API remote_transport_deadline
configuration without requiring the cluster restart.
Use `ODirectPoolSmall` buffers for inline data in PutObject.
Add a separate call for inline data that will fetch a buffer for the inline data before unmarshal.
This fixes a bug where STS Accounts map accumulates accounts in memory
and never removes expired accounts and the STS Policy mappings were not
being refreshed.
The STS purge routine now runs with every IAM credentials load instead
of every 4th time.
The listing of IAM files is now cached on every IAM load operation to
prevent re-listing for STS accounts purging/reload.
Additionally this change makes each server pick a time for IAM loading
that is randomly distributed from a 10 minute interval - this is to
prevent server from thundering while performing the IAM load.
On average, IAM loading will happen between every 5-15min after the
previous IAM load operation completes.
Fix issue [minio#19314], resolve the absence of the sed command in ubi-micro by replacing it with echo.
Signed-off-by: Andreas Bräu <ab@andi95.de>
Co-authored-by: jiuker <2818723467@qq.com>
If site replication enabled across sites, replicate the SSE-C
objects as well. These objects could be read from target sites
using the same client encryption keys.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
User doesn't need to remember and enter the server values,
rather they can select from the pre populated list.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Instead of relying on user input values, we use the DN value returned by
the LDAP server.
This handles cases like when a mapping is set on a DN value
`uid=svc.algorithm,OU=swengg,DC=min,DC=io` with a user input value (with
unicode variation) of `uid=svc﹒algorithm,OU=swengg,DC=min,DC=io`. The
LDAP server on lookup of this DN returns the normalized value where the
unicode dot character `SMALL FULL STOP` (in the user input), gets
replaced with regular full stop.
Bonus: remove persistent md5sum calculation, turn-off
sha256 as well. Instead we always enable crc32c which
is enough for payload verification also support for
trailing headers checksum.
As total drives count, online vs offline are per node basis, its
corect to select node for which graphs need to be rendered.
Set prometheus scrape jobs to fetch metrics from all nodes. A sample
scrape job for node metrics could be as below
```
- job_name: minio-job-node
bearer_token: <token>
metrics_path: /minio/v2/metrics/node
scheme: https
tls_config:
insecure_skip_verify: true
static_configs:
- targets: [tenant1-ss-0-0.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-1.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-2.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-3.tenant1-hl.tenant-ns.svc.cluster.local:9000]
```
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
we must attempt to convert all errors at storage-rest-client
into StorageErr() regardless of what functionality is being
called in, this PR fixes this for multiple callers including
some internally used functions.
- old version was unable to retain messages during config reload
- old version could not go from memory to disk during reload
- new version can batch disk queue entries to single for to reduce I/O load
- error logging has been improved, previous version would miss certain errors.
- logic for spawning/despawning additional workers has been adjusted to trigger when half capacity is reached, instead of when the log queue becomes full.
- old version would json marshall x2 and unmarshal 1x for every log item. Now we only do marshal x1 and then we GetRaw from the store and send it without having to re-marshal.
panic seen due to premature closing of slow channel while listing is still sending or
list has already closed on the sender's side:
```
panic: close of closed channel
goroutine 13666 [running]:
github.com/minio/minio/internal/ioutil.SafeClose[...](0x101ff51e4?)
/Users/kp/code/src/github.com/minio/minio/internal/ioutil/ioutil.go:425 +0x24
github.com/minio/minio/cmd.(*erasureServerPools).Walk.func1()
/Users/kp/code/src/github.com/minio/minio/cmd/erasure-server-pool.go:2142 +0x170
created by github.com/minio/minio/cmd.(*erasureServerPools).Walk in goroutine 1189
/Users/kp/code/src/github.com/minio/minio/cmd/erasure-server-pool.go:1985 +0x228
```
Object names of directory objects qualified for ExpiredObjectAllVersions
must be encoded appropriately before calling on deletePrefix on their
erasure set.
e.g., a directory object and regular objects with overlapping prefixes
could lead to the expiration of regular objects, which is not the
intention of ILM.
```
bucket/dir/ ---> directory object
bucket/dir/obj-1
```
When `bucket/dir/` qualifies for expiration, the current implementation would
remove regular objects under the prefix `bucket/dir/`, in this case,
`bucket/dir/obj-1`.
In handlers related to health diagnostics e.g. CPU, Network, Partitions,
etc, globalMinioHost was being passed as the addr, resulting in empty
value for the same in the health report.
Using globalLocalNodeName instead fixes the issue.
IAM loading is a lazy operation, allow these
fallbacks to be in place when we cannot find
in-memory state().
this allows us to honor the request even if pay
a small price for lookup and populating the data.
When objects have more versions than their ILM policy expects to retain
via NewerNoncurrentVersions, but they don't qualify for expiry due to
NoncurrentDays are configured in that rule.
In this case, applyNewerNoncurrentVersionsLimit method was enqueuing empty
tasks, which lead to a panic (panic: runtime error: index out of range [0] with
length 0) in newerNoncurrentTask.OpHash method, which assumes the task
to contain at least one version to expire.
When returning the status of a decommissioned pool, a pool with zero
time StartedTime will be considered an active pool, which is unexpected.
This commit will always ensure that a pool's canceled/failed/completed
status is returned.
This commit changes how MinIO generates the object encryption key (OEK)
when encrypting an object using server-side encryption.
This change is fully backwards compatible. Now, MinIO generates
the OEK as following:
```
Nonce = RANDOM(32) // generate 256 bit random value
OEK = HMAC-SHA256(EK, Context || Nonce)
```
Before, the OEK was computed as following:
```
Nonce = RANDOM(32) // generate 256 bit random value
OEK = SHA256(EK || Nonce)
```
The new scheme does not technically fix a security issue but
uses a more familiar scheme. The only requirement for the
OEK generation function is that it produces a (pseudo)random value
for every pair (`EK`,`Nonce`) as long as no `EK`-`Nonce` combination
is repeated. This prevents a faulty PRNG from repeating or generating
a "bad" key.
The previous scheme guarantees that the `OEK` is a (pseudo)random
value given that no pair (`EK`,`Nonce`) repeats under the assumption
that SHA256 is indistinguable from a random oracle.
The new scheme guarantees that the `OEK` is a (pseudo)random value
given that no pair (`EK`, `Nonce`) repeats under the assumption that
SHA256's underlying compression function is a PRF/PRP.
While the later is a weaker assumption, and therefore, less likely
to be false, both are considered true. SHA256 is believed to be
indistinguable from a random oracle AND its compression function
is assumed to be a PRF/PRP.
As far as the OEK generating is concerned, the OS random number
generator is not required to be pseudo-random but just non-repeating.
Apart from being more compatible to standard definitions and
descriptions for how to generate crypto. keys, this change does not
have any impact of the actual security of the OEK key generation.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
avoids error during upgrades such as
```
API: SYSTEM()
Time: 19:19:22 UTC 03/18/2024
DeploymentID: 24e4b574-b28d-4e94-9bfa-03c363a600c2
Error: Invalid api configuration: found invalid keys (expiry_workers=100 ) for 'api' sub-system, use 'mc admin config reset myminio api' to fix invalid keys (*fmt.wrapError)
11: internal/logger/logger.go:260:logger.LogIf()
...
```
we were prematurely not writing 4k pages while we
could have due to the fact that most buffers would
be multiples of 4k upto some number and there shall
be some remainder.
We only need to write the remainder without O_DIRECT.
at scale customers might start with failed drives,
causing skew in the overall usage ratio per EC set.
make this configurable such that customers can turn
this off as needed depending on how comfortable they
are.
Cosmetic change, but breaks up a big code block and will make a goroutine
dumps of streams are more readable, so it is clearer what each goroutine is doing.
Currently, the code relies on object parity to decide whether it is a
delete marker or a regular object. In the case of a delete marker, the
return quorum is half of the disks in the erasure set. However, this
calculation must be corrected with objects with EC = 0, mainly
because EC is not a one-time fixed configuration.
Though all data are correct, the manifested symptom is a 503 with an
EC=0 object. This bug was manifested after we introduced the
fast Get Object feature that does not read all data from all disks in
case of inlined objects
Metrics v3 is mainly a reorganization of metrics into smaller groups of
metrics and the removal of internal aggregation of metrics received from
peer nodes in a MinIO cluster.
This change adds the endpoint `/minio/metrics/v3` as the top-level metrics
endpoint and under this, various sub-endpoints are implemented. These
are currently documented in `docs/metrics/v3.md`
The handler will serve metrics at any path
`/minio/metrics/v3/PATH`, as follows:
when PATH is a sub-endpoint listed above => serves the group of
metrics under that path; or when PATH is a (non-empty) parent
directory of the sub-endpoints listed above => serves metrics
from each child sub-endpoint of PATH. otherwise, returns a no
resource found error
All available metrics are listed in the `docs/metrics/v3.md`. More will
be added subsequently.
When an object qualifies for both tiering and expiration rules and is
past its expiration date, it should be expired without requiring to tier
it, even when tiering event occurs before expiration.
Merging same-object - multiple versions from different pools would not always result in correct ordering.
When merging keep inputs separate.
```
λ mc ls --versions local/testbucket
------ before ------
[2024-03-05 20:17:19 CET] 228B STANDARD 1f163718-9bc5-4b01-bff7-5d8cf09caf10 v3 PUT hosts
[2024-03-05 20:19:56 CET] 19KiB STANDARD null v2 PUT hosts
[2024-03-05 20:17:15 CET] 228B STANDARD 73c9f651-f023-4566-b012-cc537fdb7ce2 v1 PUT hosts
------ after ------
λ mc ls --versions local/testbucket
[2024-03-05 20:19:56 CET] 19KiB STANDARD null v3 PUT hosts
[2024-03-05 20:17:19 CET] 228B STANDARD 1f163718-9bc5-4b01-bff7-5d8cf09caf10 v2 PUT hosts
[2024-03-05 20:17:15 CET] 228B STANDARD 73c9f651-f023-4566-b012-cc537fdb7ce2 v1 PUT hosts
```
configure batch size to send audit/logger events
in batches instead of sending one event per connection.
this is mainly to optimize the number of requests
we make to webhook endpoint.
Currently, the progress of the batch job is saved in inside the job
request object, which is normally not supported by MinIO. Though there
is no apparent bug, it is better to fix this now.
Batch progress is saved in .minio.sys/batch-jobs/reports/
Co-authored-by: Anis Eleuch <anis@min.io>
our PoolNumber calculation was costly,
while we already had this information per
endpoint, we needed to deduce it appropriately.
This PR addresses this by assigning PoolNumbers
field that carries all the pool numbers that
belong to a server.
properties.PoolNumber still carries a valid value
only when len(properties.PoolNumbers) == 1, otherwise
properties.PoolNumber is set to math.MaxInt (indicating
that this value is undefined) and then one must rely
on properties.PoolNumbers for server participation
in multiple pools.
addresses the issue originating from #11327
simplify audit webhook worker model
fixes couple of bugs like
- ping(ctx) was creating a logger without updating
number of workers leading to incorrect nWorkers
scaling, causing an additional worker that is not
tracked properly.
- h.logCh <- entry could potentially hang for when
the queue is full on heavily loaded systems.
there can be a sudden spike in tiny allocations,
due to too much auditing being done, also don't hang
on the
```
h.logCh <- entry
```
after initializing workers if you do not have a way to
dequeue for some reason.
This commits adds support for using the `--endpoint` arg when creating a
tier of type `azure`. This is needed to connect to Azure's Gov Cloud
instance. For example,
```
mc ilm tier add azure TARGET TIER_NAME \
--account-name ACCOUNT \
--account-key KEY \
--bucket CONTAINER \
--endpoint https://ACCOUNT.blob.core.usgovcloudapi.net
--prefix PREFIX \
--storage-class STORAGE_CLASS
```
Prior to this, the endpoint was hardcoded to `https://ACCOUNT.blob.core.windows.net`.
The docs were even explicit about this, stating that `--endpoint` is:
"Required for `s3` or `minio` tier types. This option has no effect for any
other value of `TIER_TYPE`."
Now, if the endpoint arg is present it will be used. If not, it will
fall back to the same default behavior of `ACCOUNT.blob.core.windows.net`.
Remove api.expiration_workers config setting which was inadvertently left behind. Per review comment
https://github.com/minio/minio/pull/18926, expiration_workers can be configured via ilm.expiration_workers.
ext4, xfs support this behavior however
btrfs, nfs may not support it properly.
in-case when we see Nlink < 2 then we know
that we need to fallback on readdir()
fixes a regression from #19100fixes#19181
The middleware sets up tracing, throttling, gzipped responses and
collecting API stats.
Additionally, this change updates the names of handler functions in
metric labels to be the same as the name derived from Go lang reflection
on the handler name.
The metric api labels are now stored in memory the same as the handler
name - they will be camelcased, e.g. `GetObject` instead of `getobject`.
For compatibility, we lowercase the metric api label values when emitting the metrics.
- Use a shared worker pool for all ILM expiry tasks
- Free version cleanup executes in a separate goroutine
- Add a free version only if removing the remote object fails
- Add ILM expiry metrics to the node namespace
- Move tier journal tasks to expiryState
- Remove unused on-disk journal for tiered objects pending deletion
- Distribute expiry tasks across workers such that the expiry of versions of
the same object serialized
- Ability to resize worker pool without server restart
- Make scaling down of expiryState workers' concurrency safe; Thanks
@klauspost
- Add error logs when expiryState and transition state are not
initialized (yet)
* metrics: Add missed tier journal entry tasks
* Initialize the ILM worker pool after the object layer
With this commit, MinIO generates root credentials automatically
and deterministically if:
- No root credentials have been set.
- A KMS (KES) is configured.
- API access for the root credentials is disabled (lockdown mode).
Before, MinIO defaults to `minioadmin` for both the access and
secret keys. Now, MinIO generates unique root credentials
automatically on startup using the KMS.
Therefore, it uses the KMS HMAC function to generate pseudo-random
values. These values never change as long as the KMS key remains
the same, and the KMS key must continue to exist since all IAM data
is encrypted with it.
Backward compatibility:
This commit should not cause existing deployments to break. It only
changes the root credentials of deployments that have a KMS configured
(KES, not a static key) but have not set any admin credentials. Such
implementations should be rare or not exist at all.
Even if the worst case would be updating root credentials in mc
or other clients used to administer the cluster. Root credentials
are anyway not intended for regular S3 operations.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
just like client-conn-read-deadline, added a new flag that does
client-conn-write-deadline as well.
Both are not configured by default, since we do not yet know
what is the right value. Allow this to be configurable if needed.
we should do this to ensure that we focus on
data healing as primary focus, fixing metadata
as part of healing must be done but making
data available is the main focus.
the main reason is metadata inconsistencies can
cause data availability issues, which must be
avoided at all cost.
will be bringing in an additional healing mechanism
that involves "metadata-only" heal, for now we do
not expect to have these checks.
continuation of #19154
Bonus: add a pro-active healthcheck to perform a connection
This change makes the label names consistent with the handler names.
This is in preparation to use reflection based API handler function
names for the api labels so they will be the same as tracing, auditing
and logging names for these API calls.
Moved different dashboards to their specific directories. Also
mentioned that these dashbards are examples of how to create
graphs using MinIO provided and metrics and customers should
change / add graphs on their specific need basis.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Streams can return errors if the cancelation is picked up before the response
stream close is picked up. Under extreme load, this could lead to missing
responses.
Send server mux ack async so a blocked send cannot block newMuxStream
call. Stream will not progress until mux has been acked.
in k8s things really do come online very asynchronously,
we need to use implementation that allows this randomness.
To facilitate this move WriteAll() as part of the
websocket layer instead.
Bonus: avoid instances of dnscache usage on k8s
New disk healing code skips/expires objects that ILM supposed to expire.
Add more visibility to the user about this activity by calculating those
objects and print it at the end of healing activity.
This PR fixes a bug that perhaps has been long introduced,
with no visible workarounds. In any deployment, if an entire
erasure set is deleted, there is no way the cluster recovers.
Currently, if one object tag matches with one lifecycle tag filter, ILM
will select it, however, this is wrong. All the Tag filters in the
lifecycle document should be satisfied.
This change is to decouple need for root credentials to match between
site replication deployments.
Also ensuring site replication config initialization is re-tried until
it succeeds, this deoendency is critical to STS flow in site replication
scenario.
Currently, we read from `/proc/diskstats` which is found to be
un-reliable in k8s environments. We can read from `sysfs` instead.
Also, cache the latest drive io stats to find the diff and update
the metrics.
* Remove lock for cached operations.
* Rename "Relax" to `ReturnLastGood`.
* Add `CacheError` to allow caching values even on errors.
* Add NoWait that will return current value with async fetching if within 2xTTL.
* Make benchmark somewhat representative.
```
Before: BenchmarkCache-12 16408370 63.12 ns/op 0 B/op
After: BenchmarkCache-12 428282187 2.789 ns/op 0 B/op
```
* Remove `storageRESTClient.scanning`. Nonsensical - RPC clients will not have any idea about scanning.
* Always fetch remote diskinfo metrics and cache them. Seems most calls are requesting metrics.
* Do async fetching of usage caches.
It also fixes a long-standing bug in expiring transitioned objects.
The expiration action was deleting the current version in the case'
of tiered objects instead of adding a delete marker.
only enable md5sum if explicitly asked by the client, otherwise
its not necessary to compute md5sum when SSE-KMS/SSE-C is enabled.
this is continuation of #17958
If network conditions have filled the output queue before a reconnect happens blocked sends could stop reconnects from happening. In short `respMu` would be held for a mux client while sending - if the queue is full this will never get released and closing the mux client will hang.
A) Use the mux client context instead of connection context for sends, so sends are unblocked when the mux client is canceled.
B) Use a `TryLock` on "close" and cancel the request if we cannot get the lock at once. This will unblock any attempts to send.
data-dir not being present is okay, however we can still
rely on the `rename()` atomic call instead of relying on
write xl.meta write which may truncate the io.EOF.
Add a new function logger.Event() to send the log to Console and
http/kafka log webhooks. This will include some internal events such as
disk healing and rebalance/decommissioning
the PR in #16541 was incorrect and hand wrong assumptions
about the overall setup, revert this since this expectation
to have offline servers is wrong and we can end up with a
bigger chicken and egg problem.
This reverts commit 5996c8c4d5.
Bonus:
- preserve disk in globalLocalDrives properly upon connectDisks()
- do not return 'nil' from newXLStorage(), getting it ready for
the next set of changes for 'format.json' loading.
The previous logic of calculating per second values for disk io stats
divides the stats by the host uptime. This doesn't work in k8s
environment as the uptime is of the pod, but the stats (from
/proc/diskstats) are from the host.
Fix this by storing the initial values of uptime and the stats at the
timme of server startup, and using the difference between current and
initial values when calculating the per second values.
globalLocalDrives seem to be not updated during the
HealFormat() leads to a requirement where the server
needs to be restarted for the healing to continue.
a/prefix
a/prefix/1.txt
where `a/prefix` is an object which does not have `/` at the end,
we do not have to aggressively recursively delete all the sub-folders
as well. Instead convert the call into self contained to deleting
'xl.meta' and then subsequently attempting to Remove the parent.
Bonus: enable audit alerts for object versions
beyond the configured value, default is '100'
versions per object beyond which scanner will
alert for each such objects.
when we expand via pools, there is no reason to stick
with the same distributionAlgo as the rest. Since the
algo only makes sense with-in a pool not across pools.
This allows for newer pools to use newer codepaths to
avoid legacy file lookups when they have a pre-existing
deployment from 2019, they can expand their new pool
to be of a newer distribution format, allowing the
pool to be more performant.
Fix reported races that are actually synchronized by network calls.
But this should add some extra safety for untimely disconnects.
Race reported:
```
WARNING: DATA RACE
Read at 0x00c00171c9c0 by goroutine 214:
github.com/minio/minio/internal/grid.(*muxClient).addResponse()
e:/gopath/src/github.com/minio/minio/internal/grid/muxclient.go:519 +0x111
github.com/minio/minio/internal/grid.(*muxClient).error()
e:/gopath/src/github.com/minio/minio/internal/grid/muxclient.go:470 +0x21d
github.com/minio/minio/internal/grid.(*Connection).handleDisconnectClientMux()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:1391 +0x15b
github.com/minio/minio/internal/grid.(*Connection).handleMsg()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:1190 +0x1ab
github.com/minio/minio/internal/grid.(*Connection).handleMessages.func1()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:981 +0x610
Previous write at 0x00c00171c9c0 by goroutine 1081:
github.com/minio/minio/internal/grid.(*muxClient).roundtrip()
e:/gopath/src/github.com/minio/minio/internal/grid/muxclient.go:94 +0x324
github.com/minio/minio/internal/grid.(*muxClient).traceRoundtrip()
e:/gopath/src/github.com/minio/minio/internal/grid/trace.go:74 +0x10e4
github.com/minio/minio/internal/grid.(*Subroute).Request()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:366 +0x230
github.com/minio/minio/internal/grid.(*SingleHandler[go.shape.*github.com/minio/minio/cmd.DiskInfoOptions,go.shape.*github.com/minio/minio/cmd.DiskInfo]).Call()
e:/gopath/src/github.com/minio/minio/internal/grid/handlers.go:554 +0x3fd
github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo()
e:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:314 +0x270
github.com/minio/minio/cmd.erasureObjects.getOnlineDisksWithHealingAndInfo.func1()
e:/gopath/src/github.com/minio/minio/cmd/erasure.go:293 +0x171
```
This read will always happen after the write, since there is a network call in between.
However a disconnect could come in while we are setting up the call, so we protect against that with extra checks.
- bucket metadata does not need to look for legacy things
anymore if b.Created is non-zero
- stagger bucket metadata loads across lots of nodes to
avoid the current thundering herd problem.
- Remove deadlines for RenameData, RenameFile - these
calls should not ever be timed out and should wait
until completion or wait for client timeout. Do not
choose timeouts for applications during the WRITE phase.
- increase R/W buffer size, increase maxMergeMessages to 30
We have observed cases where a blocked stream will block for cancellations.
This happens when response channel is blocked and we want to push an error.
This will have the response mutex locked, which will prevent all other operations until upstream is unblocked.
Make this behavior non-blocking and if blocked spawn a goroutine that will send the response and close the output.
Still a lot of "dancing". Added a test for this and reviewed.
Depending on when the context cancelation is picked up the handler may return and close the channel before `SubscribeJSON` returns, causing:
```
Feb 05 17:12:00 s3-us-node11 minio[3973657]: panic: send on closed channel
Feb 05 17:12:00 s3-us-node11 minio[3973657]: goroutine 378007076 [running]:
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub.(*PubSub[...]).SubscribeJSON.func1()
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub/pubsub.go:139 +0x12d
Feb 05 17:12:00 s3-us-node11 minio[3973657]: created by github.com/minio/minio/internal/pubsub.(*PubSub[...]).SubscribeJSON in goroutine 378010884
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub/pubsub.go:124 +0x352
```
Wait explicitly for the goroutine to exit.
Bonus: Listen for doneCh when sending to not risk getting blocked there is channel isn't being emptied.
this fixes rare bugs we have seen but never really found a
reproducer for
- PutObjectRetention() returning 503s
- PutObjectTags() returning 503s
- PutObjectMetadata() updates during replication returning 503s
These calls return errors, and this perpetuates with
no apparent fix.
This PR fixes with correct quorum requirement.
To force limit the duration of STS accounts, the user can create a new
policy, like the following:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {"NumericLessThanEquals": {"sts:DurationSeconds": "300"}}
}]
}
And force binding the policy to all OpenID users, whether using a claim name or role
ARN.
disk tokens usage is not necessary anymore with the implementation
of deadlines for storage calls and active monitoring of the drive
for I/O timeouts.
Functionality kicking off a bad drive is still supported, it's just that
we do not have to serialize I/O in the manner tokens would do.
Recycle would always be called on the dummy value `any(newRT())` instead of the actual value given to the recycle function.
Caught by race tests, but mostly harmless, except for reduced perf.
Other minor cleanups. Introduced in #18940 (unreleased)
Interpret `null` inline policy for access keys as inheriting parent
policy. Since MinIO Console currently sends this value, we need to honor it
for now. A larger fix in Console and in the server are required.
Fixes#18939.
Allow internal types to support a `Recycler` interface, which will allow for sharing of common types across handlers.
This means that all `grid.MSS` (and similar) objects are shared across in a common pool instead of a per-handler pool.
Add internal request reuse of internal types. Add for safe (pointerless) types explicitly.
Only log params for internal types. Doing Sprint(obj) is just a bit too messy.
With this change, only a user with `UpdateServiceAccountAdminAction`
permission is able to edit access keys.
We would like to let a user edit their own access keys, however the
feature needs to be re-designed for better security and integration with
external systems like AD/LDAP and OpenID.
This change prevents privilege escalation via service accounts.
for actionable, inspections we have `mc support inspect`
we do not need double logging, healing will report relevant
errors if any, in terms of quorum lost etc.
Each Put, List, Multipart operations heavily rely on making
GetBucketInfo() call to verify if bucket exists or not on
a regular basis. This has a large performance cost when there
are tons of servers involved.
We did optimize this part by vectorizing the bucket calls,
however its not enough, beyond 100 nodes and this becomes
fairly visible in terms of performance.
- healing must not set the write xattr
because that is the job of active healing
to update. what we need to preserve is
permanent deletes.
- remove older env for drive monitoring and
enable it accordingly, as a global value.
local disk metrics were polluting cluster metrics
Please remove them instead of adding relevant ones.
- batch job metrics were incorrectly kept at bucket
metrics endpoint, move it to cluster metrics.
- add tier metrics to cluster peer metrics from the node.
- fix missing set level cluster health metrics
Do not rely on `connChange` to do reconnects.
Instead, you can block while the connection is running and reconnect
when handleMessages returns.
Add fully async monitoring instead of monitoring on the main goroutine
and keep this to avoid full network lockup.
it is entirely possible that a rebalance process which was running
when it was asked to "stop" it failed to write its last statistics
to the disk.
After this a pool expansion can cause disruption and all S3 API
calls would fail at IsPoolRebalancing() function.
This PRs makes sure that we update rebalance.bin under such
conditions to avoid any runtime crashes.
add new update v2 that updates per node, allows idempotent behavior
new API ensures that
- binary is correct and can be downloaded checksummed verified
- committed to actual path
- restart returns back the relevant waiting drives
do not need to be defensive in our approach,
we should simply override anything everything
in import process, do not care about what
currently exists on the disk - backup is the
source of truth.
Right now the format.json is excluded if anything within `.minio.sys` is requested.
I assume the check was meant to exclude only if it was actually requesting it.
- Move RenameFile to websockets
- Move ReadAll that is primarily is used
for reading 'format.json' to to websockets
- Optimize DiskInfo calls, and provide a way
to make a NoOp DiskInfo call.
Add separate reconnection mutex
Give more safety around reconnects and make sure a state change isn't missed.
Tested with several runs of `λ go test -race -v -count=500`
Adds separate mutex and doesn't mix in the testing mutex.
AlmosAll uses of NewDeadlineWorker, which relied on secondary values, were used in a racy fashion,
which could lead to inconsistent errors/data being returned. It also propagates the deadline downstream.
Rewrite all these to use a generic WithDeadline caller that can return an error alongside a value.
Remove the stateful aspect of DeadlineWorker - it was racy if used - but it wasn't AFAICT.
Fixes races like:
```
WARNING: DATA RACE
Read at 0x00c130b29d10 by goroutine 470237:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).ReadVersion()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:702 +0x611
github.com/minio/minio/cmd.readFileInfo()
github.com/minio/minio/cmd/erasure-metadata-utils.go:160 +0x122
github.com/minio/minio/cmd.erasureObjects.getObjectFileInfo.func1.1()
github.com/minio/minio/cmd/erasure-object.go:809 +0x27a
github.com/minio/minio/cmd.erasureObjects.getObjectFileInfo.func1.2()
github.com/minio/minio/cmd/erasure-object.go:828 +0x61
Previous write at 0x00c130b29d10 by goroutine 470298:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).ReadVersion.func1()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:698 +0x244
github.com/minio/minio/internal/ioutil.(*DeadlineWorker).Run.func1()
github.com/minio/minio/internal/ioutil/ioutil.go:141 +0x33
WARNING: DATA RACE
Write at 0x00c0ba6e6c00 by goroutine 94507:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).StatVol.func1()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:419 +0x104
github.com/minio/minio/internal/ioutil.(*DeadlineWorker).Run.func1()
github.com/minio/minio/internal/ioutil/ioutil.go:141 +0x33
Previous read at 0x00c0ba6e6c00 by goroutine 94463:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).StatVol()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:422 +0x47e
github.com/minio/minio/cmd.getBucketInfoLocal.func1()
github.com/minio/minio/cmd/peer-s3-server.go:275 +0x122
github.com/minio/pkg/v2/sync/errgroup.(*Group).Go.func1()
```
Probably back from #17701
protection was in place. However, it covered only some
areas, so we re-arranged the code to ensure we could hold
locks properly.
Along with this, remove the DataShardFix code altogether,
in deployments with many drive replacements, this can affect
and lead to quorum loss.
Also limit the amount of concurrency when sending
binary updates to peers, avoid high network over
TX that can cause disconnection events for the
node sending updates.
Race checks would occasionally show race on handleMsgWg WaitGroup by debug messages (used in test only).
Use the `connMu` mutex to protect this against concurrent Wait/Add.
Fixes#18827
If site replication is enabled, we should still show the size and
version distribution histogram metrics at bucket level.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
New API now verifies any hung disks before restart/stop,
provides a 'per node' break down of the restart/stop results.
Provides also how many blocked syscalls are present on the
drives and what users must do about them.
Adds options to do pre-flight checks to provide information
to the user regarding any hung disks. Provides 'force' option
to forcibly attempt a restart() even with waiting syscalls
on the drives.
When rejecting incoming grid requests fill out the rejection reason and log it once.
This will give more context when startup is failing. Already logged after a retry on caller.
On a policy detach operation, if there are no policies remaining
attached to the user/group, remove the policy mapping file, instead of
leaving a file containing an empty list of policies.
Healing dangling buckets is conservative, and it is a typical use case to
fail to remove a dangling bucket because it contains some data because
healing danging bucket code is not allowed to remove data: only healing
the dangling object is allowed to do so.
reference format is constant for any lifetime of
a minio cluster, we do not have to ever replace
it during HealFormat() as it will never change.
additionally we should simply reject reference
formats that we do not understand early on.
GetActualSize() was heavily relying on o.Parts()
to be non-empty to figure out if the object is multipart or not,
However, we have many indicators of whether an object is multipart
or not.
Blindly assuming that o.Parts == nil is not a multipart, is an
incorrect expectation instead, multipart must be obtained via
- Stored metadata value indicating this is a multipart encrypted object.
- Rely on <meta>-actual-size metadata to get the object's actual size.
This value is preserved for additional reasons such as these.
- ETag != 32 length
support proxying of tagging requests in active-active replication
Note: even if proxying is successful, PutObjectTagging/DeleteObjectTagging
will continue to report a 404 since the object is not present locally.
New intervals:
[1024B, 64KiB)
[64KiB, 256KiB)
[256KiB, 512KiB)
[512KiB, 1MiB)
The new intervals helps us see object size distribution with higher
resolution for the interval [1024B, 1MiB).
- HealFormat() was leaking healthcheck goroutines for
disks, we are only interested in enabling healthcheck
for the newly formatted disk, not for existing disks.
- When disk is a root-disk a random disk monitor was
leaking while we ignored the drive.
- When loading the disk for each erasure set, we were
leaking goroutines for the prepare-storage.go disks
which were replaced via the globalLocalDrives slice
- avoid disk monitoring utilizing health tokens that
would cause exhaustion in the tokens, prematurely
which were meant for incoming I/O. This is ensured
by avoiding writing O_DIRECT aligned buffer instead
write 2048 worth of content only as O_DSYNC, which is
sufficient.
Add a hidden configuration under the scanner sub section to configure if
the scanner should sleep between two objects scan. The configuration has
only effect when there is no drive activity related to s3 requests or
healing.
By default, the code will keep the current behavior which is doing
sleep between objects.
To forcefully enable the full scan speed in idle mode, you can do this:
`mc admin config set myminio scanner idle_speed=full`
fixes#18724
A regression was introduced in #18547, that attempted
to file adding a missing `null` marker however we
should not skip returning based on versionID instead
it must be based on if we are being asked to create
a DEL marker or not.
The PR also has a side-affect for replicating `null`
marker permanent delete, as it may end up adding a
`null` marker while removing one.
This PR should address both scenarios.
NOTE: This feature is not retro-active; it will not cater to previous transactions
on existing setups.
To enable this feature, please set ` _MINIO_DRIVE_QUORUM=on` environment
variable as part of systemd service or k8s configmap.
Once this has been enabled, you need to also set `list_quorum`.
```
~ mc admin config set alias/ api list_quorum=auto`
```
A new debugging tool is available to check for any missing counters.
Following policies if present
```
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"54.240.143.0/24",
"2001:DB8:1234:5678::/64"
]
}
}
```
And client is making a request to MinIO via IPv6 can
potentially crash the server.
Workarounds are turn-off IPv6 and use only IPv4
This PR also increases per node bpool memory from 1024 entries
to 2048 entries; along with that, it also moves the byte pool
centrally instead of being per pool.
minio_node_tier_ttlb_seconds - Distribution of time to last byte for streaming objects from warm tier
minio_node_tier_requests_success - Number of requests to download object from warm tier that were successful
minio_node_tier_requests_failure - Number of requests to download object from warm tier that failed
SUBNET now has a v2 of license that is returned in the new key
`license_v2`. mc will start reading and storing the same. (The old key
`license` is deprecated but is still available in SUBNET response to
ensure that the current released version of minio doesn't break)
`(*xlStorageDiskIDCheck).CreateFile` wraps the incoming reader in `xioutil.NewDeadlineReader`.
The wrapped reader is handed to `(*xlStorage).CreateFile`. This performs a Read call via `writeAllDirect`,
which reads into an `ODirectPool` buffer.
`(*DeadlineReader).Read` spawns an async read into the buffer. If a timeout is hit while reading,
the read operation returns to `writeAllDirect`. The operation returns an error and the buffer is reused.
However, if the async `Read` call unblocks, it will write to the now recycled buffer.
Fix: Remove the `DeadlineReader` - it is inherently unsafe. Instead, rely on the network timeouts.
This is not a disk timeout, anyway.
Regression in https://github.com/minio/minio/pull/17745
This patch adds the targetID to the existing notification target metrics
and deprecates the current target metrics which points to the overall
event notification subsystem
historically, we have always kept storage-rest-server
and a local storage API separate without much trouble,
since they both can independently operate due to no
special state() between them.
however, over some time, we have added state()
such as
- drive monitoring threads now there will be "2" of
them per drive instead of just 1.
- concurrent tokens available per drive are now twice
instead of just single shared, allowing unexpectedly
high amount of I/O to go through.
- applying serialization by using walkMutexes can now
be adequately honored for both remote callers and local
callers.
Regression from #18285. CopyObject options were inheriting source MTime
for metadata timestamps if unspecified, removing this prevented metadata
updates from being applied on target.
The metrics `minio_bucket_replication_received_bytes` and
`minio_bucket_replication_sent_bytes` are additive in nature
and rendering the value as is looks fine.
Also added sort order for few graphs for better reading of tool
tips as keeping ones with highest value at top helps.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
By default the cpu load is the cumulative of all cores. Capture the
percentage load (load * 100 / cpu-count)
Also capture the percentage memory used (used * 100 / total)
use memory for async events when necessary and dequeue them as
needed, for all synchronous events customers must enable
```
MINIO_API_SYNC_EVENTS=on
```
Async events can be lost but is upto to the admin to
decide what they want, we will not create run-away number
of goroutines per event instead we will queue them properly.
Currently the max async workers is set to runtime.GOMAXPROCS(0)
which is more than sufficient in general, but it can be made
configurable in future but may not be needed.
there is potential for danglingWrites when quorum failed, where
only some drives took a successful write, generally this is left
to the healing routine to pick it up. However it is better that
we delete it right away to avoid potential for quorum issues on
version signature when there are many versions of an object.
it is okay if the warm-tier cannot keep up, we should continue
to take I/O at hot-tier, only fail hot-tier or block it when
we are disk full.
Bonus: add metrics counter for these missed tasks, we will
know for sure if one of the node is lagging behind or is
losing too many tasks during transitioning.
A disk that is not able to initialize when an instance is started
will never have a handler registered, which means a user will
need to restart the node after fixing the disk;
This will also prevent showing the wrong 'upgrade is needed.'
error message in that case.
When the disk is still failing, print an error every 30 minutes;
Disk reconnection will be retried every 30 seconds.
Co-authored-by: Anis Elleuch <anis@min.io>
`OpMuxConnectError` was not handled correctly.
Remove local checks for single request handlers so they can
run before being registered locally.
Bonus: Only log IAM bootstrap on startup.
```
using deb packager...
created package: minio-release/linux-amd64/minio_20231120224007.0.0.hotfix.e96ac7272_amd64.deb
using rpm packager...
created package: minio-release/linux-amd64/minio-20231120224007.0.0.hotfix.e96ac7272-1.x86_64.rpm
```
While healing the latest changes of expiry rules across sites
if target had pre existing transition rules, they were getting
overwritten as cloned latest expiry rules from remote site were
getting written as is. Fixed the same and added test cases as
well.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
moveToTrash() function moves a folder to .trash, for example, when
doing some object deletions: a data dir that has many parts will be
renamed to the trash folder; However, ENOSPC is a valid error from
rename(), and it can cripple a user trying to free some space in an
entire disk situation.
Therefore, this commit will try to do a recursive delete in that case.
This allows batch replication to basically do not
attempt to copy objects that do not have read quorum.
This PR also allows walk() to provide custom
values for quorum under batch replication, and
key rotation.
this PR allows following policy
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Deny a presigned URL request if the signature is more than 10 min old",
"Effect": "Deny",
"Action": "s3:*",
"Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET1/*",
"Condition": {
"NumericGreaterThan": {
"s3:signatureAge": 600000
}
}
}
]
}
```
This is to basically disable all pre-signed URLs that are older than 10 minutes.
AWS S3 closes keep-alive connections frequently
leading to frivolous logs filling up the MinIO
logs when the transition tier is an AWS S3 bucket.
Ignore such transient errors, let MinIO retry
it when it can.
When minio runs with MINIO_CI_CD=on, it is expected to communicate
with the locally running SUBNET. This is happening in the case of MinIO
via call home functionality. However, the subnet-related functionality inside the
console continues to talk to the SUBNET production URL. Because of this,
the console cannot be tested with a locally running SUBNET.
Set the env variable CONSOLE_SUBNET_URL correctly in such cases.
(The console already has code to use the value of this variable
as the subnet URL)
Optionally allows customers to enable
- Enable an external cache to catch GET/HEAD responses
- Enable skipping disks that are slow to respond in GET/HEAD
when we have already achieved a quorum
Bonus: allow replication to attempt Deletes/Puts when
the remote returns quorum errors of some kind, this is
to ensure that MinIO can rewrite the namespace with the
latest version that exists on the source.
This PR adds a WebSocket grid feature that allows servers to communicate via
a single two-way connection.
There are two request types:
* Single requests, which are `[]byte => ([]byte, error)`. This is for efficient small
roundtrips with small payloads.
* Streaming requests which are `[]byte, chan []byte => chan []byte (and error)`,
which allows for different combinations of full two-way streams with an initial payload.
Only a single stream is created between two machines - and there is, as such, no
server/client relation since both sides can initiate and handle requests. Which server
initiates the request is decided deterministically on the server names.
Requests are made through a mux client and server, which handles message
passing, congestion, cancelation, timeouts, etc.
If a connection is lost, all requests are canceled, and the calling server will try
to reconnect. Registered handlers can operate directly on byte
slices or use a higher-level generics abstraction.
There is no versioning of handlers/clients, and incompatible changes should
be handled by adding new handlers.
The request path can be changed to a new one for any protocol changes.
First, all servers create a "Manager." The manager must know its address
as well as all remote addresses. This will manage all connections.
To get a connection to any remote, ask the manager to provide it given
the remote address using.
```
func (m *Manager) Connection(host string) *Connection
```
All serverside handlers must also be registered on the manager. This will
make sure that all incoming requests are served. The number of in-flight
requests and responses must also be given for streaming requests.
The "Connection" returned manages the mux-clients. Requests issued
to the connection will be sent to the remote.
* `func (c *Connection) Request(ctx context.Context, h HandlerID, req []byte) ([]byte, error)`
performs a single request and returns the result. Any deadline provided on the request is
forwarded to the server, and canceling the context will make the function return at once.
* `func (c *Connection) NewStream(ctx context.Context, h HandlerID, payload []byte) (st *Stream, err error)`
will initiate a remote call and send the initial payload.
```Go
// A Stream is a two-way stream.
// All responses *must* be read by the caller.
// If the call is canceled through the context,
//The appropriate error will be returned.
type Stream struct {
// Responses from the remote server.
// Channel will be closed after an error or when the remote closes.
// All responses *must* be read by the caller until either an error is returned or the channel is closed.
// Canceling the context will cause the context cancellation error to be returned.
Responses <-chan Response
// Requests sent to the server.
// If the handler is defined with 0 incoming capacity this will be nil.
// Channel *must* be closed to signal the end of the stream.
// If the request context is canceled, the stream will no longer process requests.
Requests chan<- []byte
}
type Response struct {
Msg []byte
Err error
}
```
There are generic versions of the server/client handlers that allow the use of type
safe implementations for data types that support msgpack marshal/unmarshal.
With an odd number of drives per erasure set setup, the write/quorum is
the half + 1; however the decommissioning listing will still list those
objects and does not consider those as stale.
Fix it by using (N+1)/2 formula.
Co-authored-by: Anis Elleuch <anis@min.io>
Immediate transition use case and is mostly used to fill warm
backend with a lot of data when a new deployment is created
Currently, if the transition queue is complete, the transition will be
deferred to the scanner; change this behavior by blocking the PUT request
until the transition queue has a new place for a transition task.
Currently, once the audit becomes offline, there is no code that tries
to reconnect to the audit, at the same time Send() quickly returns with
an error without really trying to send a message the audit endpoint; so
the audit endpoint will never be online again.
Fixing this behavior; the current downside is that we miss printing some
logs when the audit becomes offline; however this information is
available in prometheus
Later, we can refactor internal/logger so the http endpoint can send errors to
console target.
Currently if the object does not exist in quorum disks of an erasure
set, the dangling code is never called because the returned error will
be errFileNotFound or errFileVersionNotFound;
With this commit, when errFileNotFound or errFileVersionNotFound is
returning when trying to calculate the quorum of a given object, the
code checks if a disk returned nil, which means a stale object exists in
that disk, that will trigger deleteIfDangling() function
This commit splits the liveness and readiness
handler into two separate handlers. In K8S, a
liveness probe is used to determine whether the
pod is in "live" state and functioning at all.
In contrast, the readiness probe is used to
determine whether the pod is ready to serve
requests.
A failing liveness probe causes pod restarts while
a failing readiness probe causes k8s to stop routing
traffic to the pod. Hence, a liveness probe should
be as robust as possible while a readiness probe
should be used to load balancing.
Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Signed-off-by: Andreas Auernhammer <github@aead.dev>
This patch takes care of loading the bucket configs of failed buckets
during the periodic refresh. This makes sure the event notifiers and
remote bucket targets are properly initialized.
users might use MinIO on NFS, GPFS that provide dynamic
inodes and may not even have a concept of free inodes.
to allow users to use MinIO on top of GPFS relax the
free inode check.
* creating a byte buffer for SFTP file segments
* Adding an error condition for when there are
remaining segments in the queue
* Simplification of the queue using a map
it is possible that ILM or Deletes got triggered on batch
of objects that we are attempting to batch replicate, ignore
this scenario as valid behavior.
sendfile implementation to perform DMA on all platforms
Go stdlib already supports sendfile/splice implementations
for
- Linux
- Windows
- *BSD
- Solaris
Along with this change however O_DIRECT for reads() must be
removed as well since we need to use sendfile() implementation
The main reason to add O_DIRECT for reads was to reduce the
chances of page-cache causing OOMs for MinIO, however it would
seem that avoiding buffer copies from user-space to kernel space
this issue is not a problem anymore.
There is no Go based memory allocation required, and neither
the page-cache is referenced back to MinIO. This page-
cache reference is fully owned by kernel at this point, this
essentially should solve the problem of page-cache build up.
With this now we also support SG - when NIC supports Scatter/Gather
https://en.wikipedia.org/wiki/Gather/scatter_(vector_addressing)
`monitorAndConnectEndpoints` will continue to attempt to reconnect offline disks.
Since disks were never closed, a `MarkOffline` would continue to try to check these disks forever.
Close previous disks.
replace io.Discard usage to fix NUMA copy() latencies
On NUMA systems copying from 8K buffer allocated via
io.Discard leads to large latency build-up for every
```
copy(new8kbuf, largebuf)
```
can in-cur upto 1ms worth of latencies on NUMA systems
due to memory sharding across NUMA nodes.
Fix various regressions from #18029
* If context is canceled the token is never returned. This will lead to scanner being unable to save and deadlocking.
* Fix backup not being able to get any data (hr empty)
* Reduce backup timeout.
All GitHub issues are addressed on a best-effort basis at MinIO's sole discretion. There are no Service Level Agreements (SLA) or Objectives (SLO). Remember our [Code of Conduct](https://github.com/minio/minio/blob/master/code_of_conduct.md) when engaging with MinIO Engineers and the larger community.
For urgent issues (e.g. production down, etc.), subscribe to [SUBNET](https://min.io/pricing?jmp=github) for direct to engineering support.
Silo issues are handled by community maintainers on a best-effort basis. There
is no SLA, SLO, or emergency production-support channel. Follow the local
[Code of Conduct](../code_of_conduct.md) when participating. Report suspected
vulnerabilities through the private process in [SECURITY.md](../SECURITY.md),
not in a public issue.
<!--- Provide a general summary of the issue in the Title above -->
@@ -29,7 +32,7 @@ For urgent issues (e.g. production down, etc.), subscribe to [SUBNET](https://mi
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->
<!--- and include relevant Silo logs with secrets and credentials removed -->
1.
2.
@@ -42,10 +45,10 @@ For urgent issues (e.g. production down, etc.), subscribe to [SUBNET](https://mi
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
<!-- If Yes, optionally include the Silo version, commit id, or PR that caused this regression. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
if ! git log -1 --format='%(trailers:key=Signed-off-by,valueonly)' "${sha}" |
grep -qiF "<${author_email}>"; then
echo "::error::commit ${sha} by ${author_name} <${author_email}> lacks a matching Signed-off-by trailer; sign with 'git commit -s', repair with 'git rebase --signoff'"
We have designed MinIO as an Open Source software for the Open Source software community. This requires applications to consider whether their usage of MinIO is in compliance with the GNU AGPLv3 [license](https://github.com/minio/minio/blob/master/LICENSE).
Silo is distributed under the [GNU Affero General Public License v3.0](LICENSE).
It incorporates source code from the MinIO project and preserves the original
copyright, license, and attribution notices in [`NOTICE`](NOTICE),
[`CREDITS`](CREDITS), and source-file headers.
MinIO cannot make the determination as to whether your application's usage of MinIO is in compliance with the AGPLv3 license requirements. You should instead rely on your own legal counsel or licensing specialists to audit and ensure your application is in compliance with the licenses of MinIO and all other open-source projects with which your application integrates or interacts. We understand that AGPLv3 licensing is complex and nuanced. It is for that reason we strongly encourage using experts in licensing to make any such determinations around compliance instead of relying on apocryphal or anecdotal advice.
You are responsible for determining how the AGPLv3 applies to your use,
modification, deployment, and distribution of Silo and its dependencies. The
Silo maintainers cannot provide legal advice or determine whether a particular
application or service satisfies the license. Consult qualified counsel when
the obligations are material to your deployment.
[MinIO Commercial Licensing](https://min.io/pricing) is the best option for applications that trigger AGPLv3 obligations (e.g. open sourcing your application). Applications using MinIO - or any other OSS-licensed code - without validating their usage do so at their own risk.
If you convey modified binaries or provide network access to a modified
version, review the complete AGPLv3 text and ensure that the corresponding
source and notices are made available as required. Dependency licenses and
``MinIO`` community welcomes your contribution. To make the process as seamless as possible, we recommend you read this contribution guide.
Silo welcomes focused contributions that improve security, reliability,
compatibility, packaging, tests, or maintainability. This repository preserves
MinIO-compatible interfaces and storage formats, so changes must identify and
test any compatibility impact.
## Development Workflow
Start by forking the MinIO GitHub repository, make changes in a branch and then send a pull request. We encourage pull requests to discuss code changes. Here are the steps in details:
Fork the current Silo source repository, create a topic branch, and submit a
pull request. Discuss broad or compatibility-sensitive changes in an issue
before implementation.
### Setup your MinIO GitHub Repository
Fork [MinIO upstream](https://github.com/minio/minio/fork) source repository to your own personal repository. Copy the URL of your MinIO fork (you will need it for the `git clone` command below).
Pull requests can be created via GitHub. Refer to [this document](https://help.github.com/articles/creating-a-pull-request/) for detailed steps on how to create a pull request. After a Pull Request gets peer reviewed and approved, it will be merged.
Pull requests should include motivation, reproduction steps where applicable,
test evidence, compatibility notes, and documentation impact. Public product
Silo is licensed under the [GNU AGPL v3.0 or later](LICENSE). Its core is
Copyright (c) MinIO, Inc.; the combined work can never be relicensed, and this
fork does not try to.
* **No CLA.** We do not ask you to sign a Contributor License Agreement and we
do not take your copyright. Contributions are accepted inbound=outbound: you
keep the copyright to your changes and license them under the same
AGPL-3.0-or-later as the project itself. The maintainers receive no rights
beyond the project license.
* **DCO sign-off required.** Every commit must carry a
`Signed-off-by: Your Name <you@example.com>` trailer certifying the
[Developer Certificate of Origin 1.1](https://developercertificate.org/) —
your statement that you have the right to submit the code under the project
license. Sign each commit with:
```
git commit -s
```
Forgot some? Repair your branch with `git rebase --signoff` and force-push.
CI rejects pull requests containing unsigned commits; the sign-off email
must match the commit author email. (Lowercase `-s` is the plain-text DCO
sign-off; cryptographic `-S`/GPG signing is welcome but independent.)
* **Provenance.** Only submit code you are entitled to submit. This matters
more here than in most projects: Silo carries a downstream delta over an
upstream code base, and cherry-picks from the lineage remote or other forks
are routine. When relaying a patch written by someone else, preserve original
authorship (`git cherry-pick -x`, keep the author field and any existing
`Signed-off-by` trailers) and add your own sign-off as the person passing it
along. Never import code from a proprietary distribution.
* **File headers.** Files derived from upstream keep the original MinIO
copyright header unchanged. New files added by this fork use the dual
header, followed by the standard AGPL boilerplate:
```
// Copyright (c) 2015-2025 MinIO, Inc.
// Copyright (c) 2025-2026 PGSTY
```
* **Squash merges** must keep the `Signed-off-by:` trailers in the resulting
commit message.
* **Authorship and tooling.** The human contributor is the author of the commit
and the sole signatory of its DCO sign-off. Attribution trailers for
assistive tooling (for example `Co-Authored-By:` naming an AI assistant) are
informational only: they record which tools were used, and do not create
authorship, co-authorship, or any copyright claim. Whoever signs off remains
responsible for the content of the commit, whatever produced it.
## FAQs
### How does ``MinIO`` manage dependencies?
### How does Silo manage dependencies?
``MinIO`` uses `go mod` to manage its dependencies.
Silo uses Go modules. Preserve the compatibility module and import paths in
`go.mod`; downstream forks are selected with explicit `replace` directives.
- Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
@@ -76,6 +141,7 @@ To remove a dependency
- Edit your code and remove the import reference.
- Run `go mod tidy` in the source folder to remove dependency from `go.mod` file.
### What are the coding guidelines for MinIO?
### What are the coding guidelines?
``MinIO`` is fully conformant with Golang style. Refer: [Effective Go](https://github.com/golang/go/wiki/CodeReviewComments) article from Golang project. If you observe offending code, please feel free to send a pull request or ping us on [Slack](https://slack.min.io).
Follow the existing Go style, run `gofmt` on changed Go files, and keep changes
compact. See the Go project's [code review comments](https://go.dev/wiki/CodeReviewComments).
| [@waterkip](https://github.com/waterkip) | Repointed documentation links from the upstream domain to the Silo portal | [#41](https://github.com/pgsty/silo/pull/41) | [`d495d30`](https://github.com/pgsty/silo/commit/d495d30d5) |
## Proposed changes
Pull requests that are open for review, or that were closed after informing work that shipped
differently.
| Contributor | Change | Pull request | Status |
| :-- | :-- | :-- | :-- |
| [@magicxor](https://github.com/magicxor) | `DELETE` precondition checks for the `If-Match` header | [#12](https://github.com/pgsty/silo/pull/12) | Open, queued for review |
| [@ycjlin](https://github.com/ycjlin) | `ListObjects` should return `NoSuchBucket` for a prefix on a missing bucket | [#37](https://github.com/pgsty/silo/pull/37) | Open, queued for review |
| [@davinkevin](https://github.com/davinkevin) | Distroless-based Docker image variant | [#21](https://github.com/pgsty/silo/pull/21) | Superseded by the distroless variant shipped in RELEASE.2026-08-06, which the PR anticipated by four months |
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
# Verify binary signature using public key "RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGavRUN"
RUN minisign -Vqm /go/bin/minio -x /go/bin/minio.minisig -P RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav
FROMregistry.access.redhat.com/ubi9/ubi-micro:9.2
ARG RELEASE
LABELname="MinIO"\
vendor="MinIO Inc <dev@min.io>"\
maintainer="MinIO Inc <dev@min.io>"\
version="${RELEASE}"\
release="${RELEASE}"\
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
test-replication:install-racetest-replication-2sitetest-replication-3sitetest-delete-replicationtest-sio-errortest-delete-marker-proxying## verify multi site replication
@echo "Running tests for replicating three sites"
test-site-replication-ldap:install-race## verify automatic site replication
These guidelines ensure high-quality commits in Silo's GitHub repositories, maintaining
a clear, valuable commit history for our open-source projects. They apply to all contributors,
fostering efficient reviews and robust code.
## Why Pull Requests?
Pull Requests (PRs) drive quality in Silo's codebase by:
- Enabling peer review without pair programming.
- Documenting changes for future reference.
- Ensuring commits tell a clear story of development.
**A poor commit lasts forever, even if code is refactored.**
## Crafting a Quality PR
A strong Silo PR:
- Delivers a complete, valuable change (feature, bug fix, or improvement).
- Has a concise title (e.g., `[S3] Fix bucket policy parsing #1234`) and a summary with context, referencing issues (e.g., `#1234`).
- Contains well-written, logical commits explaining *why* changes were made (e.g., “Add S3 bucket tagging support so that users can organize resources efficiently”).
- Is small, focused, and easy to review—ideally one commit, unless multiple commits better narrate complex work.
- Adheres to Silo's coding standards (e.g., Go style, error handling, testing).
PRs must flow smoothly through review to reach production. Large PRs should be split into smaller, manageable ones.
## Submitting PRs
1.**Title and Summary**:
- Use a scannable title: `[Subsystem] Action Description #Issue` (e.g., `[IAM] Add role-based access control #567`).
- Include context in the summary: what changed, why, and any issue references.
- Use `[WIP]` for in-progress PRs to avoid premature merging or choose GitHub draft PRs.
2.**Commits**:
- Write clear messages: what changed and why (e.g., “Refactor S3 API handler to reduce latency so that requests process 20% faster”).
- Rebase to tidy commits before submitting (e.g., `git rebase -i main` to squash typos or reword messages), unless multiple contributors worked on the branch.
- Keep PRs focused—one feature or fix. Split large changes into multiple PRs.
3.**Testing**:
- Include unit tests for new functionality or bug fixes.
- Ensure existing tests pass (`make test`).
- Document testing steps in the PR summary if manual testing was performed.
4.**Before Submitting**:
- Run `make verify` to check formatting, linting, and tests.
- Reference related issues (e.g., “Closes #1234”).
- Notify team members via GitHub `@mentions` if urgent or complex.
## Reviewing PRs
Reviewers ensure Silo's commit history remains a clear, reliable record. Responsibilities include:
1.**Commit Quality**:
- Verify each commit explains *why* the change was made (e.g., “So that…”).
- Request rebasing if commits are unclear, redundant, or lack context (e.g., “Please squash typo fixes into the parent commit”).
2.**Code Quality**:
- Check adherence to Silo's Go standards (e.g., error handling, documentation).
- Ensure tests cover new code and pass CI.
- Flag bugs or critical issues for immediate fixes; suggest non-blocking improvements as follow-up issues.
3.**Flow**:
- Review promptly to avoid blocking progress.
- Balance quality and speed—minor issues can be addressed later via issues, not PR blocks.
- If unable to complete the review, tag another reviewer (e.g., `@username please take over`).
4.**Shared Responsibility**:
- All Silo contributors are reviewers. The first commenter on a PR owns the review unless they delegate.
- Multiple reviewers are encouraged for complex PRs.
5.**No Self-Edits**:
- Don’t modify the PR directly (e.g., fixing bugs). Request changes from the submitter or create a follow-up PR.
- If you edit, you’re a collaborator, not a reviewer, and cannot merge.
6.**Testing**:
- Assume the submitter tested the code. If testing is unclear, ask for details (e.g., “How was this tested?”).
- Reject untested PRs unless testing is infeasible, then assist with test setup.
## Tips for Success
- **Small PRs**: Easier to review, faster to merge. Split large changes logically.
- **Clear Commits**: Use `git rebase -i` to refine history before submitting.
- **Engage Early**: Discuss complex changes in a GitHub issue before coding.
MinIO creates FIPS builds using a patched version of the Go compiler (that uses BoringCrypto, from BoringSSL, which is [FIPS 140-2 validated](https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2964.pdf)) published by the Golang Team [here](https://github.com/golang/go/tree/dev.boringcrypto/misc/boring).
MinIO FIPS executables are available at <http://dl.min.io> - they are only published for `linux-amd64` architecture as binary files with the suffix `.fips`. We also publish corresponding container images to our official image repositories.
We are not making any statements or representations about the suitability of this code or build in relation to the FIPS 140-2 standard. Interested users will have to evaluate for themselves whether this is useful for their own purposes.
<strong>S3-compatible object storage — a MinIO fork maintained by PGSTY</strong>
</p>
MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.
This README provides quickstart instructions on running MinIO on bare metal hardware, including container-based installations. For Kubernetes environments, use the [MinIO Kubernetes Operator](https://github.com/minio/operator/blob/master/README.md).
Use the following commands to run a standalone MinIO server as a container.
> [!IMPORTANT]
> **PGSTY Silo** (hereinafter “Silo”) is an independent, community-maintained fork of the open-source MinIO server, published by [Pigsty](https://pigsty.io) from [`pgsty/silo`](https://github.com/pgsty/silo). It is not affiliated with, endorsed by, or sponsored by MinIO, Inc. “MinIO” is used only to identify the upstream project and compatibility lineage.
Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication
require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically,
with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html)
for more complete documentation.
> [!NOTE]
> Renamed from `pgsty/minio` to `pgsty/silo`, default branch `master` → `main`, on 2026-08-06. Artifacts under the original MinIO identity stay published on the archived [`minio`](https://github.com/pgsty/silo/tree/minio) branch and in releases up to [`RELEASE.2026-08-04T00-00-00Z`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z).
### Stable
## Overview
Run the following command to run the latest stable image of MinIO as a container using an ephemeral data volume:
PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
```sh
podman run -p 9000:9000 -p 9001:9001 \
quay.io/minio/minio server /data --console-address ":9001"
It follows one rule — **the product and its delivery surfaces are renamed; the protocol and your data are not.** Everything else lives on [silo.pgsty.com](https://silo.pgsty.com/).
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See
[Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers,
see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
Console on <http://localhost:9001>, S3 API on <http://localhost:9000>. The image bundles the client as `mcli`:
> NOTE: To deploy MinIO on with persistent storage, you must map local persistent directories from the host OS to the container using the `podman -v` option. For example, `-v /mnt/data:/data` maps the host OS drive at `/mnt/data` to `/data` on the container.
## macOS
Use the following commands to run a standalone MinIO server on macOS.
Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html) for more complete documentation.
### Homebrew (recommended)
Run the following command to install the latest stable MinIO package using [Homebrew](https://brew.sh/). Replace ``/data`` with the path to the drive or directory in which you want MinIO to store data.
docker exec silo mcli mb local/demo && docker exec silo mcli ls local
```
> NOTE: If you previously installed minio using `brew install minio` then it is recommended that you reinstall minio from `minio/stable/minio` official repo instead.
```sh
brew uninstall minio
brew install minio/stable/minio
```
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html/> to view MinIO SDKs for supported languages.
### Binary Download
Use the following command to download and run a standalone MinIO server on macOS. Replace``/data`` with the path to the drive or directory in which you want MinIO to store data.
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
## GNU/Linux
Use the following command to run a standalone MinIO server on Linux hosts running 64-bit Intel/AMD architectures. Replace ``/data`` with the path to the drive or directory in which you want MinIO to store data.
| 64-bit ARM | <https://dl.min.io/server/minio/release/linux-arm64/minio> |
| 64-bit PowerPC LE (ppc64le) | <https://dl.min.io/server/minio/release/linux-ppc64le/minio> |
| IBM Z-Series (S390X) | <https://dl.min.io/server/minio/release/linux-s390x/minio> |
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
> NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html#) for more complete documentation.
## Microsoft Windows
To run MinIO on 64-bit Windows hosts, download the MinIO executable from the following URL:
Use the following command to run a standalone MinIO server on the Windows host. Replace ``D:\`` with the path to the drive or directory in which you want MinIO to store data. You must change the terminal or powershell directory to the location of the ``minio.exe`` executable, *or* add the path to that directory to the system ``$PATH``:
```sh
minio.exe server D:\
```
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
> NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html#) for more complete documentation.
## Install from Source
Use the following commands to compile and run a standalone MinIO server from source. Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow [How to install Golang](https://golang.org/doc/install). Minimum version required is [go1.19](https://golang.org/dl/#stable)
```sh
go install github.com/minio/minio@latest
```
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
> NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html) for morecomplete documentation.
MinIO strongly recommends *against* using compiled-from-source MinIO servers for production environments.
## Deployment Recommendations
### Allow port access for Firewalls
By default MinIO uses the port 9000 to listen for incoming connections. If your platform blocks the port by default, you may need to enable access to the port.
### ufw
For hosts with ufw enabled (Debian based distros), you can use `ufw` command to allow traffic to specific ports. Use below command to allow access to port 9000
```sh
ufw allow 9000
```
Below command enables all incoming traffic to ports ranging from 9000 to 9010.
```sh
ufw allow 9000:9010/tcp
```
### firewall-cmd
For hosts with firewall-cmd enabled (CentOS), you can use `firewall-cmd` command to allow traffic to specific ports. Use below commands to allow access to port 9000
```sh
firewall-cmd --get-active-zones
```
This command gets the active zone(s). Now, apply port rules to the relevant zones returned above. For example if the zone is `public`, use
Note that `permanent` makes sure the rules are persistent across firewall start, restart or reload. Finally reload the firewall for changes to take effect.
```sh
firewall-cmd --reload
```
### iptables
For hosts with iptables enabled (RHEL, CentOS, etc), you can use `iptables` command to enable all traffic coming to specific ports. Use below command to allow
access to port 9000
```sh
iptables -A INPUT -p tcp --dport 9000 -j ACCEPT
service iptables restart
```
Below command enables all incoming traffic to ports ranging from 9000 to 9010.
```sh
iptables -A INPUT -p tcp --dport 9000:9010 -j ACCEPT
service iptables restart
```
## Test MinIO Connectivity
### Test using MinIO Console
MinIO Server comes with an embedded web based object browser. Point your web browser to <http://127.0.0.1:9000> to ensure your server has started successfully.
> NOTE: MinIO runs console on random port by default, if you wish to choose a specific port use `--console-address` to pick a specific interface and port.
### Things to consider
MinIO redirects browser access requests to the configured server port (i.e. `127.0.0.1:9000`) to the configured Console port. MinIO uses the hostname or IP address specified in the request when building the redirect URL. The URL and port *must* be accessible by the client for the redirection to work.
For deployments behind a load balancer, proxy, or ingress rule where the MinIO host IP address or port is not public, use the `MINIO_BROWSER_REDIRECT_URL` environment variable to specify the external hostname for the redirect. The LB/Proxy must have rules for directing traffic to the Console port specifically.
For example, consider a MinIO deployment behind a proxy `https://minio.example.net`, `https://console.minio.example.net` with rules for forwarding traffic on port :9000 and :9001 to MinIO and the MinIO Console respectively on the internal network. Set `MINIO_BROWSER_REDIRECT_URL` to `https://console.minio.example.net` to ensure the browser receives a valid reachable URL.
Similarly, if your TLS certificates do not have the IP SAN for the MinIO server host, the MinIO Console may fail to validate the connection to the server. Use the `MINIO_SERVER_URL` environment variable and specify the proxy-accessible hostname of the MinIO server to allow the Console to use the MinIO server API using the TLS certificate.
For example: `export MINIO_SERVER_URL="https://minio.example.net"`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the MinIO Client [Quickstart Guide](https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart) for further instructions.
## Upgrading MinIO
Upgrades require zero downtime in MinIO, all upgrades are non-disruptive, all transactions on MinIO are atomic. So upgrading all the servers simultaneously is the recommended way to upgrade MinIO.
> NOTE: requires internet access to update directly from <https://dl.min.io>, optionally you can host any mirrors at <https://my-artifactory.example.com/minio/>
- For deployments that installed the MinIO server binary by hand, use [`mc admin update`](https://min.io/docs/minio/linux/reference/minio-mc-admin/mc-admin-update.html)
```sh
mc admin update <minio alias, e.g., myminio>
```
- For deployments without external internet access (e.g. airgapped environments), download the binary from <https://dl.min.io> and replace the existing MinIO binary let's say for example `/opt/bin/minio`, apply executable permissions `chmod +x /opt/bin/minio` and proceed to perform `mc admin service restart alias/`.
- For installations using Systemd MinIO service, upgrade via RPM/DEB packages **parallelly** on all servers or replace the binary lets say `/opt/bin/minio` on all nodes, apply executable permissions `chmod +x /opt/bin/minio` and process to perform `mc admin service restart alias/`.
### Upgrade Checklist
- Test all upgrades in a lower environment (DEV, QA, UAT) before applying to production. Performing blind upgrades in production environments carries significant risk.
- Read the release notes for MinIO *before* performing any upgrade, there is no forced requirement to upgrade to latest release upon every release. Some release may not be relevant to your setup, avoid upgrading production environments unnecessarily.
- If you plan to use `mc admin update`, MinIO process must have write access to the parent directory where the binary is present on the host system.
- `mc admin update` is not supported and should be avoided in kubernetes/container environments, please upgrade containers by upgrading relevant container images.
- **We do not recommend upgrading one MinIO server at a time, the product is designed to support parallel upgrades please follow our recommended guidelines.**
> For production, pin a release, use unique credentials and TLS, monitor the service, keep independent backups, and test recovery. Start from the [documentation](https://silo.pgsty.com/docs/).
## Install
| Method | Where |
| :-- | :-- |
| Container | [`pgsty/silo`](https://hub.docker.com/r/pgsty/silo), multi-arch for `linux/amd64` and `linux/arm64` |
| Binaries | [GitHub Releases](https://github.com/pgsty/silo/releases) — Linux, macOS, Windows on `amd64` and `arm64` |
| Packages | RPM, DEB, and APK, also via the [Pigsty repository](https://pigsty.io/docs/repo/) |
| Kubernetes | Helm chart, see [Download & Install](https://silo.pgsty.com/download/) |
Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/); migrating from upstream MinIO — taking over an existing `minio.service` and its `/etc/default/minio`, and keeping data ownership stable with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in — is covered by the [migration guide](https://silo.pgsty.com/compatibility/migration/) and the [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
## Compatibility
The S3 API, `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, the `github.com/minio/*` import paths, and the on-disk format (including `.minio.sys`) are preserved and held in place by a CI compatibility check. Only Silo-owned delivery surfaces change: the `silo` executable, package, service, Helm chart, and container image — no `minio` binary alias is installed.
Every divergence from upstream is listed in the code-verified [compatibility audit](https://silo.pgsty.com/compatibility/server/). Treat each release as a downstream upgrade: pin versions, read the [release notes](https://silo.pgsty.com/tags/silo/), and keep a rollback path.
## Security & Contributing
Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); every fix ships with a public [advisory](https://silo.pgsty.com/blog/security/). Contributions are accepted inbound=outbound under AGPL-3.0-or-later with no CLA — only DCO sign-off (`git commit -s`) is required; see [`CONTRIBUTING.md`](CONTRIBUTING.md).
GitHub does not generate a contributor graph for forks, so [`CONTRIBUTORS.md`](CONTRIBUTORS.md) — not the Insights page — is this project's attribution record. It names everyone alongside the change or report they contributed.
## Background
Upstream wound down its community edition: the web console was cut back to a stub, prebuilt community binaries stopped, and the community repository was archived. Silo exists to keep those deployments running. The fork is a means, not an identity — if upstream restores its community edition, we will narrow our scope and offer the fixes back.
The [**Manifesto**](https://silo.pgsty.com/about/manifesto/) is the project's public commitment in eleven articles, under one discipline: every article is either something already done with public evidence, or something explicitly refused. In short:
- **Compatibility contract** — the protocol and your data do not change, and every release documents its tested rollback target and path.
- **The license cannot change** — AGPLv3, no CLA, no copyright aggregation; nobody here, ourselves included, holds enough copyright to relicense on everyone else's behalf.
- **The never list**, append-only — no paywalling existing features, no registration wall on downloads, no telemetry (upstream's phone-home paths are removed outright), no CLA, no license change, no trademark enforcement against normal use.
- **Security and release discipline** — a public advisory for every fix, and a release every one to two months, at most a quarter apart. Judge both against the public record.
Essays: [MinIO Is Dead](https://silo.pgsty.com/blog/post/minio-is-dead/) · [Who Takes Over?](https://silo.pgsty.com/blog/post/minio-alternative/) · [Long Live MinIO](https://silo.pgsty.com/blog/post/minio-resurrect/) · [Promise Kept](https://silo.pgsty.com/blog/post/minio-promise-kept/)
## License & Trademark
Silo is [AGPL-3.0-or-later](LICENSE), derived from [`minio/minio`](https://github.com/minio/minio) with upstream copyright and third-party notices preserved in [`NOTICE`](NOTICE) and [`CREDITS`](CREDITS). MinIO is a trademark of MinIO, Inc.; the name is used here only to identify the upstream project and compatibility lineage.
Silo is an independent, community-maintained object-storage server derived from
the open-source MinIO server. Upstream MinIO security contacts do not handle
Silo-specific fixes or release notes.
## Supported Versions
We always provide security updates for the [latest release](https://github.com/minio/minio/releases/latest).
Whenever there is a security update you just need to upgrade to the latest version.
Security fixes are tracked on the active development branch and summarized in
[docs/security/advisories.md](docs/security/advisories.md). Only the current
Silo release line is supported unless an advisory says otherwise.
## Reporting a Vulnerability
All security bugs in [minio/minio](https://github,com/minio/minio) (or other minio/* repositories)
should be reported by email to security@min.io. Your email will be acknowledged within 48 hours,
and you'll receive a more detailed response to your email within 72 hours indicating the next steps
in handling your report.
For vulnerabilities in this fork:
Please, provide a detailed explanation of the issue. In particular, outline the type of the security
issue (DoS, authentication bypass, information disclose, ...) and the assumptions you're making (e.g. do
you need access credentials for a successful exploit).
1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
2. Prefer this repository's [private GitHub security advisory](https://github.com/pgsty/silo/security/advisories/new) workflow.
3. If private reporting is unavailable, contact the maintainers through the
repository without publishing exploit details until a private channel is
established.
4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.
If you have not received a reply to your email within 48 hours or you have not heard from the security team
for the past five days please contact the security team directly:
## Disclosure Process
- Primary security coordinator: aead@min.io
- Secondary coordinator: harsha@min.io
- If you receive no response: dev@min.io
### Disclosure Process
MinIO uses the following disclosure process:
1. Once the security report is received one member of the security team tries to verify and reproduce
the issue and determines the impact it has.
2. A member of the security team will respond and either confirm or reject the security report.
If the report is rejected the response explains why.
3. Code is audited to find any potential similar problems.
4. Fixes are prepared for the latest release.
5. On the date that the fixes are applied a security advisory will be published on <https://blog.min.io>.
Please inform us in your report email whether MinIO should mention your contribution w.r.t. fixing
the security issue. By default MinIO will **not** publish this information to protect your privacy.
This process can take some time, especially when coordination is required with maintainers of other projects.
Every effort will be made to handle the bug in as timely a manner as possible, however it's important that we
follow the process described above to ensure that disclosures are handled consistently.
Fork-specific fixes and user-visible upgrade notes are published in [docs/security/advisories.md](docs/security/advisories.md). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
"${MINIO[@]}" server "${WORK_DIR}/fs-disk" >"$WORK_DIR/fs-minio.log" 2>&1&
sleep 10
"${SILO[@]}" server "${WORK_DIR}/fs-disk" >"$WORK_DIR/fs-silo.log" 2>&1&
"${WORK_DIR}/mc" ready verify
}
function start_minio_erasure(){
"${MINIO[@]}" server "${WORK_DIR}/erasure-disk1""${WORK_DIR}/erasure-disk2""${WORK_DIR}/erasure-disk3""${WORK_DIR}/erasure-disk4" >"$WORK_DIR/erasure-minio.log" 2>&1&
sleep 15
function start_silo_erasure(){
"${SILO[@]}" server "${WORK_DIR}/erasure-disk1""${WORK_DIR}/erasure-disk2""${WORK_DIR}/erasure-disk3""${WORK_DIR}/erasure-disk4" >"$WORK_DIR/erasure-silo.log" 2>&1&
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Feng Ruohang
