Carry the already-parsed source tagging timestamp through the KMS options
constructor so replica COPY can apply newer tag updates on explicitly or
automatically encrypted destinations.
Cover all option encryption modes and signed COPY persistence for newer,
stale, duplicate and timestamp-less updates, including bucket defaults.
Preserve the real Opus 5.0/max plan review, consensus and local validation.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Restore only the six replication-specific metadata fields after trust
validation, so streaming uploads retain their actual content encoding and
Snowball entries do not inherit ordinary metadata from the outer archive.
Include helper, authenticated PUT/COPY/multipart and Snowball regressions,
plus the R7 investigation, actual Opus 5 consensus and local verification.
The production change is based on PR #187 by Mikhail Khadarenka.
Co-authored-by: Mikhail Khadarenka <chodorenko@gmail.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep one healing loop per process across replication configuration reloads. Reacquire leadership after a lease is canceled and allow shutdown while waiting, so temporary quorum loss cannot permanently stop revocation propagation.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Retain source-ordered tombstones and parent grant boundaries across both IAM backends, cache reloads, and deliberate identity recreation. Reconcile deletions through a versioned, bounded replication protocol with restart-aware acknowledgements.
Cover inherited group grants, STS retention, same-key service recreation, absolute expiration, and failures after the durable commit. Document coordinated upgrades and the remaining consistency boundaries.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Remove access-frequency pool tiering and preserve independent multi-pool correctness fixes. Reconcile ordinary addressed-version DELETE across pools, retaining existing quorum and compatibility boundaries.
Verified delivery head: 4d0693cb8c. All 11 final CI checks and three qualified Linux upgrade runs passed. Introduction, retirement decisions and historical unresolved observations are documented.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Delete every copy of an explicitly addressed UUID, null version or delete
marker under the pool lock. Preserve retention and replication callbacks,
report unreadable pools and cleanup failures, and keep movement, incoming
replication, expiration and free-version cleanup on their existing paths.
Retain the separately developed general DELETE repair and replace its
access-mover-only coverage with a real interrupted rebalance copy followed
by HTTP deletion. Cover unqualified directory-marker DELETE and document
the existing pool-order-dependent 503 behavior that this makes consistent.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Reverse the first-parent diff of a3df317ae0,
including the feature branch compatibility and mover follow-up fixes.
Retain the independent multi-pool correctness fixes from #178 and migrate
their shared test fixture away from access-tier code.
Tolerate retired ILM keys and XML, read old v9 statistics while writing v8,
and document migration without moving objects or rewriting their metadata.
Include regression coverage using a historical scanner/writer v9 fixture.
Signed-off-by: Feng Ruohang <rh@vonng.com>
An independent adversarial review of the bucket metadata convergence
work found three defects it had introduced.
GetBucketInfo overwrote the physical creation probe with cached
metadata, which a bucket that never held a configuration legitimately
lacks. The new creation-time requirement then failed every policy, tag,
SSE, quota, versioning and Object Lock write on such a bucket, with no
operator recovery path, and initial synchronization skipped it silently.
Return the physical result unchanged when metadata is not requested, as
ListBuckets already does, recover the time during initial
synchronization, and pass it to MakeBucketHook so peers adopt the same
bucket generation.
Replication status compared parsed policies statement by statement while
heal compares the canonical key. An upgraded peer that stored an
equivalent statement order was therefore reported as mismatched forever,
and heal never had anything to write. Compare the key heal compares;
per-site presence counting is unchanged.
Heal diagnostics shared one log key across four conditions, so a real
peer RPC failure could be deduplicated away by an earlier message, and
they were logged at error level for the normal transient of a peer that
does not have the bucket yet. Give each reason its own key at warning
level, report only a field state that exists and still cannot be
ordered, and diagnose nothing when no site holds a state to propagate.
The recovery test now runs against the real ObjectLayer; the stub it
replaced returned the expected time and hid the defect. The policy
status test uses a statement order the canonical encoder reorders, and
adoption coverage is extended past a real field time.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Return the committed object or part time on federation write responses and
capture it for CopyObject and UploadPartCopy without a follow-up read.
Keep successful writes compatible with targets that do not supply a time.
Reject authenticated raw SSE-C replica CopyObject across deployments before
forwarding. Extend the existing SSE fixtures to verify empty objects,
stored checksums, KMS contexts and multipart sources.
Refs: #169, #168, #171
Signed-off-by: Feng Ruohang <rh@vonng.com>
A cross-deployment CopyObject that requests
`x-amz-object-lock-legal-hold: ON` answered 200 while the destination
carried no hold. The resolved value reached the remote as ordinary user
metadata, `X-Amz-Meta-X-Amz-Object-Lock-Legal-Hold`, so nothing applied it.
Retention requested on the same copy survived, which is what made the loss
easy to miss.
The federation branch passes the resolved metadata map straight to
`Core.PutObject` as `PutObjectOptions.UserMetadata`. minio-go's `Header()`
writes the typed lock fields first, then prefixes every UserMetadata key it
does not recognise with `x-amz-meta-`; `supportedHeaders` covers
`x-amz-object-lock-mode` and `x-amz-object-lock-retain-until-date` but not
`x-amz-object-lock-legal-hold`, and `isAmzHeader` does not match it either.
Retention therefore arrives as real headers and the hold does not. The
high-level `validate()` that would have rejected the key never runs, because
`Core.PutObject` goes straight to the low-level PUT.
Carry the hold on the typed `LegalHold` option and forward a cloned map with
the raw key removed. The clone matters twice: typed fields are written before
the UserMetadata loop, so a leftover raw key would add a bogus `x-amz-meta-`
entry beside the correct header, and the proxy's own response and event
metadata are rebuilt from the resolved values rather than the forwarding map,
which no longer carries the hold.
Retention stays in the map deliberately. It already passes through as a
standard header, and moving it to the typed `RetainUntilDate` field would
format with `time.RFC3339` and truncate a retain-until date to whole seconds.
The new test asserts the wire: the remote must receive
`X-Amz-Object-Lock-Legal-Hold` and never the `x-amz-meta-` spelling, and the
destination version must actually store the hold. It fails without the change
with "legal hold forwarded as user metadata [ON]".
Fixes#166
Signed-off-by: Ayush Sharma <72848455+Aeirx@users.noreply.github.com>
Use ChangeMyPassword for the authenticated user and keep CreateUser for other users. Coordinate silo-pkg b3760f56ec23, mcli fa22b40b4eb7, Console 1b95b6cec652 and upstream minio-go 78bfa91607c2. Add legacy-policy and SDK streaming regressions plus upgrade guidance.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Add Oren Yomtov (github.com/orenyomtov) to the contributor wall in
README.md, README_ZH.md, and CONTRIBUTORS.md, and to the Issue reports
table, for the private disclosure of the unsigned-header CopyObject
cross-object read fixed as SN-2026-011 (#173). Community contributor
count 40 -> 41.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Ledger entry for the confused-deputy fix in 123325430: an unsigned
x-amz-copy-source header turned a presigned or signed PUT into a
server-side copy of any object the signing key can read. Reported by
Oren Yomtov; inherited from upstream minio/minio; CVE requested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
A presigned or signed PUT authorized for a single object could be turned
into a server-side copy of any object the signing key can read by adding
an unsigned x-amz-copy-source header, executed as the signer. SigV4
verification only walked the signed-headers list, never the headers that
actually arrived; the meta-header check matched only X-Amz-Meta- and ran
only on the presigned path, so an unsigned x-amz-* header outside the
list was never seen while the router still dispatched the PUT to
CopyObjectHandler.
Reject any x-amz-* request header not covered by the signed headers, on
both the presigned (doesPresignedSignatureMatch) and Authorization-header
(doesSignatureMatch) paths, matching AWS S3. The check tests membership
in the signed set rather than value equality, so a header whose first
value is empty (e.g. {"", "/src/secret"}) cannot slip through.
X-Amz-Content-Sha256 is exempt (payload hash: read from the query for
presigned requests and bound into the string-to-sign for signed ones, so
it is self-protected) and X-Amz-Signature-Age is exempt (an internal
scratch header written after verification, so repeated verification of
the same request stays idempotent). The synthesized X-Amz-Tagging header
in PutObjectTagging is now injected after signature verification.
Tests that previously added x-amz-copy-source and friends after signing
(relying on the vulnerable behavior) now re-sign, mirroring real S3
clients. Adds checkUnsignedHeaders unit cases and TestPresignedVerifyIdempotent.
Reported by Oren Yomtov. Inherited unchanged from upstream minio/minio.
Tracked as SN-2026-011.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Remove the eight explicit curve overrides so Go 1.27 honors tlsmlkem=0
across Server listeners, node links and outbound transports. Remove the
unused shared curve option and add wire-level regression coverage.
Document CA trust and TLS upgrade behavior, retain the investigation
artifacts, and exclude their synthetic routes from the rebrand guard.
The product compatibility baseline remains unchanged.
Validation: focused race tests, HTTP tests, lint, compatibility guard
positive/negative controls, and a fresh Linux build with three isolated
OIDC integration scenarios all pass.
Adversarial review: Claude Code Fable 5.1, max effort.
Final verdict: APPROVE FOR COMMIT.
Signed-off-by: Feng Ruohang <rh@vonng.com>
A multipart SSE-S3 source is encrypted per part, so its logical size is
the sum of the parts' decrypted sizes and the decrypting reader crosses
a part boundary. Copy such a source across the federation to a plain
and to an SSE-S3 destination and check the destination plaintext and
single-encryption size.
The test router registers routes in endpoint order and the plain
PutObject route has no query matcher, so the multipart endpoints are
listed first.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
putOptsFromReq handed the parsed kms.Context straight to
encrypt.NewSSEKMS. kms.Context implements encoding.TextMarshaler, so the
SDK serialized it as a JSON string, and a request without a context
still produced one because the nil Context is a typed nil inside the
interface value and marshals to "{}". The receiving ParseHTTP rejects
both forms, so every federated CopyObject to an SSE-KMS destination
failed with InvalidArgument once the forwarded stream was correct.
Pass a plain map, or nothing when no context was requested, and cover
SSE-KMS destinations with and without an explicit context.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The legacy etcd bucket-federation branch of CopyObjectHandler reads its
source through getObjectNInfo, which yields the decrypted and
decompressed bytes, but it also ran the destination encryption locally
and then forwarded that stream to the remote PutObject with the source's
stored size and the destination SSE option. SSE to plain and plain to
SSE therefore failed on a Content-Length mismatch, while SSE to SSE
matched by coincidence: the remote encrypted the ciphertext a second
time and stored an unreadable object, and a destination GET returned
the inner ciphertext with HTTP 200.
The remote write owns the destination's storage transformations, so
hand it the logical bytes at their logical size and let it encrypt
exactly once. Compression was already excluded on this branch; apply
the same rule to encryption, size the forwarded reader by actualSize,
and declare that size on the forwarded PutObject.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FodsDpa6VkghaeRE6WjmEe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The #105 T3 change (PR #156) tried to keep the resident metadata cache
monotonic by guarding peer-reload publication on lastUpdate(). But
lastUpdate() is the max of per-config timestamps and cannot order whole
records: a node caching {policy@20, CORS@10} that receives a newer CORS@15
still has lastUpdate()==20, so the guard rejects the legitimately-newer
record and the periodic refresh (same comparator) cannot repair it. A
paused reload could also resurrect deleted resident state.
Per the maintainer decision, revert the reload publication to its original
unconditional (acceptable-until-refresh) behavior:
- remove setReloaded and restore the plain Set plus notification/target
registry updates in LoadBucketMetadataHandler;
- restore refreshBucketsMetadataLoop's own lastUpdate() staleness check and
globalEventNotifier.set / globalBucketTargetSys.set publication;
- restore the unconditional GetConfig cache-miss publication;
- document the known freshness limitation at the reload site (the periodic
refresh is best-effort and cannot repair an equal-maximum-timestamp
divergence).
The T1 lifecycle merge-under-lock (UpdateExpiryLCConfig) and both T2 fixes
(DeleteBucket takes metadata.lock before deleting; saveMetadata and
loadBucketMetadataParseUnderLock recheck physical bucket existence) are
kept fully intact.
Tests:
- drop the T3 reproductions (overlapping-reload resident-cache test and the
peer-reload-preserves-current-targets publication test);
- add lockBucketMetadataAcquireHook, a nil-in-production atomic test hook in
the shared metadata.lock path, so tests can deterministically observe a
caller (notably DeleteBucket, whose lock is taken through its
erasureServerPools receiver and is invisible to an injected object layer)
reaching the lock;
- rewrite the T2 delete-race ghost test to hold metadata.lock MID-SAVE (past
saveMetadata's existence recheck) and synchronize on the delete's actual
lock attempt via the hook, so it isolates the lock-before-delete fix:
removing only DeleteBucket's metadata.lock (recheck kept) now fails it;
- rewrite the cancellation test to observe the delete's actual lock attempt,
then cancel and await its error while still holding the lock, so a
scheduling-delayed delete stopped by the canceled context can no longer
pass on a broken tree.
Refs #105. Follows #156.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Three residual checksum defects in the legacy etcd federation branch of
CopyObjectHandler, found by post-merge review of #157.
1. Empty-source 500 regression. A checksum-less object gains the S3 default
CRC-64NVME (WantServerSideChecksumType is set), but minio-go streams no
trailing checksum for a 0-byte body (contentLength == 0), so the remote
computed none, federatedChecksumValue was empty, hash.NewChecksumWithType
returned nil, and the handler returned 500 -- so every empty-object
federated copy failed. For a 0-byte source, forward the empty-content digest
as an ordinary checksum request header instead, so the remote validates,
persists and returns it, matching the local path (e.g. CRC32 "AAAAAA==").
2. Inherited full-object checksum dropped. When the source already carries a
full-object checksum, the local path sets dstOpts.WantChecksum, not
WantServerSideChecksumType (only multipart-composite sources are promoted).
The federated branch inspected only WantServerSideChecksumType, so a
checksum-bearing source's checksum was silently discarded on a federated
copy that requested no algorithm. Forward WantChecksum.Encoded (always a
plain digest) as a checksum header so the remote validates and persists it,
and bind the returned value, matching local persistence.
3. Multipart-suffixed remote value accepted. The bind accepted a value like
"NSRBwg==-0": NewChecksumWithType parses the "-N" as ChecksumMultipart with
WantParts 0 and the length-only validator passes, so the destination was
returned as COMPOSITE. A single forwarded PutObject must yield a full-object
digest, so reject a multipart-marked parsed value in addition to the
existing nil (missing/malformed) rejection.
A forwarded checksum request header is stripped from objInfo.UserDefined so it
is not mistaken for object metadata.
Out of scope: the SSE federated-copy corruption (srcInfo.Reader/Size mismatch
for encrypted sources) predates this work and is filed separately.
New federated regressions cover empty source with requested and default
checksum (200 + correct value + persisted), an inherited full-object checksum
preserved without a requested algorithm, and a multipart-suffixed remote value
rejected. Red/green verified for each against the merged code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Record the ten documented MINIO_ILM_ACCESS settings, the internal object
metadata stamp, and the tracker storage-path suffix introduced by this PR.
The guard places the /ilm/access string in its routes set, but the value
is a component of the tracker object prefix, not a public HTTP endpoint.
Keep all existing compatibility entries. The guard, delivery rebrand check,
and Docker entrypoint compatibility tests pass with the refreshed manifest.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Subscribe re-registered the console target on every console-log subscription, producing duplicate minio_logger_webhook_* series on each /minio/metrics/v3 scrape. Fixes#150
Signed-off-by: nikitapogromsky <129324283+nikitapogromsky@users.noreply.github.com>
Match the exact retired repository while preserving links to distinct repositories and historical issue titles in contributor credits. Continue rejecting live links to the retired repository in those credits.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Clarify SILO contribution ownership and preserve prior copyright notices. Consolidate issue templates and route the legacy credits command through the maintained generator.
Validation: make rebrand-guard; bash -n update-credits.sh; regenerated credits match CREDITS; template and link checks.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Audit of the three deferred #105 follow-ups. Each reproduces with a
deterministic red test in cmd/bucket-metadata-race_test.go, and each fix is
the minimal change that turns its test green while preserving the
<bucket>.lck -> metadata.lock -> .metadata.bin lock order established by #103.
1. Lifecycle expiry merge lost update (persistent). PeerBucketLCConfigHandler
and healBucketILMExpiry read the current lifecycle with an unlocked
GetConfigFromDisk, merged the replicated expiry rules with the local
transition rules, then wrote the pre-computed blob via Update. Any lifecycle
transition change committed between the merge read and the merge write was
silently lost. New BucketMetadataSys.UpdateExpiryLCConfig performs the read,
merge, and save under one metadata.lock; mergeExpiryWithLCConfig now takes
the locked snapshot and validates object-lock retention from it instead of
re-reading (avoids a re-entrant metadata load under the lock).
2. DeleteBucket ghost .metadata.bin (persistent). DeleteBucket took only
<bucket>.lck while config writers take only metadata.lock, so a writer that
was mid-save could re-create .metadata.bin after the prefix purge. The purge
now runs under metadata.lock, with a best-effort unlocked fallback so a
delete is never blocked from completing.
3. Overlapping peer reloads publishing a stale resident cache (freshness only;
the persisted record stays correct). LoadBucketMetadataHandler and the
GetConfig cache-miss path published with an unconditional Set, so a reload
that read an older revision could overwrite a newer resident record until the
next refresh. New BucketMetadataSys.setReloaded (and a matching GetConfig
guard) refuses to regress a newer resident record, mirroring
refreshBucketsMetadataLoop.
Verification: go build -tags kqueue,dev ./...; go vet ./cmd; gofmt clean;
rebrand-guard baseline unchanged; go test -tags kqueue,dev ./cmd (207s) green;
new tests plus the #103 metadata suite green under -race.
Refs #105. Parent #102. Foundation #103.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The legacy etcd federation branch of CopyObjectHandler forwards the copied
bytes with minio-go Core.PutObject but never asked the remote for a checksum
and discarded any it returned, so a cross-deployment whole-object copy that
requested a checksum returned 200 with an empty checksum, and a checksum-less
source did not gain the S3 default CRC-64NVME that the local path assigns. The
request was neither honored nor rejected. This is the whole-object counterpart
of #72, which repaired the same class of defect for federated UploadPartCopy.
When a server-side checksum is wanted -- explicitly requested, inherited from a
multipart source, or the CRC-64NVME default for a checksum-less object, all
already captured in dstOpts.WantServerSideChecksumType -- the forwarded
PutObject now streams a trailing checksum of that type, so the remote computes
and persists it and echoes it in the response. The value the remote reports for
that exact write is bound into objInfo.Checksum, matching how the local
CopyObject path carries checksums into the CopyObjectResult. Reading the value
from the same UploadInfo that produced the ETag keeps the pair bound to one
write.
Only the requested algorithm is returned; a malformed or absent remote value
leaves objInfo.Checksum unset, so an ordinary copy that wanted no checksum
still returns none. Two small mapping helpers convert between the server's
hash.ChecksumType and the minio-go request type and response field.
New end-to-end tests drive the real federation branch through
getRemoteInstanceClient and minio-go into a second in-process deployment and
assert that CRC32/CRC32C/SHA256/CRC64NVME and the no-algorithm default are all
returned in the CopyObjectResult and persisted on the destination, and that a
requested algorithm never leaks other algorithms into the response.
Fixes#99
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The embedded Console reaches the S3/STS API at https://127.0.0.1:<port>
(minioConfigToConsoleFeatures), a loopback endpoint whose TLS certificate is
not expected to carry a 127.0.0.1 SAN. silo-console v2.3.x began verifying
every outbound TLS peer, so the Console's STS AssumeRole handshake to that
loopback endpoint now fails certificate validation and BOTH local and LDAP
logins fail with a generic "invalid login". The failure happens in the Console
HTTP client before any request reaches a server auth/STS/LDAP handler, so no
server-side auth error is logged, matching the report.
Restore the documented loopback bypass by opting the embedded Console into its
endpoint-scoped CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY switch whenever the server
falls back to the 127.0.0.1 endpoint under TLS. The exemption is scoped to that
single loopback origin inside Console; every other HTTPS peer (IdP, Prometheus,
webhooks) stays verified, preserving the v2.3.x hardening. An explicitly
configured endpoint is reached under its own verified name and is never
exempted. initConsoleServer unsets CONSOLE_* before re-deriving them, so the
switch cannot be supplied by the operator on the embedded path; the server must
assert it.
Fixes#108
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The legacy etcd federation branch of CopyObjectHandler forwards the copied
source metadata to the remote deployment with minio-go Core.PutObject, after
removing only two reserved keys (compression and actual-size). Every small
object is stored inline, so its stored metadata also carries
X-Minio-Internal-inline-data; the remote's setRequestLimitMiddleware rejects
any request bearing a reserved-prefix header (containsReservedMetadata), so
the forwarded write failed with 400 InvalidArgument "Your metadata headers
are not supported." for the default COPY metadata directive.
A plain federated PutObject must not carry any internal storage metadata, so
strip the whole reserved-prefix class before forwarding instead of an
enumerated subset. Enumerating a third key would only defer the next leak:
besides inline-data, replication bookkeeping (replica/replication status and
timestamps) is added to the same map earlier in the handler and would be
rejected just the same. None of these keys is required by the remote for a
correct plain PutObject; they are internal storage details the remote sets
for itself. The stripping uses stringsHasPrefixFold, matching the remote's
own case-insensitive detection. Ordinary user metadata (x-amz-meta-*) is
untouched and still copied.
The pre-existing UUID ETag on the federated write (no Content-MD5 is sent) is
out of scope and left unchanged, as recorded in the issue.
A new end-to-end test drives the real federation branch through
getRemoteInstanceClient and minio-go into a second in-process deployment,
copying an inline source with the default COPY directive. It asserts the copy
now succeeds, that no reserved-prefix header reaches the remote on any
forwarded request, and that copied user metadata survives.
Fixes#100
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
PutObjectHandler's precondition callback ran DecryptObjectInfo on the
stored object before checkPreconditionsPUT, so an authenticated raw
SSE-C replica overwrite was rejected when the stored version could not
decrypt. A replica a pre-fix destination (issue #109) left as
compress(ciphertext) or a re-encrypted body has an invalid decrypted
length, so DecryptObjectInfo returned errObjectTampered and the
retransmission that repairs it never ran -- the version stayed damaged
through resync. #134's raw-replica exemption only covered the
version/ETag duplicate check inside checkPreconditionsPUT, one step too
late.
Skip the stored object's decryption precondition only for a PURE raw
SSE-C replica overwrite (a trusted SSE-C replica write with no public
precondition), keyed on the incoming request's restored SSE-C metadata,
the same predicate checkPreconditionsPUT uses. Such a write fully
replaces the object, so requiring the damaged stored version to decrypt
is both wrong and unnecessary. A conditional request keeps the check:
DecryptObjectInfo also normalizes the stored sealed ETag to the
client-visible one, and If-Match/If-None-Match must compare against
that, not the sealed ETag -- skipping it for every replica inverted both
conditions. Ordinary writes and non-SSE-C replicas are unchanged.
Adds red/green regressions: a raw retransmit over a version staged as an
undecodable body returns 500 XMinioObjectTampered before this change and
200 with full customer-key recovery after; and a conditional replica PUT
(If-Match / If-None-Match) on the client-visible ETag is honoured rather
than inverted. Fixes the single-PUT compression-damage recovery gap in
Signed-off-by: Feng Ruohang <rh@vonng.com>
#120.
Rebased onto current main. #119 made PutObjectPart derive an encrypted
part's plaintext length and reject a part that cannot be a valid sio
stream, so TestReplicaLockReconcileNullVersion's completeNullMPU fixture
(a 4-byte plaintext part under SSE-C metadata) no longer stores; build it
with sio.Encrypt like the other encrypted-part fixtures. Also regenerate
the rebrand-guard baseline for the replication SSE header the retransmit
path reintroduces (headers 84 -> 85). Mechanical integration only.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Issue #120 routes an existing SSE-C replica through PutObjectHandler and
NewMultipartUploadHandler. On main those handlers assigned the incoming retention
and legal hold directly, without the source-timestamp ordering #111 added to
CopyObjectHandler and without persisting the ordering timestamps, so in
active-active replication a retransmit carrying an older value could overwrite a
destination version's newer lock state.
Share #111's ordering decision as applyReplicatedObjectLock in
cmd/bucket-object-lock.go and call it from CopyObject, PUT and multipart
initiation. A request that is not an actual trusted replica keeps ordinary write
semantics (a validated value is applied and stamped now); only a real replica
update is ordered against the stored version, so a marker-only peer write no
longer drops a validated hold or default retention. CopyObject keeps its SSE-C
key-rotation encMetadata reconciliation inline. putReplicationOpts now emits a
stored retention ordering timestamp even when the value keys are absent, so a
removal recorded on the retransmit PUT path still replicates onward. replicateAll
marks Failed and carries the error when putReplicationOpts fails.
The handler decision is made against the version as it stands then, which a
concurrent lock update can outrun before the write commits, and for multipart
across the whole initiation-to-completion span. Close that window under the
object write lock the receiving erasure set holds: a trusted SSE-C replica full
write sets ObjectOptions.ReplicaLockReconcile, and erasureObjects.PutObject and
CompleteMultipartUpload re-run the ordering (reconcileStoredObjectLock, which
orders retention and legal hold independently by their reserved timestamps)
against the destination version read on that set before committing. Persisted
upload metadata records the null version as an empty VersionID, so completion
looks that up as the null version rather than the latest. The reconcile runs only
against an existing version; a not-found destination keeps the write's own
accepted lock, including a pre-upgrade upload that persisted values without
ordering timestamps, and a non-not-found read error fails the write. Scoped to
the SSE-C paths this issue enables; CopyObject is left as #111 wrote it.
Scope: this orders Object Lock against the destination version under the write
lock and is correct for a single erasure set. A multi-pool deployment -- where a
version can have duplicate copies across pools, object ModTime ties do not track
per-field lock timestamps, and the object namespace lock is per-pool -- needs a
cross-pool lock-safe reconcile and is deliberately out of scope here, tracked in
pgsty/silo#TBD-multipool-lock.
Tests: TestAPISSECReplicaRetransmitObjectLockOrdering and its multipart sibling;
TestAPIReplicaMultipartNewerHoldSurvivesCompletion and
TestReplicaPutObjectLockReconcileUnderWriteLock (a hold or retention reaching the
version after the handler decision, or after multipart initiation, survives the
commit; a pre-upgrade upload on an absent version keeps its lock);
TestReplicaLockReconcileNullVersion (a null-version completion reconciles the null
version, not a coexisting UUID version, and an absent null version keeps its
accepted lock); TestAPIReplicaMarkerOnlyAppliesObjectLock; TestReplicaStoredLock;
the timestamp-only putReplicationOpts round trip; and the retransmit, exemption
and target-head tests. The #111 CopyObject replica suite and the existing #120
suite stay green, as do the PUT/multipart handler and object-layer regression
suites. Compatibility: the shared helper preserves #111's CopyObject behavior; a
non-replica PUT or multipart initiation that sets Object Lock now also stamps the
reserved ordering timestamp, matching CopyObject since #111; only trusted SSE-C
replica writes take the in-lock reconcile.
Refs pgsty/silo#120
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The replication sender's target HEAD carries no SSE-C customer key, so for
an SSE-C object the target answers 400 and replicateAll fell into a
metadata-only CopyObject that fails on any non-empty SSE-C object (the
undecryptable source checksum makes the target recompute one and rewrite
the data with a plaintext-sized reader). Once a non-empty SSE-C replica
existed, tag, retention and legal-hold changes never reached it, a heal
never retransmitted, and a resync neither repaired the replica nor
counted it correctly. Forcing a full retransmit alone was not enough:
checkPreconditionsPUT rejects a write whose PreserveETag and VersionID
match the stored version, only the single-part sealed ETag is truncated
before that comparison, so a multipart SSE-C retransmit answered 412,
which the sender turns into success. Inherited from upstream ad04afe38.
Select replicateAll when the SSE-C HEAD cannot answer (the two previous
assignments were dead: rAction still forced the metadata path), exempt an
authenticated replica write that carries an SSE-C seal from the duplicate
version and ETag rejection (the predicate is the incoming write's
restored SSE-C metadata, not what the destination holds), and send the
internal replication marker on the resync accounting HEAD for SSE-C
objects so a peer answers with the replica metadata instead of 400.
Tests: TestAPISSECReplicaRetransmitOverExistingVersion (multipart replica
initiation over the same version and ETag answered 412 on main, now 200
with parts sent and plaintext readback; single-part and zero-byte writes
unchanged), TestAPISSECReplicaWriteExemptionIsKeyedOnTheIncomingWrite
(plaintext replica over an SSE-C version still 412; SSE-C replica over a
plaintext version exempted and readable), and
TestAPISSECReplicationTargetHead (keyless HEAD 400, missing key 404,
marked HEAD 200 with metadata, metadata CopyObject ExcessData on a
non-empty object) on ErasureSD and Erasure. Compatibility: every update
of an SSE-C object now retransmits its bytes; a peer that rejects the
internal marker fails the accounting HEAD as before; the #109 destination
fix must be deployed first or a retransmitted replica is transformed
again.
Fixespgsty/silo#120
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
retentionRemovedAtSource only recognized representation (1) of a removed
retention: the object lock key present with an empty value. But a removal
that arrived by replication persists representation (2): restoreRetention
(and the receiver's replica update path) writes only the retention ordering
timestamp when the mode is empty, leaving the mode and retain-until-date keys
absent. For that shape the helper returned false, so replicationActionForTarget
skipped the GetObjectRetention confirmation and let getReplicationAction's
replicateNone stand, silently dropping a needed removal when the destination
HEAD hides retention behind a permission-filtered credential.
Recognize representation (2) as well: a present retention ordering timestamp
with the mode value absent or empty is a removal. A present timestamp paired
with a non-empty mode is a retention that was set, not removed, and still
returns false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
getReplicationAction builds its source map from oi1.UserDefined, where a
removed retention is a present key with an empty value, and its target map
from the destination's HEAD headers, which can never carry those keys because
setObjectHeaders skips empty lock values and FilterObjectLockMetadata drops
both keys when the mode is invalid. The comparison then always reports a
difference, the replicateNone fast path is dead for such versions, and an
otherwise matching version re-copies its metadata on every evaluation.
Skip an entry whose value is empty and whose key is x-amz-object-lock-mode or
x-amz-object-lock-retain-until-date, case-insensitively, in both comparison
loops, using the joined value on the target side. Normalizing only the source
would regress the case where both sides hold the empty pair.
HEAD also omits a real retention from a credential without
s3:GetObjectRetention, which the documented target policy does not grant, so
that normalization alone would read a destination hiding a retention as in
sync and drop the removal. replicationActionForTarget therefore confirms with
the destination before skipping the resend: only an explicit answer, no
retention on the version, clears it. Everything else keeps today's metadata
resend, including a denied or unreachable destination, a mode the SDK does not
recognize, and InvalidRequest, which names a bucket without Object Lock but is
also what a destination answers when its own read of that configuration fails.
The null version an existing object resync excludes is never reopened.
Tests: TestGetReplicationActionEmptyObjectLockValues (eight cases, red on 2
and 3 before this change), TestRetentionRemovedAtSource,
TestTargetRetentionConfirmedAbsent,
TestReplicationActionForTargetRetentionRemoval,
TestReplicationActionForTargetNullVersionResync and
TestEmptyRetentionValuesAreOmittedFromObjectResponseHeaders.
Compatibility: sender side only, no wire or storage change, so a fixed source
converges against any destination version.
Fixespgsty/silo#117
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
DeleteObject ignored the If-Match request header and always deleted the
object (204). AWS S3 conditional deletes require that when If-Match is
provided and does not match the object's current ETag, the delete is
refused with 412 Precondition Failed and the object is left intact.
The precondition is evaluated in erasureServerPools.DeleteObject, while
the server-pool delete lock is held, before the delete-marker short-circuit
and before any version is removed. It runs against the version that will
actually be deleted: pinfo.ObjInfo for a normal delete, or the specifically
addressed version (read under the held lock) for a version-scoped delete,
since getPoolInfoExistingWithOpts strips VersionID. The check is a pure
function (no ResponseWriter writes) and returns PreConditionFailed, which
toAPIError maps to 412; CheckPrecondFn is cleared before lower layers run
so the precondition is evaluated exactly once.
Semantics:
- If-Match mismatch on a live object -> 412, object preserved.
- If-Match "*" requires a live object; a delete-marker-latest -> 412, and
an explicitly addressed delete-marker version -> 412 (getObjectInfo
returns the marker with MethodNotAllowed; the marker is the precondition
target, not a 405).
- SSE-C/SSE-KMS: compared against the public ETag derived without the
customer key, so a satisfiable condition is never falsely rejected.
- Explicit versionId -> evaluated against that version; a missing addressed
version -> NoSuchVersion whether or not the key exists; a missing object
(no versionId) -> NoSuchKey; no If-Match -> unchanged (including the
unconditional version-scoped delete's error behavior).
Scope: atomicity is guaranteed for a single erasure set (the default
deployment). Multi-pool conditional-delete atomicity (concurrent writers
across pools, cross-pool version selection) is tracked as a follow-up.
Tests: pure-helper unit test (delete marker, "*", SSE-C without key);
object-layer tests (unversioned match/mismatch/missing, versioned
delete-marker-latest and addressed delete-marker version, explicit-version
selection, missing version on present and absent keys, read-quorum loss);
handler tests (412/204/wildcard/404) across both backends, with red/green
demonstrated per guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The AWS Java SDK v2, with chunked encoding enabled (its default), sends a
PutObject as a non-trailer signed aws-chunked stream
(x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD). When a checksum
algorithm is set it puts the precomputed value in the x-amz-checksum-crc32
header, yet still advertises the checksum in x-amz-trailer even though no
trailer chunk is ever sent.
GetContentChecksum treated any x-amz-trailer-advertised checksum as trailing
with an empty value, deferring it to a trailer. For the non-trailer auth type
the handler sets req.Trailer = nil, so at EOF the hash.Reader looked the value
up in a nil trailer, got "", and returned XAmzContentChecksumMismatch (HTTP
400) even though the correct value sat in the request header. Real S3 accepts
the request, and disabling chunked encoding removed the trailer advertisement,
matching the reported symptom.
Honor the header value directly when a trailer-advertised checksum is already
present in the request headers; fall back to trailing delivery only when the
header is absent. When the header carries the checksum but it does not parse,
reject the request with ErrInvalidChecksum instead of falling through to a
no-validation path, so a malformed client-supplied checksum is never silently
dropped. This also restores the checksum echo on the response and the stored
value, while keeping genuine trailer uploads and wrong-checksum rejection
intact.
Fixes#107.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The resync worker pool runs for a single target (opts.arn), but the dispatch
loop admitted any object whose ExistingObjResync.mustResync() was true for ANY
target. On a bucket with per-target rules (A and B), a resync of A would pull
in objects that only qualify for B - even with a single active resync, since
qualification is any-target. After the outcome-based classification (previous
change) such a cross-target object leaves A absent from its per-object result
and is counted as an A failure - an object A was never responsible for.
Scope admission to the resync's own target: dispatch an object only if it must
resync for opts.arn specifically (mustResyncTarget), via a small pure helper
objectNeedsResyncForARN. Only opts.arn carries this resync's ResetID, and that
reset is already folded into its per-target decision, so the per-target check
both scopes dispatch and honors the reset. Each target has its own resyncBucket,
so no cross-target object is dropped - it is handled by that target's resync.
The classifier's absent-ARN failure path is now unreachable for normally
dispatched objects and remains only as defense-in-depth (e.g. a config/lock
error before replication is attempted).
Delete-marker/version-purge handling, the null-version exclusion, the
finalization ordering and the outcome-based classification are unchanged.
Adds a table-driven regression for the predicate: with A/B rules and a resync
of A, an object qualifying only for B is not admitted; any-target scoping
admits it and fails the test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The resync worker classified each object by whether the target version
merely existed (a tgt.StatObject HEAD), ignoring the outcome of the
replicateObject/replicateDelete call it had just made. A quota-rejected
update leaves the old version in place, so StatObject succeeded and the
resync recorded a false success - reported as Completed / N success /
0 failed and persisted across restart (issue #139). #134's SSE-C HEAD
marker made StatObject succeed for SSE-C too, exposing it there. The delete
path had the mirror flaw (a failed delete leaves the object, so the HEAD
succeeded), and FailedSize was never incremented (a failed 196,608-byte
object counted as 1 failed / 0 bytes).
replicateObject and replicateDelete already build the per-target
replicatedInfos (each replicatedTargetInfo carries Arn, ReplicationStatus
and Err) but discarded it. Return it (callers that only trigger replication
ignore the value - a Go call statement discards it, so the queue paths are
unchanged) and classify the resync from the target whose Arn == opts.arn via
a small pure helper:
- Completed without error -> replicated (+ that target's size, falling back
to the object size).
- Failed or errored -> failed (+ the object size, fixing FailedSize).
- opts.arn absent from the result (not attempted) -> failed; a resync that
cannot confirm the object reached the target is not a success.
The StatObject-existence block (including the delete-marker/MethodNotAllowed
special case, now subsumed by the delete outcome) is removed. #134's SSE-C
HEAD marker is left intact - it is needed for genuine SSE-C success.
Adds a table-driven regression for the classifier covering a completed
update, a failed update over an existing version, an errored-but-Completed
result, a delete failure, a delete-marker success (zero bytes) and an
un-attempted ARN. Classifying by existence makes the failed cases count
success and fails the test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
resyncBucket could publish and persist a Completed resync status that did
not actually cover every object, in two ways:
1. It joined only the producer workers before the deferred markStatus ran,
not the goroutine that folds each worker result into the status, so a
Completed status could omit the last object (or a failed object) until the
periodic ~1m flush (issue #136). The same finalization also closed the
result channel on early-return paths while workers were still in flight,
risking a send-on-closed-channel panic and a lost result.
2. markStatus persists under its own background context, so if the parent
context was cancelled during the drain - workers then return without
sending their computed result - or a worker dropped a result on the
resync-cancel signal, a bare Completed was still recorded with counts that
no longer matched the objects seen.
Fixes (count integrity only; the inherited cancellation deadlock, walker leak,
and single-token routing are tracked as separate follow-ups):
- Centralize shutdown in a resyncResults helper whose finish() stops the
workers (closes inputs, waits for them to exit) before closing the result
channel and waiting for the consumer to drain, then lets the deferred
markStatus persist the final counts. finish() now runs on every exit path.
- Record a dropped result via sendResyncResult (a worker consuming the
resync-cancel token returns without sending), and in the finalizer downgrade
a Completed status to Failed via finalResyncStatus when the parent context
was cancelled or a worker aborted - so a persisted Completed never
misrepresents an incomplete resync.
Deterministic tests: an on-disk round-trip of the terminal status (complete
counts stay Completed; parent-cancel-during-drain and worker-abort each
downgrade to Failed), and testing/synctest drain/worker-order assertions that
fail deterministically if a finish() wait is removed. The inherited
cancellation structure (inline Walk, the dispatch send, the worker cancel
branches) is left unchanged for the follow-ups.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestAPIGetObjectAttributesEncryptedPartLengths (from #128) built its
fixtures by PutObjectPart-ing plaintext bodies under encrypted-object
metadata with per-part sizes 5245473 and 1. Since #119, PutObjectPart
always derives an encrypted part's plaintext length from the bytes
written and rejects a part that cannot be a valid sio stream, so those
fixtures can no longer be created through a normal write and both
variants failed at write time.
Such an on-disk shape now only exists as pre-#119 data or from an old
peer, which is exactly the state the GetObjectAttributes per-part
tamper check (#128) defends. Inject that ObjectInfo directly through a
stub object layer (the setObjectLayer pattern used by the #110 tamper
test) and exercise the handler, which is what this test pins. The
handler path, the crafted part sizes, and both assertions
(separately-encrypted-parts -> ErrObjectTampered; legacy-single-stream
-> stored fragment sizes) are unchanged. Test-only; reconciles two
already-merged correct changes (#119 and #128).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The raw SSE-C seal headers moved from an inline slice in
cmd/object-multipart-handlers.go to the shared isRawSSECReplica helper
in cmd/replication-trust.go (already recorded), so regenerate the
compat-baseline allowlist to drop the three stale object-multipart
entries. No behaviour change; make rebrand-guard is green.
Refs pgsty/silo#109
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
After a restart a node's remote erasure drives that could not be connected
during startup stay uninstalled until the next connectDisks pass, about 15
seconds later. During that window the liveness, readiness and both cluster
probes answer 200, admin info shows every drive online and mcli ready agrees,
because the cluster probes aggregate each peer's report of its own local
drives rather than the drives this node has installed. A PUT through that
node can still fail with 503 SlowDownWrite and a cross-node GET can answer
404 NoSuchKey until the window closes. The behaviour is inherited from
upstream and reproduced on the 0806 and 0903 releases alike.
Document the window and the bounded data-path check (PUT through each node,
read each object through every node, fixed deadline, re-read acknowledged
objects) that automation should use instead of the probes, and correct the
readiness probe description, which also fails on request-queue overload and
an unreachable KMS. No product change: the probes keep their documented
purpose, and changing them or the reconnect cadence was judged unproven
tuning in the agreed plan.
Refs pgsty/silo#116
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
CopyObjectHandler rebuilt the destination metadata with the public Object Lock keys stripped (cmd/object-handlers.go:1708) and then restored a value only inside retentionMode.Valid() and legalHold.Status.Valid() (cmd/object-handlers.go:1715 and :1732), so a replica update that carried no retention or legal-hold value never reached the ordering comparison and silently erased whatever the destination held, however new it was; a retention removal that did win recorded no ordering timestamp either, so cmd/bucket-object-lock.go:370 later read an unparseable stored timestamp and let an older retained value back in. Each replica field is now decided on its source timestamp first and its incoming value second, and both restore helpers write the stored timestamp back before returning early on an empty stored value, which is the only way a removal timestamp survives the REPLACE metadata directive. Legal hold stays deliberately asymmetric: S3 has no legal-hold removal, an explicitly empty status is already rejected as invalid, and an absent status conveys no change even when an orphaned timestamp arrives with it, so only a valid ON or OFF can win.
Three inherited defects would have defeated that ordering, so they are fixed here too. The SSE-KMS branch of putOptsFromHeaders built its own ObjectOptions and dropped the parsed lock timestamps, leaving every replicated lock update unordered on a bucket with default KMS encryption; it now carries them. The in-place SSE-C key rotation snapshots the stored reserved metadata into encMetadata before the lock decision exists and merges it back afterwards to preserve the encryption headers, reinstating the ordering timestamp the decision had just replaced; the snapshot is now reconciled with the decision for a trusted replica. Finally, the value-less handling applies only to an actual replica: a trusted peer that sends the replication marker without REPLICA status keeps the previous behaviour, so a REPLACE copy carrying no lock headers still writes a version with no retention and no hold.
Tests: TestAPICopyObjectReplicaAbsentLockFieldsPreserveNewerState, TestAPICopyObjectReplicaRetentionRemovalKeepsOrderingTimestamp, TestAPICopyObjectReplicaObjectLockOrdering, TestAPICopyObjectReplicaRetentionRemovalUnderBucketKMS, TestAPICopyObjectReplicaLockTimestampSurvivesSSECKeyRotation and TestAPICopyObjectMarkerOnlyLeavesObjectLockUnchanged, all on ErasureSD and Erasure. Compatibility: no API, wire or stored-field change, and a field arriving with no source timestamp is unordered and now preserves destination state, so an un-upgraded 0806 peer keeps replicating safely while it still runs the old erasing receiver.
Fixespgsty/silo#111
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
The admin bucket-metadata handlers enumerate every bucket config by name, and
per-bucket CORS was never added to that enumeration: export omitted cors.xml
(cmd/admin-bucket-handlers.go:414 cfgFiles) and import ignored the entry
outright, with no case in applyImportedBucketMetadata (:598) or SetStatus
(:629), so a CORS-only archive reported 0/0 buckets imported and a restored
bucket silently lost its configuration. Export now writes the stored document
verbatim and import validates it with the same parser and validator as
PutBucketCorsHandler, merging it under the existing bucket metadata lock and
announcing it through the dedicated SRBucketMetaTypeCorsConfig event; the local
CORS timestamp rule is extracted into localCORSUpdatedAt and reused so an
imported document always lands strictly above bucket creation, which matters
because the import stamps its fields before creating any missing bucket and a
CORS event below Created is dropped as an older bucket incarnation.
The import reads one byte past the declared entry size so archive/zip reaches
EOF and verifies the entry checksum, otherwise a corrupt or over-long entry
carrying well formed XML would overwrite the stored document; and the CORS
event is sent even when the shared bucket metadata hook failed, so an
unreachable peer cannot withhold an already committed CORS document from the
reachable ones.
Tests: TestAdminBucketMetadataCORSRoundTrip and
TestAdminBucketMetadataCORSImportReplicatesPastPeerFailure (new, ErasureSD and
Erasure).
Compatibility: the ZIP gains one entry, older archives stay importable and
leave CORS untouched; no mcli or madmin-go change is needed because
madmin.BucketStatus already carries Cors and mcli copies the export ZIP
verbatim.
Fixespgsty/silo#112
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
GetObjectAttributes decided truncation by comparing the last returned
part number with the part count (cmd/object-handlers.go:720), which is
only a coincidence of contiguous numbering. Sparse parts 1/3 reported a
complete page as truncated with a marker that loops, and parts 1/3/5
with max-parts=1 stopped after part 3 and silently dropped part 5. Set
IsTruncated in the break that proves an eligible part was left
unreturned, and zero NextPartNumberMarker when the listing is complete,
as ListObjectParts already does. Also reject negative x-amz-max-parts
and x-amz-part-number-marker in getAndValidateAttributesOpts with the
same API errors ListObjectParts uses, instead of answering an invalid
request with an empty parts listing; an absent or zero max-parts still
means the default page size.
Tests: TestAPIGetObjectAttributesPartsPagination (sparse 1/3/5 and
contiguous 1/2 walks on ErasureSD and Erasure),
TestGetAndValidateAttributesOptsPartsRange, and the sparse variants of
TestAPIGetObjectAttributesMultipartLogicalPartSize. Compatibility: no
field is added or removed; IsTruncated and NextPartNumberMarker change
only where they were wrong, and negative pagination values that no SDK
sends now fail fast.
Fixespgsty/silo#115
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
GetObjectAttributes filled ObjectPart.Size from the on-disk part length
(cmd/object-handlers.go:715), so every compressed or encrypted multipart
object reported transformed sizes that do not sum to the logical
ObjectSize the same response returns from objInfo.GetActualSize().
Report each part's uploaded plaintext length instead: a compressed part
uses its recorded ActualSize, and a separately encrypted part derives the
plaintext length with sio.DecryptedSize, because ActualSize is the
ciphertext length for a replicated SSE-C part and zero for parts written
before actualSize existed.
Only parts of an encrypted multipart object are streams of their own. A
legacy encrypted object carries no multipart marker and is one continuous
stream that the erasure writer split into storage fragments, so those
fragments keep their stored size. Where a part is a stream, one whose
length cannot be a valid encrypted stream has no logical length, and the
request now fails with XMinioObjectTampered rather than reporting the
ciphertext length; DecryptObjectInfo does not catch that case, because
ObjectInfo.isMultipart gives up on the first bad part and only the object
total is then validated.
Tests: TestAPIGetObjectAttributesMultipartLogicalPartSize (plain,
compressed, SSE-C and compressed+SSE-C, consecutive and sparse part
numbers), TestAPIGetObjectAttributesCompressedEmptyTrailingPart,
TestAPIGetObjectAttributesEncryptedPartLengths, and a part-size assertion
added to TestAPISSECMultipartReplicationTrust. Compatibility: the XML
shape is unchanged and nothing is written to disk, only the value of the
existing Size element is corrected.
Fixespgsty/silo#114
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
With compression allow_encryption=on an SSE-C object is stored as
encrypt(s2(plaintext)), while replication reads it raw (NoDecryption at
cmd/erasure-object.go:257) and putReplicationOpts drops the internal
compression and actual-size headers (cmd/bucket-replication.go:786). The
replica keeps the source seal with no compression marker, so a GET with the
correct customer key returns HTTP 200 and the raw S2 stream instead of the
object, and the source records the transfer as COMPLETED.
Widen the one condition in excludeForCompression (cmd/object-api-utils.go:613)
so SSE-C is never compressed, whatever allow_encryption says. This covers all
four producers at once, PutObject, NewMultipartUpload, CopyObject and
PutObjectExtract, plus any future caller of isCompressible.
crypto.SSEC.IsRequested ignores copy-source headers, so a copy is judged on its
destination key only, and a raw SSE-C replica write is unaffected because it
carries no public SSE-C headers. allow_encryption keeps its meaning for SSE-S3
and SSE-KMS, where the server owns the key and decompresses before replicating.
Tests: TestAPISSECCompressionReplicaStaysReadable (single PUT and multipart),
TestAPISSECCompressionProducerMatrix, TestAPISSECCompressionSkippedOnCopyObject,
TestAPISSECCompressionSkippedOnSnowballExtract and the control
TestSSECBatchReplicationCannotRead in cmd/compression-ssec_test.go. Two existing
expectations pinned the removed shape and are updated:
TestAPICopyObjectSSECKeyRotationNullVersionCompressesRewrite is renamed
TestAPICopyObjectSSECKeyRotationNullVersionSkipsCompression and now expects an
uncompressed rewrite, keeping its body, checksum, version and ETag assertions;
the SSE-C compressed-encrypted variant of
TestAPICopyObjectServerSideChecksumEncryption becomes compressible-extension and
expects an uncompressed destination, its SSE-S3 sibling keeping the compressed
coverage.
Compatibility: a deliberate behaviour change. Deployments with
allow_encryption=on no longer store new or rewritten SSE-C data compressed, so
those writes cost more space; objects already stored compressed keep working on
the source and are the concern of pgsty/silo#109, which rejects them at
replication time. Multipart uploads initiated before this change keep
compressing their parts from the metadata saved at initiation. Upstream
468a9fae8 refused this combination at PUT time and a2cab0255 removed the guard;
upstream master is still unguarded, so this is a deliberate divergence.
Fixespgsty/silo#118
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
Trusted SSE-C replication uploads parts as raw ciphertext with the
ciphertext length as Content-Length, and erasureObjects.PutObjectPart only
derived the plaintext length when the caller passed a negative size, so
each replicated part persisted the ciphertext length as ActualSize (the
field defined as the uploaded size without encryption bytes). On the
replica, partNumberToRangeSpec turned those lengths into a plaintext range,
so GET/HEAD ?partNumber=N returned the wrong bytes and shifted
Content-Range (2560, 2560 and 5120 bytes for 5 MiB, 5 MiB and 1 MiB
parts), and a later decommission or rebalance re-uploaded the parts with
the stale value and recomputed the object-level actual-size from their sum,
after which a whole-object GET advertised a Content-Length larger than the
body it wrote.
Derive the plaintext length of an encrypted, uncompressed part from the
bytes actually written (sio.DecryptedSize) in PutObjectPart, the single
place a part is persisted, rejecting a length that cannot be a valid
stream before the part is committed; and derive part lengths from
part.Size in partNumberToRangeSpec for encrypted, uncompressed objects,
returning an error instead of a nil range, so replicas already on disk
read correctly without a resync. Compressed parts keep ActualSize.
Tests: TestAPISSECReplicaPartNumberReads (three-part SSE-C replica,
?partNumber=N bytes, Content-Length and Content-Range equal the source)
and TestSSECReplicaPartActualSizeDataMovement (replay through the data
movement path leaves part and object sizes at plaintext values) fail on
main and pass with the fix on ErasureSD and Erasure;
TestAPIGetObjectWithPartNumberHandler, TestAPISSECMultipartReplicationTrust
and TestAPIListObjectPartsHandler stay green. Compatibility: no wire or
API change; objects an unfixed server already moved carry a poisoned
object-level actual-size and need a rewrite or resync.
Fixespgsty/silo#119
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
An in-place SSE-C key rotation takes the fast path at cmd/object-handlers.go:1523
that only rewraps the object key, while every line that turns
x-amz-checksum-algorithm into a stored checksum lives in the re-encrypting else
branch at 1571-1609, so a requested algorithm was silently dropped and the stale
source checksum was kept and reported. Extend the canRotateKeyInPlace guard so a
client request carrying the header falls through to the copy that recomputes,
stores and reports it. Replica-trusted requests keep the fast path: getOpts
leaves their source reader encrypted, so a rewrite would hash ciphertext, and a
replica has to keep the checksum its source assigned.
Tests: TestAPICopyObjectSSECKeyRotationChecksumAlgorithm (new, red before the
guard), TestAPICopyObjectSSECKeyRotationKeepsChecksumAbsence (new, pins the
accepted limitation that a headerless rotation preserves the stored checksum
state including absence, gaining no default CRC64NVME) and
TestAPICopyObjectSSECKeyRotationReplicaKeepsFastPath (new, pins the replica
carve-out on a non-empty and on a zero byte source).
Compatibility: no API or wire change; a rotation without the header and every
replica-trusted rotation are unchanged, while a client rotation carrying the
header now rewrites the object data, so the ETag changes, a multipart source
collapses to a single part object, the copy replicates as an object rather than
as metadata, and the rewritten bytes are compressed if compression is enabled for
that object, as AWS CopyObject documents. Upstream MinIO carries the same
defect from 2718d9a43 (minio/minio#21399); this is a deliberate divergence.
Fixespgsty/silo#113
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
A raw SSE-C replica write carries the source ciphertext and the source seal
in X-Minio-Replication-Server-Side-Encryption-* headers but no public SSE-C
request headers, so crypto.Requested() was false and PutObjectHandler and
NewMultipartUploadHandler applied the destination's default encryption and
compression to bytes that were already ciphertext (upstream 468a9fae8,
"Enable replication of SSE-C objects", never exempted the raw path). With
destination default SSE-S3 the replica's IV and seal were overwritten and
GET returned 400; with destination compression the replica stored
compress(ciphertext) and GET failed, while the source reported COMPLETED.
Recognize a validated raw SSE-C replica (replicaTrusted plus a seal header,
shared helper isRawSSECReplica) and skip bucket default encryption,
compression and the encryption branch on the single PUT path, and default
encryption plus compression on the multipart initiation path, which already
skipped key generation. On the sender, reject replication of an object that
is both compressed and SSE-C, since the wire carries no compression state
and the destination would otherwise store an undetectable S2 stream, and
make replicateObject/replicateAll report a putReplicationOpts failure as
Failed instead of Completed.
Tests: TestAPISSECReplicaSkipsDestinationTransforms (single PUT and
multipart under destination default SSE-S3, compression and an explicit SSE
header, plus an untrusted control), TestAPISSECMultipartReplicaRoundTripWith
Compression, and TestPutReplicationOptsRejectsCompressedSSEC fail on main
and pass with the fix on ErasureSD and Erasure; the replication-trust,
multipart and PutObject suites stay green. Compatibility: no wire, API or
metadata change; the destination change applies only to trusted replica
writes carrying a source seal; replicas already transformed must be
rewritten from an intact source (see pgsty/silo#120 for why a resync does
not do that yet).
Fixespgsty/silo#109
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
ErrObjectTampered was mapped to http.StatusPartialContent since upstream
ca6b4773e (2017), so a GET or HEAD of an object the server cannot decode
(invalid encrypted size, malformed actual-size, bad multipart ETag shape)
answered with a success status and an XML error document that SDKs handed
back as object content; boto3 returned the XML as Body and a zero-length
HEAD as success. Every origin of errObjectTampered is a stored-state
defect, not caller input, so map the entry to 500 Internal Server Error
and keep the XMinioObjectTampered code and message. The comment records
the deliberate divergence from upstream.
Tests: TestObjectTamperedGETHEADStatus (signed GET and HEAD, ErasureSD and
Erasure) fails with 206 on main and passes with 500; TestAPIErrCode,
TestAPIErrCodeDefinition, TestAPIHeadObjectHandler,
TestAPIHeadObjectHandlerWithEncryption and TestAPIGetObjectHandler stay
green. Compatibility: only the status line of one MinIO-specific error
changes; clients now retry damaged-object reads per their 5xx policy.
Fixespgsty/silo#110
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
Adopt silo-pkg v3.13.2, the 20260903 mcli release, and the latest validated SILO Console commit. Move maintained source imports to the pgsty/silo-pkg module path, refresh the resulting dependency closure and credits, and keep only documented legacy minio/pkg transitive residue.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Flush S3 Select records queued before an error message (an inherited select
race that the CVE-2026-39414 regression test caught under the race detector
in CI), update golang.org/x/crypto to v0.56.0 for GO-2026-6354/6355 reachable
through the SFTP server, move the toolchain to Go 1.27.1, and extend the
message writer tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QwirCh4nsrJCQp2WaXoVLK
Signed-off-by: Feng Ruohang <rh@vonng.com>
Go 1.27.1 (2026-09-01) carries fixes to the runtime, compiler, net/http,
encoding/json, os and database/sql. The go directive, the dependency check
script and the image build stage follow it; CI reads the version from
go.mod. The remaining golang.org/x modules the server requires are already
at their latest releases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QwirCh4nsrJCQp2WaXoVLK
Signed-off-by: Feng Ruohang <rh@vonng.com>
Exercise the writer directly and decode the event stream with the client
parser: a buffered plus a queued record before an error, an error with
nothing queued, a record larger than the staging buffer on both the success
and the error path, and the unchanged success ordering of records, Stats and
End.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QwirCh4nsrJCQp2WaXoVLK
Signed-off-by: Feng Ruohang <rh@vonng.com>
GO-2026-6354 and GO-2026-6355 (denial of service on deadlocked SSH channels)
are reachable through the SFTP server, which listens with x/crypto/ssh.
v0.56.0 carries the fixes; govulncheck reports no reachable vulnerability.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QwirCh4nsrJCQp2WaXoVLK
Signed-off-by: Feng Ruohang <rh@vonng.com>
The writer goroutine selects between the record channel and the error
channel at random. When Evaluate had just queued a record and then reported
an error, the error case could win first; it flushed only the staging
buffer, so the queued record was dropped in the exit drain and the client
saw the error without the records that preceded it. Stage whatever is queued
before flushing and writing the error. The CVE-2026-39414 regression test
asserted this contract and failed once under the race detector in CI; a new
unit test exercises the ordering directly.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QwirCh4nsrJCQp2WaXoVLK
Signed-off-by: Feng Ruohang <rh@vonng.com>
A bucket that is resident keeps its last loaded metadata through a failed
refresh, exactly like every other bucket configuration, so the load-failure
set only ever holds buckets that were never loaded. This makes the CORS
lookup's precedence explicit: resident document first, fail-closed only for
a bucket that has no loaded document.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
A replicated CopyObject uses the REPLACE metadata directive, so the map the
handler compared replication timestamps against had already been rebuilt
from the request and filtered of Object Lock keys: the stored retention and
legal-hold timestamps were never seen, every replica update was applied
regardless of order, and the legal-hold timestamp was written under the
retention key. A stale replica could turn a newer legal hold off or shorten
a newer retention.
Capture the stored Object Lock state before the metadata is rebuilt, apply a
replica update only when its source timestamp is newer, put the stored state
back when the update is stale, and keep each timestamp under its own key.
Inherited from upstream; recorded in the advisory ledger.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Fifteen reviewed commits: streaming-trailer trust fix and a single
replication-trust evaluator, Object Lock versioning invariant, resident-only
CORS lookup with fail-closed startup and load-failure states, dead-code
removal, compatibility baseline without exported symbols, upstream minio-go
with Console 43f8447fd and mcli 20260901, advisory ledger SN-2026-006..010,
contributor records, verification scripts with a valid root password, and a
deterministic dynamic-timeout test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
A successful on-demand load makes the bucket resident, so the failure
recorded by an earlier startup or refresh no longer applies. Clearing it here
keeps the set's invariant exact instead of waiting for the next refresh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
TestDynamicTimeoutAdjustExponential and TestDynamicTimeoutAdjustNormal
seeded the global generator and then drew from it while other tests in the
package may use the same generator, so the sample was not the one the seed
promised and the exponential case failed once in a full race run. A private
source makes both tests deterministic.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Restore the load-failure set that the cleanup removed. A presigned URL is
authenticated by its signature, so for such requests the bucket's CORS
document is the only origin boundary a browser enforces; a real bucket whose
metadata failed to load must therefore not be answered with the global
policy, and without this bit it is indistinguishable from a name that is not
a bucket. Two helpers own the set's lifecycle; the resident-only lookup and
the removal of the internal-namespace special case stay.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Restore the startup guard removed by the previous cleanup: while bucket
metadata is still loading, a non-resident name may be a bucket with a
restrictive CORS document, so the request gets no CORS answer instead of the
global policy. After startup a non-resident name still falls back to the
global policy without any metadata I/O; the separate load-failure set stays
removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
The load-time normalization compared the stored lock document with the
canonical enabled document byte for byte, so a lock configuration that also
carries a default retention rule kept a suspended or prefix-excluded
versioning document. Decide from the parsed configuration instead, after it
is parsed, so every writer that goes through Save, including the site
replication versioning and heal paths, ends with plain Enabled versioning on
a locked bucket. Receiving a lock configuration on a bucket created without
lock now enables versioning as well; the test that asserted the opposite is
updated, and a new test covers a rule-bearing lock document with suspended
and prefix-excluded versioning through Update, Get, and reload.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
The rebrand shortened the scripts' root password from minio123 to silo123.
The server requires at least eight characters, so every script that starts
a server with it failed at startup and verify-build.sh then waited forever
on mc ready. No workflow runs these scripts, which is why it went unnoticed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
The silo-go fork is retired: silo-pkg v3.13.0, Console v2.3.0, and mcli
20260901 all require upstream github.com/minio/minio-go/v7 again. Drop the
replacement and require the same upstream pre-release they use
(v7.3.1-0.20260828014306-0e78d3f18efe, one commit past the fork's base).
Console moves from e07ef01 (v2.2.1 plus pins) to 43f8447fd, the last commit
of the v2.3.0 line before Console adopted the github.com/pgsty/silo-pkg/v3
module path. It carries the six v2.3.0 security fixes (forwarding-header
trust, outbound TLS verification, credential redaction, WebSocket session and
connection caps), the IAM wildcard and session-identity fixes, and their
regenerated assets, while still consuming silo-pkg through the existing
replacement. mc stays on the last commit before its own path migration;
silo-pkg stays on the last commit that declares the github.com/minio/pkg/v3
path, which differs from v3.13.0 only by that path change. CREDITS follows
the module set.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
The baseline recorded 9,051 exported symbols of the main and internal
packages, which nothing outside this module can import, and 112 request
paths that exist only in test fixtures. Both changed with almost every
functional commit and protected no compatibility promise. The guard now
records routes from non-test files only, ignores untracked files, and drops
the symbol set; the baseline shrinks from 522 KB to 46 KB. CONTRIBUTING
explains when to refresh it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Ledger entries for the zero-byte SSE-C key check, GetObjectAttributes
authentication, replication request trust, user and group status
authorization, and DeleteObjectVersion authorization, plus the Go 1.27
toolchain refresh. The ledger names pgsty/silo, CONTRIBUTORS lists the
per-bucket CORS, ChecksumType, and NoSuchBucket contributors, and the CORS
design record states its merged status without the review logs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
mcli RELEASE.2026-09-01 carries the credential redaction fixes for --debug
output. The go.mod comments now state what the Console and silo-pkg pins are:
the last commits that consume silo-pkg through the github.com/minio/pkg/v3
replacement, since silo-pkg v3.13.0 and Console v2.3.0 moved to the
pgsty/silo-pkg module path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Assigning the two pipe halves before returning them removes the gofumpt and
gofmt disagreement that needed a permanent formatter exclusion. gomodguard is
deprecated in golangci-lint v2.12; the v2 linter takes the same configuration.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
checkSSECCopySourceKey duplicated the source-key authentication that both
CopyObject read paths already perform; the DeleteObjects signature-error
branch became unreachable once signatures are verified once per request; the
discrete PostgreSQL and MySQL notification environment constants have had no
reader since DSNs became mandatory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
The pre-authentication CORS lookup stays resident-only, so client-supplied
path segments still cause no metadata I/O and no cache growth. The
fail-closed states for startup, load failures, and the internal namespace
are gone: CORS is a browser response policy rather than an authorization
boundary, and failing closed only denied browser clients CORS headers while
bucket metadata was still loading. A bucket whose stored CORS document does
not parse still gets no CORS headers. This removes the loadFailed bookkeeping
and the unused GetCorsConfig, HasAllowedOrigin, and generic Update path for
CORS; tests use the CORS-specific writer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Site adoption and ForceCreate preserved a suspended or prefix-excluded
versioning configuration while bootstrapping Object Lock, persisting a state
that PutBucketVersioning itself rejects: objects under an excluded prefix in
a WORM bucket were not versioned and escaped retention. enablePeerBucketVersioning
now takes the lock intent and replaces such configurations with plain Enabled
versioning, and metadata loading ignores prefix exclusions on a locked bucket
as it ignored suspension before. The adoption tests assert the normalized
state and keep the timestamp-preservation checks on valid documents.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
A request that does not earn replication trust continues with a clone whose
internal replication headers are removed. r.Clone copies the Trailer map, but
the streaming body reader created from the original request fills the
original map, so a trailing checksum was never seen by the hash reader and
PutObject and UploadPart with STREAMING-UNSIGNED-PAYLOAD-TRAILER failed with
XAmzContentChecksumMismatch whenever an untrusted X-Minio-Source-* header was
present. Share the trailer map with the clone, as the Snowball path already
does for its per-entry requests, and cover both handlers with a test.
The marker evaluation that was copied into six handlers now lives in
evaluateReplicationTrust so the rule (a declared replica without the
replication permission is rejected; trust needs the exact marker plus the
permission) is defined once.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvgysXDmhPBBimCReYtA8q
Signed-off-by: Feng Ruohang <rh@vonng.com>
Degrade to the loaded in-memory configuration only when the short migration lock cannot be acquired. Preserve parse, I/O, KMS, and persistence errors once the lock is held.
Signed-off-by: Feng Ruohang <rh@vonng.com>
State that old cors-config.lock and new metadata.lock do not interoperate during a rolling upgrade, while the unchanged on-disk format remains rollback-compatible.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Use a short dedicated migration lock timeout and fall back to the already loaded legacy or target configuration in memory when persistence is contended. Never fall back to an unlocked metadata save, and keep peer fan-out contexts free of stale lock markers.\n\nRefs: #102
Signed-off-by: Feng Ruohang <rh@vonng.com>
Release metadata.lock with defer and dispatch only the configuration types actually present in the import. Avoid stamping unrelated peer metadata with the import timestamp.\n\nRefs: #102
Signed-off-by: Feng Ruohang <rh@vonng.com>
Preserve existing records only for ForceCreate, reject ghost metadata on genuine creation, keep object-lock versioning invariants, and complete metadata saves after caller cancellation. Expand deterministic coverage for peer bulk, lifecycle delete, ghost creation, and cancellation.\n\nRefs: #102
Signed-off-by: Feng Ruohang <rh@vonng.com>
Apply only validated import fields to a fresh locked record, block ForceCreate after real read errors, and route legacy or target-config migration saves through the shared lock. Compute lifecycle deletion state from the locked record.\n\nRefs: #102
Signed-off-by: Feng Ruohang <rh@vonng.com>
After storage bucket creation, merge required versioning and lock defaults into the latest on-disk metadata under metadata.lock. Avoid ForceCreate and site-adoption rewrites that replaced existing bucket configuration.\n\nRefs: #102
Signed-off-by: Feng Ruohang <rh@vonng.com>
Create policy and CORS state, force-create the existing bucket, and require the original Created time and both metadata fields to survive.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Use one per-bucket metadata.lock for ordinary updates, CORS transitions, and legacy bulk replication. Persist and update the local cache while locked, then release before peer metadata reload fan-out.\n\nRefs: #102
Signed-off-by: Feng Ruohang <rh@vonng.com>
Pause one whole-record writer at the metadata PutObject boundary and let a different config writer commit from the same stale snapshot. Assert that policy+CORS and tagging+SSE both survive on disk and in the resident cache.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Authenticate DeleteObjects before validating entry count, remove the obsolete per-version auth helper, and pin the marker-only request to the ordinary authorization path.\n\nRefs: #58
Signed-off-by: Feng Ruohang <rh@vonng.com>
Document that replication targets retain the DeleteObject plus ReplicateDelete contract and extend the existing two-site test with a target user that lacks DeleteObjectVersion. Use mc for explicit version deletion so the gate no longer depends on AWS CLI.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Select DeleteObject or DeleteObjectVersion from each request's effective version ID for single and multi-delete. Authenticate multi-delete once, bind version conditions to each XML entry, and keep the established DeleteObject plus ReplicateDelete target contract with explicit version denies honored only on trusted replication.\n\nRefs: #58
Signed-off-by: Feng Ruohang <rh@vonng.com>
Prove that DeleteObjectVersion-only principals cannot delete named UUID or null versions while DeleteObject-only principals can, contrary to the S3 action mapping in issue #58.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Accept the new CORS test routes, resident getter, and replication header literals in the rebrand guard. Apply gofumpt, context-first helper ordering, and spelling fixes required by CI.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Snapshot per-entry requests after applying bucket encryption defaults but before streaming trailers are consumed. Keep authorization failures fatal while retaining Snowball ignore-errors behavior for object-lock failures.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Place synthetic permission-check objects under each enabled rule's effective prefix, so least-privilege target policies are validated against the namespace they will actually replicate.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Evaluate PutObject and ReplicateObject permissions with immutable per-entry request snapshots during concurrent Snowball extraction. Preserve the first API error without sharing mutable handler state, and cover prefix-scoped trust under the race detector.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Evaluate ReplicateObject or ReplicateDelete before returning the no-op validation response, so underprivileged target credentials fail during replication setup instead of at runtime.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Count only valid live CORS states in per-site summaries. Treat baselines and tombstones as absent, and diagnose malformed payloads or missing source timestamps.\n\nRefs: #77
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep pre-authentication CORS lookups resident-only so attacker-controlled path segments cannot trigger metadata I/O or grow the metadata cache. Preserve fail-closed behavior for startup, load failures, invalid metadata, and the internal namespace.
Centralize replication request trust after authentication, distinguish general replication from replica-only privileges, and gate SSE-C ciphertext handling, source metadata, object-lock bypasses, event suppression, delete semantics, and replica status on the appropriate permission. Add least-privilege, multipart, PostPolicy, CORS amplification, and compatibility regressions.
Signed-off-by: Feng Ruohang <rh@vonng.com>
GetObjectAttributes lets a replication peer read SSE-C attributes without
presenting the customer key. That carve-out was keyed on the
X-Minio-Source-Replication-Request header alone, which any client can set,
so a caller holding only s3:GetObject could read an SSE-C object's ETag,
plaintext size, part list, and checksums without the key -- the response was
byte-identical to one made with the correct key.
Gate the carve-out on s3:ReplicateObject for the target object, mirroring
CopyObjectHandler's existing replication check.
The pre-existing test asserted the carve-out with root credentials, which
hold every action and therefore cannot tell a gated check apart from an
ungated one. Add least-privilege cases that do: a reader without
s3:ReplicateObject is now refused, a caller that holds it keeps the
carve-out, and the ordinary key-bearing path is unchanged.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Select merged MCLI, Console, and silo-pkg source commits under GOWORK=off so the Server module graph matches the reviewed client, policy, and environment behavior.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Remove the remaining type-only canonicalization at CompleteMultipartUpload while preserving legacy uploads stored as FULL_OBJECT.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Guard only the per-bucket metadata lookup, then retain the global handler Vary and originless preflight behavior for non-CORS traffic.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Return InvalidArgument for CRC64NVME with COMPOSITE at multipart initiation and PutObject instead of silently canonicalizing the request to FULL_OBJECT.
Signed-off-by: Feng Ruohang <rh@vonng.com>
After zero-byte reads authenticate SSE-C at the object layer, equal invalid rotation keys match non-empty reads and return AccessDenied before the rotation-specific distinction.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Unseal supplied SSE-C and copy-source keys after range and request preconditions when a zero-byte read cannot build a decryptor. Preserve internal no-decryption, replication, restore, and absent-header reads.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Unseal the supplied customer key after request preconditions before returning object size, ETag, checksum, or part attributes. Cover zero-byte and non-empty objects with correct, wrong, and missing keys.\n\nRefs #84.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Serialize release and finalize work by tag, replace only one unfinalized Draft, refuse published or finalized release state, and pin GoReleaser to the checked-out tag. Add fail-closed release-state fixtures to the release pipeline.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Count only each site own valid bucket metadata, populate quota totals, and keep malformed fields from suppressing unrelated bucket statistics. Emit bounded diagnostics for invalid payloads.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Reject unimplemented x-amz-checksum value and trailer names instead of accepting uploads without verification. Apply the same contract to PutObject, multipart initiation and parts, CopyObject, and UploadPartCopy while preserving the five supported algorithms.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Populate ObjectLockConfig for live, initial-sync, and heal events, while accepting the legacy Tags field during rolling upgrades. Exercise signed admin dispatch and remote-heal transport on both object-layer backends.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Bypass per-bucket metadata work for non-CORS traffic, including admin and Console routes. Keep operational metadata errors fail-closed and pin the existing global fallback for missing buckets.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Exercise the silent compression-on-copy path, compressed SSE-C re-encryption, and the equal-invalid-key error contract on both object-layer backends.
Signed-off-by: Feng Ruohang <rh@vonng.com>
A key rotation rewraps the object key held in metadata; it never re-encrypts
the stored bytes. CopyObjectHandler took that shortcut whenever the request
looked like a same-object SSE-C rotation, on the assumption that the object
layer would then leave the stored bytes alone. That is the same assumption
copyRewritesObjectData() was added to stop making.
When the source is a null version on a bucket that gained versioning after the
object was written, the object layer cannot reference that version and falls
back to PutObject. The reader at that point holds plaintext decrypted with the
old key and no EncryptFn is set, so the destination ends up storing plaintext
under metadata that claims the object is SSE-C encrypted. A subsequent GET
failed with "sio: unsupported version".
Gate the rotation shortcut on the same prediction the compression metadata
already uses. When the object layer stores new object data the rotation falls
through to the regular re-encrypting copy, which decrypts with the old key and
re-encrypts with the new one. The source version selection moves next to the
gate because both decisions need it.
That fallback authenticates the source key through the source decryptor, which
GetObjectNInfo does not build for a zero byte object. Check the key explicitly
before the destination is written, so the gate cannot turn a rotation that the
shortcut rejected with AccessDenied into one that succeeds. The re-encrypting
copy regenerates the encrypted ETag, unlike an in-place rotation; the test
records that difference.
The other three object layer CopyObject callers that set metadataOnly -
PostRestoreObjectHandler, updateRestoreMetadata and batchKeyRotate - address
the same version on both sides and never set Versioned, so they only reach the
two in-place cases already covered by the copyRewritesObjectData table.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fk3PAD7DHCYzcyegYWqAmt
CopyObjectHandler recorded the source compression metadata whenever the copy
was metadata-only, on the assumption that the object layer would then leave
the stored bytes alone. That assumption does not hold. Both
erasureServerPools.CopyObject and erasureSets.CopyObject only skip a data
rewrite in three cases, and otherwise fall back to a full PutObject.
The reachable gap is a copy whose source is a null version on a bucket that
gained versioning after the object was written. Neither version ID is set, so
the self-referential version branch is skipped, the data is rewritten as
plaintext, and the preserved compression metadata then described bytes that
no longer exist. A subsequent GET failed with "s2: corrupt input".
Mirror the object layer's decision in copyRewritesObjectData and record the
compression metadata from it, so the metadata always describes whichever
bytes are finally stored. The source version selection that lets a versioned
metadata-only copy add a self-referential version moves next to the same
decision, since both depend on the effective metadata-only value.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Exercise no-site, local-only, remote-only, and both-site CORS summary accounting through the real siteReplicationStatus seam.
Refs #75
Signed-off-by: Feng Ruohang <rh@vonng.com>
Record the frozen B2+B3 commit, combined local gates, raw SigV4 validation, real two-site offline/delete/heal/restart evidence, and the separate public documentation QA boundary.
Refs #75
Signed-off-by: Feng Ruohang <rh@vonng.com>
Integrate the strict B3 XML, validation, checksum, wildcard, MaxAge, and Origin-null response contract with the C-prime site-replication register from #75.
Preserve fail-closed metadata behavior and rejected-preflight cache variation while keeping legacy-invalid development metadata readable and repairable through a valid CORS PUT or DELETE.
Add combined parser, handler, browser-response, namespace, replication, restart, and legacy-repair regressions, and update the internal design contract.
Refs #75
Signed-off-by: Feng Ruohang <rh@vonng.com>
Define a deterministic CORS replication register with durable tombstones, strict source timestamps, equal-time conflict ordering, full-state status, and heal convergence.
Serialize local and peer CORS transitions with a distributed namespace lock, validate canonical transport payloads, preserve initial-sync deletes, and fail closed on metadata errors.
Add adversarial, concurrent, restart, status, heal, signed admin-dispatch, protocol, and middleware coverage together with the reviewed site-replication design record.
Refs #75
Signed-off-by: Feng Ruohang <rh@vonng.com>
Site replication emitted SRBucketMetaTypeCorsConfig on PutBucketCors, but
the peer receive/apply, initial-sync, status, and heal paths did not carry
the CORS metadata. Replicated sites could therefore diverge on CORS config
even though the originating request succeeded.
Complete every site-replication path for CORS, mirroring the SSEConfig
pattern:
- peer apply: PeerBucketCorsConfigHandler + item.Cors handling in
PeerBucketMetadataUpdateHandler, with an updatedAt staleness guard
- initial sync: push existing CorsConfigXML via BucketMetaHook
- status: parse per-site CorsConfig, count/compare, surface
CorsCfgMismatch/HasCorsCfgSet/ReplicatedCorsConfig, and include CORS in
the bucket-stats aggregation filter
- heal: healCORSMetadata, including nil -> delete propagation
Also harden the request/config path:
- PutBucketCors validates the supplied Content-MD5/checksum via
validateLengthAndChecksum
- CORS validation rejects more than one wildcard per AllowedOrigin/
AllowedHeader and enforces the 255-char rule ID limit
- preflight responses Vary on Origin, Access-Control-Request-Method, and
Access-Control-Request-Headers
Add focused tests for the CORS SR transport round-trip, the metadata
equality helper, and the new validation constraints.
Signed-off-by: h5vx <h5v@protonmail.com>
Return AWS-compatible errors for CompleteMultipartUpload checksum failures without changing the global streaming checksum mapping. Compare explicit multipart checksum types symmetrically, distinguish missing composite part checksums, and preserve the CRC64NVME canonicalization pending a direct AWS probe.
Signed-off-by: Feng Ruohang <rh@vonng.com>
ListObjects shortcuts can return EOF before consulting storage, causing missing buckets to appear as empty listings. Verify bucket existence only on those shortcuts so the normal listing path retains the upstream fan-out optimization.
Cover ListObjects, ListObjectsV2, and ListObjectVersions at the object layer and verify HTTP 404 NoSuchBucket responses.
Fixes#32
Co-authored-by: Jason Lin <jason@JasondeMacBook-Air.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The legacy etcd federation branch of CopyObjectPartHandler forwards copied
bytes with minio-go Core.PutObjectPart, which can only recover a checksum
from response headers. After the server-side part checksum work, the remote
computes and persists the checksum, but an AWS-compatible UploadPart response
correctly omits a checksum the request did not supply, so the proxy had
nothing to put in CopyPartResult.
The destination now returns the non-empty checksum fields of the PartInfo
produced by that exact write, but only when the request carries the
minio-federated application token that getRemoteInstanceClient already
attaches. Ordinary UploadPart responses are unchanged, and the checksum type
is deliberately not returned because UploadPart does not carry it. The
User-Agent is a response-shape hint only: it never gates authorization,
visibility or validation, and it can expose nothing beyond the checksum of
the body the caller just uploaded.
Reading the checksum from the same PartInfo that produced the response ETag
also keeps the pair bound to one write, so a concurrent overwrite of the same
part number cannot publish another writer's checksum.
Tests cover the application token gating matrix including lookalike tokens,
the real minio-go response parser, concurrent overwrites of one part number,
and an in-process two-deployment probe that drives the federation branch
through the real getRemoteInstanceClient into a real PutObjectPartHandler for
both FULL_OBJECT and COMPOSITE uploads.
Fixes#64
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reformat cors_test.go per gofumpt and regenerate the rebrand-guard
compatibility baseline to record the per-bucket CORS feature's new
exported symbols (internal/bucket/cors types and BucketMetadata/
BucketMetadataSys additions).
Signed-off-by: h5vx <h5v@protonmail.com>
Record the new strict named-policy and service-account write behavior, the additional admin-policy validation it activates, the compatibility boundaries retained for stored/imported/replicated policies, and the required manual rewrite for legacy bare ARN prefixes.
Use silo-pkg v3.12 strict validation when creating named policies and when creating or updating service-account session policies. Keep stored policy loads, IAM import, site replication, and STS inline policies on the permissive compatibility path.
Accept checksum-enabled UploadPart and UploadPartCopy requests when clients omit optional per-part checksum headers. Compute over the logical plaintext stream, persist the result, and return it from CopyPartResult while preserving client checksum validation.\n\nRefs #46; follow-ups #63 and #64 remain out of scope.
Reject pre-KV PostgreSQL and MySQL targets that lack a canonical connection string, propagate the typed migration error to the fatal startup boundary, and stop emitting unregistered discrete connection keys.\n\nCloses the implementation for #53; release and issue closure remain separate gates.
Standardize the related SILO components on Go 1.27 tooling, etcd 3.7.1, current Go-maintained modules, shared runtime versions, and explicit security and portability pins.
Keep the shared package Go 1.26 consumer floor, isolate lint tooling from product dependency selection, and preserve upstream-compatible import paths.
Keep NVMe/HDD pool pairs useful without remote tiering: promote objects that
are read often, demote previously moved objects once they go idle, and leave
the feature off until operators set a two-pool topology.
Signed-off-by: mr javad seydi <seydi.birjand@gmail.com>
Condensing the READMEs dropped the systemd drop-in that keeps data ownership
stable when silo.service takes over an existing minio.service, which the
rebrand guard pins in both files precisely so it cannot vanish quietly. The
path is back as a clause in the install note rather than a section, so the
guard is satisfied and the migration-critical detail stays discoverable in
the repository, not only on the portal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
GitHub generates no contributor graph for forks, so this fork had no
attribution record at all. CONTRIBUTORS.md becomes that record, in three
tiers: the four contributors with code merged into main, each listed with
the pull request and the commit that carries their authorship; the five who
opened pull requests; and the twenty-two who filed the bug reports and
compatibility findings that shaped the releases.
Both READMEs are condensed and gain a Contributors section that shows the
avatars and points at the full record, with README_ZH realigned to README
section for section.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
GPG-signing the RPMs rewrites their bytes after release.yml has already
generated SBOMs, the packages checksum manifest, and attestations from
the as-built files. The new workflow_dispatch-only finalize-release lane
runs between signing and publishing: it refuses non-Draft releases,
verifies the signed RPMs against their sha256 sidecars and the committed
PGSTY public key, confirms every other package still matches the
original manifest, regenerates the RPM SBOMs and the manifest from the
published bytes, cosign-signs the manifest under the workflow identity,
and attests the finalized set before replacing the assets in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The substring grep for the archive line also captured the archive's
.sbom.json entry, whose file is deliberately not downloaded in this
lane, so the sha256sum check failed on every run. awk now matches the
manifest filename column exactly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rpm carried a bare -1 release while the deb carried none. Both now
ship the PGDG-style 1PGSTY segment: silo-VER-1PGSTY.arch.rpm and
silo_VER-1PGSTY_arch.deb, with nfpm rendering it as the RPM Release tag
and the Debian revision respectively. The apk stays bare because Alpine
pkgrel admits only -r<integer>.
sign-release-rpms.sh declares the value once as expected_release;
package-release.sh names artifacts from its own PKG_RELEASE copy, and
test-release.yml evals the signing script's value and asserts both the
download names and the packaged RPM Release/DEB Version against it, so
the copies cannot drift apart silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The go list -deps closure is GOOS-specific, so the CREDITS generated on
darwin (carrying go-m1cpu) could never match a Linux regeneration
(ethtool, numcpus), and the check-gen guard failed on its first CI run.
Generation now pins linux/amd64 with the release build tags, so the
output is identical on every host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundle mcli RELEASE.2026-08-06T00-00-00Z in the container image (pin
and both architecture digests; interop with the new server was
verified end to end during release acceptance) and align the CI
fixture default in install-mcli.sh.
Point the Helm chart defaults at the first real silo image tag:
docker.io/pgsty/silo is a fresh repository, so the inherited
RELEASE.2026-08-04 default could never pull. Chart version moves to
7.0.1 with appVersion RELEASE.2026-08-06T00-00-00Z; the packaging
tripwire in verify-helm-migration.sh follows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gocritic's ifElseChain check fails make lint on the --json/--quiet
branch introduced by b6d47b739; the quality job runs make lint on
every push to main. Behavior is unchanged (verified by re-running the
json/quiet success and failure cases byte-for-byte).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Findings from an adversarial review (Codex, gpt-5.6-sol at max effort)
of 2ff594f4b and 4c34d2309, each independently verified before fixing:
- SBOM generation: buildx attaches a provenance attestation, so every
per-arch digest names an OCI index; Syft's platform default on an
amd64 runner cannot resolve an arm64-only index and the step dies.
Pass --platform explicitly on all four Syft calls (the two classic
lanes had the same latent defect - the renamed workflow has not run
yet, which is why it never fired).
- Release ordering: the HEALTHCHECK survival check now runs against
the pushed architecture image before the versioned and rolling
multi-arch manifests are created, so a broken health config blocks
their promotion; the comment now states honestly that the
arch-suffixed tags are already public at that point.
- Gate assertions: tar's member-argument mode exits non-zero on any
missing name, which under pipefail masked a found forbidden file
when exactly one of them existed; -tv prints symlinks as
'name -> target', defeating $-anchored greps; and the licenses
check proved only one-of-three. Export the rootfs once and assert
every required and forbidden entry individually (busybox/sh and
usr/bin/mc[li] now covered), and match the image healthcheck as an
exact array instead of a substring.
- Probe target vs CLI-configured servers: a probe process cannot see
PID 1's argv, so --url gains EnvVar MINIO_HEALTHCHECK_URL as the
documented way to point the baked-in HEALTHCHECK at a server whose
address/TLS comes from command-line arguments (verified end to end:
server on --address :9010, env var alone turns the container
healthy). Baseline regenerated for the new env token.
- IPv6 zone identifiers: serialize probe URLs via url.URL.String()
so [fe80::1%eth0]:9000 becomes a valid %25-escaped URL (tests added).
- Boolean flags: read --json/--quiet via Bool() so --json=false is
false, instead of IsSet() which treats any occurrence as true.
- Docker's HEALTHCHECK timeout raised to 10s: an outer deadline equal
to the probe's own 5s always SIGKILLed the probe before it could
print its diagnostic line.
- test-release path filter now also triggers on cmd/healthcheck-main.go
and cmd/main.go, so subcommand regressions run the image gate.
Not adopted: require_text's comment-insensitivity in verify-rebrand.sh
(snapshot-tripwire by design, consistent with its other assertions -
the semantic check lives in the CI gate now), and full
staging-then-promote tag publishing (a workflow-wide redesign shared
with the classic lanes, tracked as follow-up).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Publish pgsty/silo:<RELEASE>-distroless (and a rolling :distroless tag)
alongside the classic image: gcr.io/distroless/static-debian12 plus
exactly one program, the silo binary. No shell, no mc, no curl, no
entrypoint script - the binary is the ENTRYPOINT and the baked-in
exec-form HEALTHCHECK runs 'silo healthcheck ready'. The classic image
and its mc-based health checks are deliberately unchanged.
Design: silo.pgsty.com/compatibility/feature/healthcheck/
/data is created in the image layer, world-writable, because Docker
seeds fresh volumes from the layer mountpoint and no entrypoint exists
to repair ownership at runtime (issue #55); the parent directory is
copied from a throwaway busybox stage since COPY of a directory copies
contents rather than the entry itself, which would silently leave
/data at root:0755 and break every non-root run. The MINIO_USERNAME
drop-user path is not supported in this variant; use --user.
test-release.yml now builds the real Dockerfile.distroless on every
gate run (it has no download stages, so it stays offline) and asserts:
the HEALTHCHECK survives into the image config, /data ships 0777, no
shell and no /usr/bin/minio are present, Docker's health state turns
healthy from the baked probe alone, the probe binary execs without a
shell, and SIGTERM still stops the server gracefully - as root and as
--user 1001:1001. docker-release.yml gains the distroless build lanes,
multi-arch manifests, SBOM and provenance attestations, and a
release-blocking check that the pushed manifest still carries the
HEALTHCHECK (a Docker extension absent from the OCI image spec).
Verified locally on linux/arm64: full gate assertions plus bare
'docker run ... healthcheck' exit-code semantics and --version
passthrough.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add 'silo healthcheck [live|ready|cluster|cluster-read]', a thin
anonymous HTTP client for the server's own /minio/health/* endpoints,
so containers without a shell, curl, or mc can still run health
checks. Design: silo.pgsty.com/compatibility/feature/healthcheck/
The check vocabulary maps 1:1 onto the health API paths; the probe
target is derived from the server's own --address/MINIO_ADDRESS
contract with HTTPS auto-detected from the certs directory, and can
be overridden with --url. Exit codes are 0/1 only (Docker reserves 2);
diagnostics (x-minio-server-status, quorum headers) go into a single
output line for docker inspect. The request is strictly anonymous (a
credentialed request would be rejected by the reserved-path guard),
the transport bypasses HTTP_PROXY, and certificate verification is
skipped to match kubelet HTTPS probe behavior. Cluster checks default
to a 15s deadline so the server's 10s cluster_deadline can elapse.
Compatibility notes: the preserved /minio/health/* path literals and
the MINIO_ADDRESS env var are upstream wire/config surface, reused on
purpose; the rebrand-guard baseline is regenerated for the new route
literals (tests included) with zero new exported symbols. The docker
entrypoint argv translation learns the new command name.
Verified: unit tests, entrypoint tests, go vet, plus an end-to-end
run against a live server covering all four checks, --maintenance
(412), --json, usage errors, unreachable and timeout paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RPM, DEB and APK payloads carried no license material at all, and
the OCI image copied LICENSE and CREDITS but not NOTICE. For an AGPL
distribution that is not defensible packaging. Every delivery channel
now carries the same license set:
- nfpm packages install LICENSE and NOTICE under /usr/share/doc/silo/.
The entries are declared once per packager because nfpm only honors
type: license on rpm and silently drops such entries from deb and apk
-- the rpm keeps its %license flag, deb and apk carry plain files at
the same path. CREDITS stays out of the OS packages deliberately: at
1.8MB it would dominate an otherwise ~40MB payload.
- The OCI image gains /licenses/NOTICE next to the existing LICENSE and
CREDITS, in the Dockerfile and in both context-staging paths (the
make docker target and docker-release.yml).
- The GoReleaser archives list their files explicitly; the default
globs would keep missing NOTICE.
NOTICE's first line now reads 2015-2025 to match the banner constants
fixed in b57275be3 -- it had kept upstream's stale 2015-2023 claim --
and verify-rebrand.sh pins both NOTICE copyright lines.
The payload locks move in the same commit: sign-release-rpms.sh expects
the six-file listing, and the test-release validation asserts presence,
count, the rpm %license flag, and sha256 equality of the packaged
LICENSE and NOTICE against the repository files across all three
formats and both architectures.
Also dropped the unit's WorkingDirectory=/usr/local, a leftover from
upstream's /usr/local/bin install layout. Debian policy 9.1.2 keeps
packages out of /usr/local entirely; the binary already installs to
/usr/bin and the service does not depend on its working directory.
Verified locally with nfpm v2.47.0 (the CI-pinned version): built all
six packages via package-release.sh, then ran the test-release
validation step verbatim (extracted from the workflow YAML) and the
signing script's payload comparison against them in an Ubuntu 24.04
container -- both pass. Built the arm64 image from the real
Dockerfile.goreleaser and confirmed /licenses holds CREDITS, LICENSE
and NOTICE. make rebrand-guard passes with the new NOTICE pins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The shipped CREDITS predated the fork's dependency work: 38 of the
modules in the current go.mod, among them lestrrat-go/jwx, go-tpm,
go-spiffe and the charmbracelet family, had no entry at all, while
entries lingered for modules no longer in the build. An attribution
file that silently drifts from go.mod is worse than none, so it is now
generated, not curated.
buildscripts/gen-credits.sh (make credits) rebuilds the file from the
licenses of every module go list -deps reports for the main package --
the set actually linked into the silo binary, so test-only and tool
dependencies stay out. Entries keep the established name / URL /
license-text layout. New over the old file:
- Modules replaced in go.mod are annotated with the repository that
actually serves them, so the pgsty/silo-console, pgsty/mc and
pgsty/silo-pkg forks are named next to their upstream import paths.
- Bundled NOTICE files are reproduced after the license text, which
Apache License 2.0 section 4(d) requires when redistributing; 23
modules carry one.
- minio/colorjson, minio/csvparser and minio/filepath publish no
license file at all; they repackage Go standard library code and
their sources carry the Go Authors' BSD-style header, so the Go
project license is reproduced for them with a note saying why.
- The Go license text itself comes from the pinned golang.org/x/sys
module rather than GOROOT, because Homebrew's Go omits
GOROOT/LICENSE and the module copy is version-locked.
check-gen now runs the generator and fails on a CREDITS diff, the same
treatment go.mod and go.sum already get, so dependency changes cannot
leave stale attributions behind. Output is deterministic: two runs are
byte-identical, and LC_ALL=C sorting plus version-pinned inputs keep it
that way across machines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Pins pgsty/silo-console at 72fc0a5e, the v2.1.1 tag, replacing the
v2.1.0 commit from earlier today. The pseudo-version form stays because
the compatibility module path github.com/minio/console carries no /v2
suffix, so a v2.x semver tag is not directly selectable.
v2.1.1 is 5 commits over v2.1.0. The one substantive change for the
server is a fix that keeps unresolved i18n placeholders out of the
metrics dashboard legends; the rest is the v2.1.1 changelog, dropping
the container-image advertising from the Console docs, and regenerated
embedded web assets.
Verified: go mod tidy -compat=1.26 is a clean no-op afterwards, which is
what make check-gen asserts in CI, and go build ./... and go vet ./...
pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Brings the server in line with the contribution and copyright policy
already adopted in pgsty/mc.
Copyright terms were derived from the clock: startupBanner overwrote the
ldflags-injected CopyrightYear with time.Now().Year() and printed it as
the end of MinIO, Inc.'s term, so every January would have extended the
upstream copyright claim past the 2025 end of upstream development, and
release builds silently discarded the injected year. Both banners now
credit MinIO, Inc. for 2015-2025 and PGSTY from 2025 through the
release-stamped year, falling back to the current year in source builds.
NOTICE names PGSTY as the holder of the fork's modifications, matching
the banners and the packaging vendor field.
Contributions are accepted inbound=outbound under AGPL-3.0-or-later with
no CLA - the core is Copyright MinIO, Inc., so the combined work can
never be relicensed and a CLA would buy nothing. What a fork carrying a
downstream delta does need is provenance, so DCO 1.1 sign-off is now
mandatory and enforced by a workflow that rejects unsigned non-bot
commits. CONTRIBUTING documents sign-off, repair, cherry-pick provenance,
dual copyright headers, trailer preservation across squash merges, and
that assistive-tooling trailers carry no authorship or copyright claim.
verify-rebrand.sh pins the copyright split and the policy files so
neither can regress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
actions/checkout@v4 and actions/setup-go@v5 declare runs.using: node20, so
every Go CI job printed the runner's Node 20 deprecation warning. The
release, test-release and docker-release workflows carried six more node20
actions that never appeared in that warning only because those workflows
run on a tag or a dispatch rather than on every push: upload-artifact v4,
goreleaser-action v6, build-push v6 and the docker qemu/buildx/login v3
line. All of them move to the current majors, which declare node24. The
SHA-pinned rest already qualified -- attest and sbom-action are node24,
cosign-installer is composite.
Checked against what those majors removed. build-push v7 dropped
DOCKER_BUILD_NO_SUMMARY and DOCKER_BUILD_EXPORT_RETENTION_DAYS and
setup-buildx v4 dropped its deprecated inputs, none of which are used here.
checkout v7 refuses to check out fork PRs under pull_request_target and
workflow_run, and no workflow here triggers on either.
setup-go v6 exports GOTOOLCHAIN=local, the one change with teeth. go.mod
declares `go 1.26.5` and carries no toolchain directive, every job resolves
that same version through go-version-file, and nothing in the module graph
asks for more, so there is nothing left for the go command to fetch:
`go build ./...` passes under GOTOOLCHAIN=local. The tools installed from
outside the module are in range too -- nfpm v2.47.0 requires 1.26.4 and
govulncheck v1.6.0 requires 1.25.0 -- and golangci-lint arrives as a
prebuilt binary from upstream's install script, not through the toolchain.
The runner floor that comes with node24 (v2.327.1) is met by GitHub-hosted
runners, and every job here is ubuntu-latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pins pgsty/silo-console at 343d9442, the v2.1.0 tag, replacing the v2.0.0 commit
from 2026-08-04. The pseudo-version is still the form used here because the
compatibility module path github.com/minio/console carries no /v2 suffix, so a
v2.x semver tag is not directly selectable; the comment above the directive now
names the release the commit actually corresponds to.
v2.1.0 is 23 commits over v2.0.0. The substantive ones for the server are a
hand-rolled EN/ZH i18n layer with a language toggle covering login, navigation,
dashboard, buckets and object browser, identity and configuration, event
destinations, observability and admin tools; a dashboard migration to MinIO
Metrics V3; and removal of the remaining SUBNET references from health
reporting, which matches this repository's own decision to close the upstream
telemetry surfaces. The rest is UI polish and regenerated embedded web assets.
Verified: go mod tidy -compat=1.26 is a clean no-op afterwards, which is what
make check-gen asserts in CI, and go build ./... and go vet ./... pass.
Not included here: the GitHub Action version bumps and the .gitignore entry
sitting in the working tree are someone else's change and stay uncommitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit added a rename note to both READMEs and broke the gate I
added one commit earlier. That gate rejected the string pgsty/minio anywhere
outside three guard files, which was the wrong invariant: the point is that the
old name must never be a live target, not that it must never be spoken. A README
explaining where the MinIO-named artifacts went has to name them, and doing so
is the opposite of stranding a reader on a dead repository.
Split into two rules that say what is actually meant:
- No live URL may resolve to the old repository anywhere, READMEs included -
github.com/pgsty/minio and hub.docker.com/r/pgsty/minio are rejected outright.
This is the rule that protects users.
- The bare name is allowed only in five files: the pinned pre-rebrand image
digest in the upgrade test, the two guards that refuse a legacy image, and the
two READMEs.
Both halves are negative-tested: adding a github.com/pgsty/minio link to
README.md fails rule one, and adding a bare pgsty/minio to docs/docker/README.md
fails rule two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Someone arriving at main after the rename has one question the README did not
answer: where did the MinIO-named builds go. Both READMEs now say it directly —
the archived minio branch and the releases up to RELEASE.2026-08-04T00-00-00Z
hold the artifacts maintained under the original identity, and those assets and
the docker.io/pgsty/minio image stay published and unmodified.
The note also records the rename itself with its date, and repeats the one fact
that determines whether an operator has any work to do: only product and
artifact names changed. MINIO_* variables, minio_* metrics, x-minio-* headers,
/minio/* routes and the on-disk format are the same.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The transitional references land in one commit, because they are only correct
together: the repository is pgsty/silo, its default branch is main, and nothing
in the tree should point a user at the old names.
Changed:
- Workflow branch filters. go.yml and vulncheck.yml gated on `branches: master`
for both push and pull_request, so renaming the default branch would have
taken automatic CI offline with no error and no signal - the workflows would
simply never trigger again. They now name main.
- Release target. goreleaser's `release.github.name` becomes silo, which is
what actually decides where a tagged build publishes. sign-release-rpms.sh's
GH_REPO default follows.
- The OCI `image.source` label, the Helm chart `sources` entry, the security
advisory link in the issue-template config, and the go.mod comment citing the
LDAP TLS fix.
- 115 occurrences across README, README_ZH, SECURITY, CONTRIBUTING and 30 docs
pages, including 72 links that also carried the master branch in their path.
Those matter most: GitHub redirects clone, fetch, push and web URLs after a
rename, but raw.githubusercontent.com does not, and neither follows a branch
rename - every one of those links would 404 twice over.
- Three error strings in cmd/erasure-sets.go, cmd/storage-errors.go and
internal/config/errors.go that print an issue URL to operators. These are Go
string literals inside rebrand-guard's brand allowlist, so the baseline is
regenerated. The regeneration removes exactly those three entries and adds
none; all twelve other protected sets, including the 9014 exported symbols,
are byte-identical.
- The transitional-naming disclaimers in README, README_ZH, SECURITY and
CONTRIBUTING are dropped, since they no longer describe anything.
Deliberately unchanged, all three because they exist to reject or freeze the old
name rather than to point at it:
- buildscripts/minio-upgrade.sh pins pgsty/minio@sha256:b6bfe72... - the frozen
pre-rebrand image is the control group for the MinIO-to-Silo upgrade test.
- helm-migration-guard rejects any rendered container still pulling pgsty/minio.
- verify-rebrand.sh rejects the same in the delivery surfaces.
Also unchanged: docs/config/README.md links to pgsty/mc/blob/master, and that
repository's default branch really is still master. It moves when mc does.
verify-rebrand.sh gains three assertions so this cannot silently regress: no
source reference may name pgsty/minio outside the three allowlisted guards, no
link may target pgsty/silo's master branch, and go.yml and vulncheck.yml must
filter on main. Both new rejections were negative-tested - reintroducing a
master branch filter and adding a pgsty/minio URL each fail the gate with the
specific message.
This commit assumes the rename actually happens. Until the GitHub branch and
repository renames are executed, the links it introduces do not resolve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README, README_ZH, SECURITY, COMPLIANCE, CONTRIBUTING, NOTICE,
code_of_conduct, the vulnerability and PR-etiquette documents, the GitHub issue
and pull request templates, and the docs/ tree all present Silo as the product.
The Grafana dashboards under docs/metrics/prometheus/grafana/ have their panel
titles and descriptions rebranded while every minio_* query, label and
expression is left alone, so existing alerts and recording rules keep matching.
The distinction the review demanded is applied per hit rather than by
search-and-replace:
- Product and command text becomes Silo and silo: install and run instructions,
systemd examples, compose services, download links, badges.
- Protocol and interface text keeps MinIO: MINIO_* variables, minio_* metrics,
x-minio-* headers, /minio/* routes, .minio.sys, arn:minio, and API field and
error names.
- Attribution keeps MinIO and gains the fork's own: the AGPL obligations,
original copyright, CREDITS and NOTICE stay, with the modification notice
added alongside rather than replacing them.
- Historical and third-party references are left as facts, not rewritten for
brand tidiness.
README and README_ZH each carry an explicit non-affiliation notice, document
the side-by-side package migration including the
/etc/systemd/system/silo.service.d/10-legacy-user.conf drop-in for keeping a
legacy UID/GID, and state that recursive chown is never performed. The trademark
attribution uses the policy's approved "based on MinIO technology" wording, not
the shortened form the policy rejects.
github.com/pgsty/minio links are left in place and labelled transitional. The
repository has not been renamed, and rewriting them now would produce documented
URLs that 404 until the cutover; they change in the cutover commit together with
the goreleaser release target, the OCI source label and the raw-content branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test and verification scripts invoked ./minio and pulled their tooling from
upstream infrastructure with no integrity check. Every `curl | tar` of a client
or an old server binary was an unverified execution path in a script that
regularly runs as a privileged user, and several fetched a floating "latest".
Two installers replace all of it:
- install-mcli.sh resolves a pinned pgsty/mc release, downloads the archive and
its checksum manifest, requires exactly one valid manifest entry for the
asset, verifies it, and installs. MCLI_BIN with a mandatory MCLI_SHA256 lets
an offline or air-gapped run supply its own binary, still checksum-checked.
- install-verified-fixture.sh takes source, expected SHA-256 and target, and
refuses anything that does not match. Sources may be a URL or a local file.
Every script that previously downloaded mc now calls install-mcli.sh. The three
places that genuinely need an upstream artifact - the old MinIO server binary
for the LDAP IAM upgrade-import test, the 2021 mc for the three-site
replication test, and the functional-tests.sh fixture - go through
install-verified-fixture.sh with the digest recorded inline. Those dl.min.io
URLs remain on purpose: they are historical upstream artifacts needed to prove
upgrade compatibility, and they are now pinned and verified rather than
trusted.
The scripts otherwise switch to ./silo, silo.service, the silo container and
compose service names, and SILO_CONFIG_DIR. run-multi-site-minio-idp.sh is
renamed to run-multi-site-silo-idp.sh with the Makefile target following.
buildscripts/minio-upgrade.sh keeps its name and its `minio server` argv - it
exists to test the MinIO-to-Silo upgrade, so the old side must stay old - but
it is now pinned to an image digest rather than a tag, and its `docker system
prune` and `docker volume prune` calls are removed. Those ran unfiltered
against the developer's whole Docker installation; the resiliency tests had the
same problem and lose their prune and `docker ps -q` sweeps too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A rebrand of this size cannot be reviewed by reading the diff. The risk is not
what was renamed but what was renamed by accident: an environment variable, a
metric, a header, a route, a storage marker or an exported symbol that some
deployment depends on. So the invariants get a machine check.
rebrand-guard extracts the identifiers a product rename must not touch - the
go.mod module path, github.com/minio/* imports, MINIO_* variables, minio_*
metric literals, x-minio-* headers, route string literals, the hardcoded grid
routes in internal/grid/manager.go, the reserved bucket root, .minio.sys
markers, arn:minio and minio:s3 policy values, every exported symbol in cmd/
and internal/, and every branded string literal - and compares them to a
checked-in baseline. Any removal or addition fails the build, in both
directions, so a future change cannot quietly drop one either.
The recorded delta against the pre-rebrand tree is: module path, 137 imports,
19 metrics, 15 storage markers, 58 policy values and all 9014 exported symbols
byte-identical; environment 437 -> 436, headers 85 -> 84, routes 331 -> 330.
Each of those three is accounted for. MINIO_CONFIG_DIR and MINIO_VERSION were a
shell local and a compose variable in test scripts, never server settings.
x-minio-common was a docker-compose extension key, never an HTTP header.
/api/health/upload was the SUBNET upload path in the deleted callhome.go.
verify-rebrand.sh covers what the identifier guard cannot see: that the retired
upstream delivery paths stay deleted, that the packaging and container files
name silo, that silo.service keeps its Conflicts and both EnvironmentFile
lines, that silo.env contains no active assignment that would shadow the legacy
file, that no active delivery surface still publishes pgsty/minio, and that no
non-test Go source contains a dl.min.io, subnet.min.io, api.min.io,
slack.min.io or play.min.io URL.
Both, plus the entrypoint argv test, run in go.yml, release.yml and
test-release.yml. test-release.yml also gets the path filters right: it was
watching minio.service, which no longer exists, so the packaging gate would
have gone silently dormant. It now watches the unit, defaults, sysusers,
lifecycle scripts, helm chart and both guards, and asserts the four-file
payload, the absence of any cross-name Provides/Obsoletes/Conflicts, and that a
container started with the legacy `minio server` argv really runs silo as PID 1.
release.yml and docker-release.yml additionally verify that the checked-out
revision matches GITHUB_SHA before building. GitHub's OIDC certificate records
GITHUB_SHA, not the ref passed to actions/checkout, so a manual dispatch from
the wrong ref would have produced provenance describing different source from
the bytes being published. They now emit SBOMs and attestations for archives,
packages and images, and docker-release verifies the release workflow's
attestation on every artifact it consumes. Images publish to pgsty/silo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
helm/minio becomes helm/silo: chart name silo, version 6.0.0 -> 7.0.0, the
MinIO wordmark icon replaced with the project's own, image.repository and
mcImage.repository pointing at pgsty/silo, and the container command changed to
silo. User-visible titles, comments and documentation links are rebranded. The
MINIO_* environment variables and every existing values key are kept - the
first Silo chart is a rename, not a values-schema migration.
The hard problem is that a chart rename normally rewrites Kubernetes resource
identity, and a StatefulSet's selector and volumeClaimTemplate are immutable.
An existing release upgraded carelessly would either fail or orphan its PVCs.
Two things address that:
- Templates no longer derive the container name from .Chart.Name. It comes from
a helper, so nameOverride can pin it, which means an existing release can be
upgraded with nameOverride=minio, fullnameOverride=<existing-fullname> and
serviceAccount.name=minio-sa and render byte-stable identity while switching
chart and image.
- helm-migration-guard and verify-helm-migration.sh make that a gate rather
than a documented hope. The script lints the chart, renders it in distributed
and standalone modes plus the optional templates, then renders the legacy
chart from a pinned commit and the new chart with those three overrides and
compares resource identity. The guard additionally rejects any rendered
container still pulling pgsty/minio or invoking /usr/bin/minio. It runs
through a pinned alpine/helm image when helm is not installed locally, so the
gate does not depend on the developer's machine. Currently green over 7
compared resources.
Rollback is asymmetric and the README says so: the old chart with the new image
survives via the entrypoint argv shim, but the new chart with an old MinIO
image does not, because `silo server` is not a command that binary knows. Only
`helm rollback` is supported, never an image-only downgrade.
Not addressed here: the default image tag is pgsty/silo:RELEASE.2026-08-04T00-00-00Z,
which does not exist yet. The chart must not be published until the first Silo
image is pushed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dockerfile.goreleaser now copies silo to /usr/bin/silo, defaults to
CMD ["silo"], and labels the image as Silo. MINIO_UPDATE_MINISIGN_PUBKEY is
removed from the image environment: with the updater permanently disabled it
was dead weight, and leaving upstream's verification key in a Silo image
implied a trust relationship that does not exist. The MINIO_* runtime
environment variables, ports, volume and health endpoints are unchanged.
The image keeps shipping mcli with an /usr/bin/mc symlink. That is the client,
not a MinIO-branded alias for the server binary, and the Helm post-install job
and existing container scripts call it by name.
docker-entrypoint.sh translates a legacy first argument: `minio server /data`
becomes `silo server /data`, so an existing `command: minio server ...` in
compose or a Pod spec keeps working across the image swap. The translation is
argv-level only - no file named minio is installed, and an explicitly
overridden `entrypoint: /usr/bin/minio` still fails, which is the honest
outcome since that path genuinely no longer exists.
The entrypoint also fixes an unrelated startup hazard it was already carrying:
when the image runs under an arbitrary UID, HOME points at an unreadable /root
and the server probes its default config directory during initialization. It
now falls back to /tmp when HOME is unset, /root, missing or unwritable.
docker-entrypoint_test.sh pins all of it - empty argv, legacy minio, native
silo, bare flags, and an explicit shell command that must stay explicit. It is
wired into make rebrand-guard and into the go.yml, release.yml and
test-release.yml gates, so removing the shim breaks CI rather than breaking
users. Passing locally.
download-static-curl.sh gains checksum verification for the curl it fetches
into the build stage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything a user installs is renamed, and the package finally installs enough
to be startable on a clean host.
Artifact names
- goreleaser.yml: build id, binary, archive and checksum manifest become silo_*.
release.github.name stays "minio" with a comment - the GitHub repository has
not been renamed yet, and pointing at pgsty/silo before the rename would 404.
Also adds per-archive SPDX SBOMs and a keyless cosign signature over the
checksum manifest, so the signed manifest covers archives and SBOMs together.
- nfpm.yml: package name silo, and the binary moves from /usr/local/bin/minio
to /usr/bin/silo. /usr/local is not on the default PATH of a systemd unit and
is not FHS-correct for a distribution package.
- package-release.sh, sign-release-rpms.sh and verify-build-provenance.sh follow
the new names; the RPM signing script asserts NAME=silo and the new four-file
payload. nfpm is now invoked from the repository root so relative script paths
in the config resolve regardless of the caller's directory.
Package relationships are deliberately empty
No Provides, Obsoletes, Replaces or package-level Conflicts. Obsoletes: minio
cannot distinguish a pgsty package from upstream's own identically named one,
so an unattended dnf upgrade could silently swap a different vendor's product
for this one. With no relationships, both packages coexist, their file sets do
not overlap, and migration and rollback are single explicit commands. The
mutual exclusion lives in the unit instead: silo.service carries
Conflicts=minio.service plus After=minio.service.
Payload, from two files to four
- /usr/bin/silo
- /usr/lib/systemd/system/silo.service
- /etc/default/silo, installed config|noreplace
- /usr/lib/sysusers.d/silo.conf
The old package shipped a unit referencing an account nothing created, so a
clean install could not start. postinstall.sh now creates the silo system
account through systemd-sysusers, useradd or BusyBox adduser in that order and
runs daemon-reload. It never stops a service, never chowns data and never
touches /etc/default/minio. preremove.sh disables silo.service only on a real
removal - Debian "remove", RPM 0, Alpine's dotted version - so upgrades leave
the running service alone. lifecycle_test.sh exercises both against a stubbed
PATH, so a green run cannot create an account or touch the host.
silo.service reads /etc/default/minio then /etc/default/silo, in that order, so
an existing node's MINIO_* values keep working and the new file overrides them.
The packaged silo.env therefore ships comments only: any active assignment
would shadow the legacy file with an empty value.
Makefile: build/install/install-race produce ./silo, and the docker target now
assembles a context from a locally built linux binary plus Dockerfile.goreleaser
instead of the deleted Dockerfile. The hotfix, hotfix-push, docker-hotfix and
docker-hotfix-push targets are gone - they downloaded upstream's pkger, signed
with upstream's minisign key and scp'd to dl-N.minio.io. verifiers now depends
on a new rebrand-guard target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two coupled changes that must land together, because the same files carry both:
the server now identifies itself as Silo, and every path that would have called
home to a MinIO-operated service is closed.
Product identity
- build-constants.go: store name, UA name and startup banner become Silo. The
Go identifiers (MinioStoreName, MinioBannerName, ...) keep their names on
purpose - renaming exported symbols would churn the compatibility surface for
a cosmetic gain, and the rebrand guard freezes that surface.
- main.go, server-startup-msg.go, ftp-server.go and the user-visible log, help
and error strings across cmd/ and internal/ switch to Silo. Original MinIO
copyright, LICENSE, NOTICE and CREDITS are untouched; --version now prints
the upstream copyright, the pgsty modification notice, and the trademark
policy's approved "based on MinIO technology" attribution.
- api-headers.go: the HTTP Server header becomes "Silo". This is the one
externally observable identity change, so TestCommonHeadersUseSiloProductName
pins it - probes that sniff for "MinIO" must move to capability detection.
- Prometheus metric HELP strings keep their MinIO wording. They are part of the
metrics contract the guard protects, not product copy.
Configuration directory
- config-dir.go: new installs use ~/.silo. If only ~/.minio exists it is still
read, with a one-time notice and no files moved. If both exist ~/.silo wins
and an ambiguity warning is emitted; an explicit --config-dir always wins.
Covered by TestSelectDefaultConfigDir. The internal .minio.sys layout is
never renamed - this rule applies to the user config directory only.
Upstream service lockdown
- globalInplaceUpdateDisabled is now true at initialization rather than being
set from MINIO_UPDATE. common-main.go still parses MINIO_UPDATE so upgrading
nodes do not fail on an unknown key, but warns that the value is ignored;
there is no way to re-enable the updater. TestInplaceUpdateCannotBeEnabled
guards that. Without this, an admin with mc could have overwritten
/usr/bin/silo with an upstream MinIO binary.
- verifyBinary and commitBinary refuse early; the ServerUpdate v1/v2 admin
routes and the peer-rest update endpoints stay registered and keep returning
the existing programmatic error, so clients see a stable failure rather than
a 404.
- MinioReleaseBaseURL and defaultMinisignPubkey are emptied: no dl.min.io
download root, and upstream's minisign key is no longer a trust root for
anything this fork ships.
- cmd/callhome.go is deleted and internal/config/subnet/ is reduced to parsing
its old keys and reporting that the integration is disabled. config-current.go
warns instead of failing when callhome or SUBNET settings are present, so an
upgraded node with those keys still starts.
- internal/config/errors.go replaces the MinIO Slack and support entry points
with Silo documentation and issue links. Error codes and programmatic fields
are unchanged.
Verified: the compatibility baseline is unchanged except for the deliberate
removal of the /api/health/upload SUBNET route; go build, go vet and the full
cmd/ and internal/ unit suites pass; a locally built binary starts, serves
S3/Admin/metrics on the unchanged /minio/* routes, answers with Server: Silo,
and falls back to a pre-existing ~/.minio with the expected notice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tree still carried a complete second publishing pipeline inherited from
upstream that pgsty has never operated, plus MinIO-branded artwork the fork has
no right to redistribute. None of it is a pgsty release artifact, so the
history-is-immutable rule does not protect it: nothing here was ever published
under this repository's name, and git history plus the pre-cutover tag keep the
provenance.
Removed, by category:
- charts.min.io machinery: CNAME (charts.min.io), _config.yml, index.yaml,
helm-reindex.sh and 90 helm-releases/minio-*.tgz. GitHub Pages is not enabled
on this repository, so this was dormant - but enabling Pages for any reason
would have stood up a convincing impostor of the official MinIO chart
repository from our own domain.
- Dockerfiles that consume upstream binaries: Dockerfile, Dockerfile.cicd,
Dockerfile.hotfix, Dockerfile.release, Dockerfile.release.old_cpu,
Dockerfile.scratch and docker-buildx.sh. These fetch or expect a MinIO-built
binary; a Silo image must only ever contain an artifact this repository built
from this commit. Dockerfile.goreleaser, the one the release workflow
actually uses, stays.
- buildscripts/upgrade-tests/ (compose.yml, minio.env, nginx.conf), which
pinned minio/minio images through a MINIO_VERSION compose variable. The
upgrade test is rebuilt on image digests in a later commit.
- docs/hotfixes.md, describing an upstream hotfix distribution channel that
does not exist for this fork.
- MinIO trademark artwork: .github/logo.svg (a recoloured MinIO wordmark), the
docs/screenshots/ set, the Grafana dashboard screenshots and
docs/federation/lookup/bucket-lookup.png. The dashboards themselves are kept
and rebranded separately; only the images carrying MinIO chrome go.
Deliberately not touched: published releases, tags, image digests and package
signatures remain byte-identical. This commit only removes paths that were
never part of a pgsty release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release image (Dockerfile.goreleaser) was only ever built by
docker-release.yml, which is workflow_dispatch only - so the runtime layer and
entrypoint were never exercised by CI until an actual publish, where a broken
COPY path or a signal-handling regression would surface at the worst possible
moment. MC already covers this in its test-release; the server did not.
Add an offline, deterministic smoke: assemble a minimal image from the
linux/amd64 binary goreleaser already built and the real entrypoint, then start
the server and `docker stop` it on both the default and the MINIO_USERNAME
drop-privilege paths, asserting PID 1 is minio, the exit is 0, the shutdown is
sub-timeout, and the "Exiting on signal" log is present. The mcli-download build
stage is skipped deliberately - it hits the GitHub API and would make this gate
flaky and non-reproducible; its simple, checksum-guarded logic still runs at
publish time.
This is the regression guard for the exec-into-chroot entrypoint fix. Also add
dockerscripts/docker-entrypoint.sh to the pull_request paths so an entrypoint
change actually triggers this pipeline. Verified locally end to end: the fixed
entrypoint passes both paths; reverting the exec makes the drop-privilege path
time out to exit 137 and the step fails.
Co-authored-by: Claude <noreply@anthropic.com>
Three gate-correctness fixes:
- lint: `command typos && typos ./ || echo skipping` ran typos twice (POSIX
`command` executes it) and, via `&& ... || echo`, turned a real typo finding
(nonzero exit) into the "not installed" message with exit 0 - so spelling
issues could never fail the gate. Use `command -v` to test presence and run
typos once, letting its findings surface.
- getdeps: `curl -sSfL ... | sh` took the pipeline's exit from sh, and make's
/bin/sh has no pipefail, so a failed or partial download of the golangci-lint
installer was reported as success and a truncated script could run. Download
to a temp file under `set -e` (with a trap to clean it up) and execute that,
so a curl failure aborts the target. Verified: a 404 now exits nonzero
instead of silently continuing.
- release: workflow_dispatch checked out github.ref (the branch it ran from)
while naming artifacts after the input tag, so a release could be built from
one ref and published under another. Pin checkout to the requested tag.
Not addressed here: the server's release container image is still only built
by the publish-time docker-release workflow, never in CI (MC covers this in
its test-release). Flagged for a separate decision.
Co-authored-by: Claude <noreply@anthropic.com>
nFPM placed minio.service at /lib/systemd/system. On merged-usr EL/Fedora the
file lands correctly through the symlink, but the RPM database records /lib/...,
so `rpm -qf` disowns it and - more importantly - systemd's
%transfiletriggerin watches /usr/lib/systemd/system, so the path recorded as
/lib/... never fires the automatic `daemon-reload` on install or removal. Move
the unit to the canonical /usr/lib/systemd/system, which both restores that
trigger and makes the package own the path it ships.
The destination is asserted verbatim in several places, all updated in lockstep
so the packaging gate still passes: the six name/payload/sha checks in
test-release.yml (rpm/deb/apk) and expected_payload in sign-release-rpms.sh.
Verified by building a real deb with the new config: the unit is at
./usr/lib/systemd/system/minio.service.
Also drop the `# Built for ${project.name}...` line from minio.service. nFPM
expands variables in content src paths, not file bodies, so that pkger-era
placeholder was being written verbatim into every installed unit.
Not addressed here, deliberately: the package still does not create the
minio-user account the unit references. That matches upstream MinIO's own
packages (both rely on the documented manual useradd) and is not a fork
regression, so it stays a documentation step rather than a scriptlet.
Co-authored-by: Claude <noreply@anthropic.com>
A gitignored .goreleaser.yml sat in the repo root, so a bare `goreleaser`
invocation - the default discovery path - picked it up instead of the
.github/goreleaser.yml that CI passes with --config. The two had diverged
completely: the shadow published the GitHub release directly (draft: false),
pushed pgsty/minio:latest from a dockers section, and stamped a different
vendor and artifact naming. Since RPM signing runs on the maintainer's own
machine, that shadow config was one stray `goreleaser release` away from
publishing a mislabelled release and overwriting the Docker latest tag.
Delete the file and drop its .gitignore entry so any future .goreleaser.yml
shows up as untracked and is caught by the release workflow's clean-checkout
check, rather than silently steering local builds. CI is unaffected: it names
its config explicitly. (A copy of the removed file is preserved out of tree
for reference.)
Co-authored-by: Claude <noreply@anthropic.com>
gen-ldflags injected -X cmd.GOPATH / cmd.GOROOT from the builder's environment,
baking absolute paths like /Users/<user>/go into every released binary. That
defeats -trimpath and makes the build unreproducible: a third party rebuilding
the same tag gets different bytes and cannot verify checksums.txt.
The values only seed logger.Init's source-path trim list, and under -trimpath
the binary's paths are already relative, so there is no build-machine prefix
left to trim - the trim list also still gets runtime.GOROOT() and
build.Default.GOPATH at run time. Dropping the two stamps changes no observable
logging behaviour; cmd.GOPATH/GOROOT keep the empty defaults a plain go build
leaves.
Verified: gen-ldflags output no longer contains cmd.GOPATH/GOROOT; a
-trimpath release build has zero occurrences of the builder path (was 1);
Version, ReleaseTag and CommitID stamps are intact.
Co-authored-by: Claude <noreply@anthropic.com>
The compute step spliced ${{ github.event.inputs.tag }} straight into the
run: body, so a dispatch tag containing shell metacharacters was parsed as
script - arbitrary code in a job holding a contents:write token. Pass the
input through the environment instead (INPUT_TAG), the same shape
docker-release.yml already uses, so the value reaches bash as data.
Second vector: the old format check compared the sed output against the input
to decide validity, and sed anchors ^...$ per line. A tag carrying a newline
passed the check on its first line and the remaining lines flowed into
$GITHUB_ENV, setting arbitrary variables (PATH, LD_PRELOAD, ...) for every
later step. Replace it with a bash =~ whitelist anchored to the whole string,
which rejects any multi-line value up front.
Verified: a legal RELEASE.* tag is accepted and yields the right PKG_VERSION;
a `"; touch ...; #` injection and a newline-carrying PATH/LD_PRELOAD payload
are both rejected; an empty input falls back to the git ref name.
Co-authored-by: Claude <noreply@anthropic.com>
The crosscompile gate built 15 OS/arch combinations, ten of which we never
ship (ppc64le, mips64, s390x, mips, riscv64, 386, arm, freebsd, netbsd,
openbsd). On a cold CI cache each target costs ~80s, so the full set ran ~21
min and tripped the job's 20-min timeout, cancelling the only CI run for the
release HEAD. Trim the list to the exact goos/goarch matrix the release
actually produces (see .github/goreleaser.yml): linux, darwin, windows on
amd64 and arm64.
This also closes a coverage gap: windows/arm64 is published but was not being
compile-checked. The trimmed set builds all six in well under the timeout.
Note: netbsd is no longer compile-checked here, so the go-systemd v22.6.0 pin
(kept because v22.7.0 does not build on netbsd) loses its CI guard. The pin is
retained deliberately - it is harmless on the platforms we ship and upgrading
go-systemd is a separate decision - but a netbsd regression in a dependency
would now surface only if that build is exercised out of band.
Co-authored-by: Claude <noreply@anthropic.com>
The two chroot branches that drop privileges when MINIO_USERNAME/GROUPNAME
(and optionally MINIO_UID/GID) are set ran chroot as a child of the entry
shell, leaving the shell as PID 1. A SIGTERM from `docker stop` or an
orchestrator then went to the shell, which does not forward it, so MinIO was
never asked to shut down and was killed after the stop timeout (exit 137) with
no "Exiting on signal" log - risking in-flight requests and data at the flush
boundary. The default branch already exec's; these two now do too, so MinIO
runs as PID 1 and receives the signal directly.
Verified in a faithful reproduction of the release runtime layer: all three
paths (default, USERNAME only, USERNAME+UID/GID) now stop in ~0.2s with exit 0
and log the graceful shutdown, where the two drop-privilege paths previously
timed out to exit 137.
Co-authored-by: Claude <noreply@anthropic.com>
Make the repository's actual quality contract visible in CI. Check formatting, build and vet the tree, run cmd and internal tests, enforce pinned lint and generated-file cleanliness, exercise S3 Select under the race detector, and cross-compile every supported target.
Keep each concern in a separate job so failures identify the missing guarantee instead of hiding behind a single aggregate test result.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
SelectResults spawns a parser goroutine that drains and closes the
response body when the stream ends; deferring res.Close() had the test
drain and close the same bytes.Reader concurrently, which the race
detector catches reliably. Give the test body a close signal and wait
for the parser to finish instead of competing with it.
The double-close lives in minio-go's client parser, which no server
code path uses; it remains worth an upstream report.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Expand the generated-file gate beyond *_gen.go and go.sum to cover generated tests, msgp output, stringer files, go.mod, and untracked generated artifacts.
Regenerate the 19 stale stringer outputs with the go.mod-pinned x/tools version. Runtime String output is unchanged, and a repeated generation pass now leaves the tree byte-identical.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Installing golangci-lint from the moving master branch made local and CI results depend on the day they ran. Pin v2.11.3 and install it into a versioned path so version changes cannot accidentally reuse a stale binary.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
The release workflow built and pushed the container images in the same
run that created the draft, so pgsty/minio:latest moved to a build
nobody had signed or published yet. Abandoning that draft left latest
pointing at it with no way to notice, and the images were the one
artifact of a release that escaped the draft gate entirely.
Image publishing moves to its own dispatch-triggered workflow that
takes a tag and refuses to touch anything that is not a published,
non-prerelease release, and that is not the latest one - since it moves
the latest image tag, running it for an older release would silently
roll users back. It builds from the archives attached to that release,
checked against the published checksums, rather than rebuilding from
source: the image then contains the same binary the tarball does, by
construction rather than by assumption.
GoReleaser loses its dockers and docker_manifests sections along with
the QEMU, buildx and registry-login steps that only existed to serve
them, and the release job drops packages: write, which was granted for
a GHCR push that never happened.
Releasing now has one more manual step. That is the point: tagging no
longer moves docker latest, so the tag can be cut, inspected, signed
and published before anything reaches users who pull by tag.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Refuse dirty release checkouts and verify that every GoReleaser binary records the tagged revision with vcs.modified=false before packaging or publication.
Exercise the complete nFPM output in the release test pipeline: assert the six public names, validate every checksum and package identity field, and prove that RPM, DEB, and APK payloads contain the exact source binary and systemd unit. Validate release scripts and make their identity expectations the single source of truth.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace minio/pkger's hard-coded upstream identity with an in-tree nFPM configuration. Packages now name PGSTY as vendor and maintainer, use the SILO homepage and SPDX license, and preserve the established package names, versions, payload paths, modes, and checksum format.
Resolve both the release binary and systemd unit independently of the caller's working directory. The release and test workflows share the same package script, while the signing script consumes the same final metadata contract.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
GitHub Actions cannot hold the Pigsty RPM signing key, so the release workflow must stop before publication. Create releases as append-only drafts, refuse to overwrite existing assets, and make publication an explicit action after local signing and review.
Add a maintainer-side signing command that downloads the two RPMs, verifies checksums and package identity, signs them with the Pigsty key, regenerates checksums, and only replaces draft assets when --upload is explicitly requested. Published releases are never modified.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Consolidate the release-cycle security ledger after the implementation commits are stable. Document internode containment, policy-condition source hardening, the opt-in trusted-proxy boundary, and the bucket/object authorization tightening with fork-local identifiers where no CVE exists.
Use silo-pkg v3.11.0 as the maintained dependency reference, explain compatibility and migration behavior, and link each entry to the rewritten commit that actually carries the fix.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Keep unpublished security working notes, local agent instructions, and maintainer scratch material out of the public source tree. Already tracked security documentation remains tracked; the ignore rules only prevent accidental additions.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Point both the server and bundled client jobs at the release tag being cut, keep chart appVersion aligned, and retain the maintained pgsty/minio image selected by the public chart baseline.
Stop creating the console/console123 administrator by default. Leave an explicit change-me example while rendering no user-creation job unless an operator opts in. Chart packaging and repository indexing remain separate release steps.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
ReadParts built its disk-health trace path from partMetaPaths[0] with
no length check, so a caller passing no paths at all indexed an empty
slice. xlStorage.ReadParts handles an empty list perfectly well - it
returns an empty result - so the panic came entirely from the metrics
bookkeeping wrapped around it.
The reachable caller is ReadPartsHandler, which decodes its path list
from a msgpack request body that an authenticated peer controls, and
neither the handler, the storage-REST client, nor the path guard
rejects an empty one: guardPaths ranges over the slice, so an empty
slice passes vacuously.
net/http recovers a panicking handler, so this is not a crash - which
is what makes it worth fixing rather than merely tidy. ReadPartsHandler
calls keepHTTPResponseAlive before it calls ReadParts, and that helper
spawns a goroutine whose only exit is receiving from the channel done()
writes. Panicking in between skips both done(err) and done(nil), so the
process survives and the keep-alive goroutine and its ten-second ticker
stay parked forever - one per request, driven by a request body the
caller chooses. Repeating one malformed frame exhausts the node.
The fix follows DeleteVersions in the same decorator, which already
guards the identical "merely for tracing" lookup; ReadParts was the one
method missing the pattern. It also covers the second entry point, the
per-disk errgroup in readParts, where a panic has no recover at all and
would take the process down. That path is screened at the S3 boundary
today, so this is defence in depth there.
No error is returned for the empty case. The storage layer's answer to
an empty list is an empty result, and turning that into an error would
be a behaviour change on a path that is merely degenerate.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
joinErrs ranged over its own empty accumulator string instead of the
errs slice, so the loop body never executed and the function
unconditionally returned "". Its only caller feeds the merrs tag of the
DeleteDanglingObject audit event, so every dangling deletion was
recorded without the per-drive metadata errors: the record showed what
was deleted but not which drives errored or why quorum was lost.
Range over errs instead. The existing separator logic is already right
once the loop runs, since a leading nil error appends "<nil>" and every
later element gets its comma. Upstream's open minio/minio#21580 fixes
the same bug with a strings.Builder rewrite, not taken here: the
function runs once per dangling deletion over a drive-count-sized
slice, and the one-word change is the entire defect.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
The discrete PostgreSQL connection path concatenated raw values into a libpq keyword/value string. Whitespace, quotes, or backslashes could split a value into additional parameters or make an otherwise valid configuration fail to parse; the path also used the unsupported keyword username instead of user.
Render every generated value as a single-quoted libpq parameter, escape quotes and backslashes, and use the correct user key. Keep the existing connection_string form untouched. The earlier attempt to register migrated PostgreSQL and MySQL fields is deliberately absent because those key names collide with the legacy connection-string tokenizer.
Focused tests cover ordinary values, whitespace, quotes, backslashes, and parameter-shaped input.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
GetNotifyNATS reads user_credentials, nkey_seed and tls_handshake_first
and GetNotifyAMQP reads immediate, but none of them were registered in
DefaultNATSKVS/DefaultAMQPKVS or the help schema, so CheckValidKeys
rejected any enable=on target carrying them. Worse, the legacy config
migration wrote exactly these keys - including the env var name
MINIO_NOTIFY_NATS_USER_CREDENTIALS used as a config key, because the
NATSUserCredentials constant doubled as both - so a migrated NATS config
failed validation on every load, and the FetchEnabledTargets fail-fast
then silently disabled all bucket notification targets.
- Register user_credentials/nkey_seed/tls_handshake_first (NATS) and
immediate (AMQP) in the default KVS and help schema; split
NATSUserCredentials into a real config key plus EnvNATSUserCredentials
(all env var names byte-stable)
- Fix legacy migration: SetNotifyNATS writes the proper key;
SetNotifyAMQP no longer writes cfg.Immediate under the internal key
and now carries both immediate and internal
- Tolerate the legacy MINIO_NOTIFY_NATS_USER_CREDENTIALS key written by
pre-fix migrations (NATS-scoped, load path only) with fallback read;
env > user_credentials > legacy key
- Print key names only, never values, in the invalid-keys error of both
CheckValidKeys forms; rejected values can carry credentials
- Add an AST-based audit test asserting parser reads, migration writes
and help entries stay within the registered key set for all ten notify
subsystems, with floor assertions so collector drift fails loudly
- Document (unchanged) FetchEnabledTargets fail-fast and pin it with a
characterization test
Known same-class gap left in place and pinned by the audit's allowlist:
SetNotifyPostgres/SetNotifyMySQL write five unregistered DSN-era keys;
tracked for a follow-up issue.
Closes#39
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
The address MinIO attributes a request to is read from X-Forwarded-For,
X-Real-IP or RFC 7239 Forwarded, and never from the connection unless all
three are absent. It becomes aws:SourceIp and the audit remotehost field,
so any client that can reach the API port chooses the value an IpAddress
condition is evaluated against and the address every logged action is
attributed to.
MINIO_API_TRUSTED_PROXIES now selects who may make that claim:
unset the historical behaviour, unchanged
none no forwarded header is believed; the TCP peer wins
<CIDRs> believed only from listed peers, chains read right-to-left
Reading right-to-left is what makes an appending proxy safe: each hop
appends the peer it actually saw, so an entry a client injected can only
sit to the left of one a proxy wrote. The stock nginx recipe
$proxy_add_x_forwarded_for appends, which leaves the client's entry
left-most - exactly where the untrusted path reads - so a deployment with
no direct route to the API port was forgeable too.
_MINIO_API_XFF_HEADER is deliberately untouched, in semantics and in read
timing. Widening it to mean "trust nothing" was implemented and reverted:
it is the only part of this change that could alter a deployed
configuration, and the new variable expresses the same guarantee at no
compatibility cost. Upstream's TestXFFDisabled is retained verbatim.
Notes on the allow-list mode, all covered by tests:
- it must name proxies, not the subnet they sit in; listed entries are
skipped while walking, so a range covering clients lets them forge
- a cluster must list its own nodes, because MinIO forwards between
them and a client can force a hop via the ListObjectsV2 token
- loopback is trusted as a peer, not as a chain entry, so FTP and SFTP
keep attributing their sessions
- the node-to-node forwarder drops X-Real-IP and Forwarded from a peer
not entitled to have set them
- the walk scans the header in place and stops after 100 hops, so a
long chain costs neither allocation nor unbounded work
No behaviour change for any deployment that does not set the new
variable: the untrusted path is a verbatim copy of the previous function
body, differentially verified against it over ~5.1M header combinations.
The LDAP STS allow-list now shares the list parser as pure code motion,
verified identical across every combination of 37 allow-list values and
21 peer addresses.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
A bucket policy that allows s3:DeleteObject only when s3:versionid is null
-- Condition {"Null": {"s3:versionid": "true"}}, the idiom for "let clients
delete current objects but not roll back versions" -- denied every delete,
including the version-less ones it was meant to permit (upstream issue
minio/minio#21735).
getConditionValues wrote "versionid": {""} unconditionally. The condition
engine decides Null by slice length (nullfunc.evaluate), so a present-but-
empty value reads as "key present": Null:true never matched and Null:false
always did. Absent and empty were indistinguishable.
Writing the key only when the request names a version fixes the reported
case but, alone, opens a worse one. DeleteObjects carries each object's
version in the XML body, which getConditionValues -- reading only r.Form --
never sees. A body version would then vanish from the map, read as null,
and a policy meant to protect old versions would authorize deleting a
specific one. So authorization also rebinds versionid to the effective,
server-resolved reqInfo.VersionID for DeleteObjectAction: the per-entry
body value that checkRequestAuthTypeWithVID already sets in the
DeleteObjects loop, deleting the key when that value is empty. A
query-level ?versionId on a DeleteObjects POST no longer leaks into any
entry's decision.
Finally, trim the version the condition builder reads. newContext and
getOpts both TrimSpace it before the object layer acts, so an untrimmed
value here let a padded ?versionId=V%20 present a different s3:versionid
than the version actually operated on, sidestepping a Deny keyed on
StringEquals s3:versionid. DeleteObjectAction was already immune via the
trimmed reqInfo value; this covers GetObject, tagging, retention, and the
copy-source read.
Tests: an end-to-end DeleteObjects against a Null:{s3:versionid:true}
policy over versioned objects (with a decoy query versionId proving the
per-entry body value wins), and a unit test asserting key presence,
trimming, and the copy-source fallback.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Policy evaluation mixed server-derived identity and transport values with raw headers and query parameters. A client could therefore shadow internal condition keys, synthesize LDAP or JWT resource variables, substitute request tags for stored tags, or make a condition observe a value different from the one the handler actually used.
Partition condition sources, reserve internal names, adopt exact-name lookup from silo-pkg, and bind authorization to the effective request state. Preserve compatible query forms for storage class and upload tags with explicit header precedence, while restricting signature age and existing-object tags to authenticated or server-resolved values.
Tests sweep every supported key across header and query routes and exercise LDAP/OIDC variables, object-lock spelling, STS tags, metadata extraction, and end-to-end policy decisions.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Upgrade directly to silo-pkg v3.11.0, the fork's first non-colliding release, and carry the completed minio/minio#20449 hardening without exposing the retired v3.7.0, v3.8.0, or v3.8.1 tags in the rewritten history.
Twelve sensitive bucket-level writes now require the bare bucket resource in addition to the historical bucket/ form, so an object-only bucket/* grant cannot delete a bucket or change protections. Read/list behavior, ordinary tenant self-service, Deny statements, and NotResource exclusions retain their compatible behavior; MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH restores the old matcher when migration requires it.
End-to-end tests cover direct clients, session policies, service accounts, wildcard edge cases, compatible resource pairs, and real bucket deletion.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
sort.SliceIsSorted with a strict '<' predicate treats equal neighbours
as sorted, so a completion list like [1,1] passed the order check and
the same part was assembled into the object twice: a single uploaded
5 MiB part produced a 10 MiB object. Replace the check with an explicit
strictly-increasing scan that rejects repeats with InvalidPartOrder
before anything is assembled. Gaps and lists not starting at part 1
remain legal, matching AWS semantics.
The regression test drives the real CompleteMultipartUpload handler on
both Erasure backends and asserts that rejected completions leave no
object behind and keep the upload retryable.
Closes#49
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
httpStreamResponse, streamHTTPResponse, waitForHTTPStream, and their
8k buffer pool lost their last caller when ReadMultipleHandler went
away (73ac52472). Nothing references them anywhere in the tree, and
the storage REST wire surface is untouched - these encoded a framing
no registered handler still speaks.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
ReadPartsHandler completed its keepalive stream before reporting storage failures, then tried to write an ordinary error response after the body was already owned. Clients consequently decoded the error text as msgpack and lost the real failure.
Send failures through the keepalive completion channel and mark success only after ReadParts returns cleanly, preserving the existing wire framing and error identity.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
AppendFile, DeleteVersions, and ReadFile sized memory directly from peer-controlled Content-Length or query parameters. Tiny requests could therefore reserve gigabytes before delivering a body, while negative declarations could reach make and panic.
Cap preallocation without capping accepted append bodies, grow version slices as entries decode, reject negative counts, and enforce the format-implied 5 GiB ceiling on legacy whole-file reads. Regression tests drive raw handler inputs, measure total allocations, and retain legitimate round trips.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Malformed erasure layouts can divide by zero, while negative part sizes collapse expected shard sizes to zero and make truncated data appear healthy. Boundary validation alone is insufficient because poisoned metadata may already exist on disk or arrive through local heal paths.
Reject non-positive block sizes at the sole Erasure constructor, guard the metadata arithmetic helpers and rebalance calculation, refuse negative part sizes before persistence, and make CheckParts and VerifyFile reject previously stored poison. Tests cover both shard-size implementations, construction, persistence, local verification, and the wire boundary.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Storage REST request bodies and Grid RPC frames bypass the HTTP validity middleware, allowing wire-supplied paths and malformed FileInfo values to reach xlStorage unchecked.
Wrap the remotely exposed StorageAPI with a guard that covers every path-bearing method, including nested metadata fields. Reject traversal and destructive volume-root aliases before path cleaning can erase them, and validate erasure geometry and part sizes at the same wire boundary. Keep a raw-volume check in getVolDir for peer-S3 calls that bypass the wrapper.
Reflection, fuzz, traversal, peer-S3, compatibility, and malformed-erasure tests pin the complete method surface and prove that legal object names remain accepted.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
WithDeadline runs its work function on a goroutine of its own, so a panic
inside it is reachable by no recover() the caller can install: net/http
and internal/grid each recover only on the goroutine they own. Left
unhandled it terminates the process, which turns any malformed internode
payload that trips a bug in work() into a remote node kill.
Eleven call sites run wire-derived storage work through this path. The
project already recovers panics at both request boundaries; this extends
the same policy to the one goroutine those recovers cannot reach.
Stack dumps are capped over the process lifetime. The panic is by
definition reachable from untrusted input, so writing a full stack per
occurrence would trade a node kill for unbounded log amplification, one
small request each. The error is returned every time regardless, so the
caller's own rate-limited logging still sees each occurrence.
Inherited from upstream; the fork added no lines to this path.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Mark trackingResponseWriter committed when Write or an effective Flush implicitly sends a 200 response. This keeps duplicate-response detection aligned with the actual writer chain while preserving no-op Flush behavior when unsupported.
Add direct and gzip-streaming regression coverage for implicit headers, Flush delegation, and suppression of a second response.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace bytebufferpool-backed return values with bytes.Buffer storage so
the downloaded and compressed slices remain valid after downloadBinary
returns. Close the zstd encoder on copy failure and propagate final close
errors.
Add round-trip and pool-reuse regression coverage for both returned
buffers.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Wire the preallocated pooled shard slices into parallelReader instead
of discarding them and allocating a buffer for each disk.
Keep readerToBuf as a permutation while preferred readers are reordered.
The former assignments could duplicate a buffer slot after multiple
swaps, causing concurrent writes and a possible decode stall. Add pool
aliasing and mapping regression coverage.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
hash.Checksum.AddPart returned before seeding the accumulator when the
part was empty, so a multipart object with no content at all ended up
with no checksum instead of the checksum of zero bytes. Completing such
an upload failed with XAmzContentChecksumMismatch when the client
supplied the correct object checksum, and stored an empty checksum when
it did not.
Run the type check and the first checksum seeding before the zero size
early return. Appending zero bytes still leaves an existing accumulator
unchanged, so only the all empty case changes: a zero length part
followed by content already merged correctly, because prepending no bytes
does not alter a CRC.
AddPart has a single production caller, the multipart completion path, and
its part checksum type is derived from the upload's own checksum type, so
the type check now reached for zero sized parts cannot fire there.
Add a table test over CRC32, CRC32C and CRC64NVME covering every position
an empty part can take, and an API level zero length full object upload
that exercises the persisted AppendTo/ReadCheckSums round trip.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
CompleteMultipartUpload compared the per-part checksum taken from the
request body against the stored part checksum unconditionally, so a
client that sent only PartNumber and ETag for each part failed with
InvalidPart.
AWS S3 requires part level checksums in the completion body only for
composite checksum types. For full object types the client sends the
object level checksum in the request headers instead and does not retain
per-part values - that is the point of FULL_OBJECT. Reproduced with
boto3 1.43.58: it puts x-amz-checksum-crc32 on each UploadPart and the
full object checksum on the completion headers, but emits only ETag and
PartNumber in the completion body. A caller could only get such an upload
through by collecting the per-part checksums from the UploadPart
responses and echoing them back, which is exactly the bookkeeping
FULL_OBJECT exists to avoid and which no off-the-shelf SDK call does.
minio-go does echo them, which is why mc never hit this.
Treat the part checksum as optional when the upload declared a full
object checksum type and the client sent no part checksum at all. A part
carrying any checksum is still validated against the stored one -
including one sent under the wrong algorithm, which cannot match and is
rejected - composite uploads keep requiring a checksum for every part,
and the merged object checksum is still computed from the server stored,
upload time validated part checksums, never from client supplied values,
so integrity is unchanged.
Covered by API level tests over CRC32, CRC32C and CRC64NVME on both the
single drive and erasure backends, with guards for a wrong object
checksum, a wrong part checksum, a part checksum under another algorithm,
a mix of present and omitted part checksums, an absent object checksum,
and composite uploads still requiring every part checksum.
Fixes#31
Reported-by: Christophe Bornet <cbornet@users.noreply.github.com>
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace Georg Mangold's console fork (v1.9.1) with pgsty/silo-console
at the v2.0.0 release, Pigsty's maintained console carrying the SILO
identity, the redesigned and hardened web app, and regenerated embedded
assets. The fork keeps the upstream github.com/minio/console module
path without a /v2 suffix, so the replace pins v2.0.0's tagged commit
(b952a120) as a pseudo-version rather than the tag itself.
highwayhash v1.0.4 and go-m1cpu v0.2.2 follow from the new console's
requirements; the previous console resolved a v1.0.4 pre-release and
v0.2.1. Full-tree build verified against the new pin.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace github.com/minio/mc with the pinned pgsty/mc 2026-08-01 release while preserving the upstream module identity. Refresh the module graph, including the fork-required etcd 3.6.9 patch update.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Move MinIO from v1.18.6 to v1.18.7 to pick up the GO-2026-5841
fix. The vulnerable dictionary symbols are not reachable in this tree,
but keeping the direct compression dependency patched avoids carrying
the affected release.
Verified with go mod verify, full go test and go vet runs, and
govulncheck reporting no reachable symbol or imported-package
vulnerabilities.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
thrift 0.23.0 compares an int against math.MaxUint32, which does not
compile on 32-bit platforms and has broken linux/arm and linux/386
builds since the 2026-06-18 dependency refresh; 0.24.0 carries the
upstream fix. The library only reaches us through fraugster/parquet-go
for S3 Select, whose decode tests still pass.
go-systemd v22.7.0 moved CLOCK_MONOTONIC into a file built for every
unix platform and no longer compiles on NetBSD. Pin v22.6.0 with a
replace directive - a plain require cannot hold because Console pulls
v22.7.0 back in through MVS - until upstream ships the fix. The server
only consumes daemon.SdNotify, which both versions provide.
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Require Go 1.26.5, gRPC 1.82.1, and x/text 0.39.0 while retaining the dependency graph's existing pins wherever MVS permits. Restore the blocking govulncheck job and correct the documented security advisories.
Use the maintained Silo image for both the server and post-install mc jobs. The image bundles mcli with an mc compatibility link.
Bump the chart major version because changing the default registry can affect admission and pull policies. Keep Helm repository packaging and indexes outside this source-chart change.
Replace upstream docs.min.io references with the SILO documentation domain and remove the upstream community path prefix.
Co-authored-by: waterkip <6317502+waterkip@users.noreply.github.com>
Update the module Go directive and release Docker build images from Go 1.26.2 to Go 1.26.4 so local, CI, hotfix, and release builds use the same patched toolchain.
Keep ordinary Go module requirements and replacements unchanged; this intentionally avoids a third-party dependency refresh while allowing container system packages to refresh through the newer golang Alpine base image and existing apk resolution.
Update the security advisory index to record the Go 1.26.4 toolchain bump alongside the earlier Go 1.26.2 security update.
Verified with go build ./..., go vet ./cmd/, and focused cmd tests. go mod tidy -diff was attempted as a read-only dependency drift check but could not complete because proxy.golang.org timed out while fetching uncached transitive test modules.
Co-authored-by: Codex <codex@openai.com>
The internode storage-REST ReadMultiple endpoint (/rmpl) joined
attacker-controlled Bucket/Prefix/Files into a filesystem path with no
validation, letting a peer with internode credentials read files outside
the drive root (GHSA-xh8f-g2qw-gcm7).
ReadMultiple has had no production caller since upstream #20390 removed
the last one (listParts) in Sep 2024; multipart now uses ReadParts (/rps).
Following the upstream fix, remove the whole API instead of validating
paths: route constant and registration, server handler, REST client
wrapper, the StorageAPI/xlStorage/xlStorageDiskIDCheck methods, the
storageMetricReadMultiple metric, and the ReadMultipleReq/Resp datatypes.
Regenerated the msgp and stringer outputs; storageRESTVersion stays at v63.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Route JSON Lines through the bounded PReader path so oversized records are rejected consistently instead of bypassing the limit on SIMD-capable CPUs.
Preserve S3 Select error codes in stream error events, wrap JSON parser errors as JSONParsingError, and flush completed records before returning a terminal error event. Add regression coverage for oversized JSON Lines input and error code preservation.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Remove the per-username LDAP STS throttle bucket and keep the limiter keyed only by source IP. A username bucket is shared across all clients and lets one source keep a known account's bucket drained with bad-password attempts, locking the legitimate user out before LDAP bind.
For trusted proxies, stop trusting the left-most forwarded address. Resolve X-Forwarded-For right-to-left, skip trusted proxy hops, reject catch-all trusted-proxy CIDRs, and intentionally ignore RFC 7239 Forwarded for this security-sensitive bucket. Document that X-Real-IP is trusted verbatim and must be overwritten by the trusted proxy, not passed through from clients.
Update focused limiter, source-IP, trusted-proxy, and LDAP config tests to cover source-only buckets, spoofed appended XFF, multi-hop trusted proxies, Forwarded fallback, catch-all rejection, and the X-Real-IP deployment contract.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Allows the trackingResponseWriter to properly proxy flush
requests to the underlying http.Flusher. This ensures
compatibility with streaming responses.
Update the STS, security, select, and Docker documentation to reflect the recent hardening work, including LDAP STS throttling details, OIDC JWT verification changes, and the new pgsty-specific security policy and advisory index.
Rewrite repository and raw-document links that still pointed at minio/minio so the docs consistently reference pgsty/minio instead.
The core idea is to keep the documentation aligned with the fork's actual security behavior, ownership, and upgrade guidance without mixing in unrelated code changes.
Prevent LDAP STS reservation cancel paths from over-crediting rate-limit buckets by capping refill and refund capacity against in-flight reservations.
Add an explicit trusted-proxy allowlist for LDAP STS source bucketing, prefer clean X-Real-IP values on trusted peers, and extend tests/docs for the new behavior.
Use the socket peer address for LDAP STS per-IP rate limiting instead of the generic forwarded-header-aware helper. This keeps the security-sensitive rate-limit key from trusting spoofable X-Forwarded-For, X-Real-IP, and Forwarded headers while leaving the rest of the source-IP behavior unchanged.
Add focused regression coverage for RemoteAddr parsing, header spoofing, and peer-address bucket selection.
Update go.mod and all golang build images from 1.26.1 to 1.26.2 to pick up the upstream 2026-04-07 security release.
This includes fixes for CVE-2026-32280 / CVE-2026-32281 in crypto/x509, CVE-2026-32283 in crypto/tls, and the related toolchain and standard library security fixes shipped in go1.26.2, without changing any unrelated dependencies.
Track issue #28 / GHSA-9c4q-hq6p-c237 as fake CVE-2026-40028. Close the Snowball auto-extract auth gap in PutObjectExtractHandler by treating authTypeStreamingUnsignedTrailer the same as ordinary PUTs: honor X-Amz-Decoded-Content-Length, initialize newUnsignedV4ChunkedReader(), and verify the SigV4 request before any tar bytes reach untar(). This removes the forged-signature write primitive that let a single request fan out into arbitrary extracted object creation.
Add regression coverage for forged-signature Snowball unsigned-trailer writes, anonymous Snowball requests against non-public buckets, and legitimate signed Snowball extraction with trailing CRC32 trailers. Validate the new tests against the vulnerable parent and patched tree, and confirm with containerized before/after smoke runs that the exploit succeeds pre-fix, fails post-fix, and normal signed Snowball uploads still extract correctly.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Track issue #27 / GHSA-hv4r-mvr4-25vw as fake CVE-2026-40027. Close the unsigned-trailer trust flaw that let query-string credentials skip signature verification in PutObject and PutObjectPart by moving presigned rejection and SigV4 verification into newUnsignedV4ChunkedReader(), so authTypeStreamingUnsignedTrailer can no longer silently downgrade query auth into an anonymous body read.
Add focused regression coverage for forged query-string-only unsigned-trailer PUTs and multipart uploads, mixed header/query auth rejection, and anonymous unsigned-trailer writes that remain allowed only when bucket policy explicitly permits them. Validate the new tests against the vulnerable parent and confirm with before/after live-server runs that presigned unsigned-trailer attacks are rejected while legitimate header-authenticated and policy-driven flows still work.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Enforce the 1 MiB maxCharsPerRecord limit while splitting CSV and line-delimited JSON input so oversized records are rejected before they can be buffered and parsed.
Return OverMaxRecordSize for these failures instead of collapsing them into InternalError, and preserve splitter errors in the JSON worker so oversized-record failures are not lost after successful partial decode.
Close the replication-header trust flaw that allowed ordinary PutObject and CopyObject requests to smuggle X-Minio-Replication-* headers into X-Minio-Internal-* SSE metadata and write objects into an unreadable state. Stop accepting replication-only metadata in the default extraction path, restore it only after a trusted replication write has passed ReplicateObjectAction, and tighten CopyObject by sanitizing replication-only request headers before metadata, precondition, and SSE-C source handling consume them. Also gate replica status writes on the same trusted replication path and restore replication SSE metadata in multipart and snowball upload flows so legitimate replication continues to work.
Add focused regression coverage for untrusted PUT and COPY header poisoning at the handler layer, plus helper tests for trusted vs untrusted metadata extraction and CopyObject header sanitization. Validate the new tests against both the patched tree and the vulnerable HEAD baseline, and confirm with live server before/after runs that malicious PUT/COPY requests no longer turn objects unreadable.
Co-authored-by: Codex <codex@openai.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Prevent username enumeration in AssumeRoleWithLDAPIdentity by returning the same external STS error for unknown users and invalid passwords, while preserving LDAP infrastructure failures as upstream errors so they continue to surface as 500s and remain visible in server logs.
Add a small in-memory rate limiter for LDAP STS login attempts, keyed by source IP and normalized username, and add regression coverage for auth failure classification, throttling, and Docker-backed LDAP end-to-end flows.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Close the OIDC JWT algorithm confusion flaw in AssumeRoleWithWebIdentity by restoring a JWKS-only verification path. Stop injecting the client secret into the verifier keyring and restrict accepted signing methods to the asymmetric algorithms already supported by the existing JWKS flow.
Add regression coverage to verify HS256 tokens are rejected, RS256 tokens remain valid, and JWKS refresh and retry logic cannot bypass the method allowlist.
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Code <claude-code@anthropic.com>
Replace minio/pkg/v3 with pgsty/minio-pkg/v3 v3.6.3 to fix LDAP TLS
regression where DialURL() was not passing TLS config for ldaps://
connections, causing InsecureSkipVerify and RootCAs to be silently
ignored (x509: certificate signed by unknown authority).
Pin four dependencies to avoid breaking changes introduced in 5abd9a80f:
- go-ldap/ldap/v3 v3.4.12: v3.4.13 rewrote GetLDAPError() internals
- IBM/sarama v1.45.1: v1.46.0 changed Kafka protocol version negotiation
- lib/pq v1.10.9: v1.11.0 treats nil []byte as NULL and drops PG <14
- etcd v3.6.8: stay on intermediate version per policy
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
We maintain our own release pipeline (release.yml, test-release.yml)
and have no use for the upstream test/lint/integration workflows.
They reference infrastructure and secrets we don't have, and the
PR-triggered jobs never fire since we don't take external pull requests.
Closes#14
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rework Dockerfile.goreleaser to download the latest mcli binary from
pgsty/mc GitHub releases, verify its SHA-256 checksum, and install both
mcli and mc (symlink) into the final image alongside minio and curl.
Also add download-static-curl.sh to goreleaser extra_files and enable
workflow_dispatch for the release workflow.
Go 1.26.1 tightens a few toolchain checks that older builds tolerated.\n\nCast aliased replication status values back to their defining type before calling the generated msgp helpers, and replace Sprintf+WriteString pairs with direct Fprintf calls where needed.\n\nThese are compatibility-only source changes to keep the cmd package building cleanly under the newer linker/toolchain.
Switch to community-maintained console fork (georgmangold/console v1.9.1)
and update dependencies accordingly. Fix go vet format directive in
grid_test.go and adapt test status code for Go 1.26 HTTP semantics.
- Revert console dependency from stripped v1.7.7-pre to v1.7.6,
restoring the full embedded management console
- Rewrite README disclaimer with proper trademark attribution and
nominative fair use language for AGPL compliance
- Update documentation links and Go module paths to this repository
- Restore docs removed upstream (hotfixes.md, metrics/v3.md)
- Restore feature request issue template
- Update Go version to 1.26.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix(api): Don't send responses twice.
In some cases multiple responses are being sent for one request, causing
the API server to incorrectly drop connections.
This change introduces a ResponseWriter which tracks whether a
response has already been sent. This is used to prevent a response being
sent if something already has (e.g. by a preconditions check function).
Fixes#21633.
Co-authored-by: Menno Finlay-Smits <hello@menno.io>
This fixes a security issue where sub-policy attached to a service
account or STS account is not properly validated under certain "own"
account operations (like creating new service accounts). This allowed a
service account to create new service accounts for the same user
bypassing the inline policy restriction.
fix: use correct dummy ARN for claim-based OIDC provider
When listing OIDC access keys, use the correct ARN when looking up the provider configuration for the claim-based provider. Without this it was impossible to list access keys for a claim-based provider, only for a role-policy-based provider.
Fixesminio/minio#21548
Commit b6eb8dff64 renamed compression
setting environment variables to follow consistent style.
Although it preserved backward compatibility for the most part (i.e. it
handled MINIO_COMPRESS_ALLOW_ENCRYPTION, MINIO_COMPRESS_EXTENSIONS, and
MINIO_COMPRESS_MIME_TYPES), MINIO_COMPRESS_ENABLE was left behind.
Additionally, due to incorrect fallback ordering, and DefaultKVS
containing enable=off allow_encryption=off (so kvs.Get should've been
tried last), that commit broke MINIO_COMPRESS_ALLOW_ENCRYPTION (even
though it appeared to be handled), and even older MINIO_COMPRESS, too.
The legacy MIME types and extensions variables take precedence over both
config and new variables, so they don't need fixing.
`go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...` executed.
`go generate ./...` ran afterwards to keep generated.
* Store `ctx.Done` channel in a struct instead of a `ctx`. See: https://go.dev/blog/context-and-structs
* Return from `handleListener` on `ctx` cancellation, preventing goroutine leaks
* Simplify `handleListener` by removing the `send` closure. The `handleListener` is inlined by the compiler
* Return the first error from `Close`
* Preallocate slice in `Addrs`
* Reduce duplication in handling `opts.Trace`
* http/listener: revert error propagation from Close()
* http/listener: preserve original listener address in Addr()
* Preserve the original address when calling Addr() with multiple listeners
* Remove unused listeners from the slice
RoleARN is a required parameter in AssumeRoleWithWebIdentity,
according to the standard AWS implementation, and the official
AWS SDKs and CLI will not allow you to assume a role from a JWT
without also specifying a RoleARN. This meant that it was not
possible to use the official SDKs for claim-based OIDC with Minio
(minio/minio#21421), since Minio required you to _omit_ the RoleARN in this case.
minio/minio#21468 attempted to fix this by disabling the validation
of the RoleARN when a claim-based provider was configured, but this had
the side effect of making it impossible to have a mixture of claim-based
and role-based OIDC providers configured at the same time - every
authentication would be treated as claim-based, ignoring the RoleARN entirely.
This is an alternative fix, whereby:
- _if_ the `RoleARN` is one that Minio knows about, then use the associated role policy
- if the `RoleARN` is not recognised, but there is a claim-based provider configured, then ignore the role ARN and attempt authentication with the claim-based provider
- if the `RoleARN` is not recognised, and there is _no_ claim-based provider, then return an error.
Fixes incorrect application of ilm expiry rules on versioned objects
when replication is enabled.
Regression from https://github.com/minio/minio/pull/20441 which sends
DeleteObject calls to all pools. This is a problem for replication + ilm
scenario since replicated version can end up in a pool by itself instead of
pool where remaining object versions reside.
For example, if the delete marker is set on pool1 and object versions exist on
pool2, the second rule below will cause the delete marker to be expired by ilm
policy since it is the single version present in pool1
```
{
"Rules": [
{
"ID": "cs6il1ri2hp48g71mdjg",
"NoncurrentVersionExpiration": {
"NoncurrentDays": 14
},
"Status": "Enabled"
},
{
"Expiration": {
"ExpiredObjectDeleteMarker": true
},
"ID": "cs6inj3i2hp4po19cil0",
"Status": "Enabled"
}
]
}
```
this is minor PR that supports building on RISC-V 64,
this PR is for compilation only. There is no guarantee
that code is tested and will work in production.
S3 listing call is usually sent with a 'max-keys' parameter. This
'max-keys' will also be passed to WalkDir() call. However, when ILM is
enabled in a bucket and some objects are skipped, the listing can
return IsTruncated set to false even if there are more entries in
the drives.
The reason is that drives stop feeding the listing code because it has
max-keys parameter and the listing code thinks listing is finished
because it is being fed anymore.
Ask the drives to not stop listing and relies on the context
cancellation to stop listing in the drives as fast as possible.
This commit removes FIPS 140-2 related code for the following
reasons:
- FIPS 140-2 is a compliance, not a security requirement. Being
FIPS 140-2 compliant has no security implication on its own.
From a tech. perspetive, a FIPS 140-2 compliant implementation
is not necessarily secure and a non-FIPS 140-2 compliant implementation
is not necessarily insecure. It depends on the concret design and
crypto primitives/constructions used.
- The boringcrypto branch used to achieve FIPS 140-2 compliance was never
officially supported by the Go team and is now in maintainance mode.
It is replaced by a built-in FIPS 140-3 module. It will be removed
eventually. Ref: https://github.com/golang/go/issues/69536
- FIPS 140-2 modules are no longer re-certified after Sep. 2026.
Ref: https://csrc.nist.gov/projects/cryptographic-module-validation-program
Signed-off-by: Andreas Auernhammer <github@aead.dev>
Fixes#21249
Example params: `-ftp=force-tls=true -ftp="tls-private-key=ftp/private.key" -ftp="tls-public-cert=ftp/public.crt"`
If MinIO is set up for TLS those certs will be used.
we had a chicken and egg problem with this feature even
when used with kes the credentials generation would
not work in correct sequence causing setup/deployment
disruptions.
This PR streamlines all of this properly to ensure that
this functionality works as advertised.
On buffered streams the final entries could be missing, if a lot
are delivered when stream ends.
Fixes end-of-stream cancelling return of final entries by canceling
with the StreamEOF error.
in a specific corner case when you only have dangling
objects with single shard left over, we end up a situation
where healing is unable to list this dangling object to
purge due to the fact that listing logic expected only
`len(disks)/2+1` - where as when you make this choice you
end up with a situation that the drive where this object
is present is not part of your expected disks list, causing
it to be never listed and ignored into perpetuity.
change the logic such that HealObjects() would be able
to listAndHeal() per set properly on all its drives, since
there is really no other way to do this cleanly, however
instead of "listing" on all erasure sets simultaneously, we
list on '3' at a time. So in a large enough cluster this is
fairly staggered.
To make sure that no objects were skipped for any reason,
decommissioning does a second phase of listing to check if there
are some objects that need to be decommissioned. However, the code
forgot to skip orphan delete markers since the decom code already
skips it.
Make the code ignore delete markers in in the verification phase.
Co-authored-by: Anis Eleuch <anis@min.io>
There is prefix in json, but not in the equivalent command line. Although the role of prefix has been explained in the previous example, I think it should be supplemented.
This is a security incident fix, it would seem like since
the implementation of unsigned payload trailer on PUTs,
we do not validate the signature of the incoming request.
The signature can be invalid and is totally being ignored,
this in-turn allows any arbitrary secret to upload objects
given the user has "WRITE" permissions on the bucket, since
acces-key is a public information in general exposes these
potential users with WRITE on the bucket to be used by any
arbitrary client to make a fake request to MinIO the signature
under Authorization: header is totally ignored.
A test has been added to cover this scenario and fail
appropriately.
- Move VersionPurgeStatus into replication package
- ilm: Evaluate policy w/ obj retention/replication
- lifecycle: Use Evaluator to enforce ILM in scanner
- Unit tests covering ILM, replication and retention
- Simplify NewEvaluator constructor
When decommissioning is started, the list of buckets to decommission is
calculated, however, a bucket can be removed before decommissioning reaches
it. This will cause an infinite loop of listing error complaining about
the non-existence of the bucket. This commit will ignore
errVolumeNotFound to skip the not found bucket.
Enforce a bucket count limit on metrics for v2 calls.
If people hit this limit, they should move to v3, as certain calls explode with high bucket count.
Reviewers: This *should* only affect v2 calls, but the complexity is overwhelming.
If a user attempts to authenticate with a key but does not have an
sshpubkey attribute in LDAP, the server allows the connection, which
means the server trusted the key without reason. This is now fixed,
and a test has been added for validation.
Allow multiple private keys and extract all files from streams.
Place files in the folder with `.enc` removed.
Do basic checks so streams cannot traverse outside of the folder.
This commit adds the `MINIO_KMS_REPLICATE_KEYID` env. variable.
By default - if not specified or not set to `off` - MinIO will
replicate the KMS key ID of an object.
If `MINIO_KMS_REPLICATE_KEYID=off`, MinIO does not include the
object's KMS Key ID when replicating an object. However, it always
sets the SSE-KMS encryption header. This ensures that the object
gets encrypted using SSE-KMS. The target site chooses the KMS key
ID that gets used based on the site and bucket config.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
This commit allows clients to provide a set of intermediate CA
certificates (up to `MaxIntermediateCAs`) that the server will
use as intermediate CAs when verifying the trust chain from the
client leaf certificate up to one trusted root CA.
This is required if the client leaf certificate is not issued by
a trusted CA directly but by an intermediate CA. Without this commit,
MinIO rejects such certificates.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
If object is uploaded with tags, the internal tagging-timestamp tracked
for replication will be missing. Default to ModTime in such cases to
allow tags to be synced correctly.
Also fixing a regression in fetching tags and tag comparison
* fix: proxy requests to honor global transport
Load the globalProxyEndpoint properly
also, currently, the proxy requests will fail silently for batch cancel
even if the proxy fails; instead,d properly send the corresponding error back
for such proxy failures if opted
* pass the transport to the GetProxyEnpoints function
---------
Co-authored-by: Praveen raj Mani <praveen@minio.io>
Reject new lock requests immediately when 1000 goroutines are queued
for the local lock mutex.
We do not reject unlocking, refreshing, or maintenance; they add to the count.
The limit is set to allow for bursty behavior but prevent requests from
overloading the server completely.
Currently, DeleteObjects() tries to find the object's pool before
sending a delete request. This only works well when an object has
multiple versions in different pools since looking for the pool does
not consider the version-id. When an S3 client wants to
remove a version-id that exists in pool 2, the delete request will be
directed to pool one because it has another version of the same object.
This commit will remove looking for pool logic and will send a delete
request to all pools in parallel. This should not cause any performance
regression in most of the cases since the object will unlikely exist
in only one pool, and the performance price will be similar to
getPoolIndex() in that case.
Earlier, cluster and bucket metrics were named
`minio_usage_last_activity_nano_seconds`.
The bucket level is now named as
`minio_bucket_usage_last_activity_nano_seconds`
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
When compression is enabled, the final object size is not calculated in
that case, we need to make sure that the provided buffer is always
more significant than the shard size, the bitrot will always calculate
the hash of blocks with shard size, except the last block.
Before https://github.com/minio/minio/pull/20575, files could pick up indices
from unrelated files if no index was added.
This would result in these files not being consistent across a set.
When loading, search for the compression indicators and check if they
are within the problematic date range, and clean up any parts that have
an index but shouldn't.
The test validates that the signature matches the one in files stored without an index.
Bumps xlMetaVersion, so this check doesn't have to be made for future versions.
It is possible delete marker was received on old pool as decom
move in progress, this PR allows decom retry to ensure these
delete markers are moved to new pool so that decommission can
be completed.
Fixes#20819
Add profiling potential crash wourkaround
Using admin traces could potentially crash the server (or handler more likely) due to upstream divide by 0: https://github.com/felixge/fgprof/pull/34
Ensure the profile always runs 100ms before stopping, so sample count isn't 0 (default sample rate ~10ms/sample, but allow for cpu starvation)
If one object has many parts where all parts are readable but some parts
are missing from some drives, this object can be sometimes un-healable,
which is wrong.
This commit will avoid reading from drives that have missing, corrupted or
outdated xl.meta. It will also check if any part is unreadable to avoid
healing in that case.
we do not need to hold the read locks at the higher
layer instead before reading the body, instead hold
the read locks properly at the time of renamePart()
for protection from racy part overwrites to compete
with concurrent completeMultipart().
CheckParts call can take time to verify 10k parts of a single object in a single drive.
To avoid an internal dealine of one minute in the single handler RPC, this commit will
switch to streaming RPC instead.
This API had missing permissions checking, allowing a user to change
their policy mapping by:
1. Craft iam-info.zip file: Update own user permission in
user_mappings.json
2. Upload it via `mc admin cluster iam import nobody iam-info.zip`
Here `nobody` can be a user with pretty much any kind of permission (but
not anonymous) and this ends up working.
Some more detailed steps - start from a fresh setup:
```
./minio server /tmp/d{1...4} &
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc admin user add myminio nobody nobody123
mc admin policy attach myminio readwrite nobody nobody123
mc alias set nobody http://localhost:9000 nobody nobody123
mc admin cluster iam export myminio
mkdir /tmp/x && mv myminio-iam-info.zip /tmp/x
cd /tmp/x
unzip myminio-iam-info.zip
echo '{"nobody":{"version":1,"policy":"consoleAdmin","updatedAt":"2024-08-13T19:47:10.1Z"}}' > \
iam-assets/user_mappings.json
zip -r myminio-iam-info-updated.zip iam-assets/
mc admin cluster iam import nobody ./myminio-iam-info-updated.zip
mc admin service restart nobody
```
`mc admin heal ALIAS/bucket/object` does not have any flag to heal
object noncurrent versions, this commit will make healing of the object
noncurrent versions implicitly asked.
This also fixes the 'mc admin heal ALIAS/bucket/object' that does not work
correctly when the bucket is versioned. This has been broken since Apr 2023.
Golang http.Server will call SetReadDeadline overwriting the previous
deadline configuration set after a new connection Accept in the custom
listener code. Therefore, --idle-timeout was not correctly respected.
Make http.Server read/write timeout similar to --idle-timeout.
The code assigns corrupted state to a drive for any unexpected error,
which is confusing for users. This change will make sure to assign
corrupted state only for corrupted parts or xl.meta. Use unknown state
with a explanation for any unexpected error, like canceled, deadline
errors, drive timeout, ...
Also make sure to return the bucket/object name when the object is not
found or marked not found by the heal dangling code.
* Add the policy name to the audit log tags when doing policy-based API calls
* Audit log the retention settings requested in the API call
* Audit log of retention on PutObjectRetention API path too
The RemoveUser API only removes internal users, and it reports success
when it didnt find the internal user account for deletion. When provided
with a service account, it should not report success as that is misleading.
Update github.com/cosnicolaou/pbzip2 to latest version for
significant performance improvements. This update brings a 45%
reduction in processing time.
Previously, not setting http.Config.HTTPTimeout for logger webhook
was resulting in a timeout of 0, and causing "deadline exceeded"
errors in log webhook.
This change introduces a new env variable for configuring log webhook
timeout and more importantly sets it when the config is initialised.
Manual heal can return XMinioHealInvalidClientToken if the manual
healing is started in the first node, and the next mc call to get the
heal status is landed on another node. The reason is that redirection
based on the token ID is not able to redirect requests to the first node
due to a typo.
This also affects the batch cancel command if the batch is being done in
the first node, the user will never be able to cancel it due to the same
bug.
HTTP likes to slap an infinite read deadline on a connection and
do a blocking read while the response is being written.
This effectively means that a reading deadline becomes the
request-response deadline.
Instead of enforcing our timeout, we pass it through and keep
"infinite deadline" is sticky on connections.
However, we still "record" when reads are aborted, so we never overwrite that.
The HTTP server should have `ReadTimeout` and `IdleTimeout` set for the deadline to be effective.
Use --idle-timeout for incoming connections.
Since DeadlineConn would send deadline updates directly upstream,
it would race with Read/Write operations. The stdlib will perform a read,
but do an async SetReadDeadLine(unix(1)) to cancel the Read in
`abortPendingRead`. In this case, the Read may override the
deadline intended to cancel the read.
Stop updating deadlines if a deadline in the past is seen and when Close is called.
A mutex now protects all upstream deadline calls to avoid races.
This should fix the short-term buildup of...
```
365 @ 0x44112e 0x4756b9 0x475699 0x483525 0x732286 0x737407 0x73816b 0x479601
# 0x475698 sync.runtime_notifyListWait+0x138 runtime/sema.go:569
# 0x483524 sync.(*Cond).Wait+0x84 sync/cond.go:70
# 0x732285 net/http.(*connReader).abortPendingRead+0xa5 net/http/server.go:729
# 0x737406 net/http.(*response).finishRequest+0x86 net/http/server.go:1676
# 0x73816a net/http.(*conn).serve+0x62a net/http/server.go:2050
```
AFAICT Only affects internode calls that create a connection (non-grid).
These are needed checks for the functions to be un-crashable with any input
given to `msgUnPath` (tested with fuzzing).
Both conditions would result in a crash, which prevents that. Some
additional upstream checks are needed.
Fixes#20610
This PR fixes a regression introduced in https://github.com/minio/minio/pull/19797
by restoring the healing ability of transitioned objects
Bonus: support for transitioned objects to carry original
The object name is for future reverse lookups if necessary.
Also fix parity calculation for tiered objects to n/2 for n/2 == (parity)
Existing implementation runs IAM purge routines for expired LDAP and
OIDC accounts with a probability of 0.25 after every IAM refresh. This
change ensures that they are run once in each hour.
xlStorage.Healing() returns nil if there is an error reading
.healing.bin or if this latter is empty. healing.bin update()
call returns early if .healing.bin is empty; hence, no further update
of .healing.bin is possible.
A .healing.bin can be empty if os.Open() with O_TRUNC is successful
but the next Write returns an error.
To avoid this weird situation, avoid making healingTracker.update()
to return early if .healing.bin is empty, so write again.
This commit also fixes wrong error log printing when an object is
healed in another drive in the same erasure set but not in the drive
that is actively healing by fresh drive healing code. Currently, it prints
<nil> instead of a factual error.
* heal: Scan .minio.sys metadata only during site-wide heal (#137)
mc admin heal always invoke .minio.sys heal, but sometimes, this latter
contains a lot of data, many service accounts, STS accounts etc, which
makes mc admin heal command very slow.
Only invoke .minio.sys healing when no bucket was specified in `mc admin
heal` command.
Healing a large object with a normal scan mode where no parts read
is involved can still fail after 30 seconds if an object has
There are too many parts when hard disks are being used mainly.
The reason is there is a general deadline that checks for all parts we
do a deadline per part.
When listing objects with metadata, avoid returning an "expires" time
metadata value when its value is the zero time as this means that no
expires value is set on the object.
The condition were incorrect as we were comparing the filter
value against the modification time object.
For example if created after filter date is after modification
time of object, that means object was created before the filter
time and should be skipped while replication because as per the
filter we need only the objects created after the filter date.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
When Keycloak vendor is set, the code will start to clean up service
accounts that parents do not exist anymore. However, the code will also
look for the parent user of site-replicator-0, MINIO_ROOT_USER, which
obviously does not exist in Keycloak. Therefore, the site-replicator-0
will be removed automatically.
This commit will avoid cleaning up service accounts generated from
the root user.
The verify file handler response format was changed from gob to msgp
since two months but we forgot updating the verify handler client.
VerifyFile is only called during a heal deep scan (bitrot check).
HealObject() will fail in that case and will mark all disks corrupted and
will return early (as unrecoverable object but it will also not be
removed)
It is a bit rare for HealObject to be called with a deep scan flag. It
is called when a HealObject with a normal scan (e.g. new drive healing)
detects a bitrot corruption, therefore healing objects with a detected
bitrot corruption will fail.
in cases where we cannot possibly know a way to read and
construct the object, it is impossible to achieve any form of
quorum via xl.meta while we have sufficient responses from
all the drives, we should return object not found.
- PutObjectMetadata()
- PutObjectTags()
- DeleteObjectTags()
- TransitionObject()
- RestoreTransitionObject()
Also improve the behavior of multipart code across
pool locks, hold locks only once per upload ID for
- CompleteMultipartUpload()
- AbortMultipartUpload()
- ListObjectParts() (read-lock)
- GetMultipartInfo() (read-lock)
- PutObjectPart() (read-lock)
This avoids lock attempts across pools for no
reason, this increases O(n) when there are n-pools.
multi-object deletion may or may not compete with locks
granted for other callers, causing concurrent operations
to succeed on each other.
A continuation of the PR https://github.com/minio/minio/pull/20356
When custom authorization via plugin is enabled, the console will now
render the UI as if all actions are allowed. Since server cannot
determine the exact policy allowed for a user via the plugin, this is
acceptable to do. If a particular action is actually not allowed by the
plugin the call will result in an error.
Previously the server was evaluating a policy when custom authZ is
enabled - this is fixed now.
The "unlimited" value on PPC wasn't exactly the same as amd64.
Instead compare against an "unreasonably big value".
Would cause OOM in anything using the concurrent request limit.
Add TTFB for all requests in metrics-v3 in addition to the existing
GetObject. Also for the requests that do not return a body in the
response, calculate TTFB as the HTTP status code and the headers are
sent.
A batch job will fail if the retry attempt is not provided. The reason
is that the code mistakenly gets the retry attempts from the job status
rather than the job yaml file.
This will also set a default empty prefix for batch expiration.
Also this will avoid trimming the prefix since the yaml decoder already
does that if no quotes were provided, and we should not trim if quotes
were provided and the user provided a leading or a trailing space.
Tests if imported service accounts have
required access to buckets and objects.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
- PutObject() for multi-pooled was holding large
region locks, which was not necessary. This affects
almost all slowpoke clients and lengthy uploads.
- Re-arrange locks for CompleteMultipart, PutObject
to be close to rename()
Currently, it is not possible to remove a tier if it is not accessible
or contains some data, add a force flag to make the removal successful
in that case.
When the encryption and compression are both enabled, the
the server will avoid compressing the data for no apparent reason
This commit will enable it and update unit tests.
Download static cURL into release Docker image for all supported architectures.
Currently, the static cURL executable is only downloaded for the `amd64` architecture. However, `arm64` and `ppc64le` variants are also [available](https://github.com/moparisthebest/static-curl/releases/tag/v8.7.1).
postUpload() incorrectly saves actual size as '-1'
we should save correct size when its possible.
Bonus: fix the PutObjectPart() write locker, instead
of holding a lock before we read the client stream.
We should hold it only when we need to commit the parts.
this cache will be honored only when `prefix=""` while
performing ListMultipartUploads() operation.
This is mainly to satisfy applications like alluxio
for their underfs implementation and tests.
replaces https://github.com/minio/minio/pull/20181
AFAICT we send a canceled context to unlock (and thereby releaseAll). This will cause network calls to fail.
Instead use background and add 30s timeout.
Current implementation retries forever until our
log buffer is full, and we start dropping events.
This PR allows you to set a value until we give
up on existing audit/logger batches to proceed to
process the new ones.
Bonus:
- do not blow up buffers beyond batchSize value
- do not leak the ticker if the worker returns
The items will be saved per target batch and will
be committed to the queue store when the batch is full
Also, periodically commit the batched items to the queue store
based on configured commit_timeout; default is 30s;
Bonus: compress queue store multi writes
rebalance metadata is good to have only,
if it cannot be loaded when starting MinIO
for some reason we can possibly ignore it
and move on and let user start rebalance
again if needed.
readParts requires that both part.N and part.N.meta files be present.
This change addresses an issue with how an error to return to the upper
layers was picked from most drives where a UploadPart operation
had failed.
By default, even if MINIO_BROWSER=off set code tries to get free
port available for the console.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
S3 spec does not accept an ILM XML document containing both <Filter>
and <Prefix> XML tags, even if both are empty. That is why we added
a 'set' field in some lifecycle structures to decide when and when not to
show a tag. However, we forgot to disallow marshaling of Filter when
'set' is set to false.
This will fix ILM document replication in a site replication
configuration in some cases.
When the prefix field is not provided in the remote source of a yaml
replication job, the code fails to do listing and makes replication
successful. This commit fixes it.
This change adds a consistent nonce to ensure
that multipart uploads are deterministic on a
per-part basis.
Thanks to @klauspost for the work here minio/sio@3cd3734
locks handed by different pools would become non-compete for
multi-object delete request, this is wrong for obvious
reasons.
New locking implementation and revamp will rewrite multi-object
lock anyway, this is a workaround for now.
Currently, the bucket events and replication targets are only reloaded
with buckets that failed to load during the first cluster startup,
which is wrong because if one bucket change was done in one node but
that node was not able to notify other nodes; the other nodes will
reload the bucket metadata config but fails to set the events and bucket
targets in the memory.
when a hung drive is hot-unplugged, the server might go
into a loop where the previous `format.json` is somehow
still accessible to the process, we try to re-init() drives,
but that seems to cause a previous goroutine to hang around
since it is not canceled away when the drive is closed.
Bonus: add deadline for immediate purge routine, to unblock
it if the drive is blocking mutations.
if a user policy is found, avoid reading from the drives
for missing group mappings, group mappings are not mandatory
and conditional.
This PR restores the older behavior while making sure that
if a direct user policy is not found, we would still attempt
to load from the group from the drives.
This commit simplifies and optimizes the decryption of large (multipart)
objects. This PR does two things:
- Re-write the init logic for the decryption reader
- Reduce the number of OEK decryptions
Before, the init logic copied some SSE HTTP request headers to
parse them later. This is simplified to parsing them right away. This
removes some fields from the decryption reader struct.
Further, the decryption reader decrypted the OEK using the client-provided
key (SSE-C) or the KMS (SSE-S3 / SSE-KMS) for each part. This is redundant
since the OEK is the same for all parts. In particular, a KMS call might be a
network request. Now, the OEK is decrypted once for the entire multipart object.
This should improve latency when reading encrypted multipart objects
and reduce requests to the KMS.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
Use Walk(), which is a recursive listing with versioning, to check if
the bucket has some objects before being removed. This is beneficial
because the bucket can contain multiple dangling objects in multiple
drives.
Also, this will prevent a bug where a bucket is deleted in a deployment
that has many erasure sets but the bucket contains one or few objects
not spread to enough erasure sets.
Currently, retry healing of a new drive healing does not reset
HealedBuckets means that the next healing retry will skip those
buckets. The commit will fix this behavior.
Also, the skipped objects counter will include objects uploaded
that are uploaded after the healing is started.
sftp sends local requests to the S3 port while passing the session token
header when the account corresponds to a service account. However, this
is not permitted and will throw an error: "The security token included in the
request is invalid"
This commit will avoid passing the session token to the upper layer that
initializes MinIO client to avoid this error.
Sometimes, we need historical information in .healing.bin, such as the
number of expired objects that the healing avoids to heal and that can
create drive usage disparency in the same erasure set. For that reason,
this commit will not remove .healing.bin anymore and it will have a new
field called Finished so we know healing is finished in that drive.
Services are unfrozen before `initBackgroundReplication` is finished. This means that
the globalReplicationStats write is racy. Switch to an atomic pointer.
Provide the `ReplicationPool` with the stats, so it doesn't have to be grabbed
from the atomic pointer on every use.
All other loads and checks are nil, and calls return empty values when stats
still haven't been initialized.
* Allow a maximum of 10 seconds to start profiling operations.
* Download up to 16 profiles concurrently, but only allow 10 seconds for
each (does not include write time).
* Add cluster info as the first operation.
* Ignore remote download errors.
* Stop remote profiles if the request is terminated.
If the site replication is enabled and the code tries to extract jwt
claims while the site replication service account credentials are still
not loaded yet, the code will enter an infinite loop, causing in a
high CPU usage.
Another possibility of the infinite loop is having some service accounts
created by an old deployment version where the service account JWT was
signed by the root credentials, but not anymore.
This commit will remove the possibility of the infinite loop in the code
and add root credential fallback to extract claims from old service
accounts.
move away from map[string]interface{} to map[string]string
to simplify the audit, and also provide concise information.
avoids large allocations under load(), reduces the amount
of audit information generated, as the current implementation
was a bit free-form. instead all datastructures must be
flattened.
Previously, we checked if we had a quorum on the DataDir value.
We are removing this check, which allows reading objects with different
DataDir values in a few drives (due to a rebalance-stop race bug)
provided their eTags or ModTimes match.
Since a lot of operations load from storage, do remote calls, add a 10 second timeout to each operation.
This should make `mc admin info` return values even under extreme conditions.
- optimize writing part.N.meta by writing both part.N
and its meta in sequence without network component.
- remove part.N.meta, part.N which were partially success
ful, in quorum loss situations during renamePart()
- allow for strict read quorum check arbitrated via ETag
for the given part number, this makes it double safer
upon final commit.
- return an appropriate error when read quorum is missing,
instead of returning InvalidPart{}, which is non-retryable
error. This kind of situation can happen when many
nodes are going offline in rotation, an example of such
a restart() behavior is statefulset updates in k8s.
fixes#20091
during rebalance stop, it can possibly happen that
Put() would race by overwriting the same object again.
This may very well if done "successfully" it can
potentially proceed to delete the object from the pool,
causing data loss.
This PR enhances #20233 to handle more scenarios such
as these.
Rebalance-stop can race with ongoing rebalance operations. This change
prevents these operations from overwriting objects by checking the source
and destination pool indices are different.
This commit replaces the LDAP client TLS config and
adds a custom list of TLS cipher suites which support
RSA key exchange (RSA kex).
Some LDAP server connections experience a significant slowdown
when these cipher suites are not available. The Go TLS stack
disables them by default. (Can be enabled via GODEBUG=tlsrsakex=1).
fixes https://github.com/minio/minio/issues/20214
With a custom list of TLS ciphers, Go can pick the TLS RSA key-exchange
cipher. Ref:
```
if c.CipherSuites != nil {
return c.CipherSuites
}
if tlsrsakex.Value() == "1" {
return defaultCipherSuitesWithRSAKex
}
```
Ref: https://cs.opensource.google/go/go/+/refs/tags/go1.22.5:src/crypto/tls/common.go;l=1017
Signed-off-by: Andreas Auernhammer <github@aead.dev>
this allows for de-duplicating the callers when called
concurrently, allowing for bucketmetadata reads to be
single call. All concurrent callers will get the same data
as the first one.
Fix a regression in #19733 where TTFB metrics for all APIs except
GetObject were removed in v2 and v3 metrics. This causes breakage for
existing v2 metrics users. Instead we continue to send TTFB for all APIs
in V2 but only send for GetObject in V3.
allow multipart uploads expiration to be dyamic
It would seem like the new values will take effect
only after a restart for changes in multipart_expiration.
This PR fixes this by making it dynamic as it should have
been.
deadlines per moveToTrash() allows for a more granular timeout
approach for syscalls, instead of an aggregate timeout.
This PR also enhances multipart state cleanup to be optimal by
removing 100's of multipart network rename() calls into single
network call.
context deadline was introduced to avoid a slow transfer from blocking
replication queue(s) shared by other buckets that may not be under throttling.
This PR removes this context deadline for larger objects since they are
anyway restricted to a limited set of workers. Otherwise, objects would
get dequeued when the throttle limit is exceeded and cannot proceed
within the deadline.
epoll contention on TCP causes latency build-up when
we have high volume ingress. This PR is an attempt to
relieve this pressure.
upstream issue https://github.com/golang/go/issues/65064
It seems to be a deeper problem; haven't yet tried the fix
provide in this issue, but however this change without
changing the compiler helps.
Of course, this is a workaround for now, hoping for a
more comprehensive fix from Go runtime.
- ReadVersion
- ReadFile
- ReadXL
Further changes include to
- Compact internode resource RPC paths
- Compact internode query params
To optimize on parsing by gorilla/mux as the
length of this string increases latency in
gorilla/mux - reduce to a meaningful string.
the main reason is to let Go net/http perform necessary
book keeping properly, and in essential from consistency
point of view its GETs all the way.
Deprecate sendFile() as its buggy inside Go runtime.
For a non-tiered object, MinIO requires that EcM (# of data blocks) of
xl.meta agree, corresponding to the number of data blocks needed to
read this object.
OTOH, tiered objects have metadata in the hot tier and data in the
warm tier. The data and its integrity are offloaded to the warm tier. This
allows us to reduce the read quorum from EcM (typically > N/2, where N -
erasure stripe width) to N/2 + 1. The simple majority of metadata
ensures consensus on what the object is and where it is
located.
When a drive is in a failed state when a single node multiple drives
deployment is started, a replacement of a fresh disk will not be
properly healed unless the user restarts the node.
Fix this by always adding the new fresh disk to globalLocalDrivesMap. Also
remove globalLocalDrives for simplification, a map to store local node
drives can still be used since the order of local drives of a node is
not defined.
kms: Expose API available when bucket federation is enabled
When bucket federation feature is enabled, KMS API will not work, such
as `mc admin kms key list`
The commit will fix the issue by disabling bucket forwarding when this
is a KMS request.
Tracing syscalls, opening and reading an `xl.meta` looks like this:
```
openat(AT_FDCWD, "/mnt/drive1/ss8-old/testbucket/ObjSize4MiBThreads72/(554O51H/peTb(0iztdbTKw59.csv/xl.meta", O_RDONLY|O_NOATIME|O_CLOEXEC) = 34 <0.000>
fcntl(34, F_GETFL) = 0x48000 (flags O_RDONLY|O_LARGEFILE|O_NOATIME) <0.000>
fcntl(34, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOATIME) = 0 <0.000>
epoll_ctl(4, EPOLL_CTL_ADD, 34, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=3172471557, u64=8145488475984499461}}) = -1 EPERM (Operation not permitted) <0.000>
fcntl(34, F_GETFL) = 0x48800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOATIME) <0.000>
fcntl(34, F_SETFL, O_RDONLY|O_LARGEFILE|O_NOATIME) = 0 <0.000>
fstat(34, {st_mode=S_IFREG|0644, st_size=354, ...}) = 0 <0.000>
read(34, "XL2 \1\0\3\0\306\0\0\1P\2\2\1\304$\225\304\20\0\0\0\0\0\0\0\0\0\0\0"..., 354) = 354 <0.000>
close(34) = 0 <0.000>
```
Everything until `fstat` is the `os.Open` call.
Looking at the code: https://github.com/golang/go/blob/master/src/os/file_unix.go#L212-L243
It seems for every file it "tries" to see if it is pollable. This causes `syscall.SetNonblock(fd, true)` to be called. This is the first `F_SETFL`.
It then calls `f.pfd.Init("file", true)`. This will attempt to set it as pollable using `epoll_ctl`. This will always fail for files. It therefore calls `syscall.SetNonblock(fd, false)` resulting in the second `F_SETFL`.
If we set the `O_NONBLOCK` call on the initial open, we should avoid the 4 `fcntl` syscalls per file.
I don't see any way to avoid the `epoll_ctl` call, since kind is either `kindOpenFile` or `kindNonBlock`, so "pollable" will always be true. However avoiding 4 of 6 syscalls still seems worth it.
This should not have any effect, since files will end up with "nonblock" anyway.
allow non-inlined on disk to be inlined via
an unversioned ReadVersion() call, we only
need ReadXL() to resolve objects with multiple
versions only.
The choice of this block makes it to be dynamic
and chosen by the user via `mc admin config set`
Other bonus things
- Start measuring internode TTFB performance.
- Set TCP_NODELAY, TCP_CORK for low latency
Use `runtime.Gosched()` if we have less than maxMergeMessages and the
queue is empty. Up maxMergeMessages to 50 to merge more messages into
a single write.
Add length check for an early bailout on readAllInto when we know packet length.
This commit enforces FIPS-compliant TLS ciphers in FIPS mode
by importing the `fipsonly` module.
Otherwise, MinIO still accepts non-FIPS compliant TLS connections.
removes contentious usage of mutexes in LRU, which
were never really reused in any manner; we do not
need it.
To trust hosts, the correct way is TLS certs; this PR completely
removes this dependency, which has never been useful.
```
0 0% 100% 25.83s 26.76% github.com/hashicorp/golang-lru/v2/expirable.(*LRU[...])
0 0% 100% 28.03s 29.04% github.com/hashicorp/golang-lru/v2/expirable.(*LRU[...])
```
Bonus: use `x-minio-time` as a nanosecond to avoid unnecessary
parsing logic of time strings instead of using a more
straightforward mechanism.
- Also, fix failure reporting at the end.
- Also, avoid parsing report objects when listing or resuming jobs, this
does not cause any bugs, it is only printing, not useful errors.
Split the read and write sides of handleMessages into two separate functions
Cosmetic. The only non-copy-and-paste change is that `cancel(ErrDisconnected)` is moved
into the defer on `readStream`.
add unit test for v3 metrics for all its exposed endpoints
Bonus:
- support OpenMetrics encoding
- adds boot time for prometheus
- continueOnError is better to serve as
much metrics as possible.
```
curl http://localhost:9000/minio/metrics/v3/cluster/usage/buckets
```
Did not work as documented, due to the fact that there was a typo
in the bucket usage metrics registration group. This endpoint is
a cluster endpoint and does not require any `buckets` argument.
When creating the async listing, if the first request does not return within 3
minutes, it is stopped, since it isn't being kept alive.
Keep updating `lastHandout` while we are waiting for the initial request to be fulfilled.
When a fresh drive healing is finished, add more checks for the drive listing
errors. If any, re-list and heal again. Although this is an infrequent use
case to have listPathRaw() returning nil when minDisks is set to 1, we
still need to handle all possible use cases to avoid missing healing
any object.
Also, check for HealObject result to decide of an object is healed in the
fresh disk since HealObject returns nil if an object is healed in any
disk, and not in the new fresh drive.
In regular listing, this commit will avoid showing an object when its
latest version has a pending or failed deletion. In replicated setup.
It will also prevent showing older versions in the same case.
Also, sort the error map for multiple sites in ascending order
of deployment IDs, so that the error message generated is always
definitive order and same.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Add `ConnDialer` to abstract connection creation.
- `IncomingConn(ctx context.Context, conn net.Conn)` is provided as an entry point for
incoming custom connections.
- `ConnectWS` is provided to create web socket connections.
If `SkipReader` is called with a small initial buffer it may be doing a huge number if Reads to skip the requested number of bytes. If a small buffer is provided grab a 32K buffer and use that.
Fixes slow execution of `testAPIGetObjectWithMPHandler`.
Bonuses:
* Use `-short` with `-race` test.
* Do all suite test types with `-short`.
* Enable compressed+encrypted in `testAPIGetObjectWithMPHandler`.
* Disable big file tests in `testAPIGetObjectWithMPHandler` when using `-short`.
The code was advertenly passing max openfds to debug.SetMemoryLimit(),
fixing this accelerate go test in my machine.
This is only a testing bug, since the server context has always a valid
MaxMem, so the buggy code was never called in users environments.
Currently the status of a completed or failed batch is held in the
memory, a simple restart will lose the status and the user will not
have any visibility of the job that was long running.
In addition to the metrics, add a new API that reads the batch status
from the drives. A batch job will be cleaned up three days after
completion.
Also add the batch type in the batch id, the reason is that the batch
job request is removed immediately when the job is finished, then we
do not know the type of batch job anymore, hence a difficulty to locate
the job report
Hot load a policy document when during account authorization evaluation
to avoid returning 403 during server startup, when not all policies are
already loaded.
Add this support for group policies as well.
you can attempt a rebalance first i.e, start with 2 pools.
```
mc admin rebalance start alias/
```
and after that you can add a new pool, this would
potentially crash.
```
Jun 27 09:22:19 xxx minio[7828]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 27 09:22:19 xxx minio[7828]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x22cc225]
Jun 27 09:22:19 xxx minio[7828]: goroutine 1 [running]:
Jun 27 09:22:19 xxx minio[7828]: github.com/minio/minio/cmd.(*erasureServerPools).findIndex(...)
```
without atomic load() it is possible that for
a slow receiver we would get into a hot-loop, when
logCh is full and there are many incoming callers.
to avoid this as a workaround enable BATCH_SIZE
greater than 100 to ensure that your slow receiver
receives data in bulk to avoid being throttled in
some manner.
this PR however fixes the unprotected access to
the current workers value.
specifying customer HTTP client makes the gcs SDK
ignore the passed credentials, instead let the GCS
SDK manage the transport.
this PR fixes#19922 a regression from #19565
Currently, bucket metadata is being loaded serially inside ListBuckets
Objet API. Fix that by loading the bucket metadata as the number of
erasure sets * 10, which is a good approximation.
When a reconnection happens, `handleMessages` must be able to complete and exit.
This can be prevented in a full queue.
Deadlock chain (May 10th release)
```
1 @ 0x44110e 0x453125 0x109f88c 0x109f7d5 0x10a472c 0x10a3f72 0x10a34ed 0x4795e1
# 0x109f88b github.com/minio/minio/internal/grid.(*Connection).send+0x3eb github.com/minio/minio/internal/grid/connection.go:548
# 0x109f7d4 github.com/minio/minio/internal/grid.(*Connection).queueMsg+0x334 github.com/minio/minio/internal/grid/connection.go:586
# 0x10a472b github.com/minio/minio/internal/grid.(*Connection).handleAckMux+0xab github.com/minio/minio/internal/grid/connection.go:1284
# 0x10a3f71 github.com/minio/minio/internal/grid.(*Connection).handleMsg+0x231 github.com/minio/minio/internal/grid/connection.go:1211
# 0x10a34ec github.com/minio/minio/internal/grid.(*Connection).handleMessages.func1+0x6cc github.com/minio/minio/internal/grid/connection.go:1019
---> blocks ---> via (Connection).handleMsgWg
1 @ 0x44110e 0x454165 0x454134 0x475325 0x486b08 0x10a161a 0x10a1465 0x2470e67 0x7395a9 0x20e61af 0x20e5f1f 0x7395a9 0x22f781c 0x7395a9 0x22f89a5 0x7395a9 0x22f6e82 0x7395a9 0x22f49a2 0x7395a9 0x2206e45 0x7395a9 0x22f4d9c 0x7395a9 0x210ba06 0x7395a9 0x23089c2 0x7395a9 0x22f86e9 0x7395a9 0xd42582 0x2106c04
# 0x475324 sync.runtime_Semacquire+0x24 runtime/sema.go:62
# 0x486b07 sync.(*WaitGroup).Wait+0x47 sync/waitgroup.go:116
# 0x10a1619 github.com/minio/minio/internal/grid.(*Connection).reconnected+0xb9 github.com/minio/minio/internal/grid/connection.go:857
# 0x10a1464 github.com/minio/minio/internal/grid.(*Connection).handleIncoming+0x384 github.com/minio/minio/internal/grid/connection.go:825
```
Add a queue cleaner in reconnected that will pop old messages so `handleMessages` can
send messages without blocking and exit appropriately for the connection to be re-established.
Messages are likely dropped by the remote, but we may have some that can succeed,
so we only drop when running out of space.
Add the inlined data as base64 encoded field and try to add a string version if feasible.
Example:
```
λ xl-meta -data xl.meta
{
"8e03504e-1123-4957-b272-7bc53eda0d55": {
"bitrot_valid": true,
"bytes": 58,
"data_base64": "Z29sYW5nLm9yZy94L3N5cyB2MC4xNS4wIC8=",
"data_string": "golang.org/x/sys v0.15.0 /"
}
```
The string will have quotes, newlines escaped to produce valid JSON.
If content isn't valid utf8 or the encoding otherwise fails, only the base64 data will be added.
`-export` can still be used separately to extract the data as files (including bitrot).
* Prevents blocking when losing quorum (standard on cluster restarts).
* Time out to prevent endless buildup. Timed-out remote locks will be canceled because they miss the refresh anyway.
* Reduces latency for all calls since the wall time for the roundtrip to remotes no longer adds to the requests.
since the connection is active, the
response recorder body can grow endlessly
causing leak, as this bytes buffer is
never given back to GC due to an goroutine.
skip healing properly in scanner when drive is hotplugged
due to how the state is passed around the SkipHealing
might not be the true state() of the system always, causing
a situation where we might healing from the scanner on the
same drive which is being. Due to this competing heals get
triggered that slow each other down.
due to a historic bug in CopyObject() where
an inlined object loses its metadata, the
check causes an incorrect fallback verifying
data-dir.
CopyObject() bug was fixed in ffa91f9794 however
the occurrence of this problem is historic, so
the aforementioned check is stretching too much.
Bonus: simplify fileInfoRaw() to read xl.json as well,
also recreate buckets properly.
* Multipart SSEC checksums were not transferred.
* Remove key mismatch logging. This key is user-controlled with SSEC.
* If the source is SSEC and the destination reports ErrSSEEncryptedObject,
assume replication is good.
avoid concurrent callers for LoadUser() to even initiate
object read() requests, if an on-going operation is in progress.
this avoids many callers hitting the drives causing I/O
spikes, also allows for loading credentials faster.
This commit fixes an issue in the KES client configuration
that can cause the following error when connecting to KES:
```
ERROR Failed to connect to KMS: failed to generate data key with KMS key: tls: client certificate is required
```
The Go TLS stack seems to not send a client certificate if it
thinks the client certificate cannot be validated by the peer.
In case of an API key, we don't care about this since we use
public key pinning and the X.509 certificate is just a transport
encoding.
The `GetClientCertificate` seems to be honored always such that
this error does not occur.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
the reason for this is to avoid STS mappings to be
purged without a successful load of other policies,
and all the credentials only loaded successfully
are properly handled.
This also avoids unnecessary cache store which was
implemented earlier for optimization.
Directory objects are used by applications that simulate the folder
structure of an on-disk filesystem. These are zero-byte objects with names
ending with '/'. They are only used to check whether a 'folder' exists in
the namespace.
StartSize starts with the raw free space of all disks in the given pool,
however during the status, CurrentSize is not showing the current free
raw space, as expected at least by `mc admin decom status` since it was
written.
Go's net/http is notoriously difficult to have a streaming
deadlines per READ/WRITE on the net.Conn if we add them they
interfere with the Go's internal requirements for a HTTP
connection.
Remove this support for now
fixes#19853
Add partial shard reconstruction
* Add partial shard reconstruction
* Fix padding causing the last shard to be rejected
* Add md5 checks on single parts
* Move md5 verified to `verified/filename.ext`
* Move complete (without md5) to `complete/filename.ext.partno`
It's not pretty, but at least now the md5 gives some confidence it works correctly.
In the very rare case when all drives in a erasure set need to be healed,
remove .healing.bin from all drives, otherwise it will be stuck in a
loop
Also, fix a unit test that fails sometimes due to wrong test.
since #19688 there was a regression introduced during drive
lookups for single node multi-drive setups, drive replacement
would not work correctly without this PR.
This does not fix any current issue, but merging https://github.com/minio/madmin-go/pull/282
can lose the validation of the service account expiration time.
Add more defensive code for now. In the future, we should avoid doing
validation in another library.
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.
```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```
although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
fix: authenticate LDAP via actual DN instead of normalized DN
Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.
fixes#19757
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
Add combination of multiple parts.
Parts will be reconstructed and saved separately and can manually be combined to the complete object.
Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
Do not log errors on oneway streams when sending ping fails. Instead, cancel the stream.
This also makes sure pings are sent when blocked on sending responses.
This commit will fix one rare case of a multipart object that
can be read in theory but GetObject API returned an error.
It turned out that a six years old code was marking a drive offline
when the bitrot streaming fails to read a part in a disk with any error.
This can affect reading a subsequent part, though having enough shards,
but unable to construct because one drive was marked offline earlier.
This commit will remove the drive marking offline code. It will also
close the bitrotstreaming reader before marking it as nil.
Adds `-xver` which can be used with `-export` and `-combine` to attempt to combine files across versions if data is suspected to be the same. Overlapping data is compared.
Bonus: Make `inspect` accept wildcards.
Currently, on enabling callhome (or restarting the server), the callhome
job gets scheduled. This means that one has to wait for 24hrs (the
default frequency duration) to see it in action and to figure out if it
is working as expected.
It will be a better user experience to perform the first callhome
execution immediately after enabling it (or on server start if already
enabled).
Also, generate audit event on callhome execution, setting the error
field in case the execution has failed.
* Store ModTime in the upload ID; return it when listing instead of the current time.
* Use this ModTime to expire and skip reading the file info.
* Consistent upload sorting in listing (since it now has the ModTime).
* Exclude healing disks to avoid returning an empty list.
```
==================
WARNING: DATA RACE
Read at 0x0000082be990 by goroutine 205:
github.com/minio/minio/cmd.setCommonHeaders()
Previous write at 0x0000082be990 by main goroutine:
github.com/minio/minio/cmd.lookupConfigs()
```
Recent Veeam is very picky about storage class names. Add `_MINIO_VEEAM_FORCE_SC` env var.
It will override the storage class returned by the storage backend if it is non-standard
and we detect a Veeam client by checking the User Agent.
Applies to HeadObject/GetObject/ListObject*
add deadlines that can be dynamically changed via
the drive max timeout values.
Bonus: optimize "file not found" case and hung drives/network - circuit break the check and return right
away instead of waiting.
Do not log errors on oneway streams when sending ping fails. Instead cancel the stream.
This also makes sure pings are sent when blocked on sending responses.
I will do a separate PR that includes this and adds pings to two-way streams as well as tests for pings.
as that is the only API where the TTFB metric is beneficial, and
capturing this for all APIs exponentially increases the response size in
large clusters.
Replace the `io.Pipe` from streamingBitrotWriter -> CreateFile with a fixed size ring buffer.
This will add an output buffer for encoded shards to be written to disk - potentially via RPC.
This will remove blocking when `(*streamingBitrotWriter).Write` is called, and it writes hashes and data.
With current settings, the write looks like this:
```
Outbound
┌───────────────────┐ ┌────────────────┐ ┌───────────────┐ ┌────────────────┐
│ │ Parr. │ │ (http body) │ │ │ │
│ Bitrot Hash │ Write │ Pipe │ Read │ HTTP buffer │ Write (syscall) │ TCP Buffer │
│ Erasure Shard │ ──────────► │ (unbuffered) │ ────────────► │ (64K Max) │ ───────────────────► │ (4MB) │
│ │ │ │ │ (io.Copy) │ │ │
└───────────────────┘ └────────────────┘ └───────────────┘ └────────────────┘
```
We write a Hash (32 bytes). Since the pipe is unbuffered, it will block until the 32 bytes have
been delivered to the TCP buffer, and the next Read hits the Pipe.
Then we write the shard data. This will typically be bigger than 64KB, so it will block until two blocks
have been read from the pipe.
When we insert a ring buffer:
```
Outbound
┌───────────────────┐ ┌────────────────┐ ┌───────────────┐ ┌────────────────┐
│ │ │ │ (http body) │ │ │ │
│ Bitrot Hash │ Write │ Ring Buffer │ Read │ HTTP buffer │ Write (syscall) │ TCP Buffer │
│ Erasure Shard │ ──────────► │ (2MB) │ ────────────► │ (64K Max) │ ───────────────────► │ (4MB) │
│ │ │ │ │ (io.Copy) │ │ │
└───────────────────┘ └────────────────┘ └───────────────┘ └────────────────┘
```
The hash+shard will fit within the ring buffer, so writes will not block - but will complete after a
memcopy. Reads can fill the 64KB buffer if there is data for it.
If the network is congested, the ring buffer will become filled, and all syscalls will be on full buffers.
Only when the ring buffer is filled will erasure coding start blocking.
Since there is always "space" to write output data, we remove the parallel writing since we are
always writing to memory now, and the goroutine synchronization overhead probably not worth taking.
If the output were blocked in the existing, we would still wait for it to unblock in parallel write, so it would
make no difference there - except now the ring buffer smoothes out the load.
There are some micro-optimizations we could look at later. The biggest is that, in most cases,
we could encode directly to the ring buffer - if we are not at a boundary. Also, "force filling" the
Read requests (i.e., blocking until a full read can be completed) could be investigated and maybe
allow concurrent memory on read and write.
Metrics being added:
- read_tolerance: No of drive failures that can be tolerated without
disrupting read operations
- write_tolerance: No of drive failures that can be tolerated without
disrupting write operations
- read_health: Health of the erasure set in a pool for read operations
(1=healthy, 0=unhealthy)
- write_health: Health of the erasure set in a pool for write operations
(1=healthy, 0=unhealthy)
Adds regression test for #19699
Failures are a bit luck based, since it requires objects to be placed on different sets.
However this generates a failure prior to #19699
* Revert "Revert "Fix incorrect merging of slash-suffixed objects (#19699)""
This reverts commit f30417d9a8.
* Don't override when suffix doesn't match. Instead rely on quorum for each.
Instead of having "online" and "healing" as two metrics, replace with a
single metric "health" which can have following values:
0 = offline
1 = healthy
2 = healing
If two objects share everything but one object has a slash prefix, those would be merged in listings,
with secondary properties used for a tiebreak.
Example: An object with the key `prefix/obj` would be merged with an object named `prefix/obj/`.
While this violates the [no object can be a prefix of another](https://min.io/docs/minio/linux/operations/concepts/thresholds.html#conflicting-objects), let's resolve these.
If we have an object with 'name' and a directory named 'name/' discard the directory only - but allow objects
of 'name' and 'name/' (xldir) to be uniquely returned.
Regression from #15772
canceled callers might linger around longer,
can potentially overwhelm the system. Instead
provider a caller context and canceled callers
don't hold on to them.
Bonus: we have no reason to cache errors, we should
never cache errors otherwise we can potentially have
quorum errors creeping in unexpectedly. We should
let the cache when invalidating hit the actual resources
instead.
LastPong is saved as nanoseconds after a connection or reconnection but
saved as seconds when receiving a pong message. The code deciding if
a pong is too old can be skewed since it assumes LastPong is only in
seconds.
Accept multipart uploads where the combined checksum provides the expected part count.
It seems this was added by AWS to make the API more consistent, even if the
data is entirely superfluous on multiple levels.
Improves AWS S3 compatibility.
This commit adds support for MinKMS. Now, there are three KMS
implementations in `internal/kms`: Builtin, MinIO KES and MinIO KMS.
Adding another KMS integration required some cleanup. In particular:
- Various KMS APIs that haven't been and are not used have been
removed. A lot of the code was broken anyway.
- Metrics are now monitored by the `kms.KMS` itself. For basic
metrics this is simpler than collecting metrics for external
servers. In particular, each KES server returns its own metrics
and no cluster-level view.
- The builtin KMS now uses the same en/decryption implemented by
MinKMS and KES. It still supports decryption of the previous
ciphertext format. It's backwards compatible.
- Data encryption keys now include a master key version since MinKMS
supports multiple versions (~4 billion in total and 10000 concurrent)
per key name.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
If used, 'opts.Marker` will cause many missed entries since results are returned
unsorted, and pools are serialized.
Switch to fully concurrent listing and merging across pools to return sorted entries.
It is expected that whoever is using the credentials which has
the proper set of permissions must be able to run.
`mc support perf object`
While the root login is disabled.
fixes#19648
AWS S3 returns the actual object size as part of XML
response for InvalidRange error, this is used apparently
by SDKs to retry the request without the range.
'opts.Marker` is causing many missed entries if used since results are returned unsorted. Also since pools are serialized.
Switch to do fully concurrent listing and merging across pools to return sorted entries.
Returning errors on listings is impossible with the current API, so document that.
Return an error at once if no drives are found instead of just returning an empty listing and no error.
This is to support deployments migrating from a multi-pooled
wider stripe to lower stripe. MINIO_STORAGE_CLASS_STANDARD
is still expected to be same for all pools. So you can satisfy
adding custom drive count based pools by adjusting the storage
class value.
```
version: v2
address: ':9000'
rootUser: 'minioadmin'
rootPassword: 'minioadmin'
console-address: ':9001'
pools: # Specify the nodes and drives with pools
-
args:
- 'node{11...14}.example.net/data{1...4}'
-
args:
- 'node{15...18}.example.net/data{1...4}'
-
args:
- 'node{19...22}.example.net/data{1...4}'
-
args:
- 'node{23...34}.example.net/data{1...10}'
set-drive-count: 6
```
ILM actions due to ExpiredObjectDeleteAllVersions and
DelMarkerExpiration are ignored when object locking is enabled on a
bucket.
Note: This applies to object versions which may not have retention
configured on them. This applies to all object versions in this bucket,
including those created before the retention config was applied.
Per-bucket metrics endpoints always start with /bucket and the bucket
name is appended to the path. e.g. if the collector path is /bucket/api,
the endpoint for the bucket "mybucket" would be
/minio/metrics/v3/bucket/api/mybucket
Change the existing bucket api endpoint accordingly from /api/bucket to
/bucket/api
The `Token` parameter is a sensitive value that should not be output in the Audit log for STS AssumeRoleWithCustomToken API.
Bonus: Add a simple tool that echoes audit logs to the console.
When listing, with drives returning `errFileNotFound,` `errVolumeNotFound`, or `errUnformattedDisk,`,
we could get below `minDisks` drives being left.
This would result in a quorum never being reachable for any object. Therefore, the listing
would continue, but no results would ever be produced.
Include `fnf` in the mindisk check since it is incremented on these errors. This will stop
listing when minDisks are left.
Allow `opts.minDisks` to not return errVolumeNotFound or errFileNotFound and return that.
That will allow for good results even if disks return something else.
We switch `errUnformattedDisk` to a regular error. If we have enough of those, we should just fail.
Typically not all drives are connected, so we delay 3 minutes before resuming.
This greatly reduces risk of starting to list unconnected drives, or drives we risk being disconnected soon.
This delay is not applied when starting with an admin call.
ConsoleUI like applications rely on combination of
ListServiceAccounts() and InfoServiceAccount() to populate
UI elements, however individually these calls can be slow
causing the entire UI to load sluggishly.
i.e., this rule element doesn't apply to DEL markers.
This is a breaking change to how ExpiredObejctDeleteAllVersions
functions today. This is necessary to avoid the following highly probable
footgun scenario in the future.
Scenario:
The user uses tags-based filtering to select an object's time to live(TTL).
The application sometimes deletes objects, too, making its latest
version a DEL marker. The previous implementation skipped tag-based filters
if the newest version was DEL marker, voiding the tag-based TTL. The user is
surprised to find objects that have expired sooner than expected.
* Add DelMarkerExpiration action
This ILM action removes all versions of an object if its
the latest version is a DEL marker.
```xml
<DelMarkerObjectExpiration>
<Days> 10 </Days>
</DelMarkerObjectExpiration>
```
1. Applies only to objects whose,
• The latest version is a DEL marker.
• satisfies the number of days criteria
2. Deletes all versions of this object
3. Associated rule can't have tag-based filtering
Includes,
- New bucket event type for deletion due to DelMarkerExpiration
calling a remote target remove with a perfectly
well constructed ARN can lead to a crash for a bucket
with no replication configured.
This PR fixes, and adds a crash check for ImportMetadata
as well.
Algorithms are comma separated.
Note that valid values does not in all cases represent default values.
`--sftp=pub-key-algos=...` specifies the supported client public key
authentication algorithms. Note that this doesn't include certificate types
since those use the underlying algorithm. This list is sent to the client if
it supports the server-sig-algs extension. Order is irrelevant.
Valid values
```
ssh-ed25519
sk-ssh-ed25519@openssh.comsk-ecdsa-sha2-nistp256@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
rsa-sha2-256
rsa-sha2-512
ssh-rsa
ssh-dss
```
`--sftp=kex-algos=...` specifies the supported key-exchange algorithms in preference order.
Valid values:
```
curve25519-sha256
curve25519-sha256@libssh.org
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group14-sha256
diffie-hellman-group16-sha512
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
```
`--sftp=cipher-algos=...` specifies the allowed cipher algorithms.
If unspecified then a sensible default is used.
Valid values:
```
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.comaes256-gcm@openssh.comchacha20-poly1305@openssh.com
arcfour256
arcfour128
arcfour
aes128-cbc
3des-cbc
```
`--sftp=mac-algos=...` specifies a default set of MAC algorithms in preference order.
This is based on RFC 4253, section 6.4, but with hmac-md5 variants removed because they have
reached the end of their useful life.
Valid values:
```
hmac-sha2-256-etm@openssh.comhmac-sha2-512-etm@openssh.com
hmac-sha2-256
hmac-sha2-512
hmac-sha1
hmac-sha1-96
```
This would reduce the size of data in response of metrics
listing. While graphing we can default these metrics with
a zero value if not found.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Unfreeze as soon as the incoming connection is terminated and don't wait for everything to complete.
We don't want to keep the services frozen if something becomes stuck.
- handle errFileCorrupt properly
- micro-optimization of sending done() response quicker
to close the goroutine.
- fix logger.Event() usage in a couple of places
- handle the rest of the client to return a different error other than
lastErr() when the client is closed.
listPathRaw() counts errDiskNotFound as a valid error to indicate a
listing stream end. However, storage.WalkDir() is allowed to return
errDiskNotFound anytime since grid.ErrDisconnected is converted to
errDiskNotFound.
This affects fresh disk healing and should affect S3 listing as well.
endpoint: /minio/metrics/v3/system/process
metrics:
- locks_read_total
- locks_write_total
- cpu_total_seconds
- go_routine_total
- io_rchar_bytes
- io_read_bytes
- io_wchar_bytes
- io_write_bytes
- start_time_seconds
- uptime_seconds
- file_descriptor_limit_total
- file_descriptor_open_total
- syscall_read_total
- syscall_write_total
- resident_memory_bytes
- virtual_memory_bytes
- virtual_memory_max_bytes
Since the standard process collector implements only a subset of these
metrics, remove it and implement our own custom process collector that
captures all the process metrics we need.
Since the object is being permanently deleted, the lack of read quorum should not
matter as long as sufficient disks are online to complete the deletion with parity
requirements.
If several pools have the same object with insufficient read quorum, attempt to
delete object from all the pools where it exists
At server startup, LDAP configuration is validated against the LDAP
server. If the LDAP server is down at that point, we need to cleanly
disable LDAP configuration. Previously, LDAP would remain configured but
error out in strange ways because initialization did not complete
without errors.
When importing access keys (i.e. service accounts) for LDAP accounts,
we are requiring groups to exist under one of the configured group base
DNs. This is not correct. This change fixes this by only checking for
existence and storing the normalized form of the group DN - we do not
return an error if the group is not under a base DN.
Test is updated to illustrate an import failure that would happen
without this change.
Existing IAM import logic for LDAP creates new mappings when the
normalized form of the mapping key differs from the existing mapping key
in storage. This change effectively replaces the existing mapping key by
first deleting it and then recreating with the normalized form of the
mapping key.
For e.g. if an older deployment had a policy mapped to a user DN -
`UID=alice1,OU=people,OU=hwengg,DC=min,DC=io`
instead of adding a mapping for the normalized form -
`uid=alice1,ou=people,ou=hwengg,dc=min,dc=io`
we should replace the existing mapping.
This ensures that duplicates mappings won't remain after the import.
Some additional cleanup cases are also covered. If there are multiple
mappings for the name normalized key such as:
`UID=alice1,OU=people,OU=hwengg,DC=min,DC=io`
`uid=alice1,ou=people,ou=hwengg,DC=min,DC=io`
`uid=alice1,ou=people,ou=hwengg,dc=min,dc=io`
we check if the list of policies mapped to all these keys are exactly
the same, and if so remove all of them and create a single mapping with
the normalized key. However, if the policies mapped to such keys differ,
the import operation returns an error as the server cannot automatically
pick the "right" list of policies to map.
When inspecting files like `.minio.sys/pool.bin` that may be present on multiple sets, use signature to separate them.
Also fixes null versions to actually be useful with `-export -combine`.
`minio_cluster_webhook_queue_length` was wrongly defined as `counter`
where-as it should be `gauge`
Following were wrongly defined as `gauge` when they should actually be
`counter`:
- minio_bucket_replication_sent_bytes
- minio_bucket_replication_received_bytes
- minio_bucket_replication_total_failed_bytes
- minio_bucket_replication_total_failed_count
When LDAP is enabled, previously we were:
- rejecting creation of users and groups via the IAM import functionality
- throwing a `not a valid DN` error when non-LDAP group mappings are present
This change allows for these cases as we need to support situations
where the MinIO server contains users, groups and policy mappings
created before LDAP was enabled.
instead upon any error in renameData(), we still
preserve the existing dataDir in some form for
recoverability in strange situations such as out
of disk space type errors.
Bonus: avoid running list and heal() instead allow
versions disparity to return the actual versions,
uuid to heal. Currently limit this to 100 versions
and lesser disparate objects.
an undo now reverts back the xl.meta from xl.meta.bkp
during overwrites on such flaky setups.
Bonus: Save N depth syscalls via skipping the parents
upon overwrites and versioned updates.
Flaky setup examples are stretch clusters with regular
packet drops etc, we need to add some defensive code
around to avoid dangling objects.
RenameData could start operating on inline data after timing out
and the call returned due to WithDeadline.
This could cause a buffer to write to the inline data being written.
Since no writes are in `RenameData` and the call is canceled,
this doesn't present a corruption issue. But a race is a race and
should be fixed.
Copy inline data to a fresh buffer.
This PR makes a feasible approach to handle all the scenarios
that we must face to avoid returning "panic."
Instead, we must return "errServerNotInitialized" when a
bucketMetadataSys.Get() is called, allowing the caller to
retry their operation and wait.
Bonus fix the way data-usage-cache stores the object.
Instead of storing usage-cache.bin with the bucket as
`.minio.sys/buckets`, the `buckets` must be relative
to the bucket `.minio.sys` as part of the object name.
Otherwise, there is no way to decommission entries at
`.minio.sys/buckets` and their final erasure set positions.
A bucket must never have a `/` in it. Adds code to read()
from existing data-usage.bin upon upgrade.
This PR fixes a few things
- FIPS support for missing for remote transports, causing
MinIO could end up using non-FIPS Ciphers in FIPS mode
- Avoids too many transports, they all do the same thing
to make connection pooling work properly re-use them.
- globalTCPOptions must be set before setting transport
to make sure the client conn deadlines are honored properly.
- GCS warm tier must re-use our transport
- Re-enable trailing headers support.
This reverts commit 928c0181bf.
This change was not correct, reverting.
We track 3 states with the ProxyRequest header - if replication process wants
to know if object is already replicated with a HEAD, it shouldn't proxy back
- Poorna
AWS S3 trailing header support was recently enabled on the warm tier
client connection to MinIO type remote tiers. With this enabled, we are
seeing the following error message at http transport layer.
> Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request"; err=<nil>
This is an interim fix until we identify the root cause for this behaviour in the
minio-go client package.
Keep the EC in header, so it can be retrieved easily for dynamic quorum calculations.
To not force a full metadata decode on every read the value will be 0/0 for data written in previous versions.
Size is expected to increase by 2 bytes per version, since all valid values can be represented with 1 byte each.
Example:
```
λ xl-meta xl.meta
{
"Versions": [
{
"Header": {
"EcM": 4,
"EcN": 8,
"Flags": 6,
"ModTime": "2024-04-17T11:46:25.325613+02:00",
"Signature": "0a409875",
"Type": 1,
"VersionID": "8e03504e11234957b2727bc53eda0d55"
},
...
```
Not used for operations yet.
Follow up for #19528
If there are multiple existing DN mappings for the same normalized DN,
if they all have the same policy mapping value, we pick one of them of
them instead of returning an import error.
This is a change to IAM export/import functionality. For LDAP enabled
setups, it performs additional validations:
- for policy mappings on LDAP users and groups, it ensures that the
corresponding user or group DN exists and if so uses a normalized form
of these DNs for storage
- for access keys (service accounts), it updates (i.e. validates
existence and normalizes) the internally stored parent user DN and group
DNs.
This allows for a migration path for setups in which LDAP mappings have
been stored in previous versions of the server, where the name of the
mapping file stored on drives is not in a normalized form.
An administrator needs to execute:
`mc admin iam export ALIAS`
followed by
`mc admin iam import ALIAS /path/to/export/file`
The validations are more strict and returns errors when multiple
mappings are found for the same user/group DN. This is to ensure the
mappings stored by the server are unambiguous and to reduce the
potential for confusion.
Bonus **bug fix**: IAM export of access keys (service accounts) did not
export key name, description and expiration. This is fixed in this
change too.
Reading the list metacache is not protected by a lock; the code retries when it fails
to read the metacache object, however, it forgot to re-read the metacache object
from the drives, which is necessary, especially if the metacache object is inlined.
This commit will ensure that we always re-read the metacache object from the drives
when it is retrying.
When resuming a versioned listing where `version-id-marker=null`, the `null` object would
always be returned, causing duplicate entries to be returned.
Add check against empty version
unlinking() at two different locations on a disk when there
are lots to purge, this can lead to huge IOwaits, instead
rely on rename() to .trash to avoid running multiple unlinks()
in parallel.
since mid 2018 we do not have any deployments
without deployment-id, it is time to put this
code to rest, this PR removes this old code as
its no longer valuable.
on setups with 1000's of drives these are all
quite expensive operations.
The rest of the peer clients were not consistent across nodes. So, meta cache requests
would not go to the same server if a continuation happens on a different node.
As node metrics should be scraped per node basis, use a sample
configuartion using all the nodes in targets.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
When no results match or another error occurs, add an error to the stream. Keep the "inspect-input.txt" as the only thing in the zip for reference.
Example:
```
λ mc support inspect --airgap myminio/testbucket/fjghfjh/**
mc: Using public key from C:\Users\klaus\mc\support_public.pem
File data successfully downloaded as inspect-data.enc
λ inspect inspect-data.enc
Using private key from support_private.pem
output written to inspect-data.zip
2024/04/11 14:10:51 next stream: GetRawData: No files matched the given pattern
λ unzip -l inspect-data.zip
Archive: inspect-data.zip
Length Date Time Name
--------- ---------- ----- ----
222 2024-04-11 14:10 inspect-input.txt
--------- -------
222 1 file
λ
```
Modifies inspect to read until end of stream to report the error.
Bonus: Add legacy commandline params
Add following metrics:
- used_inodes
- total_inodes
- healing
- online
- reads_per_sec
- reads_kb_per_sec
- reads_await
- writes_per_sec
- writes_kb_per_sec
- writes_await
- perc_util
To be able to calculate the `per_sec` values, we capture the IOStats-related
data in the beginning (along with the time at which they were captured),
and compare them against the current values subsequently. This is because
dividing by "time since server uptime." doesn't work in k8s environments.
the disk location never changes in the lifetime of a
MinIO cluster, even if it did validate this close to the
disk instead at the higher layer.
Return appropriate errors indicating an invalid drive, so
that the drive is not recognized as part of a valid
drive.
we have had numerous reports on some config
values not having default values, causing
features misbehaving and not having default
values set properly.
This PR tries to address all these concerns
once and for all.
Each new sub-system that gets added
- must check for invalid keys
- must have default values set
- must not "return err" when being saved into
a global state() instead collate as part of
other subsystem errors allow other sub-systems
to independently initialize.
* Allow specifying the local server, with env variable _MINIO_SERVER_LOCAL, in systems where the hostname cannot be resolved to local IP
* Limit scope of the _MINIO_SERVER_LOCAL solution to only containerized implementations
Return an error when the user specifies endpoints for both source
and target. This can generate many type of errors as the code considers
a deployment remote if its endpoint is specified.
HealObject() does not return an error in some cases, for example, when
an object is successfully reconstructed in one disk but fails with other
disks, another case is when a disk does not have the object is temporarily
disconnected
Add the After heal drives result in the audit output for better
analysis.
Set object's modTime when being restored
restored here refers to making a temporary local copy in the hot tier
for a tiered object using the RestoreObject API
we have been using an LRU caching for internode
auth tokens, migrate to using a typed implementation
and also do not cache auth tokens when its an error.
This fixes a regression from #19358 which prevents policy mappings
created in the latest release from being displayed in policy entity
listing APIs.
This is due to the possibility that the base DNs in the LDAP config are
not in a normalized form and #19358 introduced normalized of mapping
keys (user DNs and group DNs). When listing, we check if the policy
mappings are on entities that parse as valid DNs that are descendants of
the base DNs in the config.
Test added that demonstrates a failure without this fix.
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
Using oidc.redirectUri in the values.yaml only works for the deployment.
When using the statefulset the environment variable
MINIO_IDENTITY_OPENID_REDIRECT_URI is not set. This leads to errors with
oicd providers. For example keycloak throws the error 'invalid
redirect_uri'.
This pull request fixes that.
This commit replaces the `KMS.Stat` API call with a
`KMS.GenerateKey` call. This approach is more reliable
since data key generation also works when the KMS backend
is unavailable (temp. offline), but KES has cached the
key. Ref: KES offline caching.
With this change, it is less likely that MinIO readiness
checks fail in cases where the KMS backend is offline.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
Make sure to pass a nil pointer as a Transport to minio-go when the API config
is not initialized, this will make sure that we do not pass an interface
with a known type but a nil value.
This will also fix the update of the API remote_transport_deadline
configuration without requiring the cluster restart.
Use `ODirectPoolSmall` buffers for inline data in PutObject.
Add a separate call for inline data that will fetch a buffer for the inline data before unmarshal.
This fixes a bug where STS Accounts map accumulates accounts in memory
and never removes expired accounts and the STS Policy mappings were not
being refreshed.
The STS purge routine now runs with every IAM credentials load instead
of every 4th time.
The listing of IAM files is now cached on every IAM load operation to
prevent re-listing for STS accounts purging/reload.
Additionally this change makes each server pick a time for IAM loading
that is randomly distributed from a 10 minute interval - this is to
prevent server from thundering while performing the IAM load.
On average, IAM loading will happen between every 5-15min after the
previous IAM load operation completes.
Fix issue [minio#19314], resolve the absence of the sed command in ubi-micro by replacing it with echo.
Signed-off-by: Andreas Bräu <ab@andi95.de>
Co-authored-by: jiuker <2818723467@qq.com>
If site replication enabled across sites, replicate the SSE-C
objects as well. These objects could be read from target sites
using the same client encryption keys.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
User doesn't need to remember and enter the server values,
rather they can select from the pre populated list.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Instead of relying on user input values, we use the DN value returned by
the LDAP server.
This handles cases like when a mapping is set on a DN value
`uid=svc.algorithm,OU=swengg,DC=min,DC=io` with a user input value (with
unicode variation) of `uid=svc﹒algorithm,OU=swengg,DC=min,DC=io`. The
LDAP server on lookup of this DN returns the normalized value where the
unicode dot character `SMALL FULL STOP` (in the user input), gets
replaced with regular full stop.
Bonus: remove persistent md5sum calculation, turn-off
sha256 as well. Instead we always enable crc32c which
is enough for payload verification also support for
trailing headers checksum.
As total drives count, online vs offline are per node basis, its
corect to select node for which graphs need to be rendered.
Set prometheus scrape jobs to fetch metrics from all nodes. A sample
scrape job for node metrics could be as below
```
- job_name: minio-job-node
bearer_token: <token>
metrics_path: /minio/v2/metrics/node
scheme: https
tls_config:
insecure_skip_verify: true
static_configs:
- targets: [tenant1-ss-0-0.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-1.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-2.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-3.tenant1-hl.tenant-ns.svc.cluster.local:9000]
```
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
we must attempt to convert all errors at storage-rest-client
into StorageErr() regardless of what functionality is being
called in, this PR fixes this for multiple callers including
some internally used functions.
- old version was unable to retain messages during config reload
- old version could not go from memory to disk during reload
- new version can batch disk queue entries to single for to reduce I/O load
- error logging has been improved, previous version would miss certain errors.
- logic for spawning/despawning additional workers has been adjusted to trigger when half capacity is reached, instead of when the log queue becomes full.
- old version would json marshall x2 and unmarshal 1x for every log item. Now we only do marshal x1 and then we GetRaw from the store and send it without having to re-marshal.
panic seen due to premature closing of slow channel while listing is still sending or
list has already closed on the sender's side:
```
panic: close of closed channel
goroutine 13666 [running]:
github.com/minio/minio/internal/ioutil.SafeClose[...](0x101ff51e4?)
/Users/kp/code/src/github.com/minio/minio/internal/ioutil/ioutil.go:425 +0x24
github.com/minio/minio/cmd.(*erasureServerPools).Walk.func1()
/Users/kp/code/src/github.com/minio/minio/cmd/erasure-server-pool.go:2142 +0x170
created by github.com/minio/minio/cmd.(*erasureServerPools).Walk in goroutine 1189
/Users/kp/code/src/github.com/minio/minio/cmd/erasure-server-pool.go:1985 +0x228
```
Object names of directory objects qualified for ExpiredObjectAllVersions
must be encoded appropriately before calling on deletePrefix on their
erasure set.
e.g., a directory object and regular objects with overlapping prefixes
could lead to the expiration of regular objects, which is not the
intention of ILM.
```
bucket/dir/ ---> directory object
bucket/dir/obj-1
```
When `bucket/dir/` qualifies for expiration, the current implementation would
remove regular objects under the prefix `bucket/dir/`, in this case,
`bucket/dir/obj-1`.
In handlers related to health diagnostics e.g. CPU, Network, Partitions,
etc, globalMinioHost was being passed as the addr, resulting in empty
value for the same in the health report.
Using globalLocalNodeName instead fixes the issue.
IAM loading is a lazy operation, allow these
fallbacks to be in place when we cannot find
in-memory state().
this allows us to honor the request even if pay
a small price for lookup and populating the data.
When objects have more versions than their ILM policy expects to retain
via NewerNoncurrentVersions, but they don't qualify for expiry due to
NoncurrentDays are configured in that rule.
In this case, applyNewerNoncurrentVersionsLimit method was enqueuing empty
tasks, which lead to a panic (panic: runtime error: index out of range [0] with
length 0) in newerNoncurrentTask.OpHash method, which assumes the task
to contain at least one version to expire.
When returning the status of a decommissioned pool, a pool with zero
time StartedTime will be considered an active pool, which is unexpected.
This commit will always ensure that a pool's canceled/failed/completed
status is returned.
This commit changes how MinIO generates the object encryption key (OEK)
when encrypting an object using server-side encryption.
This change is fully backwards compatible. Now, MinIO generates
the OEK as following:
```
Nonce = RANDOM(32) // generate 256 bit random value
OEK = HMAC-SHA256(EK, Context || Nonce)
```
Before, the OEK was computed as following:
```
Nonce = RANDOM(32) // generate 256 bit random value
OEK = SHA256(EK || Nonce)
```
The new scheme does not technically fix a security issue but
uses a more familiar scheme. The only requirement for the
OEK generation function is that it produces a (pseudo)random value
for every pair (`EK`,`Nonce`) as long as no `EK`-`Nonce` combination
is repeated. This prevents a faulty PRNG from repeating or generating
a "bad" key.
The previous scheme guarantees that the `OEK` is a (pseudo)random
value given that no pair (`EK`,`Nonce`) repeats under the assumption
that SHA256 is indistinguable from a random oracle.
The new scheme guarantees that the `OEK` is a (pseudo)random value
given that no pair (`EK`, `Nonce`) repeats under the assumption that
SHA256's underlying compression function is a PRF/PRP.
While the later is a weaker assumption, and therefore, less likely
to be false, both are considered true. SHA256 is believed to be
indistinguable from a random oracle AND its compression function
is assumed to be a PRF/PRP.
As far as the OEK generating is concerned, the OS random number
generator is not required to be pseudo-random but just non-repeating.
Apart from being more compatible to standard definitions and
descriptions for how to generate crypto. keys, this change does not
have any impact of the actual security of the OEK key generation.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
avoids error during upgrades such as
```
API: SYSTEM()
Time: 19:19:22 UTC 03/18/2024
DeploymentID: 24e4b574-b28d-4e94-9bfa-03c363a600c2
Error: Invalid api configuration: found invalid keys (expiry_workers=100 ) for 'api' sub-system, use 'mc admin config reset myminio api' to fix invalid keys (*fmt.wrapError)
11: internal/logger/logger.go:260:logger.LogIf()
...
```
we were prematurely not writing 4k pages while we
could have due to the fact that most buffers would
be multiples of 4k upto some number and there shall
be some remainder.
We only need to write the remainder without O_DIRECT.
at scale customers might start with failed drives,
causing skew in the overall usage ratio per EC set.
make this configurable such that customers can turn
this off as needed depending on how comfortable they
are.
Cosmetic change, but breaks up a big code block and will make a goroutine
dumps of streams are more readable, so it is clearer what each goroutine is doing.
Currently, the code relies on object parity to decide whether it is a
delete marker or a regular object. In the case of a delete marker, the
return quorum is half of the disks in the erasure set. However, this
calculation must be corrected with objects with EC = 0, mainly
because EC is not a one-time fixed configuration.
Though all data are correct, the manifested symptom is a 503 with an
EC=0 object. This bug was manifested after we introduced the
fast Get Object feature that does not read all data from all disks in
case of inlined objects
Metrics v3 is mainly a reorganization of metrics into smaller groups of
metrics and the removal of internal aggregation of metrics received from
peer nodes in a MinIO cluster.
This change adds the endpoint `/minio/metrics/v3` as the top-level metrics
endpoint and under this, various sub-endpoints are implemented. These
are currently documented in `docs/metrics/v3.md`
The handler will serve metrics at any path
`/minio/metrics/v3/PATH`, as follows:
when PATH is a sub-endpoint listed above => serves the group of
metrics under that path; or when PATH is a (non-empty) parent
directory of the sub-endpoints listed above => serves metrics
from each child sub-endpoint of PATH. otherwise, returns a no
resource found error
All available metrics are listed in the `docs/metrics/v3.md`. More will
be added subsequently.
When an object qualifies for both tiering and expiration rules and is
past its expiration date, it should be expired without requiring to tier
it, even when tiering event occurs before expiration.
Merging same-object - multiple versions from different pools would not always result in correct ordering.
When merging keep inputs separate.
```
λ mc ls --versions local/testbucket
------ before ------
[2024-03-05 20:17:19 CET] 228B STANDARD 1f163718-9bc5-4b01-bff7-5d8cf09caf10 v3 PUT hosts
[2024-03-05 20:19:56 CET] 19KiB STANDARD null v2 PUT hosts
[2024-03-05 20:17:15 CET] 228B STANDARD 73c9f651-f023-4566-b012-cc537fdb7ce2 v1 PUT hosts
------ after ------
λ mc ls --versions local/testbucket
[2024-03-05 20:19:56 CET] 19KiB STANDARD null v3 PUT hosts
[2024-03-05 20:17:19 CET] 228B STANDARD 1f163718-9bc5-4b01-bff7-5d8cf09caf10 v2 PUT hosts
[2024-03-05 20:17:15 CET] 228B STANDARD 73c9f651-f023-4566-b012-cc537fdb7ce2 v1 PUT hosts
```
configure batch size to send audit/logger events
in batches instead of sending one event per connection.
this is mainly to optimize the number of requests
we make to webhook endpoint.
Currently, the progress of the batch job is saved in inside the job
request object, which is normally not supported by MinIO. Though there
is no apparent bug, it is better to fix this now.
Batch progress is saved in .minio.sys/batch-jobs/reports/
Co-authored-by: Anis Eleuch <anis@min.io>
our PoolNumber calculation was costly,
while we already had this information per
endpoint, we needed to deduce it appropriately.
This PR addresses this by assigning PoolNumbers
field that carries all the pool numbers that
belong to a server.
properties.PoolNumber still carries a valid value
only when len(properties.PoolNumbers) == 1, otherwise
properties.PoolNumber is set to math.MaxInt (indicating
that this value is undefined) and then one must rely
on properties.PoolNumbers for server participation
in multiple pools.
addresses the issue originating from #11327
simplify audit webhook worker model
fixes couple of bugs like
- ping(ctx) was creating a logger without updating
number of workers leading to incorrect nWorkers
scaling, causing an additional worker that is not
tracked properly.
- h.logCh <- entry could potentially hang for when
the queue is full on heavily loaded systems.
there can be a sudden spike in tiny allocations,
due to too much auditing being done, also don't hang
on the
```
h.logCh <- entry
```
after initializing workers if you do not have a way to
dequeue for some reason.
This commits adds support for using the `--endpoint` arg when creating a
tier of type `azure`. This is needed to connect to Azure's Gov Cloud
instance. For example,
```
mc ilm tier add azure TARGET TIER_NAME \
--account-name ACCOUNT \
--account-key KEY \
--bucket CONTAINER \
--endpoint https://ACCOUNT.blob.core.usgovcloudapi.net
--prefix PREFIX \
--storage-class STORAGE_CLASS
```
Prior to this, the endpoint was hardcoded to `https://ACCOUNT.blob.core.windows.net`.
The docs were even explicit about this, stating that `--endpoint` is:
"Required for `s3` or `minio` tier types. This option has no effect for any
other value of `TIER_TYPE`."
Now, if the endpoint arg is present it will be used. If not, it will
fall back to the same default behavior of `ACCOUNT.blob.core.windows.net`.
Remove api.expiration_workers config setting which was inadvertently left behind. Per review comment
https://github.com/minio/minio/pull/18926, expiration_workers can be configured via ilm.expiration_workers.
ext4, xfs support this behavior however
btrfs, nfs may not support it properly.
in-case when we see Nlink < 2 then we know
that we need to fallback on readdir()
fixes a regression from #19100fixes#19181
The middleware sets up tracing, throttling, gzipped responses and
collecting API stats.
Additionally, this change updates the names of handler functions in
metric labels to be the same as the name derived from Go lang reflection
on the handler name.
The metric api labels are now stored in memory the same as the handler
name - they will be camelcased, e.g. `GetObject` instead of `getobject`.
For compatibility, we lowercase the metric api label values when emitting the metrics.
- Use a shared worker pool for all ILM expiry tasks
- Free version cleanup executes in a separate goroutine
- Add a free version only if removing the remote object fails
- Add ILM expiry metrics to the node namespace
- Move tier journal tasks to expiryState
- Remove unused on-disk journal for tiered objects pending deletion
- Distribute expiry tasks across workers such that the expiry of versions of
the same object serialized
- Ability to resize worker pool without server restart
- Make scaling down of expiryState workers' concurrency safe; Thanks
@klauspost
- Add error logs when expiryState and transition state are not
initialized (yet)
* metrics: Add missed tier journal entry tasks
* Initialize the ILM worker pool after the object layer
With this commit, MinIO generates root credentials automatically
and deterministically if:
- No root credentials have been set.
- A KMS (KES) is configured.
- API access for the root credentials is disabled (lockdown mode).
Before, MinIO defaults to `minioadmin` for both the access and
secret keys. Now, MinIO generates unique root credentials
automatically on startup using the KMS.
Therefore, it uses the KMS HMAC function to generate pseudo-random
values. These values never change as long as the KMS key remains
the same, and the KMS key must continue to exist since all IAM data
is encrypted with it.
Backward compatibility:
This commit should not cause existing deployments to break. It only
changes the root credentials of deployments that have a KMS configured
(KES, not a static key) but have not set any admin credentials. Such
implementations should be rare or not exist at all.
Even if the worst case would be updating root credentials in mc
or other clients used to administer the cluster. Root credentials
are anyway not intended for regular S3 operations.
Signed-off-by: Andreas Auernhammer <github@aead.dev>
just like client-conn-read-deadline, added a new flag that does
client-conn-write-deadline as well.
Both are not configured by default, since we do not yet know
what is the right value. Allow this to be configurable if needed.
we should do this to ensure that we focus on
data healing as primary focus, fixing metadata
as part of healing must be done but making
data available is the main focus.
the main reason is metadata inconsistencies can
cause data availability issues, which must be
avoided at all cost.
will be bringing in an additional healing mechanism
that involves "metadata-only" heal, for now we do
not expect to have these checks.
continuation of #19154
Bonus: add a pro-active healthcheck to perform a connection
This change makes the label names consistent with the handler names.
This is in preparation to use reflection based API handler function
names for the api labels so they will be the same as tracing, auditing
and logging names for these API calls.
Moved different dashboards to their specific directories. Also
mentioned that these dashbards are examples of how to create
graphs using MinIO provided and metrics and customers should
change / add graphs on their specific need basis.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Streams can return errors if the cancelation is picked up before the response
stream close is picked up. Under extreme load, this could lead to missing
responses.
Send server mux ack async so a blocked send cannot block newMuxStream
call. Stream will not progress until mux has been acked.
in k8s things really do come online very asynchronously,
we need to use implementation that allows this randomness.
To facilitate this move WriteAll() as part of the
websocket layer instead.
Bonus: avoid instances of dnscache usage on k8s
New disk healing code skips/expires objects that ILM supposed to expire.
Add more visibility to the user about this activity by calculating those
objects and print it at the end of healing activity.
This PR fixes a bug that perhaps has been long introduced,
with no visible workarounds. In any deployment, if an entire
erasure set is deleted, there is no way the cluster recovers.
Currently, if one object tag matches with one lifecycle tag filter, ILM
will select it, however, this is wrong. All the Tag filters in the
lifecycle document should be satisfied.
This change is to decouple need for root credentials to match between
site replication deployments.
Also ensuring site replication config initialization is re-tried until
it succeeds, this deoendency is critical to STS flow in site replication
scenario.
Currently, we read from `/proc/diskstats` which is found to be
un-reliable in k8s environments. We can read from `sysfs` instead.
Also, cache the latest drive io stats to find the diff and update
the metrics.
* Remove lock for cached operations.
* Rename "Relax" to `ReturnLastGood`.
* Add `CacheError` to allow caching values even on errors.
* Add NoWait that will return current value with async fetching if within 2xTTL.
* Make benchmark somewhat representative.
```
Before: BenchmarkCache-12 16408370 63.12 ns/op 0 B/op
After: BenchmarkCache-12 428282187 2.789 ns/op 0 B/op
```
* Remove `storageRESTClient.scanning`. Nonsensical - RPC clients will not have any idea about scanning.
* Always fetch remote diskinfo metrics and cache them. Seems most calls are requesting metrics.
* Do async fetching of usage caches.
It also fixes a long-standing bug in expiring transitioned objects.
The expiration action was deleting the current version in the case'
of tiered objects instead of adding a delete marker.
only enable md5sum if explicitly asked by the client, otherwise
its not necessary to compute md5sum when SSE-KMS/SSE-C is enabled.
this is continuation of #17958
If network conditions have filled the output queue before a reconnect happens blocked sends could stop reconnects from happening. In short `respMu` would be held for a mux client while sending - if the queue is full this will never get released and closing the mux client will hang.
A) Use the mux client context instead of connection context for sends, so sends are unblocked when the mux client is canceled.
B) Use a `TryLock` on "close" and cancel the request if we cannot get the lock at once. This will unblock any attempts to send.
data-dir not being present is okay, however we can still
rely on the `rename()` atomic call instead of relying on
write xl.meta write which may truncate the io.EOF.
Add a new function logger.Event() to send the log to Console and
http/kafka log webhooks. This will include some internal events such as
disk healing and rebalance/decommissioning
the PR in #16541 was incorrect and hand wrong assumptions
about the overall setup, revert this since this expectation
to have offline servers is wrong and we can end up with a
bigger chicken and egg problem.
This reverts commit 5996c8c4d5.
Bonus:
- preserve disk in globalLocalDrives properly upon connectDisks()
- do not return 'nil' from newXLStorage(), getting it ready for
the next set of changes for 'format.json' loading.
The previous logic of calculating per second values for disk io stats
divides the stats by the host uptime. This doesn't work in k8s
environment as the uptime is of the pod, but the stats (from
/proc/diskstats) are from the host.
Fix this by storing the initial values of uptime and the stats at the
timme of server startup, and using the difference between current and
initial values when calculating the per second values.
globalLocalDrives seem to be not updated during the
HealFormat() leads to a requirement where the server
needs to be restarted for the healing to continue.
a/prefix
a/prefix/1.txt
where `a/prefix` is an object which does not have `/` at the end,
we do not have to aggressively recursively delete all the sub-folders
as well. Instead convert the call into self contained to deleting
'xl.meta' and then subsequently attempting to Remove the parent.
Bonus: enable audit alerts for object versions
beyond the configured value, default is '100'
versions per object beyond which scanner will
alert for each such objects.
when we expand via pools, there is no reason to stick
with the same distributionAlgo as the rest. Since the
algo only makes sense with-in a pool not across pools.
This allows for newer pools to use newer codepaths to
avoid legacy file lookups when they have a pre-existing
deployment from 2019, they can expand their new pool
to be of a newer distribution format, allowing the
pool to be more performant.
Fix reported races that are actually synchronized by network calls.
But this should add some extra safety for untimely disconnects.
Race reported:
```
WARNING: DATA RACE
Read at 0x00c00171c9c0 by goroutine 214:
github.com/minio/minio/internal/grid.(*muxClient).addResponse()
e:/gopath/src/github.com/minio/minio/internal/grid/muxclient.go:519 +0x111
github.com/minio/minio/internal/grid.(*muxClient).error()
e:/gopath/src/github.com/minio/minio/internal/grid/muxclient.go:470 +0x21d
github.com/minio/minio/internal/grid.(*Connection).handleDisconnectClientMux()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:1391 +0x15b
github.com/minio/minio/internal/grid.(*Connection).handleMsg()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:1190 +0x1ab
github.com/minio/minio/internal/grid.(*Connection).handleMessages.func1()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:981 +0x610
Previous write at 0x00c00171c9c0 by goroutine 1081:
github.com/minio/minio/internal/grid.(*muxClient).roundtrip()
e:/gopath/src/github.com/minio/minio/internal/grid/muxclient.go:94 +0x324
github.com/minio/minio/internal/grid.(*muxClient).traceRoundtrip()
e:/gopath/src/github.com/minio/minio/internal/grid/trace.go:74 +0x10e4
github.com/minio/minio/internal/grid.(*Subroute).Request()
e:/gopath/src/github.com/minio/minio/internal/grid/connection.go:366 +0x230
github.com/minio/minio/internal/grid.(*SingleHandler[go.shape.*github.com/minio/minio/cmd.DiskInfoOptions,go.shape.*github.com/minio/minio/cmd.DiskInfo]).Call()
e:/gopath/src/github.com/minio/minio/internal/grid/handlers.go:554 +0x3fd
github.com/minio/minio/cmd.(*storageRESTClient).DiskInfo()
e:/gopath/src/github.com/minio/minio/cmd/storage-rest-client.go:314 +0x270
github.com/minio/minio/cmd.erasureObjects.getOnlineDisksWithHealingAndInfo.func1()
e:/gopath/src/github.com/minio/minio/cmd/erasure.go:293 +0x171
```
This read will always happen after the write, since there is a network call in between.
However a disconnect could come in while we are setting up the call, so we protect against that with extra checks.
- bucket metadata does not need to look for legacy things
anymore if b.Created is non-zero
- stagger bucket metadata loads across lots of nodes to
avoid the current thundering herd problem.
- Remove deadlines for RenameData, RenameFile - these
calls should not ever be timed out and should wait
until completion or wait for client timeout. Do not
choose timeouts for applications during the WRITE phase.
- increase R/W buffer size, increase maxMergeMessages to 30
We have observed cases where a blocked stream will block for cancellations.
This happens when response channel is blocked and we want to push an error.
This will have the response mutex locked, which will prevent all other operations until upstream is unblocked.
Make this behavior non-blocking and if blocked spawn a goroutine that will send the response and close the output.
Still a lot of "dancing". Added a test for this and reviewed.
Depending on when the context cancelation is picked up the handler may return and close the channel before `SubscribeJSON` returns, causing:
```
Feb 05 17:12:00 s3-us-node11 minio[3973657]: panic: send on closed channel
Feb 05 17:12:00 s3-us-node11 minio[3973657]: goroutine 378007076 [running]:
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub.(*PubSub[...]).SubscribeJSON.func1()
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub/pubsub.go:139 +0x12d
Feb 05 17:12:00 s3-us-node11 minio[3973657]: created by github.com/minio/minio/internal/pubsub.(*PubSub[...]).SubscribeJSON in goroutine 378010884
Feb 05 17:12:00 s3-us-node11 minio[3973657]: github.com/minio/minio/internal/pubsub/pubsub.go:124 +0x352
```
Wait explicitly for the goroutine to exit.
Bonus: Listen for doneCh when sending to not risk getting blocked there is channel isn't being emptied.
this fixes rare bugs we have seen but never really found a
reproducer for
- PutObjectRetention() returning 503s
- PutObjectTags() returning 503s
- PutObjectMetadata() updates during replication returning 503s
These calls return errors, and this perpetuates with
no apparent fix.
This PR fixes with correct quorum requirement.
To force limit the duration of STS accounts, the user can create a new
policy, like the following:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {"NumericLessThanEquals": {"sts:DurationSeconds": "300"}}
}]
}
And force binding the policy to all OpenID users, whether using a claim name or role
ARN.
disk tokens usage is not necessary anymore with the implementation
of deadlines for storage calls and active monitoring of the drive
for I/O timeouts.
Functionality kicking off a bad drive is still supported, it's just that
we do not have to serialize I/O in the manner tokens would do.
Recycle would always be called on the dummy value `any(newRT())` instead of the actual value given to the recycle function.
Caught by race tests, but mostly harmless, except for reduced perf.
Other minor cleanups. Introduced in #18940 (unreleased)
Interpret `null` inline policy for access keys as inheriting parent
policy. Since MinIO Console currently sends this value, we need to honor it
for now. A larger fix in Console and in the server are required.
Fixes#18939.
Allow internal types to support a `Recycler` interface, which will allow for sharing of common types across handlers.
This means that all `grid.MSS` (and similar) objects are shared across in a common pool instead of a per-handler pool.
Add internal request reuse of internal types. Add for safe (pointerless) types explicitly.
Only log params for internal types. Doing Sprint(obj) is just a bit too messy.
With this change, only a user with `UpdateServiceAccountAdminAction`
permission is able to edit access keys.
We would like to let a user edit their own access keys, however the
feature needs to be re-designed for better security and integration with
external systems like AD/LDAP and OpenID.
This change prevents privilege escalation via service accounts.
for actionable, inspections we have `mc support inspect`
we do not need double logging, healing will report relevant
errors if any, in terms of quorum lost etc.
Each Put, List, Multipart operations heavily rely on making
GetBucketInfo() call to verify if bucket exists or not on
a regular basis. This has a large performance cost when there
are tons of servers involved.
We did optimize this part by vectorizing the bucket calls,
however its not enough, beyond 100 nodes and this becomes
fairly visible in terms of performance.
- healing must not set the write xattr
because that is the job of active healing
to update. what we need to preserve is
permanent deletes.
- remove older env for drive monitoring and
enable it accordingly, as a global value.
local disk metrics were polluting cluster metrics
Please remove them instead of adding relevant ones.
- batch job metrics were incorrectly kept at bucket
metrics endpoint, move it to cluster metrics.
- add tier metrics to cluster peer metrics from the node.
- fix missing set level cluster health metrics
Do not rely on `connChange` to do reconnects.
Instead, you can block while the connection is running and reconnect
when handleMessages returns.
Add fully async monitoring instead of monitoring on the main goroutine
and keep this to avoid full network lockup.
it is entirely possible that a rebalance process which was running
when it was asked to "stop" it failed to write its last statistics
to the disk.
After this a pool expansion can cause disruption and all S3 API
calls would fail at IsPoolRebalancing() function.
This PRs makes sure that we update rebalance.bin under such
conditions to avoid any runtime crashes.
add new update v2 that updates per node, allows idempotent behavior
new API ensures that
- binary is correct and can be downloaded checksummed verified
- committed to actual path
- restart returns back the relevant waiting drives
do not need to be defensive in our approach,
we should simply override anything everything
in import process, do not care about what
currently exists on the disk - backup is the
source of truth.
Right now the format.json is excluded if anything within `.minio.sys` is requested.
I assume the check was meant to exclude only if it was actually requesting it.
- Move RenameFile to websockets
- Move ReadAll that is primarily is used
for reading 'format.json' to to websockets
- Optimize DiskInfo calls, and provide a way
to make a NoOp DiskInfo call.
Add separate reconnection mutex
Give more safety around reconnects and make sure a state change isn't missed.
Tested with several runs of `λ go test -race -v -count=500`
Adds separate mutex and doesn't mix in the testing mutex.
AlmosAll uses of NewDeadlineWorker, which relied on secondary values, were used in a racy fashion,
which could lead to inconsistent errors/data being returned. It also propagates the deadline downstream.
Rewrite all these to use a generic WithDeadline caller that can return an error alongside a value.
Remove the stateful aspect of DeadlineWorker - it was racy if used - but it wasn't AFAICT.
Fixes races like:
```
WARNING: DATA RACE
Read at 0x00c130b29d10 by goroutine 470237:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).ReadVersion()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:702 +0x611
github.com/minio/minio/cmd.readFileInfo()
github.com/minio/minio/cmd/erasure-metadata-utils.go:160 +0x122
github.com/minio/minio/cmd.erasureObjects.getObjectFileInfo.func1.1()
github.com/minio/minio/cmd/erasure-object.go:809 +0x27a
github.com/minio/minio/cmd.erasureObjects.getObjectFileInfo.func1.2()
github.com/minio/minio/cmd/erasure-object.go:828 +0x61
Previous write at 0x00c130b29d10 by goroutine 470298:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).ReadVersion.func1()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:698 +0x244
github.com/minio/minio/internal/ioutil.(*DeadlineWorker).Run.func1()
github.com/minio/minio/internal/ioutil/ioutil.go:141 +0x33
WARNING: DATA RACE
Write at 0x00c0ba6e6c00 by goroutine 94507:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).StatVol.func1()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:419 +0x104
github.com/minio/minio/internal/ioutil.(*DeadlineWorker).Run.func1()
github.com/minio/minio/internal/ioutil/ioutil.go:141 +0x33
Previous read at 0x00c0ba6e6c00 by goroutine 94463:
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).StatVol()
github.com/minio/minio/cmd/xl-storage-disk-id-check.go:422 +0x47e
github.com/minio/minio/cmd.getBucketInfoLocal.func1()
github.com/minio/minio/cmd/peer-s3-server.go:275 +0x122
github.com/minio/pkg/v2/sync/errgroup.(*Group).Go.func1()
```
Probably back from #17701
protection was in place. However, it covered only some
areas, so we re-arranged the code to ensure we could hold
locks properly.
Along with this, remove the DataShardFix code altogether,
in deployments with many drive replacements, this can affect
and lead to quorum loss.
Also limit the amount of concurrency when sending
binary updates to peers, avoid high network over
TX that can cause disconnection events for the
node sending updates.
Race checks would occasionally show race on handleMsgWg WaitGroup by debug messages (used in test only).
Use the `connMu` mutex to protect this against concurrent Wait/Add.
Fixes#18827
If site replication is enabled, we should still show the size and
version distribution histogram metrics at bucket level.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
New API now verifies any hung disks before restart/stop,
provides a 'per node' break down of the restart/stop results.
Provides also how many blocked syscalls are present on the
drives and what users must do about them.
Adds options to do pre-flight checks to provide information
to the user regarding any hung disks. Provides 'force' option
to forcibly attempt a restart() even with waiting syscalls
on the drives.
When rejecting incoming grid requests fill out the rejection reason and log it once.
This will give more context when startup is failing. Already logged after a retry on caller.
On a policy detach operation, if there are no policies remaining
attached to the user/group, remove the policy mapping file, instead of
leaving a file containing an empty list of policies.
Healing dangling buckets is conservative, and it is a typical use case to
fail to remove a dangling bucket because it contains some data because
healing danging bucket code is not allowed to remove data: only healing
the dangling object is allowed to do so.
reference format is constant for any lifetime of
a minio cluster, we do not have to ever replace
it during HealFormat() as it will never change.
additionally we should simply reject reference
formats that we do not understand early on.
GetActualSize() was heavily relying on o.Parts()
to be non-empty to figure out if the object is multipart or not,
However, we have many indicators of whether an object is multipart
or not.
Blindly assuming that o.Parts == nil is not a multipart, is an
incorrect expectation instead, multipart must be obtained via
- Stored metadata value indicating this is a multipart encrypted object.
- Rely on <meta>-actual-size metadata to get the object's actual size.
This value is preserved for additional reasons such as these.
- ETag != 32 length
support proxying of tagging requests in active-active replication
Note: even if proxying is successful, PutObjectTagging/DeleteObjectTagging
will continue to report a 404 since the object is not present locally.
New intervals:
[1024B, 64KiB)
[64KiB, 256KiB)
[256KiB, 512KiB)
[512KiB, 1MiB)
The new intervals helps us see object size distribution with higher
resolution for the interval [1024B, 1MiB).
- HealFormat() was leaking healthcheck goroutines for
disks, we are only interested in enabling healthcheck
for the newly formatted disk, not for existing disks.
- When disk is a root-disk a random disk monitor was
leaking while we ignored the drive.
- When loading the disk for each erasure set, we were
leaking goroutines for the prepare-storage.go disks
which were replaced via the globalLocalDrives slice
- avoid disk monitoring utilizing health tokens that
would cause exhaustion in the tokens, prematurely
which were meant for incoming I/O. This is ensured
by avoiding writing O_DIRECT aligned buffer instead
write 2048 worth of content only as O_DSYNC, which is
sufficient.
All GitHub issues are addressed on a best-effort basis at MinIO's sole discretion. There are no Service Level Agreements (SLA) or Objectives (SLO). Remember our [Code of Conduct](https://github.com/minio/minio/blob/master/code_of_conduct.md) when engaging with MinIO Engineers and the larger community.
For urgent issues (e.g. production down, etc.), subscribe to [SUBNET](https://min.io/pricing?jmp=github) for direct to engineering support.
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->
1.
2.
3.
4.
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
if ! git log -1 --format='%(trailers:key=Signed-off-by,valueonly)' "${sha}" |
grep -qiF "<${author_email}>"; then
echo "::error::commit ${sha} by ${author_name} <${author_email}> lacks a matching Signed-off-by trailer; sign with 'git commit -s', repair with 'git rebase --signoff'"
if [ "${CHECKED_OUT_REVISION}" != "${GITHUB_SHA}" ]; then
if [ "${RECOVERY}" != "true" ] || [ "${GITHUB_REF}" != "refs/heads/${DEFAULT_BRANCH}" ]; then
echo "Checked out ${CHECKED_OUT_REVISION}, but workflow identity is ${GITHUB_SHA}. Dispatch from ${RELEASE_TAG}, or use recovery from ${DEFAULT_BRANCH}." >&2
exit 1
fi
echo "Recovery workflow ${GITHUB_SHA} is packaging published source ${CHECKED_OUT_REVISION}."
We have designed MinIO as an Open Source software for the Open Source software community. This requires applications to consider whether their usage of MinIO is in compliance with the GNU AGPLv3 [license](https://github.com/minio/minio/blob/master/LICENSE).
Silo is distributed under the [GNU Affero General Public License v3.0](LICENSE).
It incorporates source code from the MinIO project and preserves the original
copyright, license, and attribution notices in [`NOTICE`](NOTICE),
[`CREDITS`](CREDITS), and source-file headers.
MinIO cannot make the determination as to whether your application's usage of MinIO is in compliance with the AGPLv3 license requirements. You should instead rely on your own legal counsel or licensing specialists to audit and ensure your application is in compliance with the licenses of MinIO and all other open-source projects with which your application integrates or interacts. We understand that AGPLv3 licensing is complex and nuanced. It is for that reason we strongly encourage using experts in licensing to make any such determinations around compliance instead of relying on apocryphal or anecdotal advice.
You are responsible for determining how the AGPLv3 applies to your use,
modification, deployment, and distribution of Silo and its dependencies. The
Silo maintainers cannot provide legal advice or determine whether a particular
application or service satisfies the license. Consult qualified counsel when
the obligations are material to your deployment.
[MinIO Commercial Licensing](https://min.io/pricing) is the best option for applications that trigger AGPLv3 obligations (e.g. open sourcing your application). Applications using MinIO - or any other OSS-licensed code - without validating their usage do so at their own risk.
If you convey modified binaries or provide network access to a modified
version, review the complete AGPLv3 text and ensure that the corresponding
source and notices are made available as required. Dependency licenses and
``MinIO`` community welcomes your contribution. To make the process as seamless as possible, we recommend you read this contribution guide.
Silo welcomes focused contributions that improve security, reliability,
compatibility, packaging, tests, or maintainability. This repository preserves
MinIO-compatible interfaces and storage formats, so changes must identify and
test any compatibility impact.
## Development Workflow
Start by forking the MinIO GitHub repository, make changes in a branch and then send a pull request. We encourage pull requests to discuss code changes. Here are the steps in details:
Fork the current Silo source repository, create a topic branch, and submit a
pull request. Discuss broad or compatibility-sensitive changes in an issue
before implementation.
### Setup your MinIO GitHub Repository
Fork [MinIO upstream](https://github.com/minio/minio/fork) source repository to your own personal repository. Copy the URL of your MinIO fork (you will need it for the `git clone` command below).
Pull requests can be created via GitHub. Refer to [this document](https://help.github.com/articles/creating-a-pull-request/) for detailed steps on how to create a pull request. After a Pull Request gets peer reviewed and approved, it will be merged.
Pull requests should include motivation, reproduction steps where applicable,
test evidence, compatibility notes, and documentation impact. Public product
Code contributions to PGSTY SILO (`pgsty/silo`) are accepted under the
[GNU AGPL v3.0 or later](LICENSE), the same license as the server. Submit issues
and pull requests to this repository's maintainers. No separate Apache-2.0
license grant to SILO or upstream MinIO maintainers is required.
* **No CLA.** We do not ask you to sign a Contributor License Agreement and we
do not take your copyright. Contributions are accepted inbound=outbound: you
keep the copyright to your changes and license them under the same
AGPL-3.0-or-later as the project itself. The maintainers receive no rights
beyond the project license.
* **DCO sign-off required.** Every commit must carry a
`Signed-off-by: Your Name <you@example.com>` trailer certifying the
[Developer Certificate of Origin 1.1](https://developercertificate.org/) —
your statement that you have the right to submit the code under the project
license. Sign each commit with:
```
git commit -s
```
Forgot some? Repair your branch with `git rebase --signoff` and force-push.
CI rejects pull requests containing unsigned commits; the sign-off email
must match the commit author email. (Lowercase `-s` is the plain-text DCO
sign-off; cryptographic `-S`/GPG signing is welcome but independent.)
* **Provenance.** Only submit code you are entitled to submit. This matters
more here than in most projects: Silo carries a downstream delta over an
upstream code base, and cherry-picks from the lineage remote or other forks
are routine. When relaying a patch written by someone else, preserve original
authorship (`git cherry-pick -x`, keep the author field and any existing
`Signed-off-by` trailers) and add your own sign-off as the person passing it
along. Never import code from a proprietary distribution.
* **File headers.** Preserve existing copyright and license notices in inherited
and third-party files. New original files name their actual copyright holders
and use AGPL-3.0-or-later. Use a header such as the following, then append the
standard AGPL boilerplate:
```
// Copyright (c) 2026 Your Name
```
* **Separately licensed material.** Documentation contributions in `docs/`
follow its existing [CC BY 4.0 license](docs/LICENSE). Third-party components
and earlier Apache-2.0 contributions retain their original licenses and
attribution; this policy does not relicense earlier work.
* **Squash merges** must keep the `Signed-off-by:` trailers in the resulting
commit message.
* **Authorship and tooling.** The human contributor is the author of the commit
and the sole signatory of its DCO sign-off. Attribution trailers for
assistive tooling (for example `Co-Authored-By:` naming an AI assistant) are
informational only: they record which tools were used, and do not create
authorship, co-authorship, or any copyright claim. Whoever signs off remains
responsible for the content of the commit, whatever produced it.
## FAQs
### How does ``MinIO`` manage dependencies?
### How does Silo manage dependencies?
``MinIO`` uses `go mod` to manage its dependencies.
Silo uses Go modules. Preserve the compatibility module and import paths in
`go.mod`; downstream forks are selected with explicit `replace` directives.
- Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
@@ -76,6 +151,7 @@ To remove a dependency
- Edit your code and remove the import reference.
- Run `go mod tidy` in the source folder to remove dependency from `go.mod` file.
### What are the coding guidelines for MinIO?
### What are the coding guidelines?
``MinIO`` is fully conformant with Golang style. Refer: [Effective Go](https://github.com/golang/go/wiki/CodeReviewComments) article from Golang project. If you observe offending code, please feel free to send a pull request or ping us on [Slack](https://slack.min.io).
Follow the existing Go style, run `gofmt` on changed Go files, and keep changes
compact. See the Go project's [code review comments](https://go.dev/wiki/CodeReviewComments).
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service."\
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
test-replication:install-racetest-replication-2sitetest-replication-3sitetest-delete-replicationtest-sio-errortest-delete-marker-proxying## verify multi site replication
@echo "Running tests for replicating three sites"
test-site-replication-ldap:install-race## verify automatic site replication
These guidelines ensure high-quality commits in Silo's GitHub repositories, maintaining
a clear, valuable commit history for our open-source projects. They apply to all contributors,
fostering efficient reviews and robust code.
## Why Pull Requests?
Pull Requests (PRs) drive quality in Silo's codebase by:
- Enabling peer review without pair programming.
- Documenting changes for future reference.
- Ensuring commits tell a clear story of development.
**A poor commit lasts forever, even if code is refactored.**
## Crafting a Quality PR
A strong Silo PR:
- Delivers a complete, valuable change (feature, bug fix, or improvement).
- Has a concise title (e.g., `[S3] Fix bucket policy parsing #1234`) and a summary with context, referencing issues (e.g., `#1234`).
- Contains well-written, logical commits explaining *why* changes were made (e.g., “Add S3 bucket tagging support so that users can organize resources efficiently”).
- Is small, focused, and easy to review—ideally one commit, unless multiple commits better narrate complex work.
- Adheres to Silo's coding standards (e.g., Go style, error handling, testing).
PRs must flow smoothly through review to reach production. Large PRs should be split into smaller, manageable ones.
## Submitting PRs
1.**Title and Summary**:
- Use a scannable title: `[Subsystem] Action Description #Issue` (e.g., `[IAM] Add role-based access control #567`).
- Include context in the summary: what changed, why, and any issue references.
- Use `[WIP]` for in-progress PRs to avoid premature merging or choose GitHub draft PRs.
2.**Commits**:
- Write clear messages: what changed and why (e.g., “Refactor S3 API handler to reduce latency so that requests process 20% faster”).
- Rebase to tidy commits before submitting (e.g., `git rebase -i main` to squash typos or reword messages), unless multiple contributors worked on the branch.
- Keep PRs focused—one feature or fix. Split large changes into multiple PRs.
3.**Testing**:
- Include unit tests for new functionality or bug fixes.
- Ensure existing tests pass (`make test`).
- Document testing steps in the PR summary if manual testing was performed.
4.**Before Submitting**:
- Run `make verify` to check formatting, linting, and tests.
- Reference related issues (e.g., “Closes #1234”).
- Notify team members via GitHub `@mentions` if urgent or complex.
## Reviewing PRs
Reviewers ensure Silo's commit history remains a clear, reliable record. Responsibilities include:
1.**Commit Quality**:
- Verify each commit explains *why* the change was made (e.g., “So that…”).
- Request rebasing if commits are unclear, redundant, or lack context (e.g., “Please squash typo fixes into the parent commit”).
2.**Code Quality**:
- Check adherence to Silo's Go standards (e.g., error handling, documentation).
- Ensure tests cover new code and pass CI.
- Flag bugs or critical issues for immediate fixes; suggest non-blocking improvements as follow-up issues.
3.**Flow**:
- Review promptly to avoid blocking progress.
- Balance quality and speed—minor issues can be addressed later via issues, not PR blocks.
- If unable to complete the review, tag another reviewer (e.g., `@username please take over`).
4.**Shared Responsibility**:
- All Silo contributors are reviewers. The first commenter on a PR owns the review unless they delegate.
- Multiple reviewers are encouraged for complex PRs.
5.**No Self-Edits**:
- Don’t modify the PR directly (e.g., fixing bugs). Request changes from the submitter or create a follow-up PR.
- If you edit, you’re a collaborator, not a reviewer, and cannot merge.
6.**Testing**:
- Assume the submitter tested the code. If testing is unclear, ask for details (e.g., “How was this tested?”).
- Reject untested PRs unless testing is infeasible, then assist with test setup.
## Tips for Success
- **Small PRs**: Easier to review, faster to merge. Split large changes logically.
- **Clear Commits**: Use `git rebase -i` to refine history before submitting.
- **Engage Early**: Discuss complex changes in a GitHub issue before coding.
MinIO creates FIPS builds using a patched version of the Go compiler (that uses BoringCrypto, from BoringSSL, which is [FIPS 140-2 validated](https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2964.pdf)) published by the Golang Team [here](https://github.com/golang/go/tree/dev.boringcrypto/misc/boring).
MinIO FIPS executables are available at <http://dl.min.io> - they are only published for `linux-amd64` architecture as binary files with the suffix `.fips`. We also publish corresponding container images to our official image repositories.
We are not making any statements or representations about the suitability of this code or build in relation to the FIPS 140-2 standard. Interested users will have to evaluate for themselves whether this is useful for their own purposes.
<strong>S3-compatible object storage — a MinIO fork maintained by PGSTY</strong>
</p>
MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.
This README provides quickstart instructions on running MinIO on bare metal hardware, including container-based installations. For Kubernetes environments, use the [MinIO Kubernetes Operator](https://github.com/minio/operator/blob/master/README.md).
Use the following commands to run a standalone MinIO server as a container.
> [!IMPORTANT]
> **PGSTY Silo** (hereinafter “Silo”) is an independent, community-maintained fork of the open-source MinIO server, published by [Pigsty](https://pigsty.io) from [`pgsty/silo`](https://github.com/pgsty/silo). It is not affiliated with, endorsed by, or sponsored by MinIO, Inc. “MinIO” is used only to identify the upstream project and compatibility lineage.
Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication
require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically,
with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html)
for more complete documentation.
> [!NOTE]
> Renamed from `pgsty/minio` to `pgsty/silo`, default branch `master` → `main`, on 2026-08-06. Artifacts under the original MinIO identity stay published on the archived [`minio`](https://github.com/pgsty/silo/tree/minio) branch and in releases up to [`RELEASE.2026-08-04T00-00-00Z`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z).
### Stable
## Current release and main branch
Run the following command to run the latest stable image of MinIO as a container using an ephemeral data volume:
The latest published Server is [20260903](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-09-03T13-18-01Z).
As of 2026-09-13, the main branch has newer security, storage, Console and
shared-package changes that have not shipped in a Server release. See
[CHANGELOG.md](CHANGELOG.md) and the [component version matrix](https://silo.pgsty.com/compatibility/versions/)
for the exact release/source boundary, including SN-2026-011 and password-policy migration.
```sh
podman run -p 9000:9000 -p 9001:9001 \
quay.io/minio/minio server /data --console-address ":9001"
## Overview
PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
It follows one rule — **the product and its delivery surfaces are renamed; the protocol and your data are not.** Everything else lives on [silo.pgsty.com](https://silo.pgsty.com/).
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See
[Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers,
see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
Console on <http://localhost:9001>, S3 API on <http://localhost:9000>. The image bundles the client as `mcli`:
> NOTE: To deploy MinIO on with persistent storage, you must map local persistent directories from the host OS to the container using the `podman -v` option. For example, `-v /mnt/data:/data` maps the host OS drive at `/mnt/data` to `/data` on the container.
## macOS
Use the following commands to run a standalone MinIO server on macOS.
Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html) for more complete documentation.
### Homebrew (recommended)
Run the following command to install the latest stable MinIO package using [Homebrew](https://brew.sh/). Replace ``/data`` with the path to the drive or directory in which you want MinIO to store data.
```sh
brew install minio/stable/minio
minio server /data
```bash
docker exec silo mcli alias set local http://127.0.0.1:9000 minioadmin change-me-long-password
docker exec silo mcli mb local/demo && docker exec silo mcli ls local
```
> NOTE: If you previously installed minio using `brew install minio` then it is recommended that you reinstall minio from `minio/stable/minio` official repo instead.
```sh
brew uninstall minio
brew install minio/stable/minio
```
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html/> to view MinIO SDKs for supported languages.
### Binary Download
Use the following command to download and run a standalone MinIO server on macOS. Replace ``/data`` with the path to the drive or directory in which you want MinIO to store data.
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
## GNU/Linux
Use the following command to run a standalone MinIO server on Linux hosts running 64-bit Intel/AMD architectures. Replace ``/data`` with the path to the drive or directory in which you want MinIO to store data.
| 64-bit ARM | <https://dl.min.io/server/minio/release/linux-arm64/minio> |
| 64-bit PowerPC LE (ppc64le) | <https://dl.min.io/server/minio/release/linux-ppc64le/minio> |
| IBM Z-Series (S390X) | <https://dl.min.io/server/minio/release/linux-s390x/minio> |
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
> NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html#) for more complete documentation.
## Microsoft Windows
To run MinIO on 64-bit Windows hosts, download the MinIO executable from the following URL:
Use the following command to run a standalone MinIO server on the Windows host. Replace ``D:\`` with the path to the drive or directory in which you want MinIO to store data. You must change the terminal or powershell directory to the location of the ``minio.exe`` executable, *or* add the path to that directory to the system ``$PATH``:
```sh
minio.exe server D:\
```
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
> NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html#) for more complete documentation.
## Install from Source
Use the following commands to compile and run a standalone MinIO server from source. Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow [How to install Golang](https://golang.org/doc/install). Minimum version required is [go1.19](https://golang.org/dl/#stable)
```sh
go install github.com/minio/minio@latest
```
The MinIO deployment starts using default root credentials `minioadmin:minioadmin`. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to <http://127.0.0.1:9000> and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.
You can also connect using any S3-compatible tool, such as the MinIO Client `mc` commandline tool. See [Test using MinIO Client `mc`](#test-using-minio-client-mc) for more information on using the `mc` commandline tool. For application developers, see <https://min.io/docs/minio/linux/developers/minio-drivers.html> to view MinIO SDKs for supported languages.
> NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a *minimum* of 4 drives per MinIO server. See [MinIO Erasure Code Overview](https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html) for more complete documentation.
MinIO strongly recommends *against* using compiled-from-source MinIO servers for production environments.
## Deployment Recommendations
### Allow port access for Firewalls
By default MinIO uses the port 9000 to listen for incoming connections. If your platform blocks the port by default, you may need to enable access to the port.
### ufw
For hosts with ufw enabled (Debian based distros), you can use `ufw` command to allow traffic to specific ports. Use below command to allow access to port 9000
```sh
ufw allow 9000
```
Below command enables all incoming traffic to ports ranging from 9000 to 9010.
```sh
ufw allow 9000:9010/tcp
```
### firewall-cmd
For hosts with firewall-cmd enabled (CentOS), you can use `firewall-cmd` command to allow traffic to specific ports. Use below commands to allow access to port 9000
```sh
firewall-cmd --get-active-zones
```
This command gets the active zone(s). Now, apply port rules to the relevant zones returned above. For example if the zone is `public`, use
Note that `permanent` makes sure the rules are persistent across firewall start, restart or reload. Finally reload the firewall for changes to take effect.
```sh
firewall-cmd --reload
```
### iptables
For hosts with iptables enabled (RHEL, CentOS, etc), you can use `iptables` command to enable all traffic coming to specific ports. Use below command to allow
access to port 9000
```sh
iptables -A INPUT -p tcp --dport 9000 -j ACCEPT
service iptables restart
```
Below command enables all incoming traffic to ports ranging from 9000 to 9010.
```sh
iptables -A INPUT -p tcp --dport 9000:9010 -j ACCEPT
service iptables restart
```
## Test MinIO Connectivity
### Test using MinIO Console
MinIO Server comes with an embedded web based object browser. Point your web browser to <http://127.0.0.1:9000> to ensure your server has started successfully.
> NOTE: MinIO runs console on random port by default, if you wish to choose a specific port use `--console-address` to pick a specific interface and port.
### Things to consider
MinIO redirects browser access requests to the configured server port (i.e. `127.0.0.1:9000`) to the configured Console port. MinIO uses the hostname or IP address specified in the request when building the redirect URL. The URL and port *must* be accessible by the client for the redirection to work.
For deployments behind a load balancer, proxy, or ingress rule where the MinIO host IP address or port is not public, use the `MINIO_BROWSER_REDIRECT_URL` environment variable to specify the external hostname for the redirect. The LB/Proxy must have rules for directing traffic to the Console port specifically.
For example, consider a MinIO deployment behind a proxy `https://minio.example.net`, `https://console.minio.example.net` with rules for forwarding traffic on port :9000 and :9001 to MinIO and the MinIO Console respectively on the internal network. Set `MINIO_BROWSER_REDIRECT_URL` to `https://console.minio.example.net` to ensure the browser receives a valid reachable URL.
Similarly, if your TLS certificates do not have the IP SAN for the MinIO server host, the MinIO Console may fail to validate the connection to the server. Use the `MINIO_SERVER_URL` environment variable and specify the proxy-accessible hostname of the MinIO server to allow the Console to use the MinIO server API using the TLS certificate.
For example: `export MINIO_SERVER_URL="https://minio.example.net"`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the MinIO Client [Quickstart Guide](https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart) for further instructions.
## Upgrading MinIO
Upgrades require zero downtime in MinIO, all upgrades are non-disruptive, all transactions on MinIO are atomic. So upgrading all the servers simultaneously is the recommended way to upgrade MinIO.
> NOTE: requires internet access to update directly from <https://dl.min.io>, optionally you can host any mirrors at <https://my-artifactory.example.com/minio/>
- For deployments that installed the MinIO server binary by hand, use [`mc admin update`](https://min.io/docs/minio/linux/reference/minio-mc-admin/mc-admin-update.html)
```sh
mc admin update <minio alias, e.g., myminio>
```
- For deployments without external internet access (e.g. airgapped environments), download the binary from <https://dl.min.io> and replace the existing MinIO binary let's say for example `/opt/bin/minio`, apply executable permissions `chmod +x /opt/bin/minio` and proceed to perform `mc admin service restart alias/`.
- For installations using Systemd MinIO service, upgrade via RPM/DEB packages **parallelly** on all servers or replace the binary lets say `/opt/bin/minio` on all nodes, apply executable permissions `chmod +x /opt/bin/minio` and process to perform `mc admin service restart alias/`.
### Upgrade Checklist
- Test all upgrades in a lower environment (DEV, QA, UAT) before applying to production. Performing blind upgrades in production environments carries significant risk.
- Read the release notes for MinIO *before* performing any upgrade, there is no forced requirement to upgrade to latest release upon every release. Some release may not be relevant to your setup, avoid upgrading production environments unnecessarily.
- If you plan to use `mc admin update`, MinIO process must have write access to the parent directory where the binary is present on the host system.
- `mc admin update` is not supported and should be avoided in kubernetes/container environments, please upgrade containers by upgrading relevant container images.
- **We do not recommend upgrading one MinIO server at a time, the product is designed to support parallel upgrades please follow our recommended guidelines.**
> For production, pin a release, use unique credentials and TLS, monitor the service, keep independent backups, and test recovery. Start from the [documentation](https://silo.pgsty.com/docs/).
## Install
| Method | Where |
| :-- | :-- |
| Container | [`pgsty/silo`](https://hub.docker.com/r/pgsty/silo), multi-arch for `linux/amd64` and `linux/arm64` |
| Binaries | [GitHub Releases](https://github.com/pgsty/silo/releases) — Linux, macOS, Windows on `amd64` and `arm64` |
| Packages | RPM, DEB, and APK, also via the [Pigsty repository](https://pigsty.io/docs/repo/) |
| Kubernetes | Helm chart, see [Download & Install](https://silo.pgsty.com/download/) |
Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/); migrating from upstream MinIO — taking over an existing `minio.service` and its `/etc/default/minio`, and keeping data ownership stable with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in — is covered by the [migration guide](https://silo.pgsty.com/compatibility/migration/) and the [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
## Compatibility
The S3 API, `MINIO_*` variables, `minio_*` metrics, `x-minio-*` headers, `/minio/*` routes, the `github.com/minio/*` import paths, and the on-disk format (including `.minio.sys`) are preserved and held in place by a CI compatibility check. Only Silo-owned delivery surfaces change: the `silo` executable, package, service, Helm chart, and container image — no `minio` binary alias is installed.
Every divergence from upstream is listed in the code-verified [compatibility audit](https://silo.pgsty.com/compatibility/server/). Treat each release as a downstream upgrade: pin versions, read the [release notes](https://silo.pgsty.com/tags/silo/), and keep a rollback path.
### TLS and Go upgrades
TLS key exchange follows Go's defaults across the S3 listener, node links,
replication, identity providers, etcd, and external HTTP services. If an endpoint
cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges
for the process; certificate verification remains enabled. This option does not
disable ML-DSA signatures or resolve every TLS reset. Prefer updating the
incompatible endpoint before removing the temporary setting.
If only the new SecP hybrids cause problems, `GODEBUG=tlssecpmlkem=0` disables
those groups while retaining X25519MLKEM768.
For builds targeting Go 1.27, setting either `SSL_CERT_FILE` or `SSL_CERT_DIR`
on macOS replaces Keychain trust with on-disk roots and Go's verifier. Stale or
incomplete CA paths can break previously trusted connections; unset inherited
values to restore Keychain trust. Explicit certificates in the configured `CAs`
directory remain additive to the selected root pool.
Go 1.27 binaries require macOS 13 or later. See the
[Go release notes](https://go.dev/doc/go1.27) and the
Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md); every fix ships with a public [advisory](https://silo.pgsty.com/blog/security/). Contributions are accepted inbound=outbound under AGPL-3.0-or-later with no CLA — only DCO sign-off (`git commit -s`) is required; see [`CONTRIBUTING.md`](CONTRIBUTING.md).
## Contributors
**41 community contributors** build SILO, Console, mcli, shared packages, and related projects. The list includes maintainers and every human Issue or PR author, ordered by merged PRs, other PRs, then issue reports. Gold rings highlight significant contributions.
<a href="https://github.com/Kesavaambati"><img src="https://silo.pgsty.com/images/contributors/Kesavaambati.svg" width="48" height="48" alt="@Kesavaambati" title="@Kesavaambati — Asked about community support and image maintenance"></a>
</p>
[View the full contribution record](CONTRIBUTORS.md) for each person's proposals, fixes, and reports.
## Background
Upstream wound down its community edition: the web console was cut back to a stub, prebuilt community binaries stopped, and the community repository was archived. Silo exists to keep those deployments running. The fork is a means, not an identity — if upstream restores its community edition, we will narrow our scope and offer the fixes back.
The [**Manifesto**](https://silo.pgsty.com/about/manifesto/) is the project's public commitment in eleven articles, under one discipline: every article is either something already done with public evidence, or something explicitly refused. In short:
- **Compatibility contract** — the protocol and your data do not change, and every release documents its tested rollback target and path.
- **The license cannot change** — AGPLv3, no CLA, no copyright aggregation; nobody here, ourselves included, holds enough copyright to relicense on everyone else's behalf.
- **The never list**, append-only — no paywalling existing features, no registration wall on downloads, no telemetry (upstream's phone-home paths are removed outright), no CLA, no license change, no trademark enforcement against normal use.
- **Security and release discipline** — a public advisory for every fix, and a release every one to two months, at most a quarter apart. Judge both against the public record.
Essays: [MinIO Is Dead](https://silo.pgsty.com/blog/post/minio-is-dead/) · [Who Takes Over?](https://silo.pgsty.com/blog/post/minio-alternative/) · [Long Live MinIO](https://silo.pgsty.com/blog/post/minio-resurrect/) · [Promise Kept](https://silo.pgsty.com/blog/post/minio-promise-kept/)
## License & Trademark
Silo is [AGPL-3.0-or-later](LICENSE), derived from [`minio/minio`](https://github.com/minio/minio) with upstream copyright and third-party notices preserved in [`NOTICE`](NOTICE) and [`CREDITS`](CREDITS). MinIO is a trademark of MinIO, Inc.; the name is used here only to identify the upstream project and compatibility lineage.
The inherited [service-account](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/admin-handlers-users_test.go#L211-L212)
and [STS](https://github.com/pgsty/silo/blob/c1a49490c78e9c3ebcad86ba0662319138ace190/cmd/sts-handlers_test.go#L45-L46)
regression groups remain part of `go test ./cmd`; see the
All security bugs in [minio/minio](https://github,com/minio/minio) (or other minio/* repositories)
should be reported by email to security@min.io. Your email will be acknowledged within 48 hours,
and you'll receive a more detailed response to your email within 72 hours indicating the next steps
in handling your report.
For vulnerabilities in this fork:
Please, provide a detailed explanation of the issue. In particular, outline the type of the security
issue (DoS, authentication bypass, information disclose, ...) and the assumptions you're making (e.g. do
you need access credentials for a successful exploit).
1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
2. Prefer this repository's [private GitHub security advisory](https://github.com/pgsty/silo/security/advisories/new) workflow.
3. If private reporting is unavailable, contact the maintainers through the
repository without publishing exploit details until a private channel is
established.
4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.
If you have not received a reply to your email within 48 hours or you have not heard from the security team
for the past five days please contact the security team directly:
## Disclosure Process
- Primary security coordinator: aead@min.io
- Secondary coordinator: harsha@min.io
- If you receive no response: dev@min.io
### Disclosure Process
MinIO uses the following disclosure process:
1. Once the security report is received one member of the security team tries to verify and reproduce
the issue and determines the impact it has.
2. A member of the security team will respond and either confirm or reject the security report.
If the report is rejected the response explains why.
3. Code is audited to find any potential similar problems.
4. Fixes are prepared for the latest release.
5. On the date that the fixes are applied a security advisory will be published on <https://blog.min.io>.
Please inform us in your report email whether MinIO should mention your contribution w.r.t. fixing
the security issue. By default MinIO will **not** publish this information to protect your privacy.
This process can take some time, especially when coordination is required with maintainers of other projects.
Every effort will be made to handle the bug in as timely a manner as possible, however it's important that we
follow the process described above to ensure that disclosures are handled consistently.
Fork-specific fixes and user-visible upgrade notes are published in [docs/security/advisories.md](docs/security/advisories.md). The fork-specific triage and remediation process is described in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
"${PWD}/mc" stat silo/testbucket/testobject &&exit -1
# quick header write and slow bodywrite
send_put_object_request 040&&exit -1
"${PWD}/mc" stat silo/testbucket/testobject &&exit -1
# quick header and body write
send_put_object_request 11||exit -1
"${PWD}/mc" stat silo/testbucket/testobject ||exit -1
}
function main(){
exportstart_port=$(shuf -i 10000-65000 -n 1)
exportsrv_hdr_timeout=5
exportsrv_idle_timeout=5
export -f gen_put_request
test_silo_with_timeout ${start_port}
}
main "$@"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Feng Ruohang
