Commit Graph

13353 Commits

Author SHA1 Message Date
Feng Ruohang 20b3bfb6e7 Merge verified prerelease fixes into main
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:08:25 +08:00
Feng Ruohang 322cd8d2c1 fix(decom): count lifecycle removals as scan progress
Reuse the source-version lookup to count versions removed by ILM before migration cleanup so nested keys are discovered on the next pass. Keep delete-marker-only expiry from reporting physical progress, and cover both the decommission regression and the removal-count boundary.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:08:07 +08:00
Feng Ruohang c3340ec7f5 fix(auth): scope unread tag candidates to effective policies
Select cached user, group, role and session policies without lazy IAM reads. Keep alternative tag values and Deny keys when combining candidates, and bound intermediate combinations as well as authorization attempts.

Cover unrelated cached policies, intersecting tag conditions, temporary credentials, root-derived credentials and cache-only lookup regressions for issue #293.

(cherry picked from commit 65acda0114)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:32:29 +08:00
Feng Ruohang 7c91dc17b2 Merge remote-tracking branch 'origin/main'
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:31:32 +08:00
Feng Ruohang 2f93e0c6b3 Merge pull request #294 from botdeniska/fix/tag-auth-server-error
Preserve server errors for reads authorized by object tags and GetObjectAttributes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:30:24 +08:00
Feng Ruohang 65acda0114 fix(auth): scope unread tag candidates to effective policies
Select cached user, group, role and session policies without lazy IAM reads. Keep alternative tag values and Deny keys when combining candidates, and bound intermediate combinations as well as authorization attempts.

Cover unrelated cached policies, intersecting tag conditions, temporary credentials, root-derived credentials and cache-only lookup regressions for issue #293.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:22:57 +08:00
Denis Babchuk c6949a38c2 fix(api): keep GetObjectAttributes server errors and check s3:ListBucket on the bucket
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit a2ff3c2ecd)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 80dd3e3ce1 test(api): cover GetObjectAttributes errors with and without s3:ListBucket
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit 67bc32f401)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 89028732ab fix(auth): keep the server error when a read allowed by object tags cannot read the tags
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit 3c4a6296f9)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 55bd906d30 test(auth): cover reads allowed by s3:ExistingObjectTag when the object cannot be read
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit afa35d16e9)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Feng Ruohang 296c9710b1 fix(security): close release blockers
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 21:51:47 +08:00
Feng Ruohang f3b42b0eb8 Merge pull request #335 from botdeniska/fix/decom-keys-below-a-key
fix(decom): repeat the walk of a bucket while it removes source versions

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 21:30:18 +08:00
Feng Ruohang 2b62df876a docs(contributors): cite the real CVE-2026-39414 follow-up commit
Regenerated from the contributor data on silo.pgsty.com. One note named
5dc705936, which is not a commit in any repository; the June follow-up
is fd69c89d0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 17:24:28 +08:00
Feng Ruohang 518915d241 docs(contributors): credit upstream advisory researchers and fix a commit link
Regenerated from the reviewed contributor data on silo.pgsty.com. The roll
grows from 76 to 81 people: it now names the researchers credited on the
upstream advisories for inherited CVEs. The link to the Helm chart 7.0.4
commit carried a full hash that matches no commit; it now points at
47275042f.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 16:57:15 +08:00
Denis Babchuk ce5d191077 fix(decom): repeat the walk of a bucket while it removes source versions
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-11-10 11:48:42 +03:00
Denis Babchuk b404f78764 test(decom): cover keys below a key that the walk does not see
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-11-10 11:48:42 +03:00
Feng Ruohang d138996983 docs(helm): fix NOTES connection commands and document 20261109 upgrade gates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:41:49 +08:00
Feng Ruohang d52ba0290a docs: align release, TLS, healthcheck, and contributor docs with 20261109
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:41:49 +08:00
Feng Ruohang 8f3657dc52 Merge pull request #333 from pgsty/claude/cross-bucket-path-traversal-delete-5e1fc2
fix(pools): keep rejected names out of a single-pool bulk delete
2026-11-10 14:39:36 +08:00
Feng Ruohang 7efdedd5ba Merge pull request #329 from pgsty/deps/console-2.7.0
deps: pin SILO Console 2.7.0
2026-11-10 14:30:02 +08:00
Feng Ruohang a18293d0c3 fix(pools): keep rejected names out of a single-pool bulk delete
DeleteObjects checks every name with checkDelObjArgs, but a single pool
still sent all entries to the drives and cleared the error of an entry
that a drive reported as found. Local drives join the name to the bucket
directory, so a name with a ".." segment reached another bucket or the
meta bucket. An S3 DeleteObjects caller allowed to delete in one bucket
could remove a version elsewhere, including one under Object Lock
retention, and from a versioned bucket could write a delete marker on an
object of another bucket or on stored bucket metadata. The bulk lock was
taken on the normalized name as well, that is on the other object.

Entries that fail the check now keep their error, and the rest of the
request runs as a request of its own, before any lock or lookup. A
request whose names are all valid does not enter the new code. More than
one pool already skipped such entries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:24:35 +08:00
Feng Ruohang 032ac49061 Merge pull request #332 from pgsty/claude/multi-pool-ilm-version-dup-210154
fix(ilm): count a version under old and healed headers as one revision
2026-11-10 14:14:32 +08:00
Feng Ruohang 244de7ecc1 Merge pull request #331 from pgsty/claude/svcacct-inline-policy-bug-961eaa
fix(iam): keep a service account's inline policy on an edit that names none
2026-11-10 14:12:36 +08:00
Feng Ruohang 36c8d1683a fix(ilm): count a version under old and healed headers as one revision
The version history that an ILM expiry reads accepts a version only
when one revision of it is on more than half of the drives and on its
read quorum, and a revision was the header as the drive stores it plus
a digest of the metadata. A drive holds the header that the release
which wrote the version there stored: before RELEASE.2024-04-28
without erasure coding, and until RELEASE.2022-01-25 with a signature
that counted empty part ETags. A heal writes the header of its own
release and leaves the other drives alone. Reads and heal take both
for one version; the history counted two revisions. With 5 to 11 of 16
drives healed (EC:4) neither had the quorum, objectVersions failed
with a read quorum error, and every expiry that reads the history was
refused on each scanner cycle, with nothing to bring the drives
together: every expiry in a versioned bucket with several pools, and
an expiry of all versions of a key with one pool too.

versionRevision now gives the key of an object version the signature
and the erasure coding as the current release computes them from the
decoded metadata. The key of a version this release wrote carries its
header as stored, so its history decides as before, including the
order of versions of one time, which the signature breaks. The digest
covers what both fields say. The stored flags stay in the key: whether
a transitioned version uses its data directory follows the moment a
drive wrote the entry (fileInfoRevisions).

TestPoolsLifecycleHeaderEras accepts both old headers on every count
of healed drives. TestPoolsLifecycleHealedOldHeaders expires a
noncurrent version with two pools, and all versions with one, after
half of the drives were wiped and healed. Both fail before this
change. Drives that differ only in the data directory flag of a
restored version are still two revisions; that is a separate defect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 13:45:37 +08:00
Feng Ruohang c11bfda8f4 fix(iam): keep a service account's inline policy on an edit that names none
An update-service-account request without a policy removed the inline
policy of the account: the store took a nil session policy for an empty
one, and the handler turned an empty one into nil, so "not given" and
"given, empty" were one case. A status change, a secret key rotation, a
rename or a new expiry (mcli admin user svcacct disable, enable, and edit
without --policy, and the idp ldap accesskey equivalents) signed the token
again as inherited-policy, and the account then had all its parent is
allowed. A replication peer did the same with the forwarded edit.

A nil session policy now leaves the policy as it is. The handler passes an
empty policy document on as such, and only that removes the inline policy,
as mcli admin user svcacct edit --policy with {"Statement":[]} already
asks for. A JSON null policy counts as none. The peer handler passes nil
for an edit without a policy and needs no change.

What an update does with any other policy document is unchanged.

The behaviour came with upstream f0462322f (minio/minio#14847).

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 11:55:28 +08:00
Feng Ruohang d998292387 Merge pull request #330 from pgsty/codex/release-20261109-closeout
Enforce Object Lock during S3 bulk version deletion and clarify upgrade readiness.
RELEASE.2026-11-09T00-00-00Z
2026-11-10 00:09:45 +08:00
Feng Ruohang 6ef3a52fc3 Enforce Object Lock during S3 bulk version deletion
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 23:53:43 +08:00
Feng Ruohang e21b1c3a71 docs(healthcheck): say what ready does not wait for
The ready probe turns healthy before storage and IAM load, by design, so
that distributed nodes can reach each other during startup. Point
startup ordering such as Compose service_healthy at the cluster check,
and keep publishing not-ready addresses for StatefulSet peers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 22:07:59 +08:00
Feng Ruohang 86dc6ee2e2 fix(restore): return from a failed restore without waiting for its cleanup
A failed or canceled native restore cleared its ongoing marker under a
fresh one-minute deadline, while the restore itself could still hold the
object lock that the cleanup waits for. The request then stalled for
about a minute (60 s in TestNativeRestoreWithoutOuterLock and
TestRestoreRecoversStrandedMarker).

Try the cleanup for 100 ms and leave the rest to the bounded background
retries, which already exist for the pools path and now also serve the
single-set path. Each retry still takes the object lock afresh and clears
only the marker of its own attempt. The CI restore race step drops from
about 260 s to 37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 22:07:59 +08:00
Feng Ruohang 0a238589a4 ci: gate releases on the new Object Lock delete tests
The release workflow keeps its Object Lock race gate aligned with the
matching step in go.yml. Add the replicated-delete and directory bulk
delete tests there too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:26:02 +08:00
Feng Ruohang 1f731549e4 docs: prepare SILO 20261109
Record RELEASE.2026-11-09T00-00-00Z (package 20261109000000.0.0) in the
changelog: the fixes since 20261026, Console v2.7.0, mcli 20261109 and
silo-pkg v3.16.0, upgrade notes and known issues. Point the README quick
starts at the new image, replace the 20261026 release section with the
20261109 upgrade notice in both languages, refresh the contributor roll
to 2026-11-09, and add a second security contact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:05:22 +08:00
Feng Ruohang 368c52f4dc build(deps): embed the published Console v2.7.0
Console v2.7.0 was cut again at 4e91ae506e70 with the review fixes that
followed 26ce922424f1. Select that commit; no other module changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:02:16 +08:00
Feng Ruohang f0a4d76ef8 Merge #328: Console 2.7, mcli 20261109 and silo-pkg 3.16
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:00:20 +08:00
Feng Ruohang 2c06b28dbb deps: pin SILO Console 2.7.0
Pin the Console module to the v2.7.0 tag commit (4e91ae506e70), published
on 2026-11-09. It adds ten UI languages with Arabic right-to-left layout,
fixes usage-worker TLS, scheduling and shutdown, serves folder sizes over
plain HTTP, and fixes the container CA bundle and /tmp handling.

Console 2.7.0 requires silo-pkg v3.16.0 and builds with the mcli
RELEASE.2026-11-09T00-00-00Z source (pgsty/mc e1d4cc5abfe0). Go ignores
replacements declared by dependencies, so copy Console's single mc
replacement here and raise the silo-pkg requirement to match. silo-pkg
v3.16.0 changes only a policy comment and dependency floors.

Minimal version selection also raises klauspost/cpuid/v2 to v2.4.2,
go.yaml.in/yaml/v3 to v3.0.6, go-jose/v4 to v4.1.9, go-openapi/runtime to
v0.33.4 and tidwall/gjson to v1.20.1. minio-go stays at 320af050c445.
CREDITS is unchanged after regeneration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 20:49:43 +08:00
Feng Ruohang aa2d79daf2 fix(object-lock): check a directory object's lock before a bulk delete
A directory object always has the null version. DeleteObjects without a
version ID chose whether to read the object before it turned the version
into null, so with Object Lock and no replication or tiering it checked
retention against empty metadata and removed a directory object under
legal hold or retention. Choose the null version before the lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:25:33 +08:00
Feng Ruohang 7e9ee6befc fix(replication): keep Object Lock on a replicated delete
A trusted replica DELETE skipped the destination's retention check for
every request, not only for the delete marker it is meant to create. A
caller allowed s3:ReplicateDelete (root, or the replication account)
could purge a version under legal hold or unexpired retention, or name
a stored version in a delete marker, which replaces that version.

Skip the check only for a delete-marker creation, whose version is not
yet stored here, and make that request refuse to replace a protected
version with RefuseLockedOverwrite, as batch replication pulls do; the
pools layer applies it to the version in any pool. A replicated purge
goes through the normal retention check. A source purge that bypassed
GOVERNANCE retention is now refused by a target whose copy still
carries it; the source records the purge as failed and retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:13:21 +08:00
Feng Ruohang 47275042f6 chore(helm): deploy SILO and mcli 20261109 by default
The 20261026 release moved mcImage.tag but left image.tag, which the
templates deploy, at 20260916 (#296). Point the server and mcli images
and appVersion at RELEASE.2026-11-09T00-00-00Z and bump the chart to
7.0.4.

The Helm check now reads the chart version instead of a fixed file name,
and fails when image.tag differs from appVersion, so a release cannot
move one without the other again.

Upgrades that pass --reuse-values keep the old image.tag; set image.tag
explicitly when upgrading such a release.

Fixes #296

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:03:32 +08:00
Feng Ruohang d66b3889cd ci: give the native restore race tests room to finish
The lifecycle expiry and native restore race step takes about 260 s on
an M-series Mac against its 5 minute limit, and the job holding it was
already near 9 of its 15 minutes before that step was added. GitHub's
hosted runners are slower, so allow the step 10 minutes and the job 25,
as the other jobs have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:03:32 +08:00
Feng Ruohang 099b7ddf37 fix(replication): handle absent pool in worker metrics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 18:10:19 +08:00
Feng Ruohang c054569b9f Update SILO to Console 2.7 and maintained dependency fixes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 17:24:59 +08:00
Feng Ruohang 1f579afeed Merge origin/main: conditional multipart initiation (#292)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 13:39:10 +08:00
Feng Ruohang af3064f2b5 test(pools): cover initiate locking and completion conditions
Exercise concurrent overwrite and delete while initiation persists metadata, and completion after intervening object mutations. Check that failed conditions preserve uploaded parts for retry. Include conditional multipart regressions in the CI race set and record the multipart fix from Denis Babchuk in the contributor history.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 13:37:18 +08:00
Feng Ruohang 9d2be5daaa chore(batch): keep the policy snapshot out of msgp generation
batchPolicySnapshot holds the IAM store, a context and a mutex. Without
an ignore directive, go generate emitted msgp encoders for it that do not
compile, so make check-gen (and the check-gen step of CI) failed on the
generated batch-authz files. The committed generated files are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit a6571a53e9)
2026-11-09 13:37:18 +08:00
Feng Ruohang 1f3377862b Merge pull request #292 from botdeniska/fix/initiate-precondition-pools
Evaluate conditional multipart initiation against the logical current object across pools while retaining pool routing and holding the object lock through upload placement.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 13:34:09 +08:00
Feng Ruohang b14dc454fb Merge origin/main: one trust store for both server images, mcli notices (#295)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 09:47:04 +08:00
Feng Ruohang 16738c2301 build(docker): ship mcli's legal files and keep base digests current
Copy LICENSE, NOTICE and CREDITS from the same verified mcli archive as
the bundled binary into /licenses/mcli; the server's CREDITS covers only
the server's own dependencies.

Correct the mcli archive pins to the RELEASE.2026-10-26T00-00-00Z
checksums. The release workflow overrides them from the attested
checksum file, but a plain local build failed with "Published checksum
drift".

Add a Dependabot docker update so the digest-pinned bases in
Dockerfile.goreleaser and Dockerfile.distroless keep receiving fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 09:38:37 +08:00
Feng Ruohang 53a8f7583f fix(docker): make the distroless image trust the same roots as the classic image
The distroless variant inherited Debian's CA bundle, which still contains
the two roots (Comodo AAA Services root, emSign ECC Root CA - G3) that the
classic image blocklists per RHSB-2026-012. Build the same UBI bundle in a
throwaway stage and replace the only file under /etc/ssl/certs with it.

Pin the busybox and distroless bases by digest like Dockerfile.goreleaser,
and gate the shipped file in test-release: the bundle must parse and must
not contain either blocklisted subject.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 09:38:37 +08:00
Feng Ruohang bb31c26183 fix: close IAM loading and batch recovery review gaps
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-08 18:52:50 +08:00
Feng Ruohang 07ce95dfda fix(iam): protect stale creates and cache loads
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-08 14:19:49 +08:00
Feng Ruohang 7eed95bc8e fix(iam): drop a stale cached identity when its edit is refused
70914759e refused an edit whose stored record no longer matched the
caller's view, but left the stale copy in the serving server's cache in
two cases: a service-account edit bound to a parent whose account was
deleted and created again for another parent, and a replicated edit older
than what is stored, which site replication acknowledges. That server
kept authenticating the old credential until a notification or the
periodic load.

Both refusals now drop the cache entry of the addressed kind, as a
refusal for an absent or deleted identity does: the service-account
closure reports such a record as gone (callers still get
errNoSuchServiceAccount), and editStoredIdentity also drops the entry on
errIAMStaleUpdate. The LDAP group refresh again treats a stale update as
a refusal that writes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-08 14:18:41 +08:00