Commit Graph

13361 Commits

Author SHA1 Message Date
Feng Ruohang 1d714834c5 fix(iam): normalize service policies and preserve imported state
Keep omitted edit policies unchanged, reset valid empty policies to inheritance, and normalize nonempty policies without a version before storage. Preserve caller policy objects and exclude derived authorization claims from snapshots and new JWTs.

Restore imported service-account and group status, report the actual group error, and cover creation, editing, replication, import/export and raw JWT claims with focused server tests.

Refs #339, #345.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 86e4c09d47f48cb2f44fb2f63f02f222ac6ae102)
2026-11-11 11:17:55 +08:00
Feng Ruohang 14ecb222e7 fix(auth): secure proxied tagging and streaming trailers
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 11:13:26 +08:00
Feng Ruohang e51bff35a3 Merge service-account replication fix into main
Include the remaining committed IAM snapshot fix from claude/lucid-hopper-811f3d. Preserve the current contributor inventory and previously integrated fixes while adding the service-account replication release note.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:12:08 +08:00
Feng Ruohang 20b3bfb6e7 Merge verified prerelease fixes into main
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:08:25 +08:00
Feng Ruohang 322cd8d2c1 fix(decom): count lifecycle removals as scan progress
Reuse the source-version lookup to count versions removed by ILM before migration cleanup so nested keys are discovered on the next pass. Keep delete-marker-only expiry from reporting physical progress, and cover both the decommission regression and the removal-count boundary.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:08:07 +08:00
Feng Ruohang c3340ec7f5 fix(auth): scope unread tag candidates to effective policies
Select cached user, group, role and session policies without lazy IAM reads. Keep alternative tag values and Deny keys when combining candidates, and bound intermediate combinations as well as authorization attempts.

Cover unrelated cached policies, intersecting tag conditions, temporary credentials, root-derived credentials and cache-only lookup regressions for issue #293.

(cherry picked from commit 65acda0114)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:32:29 +08:00
Feng Ruohang 7c91dc17b2 Merge remote-tracking branch 'origin/main'
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:31:32 +08:00
Feng Ruohang 2f93e0c6b3 Merge pull request #294 from botdeniska/fix/tag-auth-server-error
Preserve server errors for reads authorized by object tags and GetObjectAttributes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:30:24 +08:00
Feng Ruohang 65acda0114 fix(auth): scope unread tag candidates to effective policies
Select cached user, group, role and session policies without lazy IAM reads. Keep alternative tag values and Deny keys when combining candidates, and bound intermediate combinations as well as authorization attempts.

Cover unrelated cached policies, intersecting tag conditions, temporary credentials, root-derived credentials and cache-only lookup regressions for issue #293.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:22:57 +08:00
Denis Babchuk c6949a38c2 fix(api): keep GetObjectAttributes server errors and check s3:ListBucket on the bucket
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit a2ff3c2ecd)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 80dd3e3ce1 test(api): cover GetObjectAttributes errors with and without s3:ListBucket
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit 67bc32f401)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 89028732ab fix(auth): keep the server error when a read allowed by object tags cannot read the tags
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit 3c4a6296f9)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 55bd906d30 test(auth): cover reads allowed by s3:ExistingObjectTag when the object cannot be read
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit afa35d16e9)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Feng Ruohang 296c9710b1 fix(security): close release blockers
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 21:51:47 +08:00
Feng Ruohang f3b42b0eb8 Merge pull request #335 from botdeniska/fix/decom-keys-below-a-key
fix(decom): repeat the walk of a bucket while it removes source versions

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 21:30:18 +08:00
Feng Ruohang da8d8a54b1 fix(iam): replicate a service account as the record that is stored
Site replication sent an edit of a service account as the fields the
request changed. A peer orders the items of an account by revision and
drops an older one, which is right for a whole account and loses data for
a part of one: when an edit did not reach a peer, or reached it after a
later edit, the peer kept the old secret key, status or inline policy
under the revision of the later edit, and healing took the sites for
equal. Two requests at one site sufficed, as the sender dropped the item
of the first when the second was already stored.

The first sync of a new site, healing and IAM export put an account
together from two places: the revision from storage, and the claims and
the inline policy from the cache of the server; healing took the secret
key and the status from the cache as well. A server whose cache had not
heard of an edit made through another server sent the account from
before the edit under the revision of the edit, and a peer kept that.

replicationItem now sends a create or an edit as the account stored at
that time: the whole record, read once, with its revision, as it does
for a group. serviceAccountSnapshot builds it from the record and the
token in it; IAM export uses the same function, and sync, heal and the
update handler only name the account. The wire types and the way a peer
applies an item do not change: a peer on 20261109 or 20261026 takes such
an edit as it takes a healed account, which also means that it keeps the
inline policy on an edit that names none. A site that is not upgraded
still sends its edits as before.

The hook reads what it sends before it takes the lock of the replication
configuration: loading an identity takes that lock for reading, and a
read lock asked for again behind a waiting writer is never granted. An
account whose stored token cannot be read is an error, as it was for sync
and heal, and is not taken for deleted.

A new service account of a user that was deleted and created again now
reaches the peers without waiting for healing: the create carries the
claims of the token, among them the parent revocation a peer asks for.

An edit gives every site the whole account of the site that took it.
Accounts that already differ between sites under one revision are not
repaired by this change, only by their next edit.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 20:19:54 +08:00
Feng Ruohang 2b62df876a docs(contributors): cite the real CVE-2026-39414 follow-up commit
Regenerated from the contributor data on silo.pgsty.com. One note named
5dc705936, which is not a commit in any repository; the June follow-up
is fd69c89d0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 17:24:28 +08:00
Feng Ruohang 518915d241 docs(contributors): credit upstream advisory researchers and fix a commit link
Regenerated from the reviewed contributor data on silo.pgsty.com. The roll
grows from 76 to 81 people: it now names the researchers credited on the
upstream advisories for inherited CVEs. The link to the Helm chart 7.0.4
commit carried a full hash that matches no commit; it now points at
47275042f.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 16:57:15 +08:00
Denis Babchuk ce5d191077 fix(decom): repeat the walk of a bucket while it removes source versions
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-11-10 11:48:42 +03:00
Denis Babchuk b404f78764 test(decom): cover keys below a key that the walk does not see
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-11-10 11:48:42 +03:00
Feng Ruohang d138996983 docs(helm): fix NOTES connection commands and document 20261109 upgrade gates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:41:49 +08:00
Feng Ruohang d52ba0290a docs: align release, TLS, healthcheck, and contributor docs with 20261109
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:41:49 +08:00
Feng Ruohang 8f3657dc52 Merge pull request #333 from pgsty/claude/cross-bucket-path-traversal-delete-5e1fc2
fix(pools): keep rejected names out of a single-pool bulk delete
2026-11-10 14:39:36 +08:00
Feng Ruohang 7efdedd5ba Merge pull request #329 from pgsty/deps/console-2.7.0
deps: pin SILO Console 2.7.0
2026-11-10 14:30:02 +08:00
Feng Ruohang a18293d0c3 fix(pools): keep rejected names out of a single-pool bulk delete
DeleteObjects checks every name with checkDelObjArgs, but a single pool
still sent all entries to the drives and cleared the error of an entry
that a drive reported as found. Local drives join the name to the bucket
directory, so a name with a ".." segment reached another bucket or the
meta bucket. An S3 DeleteObjects caller allowed to delete in one bucket
could remove a version elsewhere, including one under Object Lock
retention, and from a versioned bucket could write a delete marker on an
object of another bucket or on stored bucket metadata. The bulk lock was
taken on the normalized name as well, that is on the other object.

Entries that fail the check now keep their error, and the rest of the
request runs as a request of its own, before any lock or lookup. A
request whose names are all valid does not enter the new code. More than
one pool already skipped such entries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:24:35 +08:00
Feng Ruohang 032ac49061 Merge pull request #332 from pgsty/claude/multi-pool-ilm-version-dup-210154
fix(ilm): count a version under old and healed headers as one revision
2026-11-10 14:14:32 +08:00
Feng Ruohang 244de7ecc1 Merge pull request #331 from pgsty/claude/svcacct-inline-policy-bug-961eaa
fix(iam): keep a service account's inline policy on an edit that names none
2026-11-10 14:12:36 +08:00
Feng Ruohang 63650a31a1 fix(iam): keep a service account's inline policy on an edit that names none
An update-service-account request without a policy removed the inline
policy of the account: the store took a nil session policy for an empty
one, and the handler turned an empty one into nil, so "not given" and
"given, empty" were one case. A status change, a secret key rotation, a
rename or a new expiry (mcli admin user svcacct disable, enable, and edit
without --policy, and the idp ldap accesskey equivalents) signed the token
again as inherited-policy, and the account then had all its parent is
allowed. A replication peer did the same with the forwarded edit.

A nil session policy now leaves the policy as it is. The handler passes an
empty policy document on as such, and only that removes the inline policy,
as mcli admin user svcacct edit --policy with {"Statement":[]} already
asks for. A JSON null policy counts as none. The peer handler passes nil
for an edit without a policy and needs no change.

What an update does with any other policy document is unchanged.

The behaviour came with upstream f0462322f (minio/minio#14847).

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 13:47:09 +08:00
Feng Ruohang 36c8d1683a fix(ilm): count a version under old and healed headers as one revision
The version history that an ILM expiry reads accepts a version only
when one revision of it is on more than half of the drives and on its
read quorum, and a revision was the header as the drive stores it plus
a digest of the metadata. A drive holds the header that the release
which wrote the version there stored: before RELEASE.2024-04-28
without erasure coding, and until RELEASE.2022-01-25 with a signature
that counted empty part ETags. A heal writes the header of its own
release and leaves the other drives alone. Reads and heal take both
for one version; the history counted two revisions. With 5 to 11 of 16
drives healed (EC:4) neither had the quorum, objectVersions failed
with a read quorum error, and every expiry that reads the history was
refused on each scanner cycle, with nothing to bring the drives
together: every expiry in a versioned bucket with several pools, and
an expiry of all versions of a key with one pool too.

versionRevision now gives the key of an object version the signature
and the erasure coding as the current release computes them from the
decoded metadata. The key of a version this release wrote carries its
header as stored, so its history decides as before, including the
order of versions of one time, which the signature breaks. The digest
covers what both fields say. The stored flags stay in the key: whether
a transitioned version uses its data directory follows the moment a
drive wrote the entry (fileInfoRevisions).

TestPoolsLifecycleHeaderEras accepts both old headers on every count
of healed drives. TestPoolsLifecycleHealedOldHeaders expires a
noncurrent version with two pools, and all versions with one, after
half of the drives were wiped and healed. Both fail before this
change. Drives that differ only in the data directory flag of a
restored version are still two revisions; that is a separate defect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 13:45:37 +08:00
Feng Ruohang c11bfda8f4 fix(iam): keep a service account's inline policy on an edit that names none
An update-service-account request without a policy removed the inline
policy of the account: the store took a nil session policy for an empty
one, and the handler turned an empty one into nil, so "not given" and
"given, empty" were one case. A status change, a secret key rotation, a
rename or a new expiry (mcli admin user svcacct disable, enable, and edit
without --policy, and the idp ldap accesskey equivalents) signed the token
again as inherited-policy, and the account then had all its parent is
allowed. A replication peer did the same with the forwarded edit.

A nil session policy now leaves the policy as it is. The handler passes an
empty policy document on as such, and only that removes the inline policy,
as mcli admin user svcacct edit --policy with {"Statement":[]} already
asks for. A JSON null policy counts as none. The peer handler passes nil
for an edit without a policy and needs no change.

What an update does with any other policy document is unchanged.

The behaviour came with upstream f0462322f (minio/minio#14847).

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 11:55:28 +08:00
Feng Ruohang 6a65b43076 docs(helm): fix NOTES connection commands and document 20261109 upgrade gates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 11:00:05 +08:00
Feng Ruohang 3269d2214e docs: align release, TLS, healthcheck, and contributor docs with 20261109
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 11:00:05 +08:00
Feng Ruohang b3e69bca2f fix(pools): keep rejected names out of a single-pool bulk delete
DeleteObjects checks every name with checkDelObjArgs, but a single pool
still sent all entries to the drives and cleared the error of an entry
that a drive reported as found. Local drives join the name to the bucket
directory, so a name with a ".." segment reached another bucket or the
meta bucket. An S3 DeleteObjects caller allowed to delete in one bucket
could remove a version elsewhere, including one under Object Lock
retention, and from a versioned bucket could write a delete marker on an
object of another bucket or on stored bucket metadata. The bulk lock was
taken on the normalized name as well, that is on the other object.

Entries that fail the check now keep their error, and the rest of the
request runs as a request of its own, before any lock or lookup. A
request whose names are all valid does not enter the new code. More than
one pool already skipped such entries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 10:25:08 +08:00
Feng Ruohang d998292387 Merge pull request #330 from pgsty/codex/release-20261109-closeout
Enforce Object Lock during S3 bulk version deletion and clarify upgrade readiness.
RELEASE.2026-11-09T00-00-00Z
2026-11-10 00:09:45 +08:00
Feng Ruohang 6ef3a52fc3 Enforce Object Lock during S3 bulk version deletion
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 23:53:43 +08:00
Feng Ruohang e21b1c3a71 docs(healthcheck): say what ready does not wait for
The ready probe turns healthy before storage and IAM load, by design, so
that distributed nodes can reach each other during startup. Point
startup ordering such as Compose service_healthy at the cluster check,
and keep publishing not-ready addresses for StatefulSet peers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 22:07:59 +08:00
Feng Ruohang 86dc6ee2e2 fix(restore): return from a failed restore without waiting for its cleanup
A failed or canceled native restore cleared its ongoing marker under a
fresh one-minute deadline, while the restore itself could still hold the
object lock that the cleanup waits for. The request then stalled for
about a minute (60 s in TestNativeRestoreWithoutOuterLock and
TestRestoreRecoversStrandedMarker).

Try the cleanup for 100 ms and leave the rest to the bounded background
retries, which already exist for the pools path and now also serve the
single-set path. Each retry still takes the object lock afresh and clears
only the marker of its own attempt. The CI restore race step drops from
about 260 s to 37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 22:07:59 +08:00
Feng Ruohang 0a238589a4 ci: gate releases on the new Object Lock delete tests
The release workflow keeps its Object Lock race gate aligned with the
matching step in go.yml. Add the replicated-delete and directory bulk
delete tests there too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:26:02 +08:00
Feng Ruohang 1f731549e4 docs: prepare SILO 20261109
Record RELEASE.2026-11-09T00-00-00Z (package 20261109000000.0.0) in the
changelog: the fixes since 20261026, Console v2.7.0, mcli 20261109 and
silo-pkg v3.16.0, upgrade notes and known issues. Point the README quick
starts at the new image, replace the 20261026 release section with the
20261109 upgrade notice in both languages, refresh the contributor roll
to 2026-11-09, and add a second security contact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:05:22 +08:00
Feng Ruohang 368c52f4dc build(deps): embed the published Console v2.7.0
Console v2.7.0 was cut again at 4e91ae506e70 with the review fixes that
followed 26ce922424f1. Select that commit; no other module changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:02:16 +08:00
Feng Ruohang f0a4d76ef8 Merge #328: Console 2.7, mcli 20261109 and silo-pkg 3.16
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 21:00:20 +08:00
Feng Ruohang 2c06b28dbb deps: pin SILO Console 2.7.0
Pin the Console module to the v2.7.0 tag commit (4e91ae506e70), published
on 2026-11-09. It adds ten UI languages with Arabic right-to-left layout,
fixes usage-worker TLS, scheduling and shutdown, serves folder sizes over
plain HTTP, and fixes the container CA bundle and /tmp handling.

Console 2.7.0 requires silo-pkg v3.16.0 and builds with the mcli
RELEASE.2026-11-09T00-00-00Z source (pgsty/mc e1d4cc5abfe0). Go ignores
replacements declared by dependencies, so copy Console's single mc
replacement here and raise the silo-pkg requirement to match. silo-pkg
v3.16.0 changes only a policy comment and dependency floors.

Minimal version selection also raises klauspost/cpuid/v2 to v2.4.2,
go.yaml.in/yaml/v3 to v3.0.6, go-jose/v4 to v4.1.9, go-openapi/runtime to
v0.33.4 and tidwall/gjson to v1.20.1. minio-go stays at 320af050c445.
CREDITS is unchanged after regeneration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 20:49:43 +08:00
Feng Ruohang aa2d79daf2 fix(object-lock): check a directory object's lock before a bulk delete
A directory object always has the null version. DeleteObjects without a
version ID chose whether to read the object before it turned the version
into null, so with Object Lock and no replication or tiering it checked
retention against empty metadata and removed a directory object under
legal hold or retention. Choose the null version before the lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:25:33 +08:00
Feng Ruohang 7e9ee6befc fix(replication): keep Object Lock on a replicated delete
A trusted replica DELETE skipped the destination's retention check for
every request, not only for the delete marker it is meant to create. A
caller allowed s3:ReplicateDelete (root, or the replication account)
could purge a version under legal hold or unexpired retention, or name
a stored version in a delete marker, which replaces that version.

Skip the check only for a delete-marker creation, whose version is not
yet stored here, and make that request refuse to replace a protected
version with RefuseLockedOverwrite, as batch replication pulls do; the
pools layer applies it to the version in any pool. A replicated purge
goes through the normal retention check. A source purge that bypassed
GOVERNANCE retention is now refused by a target whose copy still
carries it; the source records the purge as failed and retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:13:21 +08:00
Feng Ruohang 47275042f6 chore(helm): deploy SILO and mcli 20261109 by default
The 20261026 release moved mcImage.tag but left image.tag, which the
templates deploy, at 20260916 (#296). Point the server and mcli images
and appVersion at RELEASE.2026-11-09T00-00-00Z and bump the chart to
7.0.4.

The Helm check now reads the chart version instead of a fixed file name,
and fails when image.tag differs from appVersion, so a release cannot
move one without the other again.

Upgrades that pass --reuse-values keep the old image.tag; set image.tag
explicitly when upgrading such a release.

Fixes #296

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:03:32 +08:00
Feng Ruohang d66b3889cd ci: give the native restore race tests room to finish
The lifecycle expiry and native restore race step takes about 260 s on
an M-series Mac against its 5 minute limit, and the job holding it was
already near 9 of its 15 minutes before that step was added. GitHub's
hosted runners are slower, so allow the step 10 minutes and the job 25,
as the other jobs have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 19:03:32 +08:00
Feng Ruohang 099b7ddf37 fix(replication): handle absent pool in worker metrics
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 18:10:19 +08:00
Feng Ruohang c054569b9f Update SILO to Console 2.7 and maintained dependency fixes
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 17:24:59 +08:00
Feng Ruohang 1f579afeed Merge origin/main: conditional multipart initiation (#292)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 13:39:10 +08:00
Feng Ruohang af3064f2b5 test(pools): cover initiate locking and completion conditions
Exercise concurrent overwrite and delete while initiation persists metadata, and completion after intervening object mutations. Check that failed conditions preserve uploaded parts for retry. Include conditional multipart regressions in the CI race set and record the multipart fix from Denis Babchuk in the contributor history.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-09 13:37:18 +08:00