Commit Graph

13386 Commits

Author SHA1 Message Date
Feng Ruohang 2ed720a7fb fix(crypto): return client errors for malformed KMS contexts
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-12 07:49:47 +08:00
Feng Ruohang c39605d4f8 fix(auth): require tag read permission for object tag counts
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-12 07:49:47 +08:00
Feng Ruohang 7c0a99284f fix(read): handle indexed ranges within the first encrypted block
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-12 07:49:47 +08:00
Feng Ruohang 34ba7cc1ff Prepare SILO 20261112 release candidate
Harden IAM/bucket/import/replication policy validation, enforce tag and
version authorization across read/copy/list/archive endpoints, bind SigV4
credential-scope dates and streaming trailer limits, validate HTTP SSE-KMS
contexts as strict UTF-8 JSON, add release-commit and coordinated
dependency gates, and pin silo-pkg v3.16.1, Console v2.7.1, mcli 20261112,
Go 1.27.4, and Helm chart 7.0.5.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-12 03:06:15 +08:00
Feng Ruohang bae0ba0b75 docs(changelog): sync unreleased decommission, tagging and replication notes
Align unreleased server notes with main commits 293ab78b4 and 8587e061b and the completed site-replication topology snapshot, pooled copy rewrite, and batch KMS UTF-8 fixes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:49:35 +08:00
Feng Ruohang cd3e46a2b7 fix(copy): preserve compression index on pooled copy rewrites
Forward dstOpts.IndexCB into erasureServerPools.CopyObject's PutObject options so multi-block S2 compressed copies (> compMinIndexSize) rewritten across pools persist Parts[0].Index for O(1) block-level ranged GET seeking.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:49:35 +08:00
Feng Ruohang ef6efd3120 fix(site-replication): snapshot state before bucket healing and peer hooks
Snapshot site-replication topology and release the state read lock before bucket healing, metadata healing, IAM deletion healing, and replication hooks perform storage I/O or peer RPCs. Guard remaining peer-state reads and cloning edits so concurrent configuration updates neither deadlock waiting writers nor race on state maps.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:43:41 +08:00
Feng Ruohang 4a9bdb5407 fix(batch): reject invalid UTF-8 before KMS context rotation
Reject contexts whose JSON serialization could change associated data or collapse distinct keys. Validate both submissions and persisted jobs before changing an object key; retain valid Unicode, including replacement characters.

Exercise malformed-key collisions, invalid values, manifest reload failures and readback with the real builtin KMS. Refs #346.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 2f228284f3 fix(replication): collect site status without holding topology locks
Snapshot peers before concurrent status collection and release the state lock before local metadata and IAM reads. Keep result indexing, peer credentials and errors tied to that snapshot; return a copied current peer state with cached IAM data.

Cover storage, remote requests, configuration changes, cancellation and cached peer state. Confirm the healing entry and all five helpers with deterministic writer probes under the race detector. Refs #341.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang e36ea51d9c fix(copy): preserve authoritative content when pools are moving
Bind in-place metadata updates to the same version and pool as reads. Resolve encryption and checksum rewrites before write placement while retaining the SSE-C rewrap fast path and invalid self-copy checks.

Add HTTP and object-layer regressions for moving pools, duplicate bodies, null versions, timestamp ties and active-pool rewrites. Refs #343.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 3ce07a943b test(copy): cover KMS context and moving source pools
Exercise historical-version copies with real KMS associated data, SSE-C sources in draining pools, and SSE-KMS sources in rebalancing pools. Check destination placement and read all old and new versions.

Refs #343

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 3c35cebe9e test: format attributes read-error regression
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang ae483baca1 docs: record unreleased repair and compatibility boundaries
Document archive authorization, explicit-version bulk-delete cost, migration progress, restore and copy repairs, and coordinated IAM and batch behavior without changing released history.

Fixes #348

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 99fb47fbb9 fix(batch): align variable preflight with policy evaluation
Treat unsupported variable spellings as literals, inspect only condition values, and decode BinaryEquals before checking required context. Keep missing request variables fail-closed.

Fixes #347

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 467ce760ed fix(batch): restore KMS context from persisted rotation options
Decode context without shared cached state for validation and execution, and isolate probe-only associated data. Cover manifest reload, retry and authenticated decryption after rotation.

Fixes #346

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang b5d97e3e1f fix(restore): ignore derived data-dir flag in revision identity
Keep restored versions consistent across expiry and successive heals while preserving metadata, layout and non-object revision distinctions.

Fixes #344

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang a8824b341f fix(copy): plan historical version placement before transforms
Locate exact versions for in-place updates and rewrite cross-pool references through the encryption and compression pipeline selected before copying.

Fixes #343

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang f746d07076 fix(auth): authorize attributes and archive reads with stored tags
Require both Attributes actions on the addressed version and GetObject permission for archive listings. Preserve conditional allows, missing-object masks and retryable storage errors.

Fixes #342

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 745fbf0437 fix(site-replication): release heal read lock before hooks
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 15:30:11 +08:00
Feng Ruohang 8587e061bb fix(decom): tolerate offline drives within erasure quorum
Verify migration progress and final source cleanup on every online drive, allowing disk-not-found failures within the configured read quorum. Preserve strict rejection of online metadata remnants, corruption and directory errors. Cover parent and child migration with an offline drive and reject outages beyond parity.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 13:39:38 +08:00
Feng Ruohang 293ab78b4c fix(auth): authorize tagging replicas without rejecting replication lag
Share the remote tag read and authorization path between GET and mutations. Authorize every present replica, fail closed on unreadable targets, and skip missing replicas unless all targets are absent. Gate remote reads on possible caller access and preserve retryable local read failures for DELETE tagging.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 13:39:19 +08:00
Feng Ruohang 67a90dfef4 fix(auth): authorize every tagging proxy destination
Read and authorize every selected remote version before any tagging mutation, and retain the inspected clients for the write phase. Count only attempted writes so disabled targets cannot mask remote failures. Preserve remote errors and missing-version behavior.

Cover PUT and DELETE across both erasure backends with divergent tags, invalid credentials, anonymous requests, unavailable reads, missing versions and skipped targets. Refs #337.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 11:47:56 +08:00
Feng Ruohang 62b8784428 fix: satisfy integrated server verifiers
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 11:40:24 +08:00
Feng Ruohang 5dc4bd6a1d Fix pool migration cleanup and physical progress
Count removal only after every source drive confirms version absence, including free records and concurrently deleted parents. Retire successful candidates independently with the existing source, destination and tier ownership checks. Reject ambiguous raw reads, verify remaining metadata directly, and propagate cancellation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit b1a97c7e6cc7ab5f44110b5b8258afb60a50053e)
2026-11-11 11:25:32 +08:00
Feng Ruohang cf486df661 Fix Object Lock replica reconciliation boundaries
Reconcile CopyObject lock and tag revisions with the addressed destination,
including missing versions and self-referential copies. Preserve active
COMPLIANCE across pool orderings, lifecycle decisions and set-level writes.

Require retention authorization for source retention revisions, advance local
lock and tag revisions monotonically, and fail force-delete before option
parsing when bucket lock configuration cannot be read. Preserve authorized
same-content replication and existing encryption and tier cleanup behavior.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit a08c3d149dcccad74dd68ab374c02f56379181be)
2026-11-11 11:20:19 +08:00
Feng Ruohang 1d714834c5 fix(iam): normalize service policies and preserve imported state
Keep omitted edit policies unchanged, reset valid empty policies to inheritance, and normalize nonempty policies without a version before storage. Preserve caller policy objects and exclude derived authorization claims from snapshots and new JWTs.

Restore imported service-account and group status, report the actual group error, and cover creation, editing, replication, import/export and raw JWT claims with focused server tests.

Refs #339, #345.

Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit 86e4c09d47f48cb2f44fb2f63f02f222ac6ae102)
2026-11-11 11:17:55 +08:00
Feng Ruohang 14ecb222e7 fix(auth): secure proxied tagging and streaming trailers
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 11:13:26 +08:00
Feng Ruohang e51bff35a3 Merge service-account replication fix into main
Include the remaining committed IAM snapshot fix from claude/lucid-hopper-811f3d. Preserve the current contributor inventory and previously integrated fixes while adding the service-account replication release note.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:12:08 +08:00
Feng Ruohang 20b3bfb6e7 Merge verified prerelease fixes into main
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:08:25 +08:00
Feng Ruohang 322cd8d2c1 fix(decom): count lifecycle removals as scan progress
Reuse the source-version lookup to count versions removed by ILM before migration cleanup so nested keys are discovered on the next pass. Keep delete-marker-only expiry from reporting physical progress, and cover both the decommission regression and the removal-count boundary.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-11 00:08:07 +08:00
Feng Ruohang c3340ec7f5 fix(auth): scope unread tag candidates to effective policies
Select cached user, group, role and session policies without lazy IAM reads. Keep alternative tag values and Deny keys when combining candidates, and bound intermediate combinations as well as authorization attempts.

Cover unrelated cached policies, intersecting tag conditions, temporary credentials, root-derived credentials and cache-only lookup regressions for issue #293.

(cherry picked from commit 65acda0114)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:32:29 +08:00
Feng Ruohang 7c91dc17b2 Merge remote-tracking branch 'origin/main'
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:31:32 +08:00
Feng Ruohang 2f93e0c6b3 Merge pull request #294 from botdeniska/fix/tag-auth-server-error
Preserve server errors for reads authorized by object tags and GetObjectAttributes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:30:24 +08:00
Feng Ruohang 65acda0114 fix(auth): scope unread tag candidates to effective policies
Select cached user, group, role and session policies without lazy IAM reads. Keep alternative tag values and Deny keys when combining candidates, and bound intermediate combinations as well as authorization attempts.

Cover unrelated cached policies, intersecting tag conditions, temporary credentials, root-derived credentials and cache-only lookup regressions for issue #293.

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 23:22:57 +08:00
Denis Babchuk c6949a38c2 fix(api): keep GetObjectAttributes server errors and check s3:ListBucket on the bucket
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit a2ff3c2ecd)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 80dd3e3ce1 test(api): cover GetObjectAttributes errors with and without s3:ListBucket
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit 67bc32f401)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 89028732ab fix(auth): keep the server error when a read allowed by object tags cannot read the tags
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit 3c4a6296f9)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Denis Babchuk 55bd906d30 test(auth): cover reads allowed by s3:ExistingObjectTag when the object cannot be read
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
(cherry picked from commit afa35d16e9)
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 22:33:13 +08:00
Feng Ruohang 296c9710b1 fix(security): close release blockers
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 21:51:47 +08:00
Feng Ruohang f3b42b0eb8 Merge pull request #335 from botdeniska/fix/decom-keys-below-a-key
fix(decom): repeat the walk of a bucket while it removes source versions

Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 21:30:18 +08:00
Feng Ruohang da8d8a54b1 fix(iam): replicate a service account as the record that is stored
Site replication sent an edit of a service account as the fields the
request changed. A peer orders the items of an account by revision and
drops an older one, which is right for a whole account and loses data for
a part of one: when an edit did not reach a peer, or reached it after a
later edit, the peer kept the old secret key, status or inline policy
under the revision of the later edit, and healing took the sites for
equal. Two requests at one site sufficed, as the sender dropped the item
of the first when the second was already stored.

The first sync of a new site, healing and IAM export put an account
together from two places: the revision from storage, and the claims and
the inline policy from the cache of the server; healing took the secret
key and the status from the cache as well. A server whose cache had not
heard of an edit made through another server sent the account from
before the edit under the revision of the edit, and a peer kept that.

replicationItem now sends a create or an edit as the account stored at
that time: the whole record, read once, with its revision, as it does
for a group. serviceAccountSnapshot builds it from the record and the
token in it; IAM export uses the same function, and sync, heal and the
update handler only name the account. The wire types and the way a peer
applies an item do not change: a peer on 20261109 or 20261026 takes such
an edit as it takes a healed account, which also means that it keeps the
inline policy on an edit that names none. A site that is not upgraded
still sends its edits as before.

The hook reads what it sends before it takes the lock of the replication
configuration: loading an identity takes that lock for reading, and a
read lock asked for again behind a waiting writer is never granted. An
account whose stored token cannot be read is an error, as it was for sync
and heal, and is not taken for deleted.

A new service account of a user that was deleted and created again now
reaches the peers without waiting for healing: the create carries the
claims of the token, among them the parent revocation a peer asks for.

An edit gives every site the whole account of the site that took it.
Accounts that already differ between sites under one revision are not
repaired by this change, only by their next edit.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 20:19:54 +08:00
Feng Ruohang 2b62df876a docs(contributors): cite the real CVE-2026-39414 follow-up commit
Regenerated from the contributor data on silo.pgsty.com. One note named
5dc705936, which is not a commit in any repository; the June follow-up
is fd69c89d0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 17:24:28 +08:00
Feng Ruohang 518915d241 docs(contributors): credit upstream advisory researchers and fix a commit link
Regenerated from the reviewed contributor data on silo.pgsty.com. The roll
grows from 76 to 81 people: it now names the researchers credited on the
upstream advisories for inherited CVEs. The link to the Helm chart 7.0.4
commit carried a full hash that matches no commit; it now points at
47275042f.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 16:57:15 +08:00
Denis Babchuk ce5d191077 fix(decom): repeat the walk of a bucket while it removes source versions
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-11-10 11:48:42 +03:00
Denis Babchuk b404f78764 test(decom): cover keys below a key that the walk does not see
Signed-off-by: Denis Babchuk <babchukdenis@icloud.com>
2026-11-10 11:48:42 +03:00
Feng Ruohang d138996983 docs(helm): fix NOTES connection commands and document 20261109 upgrade gates
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:41:49 +08:00
Feng Ruohang d52ba0290a docs: align release, TLS, healthcheck, and contributor docs with 20261109
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:41:49 +08:00
Feng Ruohang 8f3657dc52 Merge pull request #333 from pgsty/claude/cross-bucket-path-traversal-delete-5e1fc2
fix(pools): keep rejected names out of a single-pool bulk delete
2026-11-10 14:39:36 +08:00
Feng Ruohang 7efdedd5ba Merge pull request #329 from pgsty/deps/console-2.7.0
deps: pin SILO Console 2.7.0
2026-11-10 14:30:02 +08:00
Feng Ruohang a18293d0c3 fix(pools): keep rejected names out of a single-pool bulk delete
DeleteObjects checks every name with checkDelObjArgs, but a single pool
still sent all entries to the drives and cleared the error of an entry
that a drive reported as found. Local drives join the name to the bucket
directory, so a name with a ".." segment reached another bucket or the
meta bucket. An S3 DeleteObjects caller allowed to delete in one bucket
could remove a version elsewhere, including one under Object Lock
retention, and from a versioned bucket could write a delete marker on an
object of another bucket or on stored bucket metadata. The bulk lock was
taken on the normalized name as well, that is on the other object.

Entries that fail the check now keep their error, and the rest of the
request runs as a request of its own, before any lock or lookup. A
request whose names are all valid does not enter the new code. More than
one pool already skipped such entries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
2026-11-10 14:24:35 +08:00