Feng Ruohang
35bd75948a
With compression allow_encryption=on an SSE-C object is stored as encrypt(s2(plaintext)), while replication reads it raw (NoDecryption at cmd/erasure-object.go:257) and putReplicationOpts drops the internal compression and actual-size headers (cmd/bucket-replication.go:786). The replica keeps the source seal with no compression marker, so a GET with the correct customer key returns HTTP 200 and the raw S2 stream instead of the object, and the source records the transfer as COMPLETED. Widen the one condition in excludeForCompression (cmd/object-api-utils.go:613) so SSE-C is never compressed, whatever allow_encryption says. This covers all four producers at once, PutObject, NewMultipartUpload, CopyObject and PutObjectExtract, plus any future caller of isCompressible. crypto.SSEC.IsRequested ignores copy-source headers, so a copy is judged on its destination key only, and a raw SSE-C replica write is unaffected because it carries no public SSE-C headers. allow_encryption keeps its meaning for SSE-S3 and SSE-KMS, where the server owns the key and decompresses before replicating. Tests: TestAPISSECCompressionReplicaStaysReadable (single PUT and multipart), TestAPISSECCompressionProducerMatrix, TestAPISSECCompressionSkippedOnCopyObject, TestAPISSECCompressionSkippedOnSnowballExtract and the control TestSSECBatchReplicationCannotRead in cmd/compression-ssec_test.go. Two existing expectations pinned the removed shape and are updated: TestAPICopyObjectSSECKeyRotationNullVersionCompressesRewrite is renamed TestAPICopyObjectSSECKeyRotationNullVersionSkipsCompression and now expects an uncompressed rewrite, keeping its body, checksum, version and ETag assertions; the SSE-C compressed-encrypted variant of TestAPICopyObjectServerSideChecksumEncryption becomes compressible-extension and expects an uncompressed destination, its SSE-S3 sibling keeping the compressed coverage. Compatibility: a deliberate behaviour change. Deployments with allow_encryption=on no longer store new or rewritten SSE-C data compressed, so those writes cost more space; objects already stored compressed keep working on the source and are the concern of pgsty/silo#109, which rejects them at replication time. Multipart uploads initiated before this change keep compressing their parts from the metadata saved at initiation. Upstream468a9fae8refused this combination at PUT time anda2cab0255removed the guard; upstream master is still unguarded, so this is a deliberate divergence. Fixes pgsty/silo#118 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe Signed-off-by: Feng Ruohang <rh@vonng.com>
Automatic Site Replication
This feature allows multiple independent Silo sites (or clusters) that are using the same external IDentity Provider (IDP) to be configured as replicas. In this situation the set of replica sites are referred to as peer sites or just sites. When site-replication is enabled on a set of sites, the following changes are replicated to all other sites:
- Creation and deletion of buckets and objects
- Creation and deletion of all IAM users, groups, policies and their mappings to users or groups
- Creation of STS credentials
- Creation and deletion of service accounts (except those owned by the root user)
- Changes to Bucket features such as:
- Bucket Policies
- Bucket Tags
- Bucket Object-Lock configurations (including retention and legal hold configuration)
- Bucket Encryption configuration
NOTE: Bucket versioning is automatically enabled for all new and existing buckets on all replicated sites.
The following Bucket features will not be replicated, is designed to differ between sites:
- Bucket notification configuration
- Bucket lifecycle (ILM) configuration
Pre-requisites
-
Initially, only one of the sites added for replication may have data. After site-replication is successfully configured, this data is replicated to the other (initially empty) sites. Subsequently, objects may be written to any of the sites, and they will be replicated to all other sites.
-
Removing a site is not allowed from a set of replicated sites once configured.
-
All sites must be using the same external IDP(s) if any.
-
For SSE-S3 or SSE-KMS encryption via KMS, all sites must have access to a central KMS deployment. This can be achieved via a central KES server or multiple KES servers (say one per site) connected via a central KMS (Vault) server.
Configuring Site Replication
- Configure an alias in
mcfor each of the sites. For example if you have three Silo sites, you may run:
mc alias set silo1 https://silo1.example.com:9000 adminuser adminpassword
mc alias set silo2 https://silo2.example.com:9000 adminuser adminpassword
mc alias set silo3 https://silo3.example.com:9000 adminuser adminpassword
or
export MC_HOST_silo1=https://adminuser:adminpassword@silo1.example.com
export MC_HOST_silo2=https://adminuser:adminpassword@silo2.example.com
export MC_HOST_silo3=https://adminuser:adminpassword@silo3.example.com
- Add site replication configuration with:
mc admin replicate add silo1 silo2 silo3
- Once the above command returns success, you may query site replication configuration with:
mc admin replicate info silo1
** Note ** Previously, site replication required the root credentials of peer sites to be identical. This is no longer necessary because STS tokens are now signed with the site replicator service account credentials, thus allowing flexibility in the independent management of root accounts across sites and the ability to disable root accounts eventually.
However, this means that STS tokens signed previously by root credentials will no longer be valid upon upgrading to the latest version with this change. Please re-generate them as you usually do. Additionally, if site replication is ever removed - the STS tokens will become invalid, regenerate them as you usually do.