A native restore (POST ?restore, issue #281) marks the version ongoing,
and a background worker rehydrates it from the tier. The worker, its
upload and every status write could be reached or confused by other
callers. A public client could write into or complete the restore
upload. POST copied a whole metadata snapshot over newer protection
changes. An old or failed attempt could act on a newer one. Restore,
expiry and heal decided on whichever drive's metadata they read.
Each POST that starts a rehydration now writes an attempt ID into the
version's internal metadata, with the identity it resolved: version
ID, ETag, size and ModTime. The worker, its upload, its part writes and
its completion carry the attempt. A worker whose attempt no longer
owns the version stops without writing, completing or resetting. Every
write uses the versioning state and explicit version ID captured by
POST, so a null target stays null.
The restore upload carries an internal marker, decided over all drives.
Public multipart calls answer NoSuchUpload and listings hide it. An
unknown marker fails closed. A completion holds a commit lock from
staging its commit in the upload until the commit ends. Stale cleanup
decides on every upload past the stale age under that lock, and keeps
it until its rename ends, also past the drive deadline. Abort of a
committing upload takes the lock and removes a failed commit's
leftover only when every copy is staged and older than one hour.
POST, the worker, the reset, the ExpireRestored recheck and heal all
decide on the authoritative revision of the exact version. That is the
revision held by max(n/2+1, read quorum) drives, identified by the
version header and the normalized metadata digest. Status writes patch
only the restore fields of that revision. They write only the drives
that hold it, and put the old revision back on failure. Heal copies
restore fields only when a quorum holds the same fields.
A failed attempt resets itself with bounded in-memory retries. A later
POST takes over a stranded marker when no attempt runs. Otherwise it
answers RestoreAlreadyInProgress within a few seconds.
Older nodes do not know the marker. A new grid capability handler lets
the serving node ask every peer. POST answers 503 ServiceUnavailable
and writes nothing until all peers support native restore, and a
queued worker asks again before it creates its upload.
The single-pool lock-domain split between DELETE/CopyObject and restore
is closed by its own fix, which this change depends on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
With compression allow_encryption=on an SSE-C object is stored as
encrypt(s2(plaintext)), while replication reads it raw (NoDecryption at
cmd/erasure-object.go:257) and putReplicationOpts drops the internal
compression and actual-size headers (cmd/bucket-replication.go:786). The
replica keeps the source seal with no compression marker, so a GET with the
correct customer key returns HTTP 200 and the raw S2 stream instead of the
object, and the source records the transfer as COMPLETED.
Widen the one condition in excludeForCompression (cmd/object-api-utils.go:613)
so SSE-C is never compressed, whatever allow_encryption says. This covers all
four producers at once, PutObject, NewMultipartUpload, CopyObject and
PutObjectExtract, plus any future caller of isCompressible.
crypto.SSEC.IsRequested ignores copy-source headers, so a copy is judged on its
destination key only, and a raw SSE-C replica write is unaffected because it
carries no public SSE-C headers. allow_encryption keeps its meaning for SSE-S3
and SSE-KMS, where the server owns the key and decompresses before replicating.
Tests: TestAPISSECCompressionReplicaStaysReadable (single PUT and multipart),
TestAPISSECCompressionProducerMatrix, TestAPISSECCompressionSkippedOnCopyObject,
TestAPISSECCompressionSkippedOnSnowballExtract and the control
TestSSECBatchReplicationCannotRead in cmd/compression-ssec_test.go. Two existing
expectations pinned the removed shape and are updated:
TestAPICopyObjectSSECKeyRotationNullVersionCompressesRewrite is renamed
TestAPICopyObjectSSECKeyRotationNullVersionSkipsCompression and now expects an
uncompressed rewrite, keeping its body, checksum, version and ETag assertions;
the SSE-C compressed-encrypted variant of
TestAPICopyObjectServerSideChecksumEncryption becomes compressible-extension and
expects an uncompressed destination, its SSE-S3 sibling keeping the compressed
coverage.
Compatibility: a deliberate behaviour change. Deployments with
allow_encryption=on no longer store new or rewritten SSE-C data compressed, so
those writes cost more space; objects already stored compressed keep working on
the source and are the concern of pgsty/silo#109, which rejects them at
replication time. Multipart uploads initiated before this change keep
compressing their parts from the metadata saved at initiation. Upstream
468a9fae8 refused this combination at PUT time and a2cab0255 removed the guard;
upstream master is still unguarded, so this is a deliberate divergence.
Fixespgsty/silo#118
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L7qJqWwy8oFA6aCXWRzXQe
Signed-off-by: Feng Ruohang <rh@vonng.com>
After zero-byte reads authenticate SSE-C at the object layer, equal invalid rotation keys match non-empty reads and return AccessDenied before the rotation-specific distinction.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Exercise the silent compression-on-copy path, compressed SSE-C re-encryption, and the equal-invalid-key error contract on both object-layer backends.
Signed-off-by: Feng Ruohang <rh@vonng.com>
A key rotation rewraps the object key held in metadata; it never re-encrypts
the stored bytes. CopyObjectHandler took that shortcut whenever the request
looked like a same-object SSE-C rotation, on the assumption that the object
layer would then leave the stored bytes alone. That is the same assumption
copyRewritesObjectData() was added to stop making.
When the source is a null version on a bucket that gained versioning after the
object was written, the object layer cannot reference that version and falls
back to PutObject. The reader at that point holds plaintext decrypted with the
old key and no EncryptFn is set, so the destination ends up storing plaintext
under metadata that claims the object is SSE-C encrypted. A subsequent GET
failed with "sio: unsupported version".
Gate the rotation shortcut on the same prediction the compression metadata
already uses. When the object layer stores new object data the rotation falls
through to the regular re-encrypting copy, which decrypts with the old key and
re-encrypts with the new one. The source version selection moves next to the
gate because both decisions need it.
That fallback authenticates the source key through the source decryptor, which
GetObjectNInfo does not build for a zero byte object. Check the key explicitly
before the destination is written, so the gate cannot turn a rotation that the
shortcut rejected with AccessDenied into one that succeeds. The re-encrypting
copy regenerates the encrypted ETag, unlike an in-place rotation; the test
records that difference.
The other three object layer CopyObject callers that set metadataOnly -
PostRestoreObjectHandler, updateRestoreMetadata and batchKeyRotate - address
the same version on both sides and never set Versioned, so they only reach the
two in-place cases already covered by the copyRewritesObjectData table.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fk3PAD7DHCYzcyegYWqAmt
CopyObjectHandler recorded the source compression metadata whenever the copy
was metadata-only, on the assumption that the object layer would then leave
the stored bytes alone. That assumption does not hold. Both
erasureServerPools.CopyObject and erasureSets.CopyObject only skip a data
rewrite in three cases, and otherwise fall back to a full PutObject.
The reachable gap is a copy whose source is a null version on a bucket that
gained versioning after the object was written. Neither version ID is set, so
the self-referential version branch is skipped, the data is rewritten as
plaintext, and the preserved compression metadata then described bytes that
no longer exist. A subsequent GET failed with "s2: corrupt input".
Mirror the object layer's decision in copyRewritesObjectData and record the
compression metadata from it, so the metadata always describes whichever
bytes are finally stored. The source version selection that lets a versioned
metadata-only copy add a self-referential version moves next to the same
decision, since both depend on the effective metadata-only value.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Feng Ruohang