4 Commits

Author	SHA1	Message	Date
Feng Ruohang	b8ebaffd2e	fix(restore): name the server that keeps native restore unavailable ... The native restore gate refused a POST restore and a queued restore with one message, "unavailable during a rolling upgrade: not every server of the cluster supports it yet", whether a server answered another protocol or was simply not connected. A cluster with one server down thus reported a rolling upgrade, and the operator looked for an older binary instead of the server. The gate now returns an error that wraps errNativeRestoreUnavailable and says why: errNativeRestorePeerUnreachable names the server that is not connected ("server host:port is not reachable, so it cannot be shown to have stopped an older restore"), errNativeRestoreUnsupported keeps the rolling-upgrade text for a server that reports another protocol or reconnected while it was asked, and a check that its deadline ended carries that error. The POST handler maps the error to 503 ServiceUnavailable with the message as before, the worker still clears its marker and stops, and errors.Is(err, errNativeRestoreUnavailable) holds for every refusal. The gate remains closed in every case it was closed before. TestNativeRestoreRefusalReasons posts a restore with a fake server that is not connected and one that answers protocol 1; before this change the first answered 503 with the rolling-upgrade text and without "not reachable". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Feng Ruohang <rh@vonng.com>	2026-11-07 17:04:02 +08:00
Feng Ruohang	3268fbea7e	fix(restore): queue the heal of a partial commit under the normalized ID ... A native restore commit that reaches a write quorum but not every drive queues an MRF heal of the version. It passed the raw version ID, which is empty for the null version, while the convergence path passes the normalized "null". The heal boundary already reads an empty ID as the null version, so the old entry healed the right version; this makes the two paths queue the same identity. The header of restore-capability.go still described the multipart upload of the first native-restore design. It now describes what the capability gate protects in the two-phase design: restore ownership through the restore lock and the attempt marker, the staged commit under the object lock, and the attempt fence. It also says why an unreachable server blocks native restore: it is not proven stopped. Test: TestRestorePartialCommitQueuesNullVersionHeal fails one drive's commit on a noncurrent null version, checks the queued entry names "null", and heals it back onto that drive. It fails on the base at the identity check only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Feng Ruohang <rh@vonng.com>	2026-11-06 19:40:24 +08:00
Feng Ruohang	45b1e4bc00	fix(restore): stage native restores outside the object lock ... The unreleased native-restore fixes after RELEASE.2026-10-26T00-00-00Z let the lock decide who owns a restore. Since fa42a0982 a single-pool worker holds the object lock for the whole tier download, and since 7f38bf628 a busy lock means "a restore runs". 7f38bf628 also keeps the restore's data in a marked multipart upload. Two regressions followed: - While an old version restored, every GET, HEAD and write of the same key waited for the download. On a single pool this is new; with several pools the released code already held the lock that long. - An ordinary upload whose completion failed after staging its metadata became InvalidUploadID for every caller, and could not be aborted for an hour. The marker that hid restore uploads could not tell them apart once a completion had rewritten the upload's xl.meta. A restore now runs in two phases: - The worker erasure-codes the tier copy into the tmp volume of the version's set, without the object lock, as heal does. A drive that fails any part, or misses a part when CheckParts runs before the commit, takes no part in the commit. - A restore lock per version tells whether a restore runs. POST takes it before the object lock and hands it to the worker. It is a liveness hint: only the checks under the object lock decide what is written. - The commit takes the object lock briefly, on one pool or several. It reads the version again and checks the attempt's ownership, the version identity, the tier copy, the part layout and the erasure layout. It then publishes with RenameData drive by drive, never through erasure renameData's undo, and needs the write quorum. The replaced data directory goes only when every published drive reports the same one. - A publication that reached some drives but not the write quorum converges under the same lock hold, in bounded retries after it: back to the unrestored version on every drive that holds it, or to a committed restore when the copy turns out to be on a write quorum. Each pass first checks that every drive holds what the version held before the commit, the commit wrote, or a convergence wrote, and stops on anything else. It deletes no directory. Multipart goes back to main: erasure-multipart.go and bucket-object-lock.go are identical to 69fcf50cd. The commit lock, the marker and the restore exceptions of multipart completion and of the null-version Object Lock check are gone, and single-pool completion again takes the upload lock first and the object lock right before it commits. With several pools, abandoned-data cleanup now takes the pools lock that writes take, as heal does, so it cannot remove a directory that a commit moved in place and has not yet referenced. The restore capability becomes protocol 2, and a queued worker checks it again. Recovery after a coordinator crash during publication, and storage RPCs that complete after their caller returned, stay excluded. Main has the same exposure, and they are tracked separately. Tests: - New regression tests cover both problems. They fail on f85d4a5fa, and the multipart one also passes on main. - Fault-injection tests cover a publication below the write quorum, including seven of twelve drives on 8+4; lost answers; convergence retries and stops; the old data directory; a stale inline marker; a recreated tmp volume; a published revision read across its expiry; a second POST; a target deleted during the download; a tier 404; and pools-locked abandoned cleanup. - Thirteen tests of the upload marker from 7f38bf628 are dropped, and so is TestNativeRestoreConcurrentCompletion from fa42a0982, which completed the restore's upload from a client. Four tests are reframed for the new semantics, including the lock-domain restore test, which now gates the commit. - Three self-recursive WriteMetadata test doubles are fixed, and the helpers of the dropped tests are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Feng Ruohang <rh@vonng.com>	2026-11-06 11:30:29 +08:00
Feng Ruohang	7f38bf6282	fix(restore): give each native restore an owner and a private upload ... A native restore (POST ?restore, issue #281) marks the version ongoing, and a background worker rehydrates it from the tier. The worker, its upload and every status write could be reached or confused by other callers. A public client could write into or complete the restore upload. POST copied a whole metadata snapshot over newer protection changes. An old or failed attempt could act on a newer one. Restore, expiry and heal decided on whichever drive's metadata they read. Each POST that starts a rehydration now writes an attempt ID into the version's internal metadata, with the identity it resolved: version ID, ETag, size and ModTime. The worker, its upload, its part writes and its completion carry the attempt. A worker whose attempt no longer owns the version stops without writing, completing or resetting. Every write uses the versioning state and explicit version ID captured by POST, so a null target stays null. The restore upload carries an internal marker, decided over all drives. Public multipart calls answer NoSuchUpload and listings hide it. An unknown marker fails closed. A completion holds a commit lock from staging its commit in the upload until the commit ends. Stale cleanup decides on every upload past the stale age under that lock, and keeps it until its rename ends, also past the drive deadline. Abort of a committing upload takes the lock and removes a failed commit's leftover only when every copy is staged and older than one hour. POST, the worker, the reset, the ExpireRestored recheck and heal all decide on the authoritative revision of the exact version. That is the revision held by max(n/2+1, read quorum) drives, identified by the version header and the normalized metadata digest. Status writes patch only the restore fields of that revision. They write only the drives that hold it, and put the old revision back on failure. Heal copies restore fields only when a quorum holds the same fields. A failed attempt resets itself with bounded in-memory retries. A later POST takes over a stranded marker when no attempt runs. Otherwise it answers RestoreAlreadyInProgress within a few seconds. Older nodes do not know the marker. A new grid capability handler lets the serving node ask every peer. POST answers 503 ServiceUnavailable and writes nothing until all peers support native restore, and a queued worker asks again before it creates its upload. The single-pool lock-domain split between DELETE/CopyObject and restore is closed by its own fix, which this change depends on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Feng Ruohang <rh@vonng.com>	2026-11-05 06:47:38 +08:00