A runtime change to audit_webhook, logger_webhook or audit_kafka is
applied on every peer by serviceReloadDynamic, under a 30-second context
that is cancelled as soon as the handler returns. applyDynamicConfigForSubSys
passed that context to the logger target updates, and the targets tie
their queue worker and, with queue_dir, their disk replay to the Init
context. On every server except the one that served the admin request
the new targets therefore stopped right after they were created: records
filled the in-memory queue and were then dropped with "log buffer full"
until the server restarted.
- Build logger webhook, audit webhook and audit Kafka targets with
GlobalContext, as startup and the serving node already do. The
caller's context is still used for logging.
- Open the HTTP disk queue store before starting the queue worker, and
cancel the derived context when it cannot be opened. A worker started
before a failed Open dereferenced the nil store on the first record and
crashed the server; with the process context that would have happened
on every server.
- Let Cancel hand records still waiting in the channel to the shared
buffer of the replacement target. A target whose queue store never
opened has no worker to do this, so repairing queue_dir and applying
the configuration again used to discard the records it had accepted.
- TestDynamicWebhookOutlivesConfigContext applies each case through
applyDynamicConfigForSubSys under a short context, cancels it as the
peer handler does, and requires delivery: audit and logger webhooks
with a memory queue, a disk queue and an unopenable disk queue
(recovered by applying the configuration again), and audit Kafka with
and without queue_dir against a sarama mock broker. Each case runs in
a child process without logger and audit environment variables.
This completes earlier uncommitted maintainer work on this issue, which
covered the two HTTP call sites, the store-before-worker order and the
first version of the test.
Refs #259
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Availability-optimized writes can leave a readable null version across separate EC groups when a disk is unavailable. Recount ordinary null versions for all non-strict reads and cover version listing and complete pool removal with end-to-end tests.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Replace process-wide hash-slot mutexes with reference-counted locks keyed
by drive and path. Retain entries before waiting, acquire unique paths in
order, and reclaim entries only after all holders and waiters finish.
Add real storage API regressions for unrelated colliding objects and drives,
plus coverage for cancellation, multi-path order and waiter lifetimes.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Wait for admitted data and metadata mutations, preserve overwrite backups during rollback, and reject rollback of a different generation. Report conflicting or unreadable pool metadata as read-quorum failures instead of exposing stale copies. Add barrier-driven local/RPC and multi-pool regressions for cancellation, self-copy, rollback and old-value reads.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Feng Ruohang