astelm/minio
1
0
Fork 0
mirror of https://github.com/pgsty/minio.git synced 2026-11-10 09:35:31 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(logger): keep reloaded webhook and audit targets alive after a peer reload

Browse Source 
A runtime change to audit_webhook, logger_webhook or audit_kafka is
applied on every peer by serviceReloadDynamic, under a 30-second context
that is cancelled as soon as the handler returns. applyDynamicConfigForSubSys
passed that context to the logger target updates, and the targets tie
their queue worker and, with queue_dir, their disk replay to the Init
context. On every server except the one that served the admin request
the new targets therefore stopped right after they were created: records
filled the in-memory queue and were then dropped with "log buffer full"
until the server restarted.

- Build logger webhook, audit webhook and audit Kafka targets with
  GlobalContext, as startup and the serving node already do. The
  caller's context is still used for logging.
- Open the HTTP disk queue store before starting the queue worker, and
  cancel the derived context when it cannot be opened. A worker started
  before a failed Open dereferenced the nil store on the first record and
  crashed the server; with the process context that would have happened
  on every server.
- Let Cancel hand records still waiting in the channel to the shared
  buffer of the replacement target. A target whose queue store never
  opened has no worker to do this, so repairing queue_dir and applying
  the configuration again used to discard the records it had accepted.
- TestDynamicWebhookOutlivesConfigContext applies each case through
  applyDynamicConfigForSubSys under a short context, cancels it as the
  peer handler does, and requires delivery: audit and logger webhooks
  with a memory queue, a disk queue and an unopenable disk queue
  (recovered by applying the configuration again), and audit Kafka with
  and without queue_dir against a sarama mock broker. Each case runs in
  a child process without logger and audit environment variables.

This completes earlier uncommitted maintainer work on this issue, which
covered the two HTTP call sites, the store-before-worker order and the
first version of the test.

Refs #259

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
2026-11-05 02:20:06 +08:00
parent 69fcf50cd2
commit 0fa4a8c503
3 changed files with 263 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/config-current.go
+6 -3
View File
@@ -609,7 +609,8 @@ func applyDynamicConfigForSubSys(ctx context.Context, objAPI ObjectLayer, s conf
}
loggerCfg.HTTP[n] = l
}
if errs := logger.UpdateHTTPWebhooks(ctx, loggerCfg.HTTP); len(errs) > 0 {
// Webhook workers outlive the bounded configuration reload.
if errs := logger.UpdateHTTPWebhooks(GlobalContext, loggerCfg.HTTP); len(errs) > 0 {
configLogIf(ctx, fmt.Errorf("Unable to update logger webhook config: %v", errs))
}
case config.AuditWebhookSubSys:
@@ -627,7 +628,8 @@ func applyDynamicConfigForSubSys(ctx context.Context, objAPI ObjectLayer, s conf
loggerCfg.AuditWebhook[n] = l
}
if errs := logger.UpdateAuditWebhooks(ctx, loggerCfg.AuditWebhook); len(errs) > 0 {
// Both the queue worker and disk replay follow the server lifetime.
if errs := logger.UpdateAuditWebhooks(GlobalContext, loggerCfg.AuditWebhook); len(errs) > 0 {
configLogIf(ctx, fmt.Errorf("Unable to update audit webhook targets: %v", errs))
}
case config.AuditKafkaSubSys:
@@ -644,7 +646,8 @@ func applyDynamicConfigForSubSys(ctx context.Context, objAPI ObjectLayer, s conf
loggerCfg.AuditKafka[n] = l
}
}
if errs := logger.UpdateAuditKafkaTargets(ctx, loggerCfg); len(errs) > 0 {
// The Kafka disk replay follows the server lifetime as well.
if errs := logger.UpdateAuditKafkaTargets(GlobalContext, loggerCfg); len(errs) > 0 {
configLogIf(ctx, fmt.Errorf("Unable to update audit kafka targets: %v", errs))
}
case config.StorageClassSubSys: