minio/docs/investigations/release-readiness-20260911.md

Remaining release correctness work, 2026-09-11

This is a dated investigation, with the source and runtime boundaries recorded below. It does not establish the current dependency pins or a later release. See the current changelog and component matrix.

This records the three work items agreed after the branch/PR consolidation: OIDC #154, Linux restart/readback #116, and the related multi-pool defects #133/#144. The maintained target is SILO with the PGSTY Console, mcli and silo-pkg dependencies in the repository's current go.mod.

Multi-pool writes and conditional deletion (#133, #144)

The pools layer now holds its object write lock across PUT and multipart completion, as it already does for metadata updates and DELETE. Multipart completion acquires the object lock before the upload lock. Queued healing and drive healing use this namespace too; healing under a caller-owned lock retains that lock's cancellation context. The destination-pool allocation policy is unchanged.

Trusted replica writes resolve the addressed version in every pool, including draining and rebalancing pools. Retention, legal hold and tags retain their independent ordering timestamps. A timestamp-only removal survives a stale retransmit. Completion resolves the version persisted in the upload, including the null version, rather than taking a later latest version's metadata.

Successful replica replacement retires competing copies of that exact version, so an equal-ModTime copy in an earlier pool cannot shadow the reconciled result. Metadata updates evaluate their callback once against the merged version and update all its copies. Replica metadata COPY rechecks ordering under the lock. Cleanup failures propagate; a replacement may already have committed when cleanup fails, and a retry can finish cleanup. Data movement keeps ownership of its source cleanup.

Retiring copies preserves any remote-tier reference still held by a surviving copy, including temporarily restored objects. Only the last copy of that tier reference schedules remote contents for garbage collection. This also protects the authoritative copy if the final conditional deletion fails; unrelated tier contents remain eligible for cleanup.

Conditional DELETE evaluates its precondition against the logical latest or explicitly addressed version. It checks all pools before mutation, removes secondary copies before the authoritative one, and returns cleanup errors. Versioned DELETE without a version ID creates a delete marker and preserves version history. Retention and replication callbacks evaluate the reconciled logical version. The existing all-pool delete helper also propagates errors from non-first pools.

The deterministic two-pool, 32-drive fixtures cover version selection, duplicate removal, delete failure propagation, PUT/DELETE and completion/DELETE interleavings, independent lock winners, draining/rebalancing owners, null versions, metadata COPY, metadata/healing serialization and cleanup retry. Tiered-copy tests inspect the persisted garbage-collection markers after metadata COPY, restored COPY, successful deletion and failed primary deletion, with distinct remote references as cleanup controls. The original branch reproduced the wrong-version lookup, surviving duplicate, suppressed delete error, PUT/DELETE race, and PUT/completion lock-state failures before the fixes were applied.

Before the final tier-reference guard, local validation passed the full cmd suite (255.720 s), all internal tests, go vet ./..., generated-file checks and the branding/entrypoint checks. Focused race checks cover the pooled interleavings, SSE-C lock regressions, conditional deletion, access-tier movement and TLS defaults. The two-pool and related replica/delete/movement tests also passed as a Linux/arm64 test binary in an isolated container with an 8 GiB /tmp tmpfs. Its initial 1 GiB tmpfs was insufficient for the existing single-drive test fixtures' free-space guard. The final tier-reference guard passed all six new cases in that Linux fixture; the retained restart binary below predates this guard and has no remote tier configured.

Linux restart/readback (#116)

The runner creates four Linux/arm64 server containers on one Docker Desktop Linux VM, with separate network identities and one drive per node (EC 2+2). A holder container keeps four 256 MiB Linux tmpfs named volumes mounted across full server stops. Docker's ordinary filesystem had only 3.8 GiB free out of 2 TiB and correctly hit the server's free-space limit; the test uses the separate tmpfs filesystems without changing that threshold. This is the four-containers-on-one-Linux-host option explicitly accepted in the #116 V1 plan.

This covers process/container restart and TCP peer reconnection on one Linux host. It does not establish independent-host, host-reboot or physical-media durability. Every resource created by the runner is removed in its cleanup. The retained summary records image identities, per-coordinator observations and readback counts.

Binary	Full restart to all admin views online	Admin gate to complete 4 PUT / 16 GET round	Timed readbacks	Final readback
0806 release	2.084 s	14.449 s	660 / 660	240 / 240
0903 release	2.284 s	0.191 s	96 / 96	52 / 52
Current fix candidate	2.461 s	14.489 s	1356 / 1356	904 / 904

Each canary has one hard 60-second deadline covering setup, requests, response body reads and sleeps, with SDK retries disabled. Every acknowledged PUT uses a unique versioned key and immediately records its VersionId, size and SHA-256. The timed checks reread these same objects through all four coordinators at 15, 30 and 60 seconds after the data canary succeeds. They do not replace early acknowledgements with later writes. The final check also includes writes made during the one-node outage and subsequent rejoin canary.

All three runs passed the bounded canary, scheduled readbacks, one-node-outage read/write and rejoin readback. The candidate also showed a rejoin window: admin online at 2.490 s, complete data canary 13.977 s later. These are individual observations, not a latency guarantee or a comparison proving one version faster. Admin/health readiness still must be followed by a data-path check.

To repeat with cached images and a native mcli executable:

uv run --with boto3==1.43.92 docs/investigations/issue-116/run-linux.py \
  0806 0903 --output /tmp/silo-linux-acceptance --mcli /path/to/mcli

CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -tags kqueue -o /tmp/silo-linux .
uv run --with boto3==1.43.92 docs/investigations/issue-116/run-linux.py \
  current --output /tmp/silo-linux-candidate --mcli /path/to/mcli \
  --current-binary /tmp/silo-linux --source-revision "$(git rev-parse HEAD)"

The runner uses --pull=never; preload the two release images, alpine:3.23, and the Linux base image selected with --current-image. Match the candidate binary architecture to that image. Detailed local evidence, including the acknowledgement ledger and full logs, remains in the requested output directory.

OIDC (#154)

The TLS implementation fix was already merged in 48e1846525cc: transports honor Go's key-exchange defaults, so GODEBUG=tlsmlkem=0 can opt out of ML-KEM for an ingress that rejects it. It does not disable certificate validation or force an automatic protocol downgrade. See the existing Go 1.27 investigation and issue investigation.

A Linux build of main b32f2d9dd01a passed fresh isolated fixture checks:

• ML-KEM-intolerant IdP with tlsmlkem=0: discovery, IAM, Console login (204), authenticated bucket listing (200).
• Normal TLS 1.3 IdP without that override: the same complete login chain.
• Add OIDC through the real administration API with the compatibility setting.
• Invalid JWT signature and audience: no session cookie and bucket access 403.
• Untrusted CA: discovery fails and cluster readiness remains 503.

The affected customer's discovery URL and ingress configuration are still unavailable. This establishes the supported local fix and its negative controls, not the root cause or recovery of that hidden deployment. #154 stays open for an affected-environment retest; no release date or published artifact is implied by these checks.