minio/docs/investigations/go127-stack.md

SILO stack: Go 1.27 compatibility audit

2026-09-09. Scope: the maintained Server, silo-pkg, mcli, and Console. This extends the OIDC #154 investigation to other paths using the same TLS configuration and to adjacent standard-library changes. It records local validation performed before commit. No pushes, issue comments, releases, or production changes were made during the audit.

Findings and changes

Component	Finding	Change
Server, baseline d1105bbb3d4a0afa33b3a4ac11b821235038ed0e	Eight TLS configuration sites explicitly use the same ML-KEM-containing curve list, overriding tlsmlkem=0 in Go 1.27.	Remove the eight assignments and obsolete TLSCurveIDs helper. Retain Go defaults in HTTP clients, replication, cloud client-certificate transport, both grid links, etcd, and the S3 listener. Add wire-level regression tests.
silo-pkg, baseline a92c54d	No built-in explicit PQ curve list. Web environment transport uses defaults; LDAP/OIDC accept caller configuration.	Add a real web-environment TLS regression test and document runtime-default selection. Keep the Go 1.26 library floor.
mcli, baseline fcd5cad8	S3/Admin transport and alias/TOFU dialer use default curves. No instance of the Server's override bug found.	Add real S3/Admin transport and alias-dialer handshake tests; update Go/TLS upgrade notes.
Console, baseline c103d08ec	IdP, SILO/STS, Prometheus, and webhook clients share a transport using defaults. The HTTPS listener explicitly uses only P-256.	Add real IdP/SILO-client handshake tests; update Go/TLS upgrade notes. Retain the existing listener policy.

The Server's general external HTTP transport is also used by OpenID discovery and JWKS, identity plugins, notification/lambda checks, audit/log webhooks, and S3 cloud backends. Fixing only the two OIDC callers would leave those other paths affected. The broader correction supersedes the earlier OIDC-only candidate.

The retained defaults follow Go's implementation rather than duplicating its GODEBUG parser. Go 1.27 explicitly changed the interaction between manually selected curves and the tlsmlkem/tlssecpmlkem default controls. See the Go TLS release notes.

With no opt-out, default curves additionally include SecP256r1MLKEM768 and SecP384r1MLKEM1024. With GODEBUG=tlsmlkem=0, hybrid exchanges are disabled for default-configured TLS throughout the process. This is an intentional change from the old fixed subset. GODEBUG=tlssecpmlkem=0 disables just the SecP hybrids while retaining X25519MLKEM768. TLS versions, cipher-suite policy, certificate and hostname checks, client certificates, proxies, and HTTP/2 choices are not relaxed by this patch. No automatic fallback after a TLS error is introduced.

Reproduction and regression evidence

Before changing product code, the new Server test failed for all five tested outbound constructors with tlsmlkem=0: general external HTTP, internode HTTP, replication, cloud client certificates, and etcd. The server observed [X25519MLKEM768 X25519 P256 P384 P521] in every case. Both TLS 1.2 and TLS 1.3 peers reproduced the problem. The inbound Server also accepted a PQ-only client despite the same opt-out.

After the change, those tests pass on darwin/arm64 and linux/arm64. They exercise 20 outbound combinations (five constructors, two peer TLS versions, two debug settings), plus inbound classical/PQ-only peers with the opt-out enabled and disabled. The inbound opt-out case rejects a PQ-only peer while continuing to accept P-256. The etcd test exercises its actual TLS configuration, not an etcd cluster; the client-certificate test exercises construction/loading, not a full mutual-authentication service.

The mc, Console, and silo-pkg handshake tests pass without product code changes. All use a trusted synthetic certificate and inspect a real ClientHello. They check both disabling and retaining ML-KEM. Console also retains its existing unknown-CA, hostname, and endpoint-scoping regression checks.

A freshly source-built complete Linux Server then passed three isolated integration scenarios using the existing synthetic IdP fixture:

Scenario	Result
ML-KEM-intolerant IdP + tlsmlkem=0	Discovery/JWKS, IAM, Console, and authenticated Admin calls succeed; login 204 and bucket list 200.
Normal TLS 1.3 IdP, no opt-out	Same successful login, token exchange, STS, session, and bucket-list chain.
Add OIDC to a running Server	Actual Admin API accepts the provider under the compatibility setting.

Both login scenarios reject modified JWT signatures and a wrong audience: no session cookie is issued and authenticated bucket access returns 403. Login currently reports these authentication failures as 500; that existing error mapping is outside this TLS change. Curl in the same namespace returns HTTP/2 200. The containers use a pre-existing generic Debian image, --network none, and no published ports; no Server or Console image was downloaded or used.

The fixture drives real Console HTTP APIs, not rendered browser interaction. This is a conditional interoperability reproduction, not proof of the actual customer ingress behavior. Grid's existing tests pass, but a production-style distributed TLS cluster, external cloud providers, and real etcd/LDAP/Keycloak deployments were not exercised. Disabling ML-KEM does not disable new ML-DSA signature offers and does not repair an ingress that rejects those offers.

Other Go changes checked

macOS root certificates: a confirmed upgrade-visible change

Using the same public synthetic CA and certs.GetRootCAs, fresh-process probes produce the following results when SSL_CERT_FILE points at that CA and SSL_CERT_DIR points at an empty directory:

Compiler / main module	CA from environment trusted?	Explicit CA argument trusted?
Go 1.26.5 / go 1.26.0	No	Yes
Go 1.27.1 / go 1.26.0	No	Yes
Go 1.27.1 / go 1.27.1	Yes	Yes

The Go 1.26 module built by Go 1.27 carries DefaultGODEBUG=...x509sslcertoverrideplatform=0; explicitly setting that option to 1 enables the new behavior. A Go 1.27 application can explicitly set it to 0 to recover the previous platform behavior. The diagnostic source is cert-roots.go. The consuming application's defaults apply to library calls as well, so silo-pkg's older go directive does not prevent the behavior in Server, mc, or Console.

This is expected standard-library behavior, not a reason to silently discard configured CA variables or skip verification. Setting either variable replaces Keychain trust with on-disk roots and Go's verifier; stale or incomplete paths can break previously trusted connections. Unset inherited variables to restore Keychain trust; explicit additional CAs still work. The three application READMEs and the package README now document this. The package's Windows loader enumerates the native ROOT store directly and does not call SystemCertPool; it does not inherit this particular new setting. Windows behavior was reviewed in source, not runtime-tested.

JSON, HTTP, timers, and other compatibility controls

• JSON: checked owned JSON error handling and exercised policy/condition, config, and authentication tests. quick uses typed SyntaxError and UnmarshalTypeError; no owned decision depending on changed standard JSON error text was found. silo-pkg's full suite passes with both Go compilers; mc's command and Console's API/auth suites pass under Go 1.27. No broad nojsonv2 opt-out or serialization rewrite is justified by these results.
• HTTP response closing: Go 1.27's standard-library drain is bounded at 256 KiB and 50 ms. mc's actual early-close/cancel regression passes, including the compressed S3 Select stream. Existing owned drain helpers can still have independent timeout concerns; those are not newly caused by this Go change.
• ALPN and custom connections: mc's TLS dialer returns a real *tls.Conn; its deadline wrapper is on the TCP dial path. No new accidental HTTP/2 opt-in from the expanded ConnectionState interface support was identified in these transports. Console keeps its existing transport policy.
• Removed switches: no maintained runtime/config reliance on tlsrsakex, tls3des, tls10server, x509keypairleaf, or asynctimerchan was found. Existing explicit cipher policies continue to be explicit. Timer uses in package certificate reload and license refresh do not depend on buffered timer-channel length/capacity. Unix EOF error changes do not expose a matching owned error-type assumption on the reviewed Unix-socket paths.
• Platform floor: application READMEs now identify macOS 13 as the minimum for Go 1.27 binaries. No Windows, old macOS, or PowerPC runtime claim is made.

Validation and delivery

• Server: focused TLS tests on macOS and Linux; macOS race run; crypto, HTTP, grid, and all internal/config/... tests; full Linux Server build and the three integration scenarios above.
• silo-pkg: make test (lint and full race suite) on Go 1.27.1; full suite on Go 1.26.5, preserving the library floor.
• mc: complete ./cmd suite, including the new TLS test and existing S3 Select early-cancel test.
• Console: complete ./api/... ./pkg/... suites, including identity and STS validation and the new handshake tests.

Go lint reports zero issues in all four repositories. Server's optional spelling check is skipped because typos is not installed. Its lint target used the already-installed matching golangci-lint v2.13.1 after a redundant download was stopped; mc's lint was rerun serially after the global linter lock prevented the first concurrent attempt. These tooling retries did not require source changes.

An independent Fable 5.1 Max review reproduced the old-code failures and ran the new TLS tests with the race detector. Its shuffled mc and Console suites passed. One shuffled silo-pkg run failed the untouched certs.TestValidPairAfterWrite; that test passed three plain reruns, the same shuffle seed, and the full certs package rerun. The certs package runs in a separate test binary from the changed env tests; this was classified as an existing timing flake.

The review also found that committing the diagnostic fixture made rebrand CI count synthetic IdP paths as product routes. The guard now excludes docs/investigations/; the product compatibility baseline is unchanged. The full proposed file set passes the guard, and a negative control adding a product route still fails. After the review corrections, a fresh Linux Server build passed all three integration scenarios again; the new binary identity and rerun results are recorded in the evidence file's post_review_validation.

All source and dependency choices remain those of the maintained PGSTY stack. go.mod and go.sum are unchanged in every repository. The Server uses its existing pinned Console/pkg/mc modules; the companion changes are tests and documentation, so no replacement graph or unpublished dependency version is needed to build the runtime fix.

Validation used separate worktrees for Server, silo-pkg, mc, and Console, leaving the original repositories and their main branches untouched. Build identities, fixture results, and command outcomes are recorded in go127-stack-evidence.json. Absolute paths and branch names in the captured evidence identify the environment at recording time. The recorded module graph identifies the dependencies selected for those builds; go.sum also retains checksums for unselected versions. Local image IDs and temporary paths are historical evidence, not portable setup instructions.