feat: add a trusted-proxy boundary for the client source address

The address MinIO attributes a request to is read from X-Forwarded-For,
X-Real-IP or RFC 7239 Forwarded, and never from the connection unless all
three are absent. It becomes aws:SourceIp and the audit remotehost field,
so any client that can reach the API port chooses the value an IpAddress
condition is evaluated against and the address every logged action is
attributed to.

MINIO_API_TRUSTED_PROXIES now selects who may make that claim:

  unset     the historical behaviour, unchanged
  none      no forwarded header is believed; the TCP peer wins
  <CIDRs>   believed only from listed peers, chains read right-to-left

Reading right-to-left is what makes an appending proxy safe: each hop
appends the peer it actually saw, so an entry a client injected can only
sit to the left of one a proxy wrote. The stock nginx recipe
$proxy_add_x_forwarded_for appends, which leaves the client's entry
left-most - exactly where the untrusted path reads - so a deployment with
no direct route to the API port was forgeable too.

_MINIO_API_XFF_HEADER is deliberately untouched, in semantics and in read
timing. Widening it to mean "trust nothing" was implemented and reverted:
it is the only part of this change that could alter a deployed
configuration, and the new variable expresses the same guarantee at no
compatibility cost. Upstream's TestXFFDisabled is retained verbatim.

Notes on the allow-list mode, all covered by tests:

  - it must name proxies, not the subnet they sit in; listed entries are
    skipped while walking, so a range covering clients lets them forge
  - a cluster must list its own nodes, because MinIO forwards between
    them and a client can force a hop via the ListObjectsV2 token
  - loopback is trusted as a peer, not as a chain entry, so FTP and SFTP
    keep attributing their sessions
  - the node-to-node forwarder drops X-Real-IP and Forwarded from a peer
    not entitled to have set them
  - the walk scans the header in place and stops after 100 hops, so a
    long chain costs neither allocation nor unbounded work

No behaviour change for any deployment that does not set the new
variable: the untrusted path is a verbatim copy of the previous function
body, differentially verified against it over ~5.1M header combinations.
The LDAP STS allow-list now shares the list parser as pure code motion,
verified identical across every combination of 37 allow-list values and
21 peer addresses.

Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Feng Ruohang
2026-08-04 13:09:06 +08:00
parent 744a9dcd71
commit fe6dc47804
10 changed files with 1369 additions and 74 deletions
+11
View File
@@ -181,6 +181,17 @@ func ipv6fix(clientIP string) string {
}
func (rw *headerRewriter) Rewrite(req *http.Request) {
// Forwarding this request attaches the node's own identity to it, so anything
// the peer was not entitled to claim has to go first. X-Forwarded-For needs no
// such handling: the reverse proxy appends the peer we actually saw, and the
// receiving node reaches that entry before any the client injected to its
// left. X-Real-IP and RFC 7239 Forwarded carry no chain, so a client's copy
// would otherwise arrive at the next node vouched for by this one.
if !TrustsForwardedHeaders(req) {
req.Header.Del(xRealIP)
req.Header.Del(forwarded)
}
if clientIP, _, err := net.SplitHostPort(req.RemoteAddr); err == nil {
clientIP = ipv6fix(clientIP)
if req.Header.Get(xRealIP) == "" {
+338 -14
View File
@@ -54,8 +54,116 @@ var (
protoRegex = regexp.MustCompile(`(?i)^(;|,| )+(?:proto=)(https|http)`)
)
// Used to disable all processing of the X-Forwarded-For header in source IP discovery.
var enableXFFHeader = env.Get("_MINIO_API_XFF_HEADER", config.EnableOn) == config.EnableOn
// Environment variables governing how much of a request's claimed source address
// the server is willing to believe.
const (
// EnvXFFHeader disables processing of X-Forwarded-For, and only of
// X-Forwarded-For. Inherited from upstream with its meaning deliberately
// unchanged: it is a parsing switch, not a trust boundary. Setting it to
// "off" still leaves X-Real-IP and RFC 7239 Forwarded honored, so it is not
// a way to stop a client naming its own address - EnvTrustedProxies is.
EnvXFFHeader = "_MINIO_API_XFF_HEADER"
// EnvTrustedProxies selects the trust policy. Unset honors forwarded headers
// from any peer, which is the historical behavior; TrustNoProxies believes
// none of them; anything else is a list of peer addresses and CIDR blocks
// whose headers are honored, which turns the source address from a claim any
// client can make into one only a named proxy can make.
EnvTrustedProxies = "MINIO_API_TRUSTED_PROXIES"
// TrustNoProxies is the EnvTrustedProxies value that believes no forwarded
// source-address header from anyone, whichever of the three it arrives in.
TrustNoProxies = "none"
)
// sourceIPTrust decides which peers may tell the server where a request came
// from. The address they choose becomes aws:SourceIp and the audit client
// address, so this is an access-control decision, not a logging preference.
type sourceIPTrust int
const (
// trustAnyPeer honors forwarded headers from whoever sent them. Historical
// default, sound only where every route to the API port passes through a
// proxy that overwrites those headers.
trustAnyPeer sourceIPTrust = iota
// trustNoPeer ignores forwarded headers; the source address is the TCP peer.
trustNoPeer
// trustListedPeers honors forwarded headers only from allow-listed peers.
trustListedPeers
)
var (
sourceIPPolicy sourceIPTrust
trustedProxies config.TrustedProxies
)
// enableXFFHeader carries upstream's X-Forwarded-For parsing switch. It applies
// within whichever trust policy is in force, and is orthogonal to it.
//
// Read at package initialisation, exactly as upstream does, and deliberately not
// re-read by ConfigureSourceIPTrust. Environment files are loaded after this
// point, so upstream silently ignores the setting when it is written there;
// picking it up would make an already-deployed setting start taking effect,
// which is a behavior change this fork has no reason to make on its way past.
var enableXFFHeader = env.Get(EnvXFFHeader, config.EnableOn) == config.EnableOn
// init establishes a policy from the process environment so that no code path
// runs without one. A server re-applies it from ConfigureSourceIPTrust once the
// environment is complete; an error here is dropped because the failure mode it
// leaves behind - trustNoPeer - is the safe one, and it is reported there.
func init() {
_ = ConfigureSourceIPTrust()
}
// ConfigureSourceIPTrust reads the trust policy out of the environment and
// installs it. The server calls this after loading MINIO_CONFIG_ENV_FILE, which
// happens long after package initialisation: a policy read only at init would
// miss every deployment that configures MinIO through an environment file and
// would silently leave the historical trust-any-peer mode in place.
//
// Not safe to call once requests are being served.
func ConfigureSourceIPTrust() error {
// Read through LookupEnv rather than env.Get, which discards the error from a
// remote env:// lookup and hands back the empty string. That would read here
// as "unset" and quietly reinstate the trust-any-peer default: a fetch that
// failed is not a statement that no proxy is trusted.
value, _, _, err := env.LookupEnv(EnvTrustedProxies)
if err != nil {
sourceIPPolicy, trustedProxies = trustNoPeer, nil
return config.Errorf("%s could not be read: %v", EnvTrustedProxies, err)
}
policy, prefixes, err := lookupSourceIPTrust(value)
sourceIPPolicy, trustedProxies = policy, prefixes
return err
}
// lookupSourceIPTrust derives the trust policy from EnvTrustedProxies. A
// malformed allow-list yields trustNoPeer alongside the error, so that a caller
// which fails to check the error still fails closed.
func lookupSourceIPTrust(proxies string) (sourceIPTrust, config.TrustedProxies, error) {
switch strings.ToLower(strings.TrimSpace(proxies)) {
case "":
return trustAnyPeer, nil, nil
case TrustNoProxies, config.EnableOff:
return trustNoPeer, nil, nil
}
prefixes, err := config.ParseTrustedProxies(proxies, EnvTrustedProxies)
if err != nil {
return trustNoPeer, nil, err
}
if len(prefixes) == 0 {
// Separators and nothing else. The value is not blank, so it was written
// on purpose, yet it names no proxy. Falling back to the permissive
// default here would answer a deliberate configuration with the one
// behavior it cannot have been asking for.
return trustNoPeer, nil, config.Errorf("%s %q names no proxy", EnvTrustedProxies, proxies)
}
return trustListedPeers, prefixes, nil
}
// GetSourceScheme retrieves the scheme from the X-Forwarded-Proto and RFC7239
// Forwarded headers (in that order).
@@ -85,20 +193,64 @@ func GetSourceScheme(r *http.Request) string {
return scheme
}
// SECURITY NOTE: these headers are trusted from any peer. There is no
// trusted-proxy boundary, X-Forwarded-For is honoured by default, and X-Real-IP
// and Forwarded are not gated at all, so any client that can reach the server
// directly can set the address the rest of the process believes it came from.
// That includes aws:SourceIp, which means an IpAddress policy condition is not
// enforceable on a directly reachable deployment - put MinIO behind a proxy
// that overwrites these headers, or set _MINIO_API_XFF_HEADER=off and keep the
// other two out at the edge. Adding a trusted-proxy allowlist here would change
// what every deployment behind a load balancer resolves to, so it is recorded
// rather than changed.
// GetSourceIPFromHeaders retrieves the client address a request claims to come
// from, or the empty string when no claim may be believed and the caller should
// fall back to the TCP peer.
//
// GetSourceIPFromHeaders retrieves the IP from the X-Forwarded-For, X-Real-IP
// and RFC7239 Forwarded headers (in that order)
// SECURITY CONTRACT. The value returned here becomes aws:SourceIp and the audit
// log's client address, so whoever controls it controls both IP-based policy
// decisions and the attribution of every logged action. Which of the three
// interchangeable source-address headers - X-Forwarded-For, X-Real-IP, RFC 7239
// Forwarded - a client sends is irrelevant; they are equally forgeable, so the
// trust decision is taken over all three at once by EnvTrustedProxies:
//
// - Unset (default). Any peer may set the headers, and the left-most
// X-Forwarded-For entry wins. aws:SourceIp is then only as trustworthy as
// the network: any client that can open a connection to the API port can
// name its own address. An IpAddress condition is not enforceable under this
// mode unless every route to the port passes through a proxy that overwrites
// all three headers. Note that the stock nginx recipe,
// proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for, appends
// rather than overwrites, and so leaves a client-supplied entry in the
// left-most position where this mode will read it.
//
// - TrustNoProxies. No header is believed; the source address is always the
// TCP peer.
//
// - A list of addresses and CIDR blocks. Headers are believed only when the
// TCP peer is on the allow-list, and the chains are read right-to-left. This
// is the only mode under which aws:SourceIp is enforceable against a client
// with direct network access.
//
// EnvXFFHeader is not one of these modes. It suppresses parsing of
// X-Forwarded-For within whichever mode is in force, leaving X-Real-IP and
// Forwarded honored, so it cannot stop a client naming its own address - a
// client refused one header simply sends another. It is kept at its upstream
// meaning rather than widened into a trust switch, because widening it would
// change what an already-deployed setting resolves to; TrustNoProxies is the
// setting that means what it says.
//
// The scheme headers are deliberately not covered: GetSourceScheme feeds the
// Location URL rather than a policy decision, and suppressing it would hand
// http:// URLs to every deployment terminating TLS at a proxy.
func GetSourceIPFromHeaders(r *http.Request) string {
switch sourceIPPolicy {
case trustNoPeer:
return ""
case trustListedPeers:
if !peerMayForward(r) {
return ""
}
return forwardedSourceIP(r)
default:
return unverifiedSourceIP(r)
}
}
// unverifiedSourceIP reads the headers the way MinIO always has, taking the
// left-most X-Forwarded-For entry and falling back through X-Real-IP to RFC 7239
// Forwarded. Every value here is a claim by whoever sent it.
func unverifiedSourceIP(r *http.Request) string {
var addr string
if enableXFFHeader {
@@ -136,6 +288,178 @@ func GetSourceIPFromHeaders(r *http.Request) string {
return addr
}
// forwardedSourceIP resolves the client address for a request whose peer is an
// allow-listed proxy.
//
// The forwarding chains are read right-to-left, stepping over entries that name
// a configured proxy, and the first remaining address wins. That direction is
// what makes the header usable: each proxy appends the peer it actually saw, so
// an entry a client injected sits to the left of the one its proxy wrote, and
// the walk stops before reaching it.
//
// That holds only while the allow-list names proxies. A list broad enough to
// cover addresses clients also occupy makes those clients skippable too, and the
// walk then continues past a real client into whatever it placed to the left. A
// broad list therefore does not merely trust more peers - it lets those peers
// forge. Configure proxy addresses, not the subnet the proxies sit in.
//
// X-Real-IP carries no chain and so cannot be checked against the allow-list; it
// is taken at face value, and only when the chain headers yield nothing. The
// deployment contract is that a configured proxy overwrites whichever headers it
// sets. A proxy that instead relays a client's copy is choosing to let the
// client answer this question, and no amount of parsing here can undo that.
//
// Note this orders the headers differently from getSTSLDAPTrustedProxySourceIP
// (cmd/sts-handlers.go), which prefers X-Real-IP. Neither order is safe for
// every proxy - preferring X-Real-IP is wrong where the proxy authors only
// X-Forwarded-For and relays the client's X-Real-IP (AWS ALB), and preferring
// X-Forwarded-For is wrong in the mirror case (an nginx that sets only
// X-Real-IP). The chain-validated header is preferred here because this decides
// access control rather than rate-limit bucketing, so the value that can be
// checked against the allow-list should win; it also keeps the header precedence
// identical to the default mode. Deployments whose proxy authors only X-Real-IP
// must strip X-Forwarded-For at the edge.
func forwardedSourceIP(r *http.Request) string {
if enableXFFHeader {
if addr := untrustedHop(r.Header.Values(xForwardedFor), canonicalSourceIP); addr != "" {
return addr
}
}
if addr := canonicalSourceIP(lastValue(r.Header.Values(xRealIP))); addr != "" {
return addr
}
return untrustedHop(r.Header.Values(forwarded), forwardedForAddr)
}
// maxForwardedHops bounds how far back along a chain the walk will look.
//
// Real chains are a handful of hops and the answer sits at the right-hand end,
// so this is far above anything a deployment produces. It exists because the
// chain arrives from the network: without it, a client behind a trusted proxy
// could spend a megabyte of header on a walk this server has to finish. Running
// out of budget yields no address, so the request falls back to the peer - the
// same safe direction as a chain of entirely trusted hops.
const maxForwardedHops = 100
// lastValue returns the final line of a repeated header. X-Real-IP carries no
// chain to walk, so where a client's line and a proxy's line both survive, the
// later one is the one added closer to this server.
func lastValue(values []string) string {
if len(values) == 0 {
return ""
}
return values[len(values)-1]
}
// untrustedHop walks a forwarding chain from the right and returns the first
// address that is not itself a trusted hop, using addrOf to read one element.
//
// values holds the header's lines in the order received; repeated field lines
// are equivalent to one comma-joined line, and proxies disagree on which they
// emit - Go's reverse proxy and nginx rewrite a single line, while HAProxy's
// forwardfor adds a second. Reading only the first would leave a client's own
// line ahead of the proxy's, which is the position this walk exists to step over.
//
// The scan runs backwards over the raw text rather than over a split slice, so
// that a long chain costs no allocation.
func untrustedHop(values []string, addrOf func(string) string) string {
budget := maxForwardedHops
for i := len(values) - 1; i >= 0 && budget > 0; i-- {
for s := values[i]; len(s) > 0 && budget > 0; budget-- {
element := s
if comma := strings.LastIndexByte(s, ','); comma >= 0 {
element, s = s[comma+1:], s[:comma]
} else {
s = ""
}
if addr := addrOf(element); addr != "" && !isTrustedHop(addr) {
return addr
}
}
}
return ""
}
// forwardedForAddr reads the for= address out of one RFC 7239 Forwarded element.
func forwardedForAddr(element string) string {
match := forRegex.FindStringSubmatch(element)
if len(match) <= 1 {
return ""
}
return canonicalSourceIP(strings.Trim(match[1], `"`))
}
// canonicalSourceIP reduces one chain element to a bare IP address, or to the
// empty string when it does not hold one. Ports, brackets and surrounding space
// are stripped. Values an allow-list cannot reason about - a hostname, or an
// RFC 7239 obfuscated identifier such as for=_gazonk - are discarded rather than
// passed on, since a trust decision cannot be made about them.
func canonicalSourceIP(addr string) string {
addr = strings.TrimSpace(addr)
if addr == "" {
return ""
}
if host, _, err := net.SplitHostPort(addr); err == nil {
addr = host
}
addr = strings.TrimPrefix(addr, "[")
addr = strings.TrimSuffix(addr, "]")
// A link-local peer arrives with a zone ("fe80::1%eth0"), which net.ParseIP
// rejects outright - so without this the address resolves to nothing and the
// peer could never be a configured proxy.
if zone := strings.IndexByte(addr, '%'); zone != -1 {
addr = addr[:zone]
}
if ip := net.ParseIP(addr); ip != nil {
return ip.String()
}
return ""
}
// isTrustedHop reports whether a chain entry names one of the configured
// proxies, and so is an address to step over rather than attribute a request to.
//
// This deliberately does not extend the loopback exemption peerMayForward
// grants. Loopback is trusted as a *peer* because the local front-ends connect
// from there; a 127.0.0.1 entry inside a forwarding chain is just an address,
// and stepping over it would discard a real answer in favor of whatever sits
// further left.
func isTrustedHop(addr string) bool {
return trustedProxies.Contains(addr)
}
// peerMayForward reports whether the request's TCP peer is allowed to speak for
// someone else.
//
// Loopback is always allowed, configured or not. The FTP and SFTP front-ends
// reach the S3 layer over 127.0.0.1 and declare their session's client with
// X-Forwarded-For (see cmd/sftp-server-driver.go), so excluding loopback would
// attribute every FTP and SFTP request to the server itself.
func peerMayForward(r *http.Request) bool {
peer := canonicalSourceIP(r.RemoteAddr)
if peer == "" {
return false
}
if ip := net.ParseIP(peer); ip != nil && ip.IsLoopback() {
return true
}
return trustedProxies.Contains(peer)
}
// TrustsForwardedHeaders reports whether the source-address headers already
// present on a request may be believed. The node-to-node forwarder uses this to
// decide whether to relay what it received or overwrite it.
func TrustsForwardedHeaders(r *http.Request) bool {
switch sourceIPPolicy {
case trustNoPeer:
return false
case trustListedPeers:
return peerMayForward(r)
default:
return true
}
}
// GetSourceIPRaw retrieves the IP from the request headers
// and falls back to r.RemoteAddr when necessary.
// however returns without bracketing.
+568
View File
@@ -19,7 +19,11 @@ package handlers
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/minio/minio/internal/config"
)
type headerTest struct {
@@ -85,6 +89,33 @@ func TestGetSourceIP(t *testing.T) {
}
}
// withSourceIPTrust installs the trust policy the given EnvTrustedProxies value
// would produce and restores the previous one when the test ends.
func withSourceIPTrust(t *testing.T, proxies string) {
t.Helper()
policy, prefixes, err := lookupSourceIPTrust(proxies)
if err != nil {
t.Fatalf("%s=%q: unexpected error: %v", EnvTrustedProxies, proxies, err)
}
prevPolicy, prevProxies := sourceIPPolicy, trustedProxies
sourceIPPolicy, trustedProxies = policy, prefixes
t.Cleanup(func() {
sourceIPPolicy, trustedProxies = prevPolicy, prevProxies
})
}
func requestFrom(peer string, header http.Header) *http.Request {
if header == nil {
header = http.Header{}
}
return &http.Request{RemoteAddr: peer, Header: header}
}
// Upstream's test, unchanged: _MINIO_API_XFF_HEADER keeps its original meaning,
// suppressing X-Forwarded-For alone and leaving X-Real-IP to answer instead.
// That behavior is exactly why it is not the trust switch - see the test below.
func TestXFFDisabled(t *testing.T) {
req := &http.Request{
Header: http.Header{
@@ -108,3 +139,540 @@ func TestXFFDisabled(t *testing.T) {
t.Errorf("wrong header, xff is disabled: got %s, want: 1.1.1.1", res)
}
}
// TestTrustNoProxiesIgnoresEveryForwardedHeader pins the property the setting
// exists for: an operator who turns forwarded-header trust off cannot be talked
// out of it by switching to another header. This is the guarantee
// _MINIO_API_XFF_HEADER=off does not provide, since suppressing one of three
// interchangeable headers only moves the answer to the next one.
func TestTrustNoProxiesIgnoresEveryForwardedHeader(t *testing.T) {
const peer = "203.0.113.9:44321"
forged := []headerTest{
{xForwardedFor, "8.8.8.8", "203.0.113.9"},
{xRealIP, "8.8.8.8", "203.0.113.9"},
{forwarded, "for=8.8.8.8", "203.0.113.9"},
}
// Default mode honors every one of them, which is the behavior being opted
// out of.
withSourceIPTrust(t, "")
for _, v := range forged {
if res := GetSourceIP(requestFrom(peer, http.Header{v.key: []string{v.val}})); res != "8.8.8.8" {
t.Errorf("%s: default mode should honor the header: got %s, want 8.8.8.8", v.key, res)
}
}
withSourceIPTrust(t, TrustNoProxies)
for _, v := range forged {
res := GetSourceIP(requestFrom(peer, http.Header{v.key: []string{v.val}}))
if res != v.expected {
t.Errorf("%s should be ignored under %s: got %s, want %s", v.key, TrustNoProxies, res, v.expected)
}
}
// All three at once, in case one merely shadows another.
res := GetSourceIP(requestFrom(peer, http.Header{
xForwardedFor: []string{"8.8.8.8"},
xRealIP: []string{"1.1.1.1"},
forwarded: []string{"for=9.9.9.9"},
}))
if res != "203.0.113.9" {
t.Errorf("wrong source with all headers set and no proxies trusted: got %s, want 203.0.113.9", res)
}
}
// TestTrustedProxiesResolution covers the allow-list mode, where the peer decides
// whether the request may speak for anyone else.
func TestTrustedProxiesResolution(t *testing.T) {
withSourceIPTrust(t, "10.0.0.0/8, 192.0.2.7")
tests := []struct {
name string
peer string
header http.Header
want string
}{{
name: "trusted proxy overwrote the header",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"1.2.3.4"}},
want: "1.2.3.4",
}, {
// The stock nginx $proxy_add_x_forwarded_for recipe appends, so a client
// that sends its own X-Forwarded-For keeps the left-most slot. Reading
// right-to-left is what steps over it.
name: "client injected the left-most entry",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"9.9.9.9, 1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "two proxy hops after the client",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"9.9.9.9, 1.2.3.4, 10.0.0.5"}},
want: "1.2.3.4",
}, {
name: "untrusted peer may not speak for anyone",
peer: "203.0.113.9:44321",
header: http.Header{xForwardedFor: []string{"1.2.3.4"}},
want: "203.0.113.9",
}, {
name: "untrusted peer cannot fall back to X-Real-IP either",
peer: "203.0.113.9:44321",
header: http.Header{xRealIP: []string{"1.2.3.4"}},
want: "203.0.113.9",
}, {
name: "untrusted peer cannot fall back to Forwarded either",
peer: "203.0.113.9:44321",
header: http.Header{forwarded: []string{"for=1.2.3.4"}},
want: "203.0.113.9",
}, {
name: "X-Real-IP from a trusted proxy",
peer: "192.0.2.7:9000",
header: http.Header{xRealIP: []string{"1.2.3.4"}},
want: "1.2.3.4",
}, {
// FTP and SFTP reach the S3 layer over loopback and declare the session's
// client with X-Forwarded-For.
name: "loopback front-end is trusted implicitly",
peer: "127.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "Forwarded chain is also read right-to-left",
peer: "10.0.0.1:9000",
header: http.Header{forwarded: []string{"for=9.9.9.9, for=1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "unparseable chain entries are skipped",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"_gazonk, workstation.local, 1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "a chain of nothing but trusted hops falls back to the peer",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"10.0.0.2, 10.0.0.3"}},
want: "10.0.0.1",
}, {
name: "X-Forwarded-For outranks X-Real-IP, as in the untrusted path",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"1.2.3.4"}, xRealIP: []string{"9.9.9.9"}},
want: "1.2.3.4",
}, {
name: "IPv6 client keeps its bracketed form",
peer: "10.0.0.1:9000",
header: http.Header{xForwardedFor: []string{"2001:db8::1"}},
want: "[2001:db8::1]",
}, {
name: "no headers at all",
peer: "10.0.0.1:9000",
want: "10.0.0.1",
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if res := GetSourceIP(requestFrom(tt.peer, tt.header)); res != tt.want {
t.Errorf("got %s, want %s", res, tt.want)
}
})
}
}
// TestTrustedProxiesRepeatedHeaderLines covers proxies that add a second header
// line instead of extending the first. HAProxy's `option forwardfor` does this,
// and Header.Get would return only the client's line - putting the value the
// right-to-left walk exists to step over back in front of it.
func TestTrustedProxiesRepeatedHeaderLines(t *testing.T) {
withSourceIPTrust(t, "10.0.0.0/8")
tests := []struct {
name string
header http.Header
want string
}{{
name: "client line then proxy line",
header: http.Header{xForwardedFor: []string{"9.9.9.9", "1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "client sends several lines",
header: http.Header{xForwardedFor: []string{"9.9.9.9", "8.8.8.8", "1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "mixed: a comma chain and an added line",
header: http.Header{xForwardedFor: []string{"9.9.9.9, 8.8.8.8", "1.2.3.4"}},
want: "1.2.3.4",
}, {
name: "repeated Forwarded lines",
header: http.Header{forwarded: []string{"for=9.9.9.9", "for=1.2.3.4"}},
want: "1.2.3.4",
}, {
// The proxy's own line is the later one; a client's cannot displace it.
name: "repeated X-Real-IP takes the last line",
header: http.Header{xRealIP: []string{"9.9.9.9", "1.2.3.4"}},
want: "1.2.3.4",
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if res := GetSourceIP(requestFrom("10.0.0.1:9000", tt.header)); res != tt.want {
t.Errorf("got %s, want %s", res, tt.want)
}
})
}
}
// The chain arrives from the network, so the work it costs has to be bounded and
// must not scale into the heap. A client sitting behind a trusted proxy can make
// this header as long as the server's header limit allows.
func TestTrustedProxiesBoundsChainWork(t *testing.T) {
withSourceIPTrust(t, "10.0.0.0/8")
// The budget runs out before reaching the left-hand entry, so the request
// falls back to the peer - the same safe direction as an all-trusted chain.
hops := []string{"1.2.3.4"}
for range maxForwardedHops + 5 {
hops = append(hops, "10.0.0.9")
}
res := GetSourceIP(requestFrom("10.0.0.1:9000", http.Header{
xForwardedFor: []string{strings.Join(hops, ",")},
}))
if res != "10.0.0.1" {
t.Errorf("an over-long chain should fall back to the peer: got %s, want 10.0.0.1", res)
}
// A chain just inside the budget must still resolve normally.
shortEnough := []string{"1.2.3.4"}
for range 10 {
shortEnough = append(shortEnough, "10.0.0.9")
}
res = GetSourceIP(requestFrom("10.0.0.1:9000", http.Header{
xForwardedFor: []string{strings.Join(shortEnough, ",")},
}))
if res != "1.2.3.4" {
t.Errorf("a normal chain regressed: got %s, want 1.2.3.4", res)
}
// Scanning must happen in place. Splitting a megabyte of separators would
// turn one request's header into tens of megabytes of slice headers.
huge := []string{strings.Repeat(",", 1<<20)}
if allocs := testing.AllocsPerRun(2, func() {
untrustedHop(huge, canonicalSourceIP)
}); allocs > 0 {
t.Errorf("walking a chain allocated %v times; it must scan the header in place", allocs)
}
}
// TestTrustedProxiesDoNotChangeUntrustedMode guards the release-window promise:
// with neither setting present, resolution is exactly what it was before.
func TestTrustedProxiesDoNotChangeUntrustedMode(t *testing.T) {
withSourceIPTrust(t, "")
tests := []struct {
header http.Header
want string
}{
{http.Header{xForwardedFor: []string{"9.9.9.9, 1.2.3.4"}}, "9.9.9.9"},
{http.Header{xForwardedFor: []string{"_gazonk"}}, "_gazonk"},
{http.Header{xRealIP: []string{"workstation.local"}}, "workstation.local"},
{http.Header{forwarded: []string{`for="[2001:db8:cafe::17]:4711`}}, "[2001:db8:cafe::17]"},
}
for _, tt := range tests {
if res := GetSourceIP(requestFrom("203.0.113.9:44321", tt.header)); res != tt.want {
t.Errorf("untrusted mode changed for %v: got %s, want %s", tt.header, res, tt.want)
}
}
}
// A chain entry naming loopback is an ordinary address, not a hop to step over.
// Stepping over it would discard the real answer for whatever a client put to
// its left - the same failure a too-broad allow-list produces.
func TestTrustedProxiesDoesNotSkipLoopbackChainEntries(t *testing.T) {
withSourceIPTrust(t, "10.0.0.0/8")
res := GetSourceIP(requestFrom("10.0.0.1:9000", http.Header{
xForwardedFor: []string{"8.8.8.8, 127.0.0.1"},
}))
if res != "127.0.0.1" {
t.Errorf("loopback chain entry was skipped: got %s, want 127.0.0.1", res)
}
}
// A link-local peer presents with a zone. net.ParseIP rejects those outright, so
// without stripping it the peer resolves to nothing and could never be trusted.
func TestTrustedProxiesAcceptsZonedIPv6Peer(t *testing.T) {
withSourceIPTrust(t, "fe80::1")
res := GetSourceIP(requestFrom("[fe80::1%eth0]:9000", http.Header{
xForwardedFor: []string{"1.2.3.4"},
}))
if res != "1.2.3.4" {
t.Errorf("zoned IPv6 peer was not trusted: got %s, want 1.2.3.4", res)
}
}
// An allow-list entry written in IPv4-mapped form grants no trust: it is a
// 128-bit prefix, and the peer address has already been reduced to its plain
// form by the time it is matched. This is a wart, and it is deliberately left
// alone - rewriting such entries was tried and reverted, because the rewrite
// reached no real request and changed what the shared parser means for the LDAP
// STS allow-list that was using it first. The direction is fail-closed: the
// peer is simply not trusted, and the request is attributed to it.
func TestTrustedProxiesIgnoreMappedIPv4AllowListEntries(t *testing.T) {
withSourceIPTrust(t, "::ffff:192.168.1.10")
res := GetSourceIP(requestFrom("192.168.1.10:9000", http.Header{
xForwardedFor: []string{"1.2.3.4"},
}))
if res != "192.168.1.10" {
t.Errorf("a mapped entry granted trust: got %s, want the peer itself (192.168.1.10)", res)
}
}
// The policy has to be re-read after the server loads MINIO_CONFIG_ENV_FILE.
// Reading it only at package initialisation left every environment-file
// deployment on the historical trust-any-peer mode with nothing reported.
func TestConfigureSourceIPTrustReadsEnvironmentLate(t *testing.T) {
prevPolicy, prevProxies := sourceIPPolicy, trustedProxies
t.Cleanup(func() {
sourceIPPolicy, trustedProxies = prevPolicy, prevProxies
})
t.Setenv(EnvTrustedProxies, "10.0.0.0/8")
if err := ConfigureSourceIPTrust(); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if sourceIPPolicy != trustListedPeers {
t.Fatalf("policy = %v, want trustListedPeers", sourceIPPolicy)
}
if res := GetSourceIP(requestFrom("203.0.113.9:44321", http.Header{
xForwardedFor: []string{"8.8.8.8"},
})); res != "203.0.113.9" {
t.Errorf("late-configured allow-list not in effect: got %s, want 203.0.113.9", res)
}
// A malformed value must be reported, not silently ignored.
t.Setenv(EnvTrustedProxies, "nonsense")
if err := ConfigureSourceIPTrust(); err == nil {
t.Error("malformed allow-list was accepted")
}
if sourceIPPolicy != trustNoPeer {
t.Errorf("policy = %v after a malformed value, want trustNoPeer", sourceIPPolicy)
}
}
// A value written as a remote env:// reference whose fetch fails must not be
// mistaken for an unset variable. env.Get drops that error and returns "", which
// would land on the permissive default at exactly the moment the operator's
// intent could not be read.
func TestConfigureSourceIPTrustRejectsUnreadableValue(t *testing.T) {
prevPolicy, prevProxies := sourceIPPolicy, trustedProxies
t.Cleanup(func() {
sourceIPPolicy, trustedProxies = prevPolicy, prevProxies
})
// Port 1 on loopback refuses immediately, and no cached _-prefixed fallback
// exists for this key.
t.Setenv(EnvTrustedProxies, "env://user:pass@127.0.0.1:1/webhook/v1/getenv")
if err := ConfigureSourceIPTrust(); err == nil {
t.Fatal("an unreadable value was accepted")
}
if sourceIPPolicy != trustAnyPeer {
return // failed closed, which is the point
}
t.Error("an unreadable value resolved to trustAnyPeer; any peer may forge the source address")
}
// _MINIO_API_XFF_HEADER must keep upstream's read timing. Upstream takes it at
// package initialisation, before environment files are loaded, so a deployment
// that writes it into MINIO_CONFIG_ENV_FILE has it silently ignored. Picking it
// up in ConfigureSourceIPTrust would make that already-deployed setting start
// taking effect, which is the one kind of change this rework exists to avoid.
func TestConfigureSourceIPTrustLeavesXFFSwitchAlone(t *testing.T) {
prevXFF := enableXFFHeader
t.Cleanup(func() { enableXFFHeader = prevXFF })
enableXFFHeader = true
t.Setenv(EnvXFFHeader, config.EnableOff)
if err := ConfigureSourceIPTrust(); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !enableXFFHeader {
t.Error("the X-Forwarded-For switch was re-read from the environment")
}
}
// forwardThroughNode sends a client request through the node-to-node forwarder
// and reports what the receiving node resolves as the source address. The
// receiving node's peer is the forwarding node, not the client, which is where
// the trust policy has its least obvious consequence.
func forwardThroughNode(t *testing.T, proxies, clientPeer, forwardingNode string) string {
t.Helper()
withSourceIPTrust(t, proxies)
var resolved string
receiving := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
// The test client's real peer is loopback, which is trusted
// unconditionally; substitute the address the forwarding node would
// present so the allow-list decision is the one under test.
r.RemoteAddr = forwardingNode
resolved = GetSourceIPRaw(r)
}))
defer receiving.Close()
in := httptest.NewRequest(http.MethodGet, receiving.URL+"/probe", nil)
in.RemoteAddr = clientPeer
in.RequestURI = "/probe"
in.URL.Scheme = "http"
in.URL.Host = strings.TrimPrefix(receiving.URL, "http://")
NewForwarder(&Forwarder{PassHost: true}).ServeHTTP(httptest.NewRecorder(), in)
return resolved
}
func TestForwardedBetweenNodesAttribution(t *testing.T) {
const (
client = "203.0.113.77:44321"
forwardingNode = "10.10.0.2:36000"
)
tests := []struct {
name string
proxies string
want string
}{{
name: "default mode carries the client through",
want: "203.0.113.77",
}, {
// Believing no header means believing nothing about the forwarding node
// either, so an internally forwarded request is attributed to it. There
// is no configuration that corrects this, which is why the documentation
// steers multi-node clusters to the allow-list instead.
name: "trusting nobody attributes to the forwarding node",
proxies: TrustNoProxies,
want: "10.10.0.2",
}, {
name: "allow-list omitting the cluster's own nodes",
proxies: "192.168.1.0/24",
want: "10.10.0.2",
}, {
name: "allow-list including the cluster's own nodes",
proxies: "192.168.1.0/24,10.10.0.2",
want: "203.0.113.77",
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := forwardThroughNode(t, tt.proxies, client, forwardingNode); got != tt.want {
t.Errorf("receiving node resolved %q, want %q", got, tt.want)
}
})
}
}
func TestLookupSourceIPTrust(t *testing.T) {
tests := []struct {
name string
proxies string
want sourceIPTrust
wantErr bool
}{
{name: "unset is the historical default", want: trustAnyPeer},
{name: "none", proxies: TrustNoProxies, want: trustNoPeer},
{name: "none is case-insensitive", proxies: "None", want: trustNoPeer},
{name: "off is accepted as a synonym", proxies: "off", want: trustNoPeer},
{name: "surrounding space is ignored", proxies: " none ", want: trustNoPeer},
{name: "allow-list", proxies: "10.0.0.0/8", want: trustListedPeers},
{name: "whitespace only is indistinguishable from unset", proxies: " ", want: trustAnyPeer},
// Written on purpose, but naming nobody. Falling back to the permissive
// default would be the one answer the operator cannot have wanted.
{name: "separators only rejected", proxies: ",", want: trustNoPeer, wantErr: true},
{name: "assorted separators rejected", proxies: " ,; ", want: trustNoPeer, wantErr: true},
// A catch-all would trust every peer and quietly undo the allow-list.
{name: "catch-all v4 rejected", proxies: "0.0.0.0/0", want: trustNoPeer, wantErr: true},
{name: "catch-all v6 rejected", proxies: "::/0", want: trustNoPeer, wantErr: true},
{name: "catch-all hidden in a list", proxies: "10.0.0.0/8,0.0.0.0/0", want: trustNoPeer, wantErr: true},
// A mapped spelling of the whole IPv4 space stays a /96 here, and so is
// accepted rather than rejected. It matches nothing, because a peer never
// reaches Contains in mapped form - the breadth guard covers what an
// operator would actually write, not every way to write it.
{name: "mapped whole-IPv4 space is a /96, not a catch-all", proxies: "::ffff:0:0/96", want: trustListedPeers},
{name: "garbage rejected", proxies: "10.0.0.0/8,nonsense", want: trustNoPeer, wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, _, err := lookupSourceIPTrust(tt.proxies)
if (err != nil) != tt.wantErr {
t.Fatalf("error = %v, wantErr %v", err, tt.wantErr)
}
// A rejected list must fail closed, not fall back to trusting everyone.
if got != tt.want {
t.Fatalf("policy = %v, want %v", got, tt.want)
}
})
}
}
func TestCanonicalSourceIP(t *testing.T) {
tests := []struct{ in, want string }{
{"1.2.3.4", "1.2.3.4"},
{" 1.2.3.4 ", "1.2.3.4"},
{"1.2.3.4:443", "1.2.3.4"},
{"2001:db8::1", "2001:db8::1"},
{"[2001:db8::1]", "2001:db8::1"},
{"[2001:db8::1]:443", "2001:db8::1"},
{"::ffff:1.2.3.4", "1.2.3.4"},
{"_gazonk", ""},
{"workstation.local", ""},
{"", ""},
}
for _, tt := range tests {
if got := canonicalSourceIP(tt.in); got != tt.want {
t.Errorf("canonicalSourceIP(%q) = %q, want %q", tt.in, got, tt.want)
}
}
}
// TestRewriteDropsUnvouchedClaims covers the node-to-node forwarder. Relaying a
// header the peer was not entitled to set would hand the client this node's
// authority at the next hop, which does trust it.
func TestRewriteDropsUnvouchedClaims(t *testing.T) {
rw := &headerRewriter{}
withSourceIPTrust(t, "10.0.0.0/8")
untrusted := requestFrom("203.0.113.9:44321", http.Header{
xRealIP: []string{"1.2.3.4"},
forwarded: []string{"for=1.2.3.4"},
})
rw.Rewrite(untrusted)
if got := untrusted.Header.Get(xRealIP); got != "203.0.113.9" {
t.Errorf("X-Real-IP from an untrusted peer should be replaced: got %s, want 203.0.113.9", got)
}
if got := untrusted.Header.Get(forwarded); got != "" {
t.Errorf("Forwarded from an untrusted peer should be dropped: got %s", got)
}
// A trusted proxy's own chain must survive, or a two-hop deployment loses the
// client it correctly identified.
trusted := requestFrom("10.0.0.1:9000", http.Header{
xRealIP: []string{"1.2.3.4"},
forwarded: []string{"for=1.2.3.4"},
})
rw.Rewrite(trusted)
if got := trusted.Header.Get(xRealIP); got != "1.2.3.4" {
t.Errorf("X-Real-IP from a trusted peer should be relayed: got %s, want 1.2.3.4", got)
}
if got := trusted.Header.Get(forwarded); got != "for=1.2.3.4" {
t.Errorf("Forwarded from a trusted peer should be relayed: got %s", got)
}
// Default mode must behave exactly as it did: fill in only what is missing.
withSourceIPTrust(t, "")
legacy := requestFrom("203.0.113.9:44321", http.Header{xRealIP: []string{"1.2.3.4"}})
rw.Rewrite(legacy)
if got := legacy.Header.Get(xRealIP); got != "1.2.3.4" {
t.Errorf("default mode should relay X-Real-IP untouched: got %s, want 1.2.3.4", got)
}
}