Compare commits

...

18 Commits

Author SHA1 Message Date
Alexey 9d5b896bb6 Merge pull request #949 from telemt/me-pool-convergence
ME: Autonomous&Lifecycle-bounded Pool Convergence
2026-10-05 11:27:50 +03:00
Alexey 556affc63a ME: Autonomous&Lifecycle-bounded Pool Convergence
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-05 06:02:08 +03:00
Alexey 9c31e970e6 Merge pull request #948 from telemt/web-resolve
WEB: Resolve opt-in decoy hostnames during config preparation
2026-10-04 14:43:23 +03:00
Alexey 3b924ab40c Bump -> 3.5.14 2026-10-04 14:32:35 +03:00
Alexey bc1c130f38 WEB: Resolve opt-in decoy hostnames during config preparation #929
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-04 14:32:06 +03:00
Alexey d3de9865cf Merge pull request #946 from telemt/http-conveyor
Bounded WEB HTTP conveyor with replay-safe streaming downlink added
2026-10-03 20:44:58 +03:00
Alexey e2f1c3f3a9 Bridge JS semicolon warnings fixed 2026-10-03 20:29:18 +03:00
Alexey dcce897d8b Bump -> 3.5.13 2026-10-03 16:05:22 +03:00
Alexey 823c6cbc7f Bounded WEB HTTP conveyor with replay-safe streaming downlink added
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-03 16:02:18 +03:00
Alexey c4555e25f3 Merge pull request #945 from telemt/bump/3.5.12
Bump -> 3.5.12
2026-10-03 13:01:07 +03:00
Alexey d15d50e92a Bump -> 3.5.12 2026-10-03 12:59:58 +03:00
Alexey 557ffe9ee0 Merge pull request #944 from telemt/me-nat-discovery
ME NAT Discovery Cancellation correctness
2026-10-03 12:56:57 +03:00
Alexey 84d5e98c11 ME NAT Discovery Cancellation correctness
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-03 12:36:29 +03:00
Alexey 642f166c98 Merge pull request #943 from telemt/carrier-methods
Configurable HTTP Methods for Bridge Carriers + Fix autonomous ME recovery from Direct fallback
2026-10-03 11:14:09 +03:00
Alexey d65efcc479 Fix autonomous ME recovery from Direct fallback
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-02 23:51:14 +03:00
Alexey ca5ebd1308 Merge pull request #942 from nomah4/docs/web-proxy-caddy
docs(web): add Caddy TLS termination example
2026-10-02 23:46:47 +03:00
Mikhail Gorbachev e0bf6fa3a0 docs(web): add Caddy TLS termination example
Add a Caddy section next to the NGINX and HAProxy examples in
WEB_PROXY.{en,ru,de}.md: minimal config, prefix-only cohosting with
handle (not handle_path), access-log redaction for the bridge capability
and bearer credentials, and notes on X-Forwarded-For, buffering,
timeouts, retries and WebSocket lanes across a Caddy reload.

Closes #914
2026-10-02 09:19:40 +07:00
Alexey fca6506449 Configurable HTTP Methods for Bridge Carriers
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-01 23:40:08 +03:00
128 changed files with 9689 additions and 1174 deletions
Generated
+1 -1
View File
@@ -2780,7 +2780,7 @@ dependencies = [
[[package]]
name = "telemt"
version = "3.5.11"
version = "3.5.14"
dependencies = [
"aes",
"anyhow",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "telemt"
version = "3.5.11"
version = "3.5.14"
edition = "2024"
rust-version = "1.88"
+63 -3
View File
@@ -2,7 +2,7 @@
[English](WEB_PROXY.en.md) | [Русский](WEB_PROXY.ru.md) | [Deutsch](WEB_PROXY.de.md)
Der WEB-Modus transportiert gewöhnliche MTProxy-Streams über begrenzte HTTPS- oder WebSocket-Carrier, die mit dem Proxy-Typ `WEB` von Telegram Desktop kompatibel sind. Telemt terminiert TLS nicht selbst: NGINX oder HAProxy verwaltet das öffentliche Zertifikat und leitet unverschlüsseltes HTTP/1.1 an einen privaten Telemt-Listener weiter.
Der WEB-Modus transportiert gewöhnliche MTProxy-Streams über begrenzte HTTPS- oder WebSocket-Carrier, die mit dem Proxy-Typ `WEB` von Telegram Desktop kompatibel sind. Telemt terminiert TLS nicht selbst: NGINX, HAProxy oder Caddy verwaltet das öffentliche Zertifikat und leitet unverschlüsseltes HTTP/1.1 an einen privaten Telemt-Listener weiter.
> [!IMPORTANT]
>
@@ -14,7 +14,7 @@ Der WEB-Modus transportiert gewöhnliche MTProxy-Streams über begrenzte HTTPS-
Telegram Desktop
| HTTPS oder WSS :443
v
NGINX oder HAProxy (TLS-Terminierung, kanonischer Host und eine X-Forwarded-For-Adresse)
NGINX, HAProxy oder Caddy (TLS-Terminierung, kanonischer Host und eine X-Forwarded-For-Adresse)
| unverschlüsseltes HTTP/1.1 in einem privaten Netz
v
Telemt-WEB-Listener
@@ -270,6 +270,66 @@ backend telemt_web
Im Frontend oder im Abschnitt `defaults` muss für das standardmäßige WebSocket-Liveness-Intervall auch `timeout client 65s` oder länger gesetzt sein. Für `https-lanes` muss das öffentliche HAProxy-ALPN `h2`, für WebSocket-Upgrade außerdem `http/1.1` enthalten. Bewahren Sie `Connection`, `Upgrade` und `Sec-WebSocket-*` unverändert; Pfad, Raw Query, Body sowie die Carrier-Header `Authorization`, `Content-Type`, `X-Up-Seq`, `X-Down-Cursor` und `X-Lane-ID` dürfen nicht umgeschrieben werden. Fügen Sie für Prefix-only-Cohosting `acl telemt_web_path path_beg /telegram/web/` hinzu und verlangen Sie in `use_backend` sowohl Host- als auch Pfad-ACL; entfernen Sie das Präfix nicht.
## TLS-Terminierung mit Caddy
```caddyfile
{
servers {
protocols h1 h2
}
}
proxy.example.com {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
```
Caddy bezieht und erneuert das öffentliche Zertifikat für die Site-Adresse automatisch. Den ursprünglichen `Host` sowie die Header `Connection`, `Upgrade` und `Sec-WebSocket-*` leitet Caddy unverändert weiter, WebSocket-Upgrade wird ohne zusätzliche Direktiven proxied; fügen Sie kein `header_up Host` hinzu. Öffentliches HTTP/2 (für `https-lanes`) und HTTP/1.1 (für WebSocket-Upgrade) sind standardmäßig aktiv, der private Abschnitt Caddy → Telemt bleibt HTTP/1.1. Die globale Option `protocols h1 h2` hält HTTP/3 aus dem WEB-Pfad heraus, weil der WEB-Vertrag nur HTTP/1.1 und HTTP/2 abdeckt. Sie gilt für alle Sites dieses Caddy-Servers.
`header_up X-Forwarded-For {remote_host}` überschreibt den Header mit genau einer Adresse. Ohne diese Zeile hängt Caddy bei Peers aus `trusted_proxies` an einen eingehenden `X-Forwarded-For` an, Telemt akzeptiert jedoch nur eine parsebare IP-Adresse. Steht Caddy selbst hinter einem L4-Balancer mit `listener_wrappers { proxy_protocol }`, ist `{remote_host}` die Client-Adresse aus dem PROXY-Header. `flush_interval -1` deaktiviert die Antwortpufferung für Long Polls und Downlinks. Request-Bodies streamt Caddy standardmäßig; aktivieren Sie kein `request_buffers`.
`reverse_proxy` hat in Caddy standardmäßig weder Read-/Write-Timeouts noch ein Body-Limit, daher passieren der 25-Sekunden-Long-Poll und das WebSocket-Liveness-Intervall unverändert. Sind serverweite `timeouts` oder `request_body max_size` gesetzt, müssen sie über 65 Sekunden bzw. mindestens bei `web.limits.max_body_bytes` liegen. Mit einem einzigen Upstream wiederholt Caddy fehlgeschlagene Anfragen nicht; fügen Sie weder `lb_retries` noch `lb_try_duration` hinzu. Bei einem Konfigurations-Reload schließt Caddy upgegradete Verbindungen standardmäßig sofort. `stream_close_delay` hält bestehende WebSocket-Lanes nach einem Reload für die angegebene Zeit offen.
Leiten Sie für Prefix-only-Cohosting mit `base_path = "telegram/web"` den Teilbaum über `handle` und nicht über `handle_path` weiter, denn `handle_path` entfernt das Präfix:
```caddyfile
proxy.example.com {
handle /telegram/web/* {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
handle {
# gewöhnliche Site
}
}
```
Das Routing von Caddy fügt keine Weiterleitung mit angehängtem Slash hinzu: `/telegram/web` ohne abschließenden Slash landet im Handler der gewöhnlichen Site. Stellen Sie sicher, dass auch dieser Handler nicht weiterleitet; `file_server` leitet etwa eine Verzeichnisanfrage auf die Variante mit Slash um.
Ohne `log`-Direktive schreibt Caddy kein Access-Log. Wird eines benötigt, schwärzen Sie Bridge-Capability und Bearer-Credentials:
```caddyfile
log {
format filter {
request>uri query {
replace bridge REDACTED
}
request>headers>Authorization delete
request>headers>Sec-Websocket-Protocol delete
}
}
```
Tragen Sie in `web_trusted_proxy_cidrs` die Adresse ein, von der aus Caddy Telemt erreicht, für die obige Konfiguration also `["127.0.0.1/32"]`.
## Lebenszyklus und Reload-Verhalten
| Konfiguration | Runtime-Verhalten |
@@ -431,7 +491,7 @@ Der vollständige Vertrag für Requests, Revisionen, Fehler und alle Benutzer-En
| `/telegram/web` leitet auf `/telegram/web/` um | Fügen Sie für den Pfad ohne Schrägstrich einen exakten Non-WEB-Handler hinzu. Nur der mit Schrägstrich abgeschlossene konfigurierte Teilbaum gehört zum WEB-Vertrag von Telemt. |
| Ein konkurrierender `https-lanes`-Downlink erreicht den Decoy mit `404` | Prüfen Sie, dass er mit `X-Down-Cursor: 0` beginnt, bewahren Sie `X-Lane-ID` und setzen Sie `lane_open_wait_secs` über den beobachteten Abstand zwischen Downlink und `OPEN`. Fortgeschrittene Cursor fehlender Lanes schlagen absichtlich fail-closed fehl. |
| Auto-Negotiation wechselt weiter, nachdem Daten bereits akzeptiert wurden | Das ist ungültig. Prüfen Sie das authentifizierte `X-Carrier-State`-Replay und das Carrier-Commit-Lifecycle-Ereignis; `committed` oder `healthy` ist terminal und erfordert eine neue Sitzung. |
| Long Polls werden nach einem festen Intervall getrennt | Setzen Sie Client-, Server-, Sende- und Lese-Timeouts von NGINX/HAProxy über `web.timeouts.long_poll_secs`. |
| Long Polls werden nach einem festen Intervall getrennt | Setzen Sie Client-, Server-, Sende- und Lese-Timeouts von NGINX/HAProxy/Caddy über `web.timeouts.long_poll_secs`. |
| WebSocket-Upgrade erreicht statt `101` den Decoy | Bewahren Sie HTTP/1.1 `Connection: Upgrade`, `Upgrade: websocket`, das einzelne exakte `Sec-WebSocket-Protocol` und den kanonischen bodylosen Request an der konfigurierten Basis plus `/api/v1/ws`. Prüfen Sie außerdem Carrier-/Session-Kompatibilität und die Prozess-Verbindungsreserve. |
| Ein `websocket-lanes`-Stream wurde geschlossen, Geschwister bleiben aber verbunden | Dies ist die beabsichtigte Fehlergrenze. Prüfen Sie die Message-/Frame-Zeilen dieser Lane in `/web-status`; fehlerhafte oder lane-fremde Frames, Write-Timeouts und Backend-Close schließen nur die betroffene Lane. |
| `/web-status` ist leer | Prüfen Sie, dass `[web.debug].enabled = true` gesetzt ist, wenden Sie die Konfiguration an, wählen Sie ein Fenster innerhalb von `max_window_secs` und erzeugen Sie nach der Policy-Änderung neuen WEB-Datenverkehr. |
+63 -3
View File
@@ -2,7 +2,7 @@
[English](WEB_PROXY.en.md) | [Русский](WEB_PROXY.ru.md) | [Deutsch](WEB_PROXY.de.md)
WEB mode carries ordinary MTProxy streams through bounded HTTPS or WebSocket carriers compatible with Telegram Desktop's `WEB` proxy type. Telemt does not terminate TLS: NGINX or HAProxy owns the public certificate and forwards plain HTTP/1.1 to a private Telemt listener.
WEB mode carries ordinary MTProxy streams through bounded HTTPS or WebSocket carriers compatible with Telegram Desktop's `WEB` proxy type. Telemt does not terminate TLS: NGINX, HAProxy, or Caddy owns the public certificate and forwards plain HTTP/1.1 to a private Telemt listener.
> [!IMPORTANT]
>
@@ -14,7 +14,7 @@ WEB mode carries ordinary MTProxy streams through bounded HTTPS or WebSocket car
Telegram Desktop
| HTTPS or WSS :443
v
NGINX or HAProxy (TLS termination, canonical Host and one X-Forwarded-For address)
NGINX, HAProxy, or Caddy (TLS termination, canonical Host and one X-Forwarded-For address)
| plain HTTP/1.1 on a private network
v
Telemt WEB listener
@@ -270,6 +270,66 @@ backend telemt_web
The frontend or `defaults` section must also set `timeout client 65s` or longer for the default WebSocket liveness interval. HAProxy's public ALPN must include `h2` for `https-lanes` and `http/1.1` for WebSocket Upgrade. Preserve `Connection`, `Upgrade`, and `Sec-WebSocket-*`; do not rewrite the path, raw query, body, or the `Authorization`, `Content-Type`, `X-Up-Seq`, `X-Down-Cursor`, and `X-Lane-ID` carrier headers. For prefix-only cohosting, add `acl telemt_web_path path_beg /telegram/web/` and require both the host and path ACLs on `use_backend`; do not remove the prefix.
## Caddy TLS termination
```caddyfile
{
servers {
protocols h1 h2
}
}
proxy.example.com {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
```
Caddy obtains and renews the public certificate for the site address automatically. It forwards the original `Host` and the `Connection`, `Upgrade`, and `Sec-WebSocket-*` headers unchanged and proxies WebSocket Upgrade without extra directives; do not add `header_up Host`. Public HTTP/2 (needed for `https-lanes`) and HTTP/1.1 (needed for WebSocket Upgrade) are both enabled by default, while the private Caddy-to-Telemt hop stays HTTP/1.1. The global `protocols h1 h2` option keeps HTTP/3 out of the WEB path, because the WEB contract only covers HTTP/1.1 and HTTP/2. It applies to every site of that Caddy server.
`header_up X-Forwarded-For {remote_host}` overwrites the header with one address. Without it, Caddy appends to an incoming `X-Forwarded-For` from peers listed in `trusted_proxies`, and Telemt accepts only one parseable IP address. If Caddy itself sits behind an L4 balancer with `listener_wrappers { proxy_protocol }`, `{remote_host}` is the client address taken from the PROXY header. `flush_interval -1` disables response buffering for long polls and downlinks. Request bodies are streamed by default; do not enable `request_buffers`.
Caddy's `reverse_proxy` has no read or write timeout and no request-body limit by default, so the 25-second long poll and the WebSocket liveness interval pass unchanged. If server-wide `timeouts` or `request_body max_size` are configured, keep them above 65 seconds and at least `web.limits.max_body_bytes` respectively. With a single upstream Caddy does not retry failed requests; do not add `lb_retries` or `lb_try_duration`. On a configuration reload Caddy closes upgraded connections immediately by default. `stream_close_delay` keeps established WebSocket lanes open across a reload for the given time.
For prefix-only cohosting with `base_path = "telegram/web"`, route the subtree with `handle`, not `handle_path`, because `handle_path` strips the prefix:
```caddyfile
proxy.example.com {
handle /telegram/web/* {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
handle {
# ordinary site
}
}
```
Caddy's routing does not add a slash-appending redirect: `/telegram/web` without the trailing slash falls through to the ordinary site handler. Make sure that handler does not redirect it either; for example, `file_server` redirects a directory request to its slash-terminated form.
Caddy writes no access log unless a `log` directive is present. If one is needed, redact the bridge capability and bearer credentials:
```caddyfile
log {
format filter {
request>uri query {
replace bridge REDACTED
}
request>headers>Authorization delete
request>headers>Sec-Websocket-Protocol delete
}
}
```
Set `web_trusted_proxy_cidrs` to the address Caddy uses to connect to Telemt, for example `["127.0.0.1/32"]` for the configuration above.
## Lifecycle and reload behavior
| Configuration | Runtime behavior |
@@ -431,7 +491,7 @@ See the complete [Control API contract](../Architecture/API/API.md) for request
| `/telegram/web` redirects to `/telegram/web/` | Add an exact non-WEB handler for the no-slash path. Only the slash-terminated configured subtree belongs to Telemt's WEB contract. |
| A racing `https-lanes` downlink reaches the decoy with `404` | Confirm it starts at `X-Down-Cursor: 0`, preserve `X-Lane-ID`, and set `lane_open_wait_secs` above the observed down-before-`OPEN` skew. Advanced cursors for missing lanes intentionally fail closed. |
| Auto-negotiation advances after traffic was already accepted | This is not valid behavior. Inspect the authenticated `X-Carrier-State` replay and the carrier commit lifecycle row; a committed or healthy response is terminal and requires a new session. |
| Long polls disconnect near a fixed interval | Raise NGINX/HAProxy client, server, send, and read timeouts above `web.timeouts.long_poll_secs`. |
| Long polls disconnect near a fixed interval | Raise NGINX/HAProxy/Caddy client, server, send, and read timeouts above `web.timeouts.long_poll_secs`. |
| WebSocket Upgrade reaches the decoy instead of returning `101` | Preserve HTTP/1.1 `Connection: Upgrade`, `Upgrade: websocket`, the single exact `Sec-WebSocket-Protocol`, and the canonical bodyless request at the configured base plus `/api/v1/ws`. Also check carrier/session compatibility and the process connection reserve. |
| One `websocket-lanes` stream closes while siblings stay connected | This is the intended failure boundary. Inspect that lane's message/frame rows in `/web-status`; malformed, cross-lane, write-timeout, and backend-close paths terminate only the affected lane. |
| `/web-status` is empty | Confirm `[web.debug].enabled = true`, apply the configuration, select a window within `max_window_secs`, and generate new WEB traffic after the policy change. |
+63 -3
View File
@@ -2,7 +2,7 @@
[English](WEB_PROXY.en.md) | [Русский](WEB_PROXY.ru.md) | [Deutsch](WEB_PROXY.de.md)
WEB-режим переносит обычные MTProxy-потоки через bounded HTTPS или WebSocket carriers, совместимые с типом прокси `WEB` в Telegram Desktop. Telemt не терминирует TLS: публичный сертификат обслуживает NGINX или HAProxy, который передаёт обычный HTTP/1.1 на приватный listener Telemt.
WEB-режим переносит обычные MTProxy-потоки через bounded HTTPS или WebSocket carriers, совместимые с типом прокси `WEB` в Telegram Desktop. Telemt не терминирует TLS: публичный сертификат обслуживает NGINX, HAProxy или Caddy, который передаёт обычный HTTP/1.1 на приватный listener Telemt.
> [!IMPORTANT]
>
@@ -14,7 +14,7 @@ WEB-режим переносит обычные MTProxy-потоки через
Telegram Desktop
| HTTPS или WSS :443
v
NGINX или HAProxy (TLS termination, канонический Host и один адрес X-Forwarded-For)
NGINX, HAProxy или Caddy (TLS termination, канонический Host и один адрес X-Forwarded-For)
| обычный HTTP/1.1 в приватной сети
v
WEB-listener Telemt
@@ -270,6 +270,66 @@ backend telemt_web
Во frontend или секции `defaults` также задайте `timeout client 65s` или больше для default WebSocket liveness interval. Для `https-lanes` публичный ALPN HAProxy должен содержать `h2`, а для WebSocket Upgrade — `http/1.1`. Сохраняйте `Connection`, `Upgrade` и `Sec-WebSocket-*`; не переписывайте path, raw query, body и carrier headers `Authorization`, `Content-Type`, `X-Up-Seq`, `X-Down-Cursor`, `X-Lane-ID`. Для prefix-only cohosting добавьте `acl telemt_web_path path_beg /telegram/web/` и потребуйте одновременно host- и path-ACL в `use_backend`; не удаляйте prefix.
## Терминация TLS на Caddy
```caddyfile
{
servers {
protocols h1 h2
}
}
proxy.example.com {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
```
Caddy сам получает и продлевает публичный сертификат для адреса сайта. Он передаёт исходный `Host` и заголовки `Connection`, `Upgrade` и `Sec-WebSocket-*` без изменений и проксирует WebSocket Upgrade без дополнительных директив; не добавляйте `header_up Host`. Публичные HTTP/2 (нужен для `https-lanes`) и HTTP/1.1 (нужен для WebSocket Upgrade) включены по умолчанию, а приватный участок Caddy → Telemt остаётся HTTP/1.1. Глобальная опция `protocols h1 h2` убирает HTTP/3 с пути WEB: контракт WEB описывает только HTTP/1.1 и HTTP/2. Она действует на все сайты этого сервера Caddy.
`header_up X-Forwarded-For {remote_host}` перезаписывает заголовок одним адресом. Без неё Caddy дописывает адрес к входящему `X-Forwarded-For` от пиров из `trusted_proxies`, а Telemt принимает только один разбираемый IP-адрес. Если сам Caddy стоит за L4-балансировщиком с `listener_wrappers { proxy_protocol }`, `{remote_host}` — адрес клиента из заголовка PROXY. `flush_interval -1` отключает буферизацию ответа для long polls и downlink. Тела запросов Caddy по умолчанию передаёт потоком; не включайте `request_buffers`.
У `reverse_proxy` в Caddy по умолчанию нет read/write timeout и лимита тела запроса, поэтому 25-секундный long poll и WebSocket liveness interval проходят без изменений. Если заданы серверные `timeouts` или `request_body max_size`, держите их выше 65 секунд и не меньше `web.limits.max_body_bytes` соответственно. С одним upstream Caddy не повторяет неудачные запросы; не добавляйте `lb_retries` и `lb_try_duration`. При reload конфигурации Caddy по умолчанию сразу закрывает upgraded-соединения. `stream_close_delay` держит установленные WebSocket lanes открытыми на заданное время после reload.
Для prefix-only cohosting с `base_path = "telegram/web"` направляйте поддерево через `handle`, а не `handle_path`: `handle_path` срезает prefix.
```caddyfile
proxy.example.com {
handle /telegram/web/* {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
handle {
# обычный сайт
}
}
```
Маршрутизация Caddy не добавляет редирект со слэшем: `/telegram/web` без завершающего слэша уходит в обработчик обычного сайта. Убедитесь, что и этот обработчик не редиректит его; например, `file_server` перенаправляет запрос каталога на вариант со слэшем.
Caddy не пишет access log без директивы `log`. Если журнал нужен, скройте bridge capability и bearer credentials:
```caddyfile
log {
format filter {
request>uri query {
replace bridge REDACTED
}
request>headers>Authorization delete
request>headers>Sec-Websocket-Protocol delete
}
}
```
В `web_trusted_proxy_cidrs` укажите адрес, с которого Caddy подключается к Telemt, например `["127.0.0.1/32"]` для конфигурации выше.
## Lifecycle и reload
| Конфигурация | Поведение runtime |
@@ -431,7 +491,7 @@ curl -sS -X POST http://127.0.0.1:9091/v1/users/web-user/rotate-secret \
| `/telegram/web` перенаправляет на `/telegram/web/` | Добавьте точный non-WEB handler для path без слеша. В WEB-контракт Telemt входит только настроенное поддерево с завершающим слешем. |
| Downlink `https-lanes`, участвующий в гонке, попадает в decoy с `404` | Убедитесь, что он начинается с `X-Down-Cursor: 0`, сохраняйте `X-Lane-ID` и задайте `lane_open_wait_secs` выше наблюдаемого разрыва down-before-`OPEN`. Продвинутый cursor отсутствующей lane намеренно закрывается fail-closed. |
| Auto-negotiation переходит дальше после уже принятого трафика | Такое поведение некорректно. Проверьте аутентифицированный replay `X-Carrier-State` и lifecycle row commit carrier; ответ `committed` или `healthy` terminal и требует новой сессии. |
| Long polls разрываются через фиксированный интервал | Поднимите client, server, send и read timeouts NGINX/HAProxy выше `web.timeouts.long_poll_secs`. |
| Long polls разрываются через фиксированный интервал | Поднимите client, server, send и read timeouts NGINX/HAProxy/Caddy выше `web.timeouts.long_poll_secs`. |
| WebSocket Upgrade попадает в decoy вместо `101` | Сохраните HTTP/1.1 `Connection: Upgrade`, `Upgrade: websocket`, единственный точный `Sec-WebSocket-Protocol` и канонический bodyless request по настроенному base плюс `/api/v1/ws`. Также проверьте соответствие carrier/session и process connection reserve. |
| Один stream `websocket-lanes` закрылся, а siblings остались подключены | Это штатная failure boundary. Проверьте message/frame rows этой lane в `/web-status`; malformed, cross-lane, write-timeout и backend-close закрывают только затронутую lane. |
| `/web-status` пуст | Убедитесь, что `[web.debug].enabled = true`, примените конфигурацию, выберите окно в пределах `max_window_secs` и создайте новый WEB-трафик после изменения policy. |
+13 -3
View File
@@ -158,7 +158,7 @@ async fn prepare_patch_to_path(
) -> Result<PreparedConfigPatch, ApiFailure> {
// 1. optimistic concurrency
let loaded = load_config_snapshot(config_path, false).await?;
let current = compute_snapshot_revision(&loaded);
let current = compute_snapshot_revision(&loaded.source_contents);
if expected_revision.is_some_and(|expected| expected != current) {
return Err(ApiFailure::new(
hyper::StatusCode::CONFLICT,
@@ -208,6 +208,8 @@ async fn prepare_patch_to_path(
let mut merged = Toml::try_from(&old_cfg)
.map_err(|e| ApiFailure::internal(format!("failed to serialize config: {}", e)))?;
deep_merge(&mut merged, &patch_toml);
ProxyConfig::validate_decoy_source_keys(&merged)
.map_err(|error| ApiFailure::bad_request(format!("invalid patched config: {error}")))?;
let requested_cfg: ProxyConfig = merged
.clone()
@@ -263,7 +265,7 @@ async fn prepare_patch_to_path(
}
// 4. classify the validated, normalized candidate.
let revision = compute_snapshot_revision(&candidate);
let revision = compute_snapshot_revision(&candidate.source_contents);
let new_cfg = candidate.config;
let class = classify_config_changes(&old_cfg, &new_cfg);
let deferred_process_fields =
@@ -306,7 +308,7 @@ fn reload_submit_failure(error: ReloadSubmitError) -> ApiFailure {
/// Returns only the editable config sections and current revision.
pub(super) async fn read_managed_config(config_path: &Path) -> Result<(Toml, String), ApiFailure> {
let loaded = load_config_snapshot(config_path, false).await?;
let revision = compute_snapshot_revision(&loaded);
let revision = compute_snapshot_revision(&loaded.source_contents);
let parsed = Toml::try_from(&loaded.config)
.map_err(|error| ApiFailure::internal(format!("failed to serialize config: {error}")))?;
@@ -453,3 +455,11 @@ mod base_path_tests;
#[cfg(test)]
#[path = "config_edit/tests.rs"]
mod tests;
#[cfg(test)]
#[path = "config_edit/conveyor_tests.rs"]
mod conveyor_tests;
#[cfg(test)]
#[path = "config_edit/decoy_dns_tests.rs"]
mod decoy_dns_tests;
+27
View File
@@ -0,0 +1,27 @@
use super::*;
#[tokio::test]
async fn conveyor_api_is_boolean_hot_and_rejects_invalid_patch_atomically() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
std::fs::write(&path, "[web]\nconveyor = true\n").unwrap();
for enabled in [false, true] {
let active = ProxyConfig::load(&path).unwrap();
let patch = serde_json::json!({"web": {"conveyor": enabled}});
let mut response = apply_patch_to_path(&path, &patch, None).await.unwrap();
let desired = ProxyConfig::load(&path).unwrap();
let resolved = reconcile_runtime_effect(&mut response, &active, &desired).unwrap();
assert_eq!(resolved.effective.web.conveyor, enabled);
assert!(response.runtime_reload_required);
assert!(!response.process_restart_required);
let (managed, _) = read_managed_config(&path).await.unwrap();
assert_eq!(managed["web"]["conveyor"].as_bool(), Some(enabled));
let previous = std::fs::read(&path).unwrap();
let invalid = serde_json::json!({"web": {"conveyor": "false"}});
let error = apply_patch_to_path(&path, &invalid, None)
.await
.unwrap_err();
assert_eq!(error.status, hyper::StatusCode::BAD_REQUEST);
assert_eq!(std::fs::read(&path).unwrap(), previous);
}
}
+105
View File
@@ -0,0 +1,105 @@
use super::*;
const SOURCE: &str = r#"
[access.users]
alice = "000102030405060708090a0b0c0d0e0f"
[[server.listeners]]
ip = "127.0.0.1"
port = 18080
transport = "web"
web_trusted_proxy_cidrs = ["127.0.0.1/32"]
[web]
enabled = true
[[web.vhosts]]
host = "proxy.example.com"
public_addr = "203.0.113.10:443"
[web.vhosts.decoy]
mode = "http_upstream"
upstream = "http://localhost:18081"
resolve = "startup"
[[web.vhosts.profiles]]
user = "alice"
secret_mode = "dd"
"#;
#[tokio::test]
async fn decoy_dns_api_get_and_mutation_base_need_no_prepared_runtime() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
std::fs::write(
&path,
SOURCE.replace("localhost", "unresolvable.example.invalid"),
)
.unwrap();
let (managed, revision) = read_managed_config(&path).await.unwrap();
assert_eq!(
managed["web"]["vhosts"][0]["decoy"]["resolve"].as_str(),
Some("startup")
);
assert!(managed["web"].get("decoy_dns").is_none());
let (base, base_revision) = crate::api::config_store::load_config_for_mutation(&path, None)
.await
.unwrap();
assert_eq!(revision, base_revision);
assert!(base.web.runtime.is_none());
assert!(base.web.decoy_dns.origins.is_empty());
}
#[tokio::test]
async fn decoy_dns_api_patch_without_reload_prepares_before_disk_write() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
std::fs::write(&path, SOURCE).unwrap();
// This current-thread runtime exercises the real Tokio resolver without nested block_on.
let active = ProxyConfig::load_prepared(path.clone())
.await
.unwrap()
.config;
let before = std::fs::read(&path).unwrap();
let mut vhosts = serde_json::to_value(&active.web.vhosts).unwrap();
vhosts[0]["decoy"]["upstream"] = Json::String("http://localhost:18080".to_string());
let error = apply_patch_to_path(&path, &serde_json::json!({"web": {"vhosts": vhosts}}), None)
.await
.unwrap_err();
assert!(error.message.contains("overlaps WEB listener"));
assert_eq!(std::fs::read(&path).unwrap(), before);
assert!(active.web.runtime.is_some());
let prepared = prepare_patch_to_path(
&path,
&serde_json::json!({"web": {"conveyor": false}}),
None,
)
.await
.unwrap();
assert_eq!(prepared.desired_config.web.decoy_dns.origins.len(), 1);
assert!(prepared.desired_config.web.runtime.is_some());
assert_eq!(std::fs::read(&path).unwrap(), before);
apply_patch_to_path(
&path,
&serde_json::json!({"web": {"conveyor": false}}),
None,
)
.await
.unwrap();
let reloaded = ProxyConfig::load_prepared(path).await.unwrap();
assert!(!reloaded.config.web.conveyor);
}
#[tokio::test]
async fn decoy_dns_api_rejects_resolve_on_static_mode_before_serde_drops_it() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
std::fs::write(&path, SOURCE).unwrap();
let base = ProxyConfig::parse_source(&path).unwrap();
let mut vhosts = serde_json::to_value(&base.config.web.vhosts).unwrap();
vhosts[0]["decoy"] = serde_json::json!({
"mode": "static_directory", "directory": directory.path(), "resolve": "startup"
});
let before = std::fs::read(&path).unwrap();
let error = apply_patch_to_path(&path, &serde_json::json!({"web": {"vhosts": vhosts}}), None)
.await
.unwrap_err();
assert!(error.message.contains("only valid for mode=http_upstream"));
assert_eq!(std::fs::read(&path).unwrap(), before);
}
+51 -1
View File
@@ -1,5 +1,54 @@
use super::*;
#[tokio::test]
async fn carrier_method_api_defaults_and_patches_are_hot() {
let (path, _directory) = temp_config("[web]\nenabled = false\n");
let (value, _) = read_managed_config(&path).await.unwrap();
assert_eq!(value["web"]["carrier_method"].as_str(), Some("post"));
for token in ["put", "post"] {
let active = ProxyConfig::load(&path).unwrap();
let patch = serde_json::json!({"web": {"carrier_method": token}});
let mut response = apply_patch_to_path(&path, &patch, None).await.unwrap();
let desired = ProxyConfig::load(&path).unwrap();
reconcile_runtime_effect(&mut response, &active, &desired).unwrap();
assert!(!response.restart_required);
assert!(response.runtime_reload_required);
assert!(!response.process_restart_required);
assert!(response.deferred_process_fields.is_empty());
let (value, revision) = read_managed_config(&path).await.unwrap();
assert_eq!(value["web"]["carrier_method"].as_str(), Some(token));
assert_eq!(revision, response.revision);
let written = tokio::fs::read_to_string(&path).await.unwrap();
assert!(written.contains(&format!("carrier_method = \"{token}\"")));
}
}
#[tokio::test]
async fn carrier_method_api_rejects_invalid_values_without_writing() {
let (path, _directory) = temp_config("[web]\ncarrier_method = \"put\"\n");
let original = tokio::fs::read(&path).await.unwrap();
let revision = crate::api::config_store::current_revision(&path)
.await
.unwrap();
for value in [
serde_json::json!("PUT"),
serde_json::json!("patch"),
serde_json::json!(true),
serde_json::json!(42),
] {
let patch = serde_json::json!({"web": {"carrier_method": value}});
let error = apply_patch_to_path(&path, &patch, None).await.unwrap_err();
assert_eq!(error.status, hyper::StatusCode::BAD_REQUEST);
assert_eq!(tokio::fs::read(&path).await.unwrap(), original);
assert_eq!(
crate::api::config_store::current_revision(&path)
.await
.unwrap(),
revision
);
}
}
#[test]
fn json_object_converts_to_toml_table() {
let j: Json = serde_json::json!({"censorship": {"tls_domain": "a.com"}, "default_dc": 2});
@@ -127,7 +176,8 @@ async fn patch_web_debug_is_hot_and_limits_are_process_deferred() {
assert!(desired.web.debug.sideband);
let limits_patch: Json = serde_json::json!({
"web": {"limits": {"max_http_connections": 2049}}
// Extra HTTP heads must fit alongside the default-on conveyor metadata reservation.
"web": {"limits": {"max_http_connections": 2049, "memory_envelope_bytes": 1610612736u64}}
});
let limits = apply_patch_to_path(&path, &limits_patch, None)
.await
+27 -15
View File
@@ -4,7 +4,7 @@ use std::path::{Path, PathBuf};
use hyper::header::IF_MATCH;
use sha2::{Digest, Sha256};
use crate::config::{ConfigSourceGraph, LoadedConfig, ProxyConfig};
use crate::config::{ConfigSourceGraph, LoadedConfig, ParsedConfigSource, ProxyConfig};
use super::model::ApiFailure;
@@ -63,7 +63,7 @@ pub(super) async fn load_config_for_mutation(
expected_revision: Option<&str>,
) -> Result<(ProxyConfig, String), ApiFailure> {
let loaded = load_config_snapshot(config_path, false).await?;
let revision = compute_snapshot_revision(&loaded);
let revision = compute_snapshot_revision(&loaded.source_contents);
if expected_revision.is_some_and(|expected| expected != revision) {
return Err(ApiFailure::new(
hyper::StatusCode::CONFLICT,
@@ -98,9 +98,10 @@ pub(super) fn compute_revision(content: &str) -> String {
hex::encode(hasher.finalize())
}
pub(super) fn compute_snapshot_revision(loaded: &LoadedConfig) -> String {
/// Hashes source documents only; prepared DNS answers never change a revision.
pub(super) fn compute_snapshot_revision(source_contents: &BTreeMap<PathBuf, String>) -> String {
compute_source_revision(&ConfigSourceGraph {
source_contents: loaded.source_contents.clone(),
source_contents: source_contents.clone(),
rendered: String::new(),
})
}
@@ -118,12 +119,13 @@ pub(super) fn compute_source_revision(graph: &ConfigSourceGraph) -> String {
hex::encode(hasher.finalize())
}
/// Reads a normalized source snapshot without resolving decoys or loading static assets.
pub(super) async fn load_config_snapshot(
config_path: &Path,
invalid_is_bad_request: bool,
) -> Result<LoadedConfig, ApiFailure> {
) -> Result<ParsedConfigSource, ApiFailure> {
let config_path = config_path.to_path_buf();
tokio::task::spawn_blocking(move || ProxyConfig::load_with_metadata(config_path))
tokio::task::spawn_blocking(move || ProxyConfig::parse_source(config_path))
.await
.map_err(|error| ApiFailure::internal(format!("failed to join config loader: {error}")))?
.map_err(|error| {
@@ -135,8 +137,9 @@ pub(super) async fn load_config_snapshot(
})
}
/// Selects the single writable source document owning all requested sections.
pub(super) fn resolve_single_source_owner(
loaded: &LoadedConfig,
loaded: &ParsedConfigSource,
config_path: &Path,
targets: &[&str],
) -> Result<PathBuf, ApiFailure> {
@@ -227,6 +230,7 @@ fn has_include_inside_table(content: &str) -> bool {
false
}
/// Prepares the entire mutation candidate before any source file may be replaced.
pub(super) async fn load_candidate_snapshot(
config_path: &Path,
base_sources: &BTreeMap<PathBuf, String>,
@@ -236,12 +240,16 @@ pub(super) async fn load_candidate_snapshot(
let config_path = config_path.to_path_buf();
let mut overrides = base_sources.clone();
overrides.insert(owner_path, owner_contents);
tokio::task::spawn_blocking(move || {
ProxyConfig::load_with_source_overrides(config_path, &overrides)
let parsed = tokio::task::spawn_blocking(move || {
ProxyConfig::parse_source_with_overrides(config_path, &overrides)
})
.await
.map_err(|error| ApiFailure::internal(format!("failed to join config loader: {error}")))?
.map_err(|error| ApiFailure::bad_request(format!("invalid patched config: {error}")))
.map_err(|error| ApiFailure::bad_request(format!("invalid patched config: {error}")))?;
parsed
.prepare()
.await
.map_err(|error| ApiFailure::bad_request(format!("invalid patched config: {error}")))
}
fn normalize_source_path(path: &Path) -> PathBuf {
@@ -265,19 +273,23 @@ fn normalize_source_path(path: &Path) -> PathBuf {
normalized
}
/// Reads desired user and quota policy without acquiring runtime decoy state.
pub(super) async fn load_config_from_disk(config_path: &Path) -> Result<ProxyConfig, ApiFailure> {
let config_path = config_path.to_path_buf();
tokio::task::spawn_blocking(move || ProxyConfig::load(config_path))
load_config_snapshot(config_path, false)
.await
.map_err(|e| ApiFailure::internal(format!("failed to join config loader: {}", e)))?
.map_err(|e| ApiFailure::internal(format!("failed to load config: {}", e)))
.map(|source| source.config)
}
/// Acquires fresh decoy evidence for an explicit API reload, even at the same revision.
pub(super) async fn load_config_for_reload(
config_path: &Path,
) -> Result<(ProxyConfig, String), ApiFailure> {
let loaded = load_config_snapshot(config_path, true).await?;
let revision = compute_snapshot_revision(&loaded);
let revision = compute_snapshot_revision(&loaded.source_contents);
let loaded = loaded
.prepare()
.await
.map_err(|error| ApiFailure::bad_request(format!("invalid runtime config: {error}")))?;
Ok((loaded.config, revision))
}
+2 -2
View File
@@ -113,7 +113,7 @@ pub(in crate::api) async fn save_access_sections_to_disk_if_revision(
expected_revision: Option<&str>,
) -> Result<String, ApiFailure> {
let loaded = load_config_snapshot(config_path, false).await?;
let loaded_revision = compute_snapshot_revision(&loaded);
let loaded_revision = compute_snapshot_revision(&loaded.source_contents);
if expected_revision.is_some_and(|expected| expected != loaded_revision) {
return Err(revision_conflict());
}
@@ -159,7 +159,7 @@ pub(in crate::api) async fn save_access_sections_to_disk_if_revision(
owner_contents.clone(),
)
.await?;
let _candidate_revision = compute_snapshot_revision(&candidate);
let _candidate_revision = compute_snapshot_revision(&candidate.source_contents);
let revision = write_atomic_if_unchanged(
config_path.to_path_buf(),
loaded_revision,
+2 -1
View File
@@ -59,7 +59,7 @@ const DEFAULT_ME_POOL_DRAIN_SOFT_EVICT_BUDGET_PER_CORE: u16 = 16;
const DEFAULT_ME_POOL_DRAIN_SOFT_EVICT_COOLDOWN_MS: u64 = 1000;
const DEFAULT_USER_MAX_UNIQUE_IPS_WINDOW_SECS: u64 = 30;
const DEFAULT_ACCEPT_PERMIT_TIMEOUT_MS: u64 = 250;
const DEFAULT_CONNTRACK_CONTROL_ENABLED: bool = true;
const DEFAULT_CONNTRACK_CONTROL_ENABLED: bool = false;
const DEFAULT_CONNTRACK_PRESSURE_HIGH_WATERMARK_PCT: u8 = 85;
const DEFAULT_CONNTRACK_PRESSURE_LOW_WATERMARK_PCT: u8 = 70;
const DEFAULT_CONNTRACK_DELETE_BUDGET_PER_SEC: u64 = 4096;
@@ -244,6 +244,7 @@ pub(crate) fn default_accept_permit_timeout_ms() -> u64 {
DEFAULT_ACCEPT_PERMIT_TIMEOUT_MS
}
/// Keeps privileged conntrack control disabled unless explicitly requested.
pub(crate) fn default_conntrack_control_enabled() -> bool {
DEFAULT_CONNTRACK_CONTROL_ENABLED
}
+9
View File
@@ -17,6 +17,7 @@
//! | `network` | `dns_overrides` | Applied immediately |
//! | `access` | All user/quota fields | Effective immediately |
//! | `web` | Carrier, timing, and debug policy | Applied to newly issued sessions |
//! | `web` | `carrier_method` | Applied to newly rendered bridge pages |
//! Fields that require re-binding sockets (`server.listeners`, legacy
//! `server.port`, `censorship.*`, `network.*`, `use_middle_proxy`) are **not**
//! applied; a warning is emitted. SYN limiter rules are process-owned and are
@@ -67,3 +68,11 @@ use watcher::{ReloadState, reload_config};
mod base_path_tests;
#[cfg(test)]
mod tests;
// Carrier method reloads preserve page-owned requests and process-owned limits.
#[cfg(test)]
#[path = "hot_reload/carrier_method_tests.rs"]
mod carrier_method_tests;
#[cfg(test)]
#[path = "hot_reload/conveyor_tests.rs"]
mod conveyor_tests;
@@ -0,0 +1,50 @@
use super::*;
use crate::config::WebCarrierMethod;
#[test]
fn carrier_method_reload_is_hot_and_preserves_process_limits() {
let mut active = ProxyConfig::default();
for method in [WebCarrierMethod::Put, WebCarrierMethod::Post] {
let mut desired = active.clone();
desired.web.carrier_method = method;
assert_eq!(classify_config_changes(&active, &desired).changed, ["web"]);
assert!(!classify_config_changes(&active, &desired).restart_required);
desired.web.limits.max_http_connections += 1;
let applied = overlay_hot_fields(&active, &desired);
assert_eq!(applied.web.carrier_method, method);
assert_eq!(
applied.web.limits.max_http_connections,
active.web.limits.max_http_connections
);
active = applied;
}
}
#[test]
fn carrier_method_reload_publishes_both_directions_and_keeps_last_good_value() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
let source = |method| format!("[web]\ncarrier_method = \"{method}\"\n");
std::fs::write(&path, source("post")).unwrap();
let initial = Arc::new(ProxyConfig::load(&path).unwrap());
let initial_hash = ProxyConfig::load_with_metadata(&path)
.unwrap()
.rendered_hash;
let (config_tx, _config_rx) = watch::channel(Arc::clone(&initial));
let (log_tx, _log_rx) = watch::channel(initial.general.log_level.clone());
let mut reload_state = ReloadState::new(Some(initial_hash));
for (token, method) in [
("put", WebCarrierMethod::Put),
("post", WebCarrierMethod::Post),
] {
std::fs::write(&path, source(token)).unwrap();
reload_config(&path, &config_tx, &log_tx, None, None, &mut reload_state).unwrap();
let applied = config_tx.borrow().clone();
assert_eq!(applied.web.carrier_method, method);
std::fs::write(&path, source("PATCH")).unwrap();
reload_config(&path, &config_tx, &log_tx, None, None, &mut reload_state);
let unchanged = config_tx.borrow().clone();
assert!(Arc::ptr_eq(&unchanged, &applied));
}
}
+20
View File
@@ -0,0 +1,20 @@
use super::*;
#[test]
fn conveyor_reload_is_hot_in_both_directions_and_preserves_process_limits() {
let mut active = ProxyConfig::default();
for enabled in [false, true] {
let mut desired = active.clone();
desired.web.conveyor = enabled;
assert_eq!(classify_config_changes(&active, &desired).changed, ["web"]);
assert!(!classify_config_changes(&active, &desired).restart_required);
desired.web.limits.max_body_readers += 1;
let applied = overlay_hot_fields(&active, &desired);
assert_eq!(applied.web.conveyor, enabled);
assert_eq!(
applied.web.limits.max_body_readers,
active.web.limits.max_body_readers
);
active = applied;
}
}
+43 -42
View File
@@ -1,5 +1,8 @@
use super::*;
// Source polling and explicit reloads have distinct DNS preparation lifecycles.
mod preparation;
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub(super) struct WatchManifest {
files: BTreeSet<PathBuf>,
@@ -132,9 +135,21 @@ fn apply_watch_manifest<W1: Watcher, W2: Watcher>(
}
}
/// Load config, validate, diff against current, and broadcast if changed.
/// Builds the effective WEB snapshot without changing live state.
fn prepare_effective_config(
old_cfg: &ProxyConfig,
new_cfg: &ProxyConfig,
) -> crate::error::Result<ProxyConfig> {
new_cfg.validate()?;
let mut applied_cfg = overlay_hot_fields(old_cfg, new_cfg);
applied_cfg.validate_effective_web()?;
applied_cfg.rebuild_runtime_web()?;
Ok(applied_cfg)
}
/// Diffs and publishes a fully prepared configuration on the watcher owner task.
fn reload_config_with_resolver(
config_path: &PathBuf,
prepared: (LoadedConfig, ProxyConfig),
config_tx: &watch::Sender<Arc<ProxyConfig>>,
log_tx: &watch::Sender<LogLevel>,
detected_ip_v4: Option<IpAddr>,
@@ -142,13 +157,7 @@ fn reload_config_with_resolver(
reload_state: &mut ReloadState,
dns_resolver: Option<&crate::network::dns_overrides::GenerationDnsResolver>,
) -> Option<WatchManifest> {
let loaded = match ProxyConfig::load_with_metadata(config_path) {
Ok(loaded) => loaded,
Err(e) => {
error!("config reload: failed to parse {:?}: {}", config_path, e);
return None;
}
};
let (loaded, applied_cfg) = prepared;
let LoadedConfig {
config: new_cfg,
source_files,
@@ -157,34 +166,12 @@ fn reload_config_with_resolver(
} = loaded;
let next_manifest = WatchManifest::from_source_files(&source_files);
if let Err(e) = new_cfg.validate() {
error!(
"config reload: validation failed: {}; keeping old config",
e
);
return Some(next_manifest);
}
if reload_state.is_applied(rendered_hash) {
return Some(next_manifest);
}
let old_cfg = config_tx.borrow().clone();
let mut applied_cfg = overlay_hot_fields(&old_cfg, &new_cfg);
if let Err(error) = applied_cfg
.validate_effective_web()
.and_then(|_| applied_cfg.rebuild_runtime_web())
{
error!(
"config reload: effective WEB validation failed: {}; keeping old config",
error
);
return Some(next_manifest);
}
let old_hot = HotFields::from_config(&old_cfg);
let applied_hot = HotFields::from_config(&applied_cfg);
let non_hot_changed = !config_equal(&applied_cfg, &new_cfg);
let hot_changed = !config_equal(&old_cfg, &applied_cfg);
let hot_changed =
!config_equal(&old_cfg, &applied_cfg) || !old_cfg.web_decoy_endpoints_equal(&applied_cfg);
if non_hot_changed {
warn_non_hot_changes(&old_cfg, &new_cfg, non_hot_changed);
@@ -228,8 +215,13 @@ pub(super) fn reload_config(
detected_ip_v6: Option<IpAddr>,
reload_state: &mut ReloadState,
) -> Option<WatchManifest> {
let loaded = ProxyConfig::load_with_metadata(config_path).ok()?;
if reload_state.is_applied(loaded.rendered_hash) {
return Some(WatchManifest::from_source_files(&loaded.source_files));
}
let applied = prepare_effective_config(&config_tx.borrow(), &loaded.config).ok()?;
reload_config_with_resolver(
config_path,
(loaded, applied),
config_tx,
log_tx,
detected_ip_v4,
@@ -282,7 +274,7 @@ pub fn spawn_config_watcher(
}
}
}
let initial_loaded = ProxyConfig::load_with_metadata(&config_path).ok();
let initial_loaded = preparation::read_source(&config_path).await;
let initial_manifest = initial_loaded
.as_ref()
.map(|loaded| WatchManifest::from_source_files(&loaded.source_files))
@@ -291,7 +283,7 @@ pub fn spawn_config_watcher(
});
let initial_matches_disk = initial_loaded
.as_ref()
.is_some_and(|loaded| config_equal(config_tx.borrow().as_ref(), &loaded.config));
.is_some_and(|source| preparation::source_matches_active(&config_tx.borrow(), source));
let initial_snapshot_hash = initial_loaded
.as_ref()
.filter(|_| initial_matches_disk)
@@ -384,12 +376,14 @@ pub fn spawn_config_watcher(
};
loop {
let mut force = false;
#[cfg(unix)]
tokio::select! {
msg = notify_rx.recv() => {
if msg.is_none() { break; }
}
_ = sighup.recv() => {
force = true;
info!("SIGHUP received — reloading {:?}", config_path);
}
_ = cancellation.cancelled() => break,
@@ -406,27 +400,31 @@ pub fn spawn_config_watcher(
tokio::time::sleep(HOT_RELOAD_DEBOUNCE).await;
while notify_rx.try_recv().is_ok() {}
let mut next_manifest = reload_config_with_resolver(
let mut next_manifest = preparation::reload(
&config_path,
&config_tx,
&log_tx,
detected_ip_v4,
detected_ip_v6,
&mut reload_state,
dns_resolver.as_deref(),
);
dns_resolver.clone(),
force,
)
.await;
if next_manifest.is_none() {
tokio::time::sleep(HOT_RELOAD_DEBOUNCE).await;
while notify_rx.try_recv().is_ok() {}
next_manifest = reload_config_with_resolver(
next_manifest = preparation::reload(
&config_path,
&config_tx,
&log_tx,
detected_ip_v4,
detected_ip_v6,
&mut reload_state,
dns_resolver.as_deref(),
);
dns_resolver.clone(),
force,
)
.await;
}
if let Some(next_manifest) = next_manifest {
@@ -460,3 +458,6 @@ mod path_tests {
assert_eq!(normalize_watch_path(&linked), linked);
}
}
#[cfg(test)]
mod decoy_dns_tests;
@@ -0,0 +1,133 @@
use super::*;
const SOURCE: &str = r#"
[access.users]
alice = "000102030405060708090a0b0c0d0e0f"
[[server.listeners]]
ip = "127.0.0.1"
port = 18080
transport = "web"
web_trusted_proxy_cidrs = ["127.0.0.1/32"]
[web]
enabled = true
[[web.vhosts]]
host = "proxy.example.com"
public_addr = "203.0.113.10:443"
[web.vhosts.decoy]
mode = "http_upstream"
upstream = "http://localhost:18081"
resolve = "startup"
[[web.vhosts.profiles]]
user = "alice"
secret_mode = "dd"
"#;
#[tokio::test]
async fn decoy_dns_watcher_publishes_dns_only_changes_once() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(&path, SOURCE).unwrap();
let mut loaded = ProxyConfig::load_prepared(path.clone()).await.unwrap();
let (config_tx, mut config_rx) = watch::channel(Arc::new(loaded.config.clone()));
let (log_tx, _) = watch::channel(loaded.config.general.log_level.clone());
let old = config_rx.borrow().clone();
let mut state = ReloadState::new(Some(loaded.rendered_hash));
Arc::make_mut(&mut loaded.config.web.decoy_dns)
.origins
.insert(
("localhost".to_string(), 18081),
vec!["127.0.0.2:18081".parse().unwrap()],
);
loaded.config.rebuild_runtime_web().unwrap();
let applied = prepare_effective_config(&old, &loaded.config).unwrap();
reload_config_with_resolver(
(loaded.clone(), applied),
&config_tx,
&log_tx,
None,
None,
&mut state,
None,
)
.unwrap();
assert!(config_rx.has_changed().unwrap());
let published = config_rx.borrow_and_update().clone();
assert!(!old.web_decoy_endpoints_equal(&published));
let applied = prepare_effective_config(&published, &loaded.config).unwrap();
reload_config_with_resolver(
(loaded, applied),
&config_tx,
&log_tx,
None,
None,
&mut state,
None,
)
.unwrap();
assert!(!config_rx.has_changed().unwrap());
assert!(Arc::ptr_eq(&published, &config_rx.borrow()));
preparation::reload(
&path, &config_tx, &log_tx, None, None, &mut state, None, false,
)
.await
.unwrap();
assert!(!config_rx.has_changed().unwrap());
assert!(Arc::ptr_eq(&published, &config_rx.borrow()));
// Explicit reload bypasses the unchanged source hash and replaces the injected endpoint.
preparation::reload(
&path, &config_tx, &log_tx, None, None, &mut state, None, true,
)
.await
.unwrap();
assert!(config_rx.has_changed().unwrap());
assert!(!published.web_decoy_endpoints_equal(&config_rx.borrow()));
let current = config_rx.borrow_and_update().clone();
std::fs::write(&path, SOURCE.replace("localhost:18081", "localhost:18080")).unwrap();
preparation::reload(
&path, &config_tx, &log_tx, None, None, &mut state, None, false,
)
.await
.unwrap();
assert!(!config_rx.has_changed().unwrap());
assert!(Arc::ptr_eq(&current, &config_rx.borrow()));
}
#[tokio::test]
async fn decoy_dns_unchanged_watcher_source_skips_resolution() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(
&path,
SOURCE.replace("localhost", "unresolvable.example.invalid"),
)
.unwrap();
let source = preparation::read_source(&path).await.unwrap();
let initial = Arc::new(source.config.clone());
let (config_tx, config_rx) = watch::channel(initial.clone());
let (log_tx, _) = watch::channel(initial.general.log_level.clone());
let mut state = ReloadState::new(Some(source.rendered_hash));
assert!(preparation::source_matches_active(&initial, &source));
preparation::reload(
&path, &config_tx, &log_tx, None, None, &mut state, None, false,
)
.await
.unwrap();
assert!(!config_rx.has_changed().unwrap());
assert!(Arc::ptr_eq(&initial, &config_rx.borrow()));
}
#[tokio::test]
async fn decoy_dns_watcher_init_ignores_retained_restart_fields() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(&path, SOURCE).unwrap();
let loaded = ProxyConfig::load_prepared(path.clone()).await.unwrap();
let mut source = ProxyConfig::parse_source(&path).unwrap();
source.config.web.limits.max_static_file_bytes += 1;
assert!(preparation::source_matches_active(&loaded.config, &source));
source.config.web.conveyor = !source.config.web.conveyor;
assert!(!preparation::source_matches_active(&loaded.config, &source));
}
@@ -0,0 +1,85 @@
use super::*;
/// Restart-only differences must not enqueue another DNS preparation after activation.
pub(super) fn source_matches_active(
active: &ProxyConfig,
source: &crate::config::ParsedConfigSource,
) -> bool {
config_equal(active, &overlay_hot_fields(active, &source.config))
}
/// Reads only source state; watcher initialization and unchanged events never need DNS.
pub(super) async fn read_source(path: &PathBuf) -> Option<crate::config::ParsedConfigSource> {
let path = path.clone();
match tokio::task::spawn_blocking(move || ProxyConfig::parse_source(path)).await {
Ok(Ok(source)) => Some(source),
Ok(Err(error)) => {
error!("config reload: failed to parse: {}", error);
None
}
Err(error) => {
error!("config reload: source reader failed: {}", error);
None
}
}
}
/// Prepares fresh evidence only for changed sources or an explicit reload request.
pub(super) async fn reload(
path: &PathBuf,
config_tx: &watch::Sender<Arc<ProxyConfig>>,
log_tx: &watch::Sender<LogLevel>,
detected_ip_v4: Option<IpAddr>,
detected_ip_v6: Option<IpAddr>,
reload_state: &mut ReloadState,
dns_resolver: Option<Arc<crate::network::dns_overrides::GenerationDnsResolver>>,
force: bool,
) -> Option<WatchManifest> {
let source = read_source(path).await?;
let manifest = WatchManifest::from_source_files(&source.source_files);
if !force && reload_state.is_applied(source.rendered_hash) {
return Some(manifest);
}
let loaded = match source.prepare().await {
Ok(loaded) => loaded,
Err(error) => {
error!(
"config reload: preparation failed: {}; keeping old config",
error
);
return Some(manifest);
}
};
let old = config_tx.borrow().clone();
// Static-site reconstruction after the listener overlay must not block Tokio workers.
// The worker only builds data: cancellation cannot leave a detached publisher behind.
let result = tokio::task::spawn_blocking(move || {
prepare_effective_config(&old, &loaded.config).map(|applied| (loaded, applied))
})
.await;
match result {
Ok(Ok(prepared)) => reload_config_with_resolver(
prepared,
config_tx,
log_tx,
detected_ip_v4,
detected_ip_v6,
reload_state,
dns_resolver.as_deref(),
),
Ok(Err(error)) => {
error!(
"config reload: effective WEB validation failed: {}; keeping old config",
error
);
Some(manifest)
}
Err(error) => {
error!(
"config reload: runtime builder failed: {}; keeping old config",
error
);
Some(manifest)
}
}
}
+43
View File
@@ -29,6 +29,8 @@ mod runtime_web;
mod decode;
mod effective;
mod pipeline;
// Candidate-owned DNS preparation is separate from pure runtime reconstruction.
mod decoy_dns;
mod validate_core;
mod validate_me;
mod validate_runtime;
@@ -74,6 +76,19 @@ pub(crate) struct LoadedConfig {
pub(crate) rendered_hash: u64,
}
/// Normalized source configuration that has not acquired runtime DNS or static snapshots.
#[derive(Debug, Clone)]
pub(crate) struct ParsedConfigSource {
/// Source-only configuration; must be prepared before runtime publication.
pub(crate) config: ProxyConfig,
/// Normalized paths in the captured include graph.
pub(crate) source_files: Vec<PathBuf>,
/// Exact source documents used for revisions and atomic mutation checks.
pub(crate) source_contents: BTreeMap<PathBuf, String>,
/// Source-only hash, independent of DNS answers.
pub(crate) rendered_hash: u64,
}
/// Raw recursive source graph captured before typed deserialization.
#[derive(Debug, Clone)]
pub(crate) struct ConfigSourceGraph {
@@ -229,6 +244,33 @@ impl ProxyConfig {
pipeline::load_source_graph(graph)
}
/// Parses source without resolving decoys or creating runtime WEB snapshots.
pub(crate) fn parse_source<P: AsRef<Path>>(path: P) -> Result<ParsedConfigSource> {
Self::parse_source_with_overrides(path, &BTreeMap::new())
}
/// Checks decoy mode-specific keys before a source or API patch loses unknown fields.
pub(crate) fn validate_decoy_source_keys(document: &toml::Value) -> Result<()> {
decoy_dns::validate_mode_keys(document)
}
/// Parses an atomic mutation candidate without performing DNS.
pub(crate) fn parse_source_with_overrides<P: AsRef<Path>>(
path: P,
overrides: &BTreeMap<PathBuf, String>,
) -> Result<ParsedConfigSource> {
pipeline::parse_source_graph(Self::read_source_graph_with_overrides(path, overrides)?)
}
/// Loads and prepares a complete runtime candidate without blocking async workers.
pub(crate) async fn load_prepared(path: PathBuf) -> Result<LoadedConfig> {
let parsed = tokio::task::spawn_blocking(move || Self::parse_source(path))
.await
.map_err(|error| ProxyError::Config(format!("config reader failed: {error}")))??;
parsed.prepare().await
}
/// Rebuilds immutable credential lookup data for this configuration.
pub(crate) fn rebuild_runtime_user_auth(&mut self) -> Result<()> {
let snapshot = UserAuthSnapshot::from_users(&self.access.users)?;
self.runtime_user_auth = Some(Arc::new(snapshot));
@@ -250,6 +292,7 @@ impl ProxyConfig {
validate_web::validate_decoy_listener_separation(self)
}
/// Returns authentication data owned by this configuration generation.
pub(crate) fn runtime_user_auth(&self) -> Option<&UserAuthSnapshot> {
self.runtime_user_auth.as_deref()
}
+3
View File
@@ -1,5 +1,6 @@
use super::*;
/// Typed source together with its exact include graph and rendered input.
pub(super) type DecodedSourceGraph = (
ProxyConfig,
BTreeSet<PathBuf>,
@@ -7,6 +8,7 @@ pub(super) type DecodedSourceGraph = (
String,
);
/// Decodes source while retaining the metadata needed for atomic mutations.
pub(super) fn decode_source_graph(graph: ConfigSourceGraph) -> Result<DecodedSourceGraph> {
let ConfigSourceGraph {
source_contents,
@@ -17,6 +19,7 @@ pub(super) fn decode_source_graph(graph: ConfigSourceGraph) -> Result<DecodedSou
let parsed_toml: toml::Value =
toml::from_str(&processed).map_err(|e| ProxyError::Config(e.to_string()))?;
handle_unknown_config_keys(&parsed_toml)?;
ProxyConfig::validate_decoy_source_keys(&parsed_toml)?;
let general_table = parsed_toml
.get("general")
.and_then(|value| value.as_table());
+204
View File
@@ -0,0 +1,204 @@
use std::future::Future;
use std::time::Duration;
use super::*;
/// Rejects mode-specific keys even when unknown-key handling is permissive.
pub(super) fn validate_mode_keys(document: &toml::Value) -> Result<()> {
if let Some(vhosts) = document
.get("web")
.and_then(|web| web.get("vhosts"))
.and_then(toml::Value::as_array)
{
for (idx, vhost) in vhosts.iter().enumerate() {
if let Some(decoy) = vhost.get("decoy")
&& decoy.get("resolve").is_some()
&& decoy.get("mode").and_then(toml::Value::as_str) != Some("http_upstream")
{
return Err(ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.resolve is only valid for mode=http_upstream"
)));
}
}
}
Ok(())
}
/// Parses one origin and preserves the existing literal-address containment policy.
pub(super) fn parse_origin(
idx: usize,
upstream: &str,
resolve: WebDecoyResolve,
) -> Result<url::Url> {
let parsed = url::Url::parse(upstream).map_err(|error| {
ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream is invalid: {error}"
))
})?;
if parsed.scheme() != "http"
|| parsed.host_str().is_none()
|| !parsed.username().is_empty()
|| parsed.password().is_some()
|| parsed.query().is_some()
|| parsed.fragment().is_some()
|| parsed.path() != "/"
|| parsed.port() == Some(0)
{
return Err(ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream must be an http origin without credentials, path, query, or fragment"
)));
}
match parsed.host() {
Some(url::Host::Ipv4(ip)) => validate_address(idx, IpAddr::V4(ip))?,
Some(url::Host::Ipv6(ip)) => validate_address(idx, IpAddr::V6(ip))?,
Some(url::Host::Domain(_)) if resolve == WebDecoyResolve::Startup => {}
_ => {
return Err(ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream host must be a loopback or private IP literal unless resolve = \"startup\""
)));
}
}
Ok(parsed)
}
fn validate_address(idx: usize, ip: IpAddr) -> Result<()> {
let private = match ip {
IpAddr::V4(ip) => ip.is_loopback() || ip.is_private() || ip.is_link_local(),
IpAddr::V6(ip) => ip.is_loopback() || ip.is_unique_local() || ip.is_unicast_link_local(),
};
if !private {
return Err(ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream must remain inside loopback or a private network"
)));
}
Ok(())
}
/// Returns all captured answers so non-selected addresses cannot hide a listener loop.
pub(super) fn addresses(
config: &ProxyConfig,
idx: usize,
parsed: &url::Url,
require_resolved: bool,
) -> Result<Vec<SocketAddr>> {
let port = parsed
.port_or_known_default()
.ok_or_else(|| ProxyError::Config("WEB decoy port cannot be resolved".to_string()))?;
let answers = match parsed.host() {
Some(url::Host::Ipv4(ip)) => vec![SocketAddr::new(IpAddr::V4(ip), port)],
Some(url::Host::Ipv6(ip)) => vec![SocketAddr::new(IpAddr::V6(ip), port)],
Some(url::Host::Domain(host)) => {
let key = (host.to_string(), port);
match config.web.decoy_dns.origins.get(&key) {
Some(answers) if !answers.is_empty() => answers.clone(),
_ if !require_resolved => return Ok(Vec::new()),
_ => {
return Err(ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream requires async preparation for resolve = \"startup\""
)));
}
}
}
_ => return Err(ProxyError::Config("WEB decoy host is missing".to_string())),
};
for addr in &answers {
validate_address(idx, addr.ip())?;
if addr.port() != port {
return Err(ProxyError::Config(
"WEB decoy DNS snapshot port mismatch".to_string(),
));
}
}
Ok(answers)
}
/// Captures fresh resolver evidence without publishing any partially prepared configuration.
pub(super) async fn prepare<F, Fut>(config: &mut ProxyConfig, mut lookup: F) -> Result<()>
where
F: FnMut(String, u16) -> Fut,
Fut: Future<Output = std::io::Result<Vec<SocketAddr>>>,
{
let mut snapshot = WebDecoyDnsSnapshot::default();
for (idx, vhost) in config.web.vhosts.iter().enumerate() {
let WebDecoyConfig::HttpUpstream { upstream, resolve } = &vhost.decoy else {
continue;
};
let parsed = parse_origin(idx, upstream, *resolve)?;
let Some(url::Host::Domain(host)) = parsed.host() else {
continue;
};
let port = parsed
.port_or_known_default()
.ok_or_else(|| ProxyError::Config("WEB decoy port cannot be resolved".to_string()))?;
let key = (host.to_string(), port);
if snapshot.origins.contains_key(&key) {
continue;
}
// Dropping this wait does not forcibly cancel a running OS resolver call.
let answers = tokio::time::timeout(
Duration::from_secs(config.web.timeouts.decoy_resolve_secs),
lookup(key.0.clone(), port),
)
.await
.map_err(|_| {
ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream DNS resolution timed out after {} seconds",
config.web.timeouts.decoy_resolve_secs
))
})?
.map_err(|error| {
ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream DNS resolution failed: {error}"
))
})?;
if answers.is_empty() {
return Err(ProxyError::Config(format!(
"web.vhosts[{idx}].decoy.upstream DNS resolution returned no addresses"
)));
}
for addr in &answers {
validate_address(idx, addr.ip())?;
}
snapshot.origins.insert(key, answers);
}
config.web.decoy_dns = Arc::new(snapshot);
validate_web::validate_decoy_listener_separation(config)
}
/// Builds the unchanged request-time tuple from immutable configuration evidence.
pub(super) fn build_upstream(
config: &ProxyConfig,
idx: usize,
upstream: &str,
resolve: WebDecoyResolve,
) -> Result<WebRuntimeDecoy> {
let parsed = parse_origin(idx, upstream, resolve)?;
let addr = addresses(config, idx, &parsed, true)?
.first()
.copied()
.ok_or_else(|| ProxyError::Config("WEB decoy DNS snapshot is empty".to_string()))?;
// URL authority keeps the configured hostname and brackets IPv6 literals.
let authority = parsed[url::Position::BeforeHost..url::Position::AfterPort].to_string();
Ok(WebRuntimeDecoy::HttpUpstream { addr, authority })
}
impl ProxyConfig {
/// Compares selected endpoints separately from source-only configuration equality.
pub(crate) fn web_decoy_endpoints_equal(&self, other: &Self) -> bool {
fn selected(config: &ProxyConfig) -> impl Iterator<Item = (&String, SocketAddr)> {
config
.web
.runtime
.iter()
.flat_map(|runtime| runtime.vhosts.iter())
.filter_map(|(host, vhost)| match vhost.decoy {
WebRuntimeDecoy::HttpUpstream { addr, .. } => Some((host, addr)),
WebRuntimeDecoy::StaticDirectory(_) => None,
})
}
selected(self).eq(selected(other))
}
}
#[cfg(test)]
mod tests;
+330
View File
@@ -0,0 +1,330 @@
use std::sync::atomic::{AtomicUsize, Ordering};
use super::*;
const SOURCE: &str = r#"
[access.users]
alice = "000102030405060708090a0b0c0d0e0f"
[[server.listeners]]
ip = "127.0.0.1"
port = 18080
transport = "web"
web_trusted_proxy_cidrs = ["127.0.0.1/32"]
[web]
enabled = true
[[web.vhosts]]
host = "proxy.example.com"
public_addr = "203.0.113.10:443"
[web.vhosts.decoy]
mode = "http_upstream"
upstream = "http://Example.COM:18081"
resolve = "startup"
[[web.vhosts.profiles]]
user = "alice"
secret_mode = "dd"
"#;
fn source(text: &str) -> ParsedConfigSource {
pipeline::parse_source_graph(ConfigSourceGraph {
rendered: text.to_string(),
source_contents: BTreeMap::new(),
})
.unwrap()
}
async fn prepared(answers: &[&str]) -> LoadedConfig {
let mut parsed = source(SOURCE);
let answers: Vec<SocketAddr> = answers.iter().map(|addr| addr.parse().unwrap()).collect();
prepare(&mut parsed.config, |_, _| {
std::future::ready(Ok(answers.clone()))
})
.await
.unwrap();
pipeline::finish(parsed).unwrap()
}
fn endpoint(config: &ProxyConfig) -> (SocketAddr, &str) {
let runtime = config.web.runtime.as_ref().unwrap();
match &runtime.vhosts["proxy.example.com"].decoy {
WebRuntimeDecoy::HttpUpstream { addr, authority } => (*addr, authority),
_ => panic!("expected HTTP decoy"),
}
}
#[tokio::test]
async fn decoy_dns_snapshot_deduplicates_normalized_origins_and_preserves_order() {
let mut parsed = source(SOURCE);
let mut second = parsed.config.web.vhosts[0].clone();
second.host = "bsi.bund.de".to_string();
second.decoy = WebDecoyConfig::HttpUpstream {
upstream: "http://example.com:18081/".to_string(),
resolve: WebDecoyResolve::Startup,
};
parsed.config.web.vhosts.push(second);
let calls = AtomicUsize::new(0);
prepare(&mut parsed.config, |host, port| {
assert_eq!((host.as_str(), port), ("example.com", 18081));
calls.fetch_add(1, Ordering::SeqCst);
std::future::ready(Ok(vec![
"[fd00::2]:18081".parse().unwrap(),
"10.0.0.2:18081".parse().unwrap(),
]))
})
.await
.unwrap();
let mut loaded = pipeline::finish(parsed).unwrap();
for _ in 0..3 {
loaded.config.validate_effective_web().unwrap();
loaded.config.rebuild_runtime_web().unwrap();
}
assert_eq!(calls.load(Ordering::SeqCst), 1);
assert_eq!(
endpoint(&loaded.config),
("[fd00::2]:18081".parse().unwrap(), "example.com:18081")
);
assert_eq!(
loaded
.config
.web
.decoy_dns
.origins
.values()
.next()
.unwrap()
.len(),
2
);
assert!(
!serde_json::to_string(&loaded.config)
.unwrap()
.contains("fd00::2")
);
}
#[tokio::test]
async fn decoy_dns_rejects_every_public_answer_including_non_selected() {
for disallowed in [
"8.8.8.8:18081",
"[2001:4860:4860::8888]:18081",
"[::ffff:127.0.0.1]:18081",
] {
let mut parsed = source(SOURCE);
let error = prepare(&mut parsed.config, |_, _| {
std::future::ready(Ok(vec![
"127.0.0.2:18081".parse().unwrap(),
disallowed.parse().unwrap(),
]))
})
.await
.unwrap_err();
assert!(error.to_string().contains("must remain inside"));
assert!(parsed.config.web.runtime.is_none());
}
}
#[tokio::test]
async fn decoy_dns_rejects_non_selected_direct_and_wildcard_listener_loops() {
for listener in ["127.0.0.1", "0.0.0.0"] {
let mut parsed = source(&SOURCE.replace("Example.COM:18081", "example.com:18080"));
parsed.config.server.listeners[0].ip = listener.parse().unwrap();
let error = prepare(&mut parsed.config, |_, _| {
std::future::ready(Ok(vec![
"[fd00::2]:18080".parse().unwrap(),
"127.0.0.1:18080".parse().unwrap(),
]))
})
.await
.unwrap_err();
assert!(error.to_string().contains("overlaps WEB listener"));
}
}
#[tokio::test]
async fn decoy_dns_empty_answers_and_resolver_errors_fail_closed() {
let mut parsed = source(SOURCE);
let error = prepare(&mut parsed.config, |_, _| {
std::future::ready(Ok(Vec::new()))
})
.await
.unwrap_err();
assert!(error.to_string().contains("returned no addresses"));
let error = prepare(&mut parsed.config, |_, _| {
std::future::ready(Err(std::io::Error::new(
std::io::ErrorKind::NotFound,
"test resolver failure",
)))
})
.await
.unwrap_err();
assert!(error.to_string().contains("DNS resolution failed"));
}
#[tokio::test]
async fn decoy_dns_timeout_is_bounded() {
let mut parsed = source(SOURCE);
parsed.config.web.timeouts.decoy_resolve_secs = 1;
let error = tokio::time::timeout(
Duration::from_secs(3),
prepare(&mut parsed.config, |_, _| {
std::future::pending::<std::io::Result<Vec<SocketAddr>>>()
}),
)
.await
.unwrap()
.unwrap_err();
assert!(error.to_string().contains("timed out after 1 seconds"));
}
#[tokio::test]
async fn decoy_dns_literal_startup_never_looks_up_and_keeps_ipv6_authority() {
let mut parsed = source(&SOURCE.replace("Example.COM:18081", "[::1]:18081"));
prepare(&mut parsed.config, |_, _| {
std::future::ready(Err(std::io::Error::other("IP literals must not reach DNS")))
})
.await
.unwrap();
let loaded = pipeline::finish(parsed).unwrap();
assert_eq!(
endpoint(&loaded.config),
("[::1]:18081".parse().unwrap(), "[::1]:18081")
);
}
#[test]
fn decoy_dns_source_parse_is_not_a_prepared_runtime() {
let parsed = source(SOURCE);
assert!(parsed.config.web.runtime.is_none());
assert!(parsed.config.web.decoy_dns.origins.is_empty());
assert!(
pipeline::finish(parsed)
.unwrap_err()
.to_string()
.contains("requires async preparation")
);
}
#[tokio::test]
async fn decoy_dns_evidence_is_bound_to_origin_and_port() {
let loaded = prepared(&["10.0.0.2:18081"]).await;
for changed in ["http://telekom.com:18081", "http://example.com:18082"] {
let mut config = loaded.config.clone();
config.web.vhosts[0].decoy = WebDecoyConfig::HttpUpstream {
upstream: changed.to_string(),
resolve: WebDecoyResolve::Startup,
};
assert!(
config
.validate_effective_web()
.unwrap_err()
.to_string()
.contains("requires async preparation")
);
assert!(config.rebuild_runtime_web().is_err());
}
}
#[tokio::test]
async fn decoy_dns_reload_pins_old_generation_and_detects_dns_only_changes() {
let old = prepared(&["10.0.0.2:18081"]).await;
let new = prepared(&["10.0.0.3:18081"]).await;
assert_eq!(old.rendered_hash, new.rendered_hash);
assert_eq!(
serde_json::to_value(&old.config).unwrap(),
serde_json::to_value(&new.config).unwrap()
);
let resolved =
crate::maestro::runtime_build::resolve_reload_config(&old.config, &new.config).unwrap();
assert!(resolved.runtime_changed);
assert!(!old.config.web_decoy_endpoints_equal(&new.config));
assert_eq!(endpoint(&old.config).0, "10.0.0.2:18081".parse().unwrap());
let same = prepared(&["10.0.0.3:18081", "10.0.0.4:18081"]).await;
assert!(same.config.web_decoy_endpoints_equal(&new.config));
}
#[tokio::test]
async fn decoy_dns_effective_listener_overlay_rechecks_every_answer() {
let old = prepared(&["10.0.0.2:18081"]).await;
let mut parsed = source(SOURCE);
parsed.config.server.listeners[0].port = Some(18082);
parsed.config.web.vhosts[0].decoy = WebDecoyConfig::HttpUpstream {
upstream: "http://example.com:18080".to_string(),
resolve: WebDecoyResolve::Startup,
};
prepare(&mut parsed.config, |_, _| {
std::future::ready(Ok(vec![
"10.0.0.2:18080".parse().unwrap(),
"127.0.0.1:18080".parse().unwrap(),
]))
})
.await
.unwrap();
let desired = pipeline::finish(parsed).unwrap();
assert!(
crate::maestro::runtime_build::resolve_reload_config(&old.config, &desired.config)
.err()
.unwrap()
.contains("overlaps WEB listener")
);
}
#[tokio::test]
async fn decoy_dns_rejects_ipv6_direct_and_wildcard_listener_loops() {
for listener in ["::1", "::"] {
let mut parsed = source(&SOURCE.replace("Example.COM:18081", "example.com:18080"));
parsed.config.network.ipv6 = Some(true);
parsed.config.server.listeners[0].ip = listener.parse().unwrap();
let error = prepare(&mut parsed.config, |_, _| {
std::future::ready(Ok(vec![
"10.0.0.2:18080".parse().unwrap(),
"[::1]:18080".parse().unwrap(),
]))
})
.await
.unwrap_err();
assert!(error.to_string().contains("overlaps WEB listener"));
}
}
#[test]
fn decoy_dns_startup_does_not_relax_origin_url_rules() {
for url in [
"https://example.com",
"http://user@example.com",
"http://user:pass@example.com",
"http://example.com/path",
"http://example.com/?q=1",
"http://example.com/#fragment",
"http://example.com:0",
] {
assert!(
parse_origin(0, url, WebDecoyResolve::Startup).is_err(),
"accepted {url}"
);
}
}
#[tokio::test]
async fn decoy_dns_default_port_authority_and_evidence_port_fence() {
let mut parsed = source(&SOURCE.replace("Example.COM:18081", "example.com"));
prepare(&mut parsed.config, |_, port| {
assert_eq!(port, 80);
std::future::ready(Ok(vec!["127.0.0.2:80".parse().unwrap()]))
})
.await
.unwrap();
let loaded = pipeline::finish(parsed).unwrap();
assert_eq!(endpoint(&loaded.config).1, "example.com");
let mut config = loaded.config.clone();
Arc::make_mut(&mut config.web.decoy_dns).origins.insert(
("example.com".to_string(), 80),
vec!["127.0.0.2:81".parse().unwrap()],
);
assert!(
config
.validate_effective_web()
.unwrap_err()
.to_string()
.contains("port mismatch")
);
assert!(config.rebuild_runtime_web().is_err());
}
+1 -1
View File
@@ -1,5 +1,6 @@
use super::*;
/// Normalizes source policy and authentication data before external decoy preparation.
pub(super) fn apply(config: &mut ProxyConfig) -> Result<()> {
// Normalize optional TLS fetch scope: whitespace-only values disable scoped routing.
config.censorship.tls_fetch_scope = config.censorship.tls_fetch_scope.trim().to_string();
@@ -217,6 +218,5 @@ pub(super) fn apply(config: &mut ProxyConfig) -> Result<()> {
validate_logging_config(&config.logging)?;
validate_upstreams(config)?;
config.rebuild_runtime_user_auth()?;
config.rebuild_runtime_web()?;
Ok(())
}
+35 -2
View File
@@ -1,18 +1,51 @@
use super::*;
/// Completes legacy synchronous loads without performing hostname resolution.
pub(super) fn load_source_graph(graph: ConfigSourceGraph) -> Result<LoadedConfig> {
finish(parse_source_graph(graph)?)
}
/// Normalizes source while leaving DNS and immutable site acquisition to preparation.
pub(super) fn parse_source_graph(graph: ConfigSourceGraph) -> Result<ParsedConfigSource> {
let (mut config, source_files, source_contents, processed) =
decode::decode_source_graph(graph)?;
validate_core::validate(&mut config)?;
validate_runtime::validate(&mut config)?;
validate_me::validate(&mut config)?;
validate_server::validate(&mut config)?;
validate_web::validate(&mut config)?;
validate_web::validate_source(&mut config)?;
effective::apply(&mut config)?;
Ok(LoadedConfig {
Ok(ParsedConfigSource {
config,
source_files: source_files.into_iter().collect(),
source_contents,
rendered_hash: hash_rendered_snapshot(&processed),
})
}
/// Completes synchronous validation and static acquisition after DNS is captured.
pub(super) fn finish(mut parsed: ParsedConfigSource) -> Result<LoadedConfig> {
parsed.config.validate_effective_web()?;
parsed.config.rebuild_runtime_web()?;
Ok(LoadedConfig {
config: parsed.config,
source_files: parsed.source_files,
source_contents: parsed.source_contents,
rendered_hash: parsed.rendered_hash,
})
}
impl ParsedConfigSource {
/// Resolves each unique decoy origin once, then builds a fully validated candidate.
pub(crate) async fn prepare(mut self) -> Result<LoadedConfig> {
decoy_dns::prepare(&mut self.config, |host, port| async move {
tokio::net::lookup_host((host.as_str(), port))
.await
.map(|answers| answers.collect())
})
.await?;
tokio::task::spawn_blocking(move || finish(self))
.await
.map_err(|error| ProxyError::Config(format!("config preparation failed: {error}")))?
}
}
+7 -31
View File
@@ -47,8 +47,10 @@ pub(super) fn rebuild(config: &mut ProxyConfig) -> Result<()> {
let mut static_bytes = 0usize;
let carrier_candidates: Arc<[WebCarrier]> = config.web.carrier_candidates().into();
for vhost in &config.web.vhosts {
for (vhost_idx, vhost) in config.web.vhosts.iter().enumerate() {
let decoy = build_decoy(
config,
vhost_idx,
vhost,
&config.web.limits,
&mut static_files,
@@ -182,42 +184,16 @@ fn client_secret(secret: [u8; 16], mode: WebSecretMode) -> ([u8; 17], usize) {
}
fn build_decoy(
config: &ProxyConfig,
vhost_idx: usize,
vhost: &WebVhostConfig,
limits: &WebLimitsConfig,
static_files: &mut usize,
static_bytes: &mut usize,
) -> Result<WebRuntimeDecoy> {
match &vhost.decoy {
WebDecoyConfig::HttpUpstream { upstream } => {
let parsed = url::Url::parse(upstream).map_err(|error| {
ProxyError::Config(format!(
"WEB decoy upstream for `{}` is invalid: {error}",
vhost.host
))
})?;
let ip = match parsed.host() {
Some(url::Host::Ipv4(ip)) => std::net::IpAddr::V4(ip),
Some(url::Host::Ipv6(ip)) => std::net::IpAddr::V6(ip),
_ => {
return Err(ProxyError::Config(
"WEB decoy host must be an IP literal".to_string(),
));
}
};
let host = ip.to_string();
let port = parsed.port_or_known_default().ok_or_else(|| {
ProxyError::Config("WEB decoy port cannot be resolved".to_string())
})?;
let authority = match (ip, parsed.port()) {
(std::net::IpAddr::V6(_), Some(_)) => format!("[{host}]:{port}"),
(std::net::IpAddr::V6(_), None) => format!("[{host}]"),
(std::net::IpAddr::V4(_), Some(_)) => format!("{host}:{port}"),
(std::net::IpAddr::V4(_), None) => host.clone(),
};
Ok(WebRuntimeDecoy::HttpUpstream {
addr: SocketAddr::new(ip, port),
authority,
})
WebDecoyConfig::HttpUpstream { upstream, resolve } => {
decoy_dns::build_upstream(config, vhost_idx, upstream, *resolve)
}
WebDecoyConfig::StaticDirectory { directory, index } => {
let site = load_static_site(directory, index, limits, static_files, static_bytes)?;
+4 -1
View File
@@ -263,6 +263,8 @@ const LISTENER_CONFIG_KEYS: &[&str] = &[
const WEB_CONFIG_KEYS: &[&str] = &[
"enabled",
"carrier",
"carrier_method",
"conveyor",
"carriers",
"carrier_learning",
"carrier_negotiation_aggressiveness",
@@ -363,10 +365,11 @@ const WEB_TIMEOUTS_CONFIG_KEYS: &[&str] = &[
"http_overload_timeout_ms",
"shutdown_secs",
"decoy_header_secs",
"decoy_resolve_secs",
];
const WEB_VHOST_CONFIG_KEYS: &[&str] = &["host", "base_path", "public_addr", "decoy", "profiles"];
const WEB_DECOY_CONFIG_KEYS: &[&str] = &["mode", "upstream", "directory", "index"];
const WEB_DECOY_CONFIG_KEYS: &[&str] = &["mode", "upstream", "directory", "index", "resolve"];
const WEB_PROFILE_CONFIG_KEYS: &[&str] = &[
"user",
"secret_mode",
+25 -23
View File
@@ -27,6 +27,12 @@ const MAX_WEB_MEMORY_ENVELOPE_BYTES: usize = 4 * 1024 * 1024 * 1024;
/// Validates WEB policy and resource bounds before building runtime state.
pub(super) fn validate(config: &mut ProxyConfig) -> Result<()> {
validate_source(config)?;
validate_decoy_listener_separation(config)
}
/// Validates source policy without requiring DNS evidence for opted-in origins.
pub(super) fn validate_source(config: &mut ProxyConfig) -> Result<()> {
let web_listener_count = config
.server
.listeners
@@ -83,12 +89,19 @@ pub(super) fn validate(config: &mut ProxyConfig) -> Result<()> {
timeouts::validate(&config.web.timeouts)?;
websocket::validate(&carriers, &config.web.limits, &config.web.timeouts)?;
validate_vhosts(config)?;
validate_decoy_listener_separation(config)?;
validate_decoy_listener_separation_inner(config, false)?;
Ok(())
}
/// Rejects a direct decoy recursion into an effective WEB listener.
pub(super) fn validate_decoy_listener_separation(config: &ProxyConfig) -> Result<()> {
validate_decoy_listener_separation_inner(config, true)
}
fn validate_decoy_listener_separation_inner(
config: &ProxyConfig,
require_resolved: bool,
) -> Result<()> {
let web_listeners = config
.server
.listeners
@@ -101,30 +114,19 @@ pub(super) fn validate_decoy_listener_separation(config: &ProxyConfig) -> Result
.map(|listener| SocketAddr::new(listener.ip, listener.port.unwrap_or(config.server.port)))
.collect::<Vec<_>>();
for (vhost_idx, vhost) in config.web.vhosts.iter().enumerate() {
let WebDecoyConfig::HttpUpstream { upstream } = &vhost.decoy else {
let WebDecoyConfig::HttpUpstream { upstream, resolve } = &vhost.decoy else {
continue;
};
let parsed = url::Url::parse(upstream).map_err(|error| {
ProxyError::Config(format!(
"web.vhosts[{vhost_idx}].decoy.upstream is invalid: {error}"
))
})?;
let Some(port) = parsed.port_or_known_default() else {
continue;
};
let upstream_ip = match parsed.host() {
Some(url::Host::Ipv4(ip)) => IpAddr::V4(ip),
Some(url::Host::Ipv6(ip)) => IpAddr::V6(ip),
_ => continue,
};
let upstream_addr = SocketAddr::new(upstream_ip, port);
if web_listeners
.iter()
.any(|listener| listener_covers(*listener, upstream_addr))
{
return config_error(&format!(
"web.vhosts[{vhost_idx}].decoy upstream overlaps WEB listener {upstream_addr}"
));
let parsed = decoy_dns::parse_origin(vhost_idx, upstream, *resolve)?;
for upstream_addr in decoy_dns::addresses(config, vhost_idx, &parsed, require_resolved)? {
if web_listeners
.iter()
.any(|listener| listener_covers(*listener, upstream_addr))
{
return config_error(&format!(
"web.vhosts[{vhost_idx}].decoy upstream overlaps WEB listener {upstream_addr}"
));
}
}
}
Ok(())
+2 -1
View File
@@ -4,7 +4,8 @@ const WEB_DEBUG_RENDERERS: usize = 2;
const WEB_DEBUG_STATUS_PAGE_BYTES: usize = 8 * 1024 * 1024;
const WEB_DEBUG_GROUP_SCRATCH_BYTES: usize = 4 * 1024 * 1024;
const WEB_CARRIER_LEARNING_ENTRY_BYTES: usize = 512;
const WEB_LANE_STATE_BYTES: usize = 512;
// Includes the lane plus bounded conveyor slots, Notify ownership, and both hash tables.
const WEB_LANE_STATE_BYTES: usize = 1024;
const WEB_OVERLOAD_CONNECTION_BYTES: usize = 4 * 1024;
// Each profile capability is stored in its vhost and in the global containment table.
const WEB_CAPABILITY_INDEX_ENTRY_BYTES: usize = 64;
+1
View File
@@ -26,6 +26,7 @@ pub(super) fn validate(timeouts: &WebTimeoutsConfig) -> Result<()> {
("http_idle_secs", timeouts.http_idle_secs),
("shutdown_secs", timeouts.shutdown_secs),
("decoy_header_secs", timeouts.decoy_header_secs),
("decoy_resolve_secs", timeouts.decoy_resolve_secs),
];
if let Some((field, _)) = values
.into_iter()
+2 -39
View File
@@ -150,45 +150,8 @@ pub(super) fn web_host_last_label_is_numeric(host: &str) -> bool {
pub(super) fn validate_decoy(vhost_idx: usize, decoy: &WebDecoyConfig) -> Result<()> {
match decoy {
WebDecoyConfig::HttpUpstream { upstream } => {
let parsed = url::Url::parse(upstream).map_err(|error| {
ProxyError::Config(format!(
"web.vhosts[{vhost_idx}].decoy.upstream is invalid: {error}"
))
})?;
if parsed.scheme() != "http"
|| parsed.host_str().is_none()
|| !parsed.username().is_empty()
|| parsed.password().is_some()
|| parsed.query().is_some()
|| parsed.fragment().is_some()
|| parsed.path() != "/"
|| parsed.port() == Some(0)
{
return config_error(&format!(
"web.vhosts[{vhost_idx}].decoy.upstream must be an http origin without credentials, path, query, or fragment"
));
}
let ip = match parsed.host() {
Some(url::Host::Ipv4(ip)) => IpAddr::V4(ip),
Some(url::Host::Ipv6(ip)) => IpAddr::V6(ip),
_ => {
return config_error(&format!(
"web.vhosts[{vhost_idx}].decoy.upstream host must be a loopback or private IP literal"
));
}
};
let private = match ip {
IpAddr::V4(ip) => ip.is_loopback() || ip.is_private() || ip.is_link_local(),
IpAddr::V6(ip) => {
ip.is_loopback() || ip.is_unique_local() || ip.is_unicast_link_local()
}
};
if !private {
return config_error(&format!(
"web.vhosts[{vhost_idx}].decoy.upstream must remain inside loopback or a private network"
));
}
WebDecoyConfig::HttpUpstream { upstream, resolve } => {
decoy_dns::parse_origin(vhost_idx, upstream, *resolve)?;
}
WebDecoyConfig::StaticDirectory { directory, index } => {
if !directory.is_absolute() {
+1 -1
View File
@@ -6,5 +6,5 @@ mod load;
mod types;
pub use load::ProxyConfig;
pub(crate) use load::{ConfigSourceGraph, LoadedConfig};
pub(crate) use load::{ConfigSourceGraph, LoadedConfig, ParsedConfigSource};
pub use types::*;
@@ -1,5 +1,71 @@
use super::*;
#[test]
fn conntrack_control_is_opt_in_by_default() {
assert!(!default_conntrack_control_enabled());
assert!(
!ProxyConfig::default()
.server
.conntrack_control
.inline_conntrack_control
);
let cfg = load_config_from_temp_toml(
r#"
[censorship]
tls_domain = "example.com"
[access.users]
user = "00000000000000000000000000000000"
"#,
);
assert!(!cfg.server.conntrack_control.inline_conntrack_control);
assert!(
!cfg.server
.conntrack_control
.inline_conntrack_control_explicit
);
assert_eq!(
serde_json::to_value(&cfg.server.conntrack_control).unwrap()["inline_conntrack_control"],
false,
);
}
#[test]
fn conntrack_control_preserves_explicit_enablement() {
for enabled in [false, true] {
let cfg = load_config_from_temp_toml(&format!(
r#"
[server.conntrack_control]
inline_conntrack_control = {enabled}
backend = "nftables"
[censorship]
tls_domain = "example.com"
[access.users]
user = "00000000000000000000000000000000"
"#,
));
assert_eq!(
cfg.server.conntrack_control.inline_conntrack_control,
enabled
);
assert!(
cfg.server
.conntrack_control
.inline_conntrack_control_explicit
);
assert_eq!(
cfg.server.conntrack_control.backend,
ConntrackBackend::Nftables
);
assert_eq!(
serde_json::to_value(&cfg.server.conntrack_control).unwrap()["inline_conntrack_control"],
enabled,
);
}
}
#[test]
fn conntrack_pressure_high_watermark_out_of_range_is_rejected() {
let toml = r#"
@@ -3,6 +3,15 @@ use super::*;
#[path = "web_tests/base_path_tests.rs"]
mod base_path_tests;
// Method selection must not alter capability identity or accepted config literals.
#[path = "web_tests/carrier_method_tests.rs"]
mod carrier_method_tests;
#[path = "web_tests/conveyor_tests.rs"]
mod conveyor_tests;
#[path = "web_tests/decoy_dns_tests.rs"]
mod decoy_dns_tests;
const WEB_CONFIG: &str = r#"
[access.users]
alice = "000102030405060708090a0b0c0d0e0f"
@@ -0,0 +1,48 @@
use super::*;
#[test]
fn carrier_method_defaults_and_roundtrips_without_changing_capabilities() {
let default = load_config_from_temp_toml(WEB_CONFIG);
assert_eq!(default.web.carrier_method, WebCarrierMethod::Post);
let capabilities = default.web.runtime.as_ref().unwrap().capabilities.clone();
for (token, method) in [
("post", WebCarrierMethod::Post),
("put", WebCarrierMethod::Put),
] {
let configured = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
&format!("carrier = \"https-lanes\"\ncarrier_method = \"{token}\""),
);
let source = format!("[general]\nconfig_strict = true\n{configured}");
let config = load_config_from_temp_toml(&source);
assert_eq!(config.web.carrier_method, method);
assert_eq!(
config.web.runtime.as_ref().unwrap().capabilities,
capabilities
);
let json = serde_json::to_value(&config.web).unwrap();
assert_eq!(json["carrier_method"], token);
let decoded: WebConfig = serde_json::from_value(json).unwrap();
assert_eq!(decoded.carrier_method, method);
let serialized = toml::to_string(&config.web).unwrap();
let decoded: WebConfig = toml::from_str(&serialized).unwrap();
assert_eq!(decoded.carrier_method, method);
}
}
#[test]
fn carrier_method_rejects_unknown_tokens_types_and_aliases() {
for value in ["\"POST\"", "\"PUT\"", "\"patch\"", "true", "42", "[]"] {
let source = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
&format!("carrier = \"https-lanes\"\ncarrier_method = {value}"),
);
assert!(load_config_error_from_temp_toml(&source).contains("carrier_method"));
}
let configured = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
"carrier = \"https-lanes\"\nhttp_method = \"put\"",
);
let source = format!("[general]\nconfig_strict = true\n{configured}");
assert!(load_config_error_from_temp_toml(&source).contains("http_method"));
}
@@ -0,0 +1,44 @@
use super::*;
#[test]
fn conveyor_defaults_on_and_roundtrips_without_changing_capabilities() {
let default = load_config_from_temp_toml(WEB_CONFIG);
assert!(default.web.conveyor);
assert!(ProxyConfig::default().web.conveyor);
let capabilities = default.web.runtime.as_ref().unwrap().capabilities.clone();
for enabled in [false, true] {
let configured = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
&format!("carrier = \"https-lanes\"\nconveyor = {enabled}"),
);
let config =
load_config_from_temp_toml(&format!("[general]\nconfig_strict = true\n{configured}",));
assert_eq!(config.web.conveyor, enabled);
assert_eq!(
config.web.runtime.as_ref().unwrap().capabilities,
capabilities
);
let json = serde_json::to_value(&config.web).unwrap();
assert_eq!(json["conveyor"], enabled);
assert_eq!(
serde_json::from_value::<WebConfig>(json).unwrap().conveyor,
enabled
);
let toml = toml::to_string(&config.web).unwrap();
assert_eq!(
toml::from_str::<WebConfig>(&toml).unwrap().conveyor,
enabled
);
}
}
#[test]
fn conveyor_rejects_non_boolean_values() {
for value in ["\"true\"", "1", "0", "[]", "{}"] {
let configured = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
&format!("carrier = \"https-lanes\"\nconveyor = {value}"),
);
assert!(load_config_error_from_temp_toml(&configured).contains("conveyor"));
}
}
@@ -0,0 +1,57 @@
use super::*;
#[test]
fn decoy_dns_resolution_defaults_to_never() {
let decoy: WebDecoyConfig = serde_json::from_value(serde_json::json!({
"mode": "http_upstream",
"upstream": "http://127.0.0.1:18081"
}))
.unwrap();
let serialized = serde_json::to_value(decoy).unwrap();
assert_eq!(serialized["resolve"], "never");
}
#[test]
fn decoy_dns_timeout_defaults_to_five_seconds() {
let serialized = serde_json::to_value(WebTimeoutsConfig::default()).unwrap();
assert_eq!(serialized["decoy_resolve_secs"], 5);
}
#[test]
fn decoy_dns_resolution_is_opt_in_and_mode_specific() {
let hostname = WEB_CONFIG.replace("127.0.0.1:18081", "example.com:18081");
assert!(load_config_error_from_temp_toml(&hostname).contains("IP literal"));
let never = hostname.replace(
"mode = \"http_upstream\"",
"mode = \"http_upstream\"\nresolve = \"never\"",
);
assert!(load_config_error_from_temp_toml(&never).contains("IP literal"));
let unknown = WEB_CONFIG.replace(
"mode = \"http_upstream\"",
"mode = \"http_upstream\"\nresolve = \"always\"",
);
assert!(load_config_error_from_temp_toml(&unknown).contains("unknown variant"));
let wrong_mode = WEB_CONFIG.replace(
"mode = \"http_upstream\"",
"mode = \"static_directory\"\nresolve = \"startup\"\ndirectory = \"/tmp\"",
);
assert!(
load_config_error_from_temp_toml(&wrong_mode).contains("only valid for mode=http_upstream")
);
}
#[test]
fn decoy_dns_timeout_bounds_are_independent_from_request_deadlines() {
for seconds in [0, 3601] {
let text = format!("{WEB_CONFIG}\n[web.timeouts]\ndecoy_resolve_secs = {seconds}\n");
assert!(load_config_error_from_temp_toml(&text).contains("decoy_resolve_secs"));
}
let text = format!("{WEB_CONFIG}\n[web.timeouts]\ndecoy_resolve_secs = 3600\n");
assert_eq!(
load_config_from_temp_toml(&text)
.web
.timeouts
.decoy_resolve_secs,
3600
);
}
+5 -5
View File
@@ -54,16 +54,16 @@ pub use server::{
#[allow(unused_imports)]
pub use web::{
WebCarrierNegotiationAggressiveness, WebConfig, WebDecoyConfig, WebDecoyFastTrackMode,
WebHttpConnectionCapacityAction, WebLimitsConfig, WebProfileConfig, WebSecretMode,
WebTimeoutsConfig, WebVhostConfig,
WebDecoyResolve, WebHttpConnectionCapacityAction, WebLimitsConfig, WebProfileConfig,
WebSecretMode, WebTimeoutsConfig, WebVhostConfig,
};
pub(crate) use web::{
WebRuntimeConfig, WebRuntimeDecoy, WebRuntimeProfile, WebRuntimeVhost, WebStaticAsset,
WebStaticSite,
WebDecoyDnsSnapshot, WebRuntimeConfig, WebRuntimeDecoy, WebRuntimeProfile, WebRuntimeVhost,
WebStaticAsset, WebStaticSite,
};
pub(crate) use web_carrier::WEB_CARRIER_LEARNING_MIN_ENTRIES;
#[allow(unused_imports)]
pub use web_carrier::{WebCarrier, WebCarriers};
pub use web_carrier::{WebCarrier, WebCarrierMethod, WebCarriers};
pub(crate) use web_debug::web_debug_fits_limits;
pub use web_debug::{WebDebugBodyCapture, WebDebugConfig};
+24 -1
View File
@@ -6,7 +6,7 @@ use std::sync::Arc;
use bytes::Bytes;
use serde::{Deserialize, Serialize};
use super::web_carrier::{WebCarrier, WebCarriers};
use super::web_carrier::{WebCarrier, WebCarrierMethod, WebCarriers};
use super::web_debug::WebDebugConfig;
// Serialized WEB defaults remain separate from the runtime data model.
@@ -18,6 +18,10 @@ pub use fasttrack::WebDecoyFastTrackMode;
// Accepted-socket overload policy remains separate from the bulky WEB data model.
mod overload;
pub use overload::WebHttpConnectionCapacityAction;
// DNS evidence belongs to configuration generations, never request-time routing.
mod decoy_dns;
pub(crate) use decoy_dns::WebDecoyDnsSnapshot;
pub use decoy_dns::WebDecoyResolve;
/// Client-facing secret representation used to derive a WEB capability.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
@@ -55,6 +59,9 @@ pub enum WebDecoyConfig {
HttpUpstream {
/// Origin URL without a query or fragment.
upstream: String,
/// Opt-in configuration-time resolution; literal addresses never require DNS.
#[serde(default)]
resolve: WebDecoyResolve,
},
/// Serve an immutable, bounded snapshot of a local directory.
StaticDirectory {
@@ -358,6 +365,9 @@ pub struct WebTimeoutsConfig {
/// Deadline for connecting to and receiving headers from an HTTP decoy.
#[serde(default = "default_web_decoy_header_timeout_secs")]
pub decoy_header_secs: u64,
/// Maximum wait for each unique HTTP decoy hostname during preparation.
#[serde(default = "default_web_decoy_resolve_secs")]
pub decoy_resolve_secs: u64,
}
impl Default for WebTimeoutsConfig {
@@ -387,6 +397,7 @@ impl Default for WebTimeoutsConfig {
http_overload_timeout_ms: default_web_http_overload_timeout_ms(),
shutdown_secs: default_web_shutdown_secs(),
decoy_header_secs: default_web_decoy_header_timeout_secs(),
decoy_resolve_secs: default_web_decoy_resolve_secs(),
}
}
}
@@ -413,6 +424,12 @@ pub struct WebConfig {
/// Sole carrier when negotiation is disabled and final fallback when enabled.
#[serde(default)]
pub carrier: WebCarrier,
/// Page-owned HTTP method for HTTPS uplink and downlink, including retries.
#[serde(default)]
pub carrier_method: WebCarrierMethod,
/// Enables negotiated bounded HTTP uplink pipelining for new WEB sessions.
#[serde(default = "default_web_conveyor")]
pub conveyor: bool,
/// Ordered carriers considered by server-side negotiation before the fallback carrier.
#[serde(default)]
pub carriers: WebCarriers,
@@ -443,6 +460,9 @@ pub struct WebConfig {
/// Validated immutable runtime snapshot built during configuration loading.
#[serde(skip)]
pub(crate) runtime: Option<Arc<WebRuntimeConfig>>,
/// Immutable DNS evidence omitted from serialized configuration and revisions.
#[serde(skip)]
pub(crate) decoy_dns: Arc<WebDecoyDnsSnapshot>,
}
impl WebConfig {
@@ -471,6 +491,8 @@ impl Default for WebConfig {
Self {
enabled: false,
carrier: WebCarrier::default(),
carrier_method: WebCarrierMethod::default(),
conveyor: default_web_conveyor(),
carriers: WebCarriers::default(),
carrier_learning: default_web_carrier_learning(),
carrier_negotiation_aggressiveness: WebCarrierNegotiationAggressiveness::default(),
@@ -481,6 +503,7 @@ impl Default for WebConfig {
timeouts: WebTimeoutsConfig::default(),
vhosts: Vec::new(),
runtime: None,
decoy_dns: Arc::default(),
}
}
}
+22
View File
@@ -0,0 +1,22 @@
use std::collections::BTreeMap;
use std::net::SocketAddr;
use serde::{Deserialize, Serialize};
/// Configuration-time DNS policy for an HTTP decoy origin.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum WebDecoyResolve {
/// Accept IP literals only, without consulting DNS.
#[default]
Never,
/// Pin validated DNS answers for the lifetime of the prepared configuration.
Startup,
}
/// Complete resolver evidence, retained for validation against effective listeners.
#[derive(Debug, Clone, Default)]
pub(crate) struct WebDecoyDnsSnapshot {
/// Resolver-order answers keyed by canonical URL hostname and effective port.
pub(crate) origins: BTreeMap<(String, u16), Vec<SocketAddr>>,
}
+6
View File
@@ -2,6 +2,11 @@ pub(super) fn default_web_static_index() -> String {
"index.html".to_string()
}
/// Enables explicitly negotiated HTTP pipelining unless the operator disables it.
pub(super) fn default_web_conveyor() -> bool {
true
}
macro_rules! usize_default {
($name:ident, $value:expr) => {
pub(super) fn $name() -> usize {
@@ -110,3 +115,4 @@ u64_default!(default_web_http_idle_secs, 75);
u64_default!(default_web_http_overload_timeout_ms, 250);
u64_default!(default_web_shutdown_secs, 15);
u64_default!(default_web_decoy_header_timeout_secs, 30);
u64_default!(default_web_decoy_resolve_secs, 5);
+21
View File
@@ -3,6 +3,27 @@ use serde::{Deserialize, Serialize};
/// Minimum restart-owned entries required for one complete learning sample.
pub(crate) const WEB_CARRIER_LEARNING_MIN_ENTRIES: usize = 3;
/// HTTP method emitted by newly rendered HTTPS carrier bridges.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum WebCarrierMethod {
/// Preserve the original bridge request method.
#[default]
Post,
/// Allow idempotent-method connection reuse in Linux WebKitGTK.
Put,
}
impl WebCarrierMethod {
/// Returns the HTTP token embedded in the page-owned carrier policy.
pub(crate) const fn as_str(self) -> &'static str {
match self {
Self::Post => "POST",
Self::Put => "PUT",
}
}
}
/// Carrier selected for one newly issued WEB relay session.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
+56 -21
View File
@@ -26,15 +26,21 @@ const RETRY_DELAYS: [Duration; 6] = [
Duration::from_secs(30),
];
/// Reports whether a generation's desired firewall policy was confirmed.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) enum ReconcileOutcome {
/// The applied policy matches the accepted generation's desired policy.
Applied,
/// The attempt failed without confirming the desired policy.
Failed,
}
/// Associates a reconciliation result with its accepted runtime generation.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) struct ReconcileStatus {
/// Runtime generation whose desired policy was attempted.
pub(super) generation: u64,
/// Result of the latest completed attempt for this generation.
pub(super) outcome: ReconcileOutcome,
}
@@ -51,25 +57,14 @@ pub(crate) struct FirewallAuthority {
}
impl FirewallAuthority {
/// Starts the single process-owned firewall reconciler.
pub(crate) fn spawn(control_plane: &ProcessControlPlane) -> Result<Self, String> {
let (desired_tx, desired_rx) = watch::channel(None);
let (status_tx, status_rx) = watch::channel(None);
let terminal = CancellationToken::new();
let closed = Arc::new(AtomicBool::new(false));
let completed_flag = Arc::new(AtomicBool::new(false));
let cleanup_succeeded = Arc::new(AtomicBool::new(false));
let completed = Arc::new(Notify::new());
let actor = FirewallReconciler::new(
SystemCommandRunner,
desired_rx,
status_tx,
terminal.clone(),
Arc::clone(&closed),
Arc::clone(&completed_flag),
Arc::clone(&cleanup_succeeded),
Arc::clone(&completed),
);
/// Starts reconciliation only when enabled and CAP_NET_ADMIN is available.
pub(crate) fn spawn(
control_plane: &ProcessControlPlane,
config: &ProxyConfig,
) -> Result<Option<Self>, String> {
let Some((authority, actor)) = Self::prepare(config, SystemCommandRunner) else {
return Ok(None);
};
control_plane
.spawn_cooperative(move |process_cancellation| async move {
actor.run(process_cancellation).await;
@@ -78,7 +73,37 @@ impl FirewallAuthority {
"process control-plane admission closed before conntrack firewall startup"
.to_string()
})?;
Ok(Self {
Ok(Some(authority))
}
// Admission precedes allocation so an opted-out process never owns firewall cleanup.
// Keeping construction separate permits fake runners without changing async Send bounds.
fn prepare<R>(config: &ProxyConfig, runner: R) -> Option<(Self, FirewallReconciler<R>)>
where
R: FirewallCommandRunner + 'static,
{
if !config.server.conntrack_control.inline_conntrack_control || !runner.has_cap_net_admin()
{
return None;
}
let (desired_tx, desired_rx) = watch::channel(None);
let (status_tx, status_rx) = watch::channel(None);
let terminal = CancellationToken::new();
let closed = Arc::new(AtomicBool::new(false));
let completed_flag = Arc::new(AtomicBool::new(false));
let cleanup_succeeded = Arc::new(AtomicBool::new(false));
let completed = Arc::new(Notify::new());
let actor = FirewallReconciler::new(
runner,
desired_rx,
status_tx,
terminal.clone(),
Arc::clone(&closed),
Arc::clone(&completed_flag),
Arc::clone(&cleanup_succeeded),
Arc::clone(&completed),
);
let authority = Self {
desired_tx,
status_rx,
terminal,
@@ -86,7 +111,8 @@ impl FirewallAuthority {
completed_flag,
cleanup_succeeded,
completed,
})
};
Some((authority, actor))
}
/// Publishes policy only after its runtime generation becomes active.
@@ -169,6 +195,7 @@ impl Drop for CompletionGuard {
}
}
/// Serializes confirmed-state transitions and retains cleanup ownership until shutdown.
pub(super) struct FirewallReconciler<R> {
runner: R,
desired_rx: watch::Receiver<Option<DesiredState>>,
@@ -186,6 +213,7 @@ impl<R> FirewallReconciler<R>
where
R: FirewallCommandRunner + 'static,
{
/// Constructs an admitted actor with unknown state pending startup recovery.
pub(super) fn new(
runner: R,
desired_rx: watch::Receiver<Option<DesiredState>>,
@@ -214,6 +242,7 @@ where
}
}
/// Reconciles accepted generations and completes bounded cleanup on cancellation.
pub(super) async fn run(mut self, process_cancellation: CancellationToken) {
let mut current = None;
let mut retry_index = 0usize;
@@ -337,6 +366,7 @@ where
let _completion = &self.completion;
}
/// Accepts fenced publications without replacing telemetry with stale desired state.
pub(super) fn take_latest_desired(&mut self) -> Option<DesiredState> {
let next = self.desired_rx.borrow_and_update().clone()?;
if next.generation < self.last_generation {
@@ -369,3 +399,8 @@ where
Some(next)
}
}
// Exercises startup admission and cleanup ownership with fake helper processes.
#[cfg(test)]
#[path = "tests/startup_admission.rs"]
mod startup_admission_tests;
@@ -0,0 +1,267 @@
use std::sync::Mutex;
use std::sync::atomic::AtomicUsize;
use crate::config::{ConntrackBackend, ConntrackMode};
use super::super::command::{CommandErrorKind, CommandSpec, classify_command_error};
use super::super::transaction::reconcile_once;
use super::*;
const NETLINK_ERROR: &str =
"src/mnl.c:68: Unable to initialize Netlink socket: Address family not supported by protocol";
const PERMISSION_ERROR: &str = "iptables v1.8.11 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)";
#[derive(Clone)]
struct StartupRunner {
has_cap_net_admin: bool,
cap_probes: Arc<AtomicUsize>,
calls: Arc<Mutex<Vec<CommandSpec>>>,
failure: Option<&'static str>,
}
impl StartupRunner {
fn new(has_cap_net_admin: bool, failure: Option<&'static str>) -> Self {
Self {
has_cap_net_admin,
cap_probes: Arc::new(AtomicUsize::new(0)),
calls: Arc::new(Mutex::new(Vec::new())),
failure,
}
}
fn calls(&self) -> Vec<CommandSpec> {
self.calls.lock().unwrap().clone()
}
}
impl FirewallCommandRunner for StartupRunner {
fn available(&self, _binary: &str) -> bool {
true
}
fn has_cap_net_admin(&self) -> bool {
self.cap_probes.fetch_add(1, Ordering::Relaxed);
self.has_cap_net_admin
}
async fn run(&self, spec: CommandSpec) -> Result<(), CommandError> {
self.calls.lock().unwrap().push(spec.clone());
if let Some(message) = self.failure {
return Err(CommandError {
kind: classify_command_error(spec.binary, &spec.args, message),
message: message.to_string(),
});
}
if (spec.binary == "nft" && spec.args.first().map(String::as_str) == Some("delete"))
|| (matches!(spec.binary, "iptables" | "ip6tables")
&& matches!(
spec.args.get(2).map(String::as_str),
Some("-C" | "-D" | "-F" | "-X")
))
{
return Err(CommandError {
kind: CommandErrorKind::NotFound,
message: "injected absent owned object".to_string(),
});
}
Ok(())
}
}
fn start_with_runner(
config: &ProxyConfig,
runner: StartupRunner,
scope: &ProcessControlPlane,
) -> Option<FirewallAuthority> {
let (authority, actor) = FirewallAuthority::prepare(config, runner)?;
assert!(
scope
.spawn_cooperative(move |cancellation| actor.run(cancellation))
.is_ok()
);
Some(authority)
}
#[tokio::test(start_paused = true)]
async fn disabled_startup_never_acquires_cleanup_ownership() {
for explicit in [false, true] {
for has_cap in [true, false] {
for backend in [
ConntrackBackend::Auto,
ConntrackBackend::Nftables,
ConntrackBackend::Iptables,
] {
for mode in [
ConntrackMode::Tracked,
ConntrackMode::Notrack,
ConntrackMode::Hybrid,
] {
for message in [NETLINK_ERROR, PERMISSION_ERROR] {
let mut config = ProxyConfig::default();
if explicit {
config.server.conntrack_control.inline_conntrack_control = false;
config
.server
.conntrack_control
.inline_conntrack_control_explicit = true;
}
config.server.conntrack_control.backend = backend;
config.server.conntrack_control.mode = mode;
let runner = StartupRunner::new(has_cap, Some(message));
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope);
assert!(
authority.is_none(),
"{explicit} {has_cap} {backend:?} {mode:?}"
);
assert_eq!(runner.cap_probes.load(Ordering::Relaxed), 0);
tokio::time::advance(Duration::from_secs(90)).await;
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(runner.calls().is_empty());
}
}
}
}
}
}
#[tokio::test(start_paused = true)]
async fn enabled_startup_without_capability_creates_no_firewall_work() {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
for mode in [ConntrackMode::Tracked, ConntrackMode::Notrack] {
config.server.conntrack_control.mode = mode;
let runner = StartupRunner::new(false, Some(PERMISSION_ERROR));
let scope = ProcessControlPlane::new();
assert!(start_with_runner(&config, runner.clone(), &scope).is_none());
assert_eq!(runner.cap_probes.load(Ordering::Relaxed), 1);
tokio::time::advance(Duration::from_secs(90)).await;
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(runner.calls().is_empty());
}
}
#[tokio::test(start_paused = true)]
async fn admitted_startup_preserves_recovery_idle_and_shutdown_cleanup() {
for mode in [ConntrackMode::Tracked, ConntrackMode::Notrack] {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
config.server.conntrack_control.mode = mode;
config.server.conntrack_control.backend = ConntrackBackend::Nftables;
let runner = StartupRunner::new(true, None);
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope).unwrap();
let stats = Arc::new(Stats::new());
assert!(
authority
.publish_initial(1, Arc::new(config), stats.clone())
.await
);
assert!(stats.get_conntrack_rule_apply_ok());
for binary in ["nft", "iptables", "ip6tables"] {
assert!(runner.calls().iter().any(|call| call.binary == binary));
}
let calls = runner.calls().len();
tokio::time::advance(Duration::from_secs(90)).await;
tokio::task::yield_now().await;
assert_eq!(runner.calls().len(), calls);
assert!(authority.shutdown_and_clear().await);
assert!(!stats.get_conntrack_rule_apply_ok());
assert!(runner.calls().len() > calls);
assert!(scope.shutdown(Duration::from_secs(1)).await);
}
}
#[tokio::test(start_paused = true)]
async fn admitted_process_cancellation_preserves_cleanup() {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
let runner = StartupRunner::new(true, None);
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope).unwrap();
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(authority.shutdown_and_clear().await);
assert!(!runner.calls().is_empty());
}
#[test]
fn netlink_and_permission_errors_are_not_classified_as_absence() {
for spec in [
CommandSpec::new("nft", ["delete", "table", "inet", "telemt_conntrack"]),
CommandSpec::new("nft", ["-f", "-"]),
CommandSpec::new(
"iptables",
["-t", "raw", "-D", "PREROUTING", "-j", "TELEMT_NOTRACK"],
),
CommandSpec::new("iptables", ["-t", "raw", "-F", "TELEMT_NT_A"]),
CommandSpec::new("ip6tables", ["-t", "raw", "-X", "TELEMT_NT_B"]),
CommandSpec::new("iptables-restore", ["--noflush"]),
CommandSpec::new("ip6tables-restore", ["--noflush"]),
] {
for message in [
NETLINK_ERROR,
PERMISSION_ERROR,
"iptables: Failed to initialize nft: Address family not supported by protocol",
] {
assert_eq!(
classify_command_error(spec.binary, &spec.args, message),
CommandErrorKind::Failed,
"{spec:?} {message}",
);
}
}
}
#[tokio::test(start_paused = true)]
async fn admitted_genuine_failures_remain_unknown_and_retry() {
for message in [NETLINK_ERROR, PERMISSION_ERROR] {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
config.server.conntrack_control.mode = ConntrackMode::Notrack;
config.server.conntrack_control.backend = ConntrackBackend::Nftables;
let runner = StartupRunner::new(true, Some(message));
let mut applied = AppliedState::Unknown;
let requested = DesiredState {
generation: 1,
policy: DesiredPolicy::from_config(&config),
stats: Arc::new(Stats::new()),
};
let error = reconcile_once(&runner, &runner, &mut applied, &requested)
.await
.unwrap_err();
assert!(error.message.contains(message));
assert_eq!(applied, AppliedState::Unknown);
assert!(runner.calls().iter().all(|call| call.stdin.is_none()));
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope).unwrap();
let stats = Arc::new(Stats::new());
assert!(
!authority
.publish_initial(1, Arc::new(config), stats.clone())
.await
);
assert!(!stats.get_conntrack_rule_apply_ok());
assert_eq!(stats.get_conntrack_rule_reconcile_error_total(), 1);
let mut status = authority.status_rx.clone();
assert_eq!(
status.borrow_and_update().as_ref().unwrap().outcome,
ReconcileOutcome::Failed,
);
let calls = runner.calls().len();
tokio::time::advance(Duration::from_millis(999)).await;
tokio::task::yield_now().await;
assert_eq!(runner.calls().len(), calls);
tokio::time::advance(Duration::from_millis(1)).await;
status.changed().await.unwrap();
assert_eq!(
status.borrow().as_ref().unwrap().outcome,
ReconcileOutcome::Failed,
);
assert_eq!(stats.get_conntrack_rule_reconcile_error_total(), 2);
assert!(!authority.shutdown_and_clear().await);
assert!(authority.completed_flag.load(Ordering::Acquire));
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(runner.calls().iter().all(|call| call.stdin.is_none()));
}
}
+30 -2
View File
@@ -8,13 +8,17 @@ use crate::config::ProxyConfig;
const HEALTHCHECK_RESPONSE_MAX_BYTES: u64 = 64 * 1024;
/// API health contract selected by the standalone healthcheck command.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum HealthcheckMode {
/// Require a responsive API process.
Liveness,
/// Require the active runtime to report readiness.
Ready,
}
impl HealthcheckMode {
/// Parses the supported command-line healthcheck modes.
pub(crate) fn from_cli_arg(value: &str) -> Option<Self> {
match value {
"liveness" => Some(Self::Liveness),
@@ -31,6 +35,7 @@ impl HealthcheckMode {
}
}
/// Runs the bounded API healthcheck without preparing a new proxy generation.
pub(crate) fn run(config_path: &str, mode: HealthcheckMode) -> i32 {
match run_inner(config_path, mode) {
Ok(()) => 0,
@@ -42,8 +47,9 @@ pub(crate) fn run(config_path: &str, mode: HealthcheckMode) -> i32 {
}
fn run_inner(config_path: &str, mode: HealthcheckMode) -> Result<(), String> {
let config =
ProxyConfig::load(config_path).map_err(|error| format!("config load failed: {error}"))?;
let config = ProxyConfig::parse_source(config_path)
.map_err(|error| format!("config load failed: {error}"))?
.config;
let api_cfg = &config.server.api;
if !api_cfg.enabled {
return Ok(());
@@ -229,4 +235,26 @@ mod tests {
assert!(read_response_bounded(&mut payload.as_slice()).is_err());
}
#[test]
fn decoy_dns_healthcheck_reads_source_without_resolution() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
std::fs::write(
&path,
r#"
[server.api]
enabled = false
[[web.vhosts]]
host = "proxy.example.com"
public_addr = "203.0.113.10:443"
[web.vhosts.decoy]
mode = "http_upstream"
upstream = "http://unresolvable.example.invalid:8080"
resolve = "startup"
"#,
)
.unwrap();
super::run_inner(path.to_str().unwrap(), HealthcheckMode::Liveness).unwrap();
}
}
+20 -4
View File
@@ -13,6 +13,11 @@ use super::generation::RuntimeTaskScope;
const STARTUP_FALLBACK_AFTER: Duration = Duration::from_secs(80);
const RUNTIME_FALLBACK_AFTER: Duration = Duration::from_secs(6);
// Admission regressions cover notification loss without network startup.
#[cfg(test)]
mod tests;
/// Keeps generation admission and routing synchronized with periodic ME readiness.
pub(crate) async fn configure_admission_gate(
config: &Arc<ProxyConfig>,
me_pool: Option<Arc<MePool>>,
@@ -76,11 +81,18 @@ pub(crate) async fn configure_admission_gate(
} else {
Some(Instant::now())
};
let mut config_watch_open = true;
let mut me_ready_watch_open = true;
loop {
tokio::select! {
changed = config_rx_gate.changed() => {
changed = config_rx_gate.changed(), if config_watch_open => {
if changed.is_err() {
break;
config_watch_open = false;
warn!(
watch_channel = "config",
"Admission config watch closed; continuing readiness polling with last configuration"
);
continue;
}
let cfg = config_rx_gate.borrow_and_update().clone();
admission_poll_ms = cfg.general.me_admission_poll_ms.max(1);
@@ -88,9 +100,13 @@ pub(crate) async fn configure_admission_gate(
fast_fallback_enabled = cfg.general.me2dc_fallback && cfg.general.me2dc_fast;
continue;
}
changed = me_ready_rx_gate.changed() => {
changed = me_ready_rx_gate.changed(), if me_ready_watch_open => {
if changed.is_err() {
break;
me_ready_watch_open = false;
warn!(
watch_channel = "me_ready",
"Admission ME readiness watch closed; continuing periodic polling"
);
}
}
_ = tokio::time::sleep(Duration::from_millis(admission_poll_ms)) => {}
+210
View File
@@ -0,0 +1,210 @@
use std::sync::atomic::{AtomicUsize, Ordering};
use tracing::{Event, Subscriber};
use tracing_subscriber::Layer;
use tracing_subscriber::layer::{Context, SubscriberExt};
use super::*;
use crate::transport::middle_proxy::admission_test_support::IdlePoolFixture;
const POLL_MS: u64 = 20;
struct ClosureWarnings(Arc<AtomicUsize>);
impl<S: Subscriber> Layer<S> for ClosureWarnings {
fn on_event(&self, event: &Event<'_>, _ctx: Context<'_, S>) {
struct ChannelField(bool);
impl tracing::field::Visit for ChannelField {
fn record_debug(
&mut self,
field: &tracing::field::Field,
_value: &dyn std::fmt::Debug,
) {
self.0 |= field.name() == "watch_channel";
}
}
if event.metadata().level() == &tracing::Level::WARN {
let mut visitor = ChannelField(false);
event.record(&mut visitor);
if visitor.0 {
self.0.fetch_add(1, Ordering::Relaxed);
}
}
}
}
struct GateFixture {
pool: IdlePoolFixture,
route: Arc<RouteRuntimeController>,
admission: watch::Receiver<bool>,
config_tx: Option<watch::Sender<Arc<ProxyConfig>>>,
ready_tx: Option<watch::Sender<u64>>,
config: Arc<ProxyConfig>,
scope: RuntimeTaskScope,
}
impl GateFixture {
async fn new() -> Self {
let pool = IdlePoolFixture::new(false).await;
let mut cfg = ProxyConfig::default();
cfg.general.use_middle_proxy = true;
cfg.general.me2dc_fallback = true;
cfg.general.me2dc_fast = true;
cfg.general.me_admission_poll_ms = POLL_MS;
let config = Arc::new(cfg);
let (config_tx, config_rx) = watch::channel(config.clone());
let (ready_tx, ready_rx) = watch::channel(0);
let (admission_tx, mut admission) = watch::channel(false);
let route = Arc::new(RouteRuntimeController::new(RelayRouteMode::Direct));
let scope = RuntimeTaskScope::new();
configure_admission_gate(
&config,
Some(pool.pool()),
Arc::new(RwLock::new(Some(pool.pool()))),
route.clone(),
&admission_tx,
config_rx,
ready_rx,
scope.clone(),
)
.await;
drop(admission_tx);
admission.borrow_and_update();
tokio::task::yield_now().await;
Self {
pool,
route,
admission,
config_tx: Some(config_tx),
ready_tx: Some(ready_tx),
config,
scope,
}
}
async fn poll(&self) {
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_millis(POLL_MS)).await;
tokio::task::yield_now().await;
}
async fn expect_route(&self, mode: RelayRouteMode) {
let mut rx = self.route.subscribe();
tokio::time::timeout(Duration::from_secs(1), async {
loop {
if rx.borrow_and_update().mode == mode {
return;
}
rx.changed().await.unwrap();
}
})
.await
.expect("the generation must keep sampling readiness without client traffic");
}
}
async fn recovery_after_watch_closure(close_config: bool, close_ready: bool) {
let warnings = Arc::new(AtomicUsize::new(0));
let subscriber = tracing_subscriber::registry().with(ClosureWarnings(warnings.clone()));
let _default = tracing::subscriber::set_default(subscriber);
let mut fixture = GateFixture::new().await;
assert!(*fixture.admission.borrow());
if close_config {
fixture.config_tx.take();
}
if close_ready {
fixture.ready_tx.take();
}
for _ in 0..4 {
tokio::task::yield_now().await;
}
let expected_warnings = usize::from(close_config) + usize::from(close_ready);
fixture.pool.set_ready(true);
for _ in 0..4 {
tokio::task::yield_now().await;
}
assert_eq!(fixture.route.snapshot().mode, RelayRouteMode::Direct);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
fixture.pool.set_ready(false);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Direct).await;
fixture.pool.set_ready(true);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
for _ in 0..5 {
fixture.poll().await;
}
assert_eq!(warnings.load(Ordering::Relaxed), expected_warnings);
fixture.scope.stop().await;
assert!(fixture.admission.has_changed().is_err());
}
#[tokio::test(start_paused = true)]
async fn idle_gate_recovers_after_readiness_watch_closes() {
recovery_after_watch_closure(false, true).await;
}
#[tokio::test(start_paused = true)]
async fn idle_gate_recovers_after_config_watch_closes() {
recovery_after_watch_closure(true, false).await;
}
#[tokio::test(start_paused = true)]
async fn idle_gate_recovers_after_both_watches_close_without_spinning() {
recovery_after_watch_closure(true, true).await;
}
#[tokio::test(start_paused = true)]
async fn closed_config_watch_preserves_last_effective_fallback_policy() {
let mut fixture = GateFixture::new().await;
let mut cfg = fixture.config.as_ref().clone();
cfg.general.me2dc_fallback = false;
fixture
.config_tx
.as_ref()
.unwrap()
.send_replace(Arc::new(cfg));
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
assert!(!*fixture.admission.borrow_and_update());
fixture.config_tx.take();
fixture.ready_tx.take();
fixture.poll().await;
fixture.pool.set_ready(true);
fixture.poll().await;
tokio::time::timeout(Duration::from_secs(1), fixture.admission.changed())
.await
.expect("restored readiness must open admission after notification loss")
.unwrap();
assert!(*fixture.admission.borrow_and_update());
fixture.pool.set_ready(false);
fixture.poll().await;
tokio::time::timeout(Duration::from_secs(1), fixture.admission.changed())
.await
.unwrap()
.unwrap();
assert!(!*fixture.admission.borrow_and_update());
assert_eq!(fixture.route.snapshot().mode, RelayRouteMode::Middle);
fixture.scope.stop().await;
}
#[tokio::test(start_paused = true)]
async fn generation_cancellation_stops_gate_after_notification_loss() {
let mut fixture = GateFixture::new().await;
fixture.config_tx.take();
fixture.ready_tx.take();
fixture.poll().await;
fixture.pool.set_ready(true);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
fixture.scope.stop().await;
assert!(fixture.admission.has_changed().is_err());
let state = fixture.route.snapshot();
fixture.pool.set_ready(false);
for _ in 0..3 {
fixture.poll().await;
}
assert_eq!(fixture.route.snapshot(), state);
assert!(*fixture.admission.borrow());
}
+2 -2
View File
@@ -95,8 +95,8 @@ pub(super) async fn bootstrap(
std::process::exit(1);
}
let mut config = match ProxyConfig::load(&config_path) {
Ok(c) => c,
let mut config = match ProxyConfig::load_prepared(config_path.clone()).await {
Ok(loaded) => loaded.config,
Err(e) => {
if config_path.exists() {
eprintln!("[telemt] Error: {}", e);
+10 -6
View File
@@ -375,17 +375,21 @@ pub(super) async fn run_telemt_core(
#[cfg(target_os = "linux")]
let conntrack_firewall = {
let authority = crate::conntrack_control::FirewallAuthority::spawn(&process_control_plane)
.map_err(std::io::Error::other)?;
if !authority
.publish_initial(1, runtime.config.clone(), stats.clone())
.await
let authority = crate::conntrack_control::FirewallAuthority::spawn(
&process_control_plane,
&runtime.config,
)
.map_err(std::io::Error::other)?;
if let Some(authority) = &authority
&& !authority
.publish_initial(1, runtime.config.clone(), stats.clone())
.await
{
warn!(
"Initial conntrack firewall reconciliation failed; background retries remain active"
);
}
Some(authority)
authority
};
#[cfg(not(target_os = "linux"))]
let conntrack_firewall = None::<crate::conntrack_control::FirewallAuthority>;
+1
View File
@@ -512,6 +512,7 @@ pub(crate) fn deferred_process_fields(
fn configs_equal(old: &ProxyConfig, new: &ProxyConfig) -> bool {
serde_json::to_value(old).ok() == serde_json::to_value(new).ok()
&& old.web_decoy_endpoints_equal(new)
}
#[cfg(test)]
+32 -7
View File
@@ -104,11 +104,12 @@ pub fn generate_service_file(init_system: InitSystem, opts: &ServiceOptions) ->
InitSystem::Systemd => generate_systemd_unit(opts),
InitSystem::OpenRC => generate_openrc_script(opts),
InitSystem::FreeBSDRc => generate_freebsd_rc_script(opts),
InitSystem::Unknown => generate_systemd_unit(opts), // Default to systemd format
// Preserve systemd generation when init detection is unavailable.
InitSystem::Unknown => generate_systemd_unit(opts),
}
}
/// Generates an enhanced systemd unit file.
/// Generates a hardened systemd unit with Netlink access for optional firewall helpers.
fn generate_systemd_unit(opts: &ServiceOptions) -> String {
let user_line = opts.user.map(|u| format!("User={}", u)).unwrap_or_default();
let group_line = opts
@@ -151,14 +152,14 @@ PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
MemoryDenyWriteExecute=true
LockPersonality=true
# Allow binding to privileged ports and writing to specific paths
# Allow privileged port binding, optional firewall helpers, and runtime state writes
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMIN
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMIN
ReadWritePaths=/etc/telemt /var/run /var/lib/telemt
@@ -233,9 +234,11 @@ fn generate_freebsd_rc_script(opts: &ServiceOptions) -> String {
# Add the following lines to /etc/rc.conf to enable telemt:
#
# telemt_enable="YES"
# telemt_config="/etc/telemt/config.toml" # optional
# telemt_user="telemt" # optional
# telemt_group="telemt" # optional
# The configuration path can be overridden in rc.conf.
# telemt_config="/etc/telemt/config.toml"
# The service user and group can be overridden in rc.conf.
# telemt_user="telemt"
# telemt_group="telemt"
#
. /etc/rc.subr
@@ -355,6 +358,28 @@ mod tests {
assert!(unit.contains("PIDFile="));
}
#[test]
fn systemd_unit_allows_netlink_without_removing_hardening() {
let unit = generate_systemd_unit(&ServiceOptions::default());
let families = unit
.lines()
.find_map(|line| line.strip_prefix("RestrictAddressFamilies="))
.unwrap();
assert_eq!(
families.split_whitespace().collect::<Vec<_>>(),
["AF_INET", "AF_INET6", "AF_UNIX", "AF_NETLINK"],
);
for directive in [
"NoNewPrivileges=true",
"ProtectSystem=strict",
"RestrictNamespaces=true",
"AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMIN",
"CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMIN",
] {
assert!(unit.lines().any(|line| line == directive), "{directive}");
}
}
#[test]
fn test_openrc_script_generation() {
let opts = ServiceOptions::default();
+30 -5
View File
@@ -8,6 +8,7 @@ use std::net::{IpAddr, SocketAddr};
use std::os::fd::{AsRawFd, RawFd};
#[cfg(target_os = "linux")]
use std::os::raw::c_int;
use std::sync::Arc;
use std::sync::atomic::Ordering;
use std::time::{Duration, Instant};
@@ -252,7 +253,7 @@ impl MePool {
/// Perform full ME RPC handshake on an established TCP stream.
/// Returns cipher keys/ivs and split halves; does not register writer.
pub(crate) async fn handshake_only(
&self,
self: &Arc<Self>,
stream: TcpStream,
addr: SocketAddr,
upstream_egress: Option<UpstreamEgressInfo>,
@@ -264,7 +265,6 @@ impl MePool {
let transport_peer_addr = stream.peer_addr().map_err(ProxyError::Io)?;
let peer_addr = addr;
let _ = self.maybe_detect_nat_ip(local_addr.ip()).await;
let family = if local_addr.ip().is_ipv4() {
IpFamily::V4
} else {
@@ -323,9 +323,34 @@ impl MePool {
None
};
let local_addr_nat = self.translate_our_addr_with_reflection(local_addr, reflected);
let peer_addr_nat =
SocketAddr::new(self.translate_ip_for_nat(peer_addr.ip()), peer_addr.port());
// Authoritative sources do not wait behind unrelated HTTP discovery. Capture once
// before deriving either side of the tuple so a refresh cannot mix observations.
let detected = if socks_bound_kdf_addr.is_some()
|| self.nat_runtime.nat_ip_cfg.is_some()
|| reflected.is_some()
{
self.cached_http_nat_ip()
} else {
self.maybe_detect_nat_ip(local_addr.ip()).await
};
if self.nat_runtime.nat_probe
&& local_addr.is_ipv4()
&& is_bogon(local_addr.ip())
&& socks_bound_kdf_addr.is_none()
&& self.nat_runtime.nat_ip_cfg.is_none()
&& reflected.is_none()
&& detected.is_none()
{
return Err(ProxyError::Io(std::io::Error::new(
std::io::ErrorKind::WouldBlock,
"ME NAT source observation is unavailable or expired",
)));
}
let local_addr_nat = self.translate_our_addr_with_nat(local_addr, reflected, detected);
let peer_addr_nat = SocketAddr::new(
Self::translate_ip_with_nat(peer_addr.ip(), self.nat_runtime.nat_ip_cfg.or(detected)),
peer_addr.port(),
);
let client_addr_for_kdf = socks_bound_kdf_addr.unwrap_or(local_addr_nat);
if let Some(upstream_info) = upstream_egress {
record_upstream_bnd_status(
+4
View File
@@ -20,6 +20,8 @@ use super::pool::{
MeFamilyRuntimeState, RefillTargetKey, WriterContour, WriterOpenIntent, WriterRole,
};
use super::pool_writer::WriterReplacementPurpose;
// Bounded cross-family coverage transfers preserve the donor's current base floor.
mod coverage;
// Bounds reconnect jitter to at most half of the current backoff.
const JITTER_FRAC_NUM: u64 = 2;
@@ -110,6 +112,8 @@ impl Drop for ScheduledReconnects<'_> {
// Periodic family health monitor orchestration.
mod monitor;
// Shared-family fair scheduling and independently tracked writer-operation quanta.
mod scheduler;
// Draining-writer deadline enforcement.
mod drain;
// Per-family reconnect scheduling and health state.
@@ -0,0 +1,109 @@
use super::*;
use std::sync::atomic::Ordering;
/// Selects globally because the Active cap is shared across DCs and address families.
pub(super) async fn transfer_coverage(
pool: &Arc<MePool>,
rng: &Arc<SecureRandom>,
dc: i32,
endpoint: SocketAddr,
required: usize,
) -> Option<bool> {
let capacity = pool
.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
dc,
endpoint,
)
.await?;
drop(capacity);
let authority = pool.floor_authority();
let endpoints = pool.endpoint_snapshot.load_full();
let generation = pool.current_generation();
let writers = pool.writers.read().await;
let activity = pool.registry.writer_activity_snapshot().await;
let mut counts = HashMap::new();
for writer in writers.iter() {
if !writer.draining.load(Ordering::Acquire)
&& writer.generation == generation
&& WriterContour::from_u8(writer.contour.load(Ordering::Acquire))
== WriterContour::Active
&& endpoints.contains_dc_endpoint(writer.writer_dc, writer.addr)
{
*counts
.entry((writer.writer_dc, writer.addr.is_ipv4()))
.or_insert(0usize) += 1;
}
}
let mut candidates = Vec::new();
for writer in writers.iter() {
if writer.draining.load(Ordering::Acquire)
|| WriterContour::from_u8(writer.contour.load(Ordering::Acquire))
!= WriterContour::Active
{
continue;
}
let role = WriterRole::from_writer(writer);
let obsolete = writer.generation != generation
|| !endpoints.contains_dc_endpoint(writer.writer_dc, writer.addr);
let protected = pool.required_writers_for_dc_with_floor_mode(
endpoints
.endpoints_for_dc_family(writer.writer_dc, role.family)
.len(),
false,
);
if !obsolete
&& counts
.get(&(writer.writer_dc, writer.addr.is_ipv4()))
.copied()
.unwrap_or(0)
<= protected
{
continue;
}
let busy = activity
.bound_clients_by_writer
.get(&writer.id)
.copied()
.unwrap_or(0)
> 0;
candidates.push((busy, !obsolete, writer.created_at, writer.id, role));
}
drop(writers);
candidates
.sort_unstable_by_key(|candidate| (candidate.0, candidate.1, candidate.2, candidate.3));
for (busy, _, _, id, role) in candidates {
let reservation = if busy {
pool.registry
.try_reserve_writer_replacement_preserving_clients(id)
.await
} else {
pool.registry.try_reserve_writer_replacement(id).await
};
let Some(mut reservation) = reservation else {
continue;
};
return Some(matches!(
tokio::time::timeout(
pool.reconnect_runtime.me_one_timeout,
pool.replace_writer_with_generation_contour_for_dc(
endpoint,
rng.as_ref(),
generation,
WriterContour::Active,
dc,
role,
WriterReplacementPurpose::CoverageTransfer {
authority,
receiver_floor: required
},
&mut reservation,
),
)
.await,
Ok(Ok(()))
));
}
None
}
+50 -432
View File
@@ -1,449 +1,67 @@
use super::scheduler::HealthObservation;
use super::*;
pub(super) async fn check_family(
family: IpFamily,
pool: &Arc<MePool>,
rng: &Arc<SecureRandom>,
backoff: &mut HashMap<(i32, IpFamily), u64>,
next_attempt: &mut HashMap<(i32, IpFamily), Instant>,
inflight: &mut HashMap<(i32, IpFamily), usize>,
outage_backoff: &mut HashMap<(i32, IpFamily), u64>,
outage_next_attempt: &mut HashMap<(i32, IpFamily), Instant>,
single_endpoint_outage: &mut HashSet<(i32, IpFamily)>,
shadow_rotate_deadline: &mut HashMap<(i32, IpFamily), Instant>,
idle_refresh_next_attempt: &mut HashMap<(i32, IpFamily), Instant>,
floor_warn_next_allowed: &mut HashMap<(i32, IpFamily), Instant>,
) -> bool {
let enabled = match family {
IpFamily::V4 => pool.decision.ipv4_me,
IpFamily::V6 => pool.decision.ipv6_me,
/// Takes a control-plane observation without awaiting any network operation.
pub(super) async fn check_family(family: IpFamily, pool: &Arc<MePool>) -> HealthObservation {
let authority = pool.floor_authority();
let generation = pool.current_generation();
let snapshot = pool.endpoint_snapshot.load_full();
let enabled = if family == IpFamily::V4 {
pool.decision.ipv4_me
} else {
pool.decision.ipv6_me
};
if !enabled {
return false;
}
let mut family_degraded = false;
let mut dc_endpoints = HashMap::<i32, Vec<SocketAddr>>::new();
let endpoint_snapshot = pool.endpoint_snapshot.load();
let endpoint_revision = endpoint_snapshot.revision;
let map_guard = match family {
IpFamily::V4 => &endpoint_snapshot.map_v4,
IpFamily::V6 => &endpoint_snapshot.map_v6,
let source = if family == IpFamily::V4 {
&snapshot.map_v4
} else {
&snapshot.map_v6
};
for (dc, addrs) in map_guard.iter() {
let entry = dc_endpoints.entry(*dc).or_default();
for (ip, port) in addrs.iter().copied() {
entry.push(SocketAddr::new(ip, port));
let mut endpoints = HashMap::new();
if enabled {
for (dc, addresses) in source {
let mut addresses: Vec<_> = addresses
.iter()
.map(|(ip, port)| SocketAddr::new(*ip, *port))
.collect();
addresses.sort_unstable();
addresses.dedup();
endpoints.insert(*dc, addresses);
}
}
drop(endpoint_snapshot);
for endpoints in dc_endpoints.values_mut() {
endpoints.sort_unstable();
endpoints.dedup();
}
let reconnect_budget = health_reconnect_budget(pool, dc_endpoints.len());
let reconnect_sem = Arc::new(Semaphore::new(reconnect_budget));
if pool.floor_mode() == MeFloorMode::Static {}
let mut live_addr_counts = HashMap::<(i32, SocketAddr), usize>::new();
let mut live_writer_ids_by_addr = HashMap::<(i32, SocketAddr), Vec<u64>>::new();
let active_generation = pool.current_generation();
for writer in pool
.writers
.read()
.await
.iter()
.filter(|w| !w.draining.load(std::sync::atomic::Ordering::Relaxed))
{
if !matches!(
crate::transport::middle_proxy::pool::WriterContour::from_u8(
writer.contour.load(std::sync::atomic::Ordering::Relaxed),
),
crate::transport::middle_proxy::pool::WriterContour::Active
) {
continue;
}
if writer.generation != active_generation {
continue;
}
if !dc_endpoints
.get(&writer.writer_dc)
.is_some_and(|endpoints| endpoints.contains(&writer.addr))
let mut counts = HashMap::new();
let mut ids = HashMap::<_, Vec<_>>::new();
for writer in pool.writers.read().await.iter() {
if !writer.draining.load(std::sync::atomic::Ordering::Acquire)
&& writer.generation == generation
&& WriterContour::from_u8(writer.contour.load(std::sync::atomic::Ordering::Acquire))
== WriterContour::Active
&& endpoints
.get(&writer.writer_dc)
.is_some_and(|addresses| addresses.contains(&writer.addr))
{
continue;
*counts
.entry((writer.writer_dc, writer.addr))
.or_insert(0usize) += 1;
ids.entry((writer.writer_dc, writer.addr))
.or_default()
.push(writer.id);
}
let key = (writer.writer_dc, writer.addr);
*live_addr_counts.entry(key).or_insert(0) += 1;
live_writer_ids_by_addr
.entry(key)
.or_default()
.push(writer.id);
}
let writer_idle_since = pool.registry.writer_idle_since_snapshot().await;
let bound_clients_by_writer = pool
let idle = pool.registry.writer_idle_since_snapshot().await;
let bound = pool
.registry
.writer_activity_snapshot()
.await
.bound_clients_by_writer;
let floor_plan = build_family_floor_plan(
pool,
family,
&dc_endpoints,
&live_addr_counts,
&live_writer_ids_by_addr,
&bound_clients_by_writer,
)
.await;
pool.set_adaptive_floor_runtime_caps(
floor_plan.active_cap_configured_total,
floor_plan.active_cap_effective_total,
floor_plan.warm_cap_configured_total,
floor_plan.warm_cap_effective_total,
floor_plan.target_writers_total,
floor_plan.active_writers_current,
floor_plan.warm_writers_current,
);
let floor_targets_by_dc = Arc::new(
floor_plan
.by_dc
.iter()
.map(|(dc, entry)| (*dc, entry.target_required))
.collect::<HashMap<_, _>>(),
);
let live_writer_ids_by_addr = Arc::new(live_writer_ids_by_addr);
let writer_idle_since = Arc::new(writer_idle_since);
let bound_clients_by_writer = Arc::new(bound_clients_by_writer);
let mut reconnect_set = JoinSet::<FamilyReconnectOutcome>::new();
let mut scheduled_reconnects = ScheduledReconnects {
inflight,
keys: Vec::new(),
};
for (dc, endpoints) in dc_endpoints {
if endpoints.is_empty() {
continue;
}
let key = (dc, family);
let required = floor_plan
.by_dc
.get(&dc)
.map(|entry| entry.target_required)
.unwrap_or_else(|| {
pool.required_writers_for_dc_with_floor_mode(endpoints.len(), false)
});
let alive = endpoints
.iter()
.map(|addr| *live_addr_counts.get(&(dc, *addr)).unwrap_or(&0))
.sum::<usize>();
if endpoints.len() == 1 && pool.single_endpoint_outage_mode_enabled() && alive == 0 {
family_degraded = true;
if single_endpoint_outage.insert(key) {
pool.stats.increment_me_single_endpoint_outage_enter_total();
warn!(
dc = %dc,
?family,
required,
endpoint_count = endpoints.len(),
"Single-endpoint DC outage detected"
);
}
recover_single_endpoint_outage(
pool,
rng,
key,
endpoints[0],
required,
outage_backoff,
outage_next_attempt,
&reconnect_sem,
)
.await;
continue;
}
if single_endpoint_outage.remove(&key) {
pool.stats.increment_me_single_endpoint_outage_exit_total();
outage_backoff.remove(&key);
outage_next_attempt.remove(&key);
shadow_rotate_deadline.remove(&key);
idle_refresh_next_attempt.remove(&key);
info!(
dc = %dc,
?family,
alive,
required,
endpoint_count = endpoints.len(),
"Single-endpoint DC outage recovered"
);
}
maybe_refresh_idle_writer_for_dc(
pool,
rng,
key,
dc,
family,
&endpoints,
alive,
required,
live_writer_ids_by_addr.as_ref(),
writer_idle_since.as_ref(),
bound_clients_by_writer.as_ref(),
idle_refresh_next_attempt,
)
.await;
if alive >= required {
maybe_rotate_single_endpoint_shadow(
pool,
rng,
key,
dc,
family,
&endpoints,
alive,
required,
live_writer_ids_by_addr.as_ref(),
bound_clients_by_writer.as_ref(),
shadow_rotate_deadline,
)
.await;
continue;
}
let missing = required - alive;
family_degraded = true;
let now = Instant::now();
if reconnect_sem.available_permits() == 0 {
let base_ms = pool.reconnect_runtime.me_reconnect_backoff_base.as_millis() as u64;
let next_ms = (*backoff.get(&key).unwrap_or(&base_ms)).max(base_ms);
let jitter = next_ms / JITTER_FRAC_NUM;
let wait = Duration::from_millis(next_ms)
+ Duration::from_millis(rand::rng().random_range(0..=jitter.max(1)));
next_attempt.insert(key, now + wait);
debug!(
dc = %dc,
?family,
alive,
required,
endpoint_count = endpoints.len(),
reconnect_budget,
"Skipping reconnect due to per-tick health reconnect budget"
);
continue;
}
if let Some(ts) = next_attempt.get(&key)
&& now < *ts
{
continue;
}
let max_concurrent = pool
.reconnect_runtime
.me_reconnect_max_concurrent_per_dc
.max(1) as usize;
if scheduled_reconnects.current(&key) >= max_concurrent {
continue;
}
if pool
.has_refill_inflight_for_target(RefillTargetKey {
dc,
family,
generation: pool.current_generation(),
endpoint_revision,
contour: WriterContour::Active,
})
.await
{
debug!(
dc = %dc,
?family,
alive,
required,
endpoint_count = endpoints.len(),
"Skipping health reconnect: immediate refill is already in flight for this DC group"
);
continue;
}
scheduled_reconnects.reserve(key);
let pool_for_reconnect = pool.clone();
let rng_for_reconnect = rng.clone();
let reconnect_sem_for_dc = reconnect_sem.clone();
let endpoints_for_dc = endpoints.clone();
let live_writer_ids_by_addr_for_dc = live_writer_ids_by_addr.clone();
let writer_idle_since_for_dc = writer_idle_since.clone();
let bound_clients_by_writer_for_dc = bound_clients_by_writer.clone();
let floor_targets_by_dc_for_reconnect = floor_targets_by_dc.clone();
let active_cap_effective_total = floor_plan.active_cap_effective_total;
reconnect_set.spawn(async move {
let mut restored = 0usize;
for _ in 0..missing {
let Ok(reconnect_permit) = reconnect_sem_for_dc.clone().try_acquire_owned() else {
break;
};
let base_req = pool_for_reconnect
.required_writers_for_dc_with_floor_mode(endpoints_for_dc.len(), false);
if pool_for_reconnect.active_contour_writer_count_total().await
>= active_cap_effective_total
{
let swapped = maybe_swap_idle_writer_for_cap(
&pool_for_reconnect,
&rng_for_reconnect,
dc,
family,
&endpoints_for_dc,
required,
floor_targets_by_dc_for_reconnect.as_ref(),
live_writer_ids_by_addr_for_dc.as_ref(),
writer_idle_since_for_dc.as_ref(),
bound_clients_by_writer_for_dc.as_ref(),
)
.await;
if swapped {
pool_for_reconnect
.stats
.increment_me_floor_swap_idle_total();
restored += 1;
continue;
}
if alive + restored >= base_req {
pool_for_reconnect
.stats
.increment_me_floor_cap_block_total();
pool_for_reconnect
.stats
.increment_me_floor_swap_idle_failed_total();
debug!(
dc = %dc,
?family,
alive,
required,
active_cap_effective_total,
"Adaptive floor cap reached, reconnect attempt blocked"
);
break;
}
}
pool_for_reconnect.stats.increment_me_reconnect_attempt();
let intent = if alive + restored < base_req {
WriterOpenIntent::Coverage
} else {
WriterOpenIntent::Normal
};
let res = tokio::time::timeout(
pool_for_reconnect.reconnect_runtime.me_one_timeout,
pool_for_reconnect.connect_endpoints_round_robin_with_generation_contour(
dc,
&endpoints_for_dc,
rng_for_reconnect.as_ref(),
pool_for_reconnect.current_generation(),
WriterContour::Active,
intent,
),
)
.await;
match res {
Ok(true) => {
restored += 1;
pool_for_reconnect.stats.increment_me_reconnect_success();
}
Ok(false) => {
debug!(dc = %dc, ?family, "ME round-robin reconnect failed")
}
Err(_) => {
debug!(dc = %dc, ?family, "ME reconnect timed out");
}
}
drop(reconnect_permit);
}
FamilyReconnectOutcome {
key,
dc,
family,
required,
endpoint_count: endpoints_for_dc.len(),
}
});
let floor = build_family_floor_plan(pool, family, &endpoints, &counts, &ids, &bound).await;
HealthObservation {
authority,
generation,
floor,
ids,
idle,
bound,
}
while let Some(joined) = reconnect_set.join_next().await {
let outcome = match joined {
Ok(outcome) => outcome,
Err(join_error) => {
debug!(error = %join_error, "Health reconnect task failed");
continue;
}
};
let now = Instant::now();
let now_alive = live_active_writers_for_dc_family(pool, outcome.dc, outcome.family).await;
if now_alive >= outcome.required {
info!(
dc = %outcome.dc,
family = ?outcome.family,
alive = now_alive,
required = outcome.required,
endpoint_count = outcome.endpoint_count,
"ME writer floor restored for DC"
);
backoff.insert(
outcome.key,
pool.reconnect_runtime.me_reconnect_backoff_base.as_millis() as u64,
);
let jitter = pool.reconnect_runtime.me_reconnect_backoff_base.as_millis() as u64
/ JITTER_FRAC_NUM;
let wait = pool.reconnect_runtime.me_reconnect_backoff_base
+ Duration::from_millis(rand::rng().random_range(0..=jitter.max(1)));
next_attempt.insert(outcome.key, now + wait);
} else {
let curr = *backoff
.get(&outcome.key)
.unwrap_or(&(pool.reconnect_runtime.me_reconnect_backoff_base.as_millis() as u64));
let next_ms = (curr.saturating_mul(2))
.min(pool.reconnect_runtime.me_reconnect_backoff_cap.as_millis() as u64);
backoff.insert(outcome.key, next_ms);
let jitter = next_ms / JITTER_FRAC_NUM;
let wait = Duration::from_millis(next_ms)
+ Duration::from_millis(rand::rng().random_range(0..=jitter.max(1)));
next_attempt.insert(outcome.key, now + wait);
if pool.is_runtime_ready() {
let warn_cooldown = pool.warn_rate_limit_duration();
if should_emit_rate_limited_warn(
floor_warn_next_allowed,
outcome.key,
now,
warn_cooldown,
) {
warn!(
dc = %outcome.dc,
family = ?outcome.family,
alive = now_alive,
required = outcome.required,
endpoint_count = outcome.endpoint_count,
backoff_ms = next_ms,
"DC writer floor is below required level, scheduled reconnect"
);
}
} else {
info!(
dc = %outcome.dc,
family = ?outcome.family,
alive = now_alive,
required = outcome.required,
endpoint_count = outcome.endpoint_count,
backoff_ms = next_ms,
"DC writer floor is below required level during startup, scheduled reconnect"
);
}
}
}
family_degraded
}
pub(super) fn health_reconnect_budget(pool: &Arc<MePool>, dc_groups: usize) -> usize {
+287 -53
View File
@@ -1,59 +1,293 @@
use super::scheduler::{
HealthJob, HealthObservation, HealthScheduler, JobKind, JobOutcome, idle_refresh_ready, run_job,
};
use super::*;
/// Observes family health independently of bounded, lifecycle-owned network operations.
pub async fn me_health_monitor(pool: Arc<MePool>, rng: Arc<SecureRandom>, _min_connections: usize) {
let mut backoff: HashMap<(i32, IpFamily), u64> = HashMap::new();
let mut next_attempt: HashMap<(i32, IpFamily), Instant> = HashMap::new();
let mut inflight: HashMap<(i32, IpFamily), usize> = HashMap::new();
let mut outage_backoff: HashMap<(i32, IpFamily), u64> = HashMap::new();
let mut outage_next_attempt: HashMap<(i32, IpFamily), Instant> = HashMap::new();
let mut single_endpoint_outage: HashSet<(i32, IpFamily)> = HashSet::new();
let mut shadow_rotate_deadline: HashMap<(i32, IpFamily), Instant> = HashMap::new();
let mut idle_refresh_next_attempt: HashMap<(i32, IpFamily), Instant> = HashMap::new();
let mut floor_warn_next_allowed: HashMap<(i32, IpFamily), Instant> = HashMap::new();
let mut drain_warn_next_allowed: HashMap<u64, Instant> = HashMap::new();
let mut degraded_interval = true;
let mut scheduler = HealthScheduler::default();
let mut tasks = JoinSet::new();
let mut observations = HashMap::<IpFamily, Arc<HealthObservation>>::new();
let mut drain_warn_next_allowed = HashMap::new();
let mut observe = true;
let mut health_tick = true;
let mut next_tick = tokio::time::Instant::now();
loop {
let interval = if degraded_interval {
pool.health_interval_unhealthy()
} else {
pool.health_interval_healthy()
};
tokio::time::sleep(interval).await;
pool.prune_closed_writers().await;
pool.sweep_endpoint_quarantine().await;
reap_draining_writers(&pool, &mut drain_warn_next_allowed).await;
let v4_degraded = check_family(
IpFamily::V4,
&pool,
&rng,
&mut backoff,
&mut next_attempt,
&mut inflight,
&mut outage_backoff,
&mut outage_next_attempt,
&mut single_endpoint_outage,
&mut shadow_rotate_deadline,
&mut idle_refresh_next_attempt,
&mut floor_warn_next_allowed,
)
.await;
let v6_degraded = check_family(
IpFamily::V6,
&pool,
&rng,
&mut backoff,
&mut next_attempt,
&mut inflight,
&mut outage_backoff,
&mut outage_next_attempt,
&mut single_endpoint_outage,
&mut shadow_rotate_deadline,
&mut idle_refresh_next_attempt,
&mut floor_warn_next_allowed,
)
.await;
update_family_runtime_state(&pool, IpFamily::V4, v4_degraded);
update_family_runtime_state(&pool, IpFamily::V6, v6_degraded);
degraded_interval = v4_degraded || v6_degraded;
if observe {
if health_tick {
pool.prune_closed_writers().await;
pool.sweep_endpoint_quarantine().await;
reap_draining_writers(&pool, &mut drain_warn_next_allowed).await;
}
let mut target_total = 0usize;
let mut effective_cap = 0usize;
let mut cap_observation = None;
let mut degraded = false;
for family in [IpFamily::V4, IpFamily::V6] {
let observation = Arc::new(check_family(family, &pool).await);
let family_degraded = observation
.floor
.by_dc
.values()
.any(|entry| entry.alive < entry.target_required);
if health_tick {
update_family_runtime_state(&pool, family, family_degraded);
}
degraded |= family_degraded;
target_total += observation
.floor
.by_dc
.values()
.map(|entry| entry.target_required)
.sum::<usize>();
effective_cap = effective_cap.max(observation.floor.active_cap_effective_total);
if cap_observation.is_none() {
cap_observation = Some(observation.clone());
}
observations.insert(family, observation);
}
if let Some(observation) = cap_observation
&& observations
.values()
.all(|other| other.authority == observation.authority)
{
let floor = &observation.floor;
pool.set_adaptive_floor_runtime_caps(
observation.authority,
floor.active_cap_configured_total,
effective_cap.max(target_total),
floor.warm_cap_configured_total,
floor.warm_cap_effective_total,
target_total,
floor.active_writers_current,
floor.warm_writers_current,
);
}
scheduler.observe(&observations);
for (key, state) in &mut scheduler.groups {
if health_tick {
state.deferred = false;
}
let Some(entry) = observations
.get(&key.1)
.and_then(|o| o.floor.by_dc.get(&key.0))
else {
continue;
};
let outage = entry.endpoints.len() == 1
&& entry.alive == 0
&& pool.single_endpoint_outage_mode_enabled();
if outage != state.outage {
if outage {
pool.stats.increment_me_single_endpoint_outage_enter_total();
warn!(dc = key.0, family = ?key.1, required = entry.target_required,
endpoint_count = entry.endpoints.len(), "Single-endpoint DC outage detected");
} else {
pool.stats.increment_me_single_endpoint_outage_exit_total();
info!(dc = key.0, family = ?key.1, alive = entry.alive, required = entry.target_required,
endpoint_count = entry.endpoints.len(), "Single-endpoint DC outage recovered");
}
state.outage = outage;
// Outage retries and ordinary missing-floor rounds use different bounds.
// A successful first writer must not inherit an outage failure delay.
state.backoff_ms = 0;
state.due = None;
state.round_left = 0;
state.round_active = false;
}
if state.round_active
&& (state.round_left == 0 || entry.alive >= entry.target_required)
{
let (base, cap) = if state.outage {
pool.single_endpoint_outage_backoff_bounds_ms()
} else {
(
pool.reconnect_runtime.me_reconnect_backoff_base.as_millis() as u64,
pool.reconnect_runtime.me_reconnect_backoff_cap.as_millis() as u64,
)
};
let restored = entry.alive >= entry.target_required;
state.backoff_ms = if restored {
base
} else {
state.backoff_ms.max(base).saturating_mul(2).min(cap)
};
let wait = Duration::from_millis(state.backoff_ms)
+ Duration::from_millis(
rand::rng()
.random_range(0..=(state.backoff_ms / JITTER_FRAC_NUM).max(1)),
);
state.due = Some(Instant::now() + wait);
state.round_active = false;
if restored {
info!(dc = key.0, family = ?key.1, alive = entry.alive, required = entry.target_required,
endpoint_count = entry.endpoints.len(), "ME writer floor restored for DC");
}
if !restored && state.warn_due.is_none_or(|due| due <= Instant::now()) {
state.warn_due = Some(Instant::now() + pool.warn_rate_limit_duration());
if state.outage {
warn!(dc = key.0, family = ?key.1, endpoint = %entry.endpoints[0], required = entry.target_required,
backoff_ms = state.backoff_ms, "Single-endpoint outage reconnect scheduled");
} else if pool.is_runtime_ready() {
warn!(dc = key.0, family = ?key.1, alive = entry.alive, required = entry.target_required,
endpoint_count = entry.endpoints.len(), backoff_ms = state.backoff_ms,
"DC writer floor is below required level, scheduled reconnect");
} else {
info!(dc = key.0, family = ?key.1, alive = entry.alive, required = entry.target_required,
endpoint_count = entry.endpoints.len(), backoff_ms = state.backoff_ms,
"DC writer floor is below required level during startup, scheduled reconnect");
}
}
}
}
if health_tick {
next_tick = tokio::time::Instant::now()
+ if degraded {
pool.health_interval_unhealthy()
} else {
pool.health_interval_healthy()
};
}
observe = false;
health_tick = false;
}
let budget = family::health_reconnect_budget(&pool, scheduler.groups.len());
for _ in 0..scheduler.queue.len() {
if tasks.len() >= budget {
break;
}
let Some(key) = scheduler.queue.pop_front() else {
break;
};
scheduler.queue.push_back(key);
if scheduler.busy(key) {
continue;
}
let Some(observation) = observations.get(&key.1).cloned() else {
continue;
};
if pool.floor_authority() != observation.authority
|| pool.current_generation() != observation.generation
{
continue;
}
let Some(entry) = observation.floor.by_dc.get(&key.0).cloned() else {
continue;
};
if entry.endpoints.is_empty() {
continue;
}
let Some(state) = scheduler.groups.get_mut(&key) else {
continue;
};
if state.deferred {
continue;
}
let now = Instant::now();
let recover =
entry.alive < entry.target_required && state.due.is_none_or(|due| due <= now);
let refresh = state.refresh_due.is_none_or(|due| due <= now)
&& idle_refresh_ready(&observation, &entry);
let mut kind = if refresh && (!recover || state.prefer_refresh) {
JobKind::Refresh
} else if recover {
JobKind::Recover
} else if entry.alive >= entry.target_required
&& entry.endpoints.len() == 1
&& pool.single_endpoint_shadow_rotate_interval().is_some()
&& state.shadow_due.is_none_or(|due| due <= now)
{
JobKind::Shadow
} else {
continue;
};
if kind == JobKind::Recover {
if pool
.has_refill_inflight_for_target(RefillTargetKey {
dc: key.0,
family: key.1,
generation: observation.generation,
endpoint_revision: observation.authority.0,
contour: WriterContour::Active,
})
.await
{
continue;
}
// Preflight distinguishes capacity pressure from unreachable endpoints without
// retaining an opening token while a group waits in the fair queue.
let reservation = pool
.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Coverage,
key.0,
entry.endpoints[0],
)
.await;
if reservation.is_none() {
kind = JobKind::Transfer;
}
drop(reservation);
if !state.round_active {
state.round = state.round.wrapping_add(1);
state.round_left = if state.outage {
1
} else {
entry.target_required.saturating_sub(entry.alive)
};
state.round_active = true;
}
state.prefer_refresh = true;
} else if kind == JobKind::Refresh {
state.prefer_refresh = false;
}
let job = HealthJob {
key,
kind,
observation,
entry,
round: state.round,
};
let future = run_job(pool.clone(), rng.clone(), job.clone());
let Ok(tracked) = pool.lifecycle.track_producer(future) else {
return;
};
let id = tasks.spawn(tracked).id();
scheduler.owners.insert(id, job);
}
let now = Instant::now();
let deadline = scheduler
.groups
.values()
.flat_map(|state| [state.due, state.refresh_due, state.shadow_due])
.flatten()
.filter(|due| *due > now)
.min()
.map(tokio::time::Instant::from_std)
.map_or(next_tick, |due| due.min(next_tick));
tokio::select! {
joined = tasks.join_next_with_id(), if !tasks.is_empty() => {
let Some(joined) = joined else { continue; };
let (id, output) = match joined {
Ok((id, output)) => (id, output),
Err(error) => {
debug!(error = %error, "Health writer operation failed");
(error.id(), Some(JobOutcome::Completed(None)))
}
};
if let Some(job) = scheduler.owners.remove(&id)
&& let Some(deadline) = output {
scheduler.completed(&pool, job, deadline);
}
observe = true;
}
_ = tokio::time::sleep_until(deadline) => {
observe = true;
health_tick = tokio::time::Instant::now() >= next_tick;
}
}
}
}
+24 -2
View File
@@ -1,5 +1,10 @@
use super::*;
// Outage regressions use loopback sockets and explicit family-state snapshots.
#[cfg(test)]
mod tests;
/// Reports whether replacing an endpoint could disrupt a bound client.
pub(super) fn has_bound_clients_on_endpoint(
writer_ids: &[u64],
bound_clients_by_writer: &HashMap<u64, usize>,
@@ -9,6 +14,7 @@ pub(super) fn has_bound_clients_on_endpoint(
.any(|writer_id| bound_clients_by_writer.get(writer_id).copied().unwrap_or(0) > 0)
}
/// Restores missing single-endpoint coverage under the bounded outage retry policy.
pub(super) async fn recover_single_endpoint_outage(
pool: &Arc<MePool>,
rng: &Arc<SecureRandom>,
@@ -53,13 +59,21 @@ pub(super) async fn recover_single_endpoint_outage(
pool.stats
.increment_me_single_endpoint_outage_reconnect_attempt_total();
let generation = pool.current_generation();
let bypass_quarantine = pool.single_endpoint_outage_disable_quarantine();
let attempt_ok = if bypass_quarantine {
pool.stats
.increment_me_single_endpoint_quarantine_bypass_total();
match tokio::time::timeout(
pool.reconnect_runtime.me_one_timeout,
pool.connect_one_for_dc(endpoint, key.0, rng.as_ref()),
pool.connect_one_with_generation_contour_for_dc_with_intent(
endpoint,
rng.as_ref(),
generation,
WriterContour::Active,
key.0,
WriterOpenIntent::Coverage,
),
)
.await
{
@@ -88,7 +102,14 @@ pub(super) async fn recover_single_endpoint_outage(
let one_endpoint = [endpoint];
match tokio::time::timeout(
pool.reconnect_runtime.me_one_timeout,
pool.connect_endpoints_round_robin(key.0, &one_endpoint, rng.as_ref()),
pool.connect_endpoints_round_robin_with_generation_contour(
key.0,
&one_endpoint,
rng.as_ref(),
generation,
WriterContour::Active,
WriterOpenIntent::Coverage,
),
)
.await
{
@@ -142,6 +163,7 @@ pub(super) async fn recover_single_endpoint_outage(
);
}
/// Rotates an idle single-endpoint shadow without removing the existing coverage first.
pub(super) async fn maybe_rotate_single_endpoint_shadow(
pool: &Arc<MePool>,
rng: &Arc<SecureRandom>,
@@ -0,0 +1,148 @@
use std::sync::atomic::Ordering;
use super::*;
use crate::transport::middle_proxy::admission_test_support::unregistered_writer;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool;
async fn single_endpoint_reaches_tcp_with_a_full_cap(bypass_quarantine: bool) {
let pool = make_pool().await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = listener.local_addr().unwrap();
let donor = SocketAddr::new(endpoint.ip(), 1);
pool.update_proxy_maps(
HashMap::from([
(2, vec![(endpoint.ip(), endpoint.port())]),
(3, vec![(donor.ip(), donor.port())]),
]),
None,
)
.await;
pool.floor_runtime
.me_adaptive_floor_cpu_cores_override
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_per_core
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_global
.store(1, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_outage_disable_quarantine
.store(bypass_quarantine, Ordering::Relaxed);
let (writer, _receiver) = unregistered_writer(
&pool,
1,
3,
donor,
pool.current_generation(),
WriterContour::Active,
);
pool.writers.write().await.push(writer);
if bypass_quarantine {
pool.endpoint_quarantine
.lock()
.await
.insert(endpoint, Instant::now() + Duration::from_secs(60));
}
let rng = Arc::new(SecureRandom::new());
let key = (2, IpFamily::V4);
let mut backoff = HashMap::new();
let mut next_attempt = HashMap::new();
let semaphore = Arc::new(Semaphore::new(1));
let required = pool.required_writers_for_dc(1);
// TCP acceptance proves capacity admission; closing the peer deliberately rejects the handshake.
tokio::time::timeout(Duration::from_secs(1), async {
tokio::join!(
recover_single_endpoint_outage(
&pool,
&rng,
key,
endpoint,
required,
&mut backoff,
&mut next_attempt,
&semaphore,
),
async {
let (stream, _) = listener.accept().await.unwrap();
drop(stream);
},
);
})
.await
.expect("missing-DC recovery must reach TCP even when another DC fills the cap");
assert_eq!(
pool.stats
.get_me_single_endpoint_outage_reconnect_attempt_total(),
1
);
assert_eq!(
pool.stats
.get_me_single_endpoint_outage_reconnect_success_total(),
0
);
assert_eq!(
pool.stats.get_me_single_endpoint_quarantine_bypass_total(),
u64::from(bypass_quarantine)
);
assert!(next_attempt.contains_key(&key));
assert!(backoff.contains_key(&key));
assert_eq!(semaphore.available_permits(), 1);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn single_endpoint_quarantine_bypass_recovers_coverage_at_cap() {
single_endpoint_reaches_tcp_with_a_full_cap(true).await;
}
#[tokio::test]
async fn single_endpoint_round_robin_recovers_coverage_at_cap() {
single_endpoint_reaches_tcp_with_a_full_cap(false).await;
}
#[tokio::test]
async fn restored_coverage_preserves_family_suppression_hysteresis() {
let pool = make_pool().await;
let now = MePool::now_epoch_secs();
let until = now + 3600;
pool.set_family_runtime_state(
IpFamily::V4,
MeFamilyRuntimeState::Suppressed,
now,
until,
5,
0,
);
super::super::update_family_runtime_state(&pool, IpFamily::V4, false);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Suppressed
);
assert_eq!(pool.family_suppressed_until_epoch_secs(IpFamily::V4), until);
assert_eq!(pool.family_recover_success_streak(IpFamily::V4), 0);
pool.set_family_runtime_state(
IpFamily::V4,
MeFamilyRuntimeState::Suppressed,
now,
now - 1,
5,
0,
);
super::super::update_family_runtime_state(&pool, IpFamily::V4, false);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Recovering
);
assert_eq!(pool.family_recover_success_streak(IpFamily::V4), 1);
super::super::update_family_runtime_state(&pool, IpFamily::V4, false);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Healthy
);
assert_eq!(pool.family_fail_streak(IpFamily::V4), 0);
}
@@ -0,0 +1,311 @@
use super::*;
use std::collections::VecDeque;
use tokio::task::Id;
#[cfg(test)]
mod tests;
/// Exact fairness and single-owner domain; DC sign remains part of the identity.
pub(super) type Group = (i32, IpFamily);
/// Immutable observation is fenced again before dispatch and when accepting completion.
pub(super) struct HealthObservation {
/// Endpoint and floor-policy revisions used to derive this plan.
pub(super) authority: (u64, u64),
/// Active generation whose writers contribute coverage.
pub(super) generation: u64,
/// Family-local targets and global capacity projection.
pub(super) floor: FamilyFloorPlan,
/// Current non-draining writer IDs grouped by DC and endpoint.
pub(super) ids: HashMap<(i32, SocketAddr), Vec<u64>>,
/// Registry idle timestamps used only for refresh selection.
pub(super) idle: HashMap<u64, u64>,
/// Captured bindings; commit revalidates victim idleness when required.
pub(super) bound: HashMap<u64, usize>,
}
/// Mutually exclusive network work selected for one group quantum.
#[derive(Clone, Copy, PartialEq, Eq)]
pub(super) enum JobKind {
/// Restores one missing writer through ordinary bounded admission.
Recover,
/// Restores coverage by replacing an eligible donor.
Transfer,
/// Replaces an idle writer before upstream expiry.
Refresh,
/// Applies the configured single-endpoint rotation cadence.
Shadow,
}
/// Resource deferral consumes no network retry budget and preserves the eligibility deadline.
pub(super) enum JobOutcome {
/// Work consumed its quantum and may return a refresh or shadow deadline.
Completed(Option<Instant>),
/// Capacity or authority prevented an operation; do not charge network backoff.
Deferred,
}
/// Captured authority accompanies both successful results and JoinError ownership.
#[derive(Clone)]
pub(super) struct HealthJob {
/// Group holding the unique inflight slot.
pub(super) key: Group,
/// Determines which deadline or round may change on completion.
pub(super) kind: JobKind,
/// Immutable selection evidence shared by jobs from one observation.
pub(super) observation: Arc<HealthObservation>,
/// The selected group's target and endpoint candidates.
pub(super) entry: DcFloorPlanEntry,
/// Recovery round identity; refresh and shadow operations do not consume it.
pub(super) round: u64,
}
/// Retry and cooldown state owned exclusively by the monitor.
#[derive(Default)]
pub(super) struct GroupState {
/// Endpoint, policy and generation fence for every mutable field below.
pub(super) authority: Option<((u64, u64), u64)>,
/// Earliest recovery eligibility; resource deferral leaves it unchanged.
pub(super) due: Option<Instant>,
/// Idle-refresh cooldown independent of missing-floor recovery.
pub(super) refresh_due: Option<Instant>,
/// Single-endpoint rotation cooldown.
pub(super) shadow_due: Option<Instant>,
/// Backoff advances once per completed recovery round.
pub(super) backoff_ms: u64,
/// Remaining writer operations in the current round.
pub(super) round_left: usize,
/// Distinguishes a finalized round from one that has no operations left yet.
pub(super) round_active: bool,
/// Monotonic identity prevents late outcomes from changing a newer round.
pub(super) round: u64,
/// Alternates independently eligible refresh and recovery work.
pub(super) prefer_refresh: bool,
/// Selects the single-endpoint outage retry namespace.
pub(super) outage: bool,
/// Rate limit for missing-floor warnings.
pub(super) warn_due: Option<Instant>,
/// Resource-blocked groups wait until the next health tick.
pub(super) deferred: bool,
}
/// One queue and one owner table bound work across both address families.
#[derive(Default)]
pub(super) struct HealthScheduler {
/// Persistent cursor order survives observations and worker completions.
pub(super) queue: VecDeque<Group>,
/// Coalesced work and deadlines for configured groups only.
pub(super) groups: HashMap<Group, GroupState>,
/// Task IDs release group ownership even after panic or abort.
pub(super) owners: HashMap<Id, HealthJob>,
}
impl HealthScheduler {
/// Refreshes desired groups without resetting the round-robin cursor.
pub(super) fn observe(&mut self, observations: &HashMap<IpFamily, Arc<HealthObservation>>) {
let mut current = HashSet::new();
for (family, observation) in observations {
for dc in observation.floor.by_dc.keys() {
let key = (*dc, *family);
current.insert(key);
let state = self.groups.entry(key).or_default();
let authority = (observation.authority, observation.generation);
if state.authority != Some(authority) {
*state = GroupState {
authority: Some(authority),
..GroupState::default()
};
}
if !self.queue.contains(&key) {
self.queue.push_back(key);
}
}
}
self.queue.retain(|key| current.contains(key));
self.groups.retain(|key, _| current.contains(key));
}
/// Includes obsolete tasks until their actual completion releases ownership.
pub(super) fn busy(&self, key: Group) -> bool {
self.owners.values().any(|job| job.key == key)
}
/// Accepts only an authoritative outcome from a still-active recovery round.
pub(super) fn completed(&mut self, pool: &Arc<MePool>, job: HealthJob, outcome: JobOutcome) {
let Some(state) = self.groups.get_mut(&job.key) else {
return;
};
if state.authority != Some((job.observation.authority, job.observation.generation))
|| pool.floor_authority() != job.observation.authority
|| pool.current_generation() != job.observation.generation
{
return;
}
if matches!(job.kind, JobKind::Recover | JobKind::Transfer)
&& (!state.round_active || state.round != job.round)
{
return;
}
let JobOutcome::Completed(deadline) = outcome else {
state.deferred = true;
return;
};
match job.kind {
JobKind::Refresh => {
state.refresh_due = Some(deadline.unwrap_or_else(|| {
Instant::now() + Duration::from_secs(IDLE_REFRESH_RETRY_SECS)
}))
}
JobKind::Shadow => {
state.shadow_due = Some(deadline.unwrap_or_else(|| {
Instant::now() + Duration::from_secs(SHADOW_ROTATE_RETRY_SECS)
}))
}
JobKind::Recover | JobKind::Transfer => {
state.round_left = state.round_left.saturating_sub(1);
// Round finalization uses the next fresh observation, never the job's old count.
state.due = None;
}
}
}
}
/// Finds refresh work without waiting for another group's network operation.
pub(super) fn idle_refresh_ready(
observation: &HealthObservation,
entry: &DcFloorPlanEntry,
) -> bool {
let now = MePool::now_epoch_secs();
entry.endpoints.iter().any(|endpoint| {
observation
.ids
.get(&(entry.dc, *endpoint))
.is_some_and(|ids| {
ids.iter().any(|id| {
observation.bound.get(id).copied().unwrap_or(0) == 0
&& observation.idle.get(id).is_some_and(|since| {
now.saturating_sub(*since)
>= IDLE_REFRESH_TRIGGER_BASE_SECS
+ id % (IDLE_REFRESH_TRIGGER_JITTER_SECS + 1)
})
})
})
})
}
/// Executes at most one bounded writer operation, never a whole missing-writer batch.
pub(super) async fn run_job(
pool: Arc<MePool>,
rng: Arc<SecureRandom>,
job: HealthJob,
) -> JobOutcome {
if pool.floor_authority() != job.observation.authority
|| pool.current_generation() != job.observation.generation
{
return JobOutcome::Deferred;
}
let key = job.key;
let entry = &job.entry;
let observation = &job.observation;
let mut deadlines = HashMap::new();
match job.kind {
JobKind::Refresh => {
maybe_refresh_idle_writer_for_dc(
&pool,
&rng,
key,
key.0,
key.1,
&entry.endpoints,
entry.alive,
entry.target_required,
&observation.ids,
&observation.idle,
&observation.bound,
&mut deadlines,
)
.await
}
JobKind::Shadow => {
maybe_rotate_single_endpoint_shadow(
&pool,
&rng,
key,
key.0,
key.1,
&entry.endpoints,
entry.alive,
entry.target_required,
&observation.ids,
&observation.bound,
&mut deadlines,
)
.await
}
JobKind::Transfer => {
if coverage::transfer_coverage(
&pool,
&rng,
key.0,
entry.endpoints[0],
entry.target_required,
)
.await
.is_none()
{
return JobOutcome::Deferred;
}
}
JobKind::Recover => {
let outage = entry.endpoints.len() == 1
&& entry.alive == 0
&& pool.single_endpoint_outage_mode_enabled();
pool.stats.increment_me_reconnect_attempt();
if outage {
pool.stats
.increment_me_single_endpoint_outage_reconnect_attempt_total();
}
let base = pool.required_writers_for_dc_with_floor_mode(entry.endpoints.len(), false);
let intent = if entry.alive < base {
WriterOpenIntent::Coverage
} else {
WriterOpenIntent::Normal
};
let connected = tokio::time::timeout(pool.reconnect_runtime.me_one_timeout, async {
if outage && pool.single_endpoint_outage_disable_quarantine() {
pool.stats
.increment_me_single_endpoint_quarantine_bypass_total();
pool.connect_one_with_generation_contour_for_dc_with_intent(
entry.endpoints[0],
rng.as_ref(),
observation.generation,
WriterContour::Active,
key.0,
intent,
)
.await
.is_ok()
} else {
pool.connect_endpoints_round_robin_with_generation_contour(
key.0,
&entry.endpoints,
rng.as_ref(),
observation.generation,
WriterContour::Active,
intent,
)
.await
}
})
.await
.unwrap_or(false);
if connected {
pool.stats.increment_me_reconnect_success();
if outage {
pool.stats
.increment_me_single_endpoint_outage_reconnect_success_total();
}
}
}
}
JobOutcome::Completed(deadlines.remove(&key))
}
@@ -0,0 +1,67 @@
use super::*;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool;
#[tokio::test]
async fn deferred_and_stale_completions_preserve_round_budget_and_deadlines() {
let pool = make_pool().await;
let addr: SocketAddr = "127.0.0.1:443".parse().unwrap();
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
let observation = Arc::new(check_family(IpFamily::V4, &pool).await);
let mut scheduler = HealthScheduler::default();
scheduler.observe(&HashMap::from([(IpFamily::V4, observation.clone())]));
let key = (2, IpFamily::V4);
let entry = observation.floor.by_dc[&2].clone();
let job = HealthJob {
key,
kind: JobKind::Transfer,
observation,
entry,
round: 4,
};
let due = Instant::now() - Duration::from_secs(1);
let state = scheduler.groups.get_mut(&key).unwrap();
state.round = 4;
state.round_left = 3;
state.round_active = true;
state.due = Some(due);
scheduler.completed(&pool, job.clone(), JobOutcome::Deferred);
let state = &scheduler.groups[&key];
assert!(state.deferred);
assert_eq!(state.round_left, 3);
assert_eq!(state.due, Some(due));
let state = scheduler.groups.get_mut(&key).unwrap();
state.deferred = false;
state.round_active = false;
scheduler.completed(&pool, job.clone(), JobOutcome::Completed(None));
scheduler.completed(&pool, job, JobOutcome::Deferred);
let state = &scheduler.groups[&key];
assert!(!state.deferred);
assert_eq!(state.round_left, 3);
assert_eq!(state.due, Some(due));
}
#[tokio::test]
async fn queue_survives_repeated_observations_without_starving_later_groups() {
let pool = make_pool().await;
pool.update_proxy_maps(
(1..=12)
.map(|dc| (dc, vec![("127.0.0.1".parse().unwrap(), 443)]))
.collect(),
None,
)
.await;
let observation = Arc::new(check_family(IpFamily::V4, &pool).await);
let observations = HashMap::from([(IpFamily::V4, observation)]);
let mut scheduler = HealthScheduler::default();
scheduler.observe(&observations);
let expected = scheduler.queue.iter().copied().collect::<HashSet<_>>();
let mut visited = HashSet::new();
for _ in 0..expected.len() {
let key = scheduler.queue.pop_front().unwrap();
scheduler.queue.push_back(key);
visited.insert(key);
scheduler.observe(&observations);
}
assert_eq!(visited, expected);
}
+5
View File
@@ -49,6 +49,11 @@ mod send;
mod send_adversarial_tests;
mod wire;
/// Network-free writer fixtures shared by admission and recovery regressions.
#[cfg(test)]
#[path = "tests/admission_test_support.rs"]
pub(crate) mod admission_test_support;
use bytes::Bytes;
use tokio::sync::OwnedSemaphorePermit;
+10 -5
View File
@@ -342,6 +342,8 @@ pub(super) struct ReinitCoordinatorState {
/// Latest endpoint authority revision accepted by the coordinator.
pub(super) endpoint_revision: u64,
pub(super) pending: Option<ReinitPendingState>,
/// Revision of the coherent floor and capacity policy.
pub(super) floor_policy_revision: u64,
pub(super) attempts: HashMap<u64, ReinitAttemptState>,
}
@@ -407,8 +409,9 @@ pub(super) struct BindingPolicyCore {
}
pub(super) struct NatRuntimeCore {
pub(super) discovery: super::pool_nat::NatDiscovery,
pub(super) nat_ip_cfg: Option<IpAddr>,
pub(super) nat_ip_detected: Arc<RwLock<Option<IpAddr>>>,
pub(super) nat_ip_detected: Arc<RwLock<Option<super::pool_nat::HttpNatObservation>>>,
pub(super) nat_probe: bool,
pub(super) nat_stun: Option<String>,
pub(super) nat_stun_servers: Vec<String>,
@@ -502,10 +505,10 @@ pub struct MePool {
pub(super) proxy_secret: Arc<RwLock<SecretSnapshot>>,
pub(super) default_dc: AtomicI32,
pub(super) next_writer_id: AtomicU64,
pub(super) writer_connect_active_reserved: AtomicUsize,
pub(super) writer_connect_warm_reserved: AtomicUsize,
/// Replacement connections opened but not yet committed to writer visibility.
pub(super) writer_replacement_open_reserved: AtomicUsize,
pub(super) writer_connect_active_reserved: Arc<AtomicUsize>,
pub(super) writer_connect_warm_reserved: Arc<AtomicUsize>,
/// Physical replacement overlap: preparation plus victims awaiting transport destruction.
pub(super) writer_replacement_open_reserved: Arc<AtomicUsize>,
pub(super) rtt_stats: Arc<Mutex<HashMap<u64, (f64, f64)>>>,
/// Coalesced refill state keyed by exact generation and contour ownership.
pub(super) refill_states: Arc<ParkingMutex<HashMap<RefillTargetKey, RefillTargetState>>>,
@@ -544,6 +547,8 @@ mod transport_policy;
mod selection_policy;
// Bounded writer-open admission and coverage accounting.
mod writer_admission;
// Coherent policy revisions fence derived health plans and replacement decisions.
mod floor_authority;
pub(super) use writer_admission::{WriterOpenIntent, WriterOpenReservation, WriterRole};
// Endpoint-to-DC routing and health timing policy.
mod routing;
@@ -165,6 +165,7 @@ impl MePool {
active_generation: 1,
desired_map_hash: 0,
endpoint_revision: 1,
floor_policy_revision: 1,
pending: None,
attempts: HashMap::new(),
}),
@@ -279,6 +280,7 @@ impl MePool {
me_bind_stale_ttl_secs: AtomicU64::new(me_bind_stale_ttl_secs),
}),
nat_runtime: Arc::new(NatRuntimeCore {
discovery: super::super::pool_nat::NatDiscovery::default(),
nat_ip_cfg: nat_ip,
nat_ip_detected: Arc::new(RwLock::new(None)),
nat_probe,
@@ -394,9 +396,9 @@ impl MePool {
pool_size: 2,
default_dc: AtomicI32::new(default_dc.unwrap_or(2)),
next_writer_id: AtomicU64::new(1),
writer_connect_active_reserved: AtomicUsize::new(0),
writer_connect_warm_reserved: AtomicUsize::new(0),
writer_replacement_open_reserved: AtomicUsize::new(0),
writer_connect_active_reserved: Arc::new(AtomicUsize::new(0)),
writer_connect_warm_reserved: Arc::new(AtomicUsize::new(0)),
writer_replacement_open_reserved: Arc::new(AtomicUsize::new(0)),
rtt_stats: Arc::new(Mutex::new(HashMap::new())),
refill_states: Arc::new(ParkingMutex::new(HashMap::new())),
refill_running: AtomicUsize::new(0),
@@ -0,0 +1,59 @@
use super::*;
#[cfg(test)]
mod tests;
impl MePool {
/// Captures the two independent authorities used by a health plan.
pub(in crate::transport::middle_proxy) fn floor_authority(&self) -> (u64, u64) {
let coordinator = self.reinit.coordinator.lock();
(
coordinator.endpoint_revision,
coordinator.floor_policy_revision,
)
}
/// Captures floor-affecting values under the coordinator without recursively acquiring it.
pub(super) fn floor_policy_values(&self) -> [u64; 14] {
let floor = &self.floor_runtime;
[
self.single_endpoint_runtime
.me_single_endpoint_shadow_writers
.load(Ordering::Relaxed) as u64,
floor.me_floor_mode.load(Ordering::Relaxed) as u64,
floor.me_adaptive_floor_idle_secs.load(Ordering::Relaxed),
floor
.me_adaptive_floor_min_writers_single_endpoint
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_min_writers_multi_endpoint
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_recover_grace_secs
.load(Ordering::Relaxed),
floor
.me_adaptive_floor_writers_per_core_total
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_cpu_cores_override
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_max_extra_writers_single_per_core
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_max_extra_writers_multi_per_core
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_max_active_writers_per_core
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_max_warm_writers_per_core
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_max_active_writers_global
.load(Ordering::Relaxed) as u64,
floor
.me_adaptive_floor_max_warm_writers_global
.load(Ordering::Relaxed) as u64,
]
}
}
@@ -0,0 +1,96 @@
use super::*;
use crate::config::GeneralConfig;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool;
fn reload(pool: &MePool, cfg: &GeneralConfig) {
pool.update_runtime_reinit_policy(
cfg.hardswap,
cfg.me_pool_drain_ttl_secs,
cfg.me_instadrain,
cfg.me_pool_drain_threshold,
cfg.me_pool_drain_soft_evict_enabled,
cfg.me_pool_drain_soft_evict_grace_secs,
cfg.me_pool_drain_soft_evict_per_writer,
cfg.me_pool_drain_soft_evict_budget_per_core,
cfg.me_pool_drain_soft_evict_cooldown_ms,
cfg.effective_me_pool_force_close_secs(),
cfg.me_pool_min_fresh_ratio,
cfg.me_hardswap_warmup_delay_min_ms,
cfg.me_hardswap_warmup_delay_max_ms,
cfg.me_hardswap_warmup_extra_passes,
cfg.me_hardswap_warmup_pass_backoff_base_ms,
cfg.me_bind_stale_mode,
cfg.me_bind_stale_ttl_secs,
cfg.me_secret_atomic_snapshot,
cfg.me_deterministic_writer_sort,
cfg.me_writer_pick_mode,
cfg.me_writer_pick_sample_size,
cfg.me_single_endpoint_shadow_writers,
cfg.me_single_endpoint_outage_mode_enabled,
cfg.me_single_endpoint_outage_disable_quarantine,
cfg.me_single_endpoint_outage_backoff_min_ms,
cfg.me_single_endpoint_outage_backoff_max_ms,
cfg.me_single_endpoint_shadow_rotate_every_secs,
cfg.me_floor_mode,
cfg.me_adaptive_floor_idle_secs,
cfg.me_adaptive_floor_min_writers_single_endpoint,
cfg.me_adaptive_floor_min_writers_multi_endpoint,
cfg.me_adaptive_floor_recover_grace_secs,
cfg.me_adaptive_floor_writers_per_core_total,
cfg.me_adaptive_floor_cpu_cores_override,
cfg.me_adaptive_floor_max_extra_writers_single_per_core,
cfg.me_adaptive_floor_max_extra_writers_multi_per_core,
cfg.me_adaptive_floor_max_active_writers_per_core,
cfg.me_adaptive_floor_max_warm_writers_per_core,
cfg.me_adaptive_floor_max_active_writers_global,
cfg.me_adaptive_floor_max_warm_writers_global,
cfg.me_health_interval_ms_unhealthy,
cfg.me_health_interval_ms_healthy,
cfg.me_warn_rate_limit_ms,
);
}
#[tokio::test]
async fn policy_reload_invalidates_old_caps_and_rejects_stale_plan_publication() {
let pool = make_pool().await;
let mut cfg = GeneralConfig::default();
reload(&pool, &cfg);
let authority = pool.floor_authority();
pool.set_adaptive_floor_runtime_caps(authority, 10, 12, 10, 12, 6, 6, 0);
assert_eq!(
pool.floor_runtime
.me_adaptive_floor_active_cap_effective
.load(Ordering::Acquire),
12
);
cfg.me_single_endpoint_shadow_writers += 1;
reload(&pool, &cfg);
assert_ne!(pool.floor_authority(), authority);
assert_eq!(
pool.floor_runtime
.me_adaptive_floor_active_cap_effective
.load(Ordering::Acquire),
0
);
pool.set_adaptive_floor_runtime_caps(authority, 10, 128, 10, 128, 100, 6, 0);
assert_eq!(
pool.floor_runtime
.me_adaptive_floor_active_cap_effective
.load(Ordering::Acquire),
0
);
let new_authority = pool.floor_authority();
reload(&pool, &cfg);
assert_eq!(
pool.floor_authority(),
new_authority,
"unchanged reload must not supersede work"
);
pool.set_adaptive_floor_runtime_caps(new_authority, 10, 11, 10, 11, 6, 6, 0);
assert_eq!(
pool.floor_runtime
.me_adaptive_floor_active_cap_effective
.load(Ordering::Acquire),
11
);
}
@@ -305,6 +305,8 @@ impl MePool {
if previous_writer_pick_mode != writer_pick_mode {
self.stats.increment_me_writer_pick_mode_switch_total();
}
let mut coordinator = self.reinit.coordinator.lock();
let previous_floor_policy = self.floor_policy_values();
self.single_endpoint_runtime
.me_single_endpoint_shadow_writers
.store(single_endpoint_shadow_writers, Ordering::Relaxed);
@@ -381,6 +383,17 @@ impl MePool {
self.floor_runtime
.me_adaptive_floor_max_warm_writers_global
.store(adaptive_floor_max_warm_writers_global, Ordering::Relaxed);
if previous_floor_policy != self.floor_policy_values() {
coordinator.floor_policy_revision = coordinator.floor_policy_revision.wrapping_add(1);
// Cached caps belong to the old policy until a new health plan is accepted.
self.floor_runtime
.me_adaptive_floor_active_cap_effective
.store(0, Ordering::Release);
self.floor_runtime
.me_adaptive_floor_warm_cap_effective
.store(0, Ordering::Release);
}
drop(coordinator);
self.health_runtime
.me_health_interval_ms_unhealthy
.store(me_health_interval_ms_unhealthy.max(1), Ordering::Relaxed);
@@ -230,8 +230,10 @@ impl MePool {
configured
}
/// Publishes derived caps only while both endpoint and floor-policy authority remain current.
pub(in crate::transport::middle_proxy) fn set_adaptive_floor_runtime_caps(
&self,
authority: (u64, u64),
active_cap_configured: usize,
active_cap_effective: usize,
warm_cap_configured: usize,
@@ -240,6 +242,15 @@ impl MePool {
active_writers_current: usize,
warm_writers_current: usize,
) {
let coordinator = self.reinit.coordinator.lock();
if authority
!= (
coordinator.endpoint_revision,
coordinator.floor_policy_revision,
)
{
return;
}
self.floor_runtime
.me_adaptive_floor_global_cap_raw
.store(active_cap_configured as u64, Ordering::Relaxed);
@@ -48,26 +48,26 @@ pub(in crate::transport::middle_proxy) enum WriterOpenIntent {
}
/// RAII ownership of one bounded in-flight writer open.
pub(in crate::transport::middle_proxy) struct WriterOpenReservation<'a> {
counter: Option<&'a AtomicUsize>,
pub(in crate::transport::middle_proxy) struct WriterOpenReservation {
counter: Option<Arc<AtomicUsize>>,
}
impl Drop for WriterOpenReservation<'_> {
impl Drop for WriterOpenReservation {
fn drop(&mut self) {
if let Some(counter) = self.counter {
if let Some(counter) = &self.counter {
counter.fetch_sub(1, Ordering::AcqRel);
}
}
}
impl MePool {
/// Computes the authoritative active-writer floor across enabled families and DCs.
/// Computes reservation coverage across configured families, including suppressed ones.
pub(in crate::transport::middle_proxy) async fn active_coverage_required_total(&self) -> usize {
let now_epoch_secs = Self::now_epoch_secs();
let mut required_total = 0usize;
let endpoint_snapshot = self.endpoint_snapshot.load_full();
if self.family_enabled_for_drain_coverage(IpFamily::V4, now_epoch_secs) {
// Suppression must not remove the capacity needed to recover active coverage.
if self.decision.ipv4_me {
for addrs in endpoint_snapshot.map_v4.values() {
let mut endpoints = HashSet::<SocketAddr>::new();
for (ip, port) in addrs.iter().copied() {
@@ -79,7 +79,7 @@ impl MePool {
}
}
if self.family_enabled_for_drain_coverage(IpFamily::V6, now_epoch_secs) {
if self.decision.ipv6_me {
for addrs in endpoint_snapshot.map_v6.values() {
let mut endpoints = HashSet::<SocketAddr>::new();
for (ip, port) in addrs.iter().copied() {
@@ -172,7 +172,7 @@ impl MePool {
intent: WriterOpenIntent,
writer_dc: i32,
target_addr: SocketAddr,
) -> Option<WriterOpenReservation<'_>> {
) -> Option<WriterOpenReservation> {
let counter = match contour {
WriterContour::Active => &self.writer_connect_active_reserved,
WriterContour::Warm => &self.writer_connect_warm_reserved,
@@ -182,6 +182,8 @@ impl MePool {
};
if intent == WriterOpenIntent::Replacement {
// Policy reload and replacement admission share the same linearization point.
let _coordinator = self.reinit.coordinator.lock();
let configured_cap = match contour {
WriterContour::Active => self.adaptive_floor_active_cap_configured_total(),
WriterContour::Warm => self.adaptive_floor_warm_cap_configured_total(),
@@ -220,7 +222,7 @@ impl MePool {
.is_ok()
{
return Some(WriterOpenReservation {
counter: Some(&self.writer_replacement_open_reserved),
counter: Some(self.writer_replacement_open_reserved.clone()),
});
}
}
@@ -266,7 +268,7 @@ impl MePool {
.is_ok()
{
return Some(WriterOpenReservation {
counter: Some(counter),
counter: Some(counter.clone()),
});
}
}
@@ -111,6 +111,31 @@ impl MePoolLifecycle {
self.admission.try_register()
}
/// Registers a cancellation-aware producer for a caller-owned JoinSet without a second spawn.
pub(super) fn track_producer<F>(
&self,
future: F,
) -> Result<impl Future<Output = Option<F::Output>> + Send + 'static + use<F>, F>
where
F: Future + Send + 'static,
F::Output: Send + 'static,
{
let Some(registration) = self.try_register() else {
return Err(future);
};
let cancel = self.producer_cancel.clone();
let tracked = self.producer_tasks.track_future(async move {
tokio::select! {
biased;
_ = cancel.cancelled() => None,
output = future => Some(output),
}
});
// The tracker owns destruction before shutdown can observe no registrations.
drop(registration);
Ok(tracked)
}
/// Registers and spawns one cancellation-aware ME producer.
pub(super) fn spawn_producer<F>(&self, future: F) -> Result<(), F>
where
+110 -130
View File
@@ -1,7 +1,9 @@
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::oneshot;
use tokio::task::JoinSet;
use tokio::time::timeout;
use tracing::{debug, info};
@@ -18,7 +20,21 @@ use std::time::Instant;
const STUN_BATCH_TIMEOUT: Duration = Duration::from_secs(5);
const STUN_BATCH_TCP_FALLBACK_TIMEOUT: Duration = Duration::from_secs(12);
const STUN_CACHE_TTL: Duration = Duration::from_secs(600);
const HTTP_FAILURE_TTL: Duration = Duration::from_secs(60);
// Discovery ownership and source-bound cache admission for this pool generation.
mod state;
pub(super) use state::NatDiscovery;
// Successful HTTP observations expire independently of failure backoff.
mod http;
pub(super) use http::HttpNatObservation;
// Local network regressions exercise cancellation before discovery publication.
#[cfg(test)]
mod tests;
/// Probes the configured or default STUN endpoint with both address families.
#[allow(dead_code)]
pub async fn stun_probe(stun_addr: Option<String>) -> Result<crate::network::stun::DualStunResult> {
let stun_addr = stun_addr.unwrap_or_else(|| {
@@ -35,6 +51,7 @@ pub async fn stun_probe(stun_addr: Option<String>) -> Result<crate::network::stu
stun_probe_dual_with_tcp_fallback(&stun_addr, false).await
}
/// Detects a public IPv4 address using the default HTTP endpoints.
#[allow(dead_code)]
pub async fn detect_public_ip() -> Option<IpAddr> {
let urls = crate::config::defaults::default_http_ip_detect_urls();
@@ -151,15 +168,18 @@ impl MePool {
(live_servers, best_reflected)
}
/// Applies the configured or HTTP-detected NAT IP to non-public addresses.
pub(super) fn translate_ip_for_nat(&self, ip: IpAddr) -> IpAddr {
let nat_ip = self.nat_runtime.nat_ip_cfg.or_else(|| {
Self::translate_ip_with_nat(
ip,
self.nat_runtime
.nat_ip_detected
.try_read()
.ok()
.and_then(|g| *g)
});
.nat_ip_cfg
.or_else(|| self.cached_http_nat_ip()),
)
}
/// Applies a captured NAT observation without rereading mutable discovery state.
pub(super) fn translate_ip_with_nat(ip: IpAddr, nat_ip: Option<IpAddr>) -> IpAddr {
let Some(nat_ip) = nat_ip else {
return ip;
};
@@ -177,10 +197,21 @@ impl MePool {
}
}
/// Selects the NAT source IP without replacing the actual TCP source port.
pub(super) fn translate_our_addr_with_reflection(
&self,
addr: std::net::SocketAddr,
reflected: Option<std::net::SocketAddr>,
) -> std::net::SocketAddr {
self.translate_our_addr_with_nat(addr, reflected, self.cached_http_nat_ip())
}
/// Retains the actual TCP port while using one handshake-local NAT snapshot.
pub(super) fn translate_our_addr_with_nat(
&self,
addr: std::net::SocketAddr,
reflected: Option<std::net::SocketAddr>,
detected: Option<IpAddr>,
) -> std::net::SocketAddr {
let ip = if let Some(nat_ip) = self.nat_runtime.nat_ip_cfg {
match (addr.ip(), nat_ip) {
@@ -193,130 +224,98 @@ impl MePool {
if is_bogon(addr.ip()) || addr.ip().is_loopback() || addr.ip().is_unspecified() {
r.ip()
} else {
self.translate_ip_for_nat(addr.ip())
Self::translate_ip_with_nat(addr.ip(), detected)
}
} else {
self.translate_ip_for_nat(addr.ip())
Self::translate_ip_with_nat(addr.ip(), detected)
};
// Keep the kernel-assigned TCP source port; STUN port can differ.
std::net::SocketAddr::new(ip, addr.port())
}
pub(super) async fn maybe_detect_nat_ip(&self, local_ip: IpAddr) -> Option<IpAddr> {
if self.nat_runtime.nat_ip_cfg.is_some() {
return self.nat_runtime.nat_ip_cfg;
}
if !self.nat_runtime.nat_probe {
return None;
}
if !(is_bogon(local_ip) || local_ip.is_loopback() || local_ip.is_unspecified()) {
return None;
}
if let Some(ip) = *self.nat_runtime.nat_ip_detected.read().await {
return Some(ip);
}
let Some(ip) = detect_public_ipv4_http(&self.nat_runtime.http_ip_detect_urls).await else {
return None;
};
{
let mut guard = self.nat_runtime.nat_ip_detected.write().await;
*guard = Some(IpAddr::V4(ip));
}
info!(public_ip = %ip, "Auto-detected public IP for NAT translation");
Some(IpAddr::V4(ip))
}
/// Waits for a pool-owned STUN refresh in the exact source-address context.
pub(super) async fn maybe_reflect_public_addr(
&self,
self: &Arc<Self>,
family: IpFamily,
bind_ip: Option<IpAddr>,
) -> Option<std::net::SocketAddr> {
const STUN_CACHE_TTL: Duration = Duration::from_secs(600);
let use_shared_cache = bind_ip.is_none();
if !use_shared_cache {
match (family, bind_ip) {
(IpFamily::V4, Some(IpAddr::V4(_)))
| (IpFamily::V6, Some(IpAddr::V6(_)))
| (_, None) => {}
_ => return None,
}
if let Some(bind_ip) = bind_ip {
return self.reflect_bound_addr(family, bind_ip).await;
}
// Backoff window
if use_shared_cache
&& let Some(until) = *self.nat_runtime.stun_backoff_until.read().await
&& Instant::now() < until
{
if let Ok(cache) = self.nat_runtime.nat_reflection_cache.try_lock() {
let slot = match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
};
return slot.map(|(_, addr)| addr);
}
return None;
if let Some(cached) = self.cached_shared_reflection(family).await {
return cached;
}
let singleflight = match family {
IpFamily::V4 => self.nat_runtime.nat_reflection_singleflight_v4.clone(),
IpFamily::V6 => self.nat_runtime.nat_reflection_singleflight_v6.clone(),
};
let flight = singleflight.lock_owned().await;
if let Some(cached) = self.cached_shared_reflection(family).await {
return cached;
}
if use_shared_cache && let Ok(mut cache) = self.nat_runtime.nat_reflection_cache.try_lock()
{
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
if let Some((ts, addr)) = slot
&& ts.elapsed() < STUN_CACHE_TTL
{
return Some(*addr);
}
}
let _singleflight_guard = if use_shared_cache {
Some(match family {
IpFamily::V4 => self.nat_runtime.nat_reflection_singleflight_v4.lock().await,
IpFamily::V6 => self.nat_runtime.nat_reflection_singleflight_v6.lock().await,
let pool = self.clone();
let (send, receive) = oneshot::channel();
self.lifecycle
.spawn_producer(async move {
let _flight = flight;
let attempt = pool
.nat_runtime
.nat_probe_attempts
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let reflected = pool.refresh_stun(family, None, attempt).await;
if let Some(addr) = reflected {
// Cache readers must never cause a completed refresh to be discarded.
let mut cache = pool.nat_runtime.nat_reflection_cache.lock().await;
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
*slot = Some((Instant::now(), addr));
pool.nat_runtime
.nat_probe_attempts
.store(0, std::sync::atomic::Ordering::Relaxed);
} else {
let backoff = Duration::from_secs(60 * 2u64.pow((attempt as u32).min(6)));
*pool.nat_runtime.stun_backoff_until.write().await =
Some(Instant::now() + backoff);
}
let _ = send.send(reflected);
})
} else {
None
.ok()?;
receive.await.ok().flatten()
}
async fn cached_shared_reflection(
&self,
family: IpFamily,
) -> Option<Option<std::net::SocketAddr>> {
let backoff = self
.nat_runtime
.stun_backoff_until
.read()
.await
.is_some_and(|until| Instant::now() < until);
let cache = self.nat_runtime.nat_reflection_cache.lock().await;
let slot = match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
};
if use_shared_cache
&& let Some(until) = *self.nat_runtime.stun_backoff_until.read().await
&& Instant::now() < until
{
if let Ok(cache) = self.nat_runtime.nat_reflection_cache.try_lock() {
let slot = match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
};
return slot.map(|(_, addr)| addr);
}
return None;
// Preserve the existing stale-result fallback during shared failure backoff.
if backoff {
return Some(slot.map(|(_, addr)| addr));
}
slot.filter(|(created, _)| created.elapsed() < STUN_CACHE_TTL)
.map(|(_, addr)| Some(addr))
}
if use_shared_cache && let Ok(mut cache) = self.nat_runtime.nat_reflection_cache.try_lock()
{
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
if let Some((ts, addr)) = slot
&& ts.elapsed() < STUN_CACHE_TTL
{
return Some(*addr);
}
}
let attempt = if use_shared_cache {
self.nat_runtime
.nat_probe_attempts
.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
} else {
0
};
async fn refresh_stun(
&self,
family: IpFamily,
bind_ip: Option<IpAddr>,
attempt: u8,
) -> Option<std::net::SocketAddr> {
let configured_servers = self.configured_stun_servers();
let live_snapshot = self.nat_runtime.nat_stun_live_servers.read().await.clone();
let primary_servers = if live_snapshot.is_empty() {
@@ -348,33 +347,14 @@ impl MePool {
}
if let Some(reflected_addr) = selected_reflected {
if use_shared_cache {
self.nat_runtime
.nat_probe_attempts
.store(0, std::sync::atomic::Ordering::Relaxed);
}
info!(
family = ?family,
live_servers = live_server_count,
"STUN-Quorum reached, IP: {}",
reflected_addr.ip()
);
if use_shared_cache
&& let Ok(mut cache) = self.nat_runtime.nat_reflection_cache.try_lock()
{
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
*slot = Some((Instant::now(), reflected_addr));
}
return Some(reflected_addr);
}
if use_shared_cache {
let backoff = Duration::from_secs(60 * 2u64.pow((attempt as u32).min(6)));
*self.nat_runtime.stun_backoff_until.write().await = Some(Instant::now() + backoff);
}
None
}
}
+108
View File
@@ -0,0 +1,108 @@
use super::*;
const HTTP_FRESH_TTL: Duration = Duration::from_secs(600);
const HTTP_MAX_AGE: Duration = Duration::from_secs(1200);
/// A successful HTTP observation has an immutable monotonic timestamp.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(in crate::transport::middle_proxy) struct HttpNatObservation {
/// Successful detector result, never updated by a failed refresh.
pub(in crate::transport::middle_proxy) ip: IpAddr,
/// Original observation time used for both freshness and bounded stale fallback.
pub(super) observed_at: tokio::time::Instant,
}
impl HttpNatObservation {
fn usable(self) -> Option<IpAddr> {
(self.observed_at.elapsed() < HTTP_MAX_AGE).then_some(self.ip)
}
fn fresh(self) -> bool {
self.observed_at.elapsed() < HTTP_FRESH_TTL
}
}
impl MePool {
/// Reads a bounded cache snapshot without discovery or data-plane waiting.
pub(in crate::transport::middle_proxy) fn cached_http_nat_ip(&self) -> Option<IpAddr> {
self.nat_runtime
.nat_ip_detected
.try_read()
.ok()
.and_then(|cache| cache.and_then(HttpNatObservation::usable))
}
/// Refreshes once per pool, retaining stale success only within its original grace period.
pub(in crate::transport::middle_proxy) async fn maybe_detect_nat_ip(
self: &Arc<Self>,
local_ip: IpAddr,
) -> Option<IpAddr> {
if self.nat_runtime.nat_ip_cfg.is_some() {
return self.nat_runtime.nat_ip_cfg;
}
if !self.nat_runtime.nat_probe
|| !local_ip.is_ipv4()
|| !(is_bogon(local_ip) || local_ip.is_loopback() || local_ip.is_unspecified())
{
return None;
}
if let Some(cached) = *self.nat_runtime.nat_ip_detected.read().await
&& cached.fresh()
{
return Some(cached.ip);
}
let mut flight = self.nat_runtime.discovery.http.clone().lock_owned().await;
let cached = *self.nat_runtime.nat_ip_detected.read().await;
if let Some(cached) = cached
&& cached.fresh()
{
return Some(cached.ip);
}
if flight.is_some_and(|until| Instant::now() < until) {
return cached.and_then(HttpNatObservation::usable);
}
let pool = self.clone();
let (send, receive) = oneshot::channel();
// Discovery owns the flight; cancellation of a handshake cannot cancel publication.
self.lifecycle
.spawn_producer(async move {
let detected = detect_public_ipv4_http(&pool.nat_runtime.http_ip_detect_urls)
.await
.map(IpAddr::V4);
if let Some(ip) = detected {
*pool.nat_runtime.nat_ip_detected.write().await = Some(HttpNatObservation {
ip,
observed_at: tokio::time::Instant::now(),
});
*flight = None;
info!(public_ip = %ip, "Auto-detected public IP for NAT translation");
} else {
*flight = Some(Instant::now() + HTTP_FAILURE_TTL);
}
let _ = send.send(());
})
.ok()?;
let _ = receive.await;
self.cached_http_nat_ip()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test(start_paused = true)]
async fn observation_age_is_not_renewed_by_reads_or_failure() {
let cached = HttpNatObservation {
ip: "91.108.56.1".parse().unwrap(),
observed_at: tokio::time::Instant::now(),
};
tokio::time::advance(HTTP_FRESH_TTL).await;
assert!(!cached.fresh());
for _ in 0..100 {
assert_eq!(cached.usable(), Some(cached.ip));
}
tokio::time::advance(HTTP_MAX_AGE - HTTP_FRESH_TTL).await;
assert_eq!(cached.usable(), None);
}
}
@@ -0,0 +1,119 @@
use std::collections::{BTreeSet, HashMap};
use std::net::{IpAddr, SocketAddr};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::{Mutex, oneshot};
use tokio_util::sync::CancellationToken;
use super::{IpFamily, MePool, STUN_CACHE_TTL};
// Context retirement and TTL tests use the same gates as production discovery.
#[cfg(test)]
mod tests;
/// Pool-local discovery gates; bound entries are limited to configured egress addresses.
#[derive(Default)]
pub(in crate::transport::middle_proxy) struct NatDiscovery {
pub(super) http: Arc<Mutex<Option<Instant>>>,
bound: Mutex<HashMap<(IpFamily, IpAddr), Arc<BoundDiscovery>>>,
}
#[derive(Default)]
struct BoundDiscovery {
state: Arc<Mutex<BoundDiscoveryState>>,
cancel: CancellationToken,
}
#[derive(Default)]
struct BoundDiscoveryState {
reflection: Option<(Instant, SocketAddr)>,
retry_after: Option<Instant>,
attempt: u8,
}
impl NatDiscovery {
async fn bound_context(
&self,
family: IpFamily,
bind_ip: IpAddr,
allowed: &BTreeSet<IpAddr>,
) -> Option<Arc<BoundDiscovery>> {
let mut contexts = self.bound.lock().await;
// Retired producers cannot republish into the map or a replacement context.
contexts.retain(|(entry_family, ip), context| {
let keep = *entry_family != family || allowed.contains(ip);
if !keep {
context.cancel.cancel();
}
keep
});
if !allowed.contains(&bind_ip) {
return None;
}
Some(contexts.entry((family, bind_ip)).or_default().clone())
}
}
impl MePool {
/// Refreshes one configured source without borrowing another source's cached reflection.
pub(super) async fn reflect_bound_addr(
self: &Arc<Self>,
family: IpFamily,
bind_ip: IpAddr,
) -> Option<SocketAddr> {
if bind_ip.is_ipv6() != matches!(family, IpFamily::V6) {
return None;
}
let allowed = self
.upstream
.as_ref()?
.direct_bind_ips_for_family(bind_ip.is_ipv6())
.await;
let context = self
.nat_runtime
.discovery
.bound_context(family, bind_ip, &allowed)
.await?;
let mut flight = tokio::select! {
biased;
_ = context.cancel.cancelled() => return None,
guard = context.state.clone().lock_owned() => guard,
};
if let Some((created, addr)) = flight.reflection
&& created.elapsed() < STUN_CACHE_TTL
{
return Some(addr);
}
if flight
.retry_after
.is_some_and(|until| Instant::now() < until)
{
return None;
}
let pool = self.clone();
let (send, receive) = oneshot::channel();
self.lifecycle.spawn_producer(async move {
tokio::select! {
biased;
_ = context.cancel.cancelled() => {}
_ = async {
let reflected = pool.refresh_stun(family, Some(bind_ip), flight.attempt).await;
if let Some(addr) = reflected {
flight.reflection = Some((Instant::now(), addr));
flight.retry_after = None;
flight.attempt = 0;
} else {
// Completed failures must also survive a short reconnect deadline.
let backoff = Duration::from_secs(60 * 2u64.pow(flight.attempt as u32));
flight.retry_after = Some(Instant::now() + backoff);
flight.attempt = flight.attempt.saturating_add(1).min(6);
}
let _ = send.send(reflected);
} => {}
}
}).ok()?;
receive.await.ok().flatten()
}
}
@@ -0,0 +1,146 @@
use super::*;
use crate::transport::middle_proxy::pool_nat::tests::lifecycle::set_bind_addresses;
use crate::transport::middle_proxy::pool_nat::tests::{
PUBLIC_IP, TEST_DEADLINE, nat_pool, stun_server,
};
use std::net::Ipv4Addr;
use std::sync::atomic::Ordering;
use tokio::time::timeout;
#[tokio::test]
async fn bound_positive_ttl_refreshes_without_publishing_to_default_cache() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
stun.release.add_permits(16);
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let ip = IpAddr::V4(Ipv4Addr::LOCALHOST);
let allowed = BTreeSet::from([ip]);
let context = pool
.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &allowed)
.await
.unwrap();
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
context.state.lock().await.reflection.as_mut().unwrap().0 = Instant::now() - STUN_CACHE_TTL;
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 2);
assert_eq!(pool.nat_runtime.nat_reflection_cache.lock().await.v4, None);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn context_churn_is_bounded_and_retirement_does_not_mix_families() {
let discovery = NatDiscovery::default();
let v6: IpAddr = "::1".parse().unwrap();
let v6_context = discovery
.bound_context(IpFamily::V6, v6, &BTreeSet::from([v6]))
.await
.unwrap();
let mut previous: Option<Arc<BoundDiscovery>> = None;
for suffix in 1..=1000u32 {
let ip = IpAddr::V4(Ipv4Addr::from(0x7f000000 + suffix));
let context = discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::from([ip]))
.await
.unwrap();
if let Some(old) = previous.replace(context.clone()) {
assert!(old.cancel.is_cancelled());
}
assert!(!context.cancel.is_cancelled());
assert!(!v6_context.cancel.is_cancelled());
assert_eq!(discovery.bound.lock().await.len(), 2);
}
let denied = IpAddr::V4(Ipv4Addr::LOCALHOST);
assert!(
discovery
.bound_context(IpFamily::V4, denied, &BTreeSet::new())
.await
.is_none()
);
assert!(previous.unwrap().cancel.is_cancelled());
assert_eq!(discovery.bound.lock().await.len(), 1);
}
#[tokio::test]
async fn retiring_bound_context_cancels_its_producer_and_prevents_late_publication() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let ip = IpAddr::V4(Ipv4Addr::LOCALHOST);
let context = pool
.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::from([ip]))
.await
.unwrap();
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
});
stun.received().await;
assert!(
pool.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::new())
.await
.is_none()
);
assert_eq!(timeout(TEST_DEADLINE, waiter).await.unwrap().unwrap(), None);
stun.release.add_permits(16);
assert_eq!(context.state.lock().await.reflection, None);
assert!(pool.nat_runtime.discovery.bound.lock().await.is_empty());
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn bound_failure_retry_deadline_expires_and_success_resets_backoff() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
stun.release.add_permits(16);
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let ip = IpAddr::V4(Ipv4Addr::LOCALHOST);
let context = pool
.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::from([ip]))
.await
.unwrap();
{
let mut state = context.state.lock().await;
state.retry_after = Some(Instant::now() + Duration::from_secs(60));
state.attempt = 3;
}
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_none()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 0);
context.state.lock().await.retry_after = Some(Instant::now() - Duration::from_secs(1));
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert_eq!(context.state.lock().await.attempt, 0);
assert_eq!(context.state.lock().await.retry_after, None);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
@@ -0,0 +1,299 @@
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, UdpSocket};
use tokio::sync::Semaphore;
use tokio::task::JoinHandle;
use super::*;
use crate::network::probe::NetworkDecision;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool_with_decision;
/// Public IPv4 used as source material by local discovery and encrypted ME fixtures.
pub(super) const PUBLIC_IP: Ipv4Addr = Ipv4Addr::new(91, 108, 56, 1);
/// Bounds asynchronous fixture assertions without relying on external services.
pub(super) const TEST_DEADLINE: Duration = Duration::from_secs(2);
// Concurrency and end-to-end regressions share gated local discovery servers.
mod freshness;
pub(super) mod lifecycle;
mod recovery;
/// Gated local discovery service with observable request counts.
pub(super) struct DiscoveryServer {
/// Listening address selected by the kernel.
pub(super) addr: SocketAddr,
/// Number of requests actually received from clients.
pub(super) requests: Arc<AtomicUsize>,
arrived: Arc<Semaphore>,
/// One permit releases one discovery response.
pub(super) release: Arc<Semaphore>,
task: JoinHandle<()>,
}
impl Drop for DiscoveryServer {
fn drop(&mut self) {
self.task.abort();
}
}
impl DiscoveryServer {
/// Waits until a real discovery request reaches the server.
pub(super) async fn received(&self) {
tokio::time::timeout(TEST_DEADLINE, self.arrived.acquire())
.await
.expect("discovery request must reach the local server")
.unwrap()
.forget();
}
}
/// Serves valid STUN responses with a deliberately different reflected port.
pub(super) async fn stun_server(bind: SocketAddr, reflected: IpAddr) -> DiscoveryServer {
stun_server_response(bind, Some(reflected), Duration::ZERO).await
}
async fn stun_server_response(
bind: SocketAddr,
reflected: Option<IpAddr>,
delay: Duration,
) -> DiscoveryServer {
let socket = UdpSocket::bind(bind).await.unwrap();
let addr = socket.local_addr().unwrap();
let requests = Arc::new(AtomicUsize::new(0));
let arrived = Arc::new(Semaphore::new(0));
let release = Arc::new(Semaphore::new(0));
let count = requests.clone();
let signal = arrived.clone();
let gate = release.clone();
let task = tokio::spawn(async move {
let mut request = [0u8; 512];
loop {
let (len, peer) = socket.recv_from(&mut request).await.unwrap();
assert!(len >= 20);
count.fetch_add(1, Ordering::Relaxed);
signal.add_permits(1);
gate.acquire().await.unwrap().forget();
tokio::time::sleep(delay).await;
let Some(reflected) = reflected else {
socket.send_to(&[0u8], peer).await.unwrap();
continue;
};
let (family, ip) = match reflected {
IpAddr::V4(ip) => (1u8, ip.octets().to_vec()),
IpAddr::V6(ip) => (2u8, ip.octets().to_vec()),
};
let attribute_len = 4 + ip.len();
let mut response = vec![0u8; 24 + attribute_len];
response[..2].copy_from_slice(&0x0101u16.to_be_bytes());
response[2..4].copy_from_slice(&((4 + attribute_len) as u16).to_be_bytes());
response[4..20].copy_from_slice(&request[4..20]);
response[20..22].copy_from_slice(&1u16.to_be_bytes());
response[22..24].copy_from_slice(&(attribute_len as u16).to_be_bytes());
response[25] = family;
response[26..28].copy_from_slice(&45678u16.to_be_bytes());
response[28..].copy_from_slice(&ip);
socket.send_to(&response, peer).await.unwrap();
}
});
DiscoveryServer {
addr,
requests,
arrived,
release,
task,
}
}
async fn http_server(body: &'static str) -> DiscoveryServer {
http_server_with_delay(body, Duration::ZERO).await
}
async fn http_server_with_delay(body: &'static str, delay: Duration) -> DiscoveryServer {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let requests = Arc::new(AtomicUsize::new(0));
let arrived = Arc::new(Semaphore::new(0));
let release = Arc::new(Semaphore::new(0));
let count = requests.clone();
let signal = arrived.clone();
let gate = release.clone();
let task = tokio::spawn(async move {
loop {
let (mut stream, _) = listener.accept().await.unwrap();
let mut request = [0u8; 4096];
let _ = stream.read(&mut request).await.unwrap();
count.fetch_add(1, Ordering::Relaxed);
signal.add_permits(1);
gate.acquire().await.unwrap().forget();
tokio::time::sleep(delay).await;
let response = format!(
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
body.len(),
body
);
let _ = stream.write_all(response.as_bytes()).await;
}
});
DiscoveryServer {
addr,
requests,
arrived,
release,
task,
}
}
/// Builds an isolated dual-stack pool using only caller-supplied discovery services.
pub(super) async fn nat_pool(servers: Vec<String>, urls: Vec<String>) -> Arc<MePool> {
let mut pool = make_pool_with_decision(NetworkDecision {
ipv4_me: true,
ipv6_me: true,
..NetworkDecision::default()
})
.await;
let pool_mut = Arc::get_mut(&mut pool).unwrap();
let nat = Arc::get_mut(&mut pool_mut.nat_runtime).unwrap();
nat.nat_probe = true;
nat.nat_stun_servers = servers;
nat.http_ip_detect_urls = urls;
nat.nat_probe_concurrency = 2;
pool
}
async fn wait_for_reflection(pool: &MePool, family: IpFamily) -> SocketAddr {
tokio::time::timeout(TEST_DEADLINE, async {
loop {
let cached = {
let cache = pool.nat_runtime.nat_reflection_cache.lock().await;
match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
}
};
if let Some((created, addr)) = cached
&& created.elapsed() < Duration::from_secs(600)
{
return addr;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.expect("pool-owned STUN refresh must publish after its caller is cancelled")
}
#[tokio::test]
async fn expired_stun_refresh_survives_reconnect_timeout() {
let fast = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let slow = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
fast.release.add_permits(16);
let pool = nat_pool(vec![fast.addr.to_string(), slow.addr.to_string()], vec![]).await;
pool.nat_runtime.nat_reflection_cache.lock().await.v4 = Some((
Instant::now() - Duration::from_secs(601),
SocketAddr::new(PUBLIC_IP.into(), 1),
));
let caller_pool = pool.clone();
let caller = tokio::spawn(async move {
tokio::time::timeout(
Duration::from_millis(100),
caller_pool.maybe_reflect_public_addr(IpFamily::V4, None),
)
.await
});
fast.received().await;
slow.received().await;
assert!(caller.await.unwrap().is_err());
slow.release.add_permits(16);
assert_eq!(
wait_for_reflection(&pool, IpFamily::V4).await.ip(),
PUBLIC_IP
);
assert_eq!(fast.requests.load(Ordering::Relaxed), 1);
assert_eq!(slow.requests.load(Ordering::Relaxed), 1);
assert_eq!(
pool.nat_runtime.nat_probe_attempts.load(Ordering::Relaxed),
0
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn http_discovery_survives_reconnect_timeout() {
let http = http_server("91.108.56.1").await;
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
let caller_pool = pool.clone();
let caller = tokio::spawn(async move {
tokio::time::timeout(
Duration::from_millis(100),
caller_pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()),
)
.await
});
http.received().await;
assert!(caller.await.unwrap().is_err());
http.release.add_permits(16);
tokio::time::timeout(TEST_DEADLINE, async {
while pool.nat_runtime.nat_ip_detected.read().await.is_none() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.expect("pool-owned HTTP discovery must publish after caller cancellation");
assert_eq!(
pool.nat_runtime
.nat_ip_detected
.read()
.await
.map(|observation| observation.ip),
Some(PUBLIC_IP.into())
);
assert_eq!(http.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn failed_http_discovery_is_reused_during_negative_cache_window() {
let http = http_server("not an IP address").await;
http.release.add_permits(16);
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
for _ in 0..2 {
assert!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into())
.await
.is_none()
);
}
assert_eq!(
http.requests.load(Ordering::Relaxed),
1,
"a failed HTTP lookup must not restart ahead of every STUN/handshake attempt"
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn successful_stun_publication_waits_for_cache_reader() {
let server = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(vec![server.addr.to_string()], vec![]).await;
let caller_pool = pool.clone();
let caller = tokio::spawn(async move {
caller_pool
.maybe_reflect_public_addr(IpFamily::V4, None)
.await
});
server.received().await;
let cache_reader = pool.nat_runtime.nat_reflection_cache.lock().await;
server.release.add_permits(16);
tokio::time::sleep(Duration::from_millis(50)).await;
drop(cache_reader);
assert_eq!(caller.await.unwrap().unwrap().ip(), PUBLIC_IP);
assert_eq!(
wait_for_reflection(&pool, IpFamily::V4).await.ip(),
PUBLIC_IP
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
@@ -0,0 +1,75 @@
use super::*;
#[tokio::test]
async fn failed_refresh_retains_original_age_but_never_expired_material() {
let http = http_server("invalid address").await;
http.release.add_permits(8);
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
let observed_at = tokio::time::Instant::now() - Duration::from_secs(601);
*pool.nat_runtime.nat_ip_detected.write().await = Some(HttpNatObservation {
ip: PUBLIC_IP.into(),
observed_at,
});
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
Some(PUBLIC_IP.into())
);
assert_eq!(
pool.nat_runtime
.nat_ip_detected
.read()
.await
.unwrap()
.observed_at,
observed_at
);
assert_eq!(http.requests.load(Ordering::Acquire), 1);
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
Some(PUBLIC_IP.into())
);
assert_eq!(
http.requests.load(Ordering::Acquire),
1,
"failure backoff remains singleflight"
);
pool.nat_runtime
.nat_ip_detected
.write()
.await
.as_mut()
.unwrap()
.observed_at = tokio::time::Instant::now() - Duration::from_secs(1201);
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
assert_eq!(
pool.translate_ip_for_nat(Ipv4Addr::LOCALHOST.into()),
IpAddr::V4(Ipv4Addr::LOCALHOST)
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn expired_http_observation_defers_handshake_before_sending_kdf_material() {
let pool = nat_pool(vec![], vec![]).await;
*pool.nat_runtime.nat_ip_detected.write().await = Some(HttpNatObservation {
ip: PUBLIC_IP.into(),
observed_at: tokio::time::Instant::now() - Duration::from_secs(1201),
});
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let stream = tokio::net::TcpStream::connect(addr).await.unwrap();
let (mut peer, _) = listener.accept().await.unwrap();
let error = pool
.handshake_only(stream, addr, None, pool.rng.as_ref())
.await
.err()
.unwrap();
assert!(
matches!(error, ProxyError::Io(ref error) if error.kind() == std::io::ErrorKind::WouldBlock)
);
assert_eq!(peer.read(&mut [0u8; 1]).await.unwrap(), 0);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
@@ -0,0 +1,342 @@
use super::*;
use crate::config::{UpstreamConfig, UpstreamType};
use crate::transport::UpstreamManager;
/// Gives an isolated pool the same finite bind-address authority as production egress.
pub(in crate::transport::middle_proxy::pool_nat) fn set_bind_addresses(
pool: &mut Arc<MePool>,
addresses: &[&str],
) {
let manager = UpstreamManager::new(
vec![UpstreamConfig {
upstream_type: UpstreamType::Direct {
interface: None,
bind_addresses: Some(addresses.iter().map(|value| value.to_string()).collect()),
bindtodevice: None,
},
weight: 1,
enabled: true,
scopes: String::new(),
selected_scope: String::new(),
ipv4: None,
ipv6: None,
prefer: None,
}],
1,
0,
1000,
1,
1,
true,
pool.stats.clone(),
);
Arc::get_mut(pool).unwrap().upstream = Some(Arc::new(manager));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn cancelled_waiter_storm_shares_one_http_and_stun_producer() {
let http = http_server("91.108.56.1").await;
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(
vec![stun.addr.to_string()],
vec![format!("http://{}/", http.addr)],
)
.await;
let mut waiters = JoinSet::new();
for _ in 0..32 {
let pool = pool.clone();
waiters.spawn(async move {
tokio::join!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()),
pool.maybe_reflect_public_addr(IpFamily::V4, None),
);
});
}
http.received().await;
stun.received().await;
waiters.abort_all();
while waiters.join_next().await.is_some() {}
http.release.add_permits(64);
stun.release.add_permits(64);
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
Some(PUBLIC_IP.into())
);
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, None)
.await
.unwrap()
.ip(),
PUBLIC_IP
);
assert_eq!(http.requests.load(Ordering::Relaxed), 1);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn shutdown_cancels_discovery_and_releases_all_waiters() {
let http = http_server("91.108.56.1").await;
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(
vec![stun.addr.to_string()],
vec![format!("http://{}/", http.addr)],
)
.await;
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
tokio::join!(
waiter_pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()),
waiter_pool.maybe_reflect_public_addr(IpFamily::V4, None),
)
});
http.received().await;
stun.received().await;
assert!(pool.shutdown_until(TEST_DEADLINE).await);
assert_eq!(
timeout(TEST_DEADLINE, waiter).await.unwrap().unwrap(),
(None, None)
);
http.release.add_permits(64);
stun.release.add_permits(64);
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, None).await,
None
);
assert_eq!(*pool.nat_runtime.nat_ip_detected.read().await, None);
assert_eq!(pool.nat_runtime.nat_reflection_cache.lock().await.v4, None);
assert_eq!(http.requests.load(Ordering::Relaxed), 1);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(
pool.nat_runtime
.nat_reflection_singleflight_v4
.try_lock()
.is_ok()
);
assert!(pool.nat_runtime.discovery.http.try_lock().is_ok());
}
#[tokio::test]
async fn bound_refresh_survives_cancellation_and_reuses_only_its_source() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1", "127.0.0.2"]);
let bind = Some(Ipv4Addr::LOCALHOST.into());
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, bind)
.await
});
stun.received().await;
waiter.abort();
assert!(waiter.await.unwrap_err().is_cancelled());
stun.release.add_permits(64);
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, bind)
.await
.unwrap()
.ip(),
PUBLIC_IP
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert_eq!(pool.nat_runtime.nat_reflection_cache.lock().await.v4, None);
let other = Some("127.0.0.2".parse().unwrap());
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, other)
.await
.unwrap()
.ip(),
PUBLIC_IP
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 2);
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some("127.0.0.3".parse().unwrap()))
.await
.is_none()
);
assert!(
pool.maybe_reflect_public_addr(IpFamily::V6, bind)
.await
.is_none()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 2);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn failed_bound_refresh_is_observable_after_caller_cancellation() {
let stun = stun_server_response("127.0.0.1:0".parse().unwrap(), None, Duration::ZERO).await;
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let bind = Some(Ipv4Addr::LOCALHOST.into());
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, bind)
.await
});
stun.received().await;
waiter.abort();
let _ = waiter.await;
// One response completes the original probe; a restarted probe would block forever here.
stun.release.add_permits(1);
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V4, bind)
)
.await
.unwrap(),
None
);
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V4, bind)
)
.await
.unwrap(),
None
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn http_negative_cache_expires_without_waiting_a_minute() {
let http = http_server("invalid").await;
http.release.add_permits(64);
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
*pool.nat_runtime.discovery.http.lock().await = Some(Instant::now() - Duration::from_secs(1));
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
assert_eq!(http.requests.load(Ordering::Relaxed), 2);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn ipv6_refresh_is_independent_of_blocked_ipv4_refresh() {
let v4 = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let public_v6: IpAddr = "2001:4860:4860::8888".parse().unwrap();
let v6 = stun_server("[::1]:0".parse().unwrap(), public_v6).await;
v6.release.add_permits(16);
let pool = nat_pool(vec![v4.addr.to_string(), v6.addr.to_string()], vec![]).await;
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, None)
.await
});
v4.received().await;
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V6, None)
)
.await
.unwrap()
.unwrap()
.ip(),
public_v6
);
assert!(!waiter.is_finished());
v4.release.add_permits(16);
assert_eq!(waiter.await.unwrap().unwrap().ip(), PUBLIC_IP);
assert_eq!(
pool.nat_runtime
.nat_reflection_cache
.lock()
.await
.v4
.unwrap()
.1
.ip(),
PUBLIC_IP
);
assert_eq!(
pool.nat_runtime
.nat_reflection_cache
.lock()
.await
.v6
.unwrap()
.1
.ip(),
public_v6
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn failed_shared_refresh_publishes_backoff_after_caller_cancellation() {
let stun = stun_server_response("127.0.0.1:0".parse().unwrap(), None, Duration::ZERO).await;
let pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, None)
.await
});
stun.received().await;
waiter.abort();
let _ = waiter.await;
stun.release.add_permits(1);
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V4, None)
)
.await
.unwrap(),
None
);
assert!(pool.nat_runtime.stun_backoff_until.read().await.unwrap() > Instant::now());
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, None).await,
None
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn shutdown_before_producer_first_poll_releases_owned_gates() {
let http = http_server("91.108.56.1").await;
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(
vec![stun.addr.to_string()],
vec![format!("http://{}/", http.addr)],
)
.await;
let mut http_waiter =
tokio_test::task::spawn(pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()));
let mut stun_waiter =
tokio_test::task::spawn(pool.maybe_reflect_public_addr(IpFamily::V4, None));
assert!(http_waiter.poll().is_pending());
assert!(stun_waiter.poll().is_pending());
// On this current-thread runtime neither producer can run before shutdown admission closes.
pool.begin_shutdown();
assert!(pool.shutdown_until(TEST_DEADLINE).await);
assert_eq!(http_waiter.poll(), std::task::Poll::Ready(None));
assert_eq!(stun_waiter.poll(), std::task::Poll::Ready(None));
assert_eq!(http.requests.load(Ordering::Relaxed), 0);
assert_eq!(stun.requests.load(Ordering::Relaxed), 0);
assert!(pool.nat_runtime.discovery.http.try_lock().is_ok());
assert!(
pool.nat_runtime
.nat_reflection_singleflight_v4
.try_lock()
.is_ok()
);
}
@@ -0,0 +1,331 @@
use super::*;
use crate::crypto::{SecureRandom, derive_middleproxy_keys};
use crate::transport::middle_proxy::codec::{
RpcChecksumMode, build_handshake_payload, build_nonce_payload, build_rpc_frame,
cbc_decrypt_inplace, cbc_encrypt_padded, parse_handshake_flags, parse_nonce_payload,
read_rpc_frame_plaintext, rpc_crc,
};
use crate::transport::middle_proxy::me_health_monitor;
use crate::transport::middle_proxy::pool::MeFamilyRuntimeState;
use tokio::net::TcpStream;
struct MiddlePeer {
addr: SocketAddr,
handshakes: Arc<AtomicUsize>,
cancelled: Arc<AtomicUsize>,
task: JoinHandle<()>,
}
impl Drop for MiddlePeer {
fn drop(&mut self) {
self.task.abort();
}
}
impl MiddlePeer {
async fn new(server_ip: Ipv4Addr) -> Self {
Self::with_bind("127.0.0.1:0", server_ip.into(), PUBLIC_IP.into()).await
}
async fn with_bind(bind: &str, server_ip: IpAddr, client_ip: IpAddr) -> Self {
let listener = TcpListener::bind(bind).await.unwrap();
let addr = listener.local_addr().unwrap();
let handshakes = Arc::new(AtomicUsize::new(0));
let cancelled = Arc::new(AtomicUsize::new(0));
let succeeded = handshakes.clone();
let closed = cancelled.clone();
let task = tokio::spawn(async move {
let mut pending = JoinSet::new();
let mut established = Vec::new();
loop {
tokio::select! {
accepted = listener.accept() => {
let (stream, _) = accepted.unwrap();
pending.spawn(accept_handshake(stream, server_ip, client_ip));
}
completed = pending.join_next(), if !pending.is_empty() => {
match completed.unwrap().expect("fake ME handshake task must not panic") {
Some(stream) => {
established.push(stream);
succeeded.fetch_add(1, Ordering::Release);
}
None => { closed.fetch_add(1, Ordering::Release); }
}
}
}
}
});
Self {
addr,
handshakes,
cancelled,
task,
}
}
}
async fn accept_handshake(
mut stream: TcpStream,
server_ip: IpAddr,
client_ip: IpAddr,
) -> Option<TcpStream> {
let client_port = stream.peer_addr().unwrap().port();
let server_port = stream.local_addr().unwrap().port();
let (seq, payload) = match read_rpc_frame_plaintext(&mut stream).await {
Ok(frame) => frame,
// Reconnect may time out during NAT discovery, before sending any nonce.
Err(ProxyError::Io(error)) if error.kind() == std::io::ErrorKind::UnexpectedEof => {
return None;
}
Err(error) => panic!("invalid client nonce: {error}"),
};
assert_eq!(seq, -2);
let (selector, _, timestamp, client_nonce) = parse_nonce_payload(&payload).unwrap();
let server_nonce = [0x42u8; 16];
let nonce = build_nonce_payload(selector, timestamp, &server_nonce);
stream
.write_all(&build_rpc_frame(-2, &nonce, RpcChecksumMode::Crc32))
.await
.unwrap();
let split = |ip: IpAddr| match ip {
IpAddr::V4(ip) => {
let mut octets = ip.octets();
octets.reverse();
(Some(octets), None)
}
IpAddr::V6(ip) => (None, Some(ip.octets())),
};
let (client_v4, client_v6) = split(client_ip);
let (server_v4, server_v6) = split(server_ip);
let client_ip = client_v4.unwrap_or([0; 4]);
let server_ip = server_v4.unwrap_or([0; 4]);
let (read_key, read_iv) = derive_middleproxy_keys(
&server_nonce,
&client_nonce,
&timestamp.to_le_bytes(),
server_v4.as_ref().map(|ip| ip.as_slice()),
&client_port.to_le_bytes(),
b"CLIENT",
client_v4.as_ref().map(|ip| ip.as_slice()),
&server_port.to_le_bytes(),
&[1u8; 32],
client_v6.as_ref(),
server_v6.as_ref(),
);
let (write_key, write_iv) = derive_middleproxy_keys(
&server_nonce,
&client_nonce,
&timestamp.to_le_bytes(),
server_v4.as_ref().map(|ip| ip.as_slice()),
&client_port.to_le_bytes(),
b"SERVER",
client_v4.as_ref().map(|ip| ip.as_slice()),
&server_port.to_le_bytes(),
&[1u8; 32],
client_v6.as_ref(),
server_v6.as_ref(),
);
let mut encrypted = [0u8; 48];
stream.read_exact(&mut encrypted).await.unwrap();
cbc_decrypt_inplace(&read_key, &read_iv, &mut encrypted).unwrap();
assert_eq!(u32::from_le_bytes(encrypted[..4].try_into().unwrap()), 44);
assert_eq!(i32::from_le_bytes(encrypted[4..8].try_into().unwrap()), -1);
assert_eq!(
u32::from_le_bytes(encrypted[40..44].try_into().unwrap()),
rpc_crc(RpcChecksumMode::Crc32, &encrypted[..40])
);
parse_handshake_flags(&encrypted[8..40]).unwrap();
assert_eq!(&encrypted[16..20], &client_ip);
assert_eq!(
u16::from_le_bytes(encrypted[20..22].try_into().unwrap()),
client_port
);
assert_eq!(
u16::from_le_bytes(encrypted[32..34].try_into().unwrap()),
server_port
);
let reply = build_handshake_payload(server_ip, server_port, client_ip, client_port, 0);
let (encrypted, _) = cbc_encrypt_padded(
&write_key,
&write_iv,
&build_rpc_frame(-1, &reply, RpcChecksumMode::Crc32),
)
.unwrap();
stream.write_all(&encrypted).await.unwrap();
Some(stream)
}
async fn suppressed_pool_recovers_after_delayed_discovery(http_slow: bool) {
// Every lookup exceeds me_one_timeout_ms, not just the first request of the outage.
let delay = Duration::from_millis(1500);
let stun = stun_server_response(
"127.0.0.1:0".parse().unwrap(),
Some(PUBLIC_IP.into()),
delay,
)
.await;
let http = http_server_with_delay("91.108.56.1", delay).await;
stun.release.add_permits(64);
http.release.add_permits(64);
// A valid STUN source bypasses HTTP; the peer remains the configured loopback endpoint.
let peer = MiddlePeer::new(Ipv4Addr::LOCALHOST).await;
let urls = if http_slow {
vec![format!("http://{}/", http.addr)]
} else {
vec![]
};
let pool = nat_pool(vec![stun.addr.to_string()], urls).await;
pool.health_runtime
.me_health_interval_ms_unhealthy
.store(10, Ordering::Relaxed);
pool.health_runtime
.me_health_interval_ms_healthy
.store(10, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_outage_backoff_min_ms
.store(100, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_outage_backoff_max_ms
.store(100, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_shadow_writers
.store(0, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_shadow_rotate_every_secs
.store(0, Ordering::Relaxed);
pool.update_proxy_maps(
HashMap::from([(2, vec![(peer.addr.ip(), peer.addr.port())])]),
None,
)
.await;
let generation = pool.current_generation();
let now = MePool::now_epoch_secs();
pool.set_family_runtime_state(
IpFamily::V4,
MeFamilyRuntimeState::Suppressed,
now,
now + 3600,
5,
0,
);
pool.nat_runtime.nat_reflection_cache.lock().await.v4 = Some((
if http_slow {
Instant::now()
} else {
Instant::now() - Duration::from_secs(601)
},
SocketAddr::new(PUBLIC_IP.into(), 45678),
));
let monitor = tokio::spawn(me_health_monitor(
pool.clone(),
Arc::new(SecureRandom::new()),
0,
));
if !http_slow {
stun.received().await;
timeout(Duration::from_secs(5), async {
while peer.cancelled.load(Ordering::Acquire) == 0 {
assert!(!peer.task.is_finished());
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("the actual 1200ms reconnect must cancel a pre-KDF TCP connection");
assert_eq!(peer.handshakes.load(Ordering::Acquire), 0);
}
let expected = 2 * pool.required_writers_for_dc(1);
timeout(Duration::from_secs(10), async {
while !pool.admission_ready_full_floor().await
|| peer.handshakes.load(Ordering::Acquire) < expected
{
assert!(
!peer.task.is_finished(),
"fake ME must validate every encrypted handshake"
);
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("suppressed pool must autonomously regain its full floor without restart");
monitor.abort();
let _ = monitor.await;
assert_eq!(pool.current_generation(), generation);
assert_eq!(pool.reinit.status.load().pending_hardswap_generation, 0);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert_eq!(pool.writers.read().await.len(), expected);
assert!(peer.handshakes.load(Ordering::Acquire) >= expected);
assert_eq!(http.requests.load(Ordering::Relaxed), 0);
assert_eq!(
stun.requests.load(Ordering::Relaxed),
usize::from(!http_slow)
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn suppressed_pool_recovers_full_floor_after_expired_stun_refresh() {
suppressed_pool_recovers_after_delayed_discovery(false).await;
}
#[tokio::test]
async fn suppressed_pool_recovers_full_floor_after_slow_http_with_fresh_stun() {
suppressed_pool_recovers_after_delayed_discovery(true).await;
}
#[tokio::test]
async fn stalled_ipv4_does_not_delay_real_ipv6_handshakes() {
let stalled = TcpListener::bind("127.0.0.1:0").await.unwrap();
let stalled_addr = stalled.local_addr().unwrap();
let healthy =
MiddlePeer::with_bind("[::1]:0", "::1".parse().unwrap(), "::1".parse().unwrap()).await;
let mut pool = make_pool_with_decision(NetworkDecision {
ipv4_me: true,
ipv6_me: true,
..NetworkDecision::default()
})
.await;
Arc::get_mut(&mut Arc::get_mut(&mut pool).unwrap().reconnect_runtime)
.unwrap()
.me_one_timeout = Duration::from_secs(5);
pool.health_runtime
.me_health_interval_ms_unhealthy
.store(10, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_shadow_rotate_every_secs
.store(0, Ordering::Relaxed);
pool.update_proxy_maps(
HashMap::from([(2, vec![(stalled_addr.ip(), stalled_addr.port())])]),
Some(HashMap::from([(
2,
vec![(healthy.addr.ip(), healthy.addr.port())],
)])),
)
.await;
let monitor = tokio::spawn(me_health_monitor(pool.clone(), pool.rng.clone(), 0));
let (blocked_stream, _) = timeout(TEST_DEADLINE, stalled.accept())
.await
.unwrap()
.unwrap();
timeout(TEST_DEADLINE, async {
while healthy.handshakes.load(Ordering::Acquire) < pool.required_writers_for_dc(1) {
assert!(!healthy.task.is_finished());
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("IPv6 must restore coverage while IPv4 still owns a stalled handshake");
monitor.abort();
let _ = monitor.await;
assert!(
pool.shutdown_until(TEST_DEADLINE).await,
"tracked child transports must join on shutdown"
);
drop(blocked_stream);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
@@ -491,7 +491,11 @@ impl MePool {
rand::rng().random_range(min_ms..=max_ms)
}
pub(super) fn hardswap_warmup_backoff_ms(&self, pass_idx: usize) -> u64 {
/// Returns capped exponential retry delay with bounded positive jitter.
pub(in crate::transport::middle_proxy) fn hardswap_warmup_backoff_ms(
&self,
pass_idx: usize,
) -> u64 {
let base_ms = self
.reinit
.me_hardswap_warmup_pass_backoff_base_ms
@@ -169,11 +169,21 @@ impl MePool {
}
}
/// Reconciles one immutable endpoint snapshot without draining uncovered groups.
pub async fn zero_downtime_reinit_after_map_change(
self: &Arc<Self>,
rng: &SecureRandom,
) -> bool {
let endpoint_snapshot = self.endpoint_snapshot.load_full();
self.reinit_endpoint_snapshot(rng, endpoint_snapshot).await
}
/// Reconciles exactly the revision owned by the scheduler's retry objective.
pub(in crate::transport::middle_proxy) async fn reinit_endpoint_snapshot(
self: &Arc<Self>,
rng: &SecureRandom,
endpoint_snapshot: Arc<EndpointSnapshot>,
) -> bool {
let desired_by_dc = self.desired_dc_endpoints_from_snapshot(&endpoint_snapshot);
let now_epoch_secs = Self::now_epoch_secs();
let v4_suppressed = self.is_family_temporarily_suppressed(IpFamily::V4, now_epoch_secs);
@@ -407,6 +417,7 @@ impl MePool {
true
}
/// Applies periodic rotation using the same coverage and generation fences as map changes.
pub async fn zero_downtime_reinit_periodic(self: &Arc<Self>, rng: &SecureRandom) -> bool {
self.zero_downtime_reinit_after_map_change(rng).await
}
@@ -149,6 +149,7 @@ fn stale_concurrent_attempt_cannot_regress_active_generation() {
active_generation: 1,
desired_map_hash: 22,
endpoint_revision: 7,
floor_policy_revision: 1,
pending: Some(ReinitPendingState {
generation: 3,
started_at_epoch_secs: 1,
+16 -2
View File
@@ -32,15 +32,21 @@ use super::wire::build_proxy_req_payload;
mod publication;
mod replacement;
mod runtime;
// Physical replacement overlap survives logical writer removal until transport destruction.
pub(super) mod overlap;
use overlap::WriterTransport;
// Recovery regressions exercise the serialized production publication boundary.
#[cfg(test)]
mod recovery_tests;
struct PreparedWriter<'a> {
writer: MeWriter,
tx: mpsc::Sender<WriterCommand>,
byte_budget: Arc<tokio::sync::Semaphore>,
task_registration: MeTaskRegistration<'a>,
writer_task: Pin<Box<dyn Future<Output = ()> + Send + 'static>>,
writer_task: WriterTransport,
intent: WriterOpenIntent,
_open_reservation: WriterOpenReservation<'a>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -55,6 +61,13 @@ pub(super) enum WriterReplacementPurpose {
/// Target count above which the receiver no longer needs the slot.
receiver_floor: usize,
},
/// Transfers one slot without taking a current donor below its protected base floor.
CoverageTransfer {
/// Endpoint and floor-policy authority captured before selecting the donor.
authority: (u64, u64),
/// Planned receiver target, revalidated against current policy at commit.
receiver_floor: usize,
},
/// Replaces a stale-generation writer with an active-generation writer.
GenerationConvergence,
/// Rotates a single-endpoint shadow while retaining coverage.
@@ -68,6 +81,7 @@ impl WriterReplacementPurpose {
match self {
Self::IdleRefresh => "idle_refresh",
Self::FloorRebalance { .. } => "floor_rebalance",
Self::CoverageTransfer { .. } => "coverage_transfer",
Self::GenerationConvergence => "generation_convergence",
Self::ShadowRotation => "shadow_rotation",
Self::SecretRotation => "secret_rotation",
@@ -0,0 +1,87 @@
use super::*;
use std::task::{Context, Poll};
/// Writer-local physical lifetime, independent of registry visibility.
pub(in crate::transport::middle_proxy) struct ReplacementHandoff {
state: parking_lot::Mutex<HandoffState>,
}
enum HandoffState {
Open(Option<WriterOpenReservation>),
Closed,
}
impl ReplacementHandoff {
/// Creates a handoff that may accept one physical-overlap token.
pub(in crate::transport::middle_proxy) fn new() -> Arc<Self> {
Arc::new(Self {
state: parking_lot::Mutex::new(HandoffState::Open(None)),
})
}
fn close(&self) {
*self.state.lock() = HandoffState::Closed;
}
/// Keeps attachment reversible until the final registry validation has succeeded.
pub(super) fn commit(
&self,
permit: &mut Option<WriterOpenReservation>,
validate: impl FnOnce() -> bool,
) -> bool {
let mut state = self.state.lock();
match &mut *state {
HandoffState::Open(slot) if slot.is_none() && permit.is_some() && validate() => {
*slot = permit.take();
true
}
_ => false,
}
}
}
/// Destroys transport futures before releasing either opening or retiring capacity.
pub(super) struct WriterTransport {
future: Option<Pin<Box<dyn Future<Output = ()> + Send + 'static>>>,
/// Shared with the registry but closed only by transport destruction.
pub(super) lifetime: Arc<ReplacementHandoff>,
/// Moves to the victim on replacement or is released after normal publication.
pub(super) opening: Option<WriterOpenReservation>,
}
impl WriterTransport {
/// Owns prepared transports even if their task is never polled or published.
pub(super) fn new(
future: Pin<Box<dyn Future<Output = ()> + Send + 'static>>,
opening: WriterOpenReservation,
) -> Self {
Self {
future: Some(future),
lifetime: ReplacementHandoff::new(),
opening: Some(opening),
}
}
}
impl Future for WriterTransport {
type Output = ();
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<()> {
match self.future.as_mut() {
Some(future) => future.as_mut().poll(cx),
None => Poll::Ready(()),
}
}
}
impl Drop for WriterTransport {
fn drop(&mut self) {
// Explicit order also applies to unpolled futures, panic unwinding and abort.
drop(self.future.take());
self.lifetime.close();
drop(self.opening.take());
}
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,238 @@
use super::*;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool;
use std::sync::atomic::AtomicUsize;
struct DestructionProbe {
counter: Arc<AtomicUsize>,
destroyed: Arc<AtomicBool>,
}
impl Drop for DestructionProbe {
fn drop(&mut self) {
assert!(
self.counter.load(Ordering::Acquire) > 0,
"transport must be destroyed before its overlap token"
);
self.destroyed.store(true, Ordering::Release);
}
}
async fn transport(pool: &Arc<MePool>, destroyed: Arc<AtomicBool>) -> WriterTransport {
let permit = pool
.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
2,
"127.0.0.1:443".parse().unwrap(),
)
.await
.unwrap();
let probe = DestructionProbe {
counter: pool.writer_replacement_open_reserved.clone(),
destroyed,
};
WriterTransport::new(
Box::pin(async move {
let _probe = probe;
std::future::pending::<()>().await;
}),
permit,
)
}
#[tokio::test]
async fn unpolled_transport_and_aborted_task_release_after_destruction() {
let pool = make_pool().await;
for abort in [false, true] {
let destroyed = Arc::new(AtomicBool::new(false));
let task = transport(&pool, destroyed.clone()).await;
if abort {
let task = tokio::spawn(task);
task.abort();
let _ = task.await;
} else {
drop(task);
}
assert!(destroyed.load(Ordering::Acquire));
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
0
);
}
}
#[tokio::test]
async fn retiring_transport_owns_capacity_after_logical_removal() {
let pool = make_pool().await;
let destroyed = Arc::new(AtomicBool::new(false));
let mut victim = transport(&pool, destroyed.clone()).await;
drop(victim.opening.take());
let lifetime = victim.lifetime.clone();
let mut successor = transport(&pool, Arc::new(AtomicBool::new(false))).await;
successor.future = Some(Box::pin(std::future::pending()));
assert!(lifetime.commit(&mut successor.opening, || true));
assert!(successor.opening.is_none());
// Registry removal only drops its handle; the real transport retains the token.
drop(lifetime);
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
1
);
drop(victim);
assert!(destroyed.load(Ordering::Acquire));
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
0
);
// Successor is no longer overlap after its victim has physically gone.
drop(successor);
}
#[tokio::test]
async fn failed_or_closed_handoff_keeps_the_prepared_token() {
let pool = make_pool().await;
let mut prepared = transport(&pool, Arc::new(AtomicBool::new(false))).await;
let lifetime = ReplacementHandoff::new();
assert!(!lifetime.commit(&mut prepared.opening, || false));
assert!(prepared.opening.is_some());
lifetime.close();
assert!(!lifetime.commit(&mut prepared.opening, || true));
assert!(prepared.opening.is_some());
drop(prepared);
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn overlap_limit_survives_commit_and_cap_shrink_until_teardown() {
let pool = make_pool().await;
pool.floor_runtime
.me_adaptive_floor_cpu_cores_override
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_per_core
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_global
.store(1, Ordering::Relaxed);
pool.set_adaptive_floor_runtime_caps(pool.floor_authority(), 1, 2, 1, 1, 1, 0, 0);
let mut victim = transport(&pool, Arc::new(AtomicBool::new(false))).await;
drop(victim.opening.take());
let mut successor = transport(&pool, Arc::new(AtomicBool::new(false))).await;
successor.future = Some(Box::pin(std::future::pending()));
assert!(victim.lifetime.commit(&mut successor.opening, || true));
let second = transport(&pool, Arc::new(AtomicBool::new(false))).await;
pool.set_adaptive_floor_runtime_caps(pool.floor_authority(), 1, 1, 1, 1, 1, 1, 0);
let addr = "127.0.0.1:443".parse().unwrap();
assert!(
pool.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
2,
addr
)
.await
.is_none()
);
drop(second);
assert!(
pool.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
2,
addr
)
.await
.is_none()
);
drop(victim);
assert!(
pool.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
2,
addr
)
.await
.is_some()
);
drop(successor);
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn panicking_transport_releases_capacity_after_unwinding_its_resources() {
let pool = make_pool().await;
let permit = pool
.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
2,
"127.0.0.1:443".parse().unwrap(),
)
.await
.unwrap();
let destroyed = Arc::new(AtomicBool::new(false));
let probe = DestructionProbe {
counter: pool.writer_replacement_open_reserved.clone(),
destroyed: destroyed.clone(),
};
let task = WriterTransport::new(
Box::pin(async move {
let _probe = probe;
panic!("injected transport panic");
}),
permit,
);
assert!(tokio::spawn(task).await.unwrap_err().is_panic());
assert!(destroyed.load(Ordering::Acquire));
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
0
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn racing_commit_and_close_release_each_token_exactly_once() {
let pool = make_pool().await;
for _ in 0..128 {
let permit = pool
.reserve_writer_open(
WriterContour::Active,
WriterOpenIntent::Replacement,
2,
"127.0.0.1:443".parse().unwrap(),
)
.await
.unwrap();
let lifetime = ReplacementHandoff::new();
let closing = lifetime.clone();
let close = tokio::spawn(async move {
tokio::task::yield_now().await;
closing.close();
});
let commit = tokio::spawn(async move {
let mut permit = Some(permit);
lifetime.commit(&mut permit, || true);
drop(permit);
});
close.await.unwrap();
commit.await.unwrap();
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
0
);
}
}
@@ -11,9 +11,8 @@ impl MePool {
tx,
byte_budget,
task_registration,
writer_task,
mut writer_task,
intent,
_open_reservation,
} = prepared;
let writer_id = writer.id;
let mut writers = self.writers.write().await;
@@ -22,9 +21,15 @@ impl MePool {
let contour = self.authorize_writer_publication(&writer, &coordinator)?;
self.authorize_writer_publication_capacity(&writer, contour, intent, writers.as_slice())?;
writer.contour.store(contour.as_u8(), Ordering::Release);
registry_registration.install(writer_id, tx, byte_budget);
registry_registration.install_with_lifetime(
writer_id,
tx,
byte_budget,
writer_task.lifetime.clone(),
);
writers.push(writer);
self.conn_count.fetch_add(1, Ordering::Relaxed);
drop(writer_task.opening.take());
self.lifecycle
.spawn_registered_writer(task_registration, writer_task);
drop(coordinator);
@@ -84,7 +89,17 @@ impl MePool {
crate::network::IpFamily::V6
};
let now_epoch_secs = Self::now_epoch_secs();
if !self.family_enabled_for_drain_coverage(family, now_epoch_secs) {
let configured = match family {
crate::network::IpFamily::V4 => self.decision.ipv4_me,
crate::network::IpFamily::V6 => self.decision.ipv6_me,
};
// Recovery coverage must remain publishable during suppression; the serialized
// per-group floor check below still bounds it and generation authority is unchanged.
let recovery_coverage =
contour == WriterContour::Active && intent == WriterOpenIntent::Coverage;
if !configured
|| (self.is_family_temporarily_suppressed(family, now_epoch_secs) && !recovery_coverage)
{
return Err(ProxyError::Proxy(
"ME writer family lost publication authority".into(),
));
@@ -0,0 +1,468 @@
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use super::*;
use crate::network::{IpFamily, probe::NetworkDecision};
use crate::transport::middle_proxy::admission_test_support::unregistered_writer;
use crate::transport::middle_proxy::pool::MeFamilyRuntimeState;
use crate::transport::middle_proxy::pool_writer_security_tests::{
make_pool, make_pool_with_decision,
};
fn cap_at_one(pool: &MePool) {
pool.floor_runtime
.me_adaptive_floor_cpu_cores_override
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_per_core
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_global
.store(1, Ordering::Relaxed);
}
fn suppress(pool: &MePool, family: IpFamily) {
let now = MePool::now_epoch_secs();
pool.set_family_runtime_state(
family,
MeFamilyRuntimeState::Suppressed,
now,
now + 3600,
5,
0,
);
}
fn prepared<'a>(
pool: &'a Arc<MePool>,
writer: MeWriter,
intent: WriterOpenIntent,
reservation: WriterOpenReservation,
task_started: Arc<AtomicBool>,
) -> PreparedWriter<'a> {
let cancel = writer.cancel.clone();
PreparedWriter {
tx: writer.tx.clone(),
byte_budget: writer.byte_budget.clone(),
writer,
task_registration: pool.lifecycle.try_register().unwrap(),
writer_task: WriterTransport::new(
Box::pin(async move {
task_started.store(true, Ordering::Release);
cancel.cancelled().await;
}),
reservation,
),
intent,
}
}
#[tokio::test]
async fn suppressed_active_coverage_publishes_only_the_missing_floor() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
let required = pool.required_writers_for_dc(1);
let mut receivers = Vec::new();
for id in 1..=required as u64 {
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.expect("missing coverage must have a reservation despite suppression");
let (writer, receiver) = unregistered_writer(
&pool,
id,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
receivers.push(receiver);
pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
Arc::new(AtomicBool::new(false)),
))
.await
.expect("authenticated coverage must publish without waiting for suppression expiry");
assert_eq!(pool.conn_count.load(Ordering::Relaxed), id as usize);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
assert!(
pool.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.is_none()
);
let (extra, _receiver) = unregistered_writer(
&pool,
99,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
assert!(
pool.authorize_writer_publication_capacity(
&extra,
WriterContour::Active,
WriterOpenIntent::Coverage,
&pool.writers.read().await,
)
.is_err()
);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Suppressed
);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn concurrent_suppressed_coverage_publications_cannot_overfill_a_group() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
let required = pool.required_writers_for_dc(1);
let mut receivers = Vec::new();
let mut publications = Vec::new();
for id in 1..=(required + 1) as u64 {
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, receiver) = unregistered_writer(
&pool,
id,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
receivers.push(receiver);
publications.push(pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
Arc::new(AtomicBool::new(false)),
)));
}
let guard = pool.writers.write().await;
let results = futures::future::join_all(publications);
tokio::pin!(results);
tokio::select! {
biased;
results = &mut results => panic!("publication bypassed the writer barrier: {results:?}"),
_ = tokio::task::yield_now() => {}
}
drop(guard);
let results = results.await;
assert_eq!(
results.iter().filter(|result| result.is_ok()).count(),
required
);
assert_eq!(results.iter().filter(|result| result.is_err()).count(), 1);
assert_eq!(pool.writers.read().await.len(), required);
assert_eq!(pool.conn_count.load(Ordering::Relaxed), required);
assert_eq!(
pool.registry.writer_idle_since_snapshot().await.len(),
required
);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn committed_warm_writer_uses_resolved_active_coverage_authority() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
let reservation = pool
.reserve_writer_open(WriterContour::Warm, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
addr,
pool.current_generation(),
WriterContour::Warm,
);
pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
Arc::new(AtomicBool::new(false)),
))
.await
.expect("a committed generation must resolve warm coverage to its active role");
assert_eq!(
pool.writers.read().await[0].contour.load(Ordering::Acquire),
WriterContour::Active.as_u8()
);
assert_eq!(pool.writer_connect_warm_reserved.load(Ordering::Acquire), 0);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn suppressed_families_keep_their_full_coverage_reservation_budget() {
let pool = make_pool_with_decision(NetworkDecision {
ipv4_me: true,
ipv6_me: true,
..NetworkDecision::default()
})
.await;
let v4 = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
let v6 = SocketAddr::new(IpAddr::V6(Ipv6Addr::LOCALHOST), 443);
pool.update_proxy_maps(
HashMap::from([(2, vec![(v4.ip(), v4.port())])]),
Some(HashMap::from([(2, vec![(v6.ip(), v6.port())])])),
)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
suppress(&pool, IpFamily::V6);
let required = pool.required_writers_for_dc(1);
assert_eq!(pool.active_coverage_required_total().await, required * 4);
let mut receivers = Vec::new();
let mut writers = pool.writers.write().await;
for (index, (dc, addr)) in [(2, v4), (-2, v4), (-2, v6)].into_iter().enumerate() {
for offset in 0..required {
let (writer, receiver) = unregistered_writer(
&pool,
(index * required + offset + 1) as u64,
dc,
addr,
pool.current_generation(),
WriterContour::Active,
);
writers.push(writer);
receivers.push(receiver);
}
}
drop(writers);
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, v6)
.await
.expect("other covered groups must not consume the suppressed family's floor budget");
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
1
);
drop(reservation);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn coverage_reservations_remain_bounded_and_release_on_drop() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
let limit = pool.required_writers_for_dc(1) * 2
+ pool
.reconnect_runtime
.me_reconnect_max_concurrent_per_dc
.max(1) as usize;
let mut reservations = Vec::new();
for _ in 0..limit {
reservations.push(
pool.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.expect("the configured coverage budget must remain available"),
);
}
assert!(
pool.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr,)
.await
.is_none()
);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
limit
);
drop(reservations);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn suppression_exception_does_not_admit_growth_replacement_or_pending_warm() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
for (contour, intent) in [
(WriterContour::Active, WriterOpenIntent::Normal),
(WriterContour::Active, WriterOpenIntent::Replacement),
(WriterContour::Warm, WriterOpenIntent::Coverage),
(WriterContour::Warm, WriterOpenIntent::Normal),
] {
let (writer, _receiver) =
unregistered_writer(&pool, 1, 2, addr, pool.current_generation(), contour);
assert!(
pool.authorize_writer_publication_capacity(&writer, contour, intent, &[])
.is_err()
);
}
}
#[tokio::test]
async fn coverage_exception_never_enables_a_disabled_family() {
let pool = make_pool().await;
let v4 = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
let v6 = SocketAddr::new(IpAddr::V6(Ipv6Addr::LOCALHOST), 443);
pool.update_proxy_maps(
HashMap::from([(2, vec![(v4.ip(), v4.port())])]),
Some(HashMap::from([(2, vec![(v6.ip(), v6.port())])])),
)
.await;
suppress(&pool, IpFamily::V6);
assert_eq!(
pool.active_coverage_required_total().await,
pool.required_writers_for_dc(1) * 2
);
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
v6,
pool.current_generation(),
WriterContour::Active,
);
assert!(
pool.authorize_writer_publication_capacity(
&writer,
WriterContour::Active,
WriterOpenIntent::Coverage,
&[],
)
.is_err()
);
}
#[tokio::test]
async fn suppressed_coverage_cannot_publish_a_stale_generation() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
addr,
pool.current_generation().saturating_sub(1),
WriterContour::Active,
);
let task_started = Arc::new(AtomicBool::new(false));
let error = pool
.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
task_started.clone(),
))
.await
.unwrap_err();
assert!(
error
.to_string()
.contains("generation lost publication authority")
);
assert!(pool.writers.read().await.is_empty());
assert!(pool.registry.writer_idle_since_snapshot().await.is_empty());
assert_eq!(pool.conn_count.load(Ordering::Relaxed), 0);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert!(!task_started.load(Ordering::Acquire));
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn suppressed_coverage_rechecks_endpoints_after_publication_waits() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
let task_started = Arc::new(AtomicBool::new(false));
let publication = pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
task_started.clone(),
));
tokio::pin!(publication);
let guard = pool.writers.write().await;
tokio::select! {
biased;
result = &mut publication => panic!("publication bypassed the writer barrier: {result:?}"),
_ = tokio::task::yield_now() => {}
}
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), 444)])]), None)
.await;
drop(guard);
let error = publication.await.unwrap_err();
assert!(
error
.to_string()
.contains("target changed before publication")
);
assert!(pool.writers.read().await.is_empty());
assert!(pool.registry.writer_idle_since_snapshot().await.is_empty());
assert_eq!(pool.conn_count.load(Ordering::Relaxed), 0);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert!(!task_started.load(Ordering::Acquire));
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
@@ -44,9 +44,8 @@ impl MePool {
tx,
byte_budget,
task_registration,
writer_task,
intent,
_open_reservation,
mut writer_task,
intent: _,
} = prepared;
let replacement_writer_id = writer.id;
let victim_writer_id = reservation.writer_id();
@@ -57,7 +56,25 @@ impl MePool {
let mut registry_registration = self.registry.prepare_writer_registration().await;
let coordinator = self.reinit.coordinator.lock();
let contour = self.authorize_writer_publication(&writer, &coordinator)?;
self.authorize_writer_publication_capacity(&writer, contour, intent, writers.as_slice())?;
// Only this victim-backed transaction may replace Active coverage during suppression.
let family = if writer.addr.is_ipv4() {
crate::network::IpFamily::V4
} else {
crate::network::IpFamily::V6
};
let configured = if writer.addr.is_ipv4() {
self.decision.ipv4_me
} else {
self.decision.ipv6_me
};
if !configured
|| (contour != WriterContour::Active
&& self.is_family_temporarily_suppressed(family, Self::now_epoch_secs()))
{
return Err(ProxyError::Proxy(
"ME replacement family lost publication authority".into(),
));
}
writer.contour.store(contour.as_u8(), Ordering::Release);
let Some(victim_pos) = writers
.iter()
@@ -73,6 +90,17 @@ impl MePool {
"ME replacement victim changed role before commit".into(),
));
}
if !matches!(
purpose,
WriterReplacementPurpose::FloorRebalance { .. }
| WriterReplacementPurpose::CoverageTransfer { .. }
) && (expected_victim_role.dc != writer.writer_dc
|| expected_victim_role.family != family)
{
return Err(ProxyError::Proxy(
"ME replacement purpose cannot move coverage between groups".into(),
));
}
if let WriterReplacementPurpose::FloorRebalance {
donor_floor,
receiver_floor,
@@ -122,19 +150,103 @@ impl MePool {
}) == receiver_family
})
.count();
if donor_count <= donor_floor || receiver_count >= receiver_floor {
let protected_floor = self.required_writers_for_dc_with_floor_mode(
endpoint_snapshot
.endpoints_for_dc_family(expected_victim_role.dc, expected_victim_role.family)
.len(),
false,
);
let current_receiver_floor = self.required_writers_for_dc_with_floor_mode(
endpoint_snapshot
.endpoints_for_dc_family(writer.writer_dc, receiver_family)
.len(),
false,
);
if donor_count <= donor_floor.max(protected_floor)
|| receiver_count >= receiver_floor.min(current_receiver_floor)
{
return Err(ProxyError::Proxy(
"ME floor rebalance became unnecessary before commit".into(),
));
}
}
if !registry_registration.prepare_replacement_commit(reservation) {
if let WriterReplacementPurpose::CoverageTransfer {
authority,
receiver_floor,
} = purpose
{
if authority
!= (
coordinator.endpoint_revision,
coordinator.floor_policy_revision,
)
|| contour != WriterContour::Active
|| expected_victim_role.contour != WriterContour::Active
{
return Err(ProxyError::Proxy(
"ME coverage transfer lost policy authority".into(),
));
}
let endpoints = self.endpoint_snapshot.load();
let count = |dc, family| {
writers
.iter()
.filter(|candidate| {
!candidate.draining.load(Ordering::Acquire)
&& candidate.generation == coordinator.active_generation
&& candidate.writer_dc == dc
&& WriterContour::from_u8(candidate.contour.load(Ordering::Acquire))
== WriterContour::Active
&& candidate.addr.is_ipv4()
== matches!(family, crate::network::IpFamily::V4)
&& endpoints.contains_dc_endpoint(dc, candidate.addr)
})
.count()
};
let donor_is_current = victim.generation == coordinator.active_generation
&& endpoints.contains_dc_endpoint(victim.writer_dc, victim.addr);
let donor_floor = self.required_writers_for_dc_with_floor_mode(
endpoints
.endpoints_for_dc_family(victim.writer_dc, expected_victim_role.family)
.len(),
false,
);
if (donor_is_current
&& count(victim.writer_dc, expected_victim_role.family) <= donor_floor)
|| count(writer.writer_dc, family)
>= receiver_floor.min(
self.required_writers_for_dc_with_floor_mode(
endpoints
.endpoints_for_dc_family(writer.writer_dc, family)
.len(),
false,
),
)
{
return Err(ProxyError::Proxy(
"ME coverage transfer no longer preserves floors".into(),
));
}
}
let lifetime = registry_registration
.writer_lifetime(victim_writer_id)
.ok_or_else(|| {
ProxyError::Proxy("ME replacement victim transport disappeared".into())
})?;
if !lifetime.commit(&mut writer_task.opening, || {
registry_registration.prepare_replacement_commit(reservation)
}) {
return Err(ProxyError::Proxy(
"ME replacement victim became active before commit".into(),
));
}
registry_registration.install(replacement_writer_id, tx, byte_budget);
registry_registration.install_with_lifetime(
replacement_writer_id,
tx,
byte_budget,
writer_task.lifetime.clone(),
);
writers.push(writer);
self.conn_count.fetch_add(1, Ordering::Relaxed);
writers.publish_current();
@@ -158,6 +270,7 @@ impl MePool {
#[cfg(test)]
mod tests {
mod convergence;
use std::collections::HashMap;
use std::net::{IpAddr, Ipv4Addr};
@@ -248,14 +361,54 @@ mod tests {
tx,
byte_budget,
task_registration,
writer_task: Box::pin(async move {
drop(rx);
}),
writer_task: WriterTransport::new(
Box::pin(async move {
drop(rx);
}),
open_reservation,
),
intent: WriterOpenIntent::Replacement,
_open_reservation: open_reservation,
}
}
#[tokio::test]
async fn suppressed_victim_backed_replacement_keeps_coverage() {
let pool = make_pool().await;
let addr = endpoint(8);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
let victim = install_writer(&pool, 8001, 2, addr).await;
pool.set_family_runtime_state(
crate::network::IpFamily::V4,
crate::transport::middle_proxy::pool::MeFamilyRuntimeState::Suppressed,
MePool::now_epoch_secs(),
MePool::now_epoch_secs() + 3600,
5,
0,
);
let mut reservation = pool
.registry
.try_reserve_writer_replacement(victim.id)
.await
.unwrap();
let prepared = prepared_writer(&pool, 8002, 2, addr).await;
pool.publish_prepared_replacement_writer(
prepared,
WriterRole::from_writer(&victim),
WriterReplacementPurpose::IdleRefresh,
&mut reservation,
)
.await
.expect("a reserved active victim must remain replaceable during suppression");
assert!(victim.draining.load(Ordering::Acquire));
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
1,
"overlap is owned until the victim transport is destroyed"
);
}
#[tokio::test]
async fn replacement_commit_publishes_successor_before_draining_victim() {
let pool = make_pool().await;
@@ -0,0 +1,198 @@
use super::*;
use crate::transport::middle_proxy::registry::ConnMeta;
#[tokio::test]
async fn busy_obsolete_donor_preserves_clients_and_rechecks_policy() {
for stale in [false, true] {
let pool = make_pool().await;
let obsolete = endpoint(21);
let receiver = endpoint(22);
pool.update_proxy_maps(
HashMap::from([(2, vec![(receiver.ip(), receiver.port())])]),
None,
)
.await;
let victim = install_writer(&pool, 9001, 1, obsolete).await;
let (client, _messages) = pool.registry.register().await;
assert!(
pool.registry
.bind_writer(
client,
victim.id,
ConnMeta {
target_dc: 1,
client_addr: obsolete,
our_addr: obsolete,
proto_flags: 0,
}
)
.await
);
let authority = pool.floor_authority();
let mut reservation = pool
.registry
.try_reserve_writer_replacement_preserving_clients(victim.id)
.await
.unwrap();
let prepared = prepared_writer(&pool, 9002, 2, receiver).await;
if stale {
pool.reinit.coordinator.lock().floor_policy_revision += 1;
}
let result = pool
.publish_prepared_replacement_writer(
prepared,
WriterRole::from_writer(&victim),
WriterReplacementPurpose::CoverageTransfer {
authority,
receiver_floor: pool.required_writers_for_dc(1),
},
&mut reservation,
)
.await;
assert_eq!(result.is_ok(), !stale);
assert_eq!(victim.draining.load(Ordering::Acquire), !stale);
assert_eq!(
pool.registry.get_writer(client).await.unwrap().writer_id,
victim.id
);
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
usize::from(!stale)
);
}
}
#[tokio::test]
async fn transfer_commit_matches_independent_donor_floor_model() {
// The model counts only current endpoints; obsolete sockets contribute no coverage.
for obsolete in [false, true] {
for donor_delta in [0usize, 1] {
for receiver_full in [false, true] {
let pool = make_pool().await;
let donor = endpoint(31);
let receiver = endpoint(32);
pool.update_proxy_maps(
HashMap::from([
(1, vec![(donor.ip(), donor.port())]),
(2, vec![(receiver.ip(), receiver.port())]),
]),
None,
)
.await;
let floor = pool.required_writers_for_dc(1);
let mut victim = None;
for id in 0..(floor + donor_delta) {
let writer = install_writer(&pool, 10000 + id as u64, 1, donor).await;
if id == 0 {
victim = Some(writer);
}
}
let mut victim = victim.unwrap();
if obsolete {
let mut writers = pool.writers.write().await;
writers
.iter_mut()
.find(|writer| writer.id == victim.id)
.unwrap()
.addr = endpoint(33);
victim.addr = endpoint(33);
}
if receiver_full {
for id in 0..floor {
install_writer(&pool, 11000 + id as u64, 2, receiver).await;
}
}
let mut reservation = pool
.registry
.try_reserve_writer_replacement(victim.id)
.await
.unwrap();
let prepared = prepared_writer(&pool, 12000, 2, receiver).await;
let result = pool
.publish_prepared_replacement_writer(
prepared,
WriterRole::from_writer(&victim),
WriterReplacementPurpose::CoverageTransfer {
authority: pool.floor_authority(),
receiver_floor: floor,
},
&mut reservation,
)
.await;
assert_eq!(
result.is_ok(),
!receiver_full && (obsolete || donor_delta > 0),
"obsolete={obsolete}, surplus={donor_delta}, receiver_full={receiver_full}"
);
}
}
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_transfers_cannot_overfill_receiver_or_steal_donor_floor() {
let pool = make_pool().await;
let donor = endpoint(41);
let receiver = endpoint(42);
pool.update_proxy_maps(
HashMap::from([
(1, vec![(donor.ip(), donor.port())]),
(2, vec![(receiver.ip(), receiver.port())]),
]),
None,
)
.await;
pool.set_adaptive_floor_runtime_caps(pool.floor_authority(), 128, 128, 128, 128, 6, 0, 0);
let floor = pool.required_writers_for_dc(1);
let count = floor + 8;
let mut victims = Vec::new();
for id in 0..count {
victims.push(install_writer(&pool, 20000 + id as u64, 1, donor).await);
}
let mut tasks = tokio::task::JoinSet::new();
for (index, victim) in victims.into_iter().enumerate() {
let pool = pool.clone();
tasks.spawn(async move {
let authority = pool.floor_authority();
let mut reservation = pool
.registry
.try_reserve_writer_replacement(victim.id)
.await
.unwrap();
let prepared = prepared_writer(&pool, 21000 + index as u64, 2, receiver).await;
tokio::task::yield_now().await;
pool.publish_prepared_replacement_writer(
prepared,
WriterRole::from_writer(&victim),
WriterReplacementPurpose::CoverageTransfer {
authority,
receiver_floor: floor,
},
&mut reservation,
)
.await
.is_ok()
});
}
let mut committed = 0;
while let Some(result) = tasks.join_next().await {
committed += usize::from(result.unwrap());
}
assert_eq!(committed, floor);
let writers = pool.writers.read().await;
let live = |dc| {
writers
.iter()
.filter(|writer| writer.writer_dc == dc && !writer.draining.load(Ordering::Acquire))
.count()
};
assert_eq!(live(2), floor);
assert_eq!(live(1), count - floor);
assert!(live(1) >= floor);
assert_eq!(
pool.writer_replacement_open_reserved
.load(Ordering::Acquire),
committed
);
}
@@ -291,9 +291,8 @@ impl MePool {
tx,
byte_budget,
task_registration,
writer_task,
writer_task: WriterTransport::new(writer_task, writer_open_reservation),
intent,
_open_reservation: writer_open_reservation,
})
}
+1
View File
@@ -81,6 +81,7 @@ struct WriterRoute {
tx: mpsc::Sender<WriterCommand>,
byte_budget: Arc<Semaphore>,
replacement_state: Arc<AtomicU8>,
lifetime: Arc<super::pool_writer::overlap::ReplacementHandoff>,
}
#[derive(Clone)]
@@ -99,6 +99,34 @@ impl WriterRegistrationGuard<'_> {
writer_id: u64,
tx: mpsc::Sender<WriterCommand>,
byte_budget: Arc<Semaphore>,
) {
self.install_with_lifetime(
writer_id,
tx,
byte_budget,
super::super::pool_writer::overlap::ReplacementHandoff::new(),
);
}
/// Retrieves the physical lifetime while binding ownership fences victim replacement.
pub(in crate::transport::middle_proxy) fn writer_lifetime(
&self,
writer_id: u64,
) -> Option<Arc<super::super::pool_writer::overlap::ReplacementHandoff>> {
self.registry
.writers
.map
.get(&writer_id)
.map(|route| route.lifetime.clone())
}
/// Publishes a route associated with its transport destruction authority.
pub(in crate::transport::middle_proxy) fn install_with_lifetime(
&mut self,
writer_id: u64,
tx: mpsc::Sender<WriterCommand>,
byte_budget: Arc<Semaphore>,
lifetime: Arc<super::super::pool_writer::overlap::ReplacementHandoff>,
) {
self.binding.conns_for_writer.entry(writer_id).or_default();
self.registry
@@ -117,6 +145,7 @@ impl WriterRegistrationGuard<'_> {
tx,
byte_budget,
replacement_state: Arc::new(AtomicU8::new(WriterReplacementState::Open as u8)),
lifetime,
},
);
}
+9 -21
View File
@@ -415,15 +415,10 @@ impl ConnRegistry {
.map
.get(&conn_id)
.map(|entry| entry.writer_id)?;
let writer = self
.writers
.map
.get(&writer_id)
.map(|entry| entry.value().clone())?;
Some(ConnWriter {
self.writers.map.get(&writer_id).map(|entry| ConnWriter {
writer_id,
tx: writer.tx,
byte_budget: writer.byte_budget,
tx: entry.tx.clone(),
byte_budget: entry.byte_budget.clone(),
})
}
@@ -436,19 +431,12 @@ impl ConnRegistry {
let hot = self.hot_binding.map.get(&conn_id)?;
let writer_id = hot.writer_id;
let meta = hot.meta.clone();
let writer = self
.writers
.map
.get(&writer_id)
.map(|entry| entry.value().clone())?;
Some((
ConnWriter {
writer_id,
tx: writer.tx,
byte_budget: writer.byte_budget,
},
meta,
))
let writer = self.writers.map.get(&writer_id).map(|entry| ConnWriter {
writer_id,
tx: entry.tx.clone(),
byte_budget: entry.byte_budget.clone(),
})?;
Some((writer, meta))
}
pub async fn active_conn_ids(&self) -> Vec<u64> {
+135 -42
View File
@@ -9,10 +9,17 @@ use crate::config::ProxyConfig;
use crate::crypto::SecureRandom;
use super::MePool;
// Durable scheduler intent is independent of pending generation creation.
mod objective;
#[cfg(test)]
mod retry_tests;
/// External causes that request generation reconciliation.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MeReinitTrigger {
/// The configured rotation interval elapsed.
Periodic,
/// Endpoint authority changed.
MapChanged,
}
@@ -25,6 +32,7 @@ impl MeReinitTrigger {
}
}
/// Coalesces an external request; endpoint epochs independently preserve map-change intent.
pub fn enqueue_reinit_trigger(tx: &mpsc::Sender<MeReinitTrigger>, trigger: MeReinitTrigger) {
match tx.try_send(trigger) {
Ok(()) => {}
@@ -82,6 +90,7 @@ fn trigger_reason(pending: u8) -> &'static str {
}
}
/// Retries durable reinitialization objectives until their exact endpoint revision converges.
pub async fn me_reinit_scheduler(
pool: Arc<MePool>,
rng: Arc<SecureRandom>,
@@ -90,9 +99,14 @@ pub async fn me_reinit_scheduler(
me_ready_tx: watch::Sender<u64>,
) {
info!("ME reinit scheduler started");
let mut tasks = JoinSet::<bool>::new();
let mut pending = 0u8;
let mut pending_deadline = None;
let mut tasks = JoinSet::<Option<bool>>::new();
let mut owners = std::collections::HashMap::new();
let mut objective: Option<objective::RetryObjective> = None;
let mut next_id = 1u64;
let mut epochs = pool.writer_epoch.subscribe();
let mut epoch_open = true;
let mut observed_revision = pool.endpoint_snapshot.load().revision;
let mut completed_revision = None;
let mut trigger_channel_open = true;
loop {
@@ -101,73 +115,152 @@ pub async fn me_reinit_scheduler(
pool.reinit
.max_concurrency_effective
.store(max_concurrency, std::sync::atomic::Ordering::Release);
let pending_ready = pending != 0
&& pending_deadline.is_none_or(|deadline| deadline <= tokio::time::Instant::now());
if pending_ready && tasks.len() < max_concurrency {
let reason = trigger_reason(pending);
pending = 0;
pending_deadline = None;
debug!(reason, max_concurrency, "ME reinit scheduled");
let snapshot = pool.endpoint_snapshot.load_full();
if snapshot.revision != observed_revision {
observed_revision = snapshot.revision;
objective = Some(objective::RetryObjective::new(
next_id,
observed_revision,
REINIT_TRIGGER_MAP_CHANGED,
Duration::from_millis(cfg.general.me_reinit_coalesce_window_ms),
));
next_id = next_id.wrapping_add(1);
}
let owned = objective
.as_ref()
.is_some_and(|current| owners.values().any(|identity| current.matches(*identity)));
if let Some(current) = objective.as_mut()
&& !owned
&& current.attempt > 0
{
let status = pool.reinit.status.load_full();
let authority = pool.floor_authority();
let readiness = pool.api_hardswap_snapshot().await;
// An API projection is usable only while its exact publication and policy survive
// the await. A changed snapshot is unknown, not a not-ready transition.
if Arc::ptr_eq(&status, &pool.reinit.status.load_full())
&& authority == pool.floor_authority()
{
if status.pending_hardswap_endpoint_revision == current.revision
&& readiness.pending_map_current == Some(true)
&& readiness.pending_writer_deficit == 0
{
current.ready((
status.pending_hardswap_generation,
status.pending_hardswap_map_hash,
status.pending_hardswap_endpoint_revision,
));
} else {
current.not_ready();
}
}
}
let can_launch = !owned
&& tasks.len() < max_concurrency
&& objective
.as_ref()
.is_some_and(|current| current.attempt == 0 || tasks.is_empty());
if can_launch
&& objective.as_ref().is_some_and(|current| {
current
.deadline
.is_some_and(|deadline| deadline <= tokio::time::Instant::now())
})
{
if pool.endpoint_snapshot.load().revision != snapshot.revision {
continue;
}
let Some(current) = objective.as_mut() else {
continue;
};
let identity = (current.id, current.revision);
current.deadline = None;
debug!(
reason = trigger_reason(current.triggers),
attempt = current.attempt,
endpoint_revision = current.revision,
"ME reinit scheduled"
);
let pool_clone = pool.clone();
let rng_clone = rng.clone();
pool.reinit
.scheduler_inflight
.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
let inflight = ReinitInflightGuard {
pool: Arc::clone(&pool_clone),
pool: pool_clone.clone(),
};
tasks.spawn(async move {
let future = async move {
let _inflight = inflight;
pool_clone
.zero_downtime_reinit_periodic(rng_clone.as_ref())
.reinit_endpoint_snapshot(rng_clone.as_ref(), snapshot)
.await
});
};
// Pool shutdown joins destruction of attempts as well as their writer transports.
let Ok(tracked) = pool.lifecycle.track_producer(future) else {
return;
};
let task = tasks.spawn(tracked);
owners.insert(task.id(), identity);
continue;
}
if !trigger_channel_open && pending == 0 && tasks.is_empty() {
warn!("ME reinit scheduler stopped: trigger channel closed");
if !trigger_channel_open && objective.is_none() && tasks.is_empty() {
return;
}
let timer_enabled = pending != 0 && tasks.len() < max_concurrency;
let deadline = objective.as_ref().and_then(|current| current.deadline);
tokio::select! {
trigger = trigger_rx.recv(), if trigger_channel_open => {
match trigger {
Some(trigger) => {
if pending == 0 {
pending_deadline = Some(
tokio::time::Instant::now()
+ Duration::from_millis(
cfg.general.me_reinit_coalesce_window_ms,
),
);
let revision = pool.endpoint_snapshot.load().revision;
if trigger == MeReinitTrigger::MapChanged && completed_revision == Some(revision) {
continue;
}
pending |= trigger_bit(trigger);
if let Some(current) = objective.as_mut()
&& current.revision == revision
&& (trigger == MeReinitTrigger::MapChanged
|| !owners.values().any(|identity| current.matches(*identity))) {
// Endpoint epochs and updater notifications describe the same objective.
current.triggers |= trigger_bit(trigger);
} else {
objective = Some(objective::RetryObjective::new(next_id, revision,
trigger_bit(trigger), Duration::from_millis(cfg.general.me_reinit_coalesce_window_ms)));
next_id = next_id.wrapping_add(1);
}
observed_revision = revision;
}
None => trigger_channel_open = false,
}
}
joined = tasks.join_next(), if !tasks.is_empty() => {
match joined {
Some(Ok(true)) => {
me_ready_tx.send_modify(|version| {
*version = version.saturating_add(1);
});
}
Some(Ok(false)) => {}
Some(Err(error)) => {
changed = epochs.changed(), if epoch_open => {
if changed.is_err() { epoch_open = false; }
}
joined = tasks.join_next_with_id(), if !tasks.is_empty() => {
let Some(joined) = joined else { continue; };
let (id, succeeded) = match joined {
Ok((id, Some(succeeded))) => (id, succeeded),
Ok((_, None)) => return,
Err(error) => {
warn!(error = %error, "ME reinit task failed");
(error.id(), false)
}
};
let Some(identity) = owners.remove(&id) else { continue; };
if succeeded {
me_ready_tx.send_modify(|version| *version = version.saturating_add(1));
}
if let Some(current) = objective.as_mut()
&& current.matches(identity) {
if succeeded {
completed_revision = Some(current.revision);
objective = None;
} else {
current.failed(Duration::from_millis(pool.hardswap_warmup_backoff_ms(current.attempt)));
}
None => {}
}
}
_ = async {
if let Some(deadline) = pending_deadline {
tokio::time::sleep_until(deadline).await;
}
}, if timer_enabled => {}
if let Some(deadline) = deadline { tokio::time::sleep_until(deadline).await; }
}, if can_launch && deadline.is_some() => {}
}
}
}
@@ -0,0 +1,55 @@
use std::time::Duration;
use tokio::time::Instant;
/// Durable intent exists even if a reinit returns before creating a pending generation.
pub(super) struct RetryObjective {
/// Distinguishes separate external requests for the same endpoint revision.
pub(super) id: u64,
/// Exact endpoint snapshot to reconcile.
pub(super) revision: u64,
/// Coalesced external causes used for diagnostic logging.
pub(super) triggers: u8,
/// Completed failures; the count saturates but never terminates recovery.
pub(super) attempt: usize,
/// A missing deadline means the objective currently owns a running task.
pub(super) deadline: Option<Instant>,
ready_wake: Option<(u64, u64, u64)>,
}
impl RetryObjective {
/// Creates retryable intent before any asynchronous attempt can fail.
pub(super) fn new(id: u64, revision: u64, triggers: u8, delay: Duration) -> Self {
Self {
id,
revision,
triggers,
attempt: 0,
deadline: Some(Instant::now() + delay),
ready_wake: None,
}
}
/// Rejects completions from superseded objectives, including the same endpoint revision.
pub(super) fn matches(&self, identity: (u64, u64)) -> bool {
(self.id, self.revision) == identity
}
/// Measures backoff from completion and never caps the number of retry attempts.
pub(super) fn failed(&mut self, delay: Duration) {
self.attempt = self.attempt.saturating_add(1);
self.deadline = Some(Instant::now() + delay);
}
/// Accelerates once per observed ready tuple without allowing an epoch storm to spin.
pub(super) fn ready(&mut self, key: (u64, u64, u64)) {
if self.ready_wake != Some(key) {
self.ready_wake = Some(key);
self.deadline = Some(Instant::now());
}
}
/// Re-arms acceleration only after a coherent observation of lost readiness.
pub(super) fn not_ready(&mut self) {
self.ready_wake = None;
}
}
@@ -0,0 +1,188 @@
use super::objective::RetryObjective;
use super::*;
#[tokio::test(start_paused = true)]
async fn retry_objective_survives_unlimited_failures_and_ready_storms() {
let mut objective = RetryObjective::new(1, 2, REINIT_TRIGGER_PERIODIC, Duration::ZERO);
assert!(objective.deadline.is_some());
for _ in 0..100 {
objective.deadline = None;
tokio::time::advance(Duration::from_secs(70)).await;
let completed = tokio::time::Instant::now();
objective.failed(Duration::from_secs(30));
assert_eq!(
objective.deadline,
Some(completed + Duration::from_secs(30))
);
}
objective.ready((3, 4, 2));
objective.failed(Duration::from_secs(30));
let deadline = objective.deadline;
for _ in 0..100 {
objective.ready((3, 4, 2));
}
assert_eq!(objective.deadline, deadline);
objective.not_ready();
objective.ready((3, 4, 2));
assert!(
objective.deadline < deadline,
"a genuine loss and recovery of the same floor re-arms wakeup"
);
assert!(!objective.matches((1, 1)));
assert!(!objective.matches((2, 2)));
}
#[tokio::test]
async fn pre_pending_failure_remains_live_without_a_periodic_trigger() {
let pool = crate::transport::middle_proxy::pool_writer_security_tests::make_pool().await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
pool.update_proxy_maps(
std::collections::HashMap::from([(2, vec![(addr.ip(), addr.port())])]),
None,
)
.await;
pool.set_family_runtime_state(
crate::network::IpFamily::V4,
crate::transport::middle_proxy::pool::MeFamilyRuntimeState::Suppressed,
MePool::now_epoch_secs(),
MePool::now_epoch_secs() + 3600,
5,
0,
);
pool.reinit
.me_hardswap_warmup_pass_backoff_base_ms
.store(10, std::sync::atomic::Ordering::Relaxed);
pool.reinit
.me_hardswap_warmup_delay_min_ms
.store(0, std::sync::atomic::Ordering::Relaxed);
pool.reinit
.me_hardswap_warmup_delay_max_ms
.store(0, std::sync::atomic::Ordering::Relaxed);
let mut config = ProxyConfig::default();
config.general.me_reinit_coalesce_window_ms = 0;
let (_config_tx, config_rx) = watch::channel(Arc::new(config));
let (tx, rx) = mpsc::channel(1);
let (ready_tx, _) = watch::channel(0);
tx.send(MeReinitTrigger::Periodic).await.unwrap();
drop(tx);
let scheduler = tokio::spawn(me_reinit_scheduler(
pool.clone(),
pool.rng.clone(),
config_rx,
rx,
ready_tx,
));
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(
!scheduler.is_finished(),
"false before pending creation must not discard recovery intent"
);
assert!(pool.reinit.coordinator.lock().pending.is_none());
// Family policy changes do not emit endpoint or writer epochs.
pool.set_family_runtime_state(
crate::network::IpFamily::V4,
crate::transport::middle_proxy::pool::MeFamilyRuntimeState::Healthy,
MePool::now_epoch_secs(),
0,
0,
0,
);
let (stream, _) = tokio::time::timeout(Duration::from_secs(2), listener.accept())
.await
.expect("autonomous retry must actually reach TCP without another trigger")
.unwrap();
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
tokio::time::timeout(Duration::from_secs(1), scheduler)
.await
.unwrap()
.unwrap();
assert_eq!(
pool.reinit
.scheduler_inflight
.load(std::sync::atomic::Ordering::Acquire),
0
);
drop(stream);
}
#[tokio::test]
async fn panic_retains_the_task_identity_for_retry() {
let mut tasks = JoinSet::<bool>::new();
let id = tasks.spawn(async { panic!("injected reinit panic") }).id();
let error = tasks.join_next_with_id().await.unwrap().unwrap_err();
assert_eq!(error.id(), id);
}
async fn check_inflight_trigger(trigger: MeReinitTrigger, starts_successor: bool) {
let pool = crate::transport::middle_proxy::pool_writer_security_tests::make_pool().await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
pool.update_proxy_maps(
std::collections::HashMap::from([(2, vec![(addr.ip(), addr.port())])]),
None,
)
.await;
pool.reinit
.me_hardswap_warmup_delay_min_ms
.store(0, std::sync::atomic::Ordering::Relaxed);
pool.reinit
.me_hardswap_warmup_delay_max_ms
.store(0, std::sync::atomic::Ordering::Relaxed);
let mut config = ProxyConfig::default();
config.general.me_reinit_coalesce_window_ms = 0;
config.general.me_reinit_singleflight = false;
config.general.me_reinit_max_concurrency = 2;
let (_config_tx, config_rx) = watch::channel(Arc::new(config));
let (tx, rx) = mpsc::channel(1);
let (ready_tx, _) = watch::channel(0);
tx.send(MeReinitTrigger::MapChanged).await.unwrap();
let scheduler = tokio::spawn(me_reinit_scheduler(
pool.clone(),
pool.rng.clone(),
config_rx,
rx,
ready_tx,
));
let (first, _) = tokio::time::timeout(Duration::from_secs(1), listener.accept())
.await
.unwrap()
.unwrap();
assert_eq!(
pool.reinit
.scheduler_inflight
.load(std::sync::atomic::Ordering::Acquire),
1
);
// Keep the first attempt blocked before handshake completion while delivering another trigger.
tx.send(trigger).await.unwrap();
let successor = tokio::time::timeout(Duration::from_millis(100), listener.accept()).await;
let inflight = pool
.reinit
.scheduler_inflight
.load(std::sync::atomic::Ordering::Acquire);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
tokio::time::timeout(Duration::from_secs(1), scheduler)
.await
.unwrap()
.unwrap();
assert_eq!(successor.is_ok(), starts_successor);
assert_eq!(inflight, if starts_successor { 2 } else { 1 });
assert_eq!(
pool.reinit
.scheduler_inflight
.load(std::sync::atomic::Ordering::Acquire),
0
);
drop(first);
}
#[tokio::test]
async fn duplicate_map_change_coalesces_with_its_inflight_revision() {
check_inflight_trigger(MeReinitTrigger::MapChanged, false).await;
}
#[tokio::test]
async fn periodic_trigger_retains_successor_intent_while_map_change_runs() {
check_inflight_trigger(MeReinitTrigger::Periodic, true).await;
}
@@ -0,0 +1,98 @@
use std::collections::HashMap;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU32, AtomicU64, Ordering};
use std::time::Instant;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
use super::codec::WriterCommand;
use super::pool::{MePool, MeWriter, WriterContour};
use super::pool_writer_security_tests::make_pool;
/// Owns idle writer visibility without spawning ME transports or client sessions.
pub(crate) struct IdlePoolFixture {
pool: Arc<MePool>,
draining: Vec<Arc<AtomicBool>>,
_receivers: Vec<mpsc::Receiver<WriterCommand>>,
}
impl IdlePoolFixture {
/// Builds both signed DC groups with a controllable readiness quorum.
pub(crate) async fn new(ready: bool) -> Self {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
let mut draining = Vec::new();
let mut receivers = Vec::new();
{
let mut writers = pool.writers.write().await;
for (id, dc) in [(1, 2), (2, -2)] {
let (writer, receiver) = unregistered_writer(
&pool,
id,
dc,
addr,
pool.current_generation(),
WriterContour::Active,
);
writer.draining.store(!ready, Ordering::Release);
draining.push(writer.draining.clone());
receivers.push(receiver);
writers.push(writer);
}
}
Self {
pool,
draining,
_receivers: receivers,
}
}
/// Returns the pool sampled by the production admission gate.
pub(crate) fn pool(&self) -> Arc<MePool> {
self.pool.clone()
}
/// Changes the fixture quorum without notifying the admission gate.
pub(crate) fn set_ready(&self, ready: bool) {
for draining in &self.draining {
draining.store(!ready, Ordering::Release);
}
}
}
/// Builds a writer whose command receiver remains owned by the test.
pub(super) fn unregistered_writer(
pool: &Arc<MePool>,
id: u64,
dc: i32,
addr: SocketAddr,
generation: u64,
contour: WriterContour,
) -> (MeWriter, mpsc::Receiver<WriterCommand>) {
let (tx, rx) = mpsc::channel(8);
(
MeWriter {
id,
addr,
source_ip: addr.ip(),
writer_dc: dc,
generation,
contour: Arc::new(AtomicU8::new(contour.as_u8())),
created_at: Instant::now(),
tx,
byte_budget: pool.new_writer_byte_budget(),
cancel: CancellationToken::new(),
degraded: Arc::new(AtomicBool::new(false)),
rtt_ema_ms_x10: Arc::new(AtomicU32::new(0)),
draining: Arc::new(AtomicBool::new(false)),
draining_started_at_epoch_secs: Arc::new(AtomicU64::new(0)),
drain_deadline_epoch_secs: Arc::new(AtomicU64::new(0)),
allow_drain_fallback: Arc::new(AtomicBool::new(false)),
},
rx,
)
}
+2
View File
@@ -339,6 +339,8 @@ pub struct UpstreamManager {
// Upstream manager configuration, DNS resolution, and API snapshots.
mod manager_config;
// Current direct egress addresses eligible for source-bound NAT discovery.
mod nat_contexts;
// Latency- and scope-aware upstream selection.
mod selection;
// Direct and proxied connection establishment.
+122
View File
@@ -0,0 +1,122 @@
use super::*;
impl UpstreamManager {
/// Resolves this generation's direct bind candidates without advancing round-robin state.
pub(crate) async fn direct_bind_ips_for_family(&self, ipv6: bool) -> BTreeSet<IpAddr> {
let guard = self.upstreams.read().await;
let target = SocketAddr::new(
if ipv6 {
std::net::Ipv6Addr::UNSPECIFIED.into()
} else {
std::net::Ipv4Addr::UNSPECIFIED.into()
},
0,
);
let mut allowed = BTreeSet::new();
for state in guard.iter() {
let config = &state.config;
let UpstreamType::Direct {
interface,
bind_addresses,
..
} = &config.upstream_type
else {
continue;
};
if (if ipv6 { config.ipv6 } else { config.ipv4 }) == Some(false) {
continue;
}
if let Some(addresses) = bind_addresses.as_ref().filter(|v| !v.is_empty()) {
for address in addresses {
if !address
.parse::<IpAddr>()
.is_ok_and(|ip| ip.is_ipv6() == ipv6)
{
continue;
}
if let Some(ip) = Self::resolve_bind_address(
interface,
&Some(vec![address.clone()]),
target,
None,
true,
) {
allowed.insert(ip);
}
}
} else if let Some(ip) =
Self::resolve_bind_address(interface, bind_addresses, target, None, true)
{
allowed.insert(ip);
}
}
allowed
}
}
#[cfg(test)]
mod tests {
use super::*;
fn config(source: &str) -> UpstreamConfig {
toml::from_str(source).unwrap()
}
#[tokio::test]
async fn candidates_follow_generation_policy_without_advancing_round_robin() {
let manager = UpstreamManager::new(
vec![
config("type = 'direct'\nbind_addresses = ['127.0.0.1', '127.0.0.2', '::1']"),
config("type = 'direct'\ninterface = '127.0.0.3'"),
config("type = 'direct'\ninterface = '127.0.0.4'\nenabled = false"),
config("type = 'direct'\ninterface = '127.0.0.5'\nipv4 = false"),
config("type = 'socks5'\naddress = '127.0.0.1:1080'\ninterface = '127.0.0.6'"),
],
1,
0,
1000,
1,
1,
true,
Arc::new(Stats::new()),
);
manager.upstreams.write().await[0].healthy = false;
let expected = ["127.0.0.1", "127.0.0.2", "127.0.0.3"]
.into_iter()
.map(|value| value.parse().unwrap())
.collect();
assert_eq!(manager.direct_bind_ips_for_family(false).await, expected);
assert_eq!(
manager.direct_bind_ips_for_family(true).await,
BTreeSet::from(["::1".parse().unwrap()])
);
for state in manager.upstreams.read().await.iter() {
assert_eq!(state.bind_rr.load(Ordering::Relaxed), 0);
}
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn named_interface_candidates_match_direct_socket_binding() {
let manager = UpstreamManager::new(
vec![config(
"type = 'direct'\ninterface = 'lo'\nbind_addresses = ['127.0.0.1', '::1']",
)],
1,
0,
1000,
1,
1,
true,
Arc::new(Stats::new()),
);
assert_eq!(
manager.direct_bind_ips_for_family(false).await,
BTreeSet::from(["127.0.0.1".parse().unwrap()])
);
assert_eq!(
manager.direct_bind_ips_for_family(true).await,
BTreeSet::from(["::1".parse().unwrap()])
);
}
}
+7
View File
@@ -1,5 +1,6 @@
use base64::Engine as _;
use crate::config::WebCarrierMethod;
use crate::crypto::SecureRandom;
/// Browser security policy for the transient Telegram Desktop bridge page.
@@ -34,6 +35,7 @@ pub(crate) fn render(
reconnect_grace_secs: u64,
carrier_probe_coalesce_ms: u64,
bridge_diagnostics_enabled: bool,
carrier_method: WebCarrierMethod,
rng: &SecureRandom,
) -> BridgePage {
let mut nonce = [0u8; 18];
@@ -63,6 +65,8 @@ pub(crate) fn render(
.replace("__REQUEST_RUNTIME__", REQUEST_RUNTIME)
.replace("__BUFFER_RUNTIME__", BUFFER_RUNTIME)
.replace("__RECOVERY_RUNTIME__", RECOVERY_RUNTIME)
.replace("__DOWNLINK_RUNTIME__", DOWNLINK_RUNTIME)
.replace("__CONVEYOR_RUNTIME__", CONVEYOR_RUNTIME)
.replace("__RUNTIME__", RUNTIME)
.replace(
"__DIAGNOSTIC_BINDING__;\n",
@@ -124,6 +128,7 @@ pub(crate) fn render(
.replace("__HOST__", host)
.replace("__BASE_PREFIX__", base_prefix)
.replace("__BOOTSTRAP__", bootstrap)
.replace("__CARRIER_METHOD__", carrier_method.as_str())
.replace("__BATCH_LIMIT__", &batch_limit.to_string())
.replace("__QUEUE_LIMIT__", &queue_limit.to_string())
.replace("__QUEUE_ITEMS__", &queue_items.to_string())
@@ -171,6 +176,8 @@ const RESPONSE_RUNTIME: &str = include_str!("bridge/response.js");
const REQUEST_RUNTIME: &str = include_str!("bridge/request.js");
const BUFFER_RUNTIME: &str = include_str!("bridge/buffers.js");
const RECOVERY_RUNTIME: &str = include_str!("bridge/recovery.js");
const DOWNLINK_RUNTIME: &str = include_str!("bridge/downlink.js");
const CONVEYOR_RUNTIME: &str = include_str!("bridge/conveyor.js");
const RUNTIME: &str = include_str!("bridge/runtime.js");
// Rendered wire-contract tests remain separate from the embedded document.
+265
View File
@@ -0,0 +1,265 @@
'use strict';
const assert=require('node:assert/strict');
const fs=require('node:fs');
const vm=require('node:vm');
const path=require('node:path');
function renderedPage(){
if(process.argv.includes('--stdin'))return fs.readFileSync(0,'utf8');
let page=fs.readFileSync(path.join(__dirname,'document.html'),'utf8');
const modules={RESPONSE:'response',REQUEST:'request',BUFFER:'buffers',RECOVERY:'recovery',CONVEYOR:'conveyor',DOWNLINK:'downlink'};
page=page.replace('__DIAGNOSTIC_RUNTIME__','');
for(const [key,file] of Object.entries(modules))if(page.includes('__'+key+'_RUNTIME__'))page=page.replace('__'+key+'_RUNTIME__',()=>fs.readFileSync(path.join(__dirname,file+'.js'),'utf8'));
page=page.replace('__RUNTIME__',()=>fs.readFileSync(path.join(__dirname,'runtime.js'),'utf8'));
const values={BOOTSTRAP:'A'.repeat(43),HOST:'proxy.example.com',BASE_PREFIX:'',CARRIER_METHOD:'POST',NEGOTIATION_ENABLED:'true',CANDIDATE_COUNT:4,CARRIER_DEADLINES:'3,5,8,12',LONG_POLL_SECS:25,BRIDGE_REQUEST_SECS:10,BRIDGE_RETRY_SECS:90,BRIDGE_RECOVERY_SECS:15,WEBSOCKET_OPEN_SECS:15,RECONNECT_GRACE_SECS:120,CARRIER_PROBE_COALESCE_MS:0,BATCH_LIMIT:2097152,QUEUE_LIMIT:33554432,QUEUE_ITEMS:16384,MAX_STREAMS:1024,STATUS_FUNCTION:"state=>{if(port&&!closed)port.postMessage({t:'status',state})}",HELLO_TIMEOUT_CALLBACK:"()=>fail('timeout')",PAGEHIDE_CALLBACK:'()=>close(true)'};
return page.replace(/__([A-Z_]+)__/g,(all,key)=>key.startsWith('DIAGNOSTIC_')?'':String(values[key]??''));
}
function frame(type,id=0,payload=[]){
const data=new Uint8Array(8+payload.length),view=new DataView(data.buffer);
data[0]=type;data[1]=id>>>16;data[2]=id>>>8;data[3]=id;view.setUint32(4,payload.length);data.set(payload,8);return data.buffer;
}
function join(...frames){const data=new Uint8Array(frames.reduce((n,f)=>n+f.byteLength,0));let offset=0;for(const f of frames){data.set(new Uint8Array(f),offset);offset+=f.byteLength}return data.buffer}
async function flush(){for(let i=0;i<40;i++)await Promise.resolve()}
function environment(page,android=false){
let now=1000,nextTimer=1;const timers=new Map(),events=new Map(),requests=[],received=[];
const port={onmessage:null,start(){},close(){},postMessage(value,transfer){received.push(structuredClone(value,{transfer:transfer||[]}))}};
const context={ArrayBuffer,Uint8Array,DataView,TextDecoder,TextEncoder,URL,Headers,AbortController,ReadableStream,structuredClone,console,
location:{hash:android?'#android='+'A'.repeat(43):'',pathname:'/',search:'?bridge=test'},history:{replaceState(){}},parent:{},
performance:{now:()=>now},Date:class extends Date{static now(){return now}},document:{visibilityState:'visible',addEventListener(){}},
addEventListener:(name,fn)=>events.set(name,fn),setTimeout:(fn,delay)=>{const id=nextTimer++;timers.set(id,{fn,at:now+delay});return id},clearTimeout:id=>timers.delete(id),
fetch:(url,options)=>new Promise((resolve,reject)=>{const request={url,options,resolve,reject};requests.push(request);options.signal?.addEventListener('abort',()=>reject(new Error('aborted')),{once:true})}),
WebSocket:class {static OPEN=1;static CLOSING=2;constructor(){throw new Error('unexpected websocket')}}
};
if(android)context.TelegramWebProxy={onmessage:null,postMessage(value){received.push(typeof value==='string'?JSON.parse(value):structuredClone(value))}};
vm.createContext(context);
for(const match of page.matchAll(/<script\b[^>]*>([\s\S]*?)<\/script>/g))vm.runInContext(match[1],context);
if(!android)events.get('message')({source:context.parent,origin:'http://127.0.0.1:12345',data:{t:'tproxy-init',v:1},ports:[port]});
const send=data=>android?context.TelegramWebProxy.onmessage({data}):port.onmessage({data});
const pending=suffix=>requests.filter(r=>r.url.endsWith(suffix)&&!r.answered);
const answer=(request,status,body=null,headers={})=>{request.answered=true;request.resolve({status,headers:new Headers(headers),body:body===null?null:new ReadableStream({start(c){c.enqueue(new Uint8Array(body));c.close()}})})};
return {context,send,pending,answer,received,requests,async tick(ms){const until=now+ms;for(;;){const entry=[...timers].filter(([,t])=>t.at<=until).sort((a,b)=>a[1].at-b[1].at)[0];if(!entry)break;now=entry[1].at;timers.delete(entry[0]);entry[1].fn();await flush()}now=until;await flush()},close(){events.get('pagehide')();}};
}
async function session(page,carrier='https',window=1,android=false){
const env=environment(page,android);env.send(frame(16,0,[1]));await flush();
const request=env.pending('/api/v1/session')[0];assert.ok(request,'session requested');
const headers={'X-Session-Token':'B'.repeat(43),'X-Down-Cursor':'0','X-Carrier-Mode':carrier,'X-Carrier-Attempt':'1','X-Carrier-Candidate-Count':'4','X-Carrier-Deadline':'12','X-Carrier-State':'provisional'};
if(window!==null)headers['X-Telemt-Up-Window']=String(window);
env.answer(request,200,frame(17),headers);await flush();return env;
}
const tests=[];
function test(name,run){tests.push({name,run})}
function ack(env,request){env.answer(request,204,null,{'X-Up-Ack':request.options.headers['X-Up-Seq']})}
function binary(env){return env.received.filter(value=>value instanceof ArrayBuffer&&new Uint8Array(value)[0]!==17)}
function streaming(env,request,next='1',length=null){
let controller;const body=new ReadableStream({start(value){controller=value}}),headers={'X-Down-Cursor':next};if(length!==null)headers['Content-Length']=String(length);
request.answered=true;request.resolve({status:200,headers:new Headers(headers),body});return controller;
}
test('already queued OPEN and DATA share the zero-wait startup request',async page=>{
const env=await session(page);const body=join(frame(1,1),frame(2,1,[7,8]));env.send(body);await flush();
assert.equal(env.pending('/api/v1/up')[0].options.body.byteLength,body.byteLength);env.close();await flush();
});
test('negotiated conveyor sends DATA before the OPEN acknowledgement',async page=>{
for(const android of [false,true])for(const carrier of ['https','https-lanes']){
const env=await session(page,carrier,4,android);env.send(frame(1,1));await flush();env.send(frame(2,1,[9]));await flush();
assert.equal(env.pending('/api/v1/up').length,2,carrier+' startup must not wait for ACK1');env.close();await flush();
}
});
test('unsupported server and disabled conveyor remain serialized',async page=>{
for(const window of [null,1]){
const env=await session(page,'https',window);env.send(frame(1,1));await flush();env.send(frame(2,1,[9]));await flush();
assert.equal(env.pending('/api/v1/up').length,1);env.close();await flush();
}
});
test('lost ACK1 cannot slide the window after ACK2 through ACK4',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();
for(let i=0;i<6;i++){env.send(frame(2,1,[i]));await flush()}
const up=env.pending('/api/v1/up');assert.equal(up.length,4);
for(const request of up.slice(1)){ack(env,request);await flush()}
assert.equal(env.requests.filter(r=>r.url.endsWith('/api/v1/up')).length,4);
ack(env,up[0]);await flush();
const next=env.pending('/api/v1/up')[0];assert.equal(next.options.headers['X-Up-Seq'],'5');assert.equal(next.options.headers['X-Telemt-Up-Confirmed'],'4');
env.close();await flush();
});
test('early downlink cannot deliver native data before a verified commit',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();
const down=env.pending('/api/v1/down')[0];assert.ok(down,'down starts before ACK');const data=frame(2,1,[4]);
const stream=streaming(env,down,'1',data.byteLength);stream.enqueue(new Uint8Array(data));await flush();assert.equal(binary(env).length,0);
ack(env,env.pending('/api/v1/up')[0]);await flush();assert.equal(binary(env).length,1);
assert.equal(env.requests.filter(r=>r.url.endsWith('/api/v1/down')).length,1,'cursor cannot advance before EOF');
stream.close();await flush();assert.equal(env.pending('/api/v1/down')[0].options.headers['X-Down-Cursor'],'1');env.close();await flush();
});
test('DATA and WINDOW prefixes are delivered once across a mid-body network failure',async page=>{
for(const carrier of ['https','https-lanes']){
const env=await session(page,carrier,4);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const first=frame(2,1,[1,2]),credit=frame(4,1,[0,0,0,2]),last=frame(3,1),body=join(first,credit,last);
const stream=streaming(env,env.pending('/api/v1/down')[0],'1',body.byteLength);
stream.enqueue(new Uint8Array(join(first,credit)));await flush();assert.equal(binary(env).length,2);
stream.error(new Error('connection lost'));await flush();
const replay=env.pending('/api/v1/down')[0];assert.ok(replay);assert.equal(replay.options.headers['X-Down-Cursor'],'0');
env.answer(replay,200,body,{'X-Down-Cursor':'1','Content-Length':String(body.byteLength)});await flush();
assert.deepEqual(binary(env).map(data=>new Uint8Array(data)[0]),[2,4,3]);
assert.equal(env.pending('/api/v1/down')[0].options.headers['X-Down-Cursor'],'1');
assert.equal(env.requests.filter(r=>r.url.endsWith('/api/v1/down')).length,3,'successful replay is consumed without another old-cursor request');env.close();await flush();
}
});
test('changed replay prefix fails closed without duplicate native frames',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const data=frame(2,1,[7]),tail=frame(2,1,[8]),body=join(data,tail),stream=streaming(env,env.pending('/api/v1/down')[0],'1',body.byteLength);
stream.enqueue(new Uint8Array(data));await flush();stream.error(new Error('lost'));await flush();
const changed=join(frame(2,1,[9]),tail);env.answer(env.pending('/api/v1/down')[0],200,changed,{'X-Down-Cursor':'1','Content-Length':String(changed.byteLength)});await flush();
assert.equal(binary(env).length,1);assert.ok(env.received.some(value=>value&&value.t==='close'));env.close();await flush();
});
test('partial frame boundaries never escape to native and cancellation owns every lease',async page=>{
const env=await session(page,'https-lanes',4,true);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const data=frame(2,1,[1,2,3]),stream=streaming(env,env.pending('/api/v1/down')[0],'1',data.byteLength);
stream.enqueue(new Uint8Array(data,0,5));await flush();assert.equal(binary(env).length,0);
stream.enqueue(new Uint8Array(data,5));await flush();assert.equal(binary(env).length,1);
env.close();await flush();assert.equal(binary(env).length,1);
});
test('post-commit lane batching keeps OPEN and DATA from one native message together',async page=>{
const env=await session(page,'https-lanes',1);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const body=join(frame(1,2),frame(2,2,[7]));env.send(body);await flush();
assert.equal(env.pending('/api/v1/up')[0].options.body.byteLength,body.byteLength);env.close();await flush();
});
test('page-owned HTTP method and identity survive frozen retries',async page=>{
const expected=/const carrierMethod='(POST|PUT)'/.exec(page)[1],env=await session(page,'https',4);
assert.equal(env.requests.find(r=>r.url.endsWith('/api/v1/session')).options.method,'POST');
env.send(frame(1,1));await flush();const first=env.pending('/api/v1/up')[0];ack(env,first);await flush();
env.send(frame(2,1,[7]));await flush();const original=env.pending('/api/v1/up')[0];
original.answered=true;original.reject(new Error('lost'));await flush();const replay=env.pending('/api/v1/up')[0];
assert.equal(replay.options.body,original.options.body);assert.deepEqual(replay.options.headers,original.options.headers);
assert.equal(replay.options.method,expected);ack(env,replay);await flush();
for(const request of env.requests.filter(r=>/\/api\/v1\/(up|down)$/.test(r.url)))assert.equal(request.options.method,expected);
env.close();await flush();assert.equal(env.requests.find(r=>r.options.method==='DELETE').options.method,'DELETE');
});
test('RTT scales time but adds no OPEN ACK barrier or coalescing timer',async page=>{
for(const rtt of [40,100,400]){
const env=await session(page,'https',4);env.send(frame(1,1));await flush();await env.tick(1);
env.send(frame(2,1,[5]));await flush();assert.equal(env.pending('/api/v1/up').length,2);
await env.tick(rtt-1);for(const request of env.pending('/api/v1/up'))ack(env,request);await flush();
assert.equal(env.received.filter(value=>value instanceof ArrayBuffer&&new Uint8Array(value)[0]===17).length,1);
assert.ok(env.received.filter(value=>value&&value.t==='status').every(value=>Object.keys(value).sort().join(',')==='state,t'));
env.close();await flush();
}
});
test('simultaneous failed lane polls each consume their own exact replay',async page=>{
const env=await session(page,'https-lanes',4);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
env.send(frame(1,2));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const streams=[];
for(const request of env.pending('/api/v1/down')){
const id=Number(request.options.headers['X-Lane-ID']),data=frame(2,id,[id]),body=join(data,frame(4,id,[0,0,0,1]));
const response=streaming(env,request,'1',body.byteLength);response.enqueue(new Uint8Array(data));streams.push({id,response,body});
}
await flush();assert.equal(binary(env).length,2);for(const item of streams)item.response.error(new Error('lost'));await flush();
for(let round=0;round<3;round++){
for(const request of env.pending('/api/v1/down').filter(r=>r.options.headers['X-Down-Cursor']==='0')){
const body=streams.find(item=>String(item.id)===request.options.headers['X-Lane-ID']).body;
env.answer(request,200,body,{'X-Down-Cursor':'1','Content-Length':String(body.byteLength)});
}await flush();
}
assert.equal(binary(env).length,4);assert.equal(env.pending('/api/v1/down').length,2);
assert.ok(env.pending('/api/v1/down').every(r=>r.options.headers['X-Down-Cursor']==='1'));env.close();await flush();
});
test('incremental decoder rejects changed cursor, invalid WINDOW and cross-lane frames',async page=>{
for(const invalid of ['cursor','credit','lane']){
const env=await session(page,'https-lanes',4);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const body=invalid==='credit'?frame(4,1,[0,0,0,0]):frame(2,invalid==='lane'?2:1,[1]);
env.answer(env.pending('/api/v1/down')[0],200,body,{'X-Down-Cursor':invalid==='cursor'?'2':'1','Content-Length':String(body.byteLength)});await flush();
assert.equal(binary(env).length,0);assert.ok(env.received.some(value=>value&&value.t==='close'));env.close();await flush();
}
});
test('a replay without Content-Length cannot truncate an already known batch',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const prefix=frame(2,1,[1]),tail=frame(2,1,[2]),body=join(prefix,tail),response=streaming(env,env.pending('/api/v1/down')[0],'1',body.byteLength);
response.enqueue(new Uint8Array(prefix));await flush();response.error(new Error('lost'));await flush();
env.answer(env.pending('/api/v1/down')[0],200,prefix,{'X-Down-Cursor':'1'});await flush();
assert.equal(binary(env).length,1);assert.ok(env.received.some(value=>value&&value.t==='close'));
assert.ok(!env.requests.some(r=>r.url.endsWith('/api/v1/down')&&r.options.headers['X-Down-Cursor']==='1'));env.close();await flush();
});
test('a pipelined startup flight adopts the recovery budget once another ACK commits',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();env.send(frame(2,1,[7]));await flush();
const [head,tail]=env.pending('/api/v1/up');ack(env,head);await flush();tail.answered=true;tail.reject(new Error('lost'));await flush();
const replay=env.pending('/api/v1/up')[0];assert.ok(replay,'post-commit recovery replays immediately instead of staying in the startup backoff');
assert.equal(replay.options.body,tail.options.body);ack(env,replay);await flush();env.close();await flush();
});
test('candidate rollback restores every unacknowledged OPEN and DATA without a second WELCOME',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();env.send(frame(2,1,[7]));await flush();
const retired=env.requests.filter(r=>/\/api\/v1\/(up|down)$/.test(r.url));await env.tick(3001);
const next=env.pending('/api/v1/session')[0];assert.ok(next,'candidate fallback creates a new session');
env.answer(next,200,frame(17),{'X-Session-Token':'C'.repeat(43),'X-Down-Cursor':'0','X-Carrier-Mode':'https-lanes','X-Carrier-Attempt':'2','X-Carrier-Candidate-Count':'4','X-Carrier-Deadline':'12','X-Carrier-State':'provisional','X-Telemt-Up-Window':'4'});await flush();
const first=env.requests.find(r=>r.url.endsWith('/api/v1/up')&&r.options.headers.Authorization==='Bearer '+'C'.repeat(43));
assert.ok(first);assert.deepEqual(new Uint8Array(first.options.body),new Uint8Array(join(frame(1,1),frame(2,1,[7]))));
assert.ok(retired.every(r=>r.options.signal.aborted));assert.equal(env.received.filter(value=>value instanceof ArrayBuffer&&new Uint8Array(value)[0]===17).length,1);
ack(env,first);await flush();env.close();await flush();
});
test('downlink prefix reservations are bounded, cancellable and released on retirement',async page=>{
const env=environment(page),budget=env.context.TelemtBridgeDownlink.budget(()=>16),first=await budget.acquire(16);
const controller=new AbortController();let acquired=false;
const parked=budget.acquire(8,controller.signal).then(release=>{acquired=true;release()},()=>{});await flush();assert.equal(acquired,false);
controller.abort();await parked;first();assert.equal(budget.used(),0);
const release=await budget.acquire(16);release();release();assert.equal(budget.used(),0);env.close();await flush();
});
test('two interrupted and differently chunked replays never duplicate DATA WINDOW or CLOSE',async page=>{
const env=await session(page,'https',4);env.send(frame(1,1));await flush();ack(env,env.pending('/api/v1/up')[0]);await flush();
const data=frame(2,1,[1]),credit=frame(4,1,[0,0,0,1]),close=frame(3,1),body=join(data,credit,close);
const first=streaming(env,env.pending('/api/v1/down')[0],'1',body.byteLength);
first.enqueue(new Uint8Array(data));await flush();first.error(new Error('lost once'));await flush();
const second=streaming(env,env.pending('/api/v1/down')[0],'1',body.byteLength);
const prefix=new Uint8Array(join(data,credit));second.enqueue(prefix.subarray(0,3));second.enqueue(prefix.subarray(3));await flush();
assert.deepEqual(binary(env).map(value=>new Uint8Array(value)[0]),[2,4]);second.error(new Error('lost twice'));await flush();await env.tick(1000);
const last=streaming(env,env.pending('/api/v1/down')[0],'1',body.byteLength);
for(const byte of new Uint8Array(body))last.enqueue(new Uint8Array([byte]));last.close();
for(let i=0;i<20&&(!env.pending('/api/v1/down').length);i++)await flush();
assert.deepEqual(binary(env).map(value=>new Uint8Array(value)[0]),[2,4,3]);assert.equal(env.pending('/api/v1/down')[0].options.headers['X-Down-Cursor'],'1');
env.close();await flush();
});
test('multiple precommit lanes survive fallback without losing stream ordering',async page=>{
const env=await session(page,'https-lanes',4);env.send(join(frame(1,1),frame(1,2)));await flush();
env.send(join(frame(2,2,[2]),frame(2,1,[1])));await flush();await env.tick(3001);
const next=env.pending('/api/v1/session')[0];env.answer(next,200,frame(17),{'X-Session-Token':'C'.repeat(43),'X-Down-Cursor':'0','X-Carrier-Mode':'https','X-Carrier-Attempt':'2','X-Carrier-Candidate-Count':'4','X-Carrier-Deadline':'12','X-Carrier-State':'provisional','X-Telemt-Up-Window':'4'});await flush();
const requests=env.requests.filter(r=>r.url.endsWith('/api/v1/up')&&r.options.headers.Authorization==='Bearer '+'C'.repeat(43));
const frames=[];for(const request of requests){const bytes=new Uint8Array(request.options.body);for(let offset=0;offset<bytes.length;){const view=new DataView(bytes.buffer,offset);frames.push([bytes[offset],bytes[offset+3]]);offset+=8+view.getUint32(4);}}
for(const id of [1,2])assert.deepEqual(frames.filter(value=>value[1]===id),[[1,id],[2,id]]);
assert.equal(frames.length,4);env.close();await flush();
});
test('receiver retirement releases shared prefix capacity without exposing a partial frame',async page=>{
const env=environment(page),module=env.context.TelemtBridgeDownlink,budget=module.budget(()=>18),received=[];
function receiver(id){
let cursor='0',stream;const controller=new AbortController(),response={status:200,headers:new Headers({'X-Down-Cursor':'1','Content-Length':'18'}),body:new ReadableStream({start(value){stream=value}})};
const owner=module.create({failure:(reason,message)=>Object.assign(new Error(message),{telemtReason:reason}),alive:()=>true,cursor:()=>cursor,lane:id,limit:()=>18,budget,cancel(){},ready:()=>Promise.resolve(),deliver:data=>received.push(new Uint8Array(data)),advance:value=>{cursor=value},finished(){}});
const result=owner.read(response,controller.signal).catch(()=>{});return {owner,result,stream,controller,cursor:()=>cursor};
}
const a=receiver(1);a.stream.enqueue(new Uint8Array(frame(2,1,[1])));await flush();
const b=receiver(2);b.stream.enqueue(new Uint8Array(join(frame(2,2,[2]),frame(2,2,[3]))));b.stream.close();await flush();
assert.equal(budget.used(),18);assert.equal(received.length,1);assert.equal(b.cursor(),'0');
a.owner.close();a.controller.abort();a.stream.error(new Error('retired'));await a.result;await b.result;
assert.equal(received.length,3);assert.equal(b.cursor(),'1');assert.equal(budget.used(),0);
b.owner.close();env.close();await flush();
});
test('native close cancels every HTTP request and pagehide cannot repeat cleanup',async page=>{
for(const android of [false,true])for(const carrier of ['https','https-lanes']){
const env=await session(page,carrier,4,android);env.send(frame(1,1));await flush();env.send(frame(2,1,[7]));await flush();
const up=env.pending('/api/v1/up'),down=env.pending('/api/v1/down'),requests=up.concat(down);
assert.equal(up.length,2);assert.equal(down.length,1);assert.ok(requests.every(request=>!request.options.signal.aborted));
env.send({t:'close'});await flush();
assert.ok(requests.every(request=>request.options.signal.aborted));
assert.equal(env.requests.filter(request=>request.options.method==='DELETE').length,1);
const count=env.requests.length;env.close();await env.tick(120000);
assert.equal(env.requests.filter(request=>request.options.method==='DELETE').length,1);
assert.equal(env.requests.length,count);
}
});
(async()=>{const page=renderedPage();let failed=0;for(const {name,run} of tests){let timer;try{await Promise.race([run(page),new Promise((_,reject)=>{timer=setTimeout(()=>reject(new Error('test made no bounded progress')),5000)})]);console.log('ok - '+name)}catch(error){failed++;console.error('not ok - '+name+'\n'+error.stack)}finally{clearTimeout(timer)}}if(failed)process.exitCode=1})();
+26
View File
@@ -0,0 +1,26 @@
use std::io::Write;
use std::process::{Command, Stdio};
/// Exercises the actual generated page with deterministic network and native-boundary doubles.
pub(super) fn run(page: &super::BridgePage) {
let mut child = Command::new("node")
.args(["-e", include_str!("behavior_tests.js"), "--", "--stdin"])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("Node.js 22+ is required for executable WEB bridge regressions");
child
.stdin
.take()
.unwrap()
.write_all(page.body.as_bytes())
.unwrap();
let output = child.wait_with_output().unwrap();
assert!(
output.status.success(),
"bridge regressions failed:\n{}\n{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
}
+106
View File
@@ -0,0 +1,106 @@
(()=>{'use strict';
// HTTP request ownership is separate from the native WINDOW credit protocol.
function create(settings){
const channels=new Map(),window=settings.window,token=settings.token,downBudget=globalThis.TelemtBridgeDownlink.budget(settings.queueLimit);
let closed=false,order=0,ready=settings.committed(),resolveReady;
const readyPromise=new Promise(resolve=>{resolveReady=resolve});if(ready)resolveReady();
const alive=channel=>!closed&&!channel.closed&&settings.alive(channel.lane);
const markReady=()=>{if(!ready){ready=true;resolveReady();settings.commit()}};
function waitReady(signal){
if(ready)return Promise.resolve();
return new Promise((resolve,reject)=>{const abort=()=>reject(new Error('carrier retired'));if(signal&&signal.aborted){abort();return}if(signal)signal.addEventListener('abort',abort,{once:true});readyPromise.then(()=>{if(signal)signal.removeEventListener('abort',abort);resolve()})});
}
function get(id,type){
let channel=channels.get(id);if(channel)return channel;
const lane=id===null?null:settings.lane(id,type);if(id!==null&&!lane)return null;
channel={id,lane,pending:[],flights:new Map(),acks:new Set(),next:1,confirmed:0,cursor:'0',controller:null,polling:false,closed:false};
channel.receiver=globalThis.TelemtBridgeDownlink.create({
failure:settings.failure,alive:()=>alive(channel),cursor:()=>channel.cursor,lane:id,limit:settings.batchLimit,budget:downBudget,
cancel:settings.cancel,ready:waitReady,deliver:settings.deliver,advance:next=>{channel.cursor=next},finished:()=>{if(lane)settings.finish(lane,false)}
});channels.set(id,channel);return channel;
}
function enqueue(data,defer){
if(closed)return;
const entries=settings.lanes?settings.buffers.splitFrames(data):[{id:null,type:0,data}],touched=new Set();
for(const entry of entries){
const channel=get(entry.id,entry.type);if(!channel)continue;
if(!settings.buffers.reserve(entry.data,channel.lane))throw settings.failure('capacity','carrier queue full');
channel.pending.push(entry.data);touched.add(channel);
}
if(!defer)for(const channel of touched)pump(channel);
}
function checkAck(response,sequence){
if(response.status!==204)throw settings.failure('http','uplink rejected');
if(response.headers.get('X-Up-Ack')!==String(sequence))throw settings.failure('protocol','uplink acknowledgement rejected');
}
function acknowledge(channel,sequence,lease){
if(!alive(channel)||channel.flights.get(sequence)!==lease)return;
settings.buffers.settleBatch(lease);channel.flights.delete(sequence);channel.acks.add(sequence);
while(channel.acks.delete(channel.confirmed+1))channel.confirmed++;
markReady();settings.traffic(lease.total,0);pump(channel);
}
async function send(channel,sequence,lease){
const headers={'X-Up-Seq':String(sequence)};if(channel.id!==null)headers['X-Lane-ID']=String(channel.id);
if(window>1)headers['X-Telemt-Up-Confirmed']=String(channel.confirmed);
const frozen=settings.options(settings.method,token,lease.body,headers,lease.controller.signal);
try{
while(alive(channel)&&!lease.cancelled){
try{const response=await settings.request('/api/v1/up',frozen,null,()=>ready?1:9);if(!alive(channel))return;checkAck(response,sequence);acknowledge(channel,sequence,lease);return}
catch(error){
if(!alive(channel)||lease.cancelled)return;
if(!ready){settings.failed(error);return}
let replayed=false;
const recovered=await settings.recover(error,async(signal,remaining)=>{
const response=await settings.request('/api/v1/up',Object.assign({},frozen,{signal}),remaining,2);
if(!alive(channel)||lease.cancelled)return;checkAck(response,sequence);replayed=true;acknowledge(channel,sequence,lease);
});
if(!recovered||!alive(channel)||replayed)return;
}
}
}catch(error){if(alive(channel)&&!lease.cancelled)settings.failed(error)}
}
function pump(channel){
while(alive(channel)&&channel.pending.length&&channel.next<=channel.confirmed+window){
const lease=settings.buffers.takeBatch(channel.pending,channel.lane),sequence=channel.next++;
lease.controller=new AbortController();lease.order=order++;channel.flights.set(sequence,lease);send(channel,sequence,lease);
if(!channel.polling)poll(channel);
}
}
async function poll(channel){
if(channel.polling||!alive(channel))return;channel.polling=true;
try{
while(alive(channel)){
const cursor=channel.cursor,headers={'X-Down-Cursor':cursor};if(channel.id!==null)headers['X-Lane-ID']=String(channel.id);
channel.controller=new AbortController();const frozen=settings.options(settings.method,token,null,headers,channel.controller.signal);
try{
const response=await settings.request('/api/v1/down',frozen,null,1,channel.receiver.read);
if(!alive(channel))return;if(response.status!==200&&response.status!==204)throw settings.failure('http','downlink rejected');
if(ready)settings.connected();
}catch(error){
if(!alive(channel))return;
if(!ready){settings.failed(error);return}
const recovered=await settings.recover(error,async(signal,remaining)=>{
const response=await settings.request('/api/v1/down',Object.assign({},frozen,{signal}),remaining,2,channel.receiver.read);
if(response.status!==200&&response.status!==204)throw settings.failure('http','downlink replay rejected');
});
if(!recovered||!alive(channel))return;
}
}
}catch(error){if(alive(channel))settings.failed(error)}
finally{channel.polling=false;channel.controller=null}
}
function retire(channel,restore){
channel.closed=true;if(channel.controller)channel.controller.abort();channel.receiver.close();
const saved=[];
for(const lease of channel.flights.values()){if(restore)saved.push({order:lease.order,data:lease.body});settings.buffers.cancelBatch(lease);}
channel.flights.clear();channel.acks.clear();
for(const data of channel.pending)if(restore)saved.push({order:order++,data});
settings.buffers.releasePending(channel.pending,channel.lane);return saved;
}
return {enqueue,flush(){for(const channel of channels.values())pump(channel)},finish(lane){const channel=channels.get(lane.id);if(channel){channels.delete(lane.id);retire(channel,false)}},close(restore){
if(closed)return [];closed=true;const saved=[];for(const channel of channels.values())saved.push(...retire(channel,restore));channels.clear();
return saved.sort((a,b)=>a.order-b.order).map(entry=>entry.data);
}};
}
globalThis.TelemtBridgeConveyor=Object.freeze({create});
})();

Some files were not shown because too many files have changed in this diff Show More