Compare commits

..

12 Commits

Author SHA1 Message Date
Alexey c4555e25f3 Merge pull request #945 from telemt/bump/3.5.12
Bump -> 3.5.12
2026-10-03 13:01:07 +03:00
Alexey d15d50e92a Bump -> 3.5.12 2026-10-03 12:59:58 +03:00
Alexey 557ffe9ee0 Merge pull request #944 from telemt/me-nat-discovery
ME NAT Discovery Cancellation correctness
2026-10-03 12:56:57 +03:00
Alexey 84d5e98c11 ME NAT Discovery Cancellation correctness
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-03 12:36:29 +03:00
Alexey 642f166c98 Merge pull request #943 from telemt/carrier-methods
Configurable HTTP Methods for Bridge Carriers + Fix autonomous ME recovery from Direct fallback
2026-10-03 11:14:09 +03:00
Alexey d65efcc479 Fix autonomous ME recovery from Direct fallback
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-02 23:51:14 +03:00
Alexey ca5ebd1308 Merge pull request #942 from nomah4/docs/web-proxy-caddy
docs(web): add Caddy TLS termination example
2026-10-02 23:46:47 +03:00
Mikhail Gorbachev e0bf6fa3a0 docs(web): add Caddy TLS termination example
Add a Caddy section next to the NGINX and HAProxy examples in
WEB_PROXY.{en,ru,de}.md: minimal config, prefix-only cohosting with
handle (not handle_path), access-log redaction for the bridge capability
and bearer credentials, and notes on X-Forwarded-For, buffering,
timeouts, retries and WebSocket lanes across a Caddy reload.

Closes #914
2026-10-02 09:19:40 +07:00
Alexey fca6506449 Configurable HTTP Methods for Bridge Carriers
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-01 23:40:08 +03:00
Alexey 94f4f7d540 Merge pull request #939 from telemt/conntrack
Сonntrack recovery on absent iptables-nft chains fixes
2026-10-01 23:03:27 +03:00
Alexey c10677ed97 Bump -> 3.5.11 2026-10-01 22:54:07 +03:00
Alexey 86912e34b2 Сonntrack recovery on absent iptables-nft chains fixes
Co-Authored-By: brekotis <93345790+brekotis@users.noreply.github.com>
2026-10-01 22:50:50 +03:00
52 changed files with 4407 additions and 183 deletions
Generated
+1 -1
View File
@@ -2780,7 +2780,7 @@ dependencies = [
[[package]]
name = "telemt"
version = "3.5.10"
version = "3.5.12"
dependencies = [
"aes",
"anyhow",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "telemt"
version = "3.5.10"
version = "3.5.12"
edition = "2024"
rust-version = "1.88"
+63 -3
View File
@@ -2,7 +2,7 @@
[English](WEB_PROXY.en.md) | [Русский](WEB_PROXY.ru.md) | [Deutsch](WEB_PROXY.de.md)
Der WEB-Modus transportiert gewöhnliche MTProxy-Streams über begrenzte HTTPS- oder WebSocket-Carrier, die mit dem Proxy-Typ `WEB` von Telegram Desktop kompatibel sind. Telemt terminiert TLS nicht selbst: NGINX oder HAProxy verwaltet das öffentliche Zertifikat und leitet unverschlüsseltes HTTP/1.1 an einen privaten Telemt-Listener weiter.
Der WEB-Modus transportiert gewöhnliche MTProxy-Streams über begrenzte HTTPS- oder WebSocket-Carrier, die mit dem Proxy-Typ `WEB` von Telegram Desktop kompatibel sind. Telemt terminiert TLS nicht selbst: NGINX, HAProxy oder Caddy verwaltet das öffentliche Zertifikat und leitet unverschlüsseltes HTTP/1.1 an einen privaten Telemt-Listener weiter.
> [!IMPORTANT]
>
@@ -14,7 +14,7 @@ Der WEB-Modus transportiert gewöhnliche MTProxy-Streams über begrenzte HTTPS-
Telegram Desktop
| HTTPS oder WSS :443
v
NGINX oder HAProxy (TLS-Terminierung, kanonischer Host und eine X-Forwarded-For-Adresse)
NGINX, HAProxy oder Caddy (TLS-Terminierung, kanonischer Host und eine X-Forwarded-For-Adresse)
| unverschlüsseltes HTTP/1.1 in einem privaten Netz
v
Telemt-WEB-Listener
@@ -270,6 +270,66 @@ backend telemt_web
Im Frontend oder im Abschnitt `defaults` muss für das standardmäßige WebSocket-Liveness-Intervall auch `timeout client 65s` oder länger gesetzt sein. Für `https-lanes` muss das öffentliche HAProxy-ALPN `h2`, für WebSocket-Upgrade außerdem `http/1.1` enthalten. Bewahren Sie `Connection`, `Upgrade` und `Sec-WebSocket-*` unverändert; Pfad, Raw Query, Body sowie die Carrier-Header `Authorization`, `Content-Type`, `X-Up-Seq`, `X-Down-Cursor` und `X-Lane-ID` dürfen nicht umgeschrieben werden. Fügen Sie für Prefix-only-Cohosting `acl telemt_web_path path_beg /telegram/web/` hinzu und verlangen Sie in `use_backend` sowohl Host- als auch Pfad-ACL; entfernen Sie das Präfix nicht.
## TLS-Terminierung mit Caddy
```caddyfile
{
servers {
protocols h1 h2
}
}
proxy.example.com {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
```
Caddy bezieht und erneuert das öffentliche Zertifikat für die Site-Adresse automatisch. Den ursprünglichen `Host` sowie die Header `Connection`, `Upgrade` und `Sec-WebSocket-*` leitet Caddy unverändert weiter, WebSocket-Upgrade wird ohne zusätzliche Direktiven proxied; fügen Sie kein `header_up Host` hinzu. Öffentliches HTTP/2 (für `https-lanes`) und HTTP/1.1 (für WebSocket-Upgrade) sind standardmäßig aktiv, der private Abschnitt Caddy → Telemt bleibt HTTP/1.1. Die globale Option `protocols h1 h2` hält HTTP/3 aus dem WEB-Pfad heraus, weil der WEB-Vertrag nur HTTP/1.1 und HTTP/2 abdeckt. Sie gilt für alle Sites dieses Caddy-Servers.
`header_up X-Forwarded-For {remote_host}` überschreibt den Header mit genau einer Adresse. Ohne diese Zeile hängt Caddy bei Peers aus `trusted_proxies` an einen eingehenden `X-Forwarded-For` an, Telemt akzeptiert jedoch nur eine parsebare IP-Adresse. Steht Caddy selbst hinter einem L4-Balancer mit `listener_wrappers { proxy_protocol }`, ist `{remote_host}` die Client-Adresse aus dem PROXY-Header. `flush_interval -1` deaktiviert die Antwortpufferung für Long Polls und Downlinks. Request-Bodies streamt Caddy standardmäßig; aktivieren Sie kein `request_buffers`.
`reverse_proxy` hat in Caddy standardmäßig weder Read-/Write-Timeouts noch ein Body-Limit, daher passieren der 25-Sekunden-Long-Poll und das WebSocket-Liveness-Intervall unverändert. Sind serverweite `timeouts` oder `request_body max_size` gesetzt, müssen sie über 65 Sekunden bzw. mindestens bei `web.limits.max_body_bytes` liegen. Mit einem einzigen Upstream wiederholt Caddy fehlgeschlagene Anfragen nicht; fügen Sie weder `lb_retries` noch `lb_try_duration` hinzu. Bei einem Konfigurations-Reload schließt Caddy upgegradete Verbindungen standardmäßig sofort. `stream_close_delay` hält bestehende WebSocket-Lanes nach einem Reload für die angegebene Zeit offen.
Leiten Sie für Prefix-only-Cohosting mit `base_path = "telegram/web"` den Teilbaum über `handle` und nicht über `handle_path` weiter, denn `handle_path` entfernt das Präfix:
```caddyfile
proxy.example.com {
handle /telegram/web/* {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
handle {
# gewöhnliche Site
}
}
```
Das Routing von Caddy fügt keine Weiterleitung mit angehängtem Slash hinzu: `/telegram/web` ohne abschließenden Slash landet im Handler der gewöhnlichen Site. Stellen Sie sicher, dass auch dieser Handler nicht weiterleitet; `file_server` leitet etwa eine Verzeichnisanfrage auf die Variante mit Slash um.
Ohne `log`-Direktive schreibt Caddy kein Access-Log. Wird eines benötigt, schwärzen Sie Bridge-Capability und Bearer-Credentials:
```caddyfile
log {
format filter {
request>uri query {
replace bridge REDACTED
}
request>headers>Authorization delete
request>headers>Sec-Websocket-Protocol delete
}
}
```
Tragen Sie in `web_trusted_proxy_cidrs` die Adresse ein, von der aus Caddy Telemt erreicht, für die obige Konfiguration also `["127.0.0.1/32"]`.
## Lebenszyklus und Reload-Verhalten
| Konfiguration | Runtime-Verhalten |
@@ -431,7 +491,7 @@ Der vollständige Vertrag für Requests, Revisionen, Fehler und alle Benutzer-En
| `/telegram/web` leitet auf `/telegram/web/` um | Fügen Sie für den Pfad ohne Schrägstrich einen exakten Non-WEB-Handler hinzu. Nur der mit Schrägstrich abgeschlossene konfigurierte Teilbaum gehört zum WEB-Vertrag von Telemt. |
| Ein konkurrierender `https-lanes`-Downlink erreicht den Decoy mit `404` | Prüfen Sie, dass er mit `X-Down-Cursor: 0` beginnt, bewahren Sie `X-Lane-ID` und setzen Sie `lane_open_wait_secs` über den beobachteten Abstand zwischen Downlink und `OPEN`. Fortgeschrittene Cursor fehlender Lanes schlagen absichtlich fail-closed fehl. |
| Auto-Negotiation wechselt weiter, nachdem Daten bereits akzeptiert wurden | Das ist ungültig. Prüfen Sie das authentifizierte `X-Carrier-State`-Replay und das Carrier-Commit-Lifecycle-Ereignis; `committed` oder `healthy` ist terminal und erfordert eine neue Sitzung. |
| Long Polls werden nach einem festen Intervall getrennt | Setzen Sie Client-, Server-, Sende- und Lese-Timeouts von NGINX/HAProxy über `web.timeouts.long_poll_secs`. |
| Long Polls werden nach einem festen Intervall getrennt | Setzen Sie Client-, Server-, Sende- und Lese-Timeouts von NGINX/HAProxy/Caddy über `web.timeouts.long_poll_secs`. |
| WebSocket-Upgrade erreicht statt `101` den Decoy | Bewahren Sie HTTP/1.1 `Connection: Upgrade`, `Upgrade: websocket`, das einzelne exakte `Sec-WebSocket-Protocol` und den kanonischen bodylosen Request an der konfigurierten Basis plus `/api/v1/ws`. Prüfen Sie außerdem Carrier-/Session-Kompatibilität und die Prozess-Verbindungsreserve. |
| Ein `websocket-lanes`-Stream wurde geschlossen, Geschwister bleiben aber verbunden | Dies ist die beabsichtigte Fehlergrenze. Prüfen Sie die Message-/Frame-Zeilen dieser Lane in `/web-status`; fehlerhafte oder lane-fremde Frames, Write-Timeouts und Backend-Close schließen nur die betroffene Lane. |
| `/web-status` ist leer | Prüfen Sie, dass `[web.debug].enabled = true` gesetzt ist, wenden Sie die Konfiguration an, wählen Sie ein Fenster innerhalb von `max_window_secs` und erzeugen Sie nach der Policy-Änderung neuen WEB-Datenverkehr. |
+63 -3
View File
@@ -2,7 +2,7 @@
[English](WEB_PROXY.en.md) | [Русский](WEB_PROXY.ru.md) | [Deutsch](WEB_PROXY.de.md)
WEB mode carries ordinary MTProxy streams through bounded HTTPS or WebSocket carriers compatible with Telegram Desktop's `WEB` proxy type. Telemt does not terminate TLS: NGINX or HAProxy owns the public certificate and forwards plain HTTP/1.1 to a private Telemt listener.
WEB mode carries ordinary MTProxy streams through bounded HTTPS or WebSocket carriers compatible with Telegram Desktop's `WEB` proxy type. Telemt does not terminate TLS: NGINX, HAProxy, or Caddy owns the public certificate and forwards plain HTTP/1.1 to a private Telemt listener.
> [!IMPORTANT]
>
@@ -14,7 +14,7 @@ WEB mode carries ordinary MTProxy streams through bounded HTTPS or WebSocket car
Telegram Desktop
| HTTPS or WSS :443
v
NGINX or HAProxy (TLS termination, canonical Host and one X-Forwarded-For address)
NGINX, HAProxy, or Caddy (TLS termination, canonical Host and one X-Forwarded-For address)
| plain HTTP/1.1 on a private network
v
Telemt WEB listener
@@ -270,6 +270,66 @@ backend telemt_web
The frontend or `defaults` section must also set `timeout client 65s` or longer for the default WebSocket liveness interval. HAProxy's public ALPN must include `h2` for `https-lanes` and `http/1.1` for WebSocket Upgrade. Preserve `Connection`, `Upgrade`, and `Sec-WebSocket-*`; do not rewrite the path, raw query, body, or the `Authorization`, `Content-Type`, `X-Up-Seq`, `X-Down-Cursor`, and `X-Lane-ID` carrier headers. For prefix-only cohosting, add `acl telemt_web_path path_beg /telegram/web/` and require both the host and path ACLs on `use_backend`; do not remove the prefix.
## Caddy TLS termination
```caddyfile
{
servers {
protocols h1 h2
}
}
proxy.example.com {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
```
Caddy obtains and renews the public certificate for the site address automatically. It forwards the original `Host` and the `Connection`, `Upgrade`, and `Sec-WebSocket-*` headers unchanged and proxies WebSocket Upgrade without extra directives; do not add `header_up Host`. Public HTTP/2 (needed for `https-lanes`) and HTTP/1.1 (needed for WebSocket Upgrade) are both enabled by default, while the private Caddy-to-Telemt hop stays HTTP/1.1. The global `protocols h1 h2` option keeps HTTP/3 out of the WEB path, because the WEB contract only covers HTTP/1.1 and HTTP/2. It applies to every site of that Caddy server.
`header_up X-Forwarded-For {remote_host}` overwrites the header with one address. Without it, Caddy appends to an incoming `X-Forwarded-For` from peers listed in `trusted_proxies`, and Telemt accepts only one parseable IP address. If Caddy itself sits behind an L4 balancer with `listener_wrappers { proxy_protocol }`, `{remote_host}` is the client address taken from the PROXY header. `flush_interval -1` disables response buffering for long polls and downlinks. Request bodies are streamed by default; do not enable `request_buffers`.
Caddy's `reverse_proxy` has no read or write timeout and no request-body limit by default, so the 25-second long poll and the WebSocket liveness interval pass unchanged. If server-wide `timeouts` or `request_body max_size` are configured, keep them above 65 seconds and at least `web.limits.max_body_bytes` respectively. With a single upstream Caddy does not retry failed requests; do not add `lb_retries` or `lb_try_duration`. On a configuration reload Caddy closes upgraded connections immediately by default. `stream_close_delay` keeps established WebSocket lanes open across a reload for the given time.
For prefix-only cohosting with `base_path = "telegram/web"`, route the subtree with `handle`, not `handle_path`, because `handle_path` strips the prefix:
```caddyfile
proxy.example.com {
handle /telegram/web/* {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
handle {
# ordinary site
}
}
```
Caddy's routing does not add a slash-appending redirect: `/telegram/web` without the trailing slash falls through to the ordinary site handler. Make sure that handler does not redirect it either; for example, `file_server` redirects a directory request to its slash-terminated form.
Caddy writes no access log unless a `log` directive is present. If one is needed, redact the bridge capability and bearer credentials:
```caddyfile
log {
format filter {
request>uri query {
replace bridge REDACTED
}
request>headers>Authorization delete
request>headers>Sec-Websocket-Protocol delete
}
}
```
Set `web_trusted_proxy_cidrs` to the address Caddy uses to connect to Telemt, for example `["127.0.0.1/32"]` for the configuration above.
## Lifecycle and reload behavior
| Configuration | Runtime behavior |
@@ -431,7 +491,7 @@ See the complete [Control API contract](../Architecture/API/API.md) for request
| `/telegram/web` redirects to `/telegram/web/` | Add an exact non-WEB handler for the no-slash path. Only the slash-terminated configured subtree belongs to Telemt's WEB contract. |
| A racing `https-lanes` downlink reaches the decoy with `404` | Confirm it starts at `X-Down-Cursor: 0`, preserve `X-Lane-ID`, and set `lane_open_wait_secs` above the observed down-before-`OPEN` skew. Advanced cursors for missing lanes intentionally fail closed. |
| Auto-negotiation advances after traffic was already accepted | This is not valid behavior. Inspect the authenticated `X-Carrier-State` replay and the carrier commit lifecycle row; a committed or healthy response is terminal and requires a new session. |
| Long polls disconnect near a fixed interval | Raise NGINX/HAProxy client, server, send, and read timeouts above `web.timeouts.long_poll_secs`. |
| Long polls disconnect near a fixed interval | Raise NGINX/HAProxy/Caddy client, server, send, and read timeouts above `web.timeouts.long_poll_secs`. |
| WebSocket Upgrade reaches the decoy instead of returning `101` | Preserve HTTP/1.1 `Connection: Upgrade`, `Upgrade: websocket`, the single exact `Sec-WebSocket-Protocol`, and the canonical bodyless request at the configured base plus `/api/v1/ws`. Also check carrier/session compatibility and the process connection reserve. |
| One `websocket-lanes` stream closes while siblings stay connected | This is the intended failure boundary. Inspect that lane's message/frame rows in `/web-status`; malformed, cross-lane, write-timeout, and backend-close paths terminate only the affected lane. |
| `/web-status` is empty | Confirm `[web.debug].enabled = true`, apply the configuration, select a window within `max_window_secs`, and generate new WEB traffic after the policy change. |
+63 -3
View File
@@ -2,7 +2,7 @@
[English](WEB_PROXY.en.md) | [Русский](WEB_PROXY.ru.md) | [Deutsch](WEB_PROXY.de.md)
WEB-режим переносит обычные MTProxy-потоки через bounded HTTPS или WebSocket carriers, совместимые с типом прокси `WEB` в Telegram Desktop. Telemt не терминирует TLS: публичный сертификат обслуживает NGINX или HAProxy, который передаёт обычный HTTP/1.1 на приватный listener Telemt.
WEB-режим переносит обычные MTProxy-потоки через bounded HTTPS или WebSocket carriers, совместимые с типом прокси `WEB` в Telegram Desktop. Telemt не терминирует TLS: публичный сертификат обслуживает NGINX, HAProxy или Caddy, который передаёт обычный HTTP/1.1 на приватный listener Telemt.
> [!IMPORTANT]
>
@@ -14,7 +14,7 @@ WEB-режим переносит обычные MTProxy-потоки через
Telegram Desktop
| HTTPS или WSS :443
v
NGINX или HAProxy (TLS termination, канонический Host и один адрес X-Forwarded-For)
NGINX, HAProxy или Caddy (TLS termination, канонический Host и один адрес X-Forwarded-For)
| обычный HTTP/1.1 в приватной сети
v
WEB-listener Telemt
@@ -270,6 +270,66 @@ backend telemt_web
Во frontend или секции `defaults` также задайте `timeout client 65s` или больше для default WebSocket liveness interval. Для `https-lanes` публичный ALPN HAProxy должен содержать `h2`, а для WebSocket Upgrade — `http/1.1`. Сохраняйте `Connection`, `Upgrade` и `Sec-WebSocket-*`; не переписывайте path, raw query, body и carrier headers `Authorization`, `Content-Type`, `X-Up-Seq`, `X-Down-Cursor`, `X-Lane-ID`. Для prefix-only cohosting добавьте `acl telemt_web_path path_beg /telegram/web/` и потребуйте одновременно host- и path-ACL в `use_backend`; не удаляйте prefix.
## Терминация TLS на Caddy
```caddyfile
{
servers {
protocols h1 h2
}
}
proxy.example.com {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
```
Caddy сам получает и продлевает публичный сертификат для адреса сайта. Он передаёт исходный `Host` и заголовки `Connection`, `Upgrade` и `Sec-WebSocket-*` без изменений и проксирует WebSocket Upgrade без дополнительных директив; не добавляйте `header_up Host`. Публичные HTTP/2 (нужен для `https-lanes`) и HTTP/1.1 (нужен для WebSocket Upgrade) включены по умолчанию, а приватный участок Caddy → Telemt остаётся HTTP/1.1. Глобальная опция `protocols h1 h2` убирает HTTP/3 с пути WEB: контракт WEB описывает только HTTP/1.1 и HTTP/2. Она действует на все сайты этого сервера Caddy.
`header_up X-Forwarded-For {remote_host}` перезаписывает заголовок одним адресом. Без неё Caddy дописывает адрес к входящему `X-Forwarded-For` от пиров из `trusted_proxies`, а Telemt принимает только один разбираемый IP-адрес. Если сам Caddy стоит за L4-балансировщиком с `listener_wrappers { proxy_protocol }`, `{remote_host}` — адрес клиента из заголовка PROXY. `flush_interval -1` отключает буферизацию ответа для long polls и downlink. Тела запросов Caddy по умолчанию передаёт потоком; не включайте `request_buffers`.
У `reverse_proxy` в Caddy по умолчанию нет read/write timeout и лимита тела запроса, поэтому 25-секундный long poll и WebSocket liveness interval проходят без изменений. Если заданы серверные `timeouts` или `request_body max_size`, держите их выше 65 секунд и не меньше `web.limits.max_body_bytes` соответственно. С одним upstream Caddy не повторяет неудачные запросы; не добавляйте `lb_retries` и `lb_try_duration`. При reload конфигурации Caddy по умолчанию сразу закрывает upgraded-соединения. `stream_close_delay` держит установленные WebSocket lanes открытыми на заданное время после reload.
Для prefix-only cohosting с `base_path = "telegram/web"` направляйте поддерево через `handle`, а не `handle_path`: `handle_path` срезает prefix.
```caddyfile
proxy.example.com {
handle /telegram/web/* {
reverse_proxy 127.0.0.1:18080 {
header_up X-Forwarded-For {remote_host}
flush_interval -1
stream_close_delay 5m
}
}
handle {
# обычный сайт
}
}
```
Маршрутизация Caddy не добавляет редирект со слэшем: `/telegram/web` без завершающего слэша уходит в обработчик обычного сайта. Убедитесь, что и этот обработчик не редиректит его; например, `file_server` перенаправляет запрос каталога на вариант со слэшем.
Caddy не пишет access log без директивы `log`. Если журнал нужен, скройте bridge capability и bearer credentials:
```caddyfile
log {
format filter {
request>uri query {
replace bridge REDACTED
}
request>headers>Authorization delete
request>headers>Sec-Websocket-Protocol delete
}
}
```
В `web_trusted_proxy_cidrs` укажите адрес, с которого Caddy подключается к Telemt, например `["127.0.0.1/32"]` для конфигурации выше.
## Lifecycle и reload
| Конфигурация | Поведение runtime |
@@ -431,7 +491,7 @@ curl -sS -X POST http://127.0.0.1:9091/v1/users/web-user/rotate-secret \
| `/telegram/web` перенаправляет на `/telegram/web/` | Добавьте точный non-WEB handler для path без слеша. В WEB-контракт Telemt входит только настроенное поддерево с завершающим слешем. |
| Downlink `https-lanes`, участвующий в гонке, попадает в decoy с `404` | Убедитесь, что он начинается с `X-Down-Cursor: 0`, сохраняйте `X-Lane-ID` и задайте `lane_open_wait_secs` выше наблюдаемого разрыва down-before-`OPEN`. Продвинутый cursor отсутствующей lane намеренно закрывается fail-closed. |
| Auto-negotiation переходит дальше после уже принятого трафика | Такое поведение некорректно. Проверьте аутентифицированный replay `X-Carrier-State` и lifecycle row commit carrier; ответ `committed` или `healthy` terminal и требует новой сессии. |
| Long polls разрываются через фиксированный интервал | Поднимите client, server, send и read timeouts NGINX/HAProxy выше `web.timeouts.long_poll_secs`. |
| Long polls разрываются через фиксированный интервал | Поднимите client, server, send и read timeouts NGINX/HAProxy/Caddy выше `web.timeouts.long_poll_secs`. |
| WebSocket Upgrade попадает в decoy вместо `101` | Сохраните HTTP/1.1 `Connection: Upgrade`, `Upgrade: websocket`, единственный точный `Sec-WebSocket-Protocol` и канонический bodyless request по настроенному base плюс `/api/v1/ws`. Также проверьте соответствие carrier/session и process connection reserve. |
| Один stream `websocket-lanes` закрылся, а siblings остались подключены | Это штатная failure boundary. Проверьте message/frame rows этой lane в `/web-status`; malformed, cross-lane, write-timeout и backend-close закрывают только затронутую lane. |
| `/web-status` пуст | Убедитесь, что `[web.debug].enabled = true`, примените конфигурацию, выберите окно в пределах `max_window_secs` и создайте новый WEB-трафик после изменения policy. |
+49
View File
@@ -1,5 +1,54 @@
use super::*;
#[tokio::test]
async fn carrier_method_api_defaults_and_patches_are_hot() {
let (path, _directory) = temp_config("[web]\nenabled = false\n");
let (value, _) = read_managed_config(&path).await.unwrap();
assert_eq!(value["web"]["carrier_method"].as_str(), Some("post"));
for token in ["put", "post"] {
let active = ProxyConfig::load(&path).unwrap();
let patch = serde_json::json!({"web": {"carrier_method": token}});
let mut response = apply_patch_to_path(&path, &patch, None).await.unwrap();
let desired = ProxyConfig::load(&path).unwrap();
reconcile_runtime_effect(&mut response, &active, &desired).unwrap();
assert!(!response.restart_required);
assert!(response.runtime_reload_required);
assert!(!response.process_restart_required);
assert!(response.deferred_process_fields.is_empty());
let (value, revision) = read_managed_config(&path).await.unwrap();
assert_eq!(value["web"]["carrier_method"].as_str(), Some(token));
assert_eq!(revision, response.revision);
let written = tokio::fs::read_to_string(&path).await.unwrap();
assert!(written.contains(&format!("carrier_method = \"{token}\"")));
}
}
#[tokio::test]
async fn carrier_method_api_rejects_invalid_values_without_writing() {
let (path, _directory) = temp_config("[web]\ncarrier_method = \"put\"\n");
let original = tokio::fs::read(&path).await.unwrap();
let revision = crate::api::config_store::current_revision(&path)
.await
.unwrap();
for value in [
serde_json::json!("PUT"),
serde_json::json!("patch"),
serde_json::json!(true),
serde_json::json!(42),
] {
let patch = serde_json::json!({"web": {"carrier_method": value}});
let error = apply_patch_to_path(&path, &patch, None).await.unwrap_err();
assert_eq!(error.status, hyper::StatusCode::BAD_REQUEST);
assert_eq!(tokio::fs::read(&path).await.unwrap(), original);
assert_eq!(
crate::api::config_store::current_revision(&path)
.await
.unwrap(),
revision
);
}
}
#[test]
fn json_object_converts_to_toml_table() {
let j: Json = serde_json::json!({"censorship": {"tls_domain": "a.com"}, "default_dc": 2});
+2 -1
View File
@@ -59,7 +59,7 @@ const DEFAULT_ME_POOL_DRAIN_SOFT_EVICT_BUDGET_PER_CORE: u16 = 16;
const DEFAULT_ME_POOL_DRAIN_SOFT_EVICT_COOLDOWN_MS: u64 = 1000;
const DEFAULT_USER_MAX_UNIQUE_IPS_WINDOW_SECS: u64 = 30;
const DEFAULT_ACCEPT_PERMIT_TIMEOUT_MS: u64 = 250;
const DEFAULT_CONNTRACK_CONTROL_ENABLED: bool = true;
const DEFAULT_CONNTRACK_CONTROL_ENABLED: bool = false;
const DEFAULT_CONNTRACK_PRESSURE_HIGH_WATERMARK_PCT: u8 = 85;
const DEFAULT_CONNTRACK_PRESSURE_LOW_WATERMARK_PCT: u8 = 70;
const DEFAULT_CONNTRACK_DELETE_BUDGET_PER_SEC: u64 = 4096;
@@ -244,6 +244,7 @@ pub(crate) fn default_accept_permit_timeout_ms() -> u64 {
DEFAULT_ACCEPT_PERMIT_TIMEOUT_MS
}
/// Keeps privileged conntrack control disabled unless explicitly requested.
pub(crate) fn default_conntrack_control_enabled() -> bool {
DEFAULT_CONNTRACK_CONTROL_ENABLED
}
+5
View File
@@ -17,6 +17,7 @@
//! | `network` | `dns_overrides` | Applied immediately |
//! | `access` | All user/quota fields | Effective immediately |
//! | `web` | Carrier, timing, and debug policy | Applied to newly issued sessions |
//! | `web` | `carrier_method` | Applied to newly rendered bridge pages |
//! Fields that require re-binding sockets (`server.listeners`, legacy
//! `server.port`, `censorship.*`, `network.*`, `use_middle_proxy`) are **not**
//! applied; a warning is emitted. SYN limiter rules are process-owned and are
@@ -67,3 +68,7 @@ use watcher::{ReloadState, reload_config};
mod base_path_tests;
#[cfg(test)]
mod tests;
// Carrier method reloads preserve page-owned requests and process-owned limits.
#[cfg(test)]
#[path = "hot_reload/carrier_method_tests.rs"]
mod carrier_method_tests;
@@ -0,0 +1,50 @@
use super::*;
use crate::config::WebCarrierMethod;
#[test]
fn carrier_method_reload_is_hot_and_preserves_process_limits() {
let mut active = ProxyConfig::default();
for method in [WebCarrierMethod::Put, WebCarrierMethod::Post] {
let mut desired = active.clone();
desired.web.carrier_method = method;
assert_eq!(classify_config_changes(&active, &desired).changed, ["web"]);
assert!(!classify_config_changes(&active, &desired).restart_required);
desired.web.limits.max_http_connections += 1;
let applied = overlay_hot_fields(&active, &desired);
assert_eq!(applied.web.carrier_method, method);
assert_eq!(
applied.web.limits.max_http_connections,
active.web.limits.max_http_connections
);
active = applied;
}
}
#[test]
fn carrier_method_reload_publishes_both_directions_and_keeps_last_good_value() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("config.toml");
let source = |method| format!("[web]\ncarrier_method = \"{method}\"\n");
std::fs::write(&path, source("post")).unwrap();
let initial = Arc::new(ProxyConfig::load(&path).unwrap());
let initial_hash = ProxyConfig::load_with_metadata(&path)
.unwrap()
.rendered_hash;
let (config_tx, _config_rx) = watch::channel(Arc::clone(&initial));
let (log_tx, _log_rx) = watch::channel(initial.general.log_level.clone());
let mut reload_state = ReloadState::new(Some(initial_hash));
for (token, method) in [
("put", WebCarrierMethod::Put),
("post", WebCarrierMethod::Post),
] {
std::fs::write(&path, source(token)).unwrap();
reload_config(&path, &config_tx, &log_tx, None, None, &mut reload_state).unwrap();
let applied = config_tx.borrow().clone();
assert_eq!(applied.web.carrier_method, method);
std::fs::write(&path, source("PATCH")).unwrap();
reload_config(&path, &config_tx, &log_tx, None, None, &mut reload_state);
let unchanged = config_tx.borrow().clone();
assert!(Arc::ptr_eq(&unchanged, &applied));
}
}
+1
View File
@@ -263,6 +263,7 @@ const LISTENER_CONFIG_KEYS: &[&str] = &[
const WEB_CONFIG_KEYS: &[&str] = &[
"enabled",
"carrier",
"carrier_method",
"carriers",
"carrier_learning",
"carrier_negotiation_aggressiveness",
@@ -1,5 +1,71 @@
use super::*;
#[test]
fn conntrack_control_is_opt_in_by_default() {
assert!(!default_conntrack_control_enabled());
assert!(
!ProxyConfig::default()
.server
.conntrack_control
.inline_conntrack_control
);
let cfg = load_config_from_temp_toml(
r#"
[censorship]
tls_domain = "example.com"
[access.users]
user = "00000000000000000000000000000000"
"#,
);
assert!(!cfg.server.conntrack_control.inline_conntrack_control);
assert!(
!cfg.server
.conntrack_control
.inline_conntrack_control_explicit
);
assert_eq!(
serde_json::to_value(&cfg.server.conntrack_control).unwrap()["inline_conntrack_control"],
false,
);
}
#[test]
fn conntrack_control_preserves_explicit_enablement() {
for enabled in [false, true] {
let cfg = load_config_from_temp_toml(&format!(
r#"
[server.conntrack_control]
inline_conntrack_control = {enabled}
backend = "nftables"
[censorship]
tls_domain = "example.com"
[access.users]
user = "00000000000000000000000000000000"
"#,
));
assert_eq!(
cfg.server.conntrack_control.inline_conntrack_control,
enabled
);
assert!(
cfg.server
.conntrack_control
.inline_conntrack_control_explicit
);
assert_eq!(
cfg.server.conntrack_control.backend,
ConntrackBackend::Nftables
);
assert_eq!(
serde_json::to_value(&cfg.server.conntrack_control).unwrap()["inline_conntrack_control"],
enabled,
);
}
}
#[test]
fn conntrack_pressure_high_watermark_out_of_range_is_rejected() {
let toml = r#"
@@ -3,6 +3,10 @@ use super::*;
#[path = "web_tests/base_path_tests.rs"]
mod base_path_tests;
// Method selection must not alter capability identity or accepted config literals.
#[path = "web_tests/carrier_method_tests.rs"]
mod carrier_method_tests;
const WEB_CONFIG: &str = r#"
[access.users]
alice = "000102030405060708090a0b0c0d0e0f"
@@ -0,0 +1,48 @@
use super::*;
#[test]
fn carrier_method_defaults_and_roundtrips_without_changing_capabilities() {
let default = load_config_from_temp_toml(WEB_CONFIG);
assert_eq!(default.web.carrier_method, WebCarrierMethod::Post);
let capabilities = default.web.runtime.as_ref().unwrap().capabilities.clone();
for (token, method) in [
("post", WebCarrierMethod::Post),
("put", WebCarrierMethod::Put),
] {
let configured = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
&format!("carrier = \"https-lanes\"\ncarrier_method = \"{token}\""),
);
let source = format!("[general]\nconfig_strict = true\n{configured}");
let config = load_config_from_temp_toml(&source);
assert_eq!(config.web.carrier_method, method);
assert_eq!(
config.web.runtime.as_ref().unwrap().capabilities,
capabilities
);
let json = serde_json::to_value(&config.web).unwrap();
assert_eq!(json["carrier_method"], token);
let decoded: WebConfig = serde_json::from_value(json).unwrap();
assert_eq!(decoded.carrier_method, method);
let serialized = toml::to_string(&config.web).unwrap();
let decoded: WebConfig = toml::from_str(&serialized).unwrap();
assert_eq!(decoded.carrier_method, method);
}
}
#[test]
fn carrier_method_rejects_unknown_tokens_types_and_aliases() {
for value in ["\"POST\"", "\"PUT\"", "\"patch\"", "true", "42", "[]"] {
let source = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
&format!("carrier = \"https-lanes\"\ncarrier_method = {value}"),
);
assert!(load_config_error_from_temp_toml(&source).contains("carrier_method"));
}
let configured = WEB_CONFIG.replace(
"carrier = \"https-lanes\"",
"carrier = \"https-lanes\"\nhttp_method = \"put\"",
);
let source = format!("[general]\nconfig_strict = true\n{configured}");
assert!(load_config_error_from_temp_toml(&source).contains("http_method"));
}
+1 -1
View File
@@ -63,7 +63,7 @@ pub(crate) use web::{
};
pub(crate) use web_carrier::WEB_CARRIER_LEARNING_MIN_ENTRIES;
#[allow(unused_imports)]
pub use web_carrier::{WebCarrier, WebCarriers};
pub use web_carrier::{WebCarrier, WebCarrierMethod, WebCarriers};
pub(crate) use web_debug::web_debug_fits_limits;
pub use web_debug::{WebDebugBodyCapture, WebDebugConfig};
+5 -1
View File
@@ -6,7 +6,7 @@ use std::sync::Arc;
use bytes::Bytes;
use serde::{Deserialize, Serialize};
use super::web_carrier::{WebCarrier, WebCarriers};
use super::web_carrier::{WebCarrier, WebCarrierMethod, WebCarriers};
use super::web_debug::WebDebugConfig;
// Serialized WEB defaults remain separate from the runtime data model.
@@ -413,6 +413,9 @@ pub struct WebConfig {
/// Sole carrier when negotiation is disabled and final fallback when enabled.
#[serde(default)]
pub carrier: WebCarrier,
/// Page-owned HTTP method for HTTPS uplink and downlink, including retries.
#[serde(default)]
pub carrier_method: WebCarrierMethod,
/// Ordered carriers considered by server-side negotiation before the fallback carrier.
#[serde(default)]
pub carriers: WebCarriers,
@@ -471,6 +474,7 @@ impl Default for WebConfig {
Self {
enabled: false,
carrier: WebCarrier::default(),
carrier_method: WebCarrierMethod::default(),
carriers: WebCarriers::default(),
carrier_learning: default_web_carrier_learning(),
carrier_negotiation_aggressiveness: WebCarrierNegotiationAggressiveness::default(),
+21
View File
@@ -3,6 +3,27 @@ use serde::{Deserialize, Serialize};
/// Minimum restart-owned entries required for one complete learning sample.
pub(crate) const WEB_CARRIER_LEARNING_MIN_ENTRIES: usize = 3;
/// HTTP method emitted by newly rendered HTTPS carrier bridges.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum WebCarrierMethod {
/// Preserve the original bridge request method.
#[default]
Post,
/// Allow idempotent-method connection reuse in Linux WebKitGTK.
Put,
}
impl WebCarrierMethod {
/// Returns the HTTP token embedded in the page-owned carrier policy.
pub(crate) const fn as_str(self) -> &'static str {
match self {
Self::Post => "POST",
Self::Put => "PUT",
}
}
}
/// Carrier selected for one newly issued WEB relay session.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
+56 -21
View File
@@ -26,15 +26,21 @@ const RETRY_DELAYS: [Duration; 6] = [
Duration::from_secs(30),
];
/// Reports whether a generation's desired firewall policy was confirmed.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) enum ReconcileOutcome {
/// The applied policy matches the accepted generation's desired policy.
Applied,
/// The attempt failed without confirming the desired policy.
Failed,
}
/// Associates a reconciliation result with its accepted runtime generation.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) struct ReconcileStatus {
/// Runtime generation whose desired policy was attempted.
pub(super) generation: u64,
/// Result of the latest completed attempt for this generation.
pub(super) outcome: ReconcileOutcome,
}
@@ -51,25 +57,14 @@ pub(crate) struct FirewallAuthority {
}
impl FirewallAuthority {
/// Starts the single process-owned firewall reconciler.
pub(crate) fn spawn(control_plane: &ProcessControlPlane) -> Result<Self, String> {
let (desired_tx, desired_rx) = watch::channel(None);
let (status_tx, status_rx) = watch::channel(None);
let terminal = CancellationToken::new();
let closed = Arc::new(AtomicBool::new(false));
let completed_flag = Arc::new(AtomicBool::new(false));
let cleanup_succeeded = Arc::new(AtomicBool::new(false));
let completed = Arc::new(Notify::new());
let actor = FirewallReconciler::new(
SystemCommandRunner,
desired_rx,
status_tx,
terminal.clone(),
Arc::clone(&closed),
Arc::clone(&completed_flag),
Arc::clone(&cleanup_succeeded),
Arc::clone(&completed),
);
/// Starts reconciliation only when enabled and CAP_NET_ADMIN is available.
pub(crate) fn spawn(
control_plane: &ProcessControlPlane,
config: &ProxyConfig,
) -> Result<Option<Self>, String> {
let Some((authority, actor)) = Self::prepare(config, SystemCommandRunner) else {
return Ok(None);
};
control_plane
.spawn_cooperative(move |process_cancellation| async move {
actor.run(process_cancellation).await;
@@ -78,7 +73,37 @@ impl FirewallAuthority {
"process control-plane admission closed before conntrack firewall startup"
.to_string()
})?;
Ok(Self {
Ok(Some(authority))
}
// Admission precedes allocation so an opted-out process never owns firewall cleanup.
// Keeping construction separate permits fake runners without changing async Send bounds.
fn prepare<R>(config: &ProxyConfig, runner: R) -> Option<(Self, FirewallReconciler<R>)>
where
R: FirewallCommandRunner + 'static,
{
if !config.server.conntrack_control.inline_conntrack_control || !runner.has_cap_net_admin()
{
return None;
}
let (desired_tx, desired_rx) = watch::channel(None);
let (status_tx, status_rx) = watch::channel(None);
let terminal = CancellationToken::new();
let closed = Arc::new(AtomicBool::new(false));
let completed_flag = Arc::new(AtomicBool::new(false));
let cleanup_succeeded = Arc::new(AtomicBool::new(false));
let completed = Arc::new(Notify::new());
let actor = FirewallReconciler::new(
runner,
desired_rx,
status_tx,
terminal.clone(),
Arc::clone(&closed),
Arc::clone(&completed_flag),
Arc::clone(&cleanup_succeeded),
Arc::clone(&completed),
);
let authority = Self {
desired_tx,
status_rx,
terminal,
@@ -86,7 +111,8 @@ impl FirewallAuthority {
completed_flag,
cleanup_succeeded,
completed,
})
};
Some((authority, actor))
}
/// Publishes policy only after its runtime generation becomes active.
@@ -169,6 +195,7 @@ impl Drop for CompletionGuard {
}
}
/// Serializes confirmed-state transitions and retains cleanup ownership until shutdown.
pub(super) struct FirewallReconciler<R> {
runner: R,
desired_rx: watch::Receiver<Option<DesiredState>>,
@@ -186,6 +213,7 @@ impl<R> FirewallReconciler<R>
where
R: FirewallCommandRunner + 'static,
{
/// Constructs an admitted actor with unknown state pending startup recovery.
pub(super) fn new(
runner: R,
desired_rx: watch::Receiver<Option<DesiredState>>,
@@ -214,6 +242,7 @@ where
}
}
/// Reconciles accepted generations and completes bounded cleanup on cancellation.
pub(super) async fn run(mut self, process_cancellation: CancellationToken) {
let mut current = None;
let mut retry_index = 0usize;
@@ -337,6 +366,7 @@ where
let _completion = &self.completion;
}
/// Accepts fenced publications without replacing telemetry with stale desired state.
pub(super) fn take_latest_desired(&mut self) -> Option<DesiredState> {
let next = self.desired_rx.borrow_and_update().clone()?;
if next.generation < self.last_generation {
@@ -369,3 +399,8 @@ where
Some(next)
}
}
// Exercises startup admission and cleanup ownership with fake helper processes.
#[cfg(test)]
#[path = "tests/startup_admission.rs"]
mod startup_admission_tests;
+104 -5
View File
@@ -10,14 +10,19 @@ use crate::util::trusted_command::{resolve_trusted_helper, trusted_helper_comman
const COMMAND_TIMEOUT: Duration = Duration::from_secs(30);
/// A privileged helper invocation with arguments kept separate from shell syntax.
#[derive(Clone, Debug, Eq, PartialEq)]
pub(super) struct CommandSpec {
/// Logical helper name resolved by the trusted executable policy.
pub(super) binary: &'static str,
/// Arguments passed directly to the helper process.
pub(super) args: Vec<String>,
/// Optional restore script supplied on standard input.
pub(super) stdin: Option<String>,
}
impl CommandSpec {
/// Creates a helper invocation without an input script.
pub(super) fn new(binary: &'static str, args: impl IntoIterator<Item = &'static str>) -> Self {
Self {
binary,
@@ -26,6 +31,7 @@ impl CommandSpec {
}
}
/// Creates a helper invocation that receives a restore script.
pub(super) fn with_stdin(
binary: &'static str,
args: impl IntoIterator<Item = &'static str>,
@@ -39,22 +45,32 @@ impl CommandSpec {
}
}
/// Distinguishes idempotent absence from transaction and execution failures.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) enum CommandErrorKind {
/// The trusted helper executable is unavailable.
Missing,
/// The requested firewall object or rule is absent.
NotFound,
/// A terminal or process cancellation interrupted the invocation.
Cancelled,
/// The helper exceeded its execution deadline.
Timeout,
/// A failure that must not be treated as successful cleanup.
Failed,
}
/// A classified helper failure retaining its diagnostic for reconciliation.
#[derive(Clone, Debug, Eq, PartialEq)]
pub(super) struct CommandError {
/// Recovery semantics associated with the failure.
pub(super) kind: CommandErrorKind,
/// Original helper diagnostic or an execution failure description.
pub(super) message: String,
}
impl CommandError {
/// Creates the cancellation failure used by interruptible transactions.
pub(super) fn cancelled() -> Self {
Self {
kind: CommandErrorKind::Cancelled,
@@ -62,6 +78,7 @@ impl CommandError {
}
}
/// Creates a failure that prevents idempotent cleanup from claiming success.
pub(super) fn failed(message: impl Into<String>) -> Self {
Self {
kind: CommandErrorKind::Failed,
@@ -76,14 +93,19 @@ impl std::fmt::Display for CommandError {
}
}
/// Executes firewall commands through a production or deterministic test runner.
pub(super) trait FirewallCommandRunner: Send + Sync {
/// Reports whether a helper can be resolved under the runner's trust policy.
fn available(&self, binary: &str) -> bool;
/// Reports whether the process has the capability required to alter firewall rules.
fn has_cap_net_admin(&self) -> bool;
/// Executes one invocation while preserving classified failure semantics.
async fn run(&self, spec: CommandSpec) -> Result<(), CommandError>;
}
/// Runs trusted system helpers with bounded execution and captured diagnostics.
#[derive(Clone, Copy, Default)]
pub(super) struct SystemCommandRunner;
@@ -187,18 +209,95 @@ impl FirewallCommandRunner for SystemCommandRunner {
} else {
stderr
};
let kind = if is_not_found_error(&message) {
CommandErrorKind::NotFound
} else {
CommandErrorKind::Failed
};
let kind = classify_command_error(binary, &spec.args, &message);
Err(CommandError { kind, message })
}
}
}
/// Recognizes the existing legacy iptables and native nftables absence formats.
pub(super) fn is_not_found_error(message: &str) -> bool {
message.contains("No chain/target/match by that name")
|| message.contains("Bad rule (does a matching rule exist in that chain?)")
|| message.contains("Could not process rule: No such file or directory")
}
/// Bounds additional iptables-nft absence diagnostics to owned cleanup and checks.
pub(super) fn classify_command_error(
binary: &str,
args: &[String],
message: &str,
) -> CommandErrorKind {
if is_not_found_error(message) || is_missing_owned_iptables_chain(binary, args, message) {
CommandErrorKind::NotFound
} else {
CommandErrorKind::Failed
}
}
fn is_missing_owned_iptables_chain(binary: &str, args: &[String], message: &str) -> bool {
if !matches!(binary, "iptables" | "ip6tables") {
return false;
}
// An absent jump target is harmless for deletion, but not for installation.
let chain = match args {
[flag, table, operation, source, jump, target]
if flag == "-t"
&& table == "raw"
&& matches!(operation.as_str(), "-C" | "-D")
&& source == "PREROUTING"
&& jump == "-j"
&& target == "TELEMT_NOTRACK" =>
{
target.as_str()
}
[flag, table, operation, target]
if flag == "-t"
&& table == "raw"
&& matches!(operation.as_str(), "-F" | "-X")
&& matches!(
target.as_str(),
"TELEMT_NOTRACK" | "TELEMT_NT_A" | "TELEMT_NT_B"
) =>
{
target.as_str()
}
_ => return false,
};
let mut lines = message.lines();
let Some(first) = lines.next() else {
return false;
};
let diagnostic = if first.starts_with("Chain '") {
first
} else {
let Some((version, diagnostic)) = first
.strip_prefix(binary)
.and_then(|line| line.strip_prefix(" v"))
.and_then(|line| line.split_once(" (nf_tables): "))
else {
return false;
};
if version.is_empty() || version.bytes().any(|byte| byte.is_ascii_whitespace()) {
return false;
}
diagnostic
};
if diagnostic
.strip_prefix("Chain '")
.and_then(|line| line.strip_suffix("' does not exist"))
!= Some(chain)
{
return false;
}
// Reject mixed diagnostics instead of hiding another failure after an absence message.
let Some(help) = lines.next() else {
return true;
};
help.strip_prefix("Try `")
.and_then(|line| line.strip_prefix(binary))
.and_then(|line| line.strip_prefix(" -h' or '"))
.and_then(|line| line.strip_prefix(binary))
== Some(" --help' for more information.")
&& lines.next().is_none()
}
+4
View File
@@ -20,6 +20,10 @@ use super::transaction::{InterruptibleRunner, reconcile_once};
#[path = "tests/model_tests.rs"]
mod model_tests;
// Replays helper diagnostics through command classification and recovery.
#[path = "tests/recovery_errors.rs"]
mod recovery_errors;
#[derive(Clone)]
struct FailureRule {
binary: &'static str,
@@ -0,0 +1,512 @@
use super::super::command::classify_command_error;
use super::super::transaction::recover_to_empty;
use super::*;
const OWNED_CHAINS: [&str; 3] = ["TELEMT_NOTRACK", "TELEMT_NT_A", "TELEMT_NT_B"];
type ChainKey = (&'static str, String, String);
type ChainRules = BTreeMap<ChainKey, BTreeSet<String>>;
fn missing_chain(binary: &str, chain: &str) -> String {
format!(
"{binary} v1.8.10 (nf_tables): Chain '{chain}' does not exist\n\
Try `{binary} -h' or '{binary} --help' for more information."
)
}
fn classified_error(spec: &CommandSpec, message: &str) -> CommandError {
CommandError {
kind: classify_command_error(spec.binary, &spec.args, message),
message: message.to_string(),
}
}
#[derive(Default)]
struct FixtureState {
calls: Vec<CommandSpec>,
chains: ChainRules,
nft_tables: BTreeSet<String>,
failure: Option<&'static str>,
}
#[derive(Clone)]
struct FixtureRunner {
state: Arc<Mutex<FixtureState>>,
}
impl FixtureRunner {
fn new(failure: Option<&'static str>) -> Self {
let mut state = FixtureState {
failure,
..FixtureState::default()
};
for binary in ["iptables", "ip6tables"] {
for (table, chain, rules) in [
("raw", "PREROUTING", vec!["-j FOREIGN_RAW"]),
("raw", "FOREIGN_RAW", vec!["-j ACCEPT"]),
("filter", "INPUT", vec!["-j MTPR_SYNFIX", "-j TMT_SYN_TEST"]),
("filter", "MTPR_SYNFIX", vec!["-p tcp --syn -j DROP"]),
("filter", "TMT_SYN_TEST", vec!["-p tcp --syn -j DROP"]),
] {
state.chains.insert(
(binary, table.to_string(), chain.to_string()),
rules.into_iter().map(str::to_string).collect(),
);
}
}
state.nft_tables.extend([
"mtpr_synfix".to_string(),
"telemt_synlimit_test".to_string(),
"foreign_table".to_string(),
]);
Self {
state: Arc::new(Mutex::new(state)),
}
}
fn calls(&self) -> Vec<CommandSpec> {
self.state.lock().unwrap().calls.clone()
}
fn foreign_snapshot(&self) -> (ChainRules, BTreeSet<String>) {
let state = self.state.lock().unwrap();
let mut chains = state.chains.clone();
chains.retain(|(_, table, chain), _| {
table != "raw" || !OWNED_CHAINS.contains(&chain.as_str())
});
for ((_, table, chain), rules) in &mut chains {
if table == "raw" && chain == "PREROUTING" {
rules.remove("-j TELEMT_NOTRACK");
}
}
(chains, state.nft_tables.clone())
}
fn assert_owned_policy(&self, enabled: bool) {
let state = self.state.lock().unwrap();
for (binary, ip) in [("iptables", "192.0.2.10"), ("ip6tables", "2001:db8::10")] {
for chain in OWNED_CHAINS {
let key = (binary, "raw".to_string(), chain.to_string());
if !enabled {
assert!(!state.chains.contains_key(&key));
continue;
}
let expected = match chain {
"TELEMT_NOTRACK" => BTreeSet::from(["-j TELEMT_NT_A".to_string()]),
"TELEMT_NT_A" => {
BTreeSet::from([format!("-p tcp --dport 443 -d {ip} -j CT --notrack")])
}
"TELEMT_NT_B" => BTreeSet::new(),
_ => unreachable!(),
};
assert_eq!(state.chains.get(&key), Some(&expected));
}
let prerouting = (binary, "raw".to_string(), "PREROUTING".to_string());
assert_eq!(
state.chains[&prerouting].contains("-j TELEMT_NOTRACK"),
enabled
);
}
}
}
impl FirewallCommandRunner for FixtureRunner {
fn available(&self, _binary: &str) -> bool {
true
}
fn has_cap_net_admin(&self) -> bool {
true
}
async fn run(&self, spec: CommandSpec) -> Result<(), CommandError> {
let mut state = self.state.lock().unwrap();
state.calls.push(spec.clone());
if spec.binary == "nft" {
assert_eq!(&spec.args[..3], ["delete", "table", "inet"]);
return if state.nft_tables.remove(&spec.args[3]) {
Ok(())
} else {
Err(classified_error(
&spec,
"Error: Could not process rule: No such file or directory",
))
};
}
if let Some(script) = &spec.stdin {
let binary = match spec.binary {
"iptables-restore" => "iptables",
"ip6tables-restore" => "ip6tables",
_ => panic!("unexpected restore helper"),
};
assert_eq!(spec.args, ["--noflush"]);
for line in script.lines().filter(|line| line.starts_with('-')) {
let fields = line.split_whitespace().collect::<Vec<_>>();
let key = (binary, "raw".to_string(), fields[1].to_string());
let rules = state.chains.get_mut(&key).expect("restore target exists");
match fields[0] {
"-F" => rules.clear(),
"-A" => {
rules.insert(fields[2..].join(" "));
}
_ => panic!("unexpected restore operation"),
}
}
return Ok(());
}
assert!(matches!(spec.binary, "iptables" | "ip6tables"));
assert_eq!(spec.args[0], "-t");
let operation = spec.args[2].as_str();
let key = (spec.binary, spec.args[1].clone(), spec.args[3].clone());
match operation {
"-N" => {
if state.chains.contains_key(&key) {
Err(classified_error(&spec, "iptables: Chain already exists."))
} else {
state.chains.insert(key, BTreeSet::new());
Ok(())
}
}
"-C" | "-D" => {
if operation == "-D"
&& let Some(message) = state.failure.take()
{
return Err(classified_error(&spec, message));
}
let target = (spec.binary, spec.args[1].clone(), spec.args[5].clone());
if !state.chains.contains_key(&target) {
return Err(classified_error(
&spec,
&missing_chain(spec.binary, &spec.args[5]),
));
}
let rules = state.chains.get_mut(&key).expect("builtin chain exists");
let rule = spec.args[4..].join(" ");
if !rules.contains(&rule) {
return Err(classified_error(
&spec,
"Bad rule (does a matching rule exist in that chain?).",
));
}
if operation == "-D" {
rules.remove(&rule);
}
Ok(())
}
"-I" => {
let target = (spec.binary, spec.args[1].clone(), spec.args[6].clone());
if !state.chains.contains_key(&target) {
return Err(classified_error(
&spec,
&missing_chain(spec.binary, &spec.args[6]),
));
}
state
.chains
.get_mut(&key)
.unwrap()
.insert(spec.args[5..].join(" "));
Ok(())
}
"-F" | "-X" => {
let Some(rules) = state.chains.get_mut(&key) else {
return Err(classified_error(
&spec,
"iptables: No chain/target/match by that name.",
));
};
if operation == "-F" {
rules.clear();
} else {
assert!(rules.is_empty());
state.chains.remove(&key);
}
Ok(())
}
_ => panic!("unexpected iptables operation"),
}
}
}
#[test]
fn quoted_missing_chains_are_scoped_to_owned_commands() {
for binary in ["iptables", "ip6tables"] {
for operation in ["-C", "-D"] {
let spec = CommandSpec::new(
binary,
["-t", "raw", operation, "PREROUTING", "-j", "TELEMT_NOTRACK"],
);
let message = missing_chain(binary, "TELEMT_NOTRACK");
let error = classified_error(&spec, &message);
assert_eq!(error.kind, CommandErrorKind::NotFound, "{spec:?}");
assert_eq!(error.message, message);
}
for operation in ["-F", "-X"] {
for chain in OWNED_CHAINS {
let spec = CommandSpec::new(binary, ["-t", "raw", operation, chain]);
assert_eq!(
classified_error(&spec, &missing_chain(binary, chain)).kind,
CommandErrorKind::NotFound,
);
}
}
let spec = CommandSpec::new(binary, ["-t", "raw", "-F", "TELEMT_NOTRACK"]);
for message in [
"Chain 'TELEMT_NOTRACK' does not exist".to_string(),
format!("{binary} v1.8.11 (nf_tables): Chain 'TELEMT_NOTRACK' does not exist"),
] {
assert_eq!(
classified_error(&spec, &message).kind,
CommandErrorKind::NotFound
);
}
}
}
#[test]
fn unrelated_commands_and_diagnostics_remain_failures() {
let message = missing_chain("iptables", "TELEMT_NOTRACK");
for spec in [
CommandSpec::new("iptables", ["-t", "raw", "-N", "TELEMT_NOTRACK"]),
CommandSpec::new(
"iptables",
["-t", "raw", "-A", "PREROUTING", "-j", "TELEMT_NOTRACK"],
),
CommandSpec::new(
"iptables",
["-t", "raw", "-I", "PREROUTING", "1", "-j", "TELEMT_NOTRACK"],
),
CommandSpec::new("iptables", ["-t", "filter", "-F", "TELEMT_NOTRACK"]),
CommandSpec::new("iptables", ["-t", "raw", "-F", "TELEMT_NT_A"]),
CommandSpec::new(
"iptables",
["-t", "raw", "-D", "PREROUTING", "-j", "TELEMT_NT_A"],
),
CommandSpec::new(
"iptables",
["-t", "raw", "-D", "INPUT", "-j", "TELEMT_NOTRACK"],
),
CommandSpec::new(
"iptables",
["-t", "raw", "-D", "PREROUTING", "-g", "TELEMT_NOTRACK"],
),
CommandSpec::new("iptables", ["-F", "TELEMT_NOTRACK"]),
CommandSpec::new(
"iptables",
["-t", "raw", "-F", "TELEMT_NOTRACK", "--invalid"],
),
CommandSpec::new("iptables-restore", ["--noflush"]),
CommandSpec::new("ip6tables-restore", ["--noflush"]),
CommandSpec::new("nft", ["delete", "table", "inet", "telemt_conntrack"]),
CommandSpec::new("conntrack", ["-D"]),
] {
assert_eq!(
classified_error(&spec, &message).kind,
CommandErrorKind::Failed,
"{spec:?}",
);
}
for chain in [
"TELEMT_NT_C",
"TELEMT_NOTRACK_EXTRA",
"MTPR_SYNFIX",
"PREROUTING",
] {
let spec = CommandSpec::new("iptables", ["-t", "raw", "-F", chain]);
assert_eq!(
classified_error(&spec, &missing_chain("iptables", chain)).kind,
CommandErrorKind::Failed,
);
}
let spec = CommandSpec::new(
"iptables",
["-t", "raw", "-D", "PREROUTING", "-j", "TELEMT_NOTRACK"],
);
for message in [
"Permission denied",
"can't initialize iptables table `raw': Table does not exist",
"Another app is currently holding the xtables lock",
"Chain is not empty",
"Directory not empty",
"Can't delete chain with references left",
"unknown option --invalid",
"command timed out",
] {
assert_eq!(
classified_error(&spec, message).kind,
CommandErrorKind::Failed
);
}
for message in [
format!("{message}\nPermission denied"),
format!("Permission denied\n{message}"),
"Chain 'TELEMT_NOTRACK' does not exist\nPermission denied".to_string(),
"Chain 'TELEMT_NOTRACK' does not exist\n\
Try `ip6tables -h' or 'ip6tables --help' for more information."
.to_string(),
"iptables v1.8.10 (legacy): Chain 'TELEMT_NOTRACK' does not exist".to_string(),
"iptables v (nf_tables): Chain 'TELEMT_NOTRACK' does not exist".to_string(),
missing_chain("ip6tables", "TELEMT_NOTRACK"),
missing_chain("iptables", "TELEMT_NT_A"),
] {
assert_eq!(
classified_error(&spec, &message).kind,
CommandErrorKind::Failed
);
}
for binary in ["nft", "conntrack", "iptables-restore", "ip6tables-restore"] {
let spec = CommandSpec::new(binary, ["-t", "raw", "-F", "TELEMT_NOTRACK"]);
assert_eq!(
classified_error(&spec, "Chain 'TELEMT_NOTRACK' does not exist").kind,
CommandErrorKind::Failed,
);
}
}
#[tokio::test]
async fn unknown_recovery_converges_without_touching_foreign_rules() {
for policy in [DesiredPolicy::Empty, dual_stack_policy(443)] {
let runner = FixtureRunner::new(None);
let foreign = runner.foreign_snapshot();
let mut applied = AppliedState::Unknown;
let requested = desired(1, policy);
reconcile_once(&runner, &runner, &mut applied, &requested)
.await
.unwrap();
assert!(matches!(
&applied,
AppliedState::Known(plan) if plan.matches_policy(&requested.policy)
));
runner.assert_owned_policy(matches!(requested.policy, DesiredPolicy::Rules { .. }));
assert_eq!(runner.foreign_snapshot(), foreign);
for binary in ["iptables", "ip6tables"] {
let calls = runner.calls();
assert_eq!(
calls
.iter()
.filter(|call| call.binary == binary && call.args[2] == "-D")
.count(),
1,
);
assert_eq!(
calls
.iter()
.filter(|call| {
call.binary == binary && matches!(call.args[2].as_str(), "-F" | "-X")
})
.count(),
6,
);
}
let calls = runner.calls().len();
reconcile_once(
&runner,
&runner,
&mut applied,
&desired(2, requested.policy.clone()),
)
.await
.unwrap();
assert_eq!(runner.calls().len(), calls);
recover_to_empty(&runner).await.unwrap();
recover_to_empty(&runner).await.unwrap();
runner.assert_owned_policy(false);
assert_eq!(runner.foreign_snapshot(), foreign);
}
}
#[tokio::test]
async fn genuine_recovery_failures_leave_state_unknown_and_do_not_install() {
for message in [
"Permission denied",
"can't initialize iptables table `raw': Table does not exist",
"Another app is currently holding the xtables lock",
"unknown firewall failure",
] {
let runner = FixtureRunner::new(Some(message));
let mut applied = AppliedState::Unknown;
let error = reconcile_once(
&runner,
&runner,
&mut applied,
&desired(1, dual_stack_policy(443)),
)
.await
.unwrap_err();
assert!(error.message.contains(message));
assert_eq!(applied, AppliedState::Unknown);
assert!(runner.calls().iter().all(|call| call.stdin.is_none()));
}
}
#[tokio::test(start_paused = true)]
async fn actor_converges_then_idles_and_preserves_foreign_rules_on_shutdown() {
for failure in [None, Some("Permission denied")] {
let runner = FixtureRunner::new(failure);
let observed = runner.clone();
let foreign = observed.foreign_snapshot();
let (desired_tx, desired_rx) = watch::channel(None);
let (status_tx, mut status_rx) = watch::channel(None);
let terminal = CancellationToken::new();
let closed = Arc::new(std::sync::atomic::AtomicBool::new(false));
let completed = Arc::new(std::sync::atomic::AtomicBool::new(false));
let cleanup = Arc::new(std::sync::atomic::AtomicBool::new(false));
let reconciler = FirewallReconciler::new(
runner,
desired_rx,
status_tx,
terminal.clone(),
closed.clone(),
completed.clone(),
cleanup.clone(),
Arc::new(Notify::new()),
);
let task = tokio::spawn(reconciler.run(CancellationToken::new()));
let requested = desired(1, dual_stack_policy(443));
let stats = requested.stats.clone();
desired_tx.send_replace(Some(requested));
status_rx.changed().await.unwrap();
if failure.is_some() {
assert_eq!(
status_rx.borrow().as_ref().unwrap().outcome,
ReconcileOutcome::Failed,
);
assert!(!stats.get_conntrack_rule_apply_ok());
assert_eq!(stats.get_conntrack_rule_reconcile_error_total(), 1);
let calls = observed.calls().len();
tokio::time::advance(Duration::from_millis(999)).await;
tokio::task::yield_now().await;
assert_eq!(observed.calls().len(), calls);
tokio::time::advance(Duration::from_millis(1)).await;
status_rx.changed().await.unwrap();
}
assert_eq!(
status_rx.borrow().as_ref().unwrap().outcome,
ReconcileOutcome::Applied,
);
assert_eq!(status_rx.borrow().as_ref().unwrap().generation, 1);
assert!(stats.get_conntrack_rule_apply_ok());
assert_eq!(stats.get_conntrack_rule_reconcile_success_total(), 1);
assert_eq!(
stats.get_conntrack_rule_reconcile_error_total(),
u64::from(failure.is_some()),
);
let calls = observed.calls().len();
tokio::time::advance(Duration::from_secs(90)).await;
tokio::task::yield_now().await;
assert_eq!(observed.calls().len(), calls);
assert!(!status_rx.has_changed().unwrap());
observed.assert_owned_policy(true);
assert_eq!(observed.foreign_snapshot(), foreign);
terminal.cancel();
tokio::time::timeout(Duration::from_secs(1), task)
.await
.unwrap()
.unwrap();
assert!(closed.load(Ordering::Acquire));
assert!(completed.load(Ordering::Acquire));
assert!(cleanup.load(Ordering::Acquire));
assert!(!stats.get_conntrack_rule_apply_ok());
observed.assert_owned_policy(false);
assert_eq!(observed.foreign_snapshot(), foreign);
}
}
@@ -0,0 +1,267 @@
use std::sync::Mutex;
use std::sync::atomic::AtomicUsize;
use crate::config::{ConntrackBackend, ConntrackMode};
use super::super::command::{CommandErrorKind, CommandSpec, classify_command_error};
use super::super::transaction::reconcile_once;
use super::*;
const NETLINK_ERROR: &str =
"src/mnl.c:68: Unable to initialize Netlink socket: Address family not supported by protocol";
const PERMISSION_ERROR: &str = "iptables v1.8.11 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)";
#[derive(Clone)]
struct StartupRunner {
has_cap_net_admin: bool,
cap_probes: Arc<AtomicUsize>,
calls: Arc<Mutex<Vec<CommandSpec>>>,
failure: Option<&'static str>,
}
impl StartupRunner {
fn new(has_cap_net_admin: bool, failure: Option<&'static str>) -> Self {
Self {
has_cap_net_admin,
cap_probes: Arc::new(AtomicUsize::new(0)),
calls: Arc::new(Mutex::new(Vec::new())),
failure,
}
}
fn calls(&self) -> Vec<CommandSpec> {
self.calls.lock().unwrap().clone()
}
}
impl FirewallCommandRunner for StartupRunner {
fn available(&self, _binary: &str) -> bool {
true
}
fn has_cap_net_admin(&self) -> bool {
self.cap_probes.fetch_add(1, Ordering::Relaxed);
self.has_cap_net_admin
}
async fn run(&self, spec: CommandSpec) -> Result<(), CommandError> {
self.calls.lock().unwrap().push(spec.clone());
if let Some(message) = self.failure {
return Err(CommandError {
kind: classify_command_error(spec.binary, &spec.args, message),
message: message.to_string(),
});
}
if (spec.binary == "nft" && spec.args.first().map(String::as_str) == Some("delete"))
|| (matches!(spec.binary, "iptables" | "ip6tables")
&& matches!(
spec.args.get(2).map(String::as_str),
Some("-C" | "-D" | "-F" | "-X")
))
{
return Err(CommandError {
kind: CommandErrorKind::NotFound,
message: "injected absent owned object".to_string(),
});
}
Ok(())
}
}
fn start_with_runner(
config: &ProxyConfig,
runner: StartupRunner,
scope: &ProcessControlPlane,
) -> Option<FirewallAuthority> {
let (authority, actor) = FirewallAuthority::prepare(config, runner)?;
assert!(
scope
.spawn_cooperative(move |cancellation| actor.run(cancellation))
.is_ok()
);
Some(authority)
}
#[tokio::test(start_paused = true)]
async fn disabled_startup_never_acquires_cleanup_ownership() {
for explicit in [false, true] {
for has_cap in [true, false] {
for backend in [
ConntrackBackend::Auto,
ConntrackBackend::Nftables,
ConntrackBackend::Iptables,
] {
for mode in [
ConntrackMode::Tracked,
ConntrackMode::Notrack,
ConntrackMode::Hybrid,
] {
for message in [NETLINK_ERROR, PERMISSION_ERROR] {
let mut config = ProxyConfig::default();
if explicit {
config.server.conntrack_control.inline_conntrack_control = false;
config
.server
.conntrack_control
.inline_conntrack_control_explicit = true;
}
config.server.conntrack_control.backend = backend;
config.server.conntrack_control.mode = mode;
let runner = StartupRunner::new(has_cap, Some(message));
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope);
assert!(
authority.is_none(),
"{explicit} {has_cap} {backend:?} {mode:?}"
);
assert_eq!(runner.cap_probes.load(Ordering::Relaxed), 0);
tokio::time::advance(Duration::from_secs(90)).await;
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(runner.calls().is_empty());
}
}
}
}
}
}
#[tokio::test(start_paused = true)]
async fn enabled_startup_without_capability_creates_no_firewall_work() {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
for mode in [ConntrackMode::Tracked, ConntrackMode::Notrack] {
config.server.conntrack_control.mode = mode;
let runner = StartupRunner::new(false, Some(PERMISSION_ERROR));
let scope = ProcessControlPlane::new();
assert!(start_with_runner(&config, runner.clone(), &scope).is_none());
assert_eq!(runner.cap_probes.load(Ordering::Relaxed), 1);
tokio::time::advance(Duration::from_secs(90)).await;
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(runner.calls().is_empty());
}
}
#[tokio::test(start_paused = true)]
async fn admitted_startup_preserves_recovery_idle_and_shutdown_cleanup() {
for mode in [ConntrackMode::Tracked, ConntrackMode::Notrack] {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
config.server.conntrack_control.mode = mode;
config.server.conntrack_control.backend = ConntrackBackend::Nftables;
let runner = StartupRunner::new(true, None);
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope).unwrap();
let stats = Arc::new(Stats::new());
assert!(
authority
.publish_initial(1, Arc::new(config), stats.clone())
.await
);
assert!(stats.get_conntrack_rule_apply_ok());
for binary in ["nft", "iptables", "ip6tables"] {
assert!(runner.calls().iter().any(|call| call.binary == binary));
}
let calls = runner.calls().len();
tokio::time::advance(Duration::from_secs(90)).await;
tokio::task::yield_now().await;
assert_eq!(runner.calls().len(), calls);
assert!(authority.shutdown_and_clear().await);
assert!(!stats.get_conntrack_rule_apply_ok());
assert!(runner.calls().len() > calls);
assert!(scope.shutdown(Duration::from_secs(1)).await);
}
}
#[tokio::test(start_paused = true)]
async fn admitted_process_cancellation_preserves_cleanup() {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
let runner = StartupRunner::new(true, None);
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope).unwrap();
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(authority.shutdown_and_clear().await);
assert!(!runner.calls().is_empty());
}
#[test]
fn netlink_and_permission_errors_are_not_classified_as_absence() {
for spec in [
CommandSpec::new("nft", ["delete", "table", "inet", "telemt_conntrack"]),
CommandSpec::new("nft", ["-f", "-"]),
CommandSpec::new(
"iptables",
["-t", "raw", "-D", "PREROUTING", "-j", "TELEMT_NOTRACK"],
),
CommandSpec::new("iptables", ["-t", "raw", "-F", "TELEMT_NT_A"]),
CommandSpec::new("ip6tables", ["-t", "raw", "-X", "TELEMT_NT_B"]),
CommandSpec::new("iptables-restore", ["--noflush"]),
CommandSpec::new("ip6tables-restore", ["--noflush"]),
] {
for message in [
NETLINK_ERROR,
PERMISSION_ERROR,
"iptables: Failed to initialize nft: Address family not supported by protocol",
] {
assert_eq!(
classify_command_error(spec.binary, &spec.args, message),
CommandErrorKind::Failed,
"{spec:?} {message}",
);
}
}
}
#[tokio::test(start_paused = true)]
async fn admitted_genuine_failures_remain_unknown_and_retry() {
for message in [NETLINK_ERROR, PERMISSION_ERROR] {
let mut config = ProxyConfig::default();
config.server.conntrack_control.inline_conntrack_control = true;
config.server.conntrack_control.mode = ConntrackMode::Notrack;
config.server.conntrack_control.backend = ConntrackBackend::Nftables;
let runner = StartupRunner::new(true, Some(message));
let mut applied = AppliedState::Unknown;
let requested = DesiredState {
generation: 1,
policy: DesiredPolicy::from_config(&config),
stats: Arc::new(Stats::new()),
};
let error = reconcile_once(&runner, &runner, &mut applied, &requested)
.await
.unwrap_err();
assert!(error.message.contains(message));
assert_eq!(applied, AppliedState::Unknown);
assert!(runner.calls().iter().all(|call| call.stdin.is_none()));
let scope = ProcessControlPlane::new();
let authority = start_with_runner(&config, runner.clone(), &scope).unwrap();
let stats = Arc::new(Stats::new());
assert!(
!authority
.publish_initial(1, Arc::new(config), stats.clone())
.await
);
assert!(!stats.get_conntrack_rule_apply_ok());
assert_eq!(stats.get_conntrack_rule_reconcile_error_total(), 1);
let mut status = authority.status_rx.clone();
assert_eq!(
status.borrow_and_update().as_ref().unwrap().outcome,
ReconcileOutcome::Failed,
);
let calls = runner.calls().len();
tokio::time::advance(Duration::from_millis(999)).await;
tokio::task::yield_now().await;
assert_eq!(runner.calls().len(), calls);
tokio::time::advance(Duration::from_millis(1)).await;
status.changed().await.unwrap();
assert_eq!(
status.borrow().as_ref().unwrap().outcome,
ReconcileOutcome::Failed,
);
assert_eq!(stats.get_conntrack_rule_reconcile_error_total(), 2);
assert!(!authority.shutdown_and_clear().await);
assert!(authority.completed_flag.load(Ordering::Acquire));
assert!(scope.shutdown(Duration::from_secs(1)).await);
assert!(runner.calls().iter().all(|call| call.stdin.is_none()));
}
}
+20 -4
View File
@@ -13,6 +13,11 @@ use super::generation::RuntimeTaskScope;
const STARTUP_FALLBACK_AFTER: Duration = Duration::from_secs(80);
const RUNTIME_FALLBACK_AFTER: Duration = Duration::from_secs(6);
// Admission regressions cover notification loss without network startup.
#[cfg(test)]
mod tests;
/// Keeps generation admission and routing synchronized with periodic ME readiness.
pub(crate) async fn configure_admission_gate(
config: &Arc<ProxyConfig>,
me_pool: Option<Arc<MePool>>,
@@ -76,11 +81,18 @@ pub(crate) async fn configure_admission_gate(
} else {
Some(Instant::now())
};
let mut config_watch_open = true;
let mut me_ready_watch_open = true;
loop {
tokio::select! {
changed = config_rx_gate.changed() => {
changed = config_rx_gate.changed(), if config_watch_open => {
if changed.is_err() {
break;
config_watch_open = false;
warn!(
watch_channel = "config",
"Admission config watch closed; continuing readiness polling with last configuration"
);
continue;
}
let cfg = config_rx_gate.borrow_and_update().clone();
admission_poll_ms = cfg.general.me_admission_poll_ms.max(1);
@@ -88,9 +100,13 @@ pub(crate) async fn configure_admission_gate(
fast_fallback_enabled = cfg.general.me2dc_fallback && cfg.general.me2dc_fast;
continue;
}
changed = me_ready_rx_gate.changed() => {
changed = me_ready_rx_gate.changed(), if me_ready_watch_open => {
if changed.is_err() {
break;
me_ready_watch_open = false;
warn!(
watch_channel = "me_ready",
"Admission ME readiness watch closed; continuing periodic polling"
);
}
}
_ = tokio::time::sleep(Duration::from_millis(admission_poll_ms)) => {}
+210
View File
@@ -0,0 +1,210 @@
use std::sync::atomic::{AtomicUsize, Ordering};
use tracing::{Event, Subscriber};
use tracing_subscriber::Layer;
use tracing_subscriber::layer::{Context, SubscriberExt};
use super::*;
use crate::transport::middle_proxy::admission_test_support::IdlePoolFixture;
const POLL_MS: u64 = 20;
struct ClosureWarnings(Arc<AtomicUsize>);
impl<S: Subscriber> Layer<S> for ClosureWarnings {
fn on_event(&self, event: &Event<'_>, _ctx: Context<'_, S>) {
struct ChannelField(bool);
impl tracing::field::Visit for ChannelField {
fn record_debug(
&mut self,
field: &tracing::field::Field,
_value: &dyn std::fmt::Debug,
) {
self.0 |= field.name() == "watch_channel";
}
}
if event.metadata().level() == &tracing::Level::WARN {
let mut visitor = ChannelField(false);
event.record(&mut visitor);
if visitor.0 {
self.0.fetch_add(1, Ordering::Relaxed);
}
}
}
}
struct GateFixture {
pool: IdlePoolFixture,
route: Arc<RouteRuntimeController>,
admission: watch::Receiver<bool>,
config_tx: Option<watch::Sender<Arc<ProxyConfig>>>,
ready_tx: Option<watch::Sender<u64>>,
config: Arc<ProxyConfig>,
scope: RuntimeTaskScope,
}
impl GateFixture {
async fn new() -> Self {
let pool = IdlePoolFixture::new(false).await;
let mut cfg = ProxyConfig::default();
cfg.general.use_middle_proxy = true;
cfg.general.me2dc_fallback = true;
cfg.general.me2dc_fast = true;
cfg.general.me_admission_poll_ms = POLL_MS;
let config = Arc::new(cfg);
let (config_tx, config_rx) = watch::channel(config.clone());
let (ready_tx, ready_rx) = watch::channel(0);
let (admission_tx, mut admission) = watch::channel(false);
let route = Arc::new(RouteRuntimeController::new(RelayRouteMode::Direct));
let scope = RuntimeTaskScope::new();
configure_admission_gate(
&config,
Some(pool.pool()),
Arc::new(RwLock::new(Some(pool.pool()))),
route.clone(),
&admission_tx,
config_rx,
ready_rx,
scope.clone(),
)
.await;
drop(admission_tx);
admission.borrow_and_update();
tokio::task::yield_now().await;
Self {
pool,
route,
admission,
config_tx: Some(config_tx),
ready_tx: Some(ready_tx),
config,
scope,
}
}
async fn poll(&self) {
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_millis(POLL_MS)).await;
tokio::task::yield_now().await;
}
async fn expect_route(&self, mode: RelayRouteMode) {
let mut rx = self.route.subscribe();
tokio::time::timeout(Duration::from_secs(1), async {
loop {
if rx.borrow_and_update().mode == mode {
return;
}
rx.changed().await.unwrap();
}
})
.await
.expect("the generation must keep sampling readiness without client traffic");
}
}
async fn recovery_after_watch_closure(close_config: bool, close_ready: bool) {
let warnings = Arc::new(AtomicUsize::new(0));
let subscriber = tracing_subscriber::registry().with(ClosureWarnings(warnings.clone()));
let _default = tracing::subscriber::set_default(subscriber);
let mut fixture = GateFixture::new().await;
assert!(*fixture.admission.borrow());
if close_config {
fixture.config_tx.take();
}
if close_ready {
fixture.ready_tx.take();
}
for _ in 0..4 {
tokio::task::yield_now().await;
}
let expected_warnings = usize::from(close_config) + usize::from(close_ready);
fixture.pool.set_ready(true);
for _ in 0..4 {
tokio::task::yield_now().await;
}
assert_eq!(fixture.route.snapshot().mode, RelayRouteMode::Direct);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
fixture.pool.set_ready(false);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Direct).await;
fixture.pool.set_ready(true);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
for _ in 0..5 {
fixture.poll().await;
}
assert_eq!(warnings.load(Ordering::Relaxed), expected_warnings);
fixture.scope.stop().await;
assert!(fixture.admission.has_changed().is_err());
}
#[tokio::test(start_paused = true)]
async fn idle_gate_recovers_after_readiness_watch_closes() {
recovery_after_watch_closure(false, true).await;
}
#[tokio::test(start_paused = true)]
async fn idle_gate_recovers_after_config_watch_closes() {
recovery_after_watch_closure(true, false).await;
}
#[tokio::test(start_paused = true)]
async fn idle_gate_recovers_after_both_watches_close_without_spinning() {
recovery_after_watch_closure(true, true).await;
}
#[tokio::test(start_paused = true)]
async fn closed_config_watch_preserves_last_effective_fallback_policy() {
let mut fixture = GateFixture::new().await;
let mut cfg = fixture.config.as_ref().clone();
cfg.general.me2dc_fallback = false;
fixture
.config_tx
.as_ref()
.unwrap()
.send_replace(Arc::new(cfg));
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
assert!(!*fixture.admission.borrow_and_update());
fixture.config_tx.take();
fixture.ready_tx.take();
fixture.poll().await;
fixture.pool.set_ready(true);
fixture.poll().await;
tokio::time::timeout(Duration::from_secs(1), fixture.admission.changed())
.await
.expect("restored readiness must open admission after notification loss")
.unwrap();
assert!(*fixture.admission.borrow_and_update());
fixture.pool.set_ready(false);
fixture.poll().await;
tokio::time::timeout(Duration::from_secs(1), fixture.admission.changed())
.await
.unwrap()
.unwrap();
assert!(!*fixture.admission.borrow_and_update());
assert_eq!(fixture.route.snapshot().mode, RelayRouteMode::Middle);
fixture.scope.stop().await;
}
#[tokio::test(start_paused = true)]
async fn generation_cancellation_stops_gate_after_notification_loss() {
let mut fixture = GateFixture::new().await;
fixture.config_tx.take();
fixture.ready_tx.take();
fixture.poll().await;
fixture.pool.set_ready(true);
fixture.poll().await;
fixture.expect_route(RelayRouteMode::Middle).await;
fixture.scope.stop().await;
assert!(fixture.admission.has_changed().is_err());
let state = fixture.route.snapshot();
fixture.pool.set_ready(false);
for _ in 0..3 {
fixture.poll().await;
}
assert_eq!(fixture.route.snapshot(), state);
assert!(*fixture.admission.borrow());
}
+10 -6
View File
@@ -375,17 +375,21 @@ pub(super) async fn run_telemt_core(
#[cfg(target_os = "linux")]
let conntrack_firewall = {
let authority = crate::conntrack_control::FirewallAuthority::spawn(&process_control_plane)
.map_err(std::io::Error::other)?;
if !authority
.publish_initial(1, runtime.config.clone(), stats.clone())
.await
let authority = crate::conntrack_control::FirewallAuthority::spawn(
&process_control_plane,
&runtime.config,
)
.map_err(std::io::Error::other)?;
if let Some(authority) = &authority
&& !authority
.publish_initial(1, runtime.config.clone(), stats.clone())
.await
{
warn!(
"Initial conntrack firewall reconciliation failed; background retries remain active"
);
}
Some(authority)
authority
};
#[cfg(not(target_os = "linux"))]
let conntrack_firewall = None::<crate::conntrack_control::FirewallAuthority>;
+32 -7
View File
@@ -104,11 +104,12 @@ pub fn generate_service_file(init_system: InitSystem, opts: &ServiceOptions) ->
InitSystem::Systemd => generate_systemd_unit(opts),
InitSystem::OpenRC => generate_openrc_script(opts),
InitSystem::FreeBSDRc => generate_freebsd_rc_script(opts),
InitSystem::Unknown => generate_systemd_unit(opts), // Default to systemd format
// Preserve systemd generation when init detection is unavailable.
InitSystem::Unknown => generate_systemd_unit(opts),
}
}
/// Generates an enhanced systemd unit file.
/// Generates a hardened systemd unit with Netlink access for optional firewall helpers.
fn generate_systemd_unit(opts: &ServiceOptions) -> String {
let user_line = opts.user.map(|u| format!("User={}", u)).unwrap_or_default();
let group_line = opts
@@ -151,14 +152,14 @@ PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
MemoryDenyWriteExecute=true
LockPersonality=true
# Allow binding to privileged ports and writing to specific paths
# Allow privileged port binding, optional firewall helpers, and runtime state writes
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMIN
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMIN
ReadWritePaths=/etc/telemt /var/run /var/lib/telemt
@@ -233,9 +234,11 @@ fn generate_freebsd_rc_script(opts: &ServiceOptions) -> String {
# Add the following lines to /etc/rc.conf to enable telemt:
#
# telemt_enable="YES"
# telemt_config="/etc/telemt/config.toml" # optional
# telemt_user="telemt" # optional
# telemt_group="telemt" # optional
# The configuration path can be overridden in rc.conf.
# telemt_config="/etc/telemt/config.toml"
# The service user and group can be overridden in rc.conf.
# telemt_user="telemt"
# telemt_group="telemt"
#
. /etc/rc.subr
@@ -355,6 +358,28 @@ mod tests {
assert!(unit.contains("PIDFile="));
}
#[test]
fn systemd_unit_allows_netlink_without_removing_hardening() {
let unit = generate_systemd_unit(&ServiceOptions::default());
let families = unit
.lines()
.find_map(|line| line.strip_prefix("RestrictAddressFamilies="))
.unwrap();
assert_eq!(
families.split_whitespace().collect::<Vec<_>>(),
["AF_INET", "AF_INET6", "AF_UNIX", "AF_NETLINK"],
);
for directive in [
"NoNewPrivileges=true",
"ProtectSystem=strict",
"RestrictNamespaces=true",
"AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMIN",
"CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMIN",
] {
assert!(unit.lines().any(|line| line == directive), "{directive}");
}
}
#[test]
fn test_openrc_script_generation() {
let opts = ServiceOptions::default();
+2 -1
View File
@@ -8,6 +8,7 @@ use std::net::{IpAddr, SocketAddr};
use std::os::fd::{AsRawFd, RawFd};
#[cfg(target_os = "linux")]
use std::os::raw::c_int;
use std::sync::Arc;
use std::sync::atomic::Ordering;
use std::time::{Duration, Instant};
@@ -252,7 +253,7 @@ impl MePool {
/// Perform full ME RPC handshake on an established TCP stream.
/// Returns cipher keys/ivs and split halves; does not register writer.
pub(crate) async fn handshake_only(
&self,
self: &Arc<Self>,
stream: TcpStream,
addr: SocketAddr,
upstream_egress: Option<UpstreamEgressInfo>,
+24 -2
View File
@@ -1,5 +1,10 @@
use super::*;
// Outage regressions use loopback sockets and explicit family-state snapshots.
#[cfg(test)]
mod tests;
/// Reports whether replacing an endpoint could disrupt a bound client.
pub(super) fn has_bound_clients_on_endpoint(
writer_ids: &[u64],
bound_clients_by_writer: &HashMap<u64, usize>,
@@ -9,6 +14,7 @@ pub(super) fn has_bound_clients_on_endpoint(
.any(|writer_id| bound_clients_by_writer.get(writer_id).copied().unwrap_or(0) > 0)
}
/// Restores missing single-endpoint coverage under the bounded outage retry policy.
pub(super) async fn recover_single_endpoint_outage(
pool: &Arc<MePool>,
rng: &Arc<SecureRandom>,
@@ -53,13 +59,21 @@ pub(super) async fn recover_single_endpoint_outage(
pool.stats
.increment_me_single_endpoint_outage_reconnect_attempt_total();
let generation = pool.current_generation();
let bypass_quarantine = pool.single_endpoint_outage_disable_quarantine();
let attempt_ok = if bypass_quarantine {
pool.stats
.increment_me_single_endpoint_quarantine_bypass_total();
match tokio::time::timeout(
pool.reconnect_runtime.me_one_timeout,
pool.connect_one_for_dc(endpoint, key.0, rng.as_ref()),
pool.connect_one_with_generation_contour_for_dc_with_intent(
endpoint,
rng.as_ref(),
generation,
WriterContour::Active,
key.0,
WriterOpenIntent::Coverage,
),
)
.await
{
@@ -88,7 +102,14 @@ pub(super) async fn recover_single_endpoint_outage(
let one_endpoint = [endpoint];
match tokio::time::timeout(
pool.reconnect_runtime.me_one_timeout,
pool.connect_endpoints_round_robin(key.0, &one_endpoint, rng.as_ref()),
pool.connect_endpoints_round_robin_with_generation_contour(
key.0,
&one_endpoint,
rng.as_ref(),
generation,
WriterContour::Active,
WriterOpenIntent::Coverage,
),
)
.await
{
@@ -142,6 +163,7 @@ pub(super) async fn recover_single_endpoint_outage(
);
}
/// Rotates an idle single-endpoint shadow without removing the existing coverage first.
pub(super) async fn maybe_rotate_single_endpoint_shadow(
pool: &Arc<MePool>,
rng: &Arc<SecureRandom>,
@@ -0,0 +1,148 @@
use std::sync::atomic::Ordering;
use super::*;
use crate::transport::middle_proxy::admission_test_support::unregistered_writer;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool;
async fn single_endpoint_reaches_tcp_with_a_full_cap(bypass_quarantine: bool) {
let pool = make_pool().await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = listener.local_addr().unwrap();
let donor = SocketAddr::new(endpoint.ip(), 1);
pool.update_proxy_maps(
HashMap::from([
(2, vec![(endpoint.ip(), endpoint.port())]),
(3, vec![(donor.ip(), donor.port())]),
]),
None,
)
.await;
pool.floor_runtime
.me_adaptive_floor_cpu_cores_override
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_per_core
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_global
.store(1, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_outage_disable_quarantine
.store(bypass_quarantine, Ordering::Relaxed);
let (writer, _receiver) = unregistered_writer(
&pool,
1,
3,
donor,
pool.current_generation(),
WriterContour::Active,
);
pool.writers.write().await.push(writer);
if bypass_quarantine {
pool.endpoint_quarantine
.lock()
.await
.insert(endpoint, Instant::now() + Duration::from_secs(60));
}
let rng = Arc::new(SecureRandom::new());
let key = (2, IpFamily::V4);
let mut backoff = HashMap::new();
let mut next_attempt = HashMap::new();
let semaphore = Arc::new(Semaphore::new(1));
let required = pool.required_writers_for_dc(1);
// TCP acceptance proves capacity admission; closing the peer deliberately rejects the handshake.
tokio::time::timeout(Duration::from_secs(1), async {
tokio::join!(
recover_single_endpoint_outage(
&pool,
&rng,
key,
endpoint,
required,
&mut backoff,
&mut next_attempt,
&semaphore,
),
async {
let (stream, _) = listener.accept().await.unwrap();
drop(stream);
},
);
})
.await
.expect("missing-DC recovery must reach TCP even when another DC fills the cap");
assert_eq!(
pool.stats
.get_me_single_endpoint_outage_reconnect_attempt_total(),
1
);
assert_eq!(
pool.stats
.get_me_single_endpoint_outage_reconnect_success_total(),
0
);
assert_eq!(
pool.stats.get_me_single_endpoint_quarantine_bypass_total(),
u64::from(bypass_quarantine)
);
assert!(next_attempt.contains_key(&key));
assert!(backoff.contains_key(&key));
assert_eq!(semaphore.available_permits(), 1);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn single_endpoint_quarantine_bypass_recovers_coverage_at_cap() {
single_endpoint_reaches_tcp_with_a_full_cap(true).await;
}
#[tokio::test]
async fn single_endpoint_round_robin_recovers_coverage_at_cap() {
single_endpoint_reaches_tcp_with_a_full_cap(false).await;
}
#[tokio::test]
async fn restored_coverage_preserves_family_suppression_hysteresis() {
let pool = make_pool().await;
let now = MePool::now_epoch_secs();
let until = now + 3600;
pool.set_family_runtime_state(
IpFamily::V4,
MeFamilyRuntimeState::Suppressed,
now,
until,
5,
0,
);
super::super::update_family_runtime_state(&pool, IpFamily::V4, false);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Suppressed
);
assert_eq!(pool.family_suppressed_until_epoch_secs(IpFamily::V4), until);
assert_eq!(pool.family_recover_success_streak(IpFamily::V4), 0);
pool.set_family_runtime_state(
IpFamily::V4,
MeFamilyRuntimeState::Suppressed,
now,
now - 1,
5,
0,
);
super::super::update_family_runtime_state(&pool, IpFamily::V4, false);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Recovering
);
assert_eq!(pool.family_recover_success_streak(IpFamily::V4), 1);
super::super::update_family_runtime_state(&pool, IpFamily::V4, false);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Healthy
);
assert_eq!(pool.family_fail_streak(IpFamily::V4), 0);
}
+5
View File
@@ -49,6 +49,11 @@ mod send;
mod send_adversarial_tests;
mod wire;
/// Network-free writer fixtures shared by admission and recovery regressions.
#[cfg(test)]
#[path = "tests/admission_test_support.rs"]
pub(crate) mod admission_test_support;
use bytes::Bytes;
use tokio::sync::OwnedSemaphorePermit;
+1
View File
@@ -407,6 +407,7 @@ pub(super) struct BindingPolicyCore {
}
pub(super) struct NatRuntimeCore {
pub(super) discovery: super::pool_nat::NatDiscovery,
pub(super) nat_ip_cfg: Option<IpAddr>,
pub(super) nat_ip_detected: Arc<RwLock<Option<IpAddr>>>,
pub(super) nat_probe: bool,
@@ -279,6 +279,7 @@ impl MePool {
me_bind_stale_ttl_secs: AtomicU64::new(me_bind_stale_ttl_secs),
}),
nat_runtime: Arc::new(NatRuntimeCore {
discovery: super::super::pool_nat::NatDiscovery::default(),
nat_ip_cfg: nat_ip,
nat_ip_detected: Arc::new(RwLock::new(None)),
nat_probe,
@@ -61,13 +61,13 @@ impl Drop for WriterOpenReservation<'_> {
}
impl MePool {
/// Computes the authoritative active-writer floor across enabled families and DCs.
/// Computes reservation coverage across configured families, including suppressed ones.
pub(in crate::transport::middle_proxy) async fn active_coverage_required_total(&self) -> usize {
let now_epoch_secs = Self::now_epoch_secs();
let mut required_total = 0usize;
let endpoint_snapshot = self.endpoint_snapshot.load_full();
if self.family_enabled_for_drain_coverage(IpFamily::V4, now_epoch_secs) {
// Suppression must not remove the capacity needed to recover active coverage.
if self.decision.ipv4_me {
for addrs in endpoint_snapshot.map_v4.values() {
let mut endpoints = HashSet::<SocketAddr>::new();
for (ip, port) in addrs.iter().copied() {
@@ -79,7 +79,7 @@ impl MePool {
}
}
if self.family_enabled_for_drain_coverage(IpFamily::V6, now_epoch_secs) {
if self.decision.ipv6_me {
for addrs in endpoint_snapshot.map_v6.values() {
let mut endpoints = HashSet::<SocketAddr>::new();
for (ip, port) in addrs.iter().copied() {
+115 -103
View File
@@ -1,7 +1,9 @@
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::oneshot;
use tokio::task::JoinSet;
use tokio::time::timeout;
use tracing::{debug, info};
@@ -18,7 +20,18 @@ use std::time::Instant;
const STUN_BATCH_TIMEOUT: Duration = Duration::from_secs(5);
const STUN_BATCH_TCP_FALLBACK_TIMEOUT: Duration = Duration::from_secs(12);
const STUN_CACHE_TTL: Duration = Duration::from_secs(600);
const HTTP_FAILURE_TTL: Duration = Duration::from_secs(60);
// Discovery ownership and source-bound cache admission for this pool generation.
mod state;
pub(super) use state::NatDiscovery;
// Local network regressions exercise cancellation before discovery publication.
#[cfg(test)]
mod tests;
/// Probes the configured or default STUN endpoint with both address families.
#[allow(dead_code)]
pub async fn stun_probe(stun_addr: Option<String>) -> Result<crate::network::stun::DualStunResult> {
let stun_addr = stun_addr.unwrap_or_else(|| {
@@ -35,6 +48,7 @@ pub async fn stun_probe(stun_addr: Option<String>) -> Result<crate::network::stu
stun_probe_dual_with_tcp_fallback(&stun_addr, false).await
}
/// Detects a public IPv4 address using the default HTTP endpoints.
#[allow(dead_code)]
pub async fn detect_public_ip() -> Option<IpAddr> {
let urls = crate::config::defaults::default_http_ip_detect_urls();
@@ -151,6 +165,7 @@ impl MePool {
(live_servers, best_reflected)
}
/// Applies the configured or HTTP-detected NAT IP to non-public addresses.
pub(super) fn translate_ip_for_nat(&self, ip: IpAddr) -> IpAddr {
let nat_ip = self.nat_runtime.nat_ip_cfg.or_else(|| {
self.nat_runtime
@@ -177,6 +192,7 @@ impl MePool {
}
}
/// Selects the NAT source IP without replacing the actual TCP source port.
pub(super) fn translate_our_addr_with_reflection(
&self,
addr: std::net::SocketAddr,
@@ -203,7 +219,8 @@ impl MePool {
std::net::SocketAddr::new(ip, addr.port())
}
pub(super) async fn maybe_detect_nat_ip(&self, local_ip: IpAddr) -> Option<IpAddr> {
/// Waits for pool-owned HTTP discovery without owning its cancellation.
pub(super) async fn maybe_detect_nat_ip(self: &Arc<Self>, local_ip: IpAddr) -> Option<IpAddr> {
if self.nat_runtime.nat_ip_cfg.is_some() {
return self.nat_runtime.nat_ip_cfg;
}
@@ -220,103 +237,117 @@ impl MePool {
return Some(ip);
}
let Some(ip) = detect_public_ipv4_http(&self.nat_runtime.http_ip_detect_urls).await else {
return None;
};
{
let mut guard = self.nat_runtime.nat_ip_detected.write().await;
*guard = Some(IpAddr::V4(ip));
let mut flight = self.nat_runtime.discovery.http.clone().lock_owned().await;
if let Some(ip) = *self.nat_runtime.nat_ip_detected.read().await {
return Some(ip);
}
info!(public_ip = %ip, "Auto-detected public IP for NAT translation");
Some(IpAddr::V4(ip))
if flight.is_some_and(|until| Instant::now() < until) {
return None;
}
let pool = self.clone();
let (send, receive) = oneshot::channel();
// Transfer singleflight ownership before the producer can be cancelled or polled.
self.lifecycle
.spawn_producer(async move {
let detected = detect_public_ipv4_http(&pool.nat_runtime.http_ip_detect_urls)
.await
.map(IpAddr::V4);
if let Some(ip) = detected {
*pool.nat_runtime.nat_ip_detected.write().await = Some(ip);
*flight = None;
info!(public_ip = %ip, "Auto-detected public IP for NAT translation");
} else {
*flight = Some(Instant::now() + HTTP_FAILURE_TTL);
}
let _ = send.send(detected);
})
.ok()?;
receive.await.ok().flatten()
}
/// Waits for a pool-owned STUN refresh in the exact source-address context.
pub(super) async fn maybe_reflect_public_addr(
&self,
self: &Arc<Self>,
family: IpFamily,
bind_ip: Option<IpAddr>,
) -> Option<std::net::SocketAddr> {
const STUN_CACHE_TTL: Duration = Duration::from_secs(600);
let use_shared_cache = bind_ip.is_none();
if !use_shared_cache {
match (family, bind_ip) {
(IpFamily::V4, Some(IpAddr::V4(_)))
| (IpFamily::V6, Some(IpAddr::V6(_)))
| (_, None) => {}
_ => return None,
}
if let Some(bind_ip) = bind_ip {
return self.reflect_bound_addr(family, bind_ip).await;
}
// Backoff window
if use_shared_cache
&& let Some(until) = *self.nat_runtime.stun_backoff_until.read().await
&& Instant::now() < until
{
if let Ok(cache) = self.nat_runtime.nat_reflection_cache.try_lock() {
let slot = match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
};
return slot.map(|(_, addr)| addr);
}
return None;
if let Some(cached) = self.cached_shared_reflection(family).await {
return cached;
}
let singleflight = match family {
IpFamily::V4 => self.nat_runtime.nat_reflection_singleflight_v4.clone(),
IpFamily::V6 => self.nat_runtime.nat_reflection_singleflight_v6.clone(),
};
let flight = singleflight.lock_owned().await;
if let Some(cached) = self.cached_shared_reflection(family).await {
return cached;
}
if use_shared_cache && let Ok(mut cache) = self.nat_runtime.nat_reflection_cache.try_lock()
{
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
if let Some((ts, addr)) = slot
&& ts.elapsed() < STUN_CACHE_TTL
{
return Some(*addr);
}
}
let _singleflight_guard = if use_shared_cache {
Some(match family {
IpFamily::V4 => self.nat_runtime.nat_reflection_singleflight_v4.lock().await,
IpFamily::V6 => self.nat_runtime.nat_reflection_singleflight_v6.lock().await,
let pool = self.clone();
let (send, receive) = oneshot::channel();
self.lifecycle
.spawn_producer(async move {
let _flight = flight;
let attempt = pool
.nat_runtime
.nat_probe_attempts
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let reflected = pool.refresh_stun(family, None, attempt).await;
if let Some(addr) = reflected {
// Cache readers must never cause a completed refresh to be discarded.
let mut cache = pool.nat_runtime.nat_reflection_cache.lock().await;
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
*slot = Some((Instant::now(), addr));
pool.nat_runtime
.nat_probe_attempts
.store(0, std::sync::atomic::Ordering::Relaxed);
} else {
let backoff = Duration::from_secs(60 * 2u64.pow((attempt as u32).min(6)));
*pool.nat_runtime.stun_backoff_until.write().await =
Some(Instant::now() + backoff);
}
let _ = send.send(reflected);
})
} else {
None
.ok()?;
receive.await.ok().flatten()
}
async fn cached_shared_reflection(
&self,
family: IpFamily,
) -> Option<Option<std::net::SocketAddr>> {
let backoff = self
.nat_runtime
.stun_backoff_until
.read()
.await
.is_some_and(|until| Instant::now() < until);
let cache = self.nat_runtime.nat_reflection_cache.lock().await;
let slot = match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
};
if use_shared_cache
&& let Some(until) = *self.nat_runtime.stun_backoff_until.read().await
&& Instant::now() < until
{
if let Ok(cache) = self.nat_runtime.nat_reflection_cache.try_lock() {
let slot = match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
};
return slot.map(|(_, addr)| addr);
}
return None;
// Preserve the existing stale-result fallback during shared failure backoff.
if backoff {
return Some(slot.map(|(_, addr)| addr));
}
slot.filter(|(created, _)| created.elapsed() < STUN_CACHE_TTL)
.map(|(_, addr)| Some(addr))
}
if use_shared_cache && let Ok(mut cache) = self.nat_runtime.nat_reflection_cache.try_lock()
{
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
if let Some((ts, addr)) = slot
&& ts.elapsed() < STUN_CACHE_TTL
{
return Some(*addr);
}
}
let attempt = if use_shared_cache {
self.nat_runtime
.nat_probe_attempts
.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
} else {
0
};
async fn refresh_stun(
&self,
family: IpFamily,
bind_ip: Option<IpAddr>,
attempt: u8,
) -> Option<std::net::SocketAddr> {
let configured_servers = self.configured_stun_servers();
let live_snapshot = self.nat_runtime.nat_stun_live_servers.read().await.clone();
let primary_servers = if live_snapshot.is_empty() {
@@ -348,33 +379,14 @@ impl MePool {
}
if let Some(reflected_addr) = selected_reflected {
if use_shared_cache {
self.nat_runtime
.nat_probe_attempts
.store(0, std::sync::atomic::Ordering::Relaxed);
}
info!(
family = ?family,
live_servers = live_server_count,
"STUN-Quorum reached, IP: {}",
reflected_addr.ip()
);
if use_shared_cache
&& let Ok(mut cache) = self.nat_runtime.nat_reflection_cache.try_lock()
{
let slot = match family {
IpFamily::V4 => &mut cache.v4,
IpFamily::V6 => &mut cache.v6,
};
*slot = Some((Instant::now(), reflected_addr));
}
return Some(reflected_addr);
}
if use_shared_cache {
let backoff = Duration::from_secs(60 * 2u64.pow((attempt as u32).min(6)));
*self.nat_runtime.stun_backoff_until.write().await = Some(Instant::now() + backoff);
}
None
}
}
@@ -0,0 +1,119 @@
use std::collections::{BTreeSet, HashMap};
use std::net::{IpAddr, SocketAddr};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::{Mutex, oneshot};
use tokio_util::sync::CancellationToken;
use super::{IpFamily, MePool, STUN_CACHE_TTL};
// Context retirement and TTL tests use the same gates as production discovery.
#[cfg(test)]
mod tests;
/// Pool-local discovery gates; bound entries are limited to configured egress addresses.
#[derive(Default)]
pub(in crate::transport::middle_proxy) struct NatDiscovery {
pub(super) http: Arc<Mutex<Option<Instant>>>,
bound: Mutex<HashMap<(IpFamily, IpAddr), Arc<BoundDiscovery>>>,
}
#[derive(Default)]
struct BoundDiscovery {
state: Arc<Mutex<BoundDiscoveryState>>,
cancel: CancellationToken,
}
#[derive(Default)]
struct BoundDiscoveryState {
reflection: Option<(Instant, SocketAddr)>,
retry_after: Option<Instant>,
attempt: u8,
}
impl NatDiscovery {
async fn bound_context(
&self,
family: IpFamily,
bind_ip: IpAddr,
allowed: &BTreeSet<IpAddr>,
) -> Option<Arc<BoundDiscovery>> {
let mut contexts = self.bound.lock().await;
// Retired producers cannot republish into the map or a replacement context.
contexts.retain(|(entry_family, ip), context| {
let keep = *entry_family != family || allowed.contains(ip);
if !keep {
context.cancel.cancel();
}
keep
});
if !allowed.contains(&bind_ip) {
return None;
}
Some(contexts.entry((family, bind_ip)).or_default().clone())
}
}
impl MePool {
/// Refreshes one configured source without borrowing another source's cached reflection.
pub(super) async fn reflect_bound_addr(
self: &Arc<Self>,
family: IpFamily,
bind_ip: IpAddr,
) -> Option<SocketAddr> {
if bind_ip.is_ipv6() != matches!(family, IpFamily::V6) {
return None;
}
let allowed = self
.upstream
.as_ref()?
.direct_bind_ips_for_family(bind_ip.is_ipv6())
.await;
let context = self
.nat_runtime
.discovery
.bound_context(family, bind_ip, &allowed)
.await?;
let mut flight = tokio::select! {
biased;
_ = context.cancel.cancelled() => return None,
guard = context.state.clone().lock_owned() => guard,
};
if let Some((created, addr)) = flight.reflection
&& created.elapsed() < STUN_CACHE_TTL
{
return Some(addr);
}
if flight
.retry_after
.is_some_and(|until| Instant::now() < until)
{
return None;
}
let pool = self.clone();
let (send, receive) = oneshot::channel();
self.lifecycle.spawn_producer(async move {
tokio::select! {
biased;
_ = context.cancel.cancelled() => {}
_ = async {
let reflected = pool.refresh_stun(family, Some(bind_ip), flight.attempt).await;
if let Some(addr) = reflected {
flight.reflection = Some((Instant::now(), addr));
flight.retry_after = None;
flight.attempt = 0;
} else {
// Completed failures must also survive a short reconnect deadline.
let backoff = Duration::from_secs(60 * 2u64.pow(flight.attempt as u32));
flight.retry_after = Some(Instant::now() + backoff);
flight.attempt = flight.attempt.saturating_add(1).min(6);
}
let _ = send.send(reflected);
} => {}
}
}).ok()?;
receive.await.ok().flatten()
}
}
@@ -0,0 +1,146 @@
use super::*;
use crate::transport::middle_proxy::pool_nat::tests::lifecycle::set_bind_addresses;
use crate::transport::middle_proxy::pool_nat::tests::{
PUBLIC_IP, TEST_DEADLINE, nat_pool, stun_server,
};
use std::net::Ipv4Addr;
use std::sync::atomic::Ordering;
use tokio::time::timeout;
#[tokio::test]
async fn bound_positive_ttl_refreshes_without_publishing_to_default_cache() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
stun.release.add_permits(16);
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let ip = IpAddr::V4(Ipv4Addr::LOCALHOST);
let allowed = BTreeSet::from([ip]);
let context = pool
.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &allowed)
.await
.unwrap();
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
context.state.lock().await.reflection.as_mut().unwrap().0 = Instant::now() - STUN_CACHE_TTL;
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 2);
assert_eq!(pool.nat_runtime.nat_reflection_cache.lock().await.v4, None);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn context_churn_is_bounded_and_retirement_does_not_mix_families() {
let discovery = NatDiscovery::default();
let v6: IpAddr = "::1".parse().unwrap();
let v6_context = discovery
.bound_context(IpFamily::V6, v6, &BTreeSet::from([v6]))
.await
.unwrap();
let mut previous: Option<Arc<BoundDiscovery>> = None;
for suffix in 1..=1000u32 {
let ip = IpAddr::V4(Ipv4Addr::from(0x7f000000 + suffix));
let context = discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::from([ip]))
.await
.unwrap();
if let Some(old) = previous.replace(context.clone()) {
assert!(old.cancel.is_cancelled());
}
assert!(!context.cancel.is_cancelled());
assert!(!v6_context.cancel.is_cancelled());
assert_eq!(discovery.bound.lock().await.len(), 2);
}
let denied = IpAddr::V4(Ipv4Addr::LOCALHOST);
assert!(
discovery
.bound_context(IpFamily::V4, denied, &BTreeSet::new())
.await
.is_none()
);
assert!(previous.unwrap().cancel.is_cancelled());
assert_eq!(discovery.bound.lock().await.len(), 1);
}
#[tokio::test]
async fn retiring_bound_context_cancels_its_producer_and_prevents_late_publication() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let ip = IpAddr::V4(Ipv4Addr::LOCALHOST);
let context = pool
.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::from([ip]))
.await
.unwrap();
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
});
stun.received().await;
assert!(
pool.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::new())
.await
.is_none()
);
assert_eq!(timeout(TEST_DEADLINE, waiter).await.unwrap().unwrap(), None);
stun.release.add_permits(16);
assert_eq!(context.state.lock().await.reflection, None);
assert!(pool.nat_runtime.discovery.bound.lock().await.is_empty());
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn bound_failure_retry_deadline_expires_and_success_resets_backoff() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
stun.release.add_permits(16);
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let ip = IpAddr::V4(Ipv4Addr::LOCALHOST);
let context = pool
.nat_runtime
.discovery
.bound_context(IpFamily::V4, ip, &BTreeSet::from([ip]))
.await
.unwrap();
{
let mut state = context.state.lock().await;
state.retry_after = Some(Instant::now() + Duration::from_secs(60));
state.attempt = 3;
}
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_none()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 0);
context.state.lock().await.retry_after = Some(Instant::now() - Duration::from_secs(1));
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some(ip))
.await
.is_some()
);
assert_eq!(context.state.lock().await.attempt, 0);
assert_eq!(context.state.lock().await.retry_after, None);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
@@ -0,0 +1,294 @@
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, UdpSocket};
use tokio::sync::Semaphore;
use tokio::task::JoinHandle;
use super::*;
use crate::network::probe::NetworkDecision;
use crate::transport::middle_proxy::pool_writer_security_tests::make_pool_with_decision;
/// Public IPv4 used as source material by local discovery and encrypted ME fixtures.
pub(super) const PUBLIC_IP: Ipv4Addr = Ipv4Addr::new(91, 108, 56, 1);
/// Bounds asynchronous fixture assertions without relying on external services.
pub(super) const TEST_DEADLINE: Duration = Duration::from_secs(2);
// Concurrency and end-to-end regressions share gated local discovery servers.
pub(super) mod lifecycle;
mod recovery;
/// Gated local discovery service with observable request counts.
pub(super) struct DiscoveryServer {
/// Listening address selected by the kernel.
pub(super) addr: SocketAddr,
/// Number of requests actually received from clients.
pub(super) requests: Arc<AtomicUsize>,
arrived: Arc<Semaphore>,
/// One permit releases one discovery response.
pub(super) release: Arc<Semaphore>,
task: JoinHandle<()>,
}
impl Drop for DiscoveryServer {
fn drop(&mut self) {
self.task.abort();
}
}
impl DiscoveryServer {
/// Waits until a real discovery request reaches the server.
pub(super) async fn received(&self) {
tokio::time::timeout(TEST_DEADLINE, self.arrived.acquire())
.await
.expect("discovery request must reach the local server")
.unwrap()
.forget();
}
}
/// Serves valid STUN responses with a deliberately different reflected port.
pub(super) async fn stun_server(bind: SocketAddr, reflected: IpAddr) -> DiscoveryServer {
stun_server_response(bind, Some(reflected), Duration::ZERO).await
}
async fn stun_server_response(
bind: SocketAddr,
reflected: Option<IpAddr>,
delay: Duration,
) -> DiscoveryServer {
let socket = UdpSocket::bind(bind).await.unwrap();
let addr = socket.local_addr().unwrap();
let requests = Arc::new(AtomicUsize::new(0));
let arrived = Arc::new(Semaphore::new(0));
let release = Arc::new(Semaphore::new(0));
let count = requests.clone();
let signal = arrived.clone();
let gate = release.clone();
let task = tokio::spawn(async move {
let mut request = [0u8; 512];
loop {
let (len, peer) = socket.recv_from(&mut request).await.unwrap();
assert!(len >= 20);
count.fetch_add(1, Ordering::Relaxed);
signal.add_permits(1);
gate.acquire().await.unwrap().forget();
tokio::time::sleep(delay).await;
let Some(reflected) = reflected else {
socket.send_to(&[0u8], peer).await.unwrap();
continue;
};
let (family, ip) = match reflected {
IpAddr::V4(ip) => (1u8, ip.octets().to_vec()),
IpAddr::V6(ip) => (2u8, ip.octets().to_vec()),
};
let attribute_len = 4 + ip.len();
let mut response = vec![0u8; 24 + attribute_len];
response[..2].copy_from_slice(&0x0101u16.to_be_bytes());
response[2..4].copy_from_slice(&((4 + attribute_len) as u16).to_be_bytes());
response[4..20].copy_from_slice(&request[4..20]);
response[20..22].copy_from_slice(&1u16.to_be_bytes());
response[22..24].copy_from_slice(&(attribute_len as u16).to_be_bytes());
response[25] = family;
response[26..28].copy_from_slice(&45678u16.to_be_bytes());
response[28..].copy_from_slice(&ip);
socket.send_to(&response, peer).await.unwrap();
}
});
DiscoveryServer {
addr,
requests,
arrived,
release,
task,
}
}
async fn http_server(body: &'static str) -> DiscoveryServer {
http_server_with_delay(body, Duration::ZERO).await
}
async fn http_server_with_delay(body: &'static str, delay: Duration) -> DiscoveryServer {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let requests = Arc::new(AtomicUsize::new(0));
let arrived = Arc::new(Semaphore::new(0));
let release = Arc::new(Semaphore::new(0));
let count = requests.clone();
let signal = arrived.clone();
let gate = release.clone();
let task = tokio::spawn(async move {
loop {
let (mut stream, _) = listener.accept().await.unwrap();
let mut request = [0u8; 4096];
let _ = stream.read(&mut request).await.unwrap();
count.fetch_add(1, Ordering::Relaxed);
signal.add_permits(1);
gate.acquire().await.unwrap().forget();
tokio::time::sleep(delay).await;
let response = format!(
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
body.len(),
body
);
let _ = stream.write_all(response.as_bytes()).await;
}
});
DiscoveryServer {
addr,
requests,
arrived,
release,
task,
}
}
/// Builds an isolated dual-stack pool using only caller-supplied discovery services.
pub(super) async fn nat_pool(servers: Vec<String>, urls: Vec<String>) -> Arc<MePool> {
let mut pool = make_pool_with_decision(NetworkDecision {
ipv4_me: true,
ipv6_me: true,
..NetworkDecision::default()
})
.await;
let pool_mut = Arc::get_mut(&mut pool).unwrap();
let nat = Arc::get_mut(&mut pool_mut.nat_runtime).unwrap();
nat.nat_probe = true;
nat.nat_stun_servers = servers;
nat.http_ip_detect_urls = urls;
nat.nat_probe_concurrency = 2;
pool
}
async fn wait_for_reflection(pool: &MePool, family: IpFamily) -> SocketAddr {
tokio::time::timeout(TEST_DEADLINE, async {
loop {
let cached = {
let cache = pool.nat_runtime.nat_reflection_cache.lock().await;
match family {
IpFamily::V4 => cache.v4,
IpFamily::V6 => cache.v6,
}
};
if let Some((created, addr)) = cached
&& created.elapsed() < Duration::from_secs(600)
{
return addr;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.expect("pool-owned STUN refresh must publish after its caller is cancelled")
}
#[tokio::test]
async fn expired_stun_refresh_survives_reconnect_timeout() {
let fast = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let slow = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
fast.release.add_permits(16);
let pool = nat_pool(vec![fast.addr.to_string(), slow.addr.to_string()], vec![]).await;
pool.nat_runtime.nat_reflection_cache.lock().await.v4 = Some((
Instant::now() - Duration::from_secs(601),
SocketAddr::new(PUBLIC_IP.into(), 1),
));
let caller_pool = pool.clone();
let caller = tokio::spawn(async move {
tokio::time::timeout(
Duration::from_millis(100),
caller_pool.maybe_reflect_public_addr(IpFamily::V4, None),
)
.await
});
fast.received().await;
slow.received().await;
assert!(caller.await.unwrap().is_err());
slow.release.add_permits(16);
assert_eq!(
wait_for_reflection(&pool, IpFamily::V4).await.ip(),
PUBLIC_IP
);
assert_eq!(fast.requests.load(Ordering::Relaxed), 1);
assert_eq!(slow.requests.load(Ordering::Relaxed), 1);
assert_eq!(
pool.nat_runtime.nat_probe_attempts.load(Ordering::Relaxed),
0
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn http_discovery_survives_reconnect_timeout() {
let http = http_server("91.108.56.1").await;
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
let caller_pool = pool.clone();
let caller = tokio::spawn(async move {
tokio::time::timeout(
Duration::from_millis(100),
caller_pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()),
)
.await
});
http.received().await;
assert!(caller.await.unwrap().is_err());
http.release.add_permits(16);
tokio::time::timeout(TEST_DEADLINE, async {
while pool.nat_runtime.nat_ip_detected.read().await.is_none() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.expect("pool-owned HTTP discovery must publish after caller cancellation");
assert_eq!(
*pool.nat_runtime.nat_ip_detected.read().await,
Some(PUBLIC_IP.into())
);
assert_eq!(http.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn failed_http_discovery_is_reused_during_negative_cache_window() {
let http = http_server("not an IP address").await;
http.release.add_permits(16);
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
for _ in 0..2 {
assert!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into())
.await
.is_none()
);
}
assert_eq!(
http.requests.load(Ordering::Relaxed),
1,
"a failed HTTP lookup must not restart ahead of every STUN/handshake attempt"
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn successful_stun_publication_waits_for_cache_reader() {
let server = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(vec![server.addr.to_string()], vec![]).await;
let caller_pool = pool.clone();
let caller = tokio::spawn(async move {
caller_pool
.maybe_reflect_public_addr(IpFamily::V4, None)
.await
});
server.received().await;
let cache_reader = pool.nat_runtime.nat_reflection_cache.lock().await;
server.release.add_permits(16);
tokio::time::sleep(Duration::from_millis(50)).await;
drop(cache_reader);
assert_eq!(caller.await.unwrap().unwrap().ip(), PUBLIC_IP);
assert_eq!(
wait_for_reflection(&pool, IpFamily::V4).await.ip(),
PUBLIC_IP
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
@@ -0,0 +1,342 @@
use super::*;
use crate::config::{UpstreamConfig, UpstreamType};
use crate::transport::UpstreamManager;
/// Gives an isolated pool the same finite bind-address authority as production egress.
pub(in crate::transport::middle_proxy::pool_nat) fn set_bind_addresses(
pool: &mut Arc<MePool>,
addresses: &[&str],
) {
let manager = UpstreamManager::new(
vec![UpstreamConfig {
upstream_type: UpstreamType::Direct {
interface: None,
bind_addresses: Some(addresses.iter().map(|value| value.to_string()).collect()),
bindtodevice: None,
},
weight: 1,
enabled: true,
scopes: String::new(),
selected_scope: String::new(),
ipv4: None,
ipv6: None,
prefer: None,
}],
1,
0,
1000,
1,
1,
true,
pool.stats.clone(),
);
Arc::get_mut(pool).unwrap().upstream = Some(Arc::new(manager));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn cancelled_waiter_storm_shares_one_http_and_stun_producer() {
let http = http_server("91.108.56.1").await;
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(
vec![stun.addr.to_string()],
vec![format!("http://{}/", http.addr)],
)
.await;
let mut waiters = JoinSet::new();
for _ in 0..32 {
let pool = pool.clone();
waiters.spawn(async move {
tokio::join!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()),
pool.maybe_reflect_public_addr(IpFamily::V4, None),
);
});
}
http.received().await;
stun.received().await;
waiters.abort_all();
while waiters.join_next().await.is_some() {}
http.release.add_permits(64);
stun.release.add_permits(64);
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
Some(PUBLIC_IP.into())
);
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, None)
.await
.unwrap()
.ip(),
PUBLIC_IP
);
assert_eq!(http.requests.load(Ordering::Relaxed), 1);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn shutdown_cancels_discovery_and_releases_all_waiters() {
let http = http_server("91.108.56.1").await;
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(
vec![stun.addr.to_string()],
vec![format!("http://{}/", http.addr)],
)
.await;
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
tokio::join!(
waiter_pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()),
waiter_pool.maybe_reflect_public_addr(IpFamily::V4, None),
)
});
http.received().await;
stun.received().await;
assert!(pool.shutdown_until(TEST_DEADLINE).await);
assert_eq!(
timeout(TEST_DEADLINE, waiter).await.unwrap().unwrap(),
(None, None)
);
http.release.add_permits(64);
stun.release.add_permits(64);
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, None).await,
None
);
assert_eq!(*pool.nat_runtime.nat_ip_detected.read().await, None);
assert_eq!(pool.nat_runtime.nat_reflection_cache.lock().await.v4, None);
assert_eq!(http.requests.load(Ordering::Relaxed), 1);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(
pool.nat_runtime
.nat_reflection_singleflight_v4
.try_lock()
.is_ok()
);
assert!(pool.nat_runtime.discovery.http.try_lock().is_ok());
}
#[tokio::test]
async fn bound_refresh_survives_cancellation_and_reuses_only_its_source() {
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1", "127.0.0.2"]);
let bind = Some(Ipv4Addr::LOCALHOST.into());
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, bind)
.await
});
stun.received().await;
waiter.abort();
assert!(waiter.await.unwrap_err().is_cancelled());
stun.release.add_permits(64);
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, bind)
.await
.unwrap()
.ip(),
PUBLIC_IP
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert_eq!(pool.nat_runtime.nat_reflection_cache.lock().await.v4, None);
let other = Some("127.0.0.2".parse().unwrap());
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, other)
.await
.unwrap()
.ip(),
PUBLIC_IP
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 2);
assert!(
pool.maybe_reflect_public_addr(IpFamily::V4, Some("127.0.0.3".parse().unwrap()))
.await
.is_none()
);
assert!(
pool.maybe_reflect_public_addr(IpFamily::V6, bind)
.await
.is_none()
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 2);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn failed_bound_refresh_is_observable_after_caller_cancellation() {
let stun = stun_server_response("127.0.0.1:0".parse().unwrap(), None, Duration::ZERO).await;
let mut pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
set_bind_addresses(&mut pool, &["127.0.0.1"]);
let bind = Some(Ipv4Addr::LOCALHOST.into());
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, bind)
.await
});
stun.received().await;
waiter.abort();
let _ = waiter.await;
// One response completes the original probe; a restarted probe would block forever here.
stun.release.add_permits(1);
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V4, bind)
)
.await
.unwrap(),
None
);
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V4, bind)
)
.await
.unwrap(),
None
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn http_negative_cache_expires_without_waiting_a_minute() {
let http = http_server("invalid").await;
http.release.add_permits(64);
let pool = nat_pool(vec![], vec![format!("http://{}/", http.addr)]).await;
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
*pool.nat_runtime.discovery.http.lock().await = Some(Instant::now() - Duration::from_secs(1));
assert_eq!(
pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()).await,
None
);
assert_eq!(http.requests.load(Ordering::Relaxed), 2);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn ipv6_refresh_is_independent_of_blocked_ipv4_refresh() {
let v4 = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let public_v6: IpAddr = "2001:4860:4860::8888".parse().unwrap();
let v6 = stun_server("[::1]:0".parse().unwrap(), public_v6).await;
v6.release.add_permits(16);
let pool = nat_pool(vec![v4.addr.to_string(), v6.addr.to_string()], vec![]).await;
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, None)
.await
});
v4.received().await;
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V6, None)
)
.await
.unwrap()
.unwrap()
.ip(),
public_v6
);
assert!(!waiter.is_finished());
v4.release.add_permits(16);
assert_eq!(waiter.await.unwrap().unwrap().ip(), PUBLIC_IP);
assert_eq!(
pool.nat_runtime
.nat_reflection_cache
.lock()
.await
.v4
.unwrap()
.1
.ip(),
PUBLIC_IP
);
assert_eq!(
pool.nat_runtime
.nat_reflection_cache
.lock()
.await
.v6
.unwrap()
.1
.ip(),
public_v6
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn failed_shared_refresh_publishes_backoff_after_caller_cancellation() {
let stun = stun_server_response("127.0.0.1:0".parse().unwrap(), None, Duration::ZERO).await;
let pool = nat_pool(vec![stun.addr.to_string()], vec![]).await;
let waiter_pool = pool.clone();
let waiter = tokio::spawn(async move {
waiter_pool
.maybe_reflect_public_addr(IpFamily::V4, None)
.await
});
stun.received().await;
waiter.abort();
let _ = waiter.await;
stun.release.add_permits(1);
assert_eq!(
timeout(
TEST_DEADLINE,
pool.maybe_reflect_public_addr(IpFamily::V4, None)
)
.await
.unwrap(),
None
);
assert!(pool.nat_runtime.stun_backoff_until.read().await.unwrap() > Instant::now());
assert_eq!(
pool.maybe_reflect_public_addr(IpFamily::V4, None).await,
None
);
assert_eq!(stun.requests.load(Ordering::Relaxed), 1);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn shutdown_before_producer_first_poll_releases_owned_gates() {
let http = http_server("91.108.56.1").await;
let stun = stun_server("127.0.0.1:0".parse().unwrap(), PUBLIC_IP.into()).await;
let pool = nat_pool(
vec![stun.addr.to_string()],
vec![format!("http://{}/", http.addr)],
)
.await;
let mut http_waiter =
tokio_test::task::spawn(pool.maybe_detect_nat_ip(Ipv4Addr::LOCALHOST.into()));
let mut stun_waiter =
tokio_test::task::spawn(pool.maybe_reflect_public_addr(IpFamily::V4, None));
assert!(http_waiter.poll().is_pending());
assert!(stun_waiter.poll().is_pending());
// On this current-thread runtime neither producer can run before shutdown admission closes.
pool.begin_shutdown();
assert!(pool.shutdown_until(TEST_DEADLINE).await);
assert_eq!(http_waiter.poll(), std::task::Poll::Ready(None));
assert_eq!(stun_waiter.poll(), std::task::Poll::Ready(None));
assert_eq!(http.requests.load(Ordering::Relaxed), 0);
assert_eq!(stun.requests.load(Ordering::Relaxed), 0);
assert!(pool.nat_runtime.discovery.http.try_lock().is_ok());
assert!(
pool.nat_runtime
.nat_reflection_singleflight_v4
.try_lock()
.is_ok()
);
}
@@ -0,0 +1,266 @@
use super::*;
use crate::crypto::{SecureRandom, derive_middleproxy_keys};
use crate::transport::middle_proxy::codec::{
RpcChecksumMode, build_handshake_payload, build_nonce_payload, build_rpc_frame,
cbc_decrypt_inplace, cbc_encrypt_padded, parse_handshake_flags, parse_nonce_payload,
read_rpc_frame_plaintext, rpc_crc,
};
use crate::transport::middle_proxy::me_health_monitor;
use crate::transport::middle_proxy::pool::MeFamilyRuntimeState;
use tokio::net::TcpStream;
struct MiddlePeer {
addr: SocketAddr,
handshakes: Arc<AtomicUsize>,
cancelled: Arc<AtomicUsize>,
task: JoinHandle<()>,
}
impl Drop for MiddlePeer {
fn drop(&mut self) {
self.task.abort();
}
}
impl MiddlePeer {
async fn new(server_ip: Ipv4Addr) -> Self {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let handshakes = Arc::new(AtomicUsize::new(0));
let cancelled = Arc::new(AtomicUsize::new(0));
let succeeded = handshakes.clone();
let closed = cancelled.clone();
let task = tokio::spawn(async move {
let mut pending = JoinSet::new();
let mut established = Vec::new();
loop {
tokio::select! {
accepted = listener.accept() => {
let (stream, _) = accepted.unwrap();
pending.spawn(accept_handshake(stream, server_ip));
}
completed = pending.join_next(), if !pending.is_empty() => {
match completed.unwrap().expect("fake ME handshake task must not panic") {
Some(stream) => {
established.push(stream);
succeeded.fetch_add(1, Ordering::Release);
}
None => { closed.fetch_add(1, Ordering::Release); }
}
}
}
}
});
Self {
addr,
handshakes,
cancelled,
task,
}
}
}
async fn accept_handshake(mut stream: TcpStream, server_ip: Ipv4Addr) -> Option<TcpStream> {
let client_port = stream.peer_addr().unwrap().port();
let server_port = stream.local_addr().unwrap().port();
let (seq, payload) = match read_rpc_frame_plaintext(&mut stream).await {
Ok(frame) => frame,
// Reconnect may time out during NAT discovery, before sending any nonce.
Err(ProxyError::Io(error)) if error.kind() == std::io::ErrorKind::UnexpectedEof => {
return None;
}
Err(error) => panic!("invalid client nonce: {error}"),
};
assert_eq!(seq, -2);
let (selector, _, timestamp, client_nonce) = parse_nonce_payload(&payload).unwrap();
let server_nonce = [0x42u8; 16];
let nonce = build_nonce_payload(selector, timestamp, &server_nonce);
stream
.write_all(&build_rpc_frame(-2, &nonce, RpcChecksumMode::Crc32))
.await
.unwrap();
let mut client_ip = PUBLIC_IP.octets();
client_ip.reverse();
let mut server_ip = server_ip.octets();
server_ip.reverse();
let (read_key, read_iv) = derive_middleproxy_keys(
&server_nonce,
&client_nonce,
&timestamp.to_le_bytes(),
Some(&server_ip),
&client_port.to_le_bytes(),
b"CLIENT",
Some(&client_ip),
&server_port.to_le_bytes(),
&[1u8; 32],
None,
None,
);
let (write_key, write_iv) = derive_middleproxy_keys(
&server_nonce,
&client_nonce,
&timestamp.to_le_bytes(),
Some(&server_ip),
&client_port.to_le_bytes(),
b"SERVER",
Some(&client_ip),
&server_port.to_le_bytes(),
&[1u8; 32],
None,
None,
);
let mut encrypted = [0u8; 48];
stream.read_exact(&mut encrypted).await.unwrap();
cbc_decrypt_inplace(&read_key, &read_iv, &mut encrypted).unwrap();
assert_eq!(u32::from_le_bytes(encrypted[..4].try_into().unwrap()), 44);
assert_eq!(i32::from_le_bytes(encrypted[4..8].try_into().unwrap()), -1);
assert_eq!(
u32::from_le_bytes(encrypted[40..44].try_into().unwrap()),
rpc_crc(RpcChecksumMode::Crc32, &encrypted[..40])
);
parse_handshake_flags(&encrypted[8..40]).unwrap();
assert_eq!(&encrypted[16..20], &client_ip);
assert_eq!(
u16::from_le_bytes(encrypted[20..22].try_into().unwrap()),
client_port
);
assert_eq!(
u16::from_le_bytes(encrypted[32..34].try_into().unwrap()),
server_port
);
let reply = build_handshake_payload(server_ip, server_port, client_ip, client_port, 0);
let (encrypted, _) = cbc_encrypt_padded(
&write_key,
&write_iv,
&build_rpc_frame(-1, &reply, RpcChecksumMode::Crc32),
)
.unwrap();
stream.write_all(&encrypted).await.unwrap();
Some(stream)
}
async fn suppressed_pool_recovers_after_delayed_discovery(http_slow: bool) {
// Every lookup exceeds me_one_timeout_ms, not just the first request of the outage.
let delay = Duration::from_millis(1500);
let stun = stun_server_response(
"127.0.0.1:0".parse().unwrap(),
Some(PUBLIC_IP.into()),
delay,
)
.await;
let http = http_server_with_delay("91.108.56.1", delay).await;
stun.release.add_permits(64);
http.release.add_permits(64);
let peer = MiddlePeer::new(if http_slow {
PUBLIC_IP
} else {
Ipv4Addr::LOCALHOST
})
.await;
let urls = if http_slow {
vec![format!("http://{}/", http.addr)]
} else {
vec![]
};
let pool = nat_pool(vec![stun.addr.to_string()], urls).await;
pool.health_runtime
.me_health_interval_ms_unhealthy
.store(10, Ordering::Relaxed);
pool.health_runtime
.me_health_interval_ms_healthy
.store(10, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_outage_backoff_min_ms
.store(100, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_outage_backoff_max_ms
.store(100, Ordering::Relaxed);
pool.single_endpoint_runtime
.me_single_endpoint_shadow_writers
.store(0, Ordering::Relaxed);
pool.update_proxy_maps(
HashMap::from([(2, vec![(peer.addr.ip(), peer.addr.port())])]),
None,
)
.await;
let generation = pool.current_generation();
let now = MePool::now_epoch_secs();
pool.set_family_runtime_state(
IpFamily::V4,
MeFamilyRuntimeState::Suppressed,
now,
now + 3600,
5,
0,
);
pool.nat_runtime.nat_reflection_cache.lock().await.v4 = Some((
if http_slow {
Instant::now()
} else {
Instant::now() - Duration::from_secs(601)
},
SocketAddr::new(PUBLIC_IP.into(), 45678),
));
let monitor = tokio::spawn(me_health_monitor(
pool.clone(),
Arc::new(SecureRandom::new()),
0,
));
if http_slow {
http.received().await;
} else {
stun.received().await;
}
timeout(Duration::from_secs(5), async {
while peer.cancelled.load(Ordering::Acquire) == 0 {
assert!(!peer.task.is_finished());
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("the actual 1200ms reconnect must cancel at least one pre-KDF TCP connection");
assert_eq!(peer.handshakes.load(Ordering::Acquire), 0);
let expected = 2 * pool.required_writers_for_dc(1);
timeout(Duration::from_secs(10), async {
while !pool.admission_ready_full_floor().await
|| peer.handshakes.load(Ordering::Acquire) < expected
{
assert!(
!peer.task.is_finished(),
"fake ME must validate every encrypted handshake"
);
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("suppressed pool must autonomously regain its full floor without restart");
monitor.abort();
let _ = monitor.await;
assert_eq!(pool.current_generation(), generation);
assert_eq!(pool.reinit.status.load().pending_hardswap_generation, 0);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert_eq!(pool.writers.read().await.len(), expected);
assert!(peer.handshakes.load(Ordering::Acquire) >= expected);
assert_eq!(
http.requests.load(Ordering::Relaxed),
usize::from(http_slow)
);
assert_eq!(
stun.requests.load(Ordering::Relaxed),
usize::from(!http_slow)
);
assert!(pool.shutdown_until(TEST_DEADLINE).await);
}
#[tokio::test]
async fn suppressed_pool_recovers_full_floor_after_expired_stun_refresh() {
suppressed_pool_recovers_after_delayed_discovery(false).await;
}
#[tokio::test]
async fn suppressed_pool_recovers_full_floor_after_slow_http_with_fresh_stun() {
suppressed_pool_recovers_after_delayed_discovery(true).await;
}
@@ -33,6 +33,10 @@ mod publication;
mod replacement;
mod runtime;
// Recovery regressions exercise the serialized production publication boundary.
#[cfg(test)]
mod recovery_tests;
struct PreparedWriter<'a> {
writer: MeWriter,
tx: mpsc::Sender<WriterCommand>,
@@ -84,7 +84,17 @@ impl MePool {
crate::network::IpFamily::V6
};
let now_epoch_secs = Self::now_epoch_secs();
if !self.family_enabled_for_drain_coverage(family, now_epoch_secs) {
let configured = match family {
crate::network::IpFamily::V4 => self.decision.ipv4_me,
crate::network::IpFamily::V6 => self.decision.ipv6_me,
};
// Recovery coverage must remain publishable during suppression; the serialized
// per-group floor check below still bounds it and generation authority is unchanged.
let recovery_coverage =
contour == WriterContour::Active && intent == WriterOpenIntent::Coverage;
if !configured
|| (self.is_family_temporarily_suppressed(family, now_epoch_secs) && !recovery_coverage)
{
return Err(ProxyError::Proxy(
"ME writer family lost publication authority".into(),
));
@@ -0,0 +1,466 @@
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use super::*;
use crate::network::{IpFamily, probe::NetworkDecision};
use crate::transport::middle_proxy::admission_test_support::unregistered_writer;
use crate::transport::middle_proxy::pool::MeFamilyRuntimeState;
use crate::transport::middle_proxy::pool_writer_security_tests::{
make_pool, make_pool_with_decision,
};
fn cap_at_one(pool: &MePool) {
pool.floor_runtime
.me_adaptive_floor_cpu_cores_override
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_per_core
.store(1, Ordering::Relaxed);
pool.floor_runtime
.me_adaptive_floor_max_active_writers_global
.store(1, Ordering::Relaxed);
}
fn suppress(pool: &MePool, family: IpFamily) {
let now = MePool::now_epoch_secs();
pool.set_family_runtime_state(
family,
MeFamilyRuntimeState::Suppressed,
now,
now + 3600,
5,
0,
);
}
fn prepared<'a>(
pool: &'a Arc<MePool>,
writer: MeWriter,
intent: WriterOpenIntent,
reservation: WriterOpenReservation<'a>,
task_started: Arc<AtomicBool>,
) -> PreparedWriter<'a> {
let cancel = writer.cancel.clone();
PreparedWriter {
tx: writer.tx.clone(),
byte_budget: writer.byte_budget.clone(),
writer,
task_registration: pool.lifecycle.try_register().unwrap(),
writer_task: Box::pin(async move {
task_started.store(true, Ordering::Release);
cancel.cancelled().await;
}),
intent,
_open_reservation: reservation,
}
}
#[tokio::test]
async fn suppressed_active_coverage_publishes_only_the_missing_floor() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
let required = pool.required_writers_for_dc(1);
let mut receivers = Vec::new();
for id in 1..=required as u64 {
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.expect("missing coverage must have a reservation despite suppression");
let (writer, receiver) = unregistered_writer(
&pool,
id,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
receivers.push(receiver);
pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
Arc::new(AtomicBool::new(false)),
))
.await
.expect("authenticated coverage must publish without waiting for suppression expiry");
assert_eq!(pool.conn_count.load(Ordering::Relaxed), id as usize);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
assert!(
pool.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.is_none()
);
let (extra, _receiver) = unregistered_writer(
&pool,
99,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
assert!(
pool.authorize_writer_publication_capacity(
&extra,
WriterContour::Active,
WriterOpenIntent::Coverage,
&pool.writers.read().await,
)
.is_err()
);
assert_eq!(
pool.family_runtime_state(IpFamily::V4),
MeFamilyRuntimeState::Suppressed
);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn concurrent_suppressed_coverage_publications_cannot_overfill_a_group() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
let required = pool.required_writers_for_dc(1);
let mut receivers = Vec::new();
let mut publications = Vec::new();
for id in 1..=(required + 1) as u64 {
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, receiver) = unregistered_writer(
&pool,
id,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
receivers.push(receiver);
publications.push(pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
Arc::new(AtomicBool::new(false)),
)));
}
let guard = pool.writers.write().await;
let results = futures::future::join_all(publications);
tokio::pin!(results);
tokio::select! {
biased;
results = &mut results => panic!("publication bypassed the writer barrier: {results:?}"),
_ = tokio::task::yield_now() => {}
}
drop(guard);
let results = results.await;
assert_eq!(
results.iter().filter(|result| result.is_ok()).count(),
required
);
assert_eq!(results.iter().filter(|result| result.is_err()).count(), 1);
assert_eq!(pool.writers.read().await.len(), required);
assert_eq!(pool.conn_count.load(Ordering::Relaxed), required);
assert_eq!(
pool.registry.writer_idle_since_snapshot().await.len(),
required
);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn committed_warm_writer_uses_resolved_active_coverage_authority() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
let reservation = pool
.reserve_writer_open(WriterContour::Warm, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
addr,
pool.current_generation(),
WriterContour::Warm,
);
pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
Arc::new(AtomicBool::new(false)),
))
.await
.expect("a committed generation must resolve warm coverage to its active role");
assert_eq!(
pool.writers.read().await[0].contour.load(Ordering::Acquire),
WriterContour::Active.as_u8()
);
assert_eq!(pool.writer_connect_warm_reserved.load(Ordering::Acquire), 0);
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn suppressed_families_keep_their_full_coverage_reservation_budget() {
let pool = make_pool_with_decision(NetworkDecision {
ipv4_me: true,
ipv6_me: true,
..NetworkDecision::default()
})
.await;
let v4 = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
let v6 = SocketAddr::new(IpAddr::V6(Ipv6Addr::LOCALHOST), 443);
pool.update_proxy_maps(
HashMap::from([(2, vec![(v4.ip(), v4.port())])]),
Some(HashMap::from([(2, vec![(v6.ip(), v6.port())])])),
)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
suppress(&pool, IpFamily::V6);
let required = pool.required_writers_for_dc(1);
assert_eq!(pool.active_coverage_required_total().await, required * 4);
let mut receivers = Vec::new();
let mut writers = pool.writers.write().await;
for (index, (dc, addr)) in [(2, v4), (-2, v4), (-2, v6)].into_iter().enumerate() {
for offset in 0..required {
let (writer, receiver) = unregistered_writer(
&pool,
(index * required + offset + 1) as u64,
dc,
addr,
pool.current_generation(),
WriterContour::Active,
);
writers.push(writer);
receivers.push(receiver);
}
}
drop(writers);
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, v6)
.await
.expect("other covered groups must not consume the suppressed family's floor budget");
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
1
);
drop(reservation);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn coverage_reservations_remain_bounded_and_release_on_drop() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
cap_at_one(&pool);
suppress(&pool, IpFamily::V4);
let limit = pool.required_writers_for_dc(1) * 2
+ pool
.reconnect_runtime
.me_reconnect_max_concurrent_per_dc
.max(1) as usize;
let mut reservations = Vec::new();
for _ in 0..limit {
reservations.push(
pool.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.expect("the configured coverage budget must remain available"),
);
}
assert!(
pool.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr,)
.await
.is_none()
);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
limit
);
drop(reservations);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
}
#[tokio::test]
async fn suppression_exception_does_not_admit_growth_replacement_or_pending_warm() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
for (contour, intent) in [
(WriterContour::Active, WriterOpenIntent::Normal),
(WriterContour::Active, WriterOpenIntent::Replacement),
(WriterContour::Warm, WriterOpenIntent::Coverage),
(WriterContour::Warm, WriterOpenIntent::Normal),
] {
let (writer, _receiver) =
unregistered_writer(&pool, 1, 2, addr, pool.current_generation(), contour);
assert!(
pool.authorize_writer_publication_capacity(&writer, contour, intent, &[])
.is_err()
);
}
}
#[tokio::test]
async fn coverage_exception_never_enables_a_disabled_family() {
let pool = make_pool().await;
let v4 = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
let v6 = SocketAddr::new(IpAddr::V6(Ipv6Addr::LOCALHOST), 443);
pool.update_proxy_maps(
HashMap::from([(2, vec![(v4.ip(), v4.port())])]),
Some(HashMap::from([(2, vec![(v6.ip(), v6.port())])])),
)
.await;
suppress(&pool, IpFamily::V6);
assert_eq!(
pool.active_coverage_required_total().await,
pool.required_writers_for_dc(1) * 2
);
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
v6,
pool.current_generation(),
WriterContour::Active,
);
assert!(
pool.authorize_writer_publication_capacity(
&writer,
WriterContour::Active,
WriterOpenIntent::Coverage,
&[],
)
.is_err()
);
}
#[tokio::test]
async fn suppressed_coverage_cannot_publish_a_stale_generation() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
addr,
pool.current_generation().saturating_sub(1),
WriterContour::Active,
);
let task_started = Arc::new(AtomicBool::new(false));
let error = pool
.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
task_started.clone(),
))
.await
.unwrap_err();
assert!(
error
.to_string()
.contains("generation lost publication authority")
);
assert!(pool.writers.read().await.is_empty());
assert!(pool.registry.writer_idle_since_snapshot().await.is_empty());
assert_eq!(pool.conn_count.load(Ordering::Relaxed), 0);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert!(!task_started.load(Ordering::Acquire));
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
#[tokio::test]
async fn suppressed_coverage_rechecks_endpoints_after_publication_waits() {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
suppress(&pool, IpFamily::V4);
let reservation = pool
.reserve_writer_open(WriterContour::Active, WriterOpenIntent::Coverage, 2, addr)
.await
.unwrap();
let (writer, _receiver) = unregistered_writer(
&pool,
1,
2,
addr,
pool.current_generation(),
WriterContour::Active,
);
let task_started = Arc::new(AtomicBool::new(false));
let publication = pool.publish_connected_writer(prepared(
&pool,
writer,
WriterOpenIntent::Coverage,
reservation,
task_started.clone(),
));
tokio::pin!(publication);
let guard = pool.writers.write().await;
tokio::select! {
biased;
result = &mut publication => panic!("publication bypassed the writer barrier: {result:?}"),
_ = tokio::task::yield_now() => {}
}
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), 444)])]), None)
.await;
drop(guard);
let error = publication.await.unwrap_err();
assert!(
error
.to_string()
.contains("target changed before publication")
);
assert!(pool.writers.read().await.is_empty());
assert!(pool.registry.writer_idle_since_snapshot().await.is_empty());
assert_eq!(pool.conn_count.load(Ordering::Relaxed), 0);
assert_eq!(
pool.writer_connect_active_reserved.load(Ordering::Acquire),
0
);
assert!(!task_started.load(Ordering::Acquire));
assert!(pool.shutdown_until(Duration::from_secs(1)).await);
}
@@ -0,0 +1,98 @@
use std::collections::HashMap;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU32, AtomicU64, Ordering};
use std::time::Instant;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
use super::codec::WriterCommand;
use super::pool::{MePool, MeWriter, WriterContour};
use super::pool_writer_security_tests::make_pool;
/// Owns idle writer visibility without spawning ME transports or client sessions.
pub(crate) struct IdlePoolFixture {
pool: Arc<MePool>,
draining: Vec<Arc<AtomicBool>>,
_receivers: Vec<mpsc::Receiver<WriterCommand>>,
}
impl IdlePoolFixture {
/// Builds both signed DC groups with a controllable readiness quorum.
pub(crate) async fn new(ready: bool) -> Self {
let pool = make_pool().await;
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 443);
pool.update_proxy_maps(HashMap::from([(2, vec![(addr.ip(), addr.port())])]), None)
.await;
let mut draining = Vec::new();
let mut receivers = Vec::new();
{
let mut writers = pool.writers.write().await;
for (id, dc) in [(1, 2), (2, -2)] {
let (writer, receiver) = unregistered_writer(
&pool,
id,
dc,
addr,
pool.current_generation(),
WriterContour::Active,
);
writer.draining.store(!ready, Ordering::Release);
draining.push(writer.draining.clone());
receivers.push(receiver);
writers.push(writer);
}
}
Self {
pool,
draining,
_receivers: receivers,
}
}
/// Returns the pool sampled by the production admission gate.
pub(crate) fn pool(&self) -> Arc<MePool> {
self.pool.clone()
}
/// Changes the fixture quorum without notifying the admission gate.
pub(crate) fn set_ready(&self, ready: bool) {
for draining in &self.draining {
draining.store(!ready, Ordering::Release);
}
}
}
/// Builds a writer whose command receiver remains owned by the test.
pub(super) fn unregistered_writer(
pool: &Arc<MePool>,
id: u64,
dc: i32,
addr: SocketAddr,
generation: u64,
contour: WriterContour,
) -> (MeWriter, mpsc::Receiver<WriterCommand>) {
let (tx, rx) = mpsc::channel(8);
(
MeWriter {
id,
addr,
source_ip: addr.ip(),
writer_dc: dc,
generation,
contour: Arc::new(AtomicU8::new(contour.as_u8())),
created_at: Instant::now(),
tx,
byte_budget: pool.new_writer_byte_budget(),
cancel: CancellationToken::new(),
degraded: Arc::new(AtomicBool::new(false)),
rtt_ema_ms_x10: Arc::new(AtomicU32::new(0)),
draining: Arc::new(AtomicBool::new(false)),
draining_started_at_epoch_secs: Arc::new(AtomicU64::new(0)),
drain_deadline_epoch_secs: Arc::new(AtomicU64::new(0)),
allow_drain_fallback: Arc::new(AtomicBool::new(false)),
},
rx,
)
}
+2
View File
@@ -339,6 +339,8 @@ pub struct UpstreamManager {
// Upstream manager configuration, DNS resolution, and API snapshots.
mod manager_config;
// Current direct egress addresses eligible for source-bound NAT discovery.
mod nat_contexts;
// Latency- and scope-aware upstream selection.
mod selection;
// Direct and proxied connection establishment.
+122
View File
@@ -0,0 +1,122 @@
use super::*;
impl UpstreamManager {
/// Resolves this generation's direct bind candidates without advancing round-robin state.
pub(crate) async fn direct_bind_ips_for_family(&self, ipv6: bool) -> BTreeSet<IpAddr> {
let guard = self.upstreams.read().await;
let target = SocketAddr::new(
if ipv6 {
std::net::Ipv6Addr::UNSPECIFIED.into()
} else {
std::net::Ipv4Addr::UNSPECIFIED.into()
},
0,
);
let mut allowed = BTreeSet::new();
for state in guard.iter() {
let config = &state.config;
let UpstreamType::Direct {
interface,
bind_addresses,
..
} = &config.upstream_type
else {
continue;
};
if (if ipv6 { config.ipv6 } else { config.ipv4 }) == Some(false) {
continue;
}
if let Some(addresses) = bind_addresses.as_ref().filter(|v| !v.is_empty()) {
for address in addresses {
if !address
.parse::<IpAddr>()
.is_ok_and(|ip| ip.is_ipv6() == ipv6)
{
continue;
}
if let Some(ip) = Self::resolve_bind_address(
interface,
&Some(vec![address.clone()]),
target,
None,
true,
) {
allowed.insert(ip);
}
}
} else if let Some(ip) =
Self::resolve_bind_address(interface, bind_addresses, target, None, true)
{
allowed.insert(ip);
}
}
allowed
}
}
#[cfg(test)]
mod tests {
use super::*;
fn config(source: &str) -> UpstreamConfig {
toml::from_str(source).unwrap()
}
#[tokio::test]
async fn candidates_follow_generation_policy_without_advancing_round_robin() {
let manager = UpstreamManager::new(
vec![
config("type = 'direct'\nbind_addresses = ['127.0.0.1', '127.0.0.2', '::1']"),
config("type = 'direct'\ninterface = '127.0.0.3'"),
config("type = 'direct'\ninterface = '127.0.0.4'\nenabled = false"),
config("type = 'direct'\ninterface = '127.0.0.5'\nipv4 = false"),
config("type = 'socks5'\naddress = '127.0.0.1:1080'\ninterface = '127.0.0.6'"),
],
1,
0,
1000,
1,
1,
true,
Arc::new(Stats::new()),
);
manager.upstreams.write().await[0].healthy = false;
let expected = ["127.0.0.1", "127.0.0.2", "127.0.0.3"]
.into_iter()
.map(|value| value.parse().unwrap())
.collect();
assert_eq!(manager.direct_bind_ips_for_family(false).await, expected);
assert_eq!(
manager.direct_bind_ips_for_family(true).await,
BTreeSet::from(["::1".parse().unwrap()])
);
for state in manager.upstreams.read().await.iter() {
assert_eq!(state.bind_rr.load(Ordering::Relaxed), 0);
}
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn named_interface_candidates_match_direct_socket_binding() {
let manager = UpstreamManager::new(
vec![config(
"type = 'direct'\ninterface = 'lo'\nbind_addresses = ['127.0.0.1', '::1']",
)],
1,
0,
1000,
1,
1,
true,
Arc::new(Stats::new()),
);
assert_eq!(
manager.direct_bind_ips_for_family(false).await,
BTreeSet::from(["127.0.0.1".parse().unwrap()])
);
assert_eq!(
manager.direct_bind_ips_for_family(true).await,
BTreeSet::from(["::1".parse().unwrap()])
);
}
}
+3
View File
@@ -1,5 +1,6 @@
use base64::Engine as _;
use crate::config::WebCarrierMethod;
use crate::crypto::SecureRandom;
/// Browser security policy for the transient Telegram Desktop bridge page.
@@ -34,6 +35,7 @@ pub(crate) fn render(
reconnect_grace_secs: u64,
carrier_probe_coalesce_ms: u64,
bridge_diagnostics_enabled: bool,
carrier_method: WebCarrierMethod,
rng: &SecureRandom,
) -> BridgePage {
let mut nonce = [0u8; 18];
@@ -124,6 +126,7 @@ pub(crate) fn render(
.replace("__HOST__", host)
.replace("__BASE_PREFIX__", base_prefix)
.replace("__BOOTSTRAP__", bootstrap)
.replace("__CARRIER_METHOD__", carrier_method.as_str())
.replace("__BATCH_LIMIT__", &batch_limit.to_string())
.replace("__QUEUE_LIMIT__", &queue_limit.to_string())
.replace("__QUEUE_ITEMS__", &queue_items.to_string())
+11 -9
View File
@@ -7,6 +7,8 @@ const responseBody=globalThis.TelemtBridgeResponse;if(!responseBody)throw new Er
const requestSupport=globalThis.TelemtBridgeRequest;if(!requestSupport)throw new Error('missing request runtime');
const bufferSupport=globalThis.TelemtBridgeBuffers;if(!bufferSupport)throw new Error('missing buffer runtime');
const recoverySupport=globalThis.TelemtBridgeRecovery;if(!recoverySupport)throw new Error('missing recovery runtime');
// Keep the method page-owned so recovery and config rollback cannot change frozen retries.
const carrierMethod='__CARRIER_METHOD__';
let negotiationEnabled=__NEGOTIATION_ENABLED__,candidateCount=__CANDIDATE_COUNT__,candidateDeadlines=[__CARRIER_DEADLINES__];
let longPollMs=__LONG_POLL_SECS__*1000,bridgeRequestMs=__BRIDGE_REQUEST_SECS__*1000,bridgeRetryMs=__BRIDGE_RETRY_SECS__*1000;
let bridgeRecoveryMs=__BRIDGE_RECOVERY_SECS__*1000,websocketOpenMs=__WEBSOCKET_OPEN_SECS__*1000,reconnectGraceMs=__RECONNECT_GRACE_SECS__*1000;
@@ -226,7 +228,7 @@ async function createSession(epoch){
async function probeHttp(probe,laneID,epoch){
try{
const headers={'X-Up-Seq':'1'},token=sessionToken,controller=attemptController,body=probe.data;if(laneID!==null)headers['X-Lane-ID']=String(laneID);
const response=await request('/api/v1/up',options('POST',token,body,headers,controller.signal));
const response=await request('/api/v1/up',options(carrierMethod,token,body,headers,controller.signal));
if(closed||epoch!==attemptEpoch)return
if(response.status!==204){advanceCarrier('http',epoch);return}
if(response.headers.get('X-Up-Ack')!=='1'){advanceCarrier('protocol',epoch);return}
@@ -263,14 +265,14 @@ async function runUp(){
lease=takeBatch(upPending,null);upLease=lease;lease.controller=new AbortController();const sequence=String(upSequence),token=sessionToken;
for(;;){
try{
const response=await request('/api/v1/up',options('POST',token,lease.body,{'X-Up-Seq':sequence},lease.controller.signal),null,1);
const response=await request('/api/v1/up',options(carrierMethod,token,lease.body,{'X-Up-Seq':sequence},lease.controller.signal),null,1);
if(response.status!==204)throw failure('http','uplink rejected');
if(response.headers.get('X-Up-Ack')!==sequence)throw failure('protocol','uplink acknowledgement rejected');
break;
}catch(error){
let replayed=false;
const recovered=await recoverTransport(error,async(signal,remaining)=>{
const response=await request('/api/v1/up',options('POST',token,lease.body,{'X-Up-Seq':sequence},signal),remaining,2);
const response=await request('/api/v1/up',options(carrierMethod,token,lease.body,{'X-Up-Seq':sequence},signal),remaining,2);
if(response.status!==204)throw failure('http','uplink replay rejected');
if(response.headers.get('X-Up-Ack')!==sequence)throw failure('protocol','uplink replay acknowledgement rejected');
replayed=true;
@@ -343,7 +345,7 @@ async function poll(){
const token=sessionToken,cursor=downCursor;
try{
pollController=new AbortController();
const response=await request('/api/v1/down',options('POST',token,null,{'X-Down-Cursor':cursor},pollController.signal),null,1);
const response=await request('/api/v1/down',options(carrierMethod,token,null,{'X-Down-Cursor':cursor},pollController.signal),null,1);
if(closed||sessionToken!==token)return;
if(response.status===204){status('connected');continue}
if(response.status!==200)throw failure('http','downlink rejected');
@@ -354,7 +356,7 @@ async function poll(){
}catch(error){
if(closed)return;
const recovered=await recoverTransport(error,async(signal,remaining)=>{
const response=await request('/api/v1/down',options('POST',token,null,{'X-Down-Cursor':cursor},signal),remaining,2);
const response=await request('/api/v1/down',options(carrierMethod,token,null,{'X-Down-Cursor':cursor},signal),remaining,2);
if(response.status===204)return;
if(response.status!==200||!response.body.byteLength||!response.headers.get('X-Down-Cursor'))throw failure('http','downlink replay rejected');
});
@@ -422,14 +424,14 @@ async function runLaneUp(lane){
const sequence=String(lane.sequence),laneID=String(lane.id),token=sessionToken;
for(;;){
try{
const response=await request('/api/v1/up',options('POST',token,lease.body,{'X-Up-Seq':sequence,'X-Lane-ID':laneID},lease.controller.signal),null,1);
const response=await request('/api/v1/up',options(carrierMethod,token,lease.body,{'X-Up-Seq':sequence,'X-Lane-ID':laneID},lease.controller.signal),null,1);
if(response.status!==204)throw failure('http','lane uplink rejected');
if(response.headers.get('X-Up-Ack')!==sequence)throw failure('protocol','lane uplink acknowledgement rejected');
break;
}catch(error){
let replayed=false;
const recovered=await recoverTransport(error,async(signal,remaining)=>{
const response=await request('/api/v1/up',options('POST',token,lease.body,{'X-Up-Seq':sequence,'X-Lane-ID':laneID},signal),remaining,2);
const response=await request('/api/v1/up',options(carrierMethod,token,lease.body,{'X-Up-Seq':sequence,'X-Lane-ID':laneID},signal),remaining,2);
if(response.status!==204)throw failure('http','lane uplink replay rejected');
if(response.headers.get('X-Up-Ack')!==sequence)throw failure('protocol','lane uplink replay acknowledgement rejected');
replayed=true;
@@ -451,7 +453,7 @@ async function pollLane(lane){
while(!closed&&sessionToken&&lanes.get(lane.id)===lane){
const controller=new AbortController(),laneID=String(lane.id),token=sessionToken,cursor=lane.cursor;lane.controller=controller;
failedToken=token;failedCursor=cursor;failedLaneID=laneID;
const response=await request('/api/v1/down',options('POST',token,null,{'X-Down-Cursor':cursor,'X-Lane-ID':laneID},controller.signal),null,1);
const response=await request('/api/v1/down',options(carrierMethod,token,null,{'X-Down-Cursor':cursor,'X-Lane-ID':laneID},controller.signal),null,1);
if(closed||sessionToken!==token||lanes.get(lane.id)!==lane)return;
if(response.status===204){
if(response.headers.get('X-Lane-Closed')==='1'){finishLane(lane,false);return}
@@ -467,7 +469,7 @@ async function pollLane(lane){
}catch(error){
if(!closed&&lanes.get(lane.id)===lane){
const recovered=await recoverTransport(error,async(signal,remaining)=>{
const response=await request('/api/v1/down',options('POST',failedToken,null,{'X-Down-Cursor':failedCursor,'X-Lane-ID':failedLaneID},signal),remaining,2);
const response=await request('/api/v1/down',options(carrierMethod,failedToken,null,{'X-Down-Cursor':failedCursor,'X-Lane-ID':failedLaneID},signal),remaining,2);
if(response.status===204)return;
if(response.status!==200||!response.body.byteLength||!response.headers.get('X-Down-Cursor'))throw failure('http','lane downlink replay rejected');
});
+73
View File
@@ -1,5 +1,73 @@
use super::*;
#[test]
fn carrier_method_is_page_owned_and_used_by_every_https_request() {
for method in [WebCarrierMethod::Post, WebCarrierMethod::Put] {
let page = render(
"proxy.example.com",
"/telegram/web/",
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
2 * 1024 * 1024,
32 * 1024 * 1024,
16 * 1024,
1024,
true,
4,
[3, 5, 8, 12],
25,
10,
90,
15,
15,
120,
0,
true,
method,
&SecureRandom::new(),
);
assert!(!page.body.contains("__"));
assert!(
page.body
.contains(&format!("const carrierMethod='{}';", method.as_str()))
);
assert_eq!(page.body.matches("carrierMethod=").count(), 1);
assert_eq!(page.body.matches("options(carrierMethod,").count(), 9);
let requests: Vec<_> = page
.body
.lines()
.filter(|line| {
line.contains("request('/api/v1/up'") || line.contains("request('/api/v1/down'")
})
.collect();
assert_eq!(requests.len(), 9);
assert!(
requests
.iter()
.all(|line| line.contains("options(carrierMethod,"))
);
assert_eq!(page.body.matches("options('POST',bootstrap,").count(), 2);
assert!(
page.body
.contains("fetch(relayBase+'/api/v1/diagnostic',{method:'POST'")
);
assert!(
page.body
.contains("options('DELETE',token,null,headers,undefined,true)")
);
assert!(
page.body
.contains("method:'GET',signal:requestController.signal")
);
assert!(
page.body
.contains("exactKeys(value,['v','bootstrap','limits','timeouts','negotiation'])")
);
assert!(!page.body.contains("policy.carrier_method"));
assert!(page.body.contains("port.postMessage({t:'status',state})"));
assert!(!page.body.contains("port.postMessage({t:'status',state,"));
}
}
fn render_page(bootstrap: &str, candidate_count: usize) -> BridgePage {
render(
"proxy.example.com",
@@ -20,6 +88,7 @@ fn render_page(bootstrap: &str, candidate_count: usize) -> BridgePage {
120,
0,
false,
WebCarrierMethod::Post,
&SecureRandom::new(),
)
}
@@ -44,6 +113,7 @@ fn render_diagnostic_page(bootstrap: &str) -> BridgePage {
120,
0,
true,
WebCarrierMethod::Post,
&SecureRandom::new(),
)
}
@@ -95,6 +165,7 @@ fn rendered_page_resolves_carriers_against_the_exact_base_path() {
120,
0,
true,
WebCarrierMethod::Post,
&SecureRandom::new(),
);
@@ -148,6 +219,7 @@ fn rendered_page_embeds_the_configured_bridge_timing_policy() {
119,
4,
false,
WebCarrierMethod::Post,
&SecureRandom::new(),
);
@@ -201,6 +273,7 @@ fn disabled_negotiation_does_not_arm_a_carrier_deadline() {
120,
0,
false,
WebCarrierMethod::Post,
&SecureRandom::new(),
);
assert!(page.body.contains(
+3 -3
View File
@@ -44,9 +44,9 @@ mod recovery;
mod response;
// Session creation and replacement negotiation remain separate from request routing.
mod session;
// RFC 6455 upgrade validation and carrier drivers remain isolated from HTTP routing.
#[cfg(test)]
mod tests;
// RFC 6455 upgrade validation and carrier drivers remain isolated from HTTP routing.
mod websocket;
// Enabled-debug integration coverage remains separate from carrier behavior tests.
#[cfg(test)]
@@ -389,6 +389,7 @@ async fn handle_root(
config.web.timeouts.reconnect_grace_secs,
config.web.timeouts.carrier_probe_coalesce_ms,
config.web.debug.bridge_diagnostics_enabled(),
config.web.carrier_method,
&generation.rng,
);
let mut response = full_response(StatusCode::OK, Bytes::from(page.body));
@@ -456,14 +457,13 @@ async fn handle_api(
_ => serve_decoy(request, vhost, true, &runtime).await,
}
}
async fn handle_up(
request: Request<RequestBody>,
runtime: Arc<WebProcessRuntime>,
vhost: Arc<WebRuntimeVhost>,
token_hash: crate::web::manager::TokenHash,
) -> HttpResponse {
if request.method() != Method::POST || !binary_content_type(&request) {
if !matches!(*request.method(), Method::POST | Method::PUT) || !binary_content_type(&request) {
return serve_decoy(request, vhost, true, &runtime).await;
}
let Some(sequence) = canonical_u64_header(&request, "x-up-seq").filter(|value| *value != 0)
+480
View File
@@ -0,0 +1,480 @@
use super::super::session_policy_tests::{open_keepalive, read_http_response};
use super::super::*;
use crate::config::WebCarrierMethod;
fn carrier_request(method: &str, path: &str, token: &str, headers: &str, body: &[u8]) -> Vec<u8> {
let mut bytes = format!(
"{method} {path} HTTP/1.1\r\nHost: proxy.example.com\r\nX-Forwarded-For: 192.0.2.60\r\nAuthorization: Bearer {token}\r\n{headers}Content-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
)
.into_bytes();
bytes.extend_from_slice(body);
bytes
}
async fn bridge_page(
listener: &TcpListener,
runtime: &Arc<WebProcessRuntime>,
capability: [u8; 32],
base: &str,
) -> Vec<u8> {
let encoded = base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(capability);
request(
listener,
runtime,
format!(
"GET {base}?bridge={encoded} HTTP/1.1\r\nHost: proxy.example.com\r\nX-Forwarded-For: 192.0.2.60\r\nConnection: close\r\n\r\n"
)
.into_bytes(),
)
.await
}
fn bootstrap_from(page: &[u8]) -> &str {
let (headers, body) = split_response(page);
assert!(headers.starts_with(b"HTTP/1.1 200"));
std::str::from_utf8(body)
.unwrap()
.split_once("bootstrap=\"")
.unwrap()
.1
.split_once('"')
.unwrap()
.0
}
async fn create_session(
listener: &TcpListener,
runtime: &Arc<WebProcessRuntime>,
bootstrap: &str,
base: &str,
) -> String {
let hello = frame::encode(FrameType::Hello, 0, &[1]);
let bytes = carrier_request(
"POST",
&format!("{base}api/v1/session"),
bootstrap,
"Content-Type: application/octet-stream\r\n",
&hello,
);
let response = request(listener, runtime, bytes).await;
let (headers, _) = split_response(&response);
assert!(headers.starts_with(b"HTTP/1.1 200"));
response_header(headers, "x-session-token").to_string()
}
async fn session_token(
listener: &TcpListener,
runtime: &Arc<WebProcessRuntime>,
capability: [u8; 32],
base: &str,
) -> String {
let page = bridge_page(listener, runtime, capability, base).await;
create_session(listener, runtime, bootstrap_from(&page), base).await
}
async fn assert_put_accepted(path: &str) {
for (carrier, base, method) in [WebCarrier::Https, WebCarrier::HttpsLanes]
.into_iter()
.flat_map(|carrier| {
["/", "/telegram/web/"].into_iter().flat_map(move |base| {
[WebCarrierMethod::Post, WebCarrierMethod::Put]
.map(|method| (carrier, base, method))
})
})
{
let capability = [60; 32];
let mut config = runtime_config_with_base(capability, carrier, base);
config.web.carrier_method = method;
config.web.timeouts.long_poll_secs = 0;
let generation = test_runtime_generation(1, config);
let runtime = WebProcessRuntime::start(Arc::new(ArcSwap::from(Arc::clone(&generation))));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let token = session_token(&listener, &runtime, capability, base).await;
let lane = if carrier.uses_lanes() {
"X-Lane-ID: 0\r\n"
} else {
""
};
let pong = frame::encode(FrameType::Pong, 0, &[]);
let (headers, body) = if path.ends_with("/up") {
(
format!("Content-Type: application/octet-stream\r\nX-Up-Seq: 1\r\n{lane}"),
pong.as_ref(),
)
} else {
(format!("X-Down-Cursor: 0\r\n{lane}"), &[][..])
};
let path = format!("{base}{}", path.trim_start_matches('/'));
let bytes = carrier_request("PUT", &path, &token, &headers, body);
let response = request(&listener, &runtime, bytes).await;
assert!(
response.starts_with(b"HTTP/1.1 204"),
"{carrier:?}: {}",
String::from_utf8_lossy(&response)
);
runtime.shutdown().await;
generation.stop_sessions().await;
generation.stop_background_tasks().await;
}
}
#[tokio::test]
async fn put_uplink_is_accepted_for_both_https_carriers() {
assert_put_accepted("/api/v1/up").await;
}
#[tokio::test]
async fn put_downlink_is_accepted_for_both_https_carriers() {
assert_put_accepted("/api/v1/down").await;
}
#[tokio::test]
async fn carrier_method_mixed_retries_share_sequence_and_cursor_state() {
for carrier in [WebCarrier::Https, WebCarrier::HttpsLanes] {
for method in [WebCarrierMethod::Post, WebCarrierMethod::Put] {
let capability = [61; 32];
let mut config = runtime_config(capability, carrier);
config.web.carrier_method = method;
config.web.timeouts.long_poll_secs = 0;
// Exhausted admission queues a deterministic CLOSE without starting a relay task.
config.web.limits.max_streams_global = 0;
let generation = test_runtime_generation(1, config);
let runtime =
WebProcessRuntime::start(Arc::new(ArcSwap::from(Arc::clone(&generation))));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let token = session_token(&listener, &runtime, capability, "/").await;
for (index, stream_id) in [7, 8].into_iter().enumerate() {
let lane = if carrier.uses_lanes() {
format!("X-Lane-ID: {stream_id}\r\n")
} else {
String::new()
};
let sequence = if carrier.uses_lanes() { 1 } else { index + 1 };
let cursor = if carrier.uses_lanes() { 0 } else { index };
let open = frame::encode(FrameType::Open, stream_id, &[]);
let up_headers = format!(
"Content-Type: application/octet-stream\r\nX-Up-Seq: {sequence}\r\n{lane}"
);
let methods = if index == 0 {
["POST", "PUT"]
} else {
["PUT", "POST"]
};
for verb in methods {
let bytes = carrier_request(verb, "/api/v1/up", &token, &up_headers, &open);
let response = request(&listener, &runtime, bytes).await;
let (headers, body) = split_response(&response);
assert!(headers.starts_with(b"HTTP/1.1 204"));
assert_eq!(response_header(headers, "x-up-ack"), sequence.to_string());
assert!(body.is_empty());
}
for verb in methods {
let headers = format!("X-Down-Cursor: {cursor}\r\n{lane}");
let bytes = carrier_request(verb, "/api/v1/down", &token, &headers, &[]);
let response = request(&listener, &runtime, bytes).await;
let (headers, body) = split_response(&response);
assert!(headers.starts_with(b"HTTP/1.1 200"));
assert_eq!(
response_header(headers, "x-down-cursor"),
(cursor + 1).to_string()
);
assert_eq!(body, frame::encode(FrameType::Close, stream_id, &[]));
}
let headers = format!("X-Down-Cursor: {}\r\n{lane}", cursor + 1);
let bytes = carrier_request("PUT", "/api/v1/down", &token, &headers, &[]);
let response = request(&listener, &runtime, bytes).await;
assert!(response.starts_with(b"HTTP/1.1 204"));
}
runtime.shutdown().await;
generation.stop_sessions().await;
generation.stop_background_tasks().await;
}
}
}
#[tokio::test]
async fn carrier_method_changed_body_retry_keeps_protocol_failure() {
for carrier in [WebCarrier::Https, WebCarrier::HttpsLanes] {
let capability = [62; 32];
let generation = test_runtime_generation(1, runtime_config(capability, carrier));
let runtime = WebProcessRuntime::start(Arc::new(ArcSwap::from(Arc::clone(&generation))));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let token = session_token(&listener, &runtime, capability, "/").await;
let lane = if carrier.uses_lanes() {
"X-Lane-ID: 0\r\n"
} else {
""
};
let headers = format!("Content-Type: application/octet-stream\r\nX-Up-Seq: 1\r\n{lane}");
let pong = frame::encode(FrameType::Pong, 0, &[]);
let bytes = carrier_request("POST", "/api/v1/up", &token, &headers, &pong);
let response = request(&listener, &runtime, bytes).await;
assert!(response.starts_with(b"HTTP/1.1 204"));
let mut changed = pong.to_vec();
changed.extend_from_slice(&pong);
let bytes = carrier_request("PUT", "/api/v1/up", &token, &headers, &changed);
let response = request(&listener, &runtime, bytes).await;
assert_private_decoy(&response);
runtime.shutdown().await;
generation.stop_sessions().await;
generation.stop_background_tasks().await;
}
}
fn assert_private_decoy(response: &[u8]) {
let (headers, body) = split_response(response);
assert!(headers.starts_with(b"HTTP/1.1 404"));
assert_eq!(response_header(headers, "cache-control"), "no-store");
assert_eq!(body, b"not found\n");
let headers = std::str::from_utf8(headers).unwrap().to_ascii_lowercase();
for name in ["x-up-ack:", "x-down-cursor:", "x-session-token:", "allow:"] {
assert!(!headers.contains(name));
}
}
#[tokio::test]
async fn carrier_method_put_preserves_authenticated_request_shape_checks() {
for carrier in [WebCarrier::Https, WebCarrier::HttpsLanes] {
let capability = [63; 32];
let base = "/telegram/web/";
let generation =
test_runtime_generation(1, runtime_config_with_base(capability, carrier, base));
let runtime = WebProcessRuntime::start(Arc::new(ArcSwap::from(Arc::clone(&generation))));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let token = session_token(&listener, &runtime, capability, base).await;
let pong = frame::encode(FrameType::Pong, 0, &[]);
let lane = if carrier.uses_lanes() {
"X-Lane-ID: 0\r\n"
} else {
""
};
let up = format!("{base}api/v1/up");
let down = format!("{base}api/v1/down");
let up_headers = format!("Content-Type: application/octet-stream\r\nX-Up-Seq: 1\r\n{lane}");
let down_headers = format!("X-Down-Cursor: 0\r\n{lane}");
let mut invalid = Vec::new();
for media in [
"",
"Content-Type: text/plain\r\n",
"Content-Type: application/octet-stream\r\nContent-Type: application/octet-stream\r\n",
] {
let headers = format!("{media}X-Up-Seq: 1\r\n{lane}");
invalid.push(carrier_request("PUT", &up, &token, &headers, &pong));
}
for sequence in ["0", "01", ""] {
let sequence = if sequence.is_empty() {
String::new()
} else {
format!("X-Up-Seq: {sequence}\r\n")
};
let headers = format!("Content-Type: application/octet-stream\r\n{sequence}{lane}");
invalid.push(carrier_request("PUT", &up, &token, &headers, &pong));
}
let invalid_lane = if carrier.uses_lanes() {
""
} else {
"X-Lane-ID: 0\r\n"
};
let headers =
format!("Content-Type: application/octet-stream\r\nX-Up-Seq: 1\r\n{invalid_lane}");
invalid.push(carrier_request("PUT", &up, &token, &headers, &pong));
let headers = format!("{down_headers}Content-Type: application/octet-stream\r\n");
invalid.push(carrier_request("PUT", &down, &token, &headers, &[]));
invalid.push(carrier_request("PUT", &down, &token, &down_headers, &[0]));
let headers = format!("X-Down-Cursor: 01\r\n{lane}");
invalid.push(carrier_request("PUT", &down, &token, &headers, &[]));
for verb in ["GET", "PATCH"] {
invalid.push(carrier_request(verb, &up, &token, &up_headers, &pong));
invalid.push(carrier_request(verb, &down, &token, &down_headers, &[]));
}
for path in [
"/api/v1/up",
"/telegram/web/api/v1/up/",
"/telegram/web/api/v1/up?q=1",
"/telegram/web/api/v1//up",
"/telegram/web/api/v1/%75p",
] {
invalid.push(carrier_request("PUT", path, &token, &up_headers, &pong));
}
let wrong_host = String::from_utf8(carrier_request("PUT", &up, &token, &up_headers, &pong))
.unwrap()
.replace("Host: proxy.example.com", "Host: other.example.com")
.into_bytes();
invalid.push(wrong_host);
let path = format!("{base}api/v1/session");
invalid.push(carrier_request("PUT", &path, &token, &up_headers, &pong));
for bytes in invalid {
let response = request(&listener, &runtime, bytes).await;
assert_private_decoy(&response);
}
// Rejected shapes must not consume the first sequence or close the valid session.
let bytes = carrier_request("PUT", &up, &token, &up_headers, &pong);
let response = request(&listener, &runtime, bytes).await;
assert!(response.starts_with(b"HTTP/1.1 204"));
runtime.shutdown().await;
generation.stop_sessions().await;
generation.stop_background_tasks().await;
}
}
#[tokio::test]
async fn carrier_method_post_and_put_reuse_one_private_http_connection() {
for carrier in [WebCarrier::Https, WebCarrier::HttpsLanes] {
let capability = [64; 32];
let mut config = runtime_config(capability, carrier);
config.web.timeouts.long_poll_secs = 0;
config.web.limits.max_streams_global = 0;
let generation = test_runtime_generation(1, config);
let runtime = WebProcessRuntime::start(Arc::new(ArcSwap::from(Arc::clone(&generation))));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let token = session_token(&listener, &runtime, capability, "/").await;
let (mut client, cancellation, task) = open_keepalive(&listener, &runtime).await;
let lane = if carrier.uses_lanes() {
"X-Lane-ID: 7\r\n"
} else {
""
};
let open = frame::encode(FrameType::Open, 7, &[]);
let close = frame::encode(FrameType::Close, 7, &[]);
for verb in ["POST", "PUT", "PUT", "POST"] {
for (path, headers, body, ack_header) in [
(
"/api/v1/up",
format!("Content-Type: application/octet-stream\r\nX-Up-Seq: 1\r\n{lane}"),
open.as_ref(),
"x-up-ack",
),
(
"/api/v1/down",
format!("X-Down-Cursor: 0\r\n{lane}"),
&[][..],
"x-down-cursor",
),
] {
let bytes = carrier_request(verb, path, &token, &headers, body);
let bytes = String::from_utf8(bytes)
.unwrap()
.replace("Connection: close\r\n", "Connection: keep-alive\r\n")
.into_bytes();
client.write_all(&bytes).await.unwrap();
let response = tokio::time::timeout(
std::time::Duration::from_secs(5),
read_http_response(&mut client),
)
.await
.unwrap();
let (headers, body) = split_response(&response);
assert_eq!(response_header(headers, ack_header), "1");
assert!(
!std::str::from_utf8(headers)
.unwrap()
.to_ascii_lowercase()
.contains("connection: close")
);
if path.ends_with("/up") {
assert!(headers.starts_with(b"HTTP/1.1 204"));
assert!(body.is_empty());
} else {
assert!(headers.starts_with(b"HTTP/1.1 200"));
assert_eq!(
response_header(headers, "content-length"),
close.len().to_string()
);
assert_eq!(body, close);
}
}
}
cancellation.cancel();
task.await.unwrap();
runtime.shutdown().await;
generation.stop_sessions().await;
generation.stop_background_tasks().await;
}
}
#[tokio::test]
async fn carrier_method_reload_preserves_old_pages_bootstraps_sessions_and_recovery() {
for carrier in [WebCarrier::Https, WebCarrier::HttpsLanes] {
let capability = [65; 32];
let mut config = runtime_config(capability, carrier);
config.web.limits.max_bootstraps_per_ip = 6;
let generation = test_runtime_generation(1, config.clone());
let active_runtime = Arc::new(ArcSwap::from(Arc::clone(&generation)));
let runtime = WebProcessRuntime::start(Arc::clone(&active_runtime));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let post_page = bridge_page(&listener, &runtime, capability, "/").await;
let old_session =
create_session(&listener, &runtime, bootstrap_from(&post_page), "/").await;
let unused_post_page = bridge_page(&listener, &runtime, capability, "/").await;
config.web.carrier_method = WebCarrierMethod::Put;
let put_generation = test_runtime_generation(2, config.clone());
active_runtime.store(Arc::clone(&put_generation));
let post_session =
create_session(&listener, &runtime, bootstrap_from(&unused_post_page), "/").await;
let put_page = bridge_page(&listener, &runtime, capability, "/").await;
assert!(String::from_utf8_lossy(&put_page).contains("const carrierMethod='PUT';"));
config.web.carrier_method = WebCarrierMethod::Post;
let rollback_generation = test_runtime_generation(3, config);
active_runtime.store(Arc::clone(&rollback_generation));
let put_session = create_session(&listener, &runtime, bootstrap_from(&put_page), "/").await;
let rollback_page = bridge_page(&listener, &runtime, capability, "/").await;
assert!(String::from_utf8_lossy(&rollback_page).contains("const carrierMethod='POST';"));
assert!(String::from_utf8_lossy(&post_page).contains("const carrierMethod='POST';"));
let lane = if carrier.uses_lanes() {
"X-Lane-ID: 0\r\n"
} else {
""
};
let headers = format!("Content-Type: application/octet-stream\r\nX-Up-Seq: 1\r\n{lane}");
let pong = frame::encode(FrameType::Pong, 0, &[]);
for token in [&old_session, &post_session, &put_session] {
for verb in ["PUT", "POST"] {
let bytes = carrier_request(verb, "/api/v1/up", token, &headers, &pong);
let response = request(&listener, &runtime, bytes).await;
assert!(response.starts_with(b"HTTP/1.1 204"));
}
}
let encoded = base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(capability);
let bytes = carrier_request(
"GET",
&format!("/?bridge={encoded}"),
&put_session,
"Accept: application/vnd.telemt.web-recovery+json\r\n",
&[],
);
let recovery = request(&listener, &runtime, bytes).await;
let (recovery_headers, body) = split_response(&recovery);
assert!(recovery_headers.starts_with(b"HTTP/1.1 200"));
assert_eq!(
response_header(recovery_headers, "content-type"),
"application/vnd.telemt.web-recovery+json"
);
let document: serde_json::Value = serde_json::from_slice(body).unwrap();
let keys: std::collections::BTreeSet<_> = document
.as_object()
.unwrap()
.keys()
.map(String::as_str)
.collect();
assert_eq!(
keys,
["v", "bootstrap", "limits", "timeouts", "negotiation"]
.into_iter()
.collect()
);
let bootstrap = document["bootstrap"].as_str().unwrap();
let recovered = create_session(&listener, &runtime, bootstrap, "/").await;
let bytes = carrier_request("PUT", "/api/v1/up", &recovered, &headers, &pong);
let response = request(&listener, &runtime, bytes).await;
assert!(response.starts_with(b"HTTP/1.1 204"));
let bytes = carrier_request("PUT", "/api/v1/up", &put_session, &headers, &pong);
let retired = request(&listener, &runtime, bytes).await;
assert_private_decoy(&retired);
runtime.shutdown().await;
for generation in [generation, put_generation, rollback_generation] {
generation.stop_sessions().await;
generation.stop_background_tasks().await;
}
}
}
+1 -1
View File
@@ -22,7 +22,7 @@ pub(super) async fn handle_down(
vhost: Arc<WebRuntimeVhost>,
token_hash: TokenHash,
) -> HttpResponse {
if request.method() != hyper::Method::POST
if !matches!(*request.method(), hyper::Method::POST | hyper::Method::PUT)
|| request.headers().contains_key(header::CONTENT_TYPE)
{
return serve_decoy(request, vhost, true, &runtime).await;
+4
View File
@@ -1,5 +1,9 @@
use super::*;
// HTTP method compatibility keeps platform workarounds outside session state.
#[path = "carrier_method_tests.rs"]
mod carrier_method_tests;
#[tokio::test]
async fn windows_restricted_webview_empty_cookie_preserves_the_carrier_flow() {
for (index, carrier) in [WebCarrier::Https, WebCarrier::HttpsLanes]
+4 -2
View File
@@ -1,6 +1,7 @@
use super::*;
async fn open_keepalive(
/// Opens one reusable private HTTP connection for integration scenarios.
pub(super) async fn open_keepalive(
listener: &TcpListener,
runtime: &Arc<WebProcessRuntime>,
) -> (TcpStream, CancellationToken, tokio::task::JoinHandle<()>) {
@@ -21,7 +22,8 @@ async fn open_keepalive(
(client.unwrap(), cancellation, task)
}
async fn read_http_response(client: &mut TcpStream) -> Vec<u8> {
/// Reads one length-delimited response without waiting for connection closure.
pub(super) async fn read_http_response(client: &mut TcpStream) -> Vec<u8> {
let mut response = Vec::new();
while !response.ends_with(b"\r\n\r\n") {
assert!(response.len() < 16 * 1024);